#compactness

Today, we'd like to share a simple innovation in how we structure the JSON documents that are stored in the database - we coalesce the redundant / repeated information in the JSON doc to come up with a compact JSON document that is reduced in size without losing any textual readability in the JSON doc.  Our compactness technique is somewhere between the JSON Compression techniques and JSON Normalization techniques (IMHO, JSON Compression focuses on serializing json for storage and loses text readability and JSON Normalization techniques normalize the JSON into flat dataframe structures which is somewhat different from what we are trying to achieve)

Though, the redundancy and repeatedness of the information that we've solved for is unique to our usecase, we do believe similar patterns can be used in different situations to compact JSON documents by identifying redundancy and repeatedness in these usecases as well.

We've used AWS Dynamo DB as our database - it has worked well for the usecases that we've had and made development a breeze.  We've also experimented the same JSON document storage on the Azure Cosmos DB which also has worked like a charm.

So, without further ado, lets setup our example and see how we compact the JSON document.

Usecase

Lets say we have a Songs table in the database that stores the details about the different Songs that the users have played / are available in the library.

Our app is available in multiple marketplaces (US, Canada, UK etc) and our JSON document stores the song details for each of these marketplaces in the same JSON document (simplifies cross region song sharing usecases - a consumer in US shares a song with a friend in the UK).

From experience, we've observed that for a large majority of the songs, the song attributes do not differ between the different marketplaces. In some cases the song attributes do differ in different marketplaces. Lets suppose we have the following Song JSON document with a single albumID attribute that is a string type and we have 7 marketplaces. In the example below, marketplace1-4  have same album Id (AlbumID1) and marketplace5-7 have a different  album Id (AlbumID2).

{    "songID":  <system generated PartitionKey unique across all marketplaces>    "albumID":    {        "<marketplace1>":  "<albumID1>",        "<marketplace2>":  "<albumID1>",        "<marketplace3>":  "<albumID1>",        "<marketplace4>":  "<albumID1>",

       "<marketplace5>":  "<albumID2>",        "<marketplace6>":  "<albumID2>",        "<marketplace7>":  "<albumID2>"    } }

We compact the Song JSON document's albumID attribute by defining a defaultSet that is the largest set of marketplaces with the same value and then save the defaultSet and defaultValue for these marketplaces. The marketplaces that are are different are kept as is. The json doc with the compacted albumID attribute would look like the following:

{    "songID":  <system generated PartitionKey unique across all marketplaces>    "albumID":    {        "defaultSet":        [            "<marketplace1>", "<marketplace2>", "<marketplace3>", "<marketplace4>"        ],        "defaultValue": "<albumID1>",        "<marketplace5>":  "<albumID2>",        "<marketplace6>":  "<albumID2>",        "<marketplace7>":  "<albumID2>"    } }

There is an additional optimization that could be made - defining ranked default sets with the repeat frequency - but we did not feel the need to implement these.

{    "songID":  <system generated PartitionKey unique across all marketplaces>    "albumID":    {        "defaultSet1":        [            "<marketplace1>", "<marketplace2>", "<marketplace3>", "<marketplace4>"        ],        "defaultValue1": "<albumID1>",        "defaultSet2":        [            "<marketplace5>", "<marketplace6>", "<marketplace7>"        ],        "defaultValue2": "<albumID2>"    } }

Performance

Our app is available in 32 marketplaces and each song has approximately 16 attributes that are to be duplicated for each marketplace, the average document sizes for each song JSON document came out to ~ 405KB (song contains a bunch of song metadata specific to each marketplace etc). With the JSON Text Compactness algo, we reduced the JSON doc size to ~64 KB without any noticeable impact in serialization / deserialization performance.

Implementation

In terms of implementation, we defined a java class with the following interface (this is for storing String value type):

class StringStringDefaultMap {    

  // Serialize constructor that serializes a <marketplace, albumId> map to a compact representation   public StringStringDefaultMap(CaseInsensitiveMap<String, String> valueMap, Context context);

  // Deserialize constructor that deserializes a dynamo db attribute value compacted map to a <marketplace, albumId> map   public StringStringDefaultMap(AttributeValue stringAttributeValueMapAttrVal, Context context);

  // Get an attribute value for the compacted map represented by this class - it will create the JSON document per the DynamoDB JSON format where type is encoded into the json   public AttributeValue getMapAttributeValue(Context context);  

  // Get the <marketplaceId, albumId> value map represented by this class   public CaseInsensitiveMap<String, String> getValueMap();

  // Get the default <marketplace, albumId> key value pairs - albumId should be the same for all the marketplaces in this map since its the default map   public CaseInsensitiveMap<String, String> getDefaultMap()

  // Get the default keyset for the compacted map   public Set getDefaultKeySet(); }

The implementation for the compacted attributes for AWS Dynamo DB encodes the type into the JSON document as required by the DynamoDB JSON format. This implementation also reuses the Dynamo DB String Set type to store the defaultSet. here is what the above example would look like:

{  "songID": {    "S": "<system generated PartitionKey unique across all marketplaces> "  }  "albumID":  {    "M": {      "defaultSet1": {        "SS":        [          "<marketplace1>", "<marketplace2>", "<marketplace3>", "<marketplace4>"        ],      }      "defaultValue1": {        "S": "<albumID1>",      }      "<marketplace5>":  {        "S": "<albumID2>",      }      "<marketplace6>":  {        "S": "<albumID2>",      }      "<marketplace7>":  {        "S": "<albumID2>"      }    } }

In this example, the album ID value type String so we used a StringStringDefaultMap. There are also implementations for the different types that DynamoDB supports such as

<String marketplace, Integer attribute value> // Dynamo DB's number attribute type <String marketplace, List<Map<String, Object>>> // Dynamo DB's List Attribute Type where each list element is map attribute type <String marketplace, List<String>> // List<String> attribute type <String marketplace, List<Object>> // List<Object> attribute type <String marketplace, Set<String>> // String Set attribute type <String marketplace, Map<String,String>> // Map<String, String> attribute type

Another trick we had to implement is to replace marketplace specific references in an attribute value with a placeholder for storage in the database and resolve these at deserialization time. For example, the following data urls were coalesced to a placeholder data url:

https://www.letsresonate.net/us/albums?id=<albumId1> https://www.letsresonate.net/ca/albums?id=<albumId1> https://www.letsresonate.net/uk/albums?id=<albumId1> => https://www.letsresonate.net/{PLACE_HOLDER}/albums?id=<albumId1>

Usage

The code is shared in the repository: https://github.com/resonancedeveloper/JSONTextCompactness

The Tests in the TestDefaultMaps demonstrate on how to use each of the default map classes and serialize / deserialize into dynamo db json. Here is one such example:

   @Test    public void testStringStringDefaultMap() {

       // test default set        int defaultVal = 100;        int defaultThreshold = 10;        Set<String> defaultSet = new HashSet<>();        Set<String> countryCode = new HashSet<>();        CaseInsensitiveMap<String, String> countryCodeValueMap = new CaseInsensitiveMap<>();        for (int i = 0 ; i < 30 ; i++) {            countryCode.add("cc"+i);            if (i < defaultThreshold) {                countryCodeValueMap.put("cc" + i, "value" + defaultVal);                defaultSet.add("cc" + i);            } else {                countryCodeValueMap.put("cc" + i, "value" + i);            }        }

       Map<String, AttributeValue> attrValMap = new HashMap<>();        attrValMap.put("defaultSet", new AttributeValue().withSS(defaultSet));        attrValMap.put("default", new AttributeValue().withS("value"+defaultVal));        for (int i = 0 ; i < 30 ; i++) {            if (defaultSet.contains("cc"+i)) {                continue;            }            attrValMap.put("cc"+i, new AttributeValue().withS("value"+i));        }

       // serialize        Assert.assertEquals(new AttributeValue().withM(attrValMap), new StringStringDefaultMap(countryCodeValueMap, context).getMapAttributeValue(context));

You can get the MathJax to render if you go to the house-carpenter.tumblr.com subdomain, rather than using the dashboard.

Definition (Languages). A language is a set of objects (none equal to 0) regarded as the propositional variables of the language. For every language L, the sentences of L are defined recursively by the rules below:

Every propositional variable of L is a sentence of L.

0, which in this context we call the bottom and write as ⊥, is a sentence of L.

For every pair of sentences P and Q of L, the ordered pair (P, Q), which in this context we call the conditional with antedecent P and succedent Q, and write as P → Q, is a sentence of L.

Definition (Interpretations). For every language L, an interpretation of L is a function from L to {0, 1}. An interpretation i of L can be extended to the set of the sentences of L recursively by the rules below:

i(⊥) = 0.

For every pair of sentences P and Q of L, the value i(P → Q) is 0 if i(P) = 1 and i(Q) = 0, otherwise 1.

A sentence P of L is true under i iff i(P) = 1 and false under i iff i(P) = 0.

Definition (Tautologies). For every language L, a tautology of L is a sentence of L which is true under every interpretation of L.

Definition (Theories). For every language L, a theory of L is a set of sentences of L regarded as the axioms of the theory. The subtheories of a theory are its subsets.

Definition (Models). For every language L and every theory T of L, a model of T is an interpretation of L under which every axiom of T is true.

Theorem (Compactness Theorem). For every language L and every theory T of L, if every finite subtheory of T has a model, then so does T itself.

Proof. Let $\mathcal I$ be the Lth power of the discrete topology on {0, 1}. This topology is compact by Tychonoff’s theorem, since the discrete topology on {0, 1} (or any other finite set) is compact. This means that for every family $\mathcal J$ of closed sets in $\mathcal I$, if the intersection of every finite subfamily of $\mathcal J$ is nonempty, then the intersection of $\mathcal J$ itself is nonempty.

For every set I of interpretations of L, let L(I) be the set of the propositional variables A of L such that {i(A) : i ∈ I} = {0, 1}, i.e. there is an i ∈ I under which A is true and a i ∈ I under which A is false. By the definition of the product topology, the open sets in $\mathcal I$ are the unions of sets I of interpretations of L with L(I) cofinite in L. The closed sets in $\mathcal I$ are the complements of these sets.

For every axiom P of T, let I(P) be the set of the models of {P}, i.e. the interpretations I of L under which P is true. We shall show that I(P) is always closed in $\mathcal I$.

First of all, note if P is a tautology of L, then $I(P) = \{0, 1\}^L$, so I(P) is closed in $\mathcal I$ because the whole underlying set of a topology is always closed. So we can restrict our attention to the case where P is not a tautology of L.

Let $I = \{0, 1\}^L \setminus I(P)$. Then I is the set of the interpretations i of L under which P is false. We need to prove that I is open in $\mathcal I$. It will suffice to show that L(I) is cofinite in L. To accomplish this, we shall prove that every propositional variable of L not occurring in P is in L(I); since there are only finitely many propositional variables of L occurring in P, this will suffice.

Suppose A is a propositional variable of L not occurring in P.

First, we seek an i ∈ I under which A is false. Simply take i = ∅. Then A is false under i. Furthermore, P is false under i, because otherwise P would be a tautology of L.

Second, we seek an i ∈ I under which A is true. We know that P is not a tautology, so there is an i ∈ I under which P is false. Let j = i ∪ {A}. Then A is true under j. Moreover, j(P) = i(P), because all the propositional variables occurring in P have the same truth value under both i and j. Therefore P is false under j.

This completes the proof that I(P) is closed in $\mathcal I$.

Now, let $\mathcal J$ be the family of the sets of the form I(P), where P is an axiom of T. This is a family of closed sets in $\mathcal I$. For every subfamily $\mathcal K$ of $\mathcal J$, there is a subtheory U of T such that $\mathcal K = \{I(P): P \in U\}$. So the intersection of $\mathcal K$ is the intersection of the sets of the form I(P), where P is an axiom of U. This will be nonempty iff there is an interpretation of I under which every axiom of U is true, i.e. U has a model.

B.Sc Tuition In Noida For Real Analysis

B.Sc Tuition In Noida For Real Analysis

Call CFA Academy For B.Sc Maths Tuition Classes In Noida. Linear Algebra Tuition In Noida, Calculus Tuition In Noida, Real Analysis Tuition In Noida, Complex Analysis Tuition In Noida, Ordinary Differential Equations Tuition In Noida, Algebra Tuition In Noida, Functional Analysis Tuition In Noida, Numerical Analysis Tuition In Noida, Partial Differential…