In this blog post I'm attempting to take High Fidelity JSON (HF-JSON) out of the Merg-E language spec and define a programming-language independent version of HF-JSON: HF-JSON 2.0. While serializing Merg-E DAGs and Data-Frames remains a target application of HF-JSON, in this spec we will try to make it compatible with a wider range of programming languages.
It is important to note that while HF-JSON 2.0 allows different languages to use HF-JSON, that does not mean that all HF-JSON 2.0 data by one language will always be parsable by any other language. HF-JSON is a type-rich data format definition, and if language A is used to output data of a given type and the second language has no types to import that data, then the second language can not read it.
(If you prefer a less accurate but more accessible video rather than a long read, there is a notebookLM video available here )
JSON is a simple and powerful data serialization format that is used widely as a flexible human-readable data exchange format. It has lists, dictionaries, strings and numbers. Because it is based on the JavaScript type-system, its main problem when used by other languages that aren't JavaScript is that the numeric part of this program's type-system isn't expressible in JSON. Basically JSON has only one numeral type, and it is float64. JSON deceptively allows numbers to be written as integer literals, so people may assume that JSON has integer support, but it does not. You can accurately express a 54 bit signed integer or a 53 bit unsigned integer value in a JSON float64 number, but as soon as you think you can put a 64 bit for example cryptographic number in there, you are out of luck. Next to that if you use a float64 to store for example a 16 bit unsigned integer, that original type information is lost, and the receiver of the data will need to rely on convention between the producer and the consumer to know that that number in that specific place in the JSON data came from a 16 bit unsigned integer and should fit back into a 16 bit unsigned integer.
When working on the least-authority Web 3.0 domain specific language Merg-E, a language with a very rich numeric type system, JSON is used in many places. As intermediate parse tree format in the compiler toolchain, but also for snapshotting scheduling-tree state, and these JSON files need to maintain the numeric type richness of the Merg-E language itself. HF-JSON 1.0 was defined as a Merg-E centered data serialization format. Things that needed to be possible in Merg-E needed to be expressible in HF-JSON, things that were illegal in Merg-E were not expressible in valid HF-JSON.
We made the decision to NOT make HF-JSON 2.0 backward compatible with HF-JSON 1.0. The 1.0 version so far was only used by incomplete versions of the Merg-E compiler toolchain, so nobody except for the author currently depends on HF-JSON 1.0. That means that in this document we define HF-JSON 2.0 without accepting any weight from the first implementation.
Any HFJSON 2.0 document is a simple three key object. It should define HFJSON, VERSION and CONSTRAINS, and no other top level key is ever valid when VERSION is set to [2,0] (major and minor version), and currently only [2.0] is a valid value for that field.
{
"HFJSON" : {
...
},
"VERSION": [2,0],
"CONSTRAINTS": {
...
}
}
Let's look at the top level structure.
HFJSON: The HFJSON key is the main format fingerprint, the object body contains the deeper document meta data and data.VERSION: This field defines the specific version of the HF-JSON spec used in the creation of this document.CONSTRAINTS: The object body with this field defines document constraints and type-set usage this document promises to adhere to. Inspecting this allows an HF-JSON library to check if a document is type-system compatible.Within the HFJSON body, the NS value defines a namespace that the HF-JSON parser will need to look for throughout the document. Unless the HF-JSON producer detects a prefix collision in any part of the document, the namespace should always be set to HF. If however a prefix collision of the namespace and a colon is detected, then the HF-JSON producer should switch to an alternate non-coliding namespace. A namespace should be two up to upper case ascii letters
{
"HFJSON": {
"NS" : "HF",
"DOCTYPE": "COLS",
"BODY": {
"name": ["Rob", "Борис"],
"age": {
"HF:TYPE": "int{8,10}",
"HF:VAL": ["55", "29"]
}
}
},
"VERSION": [2,0],
"CONSTRAINTS": {
"types": {
"int": [8],
"unit": [false]
},
"tensor": [
[0,false,"int"]
]
}
}
HF-JSON defines three types of documents that can be set by DOCTYPE.
ROWS: Row oriented dataframe structured dataCOLS: Column oriented dataframe structured dataDAG: Tree- or forest-supported DAG structured dataROWS data is meant for easy porting of REST-like APIs, it is not recommended for data-intensive workflows:
{
"HFJSON": {
"DOCTYPE" : "ROWS",
"NS": "HF",
"BODY": [
{ "name" : "Rob", "age" : "HF:int{8}55"},
{ "name" : "Борис", "age": "HF:int{8}29"}
]
},
"VERSION": [2,0],
"CONSTRAINTS": {
"types": {
"int": [8],
"unit": [false]
},
"tensor": [
[0,false,"int"]
]
}
}
Note that field names in the actual data, as well as type information is duplicated.
More data oriented API's and streaming interfaces, as well as most Web 3.0 on-chain documents should likely use COLS documents for tabular data exchange.
{
"HFJSON": {
"NS" : "HF",
"DOCTYPE": "COLS",
"BODY": {
"name": ["Rob", "Борис"],
"age": {
"HF:TYPE": "int{8,10}",
"HF:VAL": ["55", "29"]
}
}
},
"VERSION": [2,0],
"CONSTRAINTS": {
"types": {
"int": [8],
"unit": [false]
},
"tensor": [
[0,false,"int"]
]
}
}
While for just two rows it might not be obvious, note that field names and type info of the actual data isn't duplicated.
And finally the DAG data structure, consisting of a tree and forest used as anchor for a more general DAG, using non-tree edges.
{
"HFJSON": {
"DOCTYPE": "DAG",
"NS": "HF",
"BODY": {
"owner": "Борис",
"exports": {
"counter": "HF:int{16}1966"
},
"cnt": "HF:link{}exports.counter"
}
},
"VERSION": [2,0],
"CONSTRAINTS": {
"types": {
"int": [8],
"unit": [false]
},
"tensor": [
[0,false,"int"]
],
"link": [
"sibling",
"sibling-descendant"
]
}
}
Let's look deeper at the CONSTRAINTS example from the previous section:
"CONSTRAINTS": {
"types": {
"int": [8]
},
"tensor": [
[0,false,"int"]
],
"link": [
"sibling",
"sibling-descendant"
]
}
The first part makes a promise in the envelope about what scalar types the document promises to limit itself to. In this case only 8 bit signed integers.
In order for a HF-JSON library to quickly assert if it parse an HF-JSON document or if the programing language is unable to match the input type in its type system, the types field in CONSTRAINTS contains a dictionary of lists of types used within the BODY. If for example a C++ library only supports signed and unsigned integers of up to 128 bits, it can immediately reject the following:
"types" : {
"int": [8,16,1024],
"whole": [64,4096]
}
So what keys does HF-JSON have as type family names?
For one we have the scalars:
Then we have tensors
And finally, both scalars and tensors can have units
So far for types constraints where tensors, units and specific scalar types are all separate. In the tensor constraints we see how the tree are combined in the data:
"tensor": [
[0,false,"int"]
],
It is a list of lists. Each list consists of three elements:
This line basically says: "There are individual integer scalars in the data that lack unit metadata".
The important thing to realize here is that a scalar and a rank-0 tensor are the same thing.
A third constraint we can define is the link constraint.
"link": [
"sibling",
"sibling-descendant"
]
To understand this we must think of the DAG document type and what these actually are. A DAG is meant to be a Directional Acyclic Graph, and a subset of DAGs is the tree-graph. With HF-JSON a DAG starts out as either a tree or a forest depending on the value type of the BODY of the DAG document. An object implies a tree and a list implies a forest.
The tree or forest can be expanded upon with named links that extend on the graph. But there are different types of links and link types one consumer can accept may be invalid for another consumer.
Note that without any constraints, it is possible to construct graphs with these that aren't acyclic. HF-JSON does not prevent this. While for Merg-E the stricter link constraints guarantee DAG, there are other ways for an implementation to guarantee DAG, by defining loose constraints in their DAC file, serializers promise they will use these other measures prior to serialization and parsers promise they will validate this.
Different programming languages have different type systems that each come with their own compatibility concerns for data exchange. The parser/serializer may come with its own quirks also. We consider any HF-JSON data solution to live in an ecosystem of different programming languages and libraries or frameworks. Because our data files contain a CONSTRAINTS section, we already have most of the tools to make different parts of the ecosystem play nice together.To do these every HF-JSON parser/serializer should distribute with an ECO file:
{
"HFECO" : {
"ID": ["merg-e"],
"types": {
"whole": [8,16,32,64,128,256,512,1024,4096,8192,16384],
"int": [8,16,32,64,128,256,512,1024,4096,8192,16384],
"rational": [16,32,64,128,256,512,1024,4096,8192,16384,32768],
"lfreal": [16,32,64,128,256],
"gaussian": [16,32,64,128,256,512,1024,4096,8192,16384,32768],
"grational": [32,64,128,256,512,1024,4096,8192,16384,32768,65536],
"complex": [32,64,128,256,512],
"unit": [false,true],
"tensor": [0,1,2,3,4,5],
"system": ["SI", "PLANCK", "DEMOTE"],
"frame": ["universal", "local", "noframe"]
},
"tensor": [
[null,false,"whole"],
[null,false,"int"],
[null,false,"rational"],
[null,false,"gaussian"],
[null,false,"grational"],
[null,true,"lfreal"],
[null,true,"complex"]
],
"link": [
"sibling",
"sibling-descendant"
]
},
"VERSION": [2,0]
}
The above is an ECO file for version 0,4 of the Merg-E language. An ECO file for a C++ library will look distinctly different, but the idea is that by combining the ECO files of the entire ecosystem and consolidating them, a max-overlap data infrastructure can be constructed for HF-JSON. That way an HF-JSON generator in Merg-E writing data meant for both a C++ and a Python HF-JSON parser will not be able to produce HF-JSON that either C++ or Python won't be able to map to its type system.
Note that while the above merg-e example has a wide range of numeric widths and units support, tensors are limited to max rank-5, and floating point numbers and complex numbers require units while other numeric types don't allow units. Further the graph constraints are very strict for Merg-E.
In a fictitious C++ library example below, the bitwidths are way more limited, units are not supported, graph constraints are fully flexible.
{
"HFECO" : {
"ID": ["cpphfjsonlib"],
"types": {
"whole": [8,16,32,64,128],
"int": [8,16,32,64,128],
"rational": [16,32,64,128,256],
"lfreal": [16,32,64,128],
"gaussian": [16,32,64,128,256],
"grational": [32,64,128,256,512],
"complex": [32,64,128,256],
"unit": [false],
"tensor": [null],
},
"tensor": [
[null,false,"whole"],
[null,false,"int"],
[null,false,"rational"],
[null,false,"gaussian"],
[null,false,"grational"],
[null,false,"lfreal"],
[null,false,"complex"],
],
"link": [
"sibling",
"sibling-descendant",
"parent",
"parent-sibling",
"nonroot-ancestor",
"treeroot",
"othertree-root",
"othertree-nonroot"
]
},
"VERSION": [2,0]
}
So what happens when we pull both ECO files through the tooling? The result is the most permissive constraints that still matches both the original ECO files:
{
"HFECO" : {
"ID": ["merg-e", "cpphfjsonlib"],
"types": {
"whole": [8,16,32,64,128],
"int": [8,16,32,64,128],
"rational": [16,32,64,128,256],
"lfreal": [],
"gaussian": [16,32,64,128,256],
"grational": [32,64,128,256,512],
"complex": [],
"unit": [false],
"tensor": [0,1,2,3,4,5],
},
"tensor": [
[null,false,"whole"],
[null,false,"int"],
[null,false,"rational"],
[null,false,"gaussian"],
[null,false,"grational"],
],
"link": [
"sibling",
"sibling-descendant"
]
},
"VERSION": [2,0]
}
Note that there are no lfreal and complex family types in the result because Merg-E requires lfreals and complex family types to have units and the fictional c++ lib doesn't support any units. For the rest, the strict DAG rules for Merg-E limit the link constraints to those of Merg-E, while the limited numerics bitwidths limits the core numeric types constraints to these bitwiths supported by the C++ type system.
We already showed examples of three document types, and we've discussed most of what was shown in these samples, but let's now zoom in to the "BODY" part of these three examples.
Let's start with a DAG body. This is most like a regular JSON body, mostly free structured.
"BODY": {
"owner": "Борис",
"exports": {
"counter": "HF:int{16}1966"
},
"cnt": "HF:link{}exports.counter"
}
}
We see the exports/counter key and the cnt key references a special string. We see the string starts with the HF: namespace indicator, telling the parser that this is a HF-JSON value. The above is the short form that is only available for unit-less rank-0 tensors and special nodes like the above use of a link. The long version with exactly the same info looks like this:
"BODY": {
"owner": "Борис",
"exports": {
"counter": {
"HF:TYPE": "int{16}",
"HF:VAL": "1966"
},
"cnt": {
"HF:TYPE": "link{}",
"HF:VAL": "exports.counter"
}
}
The HF:TYPE denotes the type of the value in a so called type-description sting, containing both type and serialization info, and HF:VAL holds the serialized value. In this case a decimal string representation of the number 1966 and the simple string "exports.counter".
Other than a single tree, a DAG document may contain a forest. Let's demonstrate with a one-tree forest:
"BODY": [
{
"owner": "Борис",
"exports": {
"counter": "HF:int{16}1966"
},
"cnt": "HF:link{}exports.counter"
}
}
]
Note that this looks almost identical to the first version, but note the body is a list now, not an object.
For a ROWS based dataframe body, the body content is less free:
"BODY": [
{ "name" : "Rob", "age" : "HF:int{8}55"},
{ "name" : "Борис", "age": "HF:int{8}29"}
]
A ROWS body is always a list, and the keys in each row must be identical. Next to this type information is repeated every single row.
When the number of rors is big, this is very inefficient, and using a COLS document instead is better:
A COLS based dataframe body:
"BODY": {
"name": ["Rob", "Борис"],
"age": {
"HF:TYPE": "int{8,10,10}",
"HF:VAL": ["55", "29"]
}
}
In this body type, each column has a name (the key) and the value is the whole column. In this sample, for name this is a simple list of strings. For age we have a column object with a HF:TYPE describing the type of all the elements in the column. HF:VAL contains the list with all of the column values.
Finally it is important to note that a dataframe (either in COLS or ROWS format) can be a leaf in a DAG document, but not the other way around.
HF-JSON defines three exact atomic scalars. whole for whole numbers, int for integers and lfreal for real numbers (specifically low-fidelity real numbers aka floating point numbers). The reason we refer to these numbers as lfreal and not just real is the fact that we use floating point numbers to express them and the set of real numbers includes all irrational numbers, numbers that by definition become low fidelity when expressed as a floating point number, as opposed to the other atomic numeric types.
Encoding an atomic whole in HF-JSON is done by using the prefix in a string value followed by the atomic type, followed by one up to two numbers:
"HF:whole{8}29"
This string defines the number 29 as an 8 bit unsigned whole number.
"HF:whole{8,10}29"
This does the same, but is explicit over encoding the number in base 10.
"HF:whole{8,16}1d"
Other than the default base-10, we can also opt for base-8, base-16, base-32, base-58 and base-64. In this case we use base-16 or hex.
Int is almost the same as whole, but for int we can have a minus sign.
"HF:int{8}-42"
An lfreal is less human readable. We avoid the precision loss that full decimal notation would bring and instead we encode the two numbers that make up the standard internal representation of floating point representation directly.
"HF:lfreal{64}-2028m19998739"
And again we can write the long version and optionally use alternate encoding of the two parts of the number.
"HF:lfreal{64,10,10}-2028m19998739"
The two last numbers don't need to be the same. You may choose for large number types to encode the mantissa as base64 but the exponent as decimal.
We define the following composite scalars:
Two examples in HF-JSON 2.0 format. A rational:
{
"HF:TYPE": "rational{16}",
"HF:VAL": ["-17", "37"]
}
Alternative the / character may be used for inline encoding:
"HF:rational{16}-17/37"
Similar for a gaussian rational,
{
"HF:TYPE": "grational{32}",
"HF:VAL": [
["-17", "37"],
["-37", "17"]
]
}
but now we need to use the | character for the complex numbers part:
"grational{32}17/32|-37/17"
In many cases numeric values in data aren't just values, they are quantities, and quantities combine a value with some kind of unit.
A unit in itself exists inside of a system of units, mostly SI, and has a specific integer value for a small number of base units within that system.
Finally, a quantity can be absolute or relative. There are basic rules for calculating things with absolute vs relative quantities that make it important to treat these quantities distinctly. If for example you subtract two absolute values from each other you get a relative quantity, and if you add or subtract a relative quantity from an absolute quantity you get an absolute quantity. But let's look at an example quantity in HF-JSON 2.0:
{
"HF:TYPE": "whole{32}",
"HF:UNIT": {
"SYS": "SI",
"ABSOLUTE": true,
"TEMP": 1
},
"HF:VAL": "300"
}
There is no short version for values with units.
Notice we use the SI system of units. In this case we store a temperature in whole degrees Kelvin by setting ABSOLUTE to true and the SI dimension TEMP to 1.
Note that because we add a unit, we need to move from the short string only value notation HF:whole{32}300 to an object notation of the quantity.
For HF-JSON 2.0 we define two systems:
These two systems have a partially similar set of base dimensions. Similar in that they use almost the same specific dimensions, but at quite different levels of scale, using quite differently sized base units. SI base units are units chosen to be meaningful to every day life of normal humans, PLANCK units are chosen as to greatly simplify physics, basically by choosing them in such a way that the main constants of nature such as the speed of light c all have a value of one, so you can just leave them out of all of the equations.
| SI | PLANCK |
|---|---|
| TEMP | TEMP |
| LEN | LEN |
| MASS | MASS |
| TIME | TIME |
| CURRENT | |
| CHARGE | |
| SUBST | |
| INTENS |
Each of these is a valid key inside of an HF:UNIT, and the matching values are just regular JSON numbers in integer notation.
The final boss of the HF-JSON values is the tensor. One thing to realize with tensors, even if they can be very big and consist of many atopic values or even composite scalars, in essence they are a single, be it very complex value. In the below example we see a three by three rank-2 tensor of gaussian values. The TYPE field denotes the rank of the tensor. The BASE defines the elemental type the tensor is made up of. In this case 128 bit gaussian numbers. Tensors exist in a frame. Just as with units and unit systems and absolute vs relative quantities, scalars that exist in different frames aren't just interchangeable. HF-JSON currently defines a few frames:
We expect this list to grow. It is permitted to use a positive integer in the JSON high fidelity integer range, as long as the value is above 127.
The SHAPE field defines the shape of the tensor, in this case 3 by 3.
And finally we have the symmetries of the tensor. In this example we define that there is a STANDARD symmetry between axis 0 and axis 1, but other symmetries can be defined:
The sample:
{
"HF:TYPE": "tensor{2}",
"HF:TENSOR": {
"SHAPE" : [3,3],
"BASE" : "gaussian{128,10}",
"FRAME" : null,
"SYM" : {
"STANDARD": [[0,1]]
}
},
"HF:UNIT": {},
"HF:VAL": [
[
["1", "133"],
["42","69"],
["17","-1"]
],[
["42","69"],
["19", "-144"],
["18","-11"]
],
[
["17","-1"],
["18", "-11"],
["-71", 0]]
]
}
In this post we defined the 2.0 version of the High-Fidelity data-exchange format HF-JSON. HF-JSON is not backward compatible with HF-JSON 1.0 that was never used outside of the development efforts of the Merg-E least authority Web 3.0 DSL, so lack of backwards compatibility only affects my own project. HF-JSON 2.0 allows Merg-E to encode its full rich numeric type-system and DAG oriented approach to data and state in JSON. While the DAG and dataframe nature of HF-JSON might not make it full general purpose, the current HF-JSON spec should allow for type-rich encoding of data in multi-language data eco-systems. The use of ECO files allows multi-language eco-systems to find the broadest possible type-rich data communication that is allowed by the set of type-systems used.
We need to note that HF-JSON 2.0, like the 1.0 version, is still designed primaraly to scratch the itch for Merg-E development within the InnuenDo stack scope, but by making it more flexible beyond the Merg-E type system, it is no longer a Merg-E only format. Where Merg-E is highly oppinionated on things like DAG guarantees through link constraints, mandatory units for anything ℝ (float) based and prohibited units for 𝕎,ℤ andd ℚ based data, HF-JSON 2.0 offers broader flexability. This flexability might make HF-JSON a valid option in broader eco systems, including those that don't include Merg-E.