JEM (JSON with Embedded Metadata) - A Simpler Alternative to JSON Schema?

I've long been a supporter of the JSON Schema initiative, and I was also happy to see developments like Orderly, which offered a simpler and less verbose format than JSON Schema. But Orderly introduces its own format, which necessitates conversion to JSON Schema before it can be used. Both approaches are unsatisfactory in their own way. One is too verbose and the other needs translation.

All of this made me wonder if we aren't approaching the problem the wrong way. JSON Schema is a conscious effort to replicate in the JSON world the descriptive capability that XML Schema brings to XML. But is this the best way to go about it?

I would like descriptive metadata about documents to be capable of being embedded inside the document itself, rather like annotations in Java programs. Indeed, this metadata should be capable of forming a "scaffold" around the data that then allows the data itself to be stripped out, leaving behind a template or schema for other data instances.

So I'm proposing something that I think is a whole lot simpler. It does require one fundamental naming convention to be followed, and that is this:

Any attribute name that begins with an underscore is metadata. Everything else is data.

Let's take this simple JSON document:

We can embed metadata about this document in two different ways. Click diagram to expand.

I'm calling the first style "Metadata Markup", where the data elements of the JSON document retain their primacy, and the metadata around them is secondary and serves to add more detail to these data elements. One can readily see that "_value" is now just one of the possible attributes of an element, and many more such attributes can therefore be added at will.

I call the second style "Metadata Description", where the primary elements are metadata, and any data elements (whether keys or values) are modelled as the values of metadata elements. Note that describing a document as an array (a nested array in the general case) rather than as a dictionary (or nested dictionary) of elements allows the default order of the elements to be retained. This is quite useful when this format is used to publish data for human consumption.

The first style, Metadata Markup, is more suitable for document instances, because a lot of detailed meta-information can accompany a document and can be hidden or stripped out at will. It is easy for a recipient to distinguish data from metadata because of the leading underscore naming convention. There is no need to pre-negotiate a dictionary of metadata elements. (Click to expand.)

The second style, Metadata Description, is more suitable for schemas, because in this format, all elements pertaining to instance data (both keys and values) are just values. If only the values representing keys are retained, we get a "scaffold" structure describing the document, and more metadata elements representing constraints can be added, turning it into a schema definition. (Click to expand.)

Obviously, this system will not work for everyone. I'm sure there are JSON documents out there that have underscores for regular data (HAL?), so adoption of this convention won't be feasible in such domains. But if a significant subset of the JSON-using crowd finds value in this approach, they're more than welcome to adopt it.

The Coming Overthrow of XML - Orderly Makes Further Strides

My feeling that XML is due to be dethroned grows stronger by the month.

A quick recap of recent history:

First, JSON offered a simpler data structure than the angle bracketed format of Unicode XML. But that still lacked rigour around data definition, so even though XML suffered the confusion of having at least three competing schema definition languages (XML Schema, RelaxNG and Schematron), the world did have a way to specify data types, formats and constraints with XML that JSON could not match.

Then, thanks to Kris Zyp, JSON Schema appeared on the scene and plugged the rigour gap. JSON Schema parsers now exist for a number of languages including Java. One of the design decisions of JSON Schema was for a schema document itself to be valid JSON, much as XML Schema is itself valid XML. Unfortunately, this meant that brevity wasn't JSON Schema's strong point because the JSON way of expressing properties is necessarily long-winded.

The third shoe has dropped now (never mind the grotesque image that conjures up of the wearer). Orderly is a new schema language developed by Lloyd Hilaiel that is far more compact than JSON Schema and yet round-trips to JSON Schema quite effectively. [There's a really cool Ajax-y screen that converts back and forth between Orderly and JSON Schema before your eyes, so you can tweak either code to see how it looks in the other representation.]

Bottomline: SOA architects can recommend the use of the simpler JSON data format instead of XML without having to worry about the lack of rigour in data definition. Data architects, designers and developers can use Orderly to design schemas without bothering about JSON Schema's cumbersome syntax. JSON parsers can work with the equivalent JSON Schema to validate a piece of JSON data without the need to understand two different syntaxes.

A great solution, and it's all come together quite nicely in time for Christmas. Thanks to Lloyd (and Kris before him) for a wonderful Christmas present to all SOA practitioners, and ultimately, everyone wrestling with XML in any capacity.

JSON Schema becomes more Orderly

I have been convinced for a while that just like REST will gradually displace its more heavyweight SOAP/WS-* equivalent, JSON will slowly displace the mighty XML in its various strongholds (today Web Services, tomorrow the world :-). But to do that, JSON first needs to incorporate some rigour into its definition, using an equivalent to XML Schema, Relax NG or Schematron.

The JSON Schema proposal seemed to fit the bill quite nicely, but I was always vaguely uneasy that it was so verbose. There was probably no escape from that, since one of the requirements was that JSON Schema should itself be valid JSON (otherwise two parsers would be needed to consume a snippet of schema-compliant JSON).

Now along comes another schema syntax for JSON called Orderly, which has the twin advantages of being succinct and being able to round-trip to JSON Schema. The syntax has already been revised with inputs from commenters, and is looking much better in its second version.

Orderly's main advantage is its human-readability and -composability. Its simplicity (with no loss of rigour) will give JSON (and JSON Schema) the impetus they need to challenge XML. If Orderly catches fire, I believe it will accelerate the adoption of JSON for serious service-oriented work.

It's overdue.