The Coming Overthrow of XML - Orderly Makes Further Strides

My feeling that XML is due to be dethroned grows stronger by the month.

A quick recap of recent history:

First, JSON offered a simpler data structure than the angle bracketed format of Unicode XML. But that still lacked rigour around data definition, so even though XML suffered the confusion of having at least three competing schema definition languages (XML Schema, RelaxNG and Schematron), the world did have a way to specify data types, formats and constraints with XML that JSON could not match.

Then, thanks to Kris Zyp, JSON Schema appeared on the scene and plugged the rigour gap. JSON Schema parsers now exist for a number of languages including Java. One of the design decisions of JSON Schema was for a schema document itself to be valid JSON, much as XML Schema is itself valid XML. Unfortunately, this meant that brevity wasn't JSON Schema's strong point because the JSON way of expressing properties is necessarily long-winded.

The third shoe has dropped now (never mind the grotesque image that conjures up of the wearer). Orderly is a new schema language developed by Lloyd Hilaiel that is far more compact than JSON Schema and yet round-trips to JSON Schema quite effectively. [There's a really cool Ajax-y screen that converts back and forth between Orderly and JSON Schema before your eyes, so you can tweak either code to see how it looks in the other representation.]

Bottomline: SOA architects can recommend the use of the simpler JSON data format instead of XML without having to worry about the lack of rigour in data definition. Data architects, designers and developers can use Orderly to design schemas without bothering about JSON Schema's cumbersome syntax. JSON parsers can work with the equivalent JSON Schema to validate a piece of JSON data without the need to understand two different syntaxes.

A great solution, and it's all come together quite nicely in time for Christmas. Thanks to Lloyd (and Kris before him) for a wonderful Christmas present to all SOA practitioners, and ultimately, everyone wrestling with XML in any capacity.

JSON Schema becomes more Orderly

I have been convinced for a while that just like REST will gradually displace its more heavyweight SOAP/WS-* equivalent, JSON will slowly displace the mighty XML in its various strongholds (today Web Services, tomorrow the world :-). But to do that, JSON first needs to incorporate some rigour into its definition, using an equivalent to XML Schema, Relax NG or Schematron.

The JSON Schema proposal seemed to fit the bill quite nicely, but I was always vaguely uneasy that it was so verbose. There was probably no escape from that, since one of the requirements was that JSON Schema should itself be valid JSON (otherwise two parsers would be needed to consume a snippet of schema-compliant JSON).

Now along comes another schema syntax for JSON called Orderly, which has the twin advantages of being succinct and being able to round-trip to JSON Schema. The syntax has already been revised with inputs from commenters, and is looking much better in its second version.

Orderly's main advantage is its human-readability and -composability. Its simplicity (with no loss of rigour) will give JSON (and JSON Schema) the impetus they need to challenge XML. If Orderly catches fire, I believe it will accelerate the adoption of JSON for serious service-oriented work.

It's overdue.

No, Really. What is SOAP? What is BPEL?

Sometimes the answers to simple questions are profound.

I have been asking myself a couple of very simple questions for some time now, and although I initially didn't want to accept some of the answers I got, I guess I can't avoid their implications.

What is SOAP? I'm deliberately not looking at any of the SOAP spec documents for the answer. I want to know the value proposition of SOAP.

Similarly, what is BPEL (or WS-BPEL, if you prefer)?

I can think of two quick answers:

1. The value proposition of SOAP is interoperability.
2. The value proposition of BPEL is portability.

To my mind, interoperability is at least an order of magnitude more important than portability.

SOAP is the "wire protocol" in one view of SOA. Within the "cloud" or "SOA Fabric", the only things one sees are SOAP messages flying past. Nobody sees BPEL within the SOA cloud. BPEL is an implementation language for the execution of processes, just like Java is an implementation language for general-purpose business logic. It runs on a node. All interactions between that node and others are through SOAP messages. A BPEL process consumes SOAP-based services and in turn exposes itself as a SOAP service.

So is BPEL really all that important in the scheme of things? Wouldn't any other portable language do? I think I've stumbled upon a candidate language, and it's not Java.

Lately, I've been playing around with the WSO2 Mashup Server, and I'm increasingly getting the excited feeling of a kid who's somehow come into possession of a machine-gun. This is not a toy, folks. Are the adults aware of what this thing can do?

Most people seem to think mashups are a cute visual gimmick. The WSO2 guys themselves don't have the air of people handing out machine-guns. The examples they bundle with the server are classic mashup fare - TomatoTube, in which you take the top-rated movies from Rotten Tomatoes and mash the list up with trailers from YouTube, enabling you to see the trailers of top-rated movies. Very cute and harmless.

But now for the machine-gun bit. The development language offered by Mashup Server is JavaScript (think general-purpose programming language). JavaScript augmented by E4X (think really easy XML manipulation). Mashup Server hides SOAP very effectively, although its interfaces to the outside world are SOAP (SOAP 1.1 and 1.2, also REST/HTTP, but more about that later). SOAP is out there in the cloud. But here, within this processing node, it's just XML representations of data and JavaScript code to process it with, thanks to the why-didn't-I-think-of-it simplicity of E4X. Surely we can do more than mashups with that kind of power...

I've found myself thinking a disturbing thought: If no one sees BPEL in the forest, then does it really exist? What if BPMN-based process modelling tools spat out E4X-enhanced JavaScript code instead of BPEL? Would anyone know or care? Take the output of the modelling tool and drop the file onto a server. The process is ready to run. All external interfaces are SOAP-based, just like with a BPEL implementation. Got any problems with that?

There's more revolutionary potential here than in the Communist Manifesto. Another of the cutesy examples bundled with Mashup Server is a REST service. You can do HTTP GETs, PUTs, POSTs and DELETEs on a simple city resource to manipulate its weather data. Very harmless, but again, the developer has very simple JavaScript-based access to REST services.

So is the WSO2 Mashup Server the one that will bring balance to the Force? A powerful programming language. Laughably easy XML manipulation. Simple access to SOAP services and REST resources. Transparent publication of itself as a service or resource in turn. Isn't this the holy grail of service composition?

Content management, process orchestration, what's in a name? I'm beginning to think BPEL is dead. Its value proposition doesn't stack up anymore. SOAP still makes sense, but not BPEL.

WSO2 Mashup Server seems to be the industry's best-kept secret for now. Keep the safety catch on, and watch where you point that thing.

Why RPC is Evil, XML-RPC is Doubly Evil and SOAP-RPC is Triply Evil

After reading my own review of Yuli Vasiliev's book, I realised I had to better explain my "SOAP-RPC is evil" comment.

Actually, SOAP-RPC is not just simply evil but triply evil, and here's why.

The root of the whole evil was the original arrogant assumption behind the notion of distributed objects - that we can make remote objects appear as if they are local. It was only much later that Martin Fowler came out with his First Law of Distributed Objects ("Don't Distribute Your Objects").

Quick question: How can we make a remote object appear to be local?
Correct answer: We can't, period.
Naive answer: Serialise the remote object, transport it over the wire and deserialise it to create a local copy.

The reason the naive answer is so badly wrong is that copies behave very differently to the original objects. To be precise, when a copy is changed, the original is not changed. So "hiding" the remoteness of an object through some object serialisation sleight-of-hand doesn't work. The component or application that gets a reference to the copy must know that it is a copy, otherwise all kinds of application errors can occur.

So that's why RPC (Remote Procedure Call) is evil. It tries to make remote objects look local. If one is not careful, this can create serious errors in applications.

The other issue is the serialisation mechanism. Even assuming it's OK to serialise an object and recreate it elsewhere from the serialised representation, what is the mechanism used for serialisation? Java serialisation works because we have Java on both ends. In recent times, XML has become popular, and some bright spark must have thought of "XML serialisation". The concept is simple. Convert a Java object to XML format, transport it over the wire to a remote host, then convert the XML document back into a Java object to create a perfect copy, a "remote object". The problem with this assumption is that there is an impedance mismatch between Java and XML. We can do things with Java that we can't do with XML, and vice-versa.

So converting a Java object into XML isn't straightforward. Things get lost in the translation. Converting from XML back into Java isn't straightforward either. More things get lost in the translation. So this "serialisation/deserialisation" using XML as the transport format doesn't result in perfect copies at the other end. This is XML-RPC, and when it's used naively, it's easy to see why it's doubly evil.

As SOAP-RPC evolved beyond XML-RPC, the wire protocol itself began to be seen as a "contract" between systems. In other words, the SOAP message could be used to decouple implementations at either end. All of a sudden, the XML document going over the wire was not expected to be just a Java serialisation mechanism, it was a technology-neutral serialisation mechanism. So now it became fashionable to think of serialising a Java object into an XML document, wrapping it in a SOAP envelope and "exposing" this as a contract to another system. This other system could pull in the SOAP object, extract the XML document from within it, then deserialise it into...a C# object! Not only is the impedance mismatch alive and well here too, there is another subtle assumption made that is violated in the execution. Can you find it?

Let me sum up.

RPC is evil because it makes developers think they can make a remote object appear local.

XML-RPC is doubly evil because it makes developers think (i) they can make a remote object appear local and (ii) it is possible to convert a Java object to XML and back without error or ambiguity.

SOAP-RPC is triply evil because it makes developers think (i) they can make a remote object appear local, (ii) it is possible to convert a Java (or C#) object to XML and back without error or ambiguity and (iii) even though the SOAP message travelling between two systems is now a "contract" to be honoured regardless of changes to implementations, it can still be generated from implementation classes, and implementation classes can be generated from it.

What's the truth, then?

Nothing glamorous or counter-intuitive.

1. A copy is a copy, not the original.
2. XML is XML and Java is Java (and C# is C#).
3. Contracts are First Class Entities, just like Domain Objects in an OO system.

SOAP-based Web Services technology today is sadder and wiser because it has internalised these truths. Because of Truth 1, we don't pretend anymore that a SOAP message is a Remote Procedure Call. We know it's just a message. Because of Truths 2 and 3, we don't believe anymore that we can generate XML documents from Java/C# objects, or Java/C# objects from XML documents. We can only map data between these two forms. If you build your Web Services abiding by these principles, you can stay out of trouble, otherwise you will be left wondering why your systems are so brittle.

(Actually, there are still people who don't understand these Truths, and who still go on about SOAP-RPC. That's what makes me despair about SOAP's chances for success.)

SOFEA and SOUI - There is a Difference, After All

Great minds think alike, or so it would appear :-).

Just a month after we published our article "Life above the Service Tier" describing SOFEA (Service-Oriented Front-End Architecture), Nolan Wright and Jeff Haynie have proposed an architecture they call SOUI (Service-Oriented User Interface). (Independently, Roger Voss blogged that web frameworks are peaking towards obsolescence.)

It's worth repeating what we said in the Conclusion section of our paper:

"Although it seems presumptuous on our part to claim that we have “solved” the end-to-end integration problem, what is probably true is that recent paradigms and technology breakthroughs have brought the SOFEA model closer to conceptualisation, and someone or the other was bound to suggest it. It happened to be us."

Well, clearly not just us. Many others are saying the same thing.

Matt Raible wondered if there was any difference between the two frameworks (SOFEA and SOUI). SOUI proponents Nolan and Jeff haven't just proposed a theoretical architecture, they've actually created a set of tools to help developers build to this model, and what's more, this set of tools is available for Java, PHP, Ruby and .NET. It's called Appcelerator, and it's a pretty impressive piece of work.

Examining Appcelerator (and it would be a fair assumption that this is what Nolan and Jeff intend SOUI to look like), it appears to satisfy all the conditions that we proposed for SOFEA, except one.

SOFEA emphasises the use of XML for Data Interchange, because one of its guiding principles is to mesh seamlessly with the service tier. Services may be built using either the REST paradigm or the SOAP messaging paradigm, but XML plays a big role in either one. SOAP requires, and REST recommends, the use of XML for Data Interchange. XML has the essential characteristic of being able to enforce data integrity (i.e., conformance to specified data structures, data types and data constraints). That's why we make a big deal about XML support in SOFEA. We considered but rejected JSON because it's only slightly better than raw HTML-over-HTTP in these respects.

Appcelerator uses JSON, not XML. So that seems to be the big difference between the two models. SOFEA requires the use of XML for Data Interchange. SOUI prefers the more lightweight JSON.

I guess Nolan and Jeff have made a valid architectural choice favouring ease-of-use over data integrity, but better XML tooling may blunt that advantage in future.

Incidentally, Ye Zhang made a comment about SOFEA on Matt Raible's blog to the effect that we seemed to be chasing buzzwords and hence tacked on the "Service-Oriented" prefix to our model. Not true. Service-Orientation is at the heart of our model, because we approached the Presentation Tier from that angle. Our emphasis on XML as the Data Interchange mechanism for SOFEA was designed to enable the Presentation Tier to interoperate with the Service Tier with no impedance mismatch at all.

At any rate, the software industry seems to be at the start of a new era, and I'm happy we were able to make a contribution to the debate.

Life above the Service Tier

Or "How to Build Application Front-ends in a Service-Oriented World"

(This is a paper written by myself with two co-authors, Rajat Taneja and Vikrant Todankar, the same team that critiqued the EJB 3.0 spec in 2004.

You can download and read the paper and an accompanying presentation.

Update Sep 2008: How SOFEA fits into a SOA Ecosystem)

But let me talk a bit about it here.

When we look at the Business Logic Tier, although the industry hasn't reached nirvana yet, the story seems to be coming together rather well. The Service-Oriented Architecture (SOA) message is being heard, with its concepts of loose coupling between systems, removal of implicit dependencies and specification of all legitimate dependencies as explicit contracts. The SOA model yields flexibility and uniformity through the specification of a definite Service Interface, which hides domain models and implementation technologies.

Unfortunately, no such clear architectural blueprint exists for the Presentation Tier that lies just above the Service Interface. There are a dozen different technologies available here. There are also the thin-client and rich-client models to choose from. Which of these is the "right" one to use? What does industry best practice say about developing the front-end of an application? What is the best way to connect the front-end neatly into the Service Interface? Developers are confused and left without guidance.

And while on the subject of connecting the front-end to the Service tier, an additional source of confusion is the fact that the Service Tier may itself follow one of two different models - SOAP-based Web Services or REST. Which one should we target? We believe that both these models will continue to co-exist, and organisations will have legacy business services in both these models. Therefore a good Presentation Tier technology will be capable of interfacing with both SOAP and REST, not just one of them. It is important to note that both SOAP and REST use (or are capable of using) XML documents for data interchange.

When we put on our architects' hats and look at the Presentation Tier, we see that regardless of whether it is a thin client or a rich client, three essential processes always take place.

The first is what we call Application Download. In order to be able to run an application on a client device, it needs to be delivered to the device in some fashion.

When the application is in use, a sequence of screens is typically seen. We call this Presentation Flow.

Finally, any non-trivial application ultimately exchanges data with some back-end resource, a process we call Data Interchange. This is usually the crux of the application's functionality.

To reiterate, the three processes occurring in the Presentation Tier of any application are Application Download (AD), Presentation Flow (PF) and Data Interchange (DI).

In thin-client applications, the web server plays a part in all three processes. Application Download occurs piecemeal, in the form of individual web pages served up by the web server. The web server also drives the application's Presentation Flow, usually with the help of a web framework like Struts or Spring MVC. Finally, the web server acts as an intermediary for Data Interchange between browser and application server.

We believe that the thin-client model as described above suffers from at least three major architectural flaws. Unless these flaws are addressed, we cannot arrive at a clean architectural blueprint for the Presentation Tier.

The first flaw is that the thin-client model does not respect data. Look at the data that goes from the browser to the web server. There is no data structure. It's just a set of name-value pairs. There are no data types. Everything is a string. And there are no data constraints. We can put any values into those name-value pairs.

Look at the data that comes back from the web server to the browser. It's highly presentation-oriented, marked-up data. The thin-client model does not respect data as data. We believe we can do much better than this in this day and age.

The second flaw is that the thin-client model tightly couples the mutually orthogonal processes of Presentation Flow and Data Interchange. It is not possible to move through the steps of the Presentation Flow without initiating what amounts to Data Interchange operations. Web pages are displayed in response to GET and POST requests sent by the browser. Even worse, every Data Interchange operation initiated by the browser willy-nilly forces a Presentation Flow. An infamous result of this tight coupling is the "browser back-button problem". When a user tries to step backwards page-by-page through the Presentation Flow, the browser re-does the Data Interchange operations that resulted in each of those pages. If any of those operations is a POST, it spells trouble, because POST operations are not safe or "idempotent". They have side-effects if re-done.

True, there are patterns such as POST-Redirect-GET that are used to work around this problem, but the inescapable fact is that the fundamental architecture is broken because of this tight coupling.

The third flaw is that the web model is request/response. It does not support peer-to-peer interaction styles that are required for server event notification.

At this juncture, one would be tempted to conclude that AJAX is the answer to these problems. Unfortunately, AJAX itself is just a raw capability and not a prescriptive model. It is possible to use AJAX and still come up with a horrible hybrid model where the web server continues to drive Presentation Flow in response to Data Interchange operations, and the AJAX interaction just hangs off to one side, so to speak.

Clearly, what we need is a prescriptive architectural model that explicitly delineates the dos and don'ts of application design.

We call this model SOFEA (Service-Oriented Front-End Architecture).

In the SOFEA model, there is an entity called a Download Server. This role may be performed by a standard web server, but its role is very strictly restricted to enabling Application Download. It stays out of the loop thereafter and does not play a role in Presentation Flow or Data Interchange. This is the most significant difference between the SOFEA model and the traditional thin-client model.

The application is downloaded onto an Application Container, which could be a browser, a Java Virtual Machine, a Flash runtime or even the native operating system itself. The nature of the Application Container does not matter. What matters is that there is an environment within which the application can run.

The application is built using a proper MVC architecture, not the Front Controller pattern used by traditional thin-client applications. The Controller drives Presentation Flow.

The Controller also asynchronously initiates Data Interchange with the Service Interface as required.

We stipulate that a SOFEA-conforming application will support both SOAP and REST-based Data Interchange through XML documents. This is the seamless interface that we envisage between the Presentation Tier and the Service Tier. These XML documents, being specified through a schema definition language, will enforce data structures, data types and data constraints at the point of capture and ensure data integrity end-to-end. The XML documents of the service interface correspond to a representation of the domain, or (to a Java programmer) Data Transfer Objects (DTOs). They're ideal for transferring data marshalled from domain objects or unmarshalled into domain objects. Thus, while XML documents are a connection point between the Presentation and Service Tiers, they also decouple the client from the domain model, which is as it should be.

For maximum flexibility, Data Interchange should support Peer-to-Peer interaction. A strict client/server delineation as in the case of traditional thin-clients is likely to be limiting.

The SOFEA model unifies the "thin" and "rich" client models. Indeed, these labels are meaningless when systems conform to the SOFEA model. The differences between technologies form a spectrum based on the application's footprint and its startup time. There is no sharp demarcation between thin and rich clients anymore.

The SOFEA model also highlights why there is a plethora of web frameworks. SOFEA explicitly repudiates Front Controller as an anti-pattern. Driving Presentation Flow from the web server is tied to a fundamental architectural flaw (the coupling to Data Interchange). Therefore no web framework can ever be satisfactory. Continued innovation in search of a "better" web framework is, we believe, completely misguided.

The SOFEA model can be implemented through any of the following modern technologies:

I DHTML/AJAX frameworks for Current Browsers

1. Handcoded with third party JavaScript libraries
2. Google Web Toolkit (GWT, GWT-Ext)
3. TIBCO General Interface Builder

II XML Dialects for Advanced Browsers

4. XForms and XHTML 2.0
5. Mozilla XUL
6. Microsoft SilverLight/XAML

III Java frameworks

7. Java WebStart (with/without Spring Rich Client)
8. JavaFX

IV Adobe Flash-based frameworks

9. Adobe Flex
10. OpenLaszlo

Obviously, different technologies will support, out of the box, different subsets of the SOFEA model. It's up to designers to consciously apply SOFEA principles to their application design, making the appropriate decisions with regard to their technology of choice.

Our contribution through the SOFEA model is the following:

1. A renewed emphasis on data integrity, which traditional thin-client technology does not and cannot enforce.
2. A cleaner architectural model that decouples the orthogonal concerns of Presentation Flow and Data Interchange.
3. Affirmation of MVC rather than Front Controller as the natural pattern to control Presentation Flow.
4. Unification of the thin-client and rich-client models, now seen as an artificial distinction.
5. Support for SOAP- and REST-based business services, and a natural integration point between the Presentation and Service Tiers.
6. Positioning the web server as a Download Server alone. The evils of web server involvement in Presentation Flow and Data Interchange are avoided.

We believe we have provided an architectural blueprint for the Presentation Tier, with clear dos and don'ts, while still supporting considerable diversity in technologies. Life above the Service Tier now lives by similar rules as life within it.

Tell us what you think.