Orwellian Truths of SOA - 2

OK, so this second post may not have the immediate shock value of a "War is Peace" kind of statement that I had in my first, but it still talks about things that people don't normally acknowledge.

Here's today's Orwellian quote:

"SOA subsumes BPM. SOA subsumes ECM."

First of all, why should this be a shocking statement? Because in many organisations, SOA, BPM and ECM are viewed as different things altogether. SOA is considered to be new-age stuff. It's supposed to be all about SOAP, Web Services, BPEL, ESB and other new-fangled technologies. BPM and ECM, on the other hand, having evolved from earlier workflow and document management products, are seen as more "legacy".

Is BPM (Business Process Management) part of SOA? Many people I talk to make a distinction between "workflow" (by which they mean business processes involving people) and process orchestration (by which they mean the automated invocation of computer-based services). The latter is typically implemented as BPEL-defined processes coordinating SOAP-based Web Services. And while BPM products have begun to support BPEL today, they're still used for only workflow tasks. Even more interestingly, large vendors who sell both BPM and process orchestration tools discourage customer organisations from using BPM tools for the latter task even if they're BPEL-capable.

I'd like to challenge this distinction. And I'm going to do this by going back to SOA's first principles, and not falling into the trap of thinking that SOA means Web Services technology.

The SOA approach requires us to analyse business processes as first-class entities. What is the business function that is being performed? The analyst breaks down the process into logical steps, perhaps rationalises it as well in a BPR kind of exercise, then defines the boundaries between process steps through contracts that define the information flow that occurs across those boundaries.

That's why SOA is often referred to as "Contract-Oriented Architecture".

Take the example of a loan approval process. One of the initial steps is a credit check. The process needs to call upon a "service" that performs this function. It needs to pass this service the necessary identifying data about the applicant so that the service can return a credit rating. At a later stage in the process, especially if the loan application is a borderline case, a loan officer may be required to assess the application and provide an approval/rejection decision.

From a SOA perspective, I would argue that these two examples are no different. In each case, the process involves a step where some data needs to be sent off to some specialised service provider in order to receive a specialised piece of information, which is then used in subsequent steps of the process. If we define the boundaries of the process steps in terms of contracts, there is nothing whatever at the logical level to distinguish one from the other on the basis that the first is automated and the second is manual. They're both identical from an interaction perspective. The first may take mere seconds while the second may take minutes, hours or days, but logically, the interaction model is the same.

This ties back to the earlier point about synchronous being asynchronous and real-time being batch. There is a standard interaction model within SOA. The process interacts with an automated credit check service asynchronously by placing applicant data (a batch of one) in a queue, and picks up credit rating data from another queue when triggered by the arrival of that data. In exactly analogous fashion, the process places application data in a queue (the loan officer's inbox) and picks up the approval/rejection decision from another queue (the loan officer's outbox).

It's a sort of Turing test within SOA. How can a process tell that a particular service step is being performed by a human and not an automated system? The answer is that it can't. In fact, it shouldn't, or else it's no longer SOA, because we violate the contract if we use implied knowledge! In that sense, it's an unfair Turing test, because we're not allowed to use any information outside of the contract to guess at the implementation. Hiding implementation behind contracts is the beauty of SOA, because it allows replacement of manual systems by automated ones, or the outsourcing of certain process steps, without breaking the process itself and impacting business continuity.

So that proves my point. SOA subsumes BPM.

ECM (Enterprise Content Management) seems to be another beast altogether. We're not talking services here. We're talking data. Surely that doesn't have much to do with SOA unless services are built on top of it?

Surprise. If we adopt the REST approach to SOA, we find that what we're doing is perfunctorily dismissing the "service" or "verb" side of the design by employing a uniform, polymorphic CRUDish interface and concentrating mainly on the modelling of the "resources" involved. The distinctive "services" that make up the application from a service consumer's point of view automatically follow from the type of resource and the polymorphic nature of the RESTian verbs. REST is effectively based on a very audacious premise that no matter what your application, you can always model it as a set of resources in such a way that the standard four RESTian verbs will let you perform all the kinds of manipulation that you could possibly hope to perform on them.

I haven't yet seen a convincing repudiation of that approach. The REST approach seems to scale to arbitrary levels of application complexity. It's almost as if service-orientation appears as a byproduct of the organisation of the application domain as a structured collection of "content". An approach to Content Management is therefore a valid approach to SOA! And that proves my second point. SOA subsumes ECM.

I'm sure these views will gradually gain currency, and that bolder organisations will start to bring all of these functions under a common umbrella. But for the near future, I'm afraid we will continue to see a rather fragmented approach to SOA, BPM and ECM.

Orwellian Truths of SOA - 1

"War is Peace. Freedom is Slavery. Ignorance is Strength."

George Orwell made those words famous in his novel Nineteen Eighty-Four.

In a discussion with a colleague today, I realised that to attain the Zen-like wisdom of SOA, we need to accept similarly ridiculous-sounding Orwellian propositions. By Orwellian, I don't imply something sinister or Big Brother-like. I'm just talking about opposites being considered the same.

Try my version:

"Local is Remote. Synchronous is Asynchronous. Real-time is Batch."

Sounds ridiculous, right? But it's all true, believe me.

Take "Local is Remote". I've already blogged here and here about why the older concept of "distributed objects" and remote procedure calls (RPC) was wrong-headed. In a nutshell, it's because copies of objects can be passed either locally or across a network but memory references can only be passed around locally and cannot be passed over a network. So working with copies of objects is a more universally applicable paradigm than working with memory references to objects. In other words, treating everything as a remote object and passing around copies of it always works (even in the local case), but treating everything as a local object and passing around references to its memory location will not work in the remote case. The SOA approach, whether using the SOAP-based messaging model or the REST-based representation-transfer model, deals with passing copies of objects around rather than passing (or pretending to pass!) memory references around. In other words, the SOA approach treats local objects as if they are remote. Local is Remote.

What about "Synchronous is Asynchronous"? When we delve into the specifics of inter-system communication, we realise that all systems fundamentally communicate through messages, and messages are asynchronous. Then what causes "synchronous" behaviour? Simply put, synchronous behaviour arises when two systems agree on certain conventions when they exchange messages. Conceptually, we can layer the capability to correlate messages on top of a pure messaging system (Give every message a unique ID that travels with it. Then we can say that this message is in response to that one). Once we have the ability to correlate messages, we have a foundation on which we can then layer the capability to coordinate actions ("I'll do an action when you send me a response to my message, and I'll then respond to your message telling you I've done it, based on which you can perform some action"). Synchronous behaviour is a kind of coordination ("I won't do anything else until you respond to my message"). Another kind of coordination is the notion of transactions. Transactions can either be of the classic all-or-nothing variety or a looser style based on compensating actions. Both of them are types of coordination between systems. So peeling back the layers, synchronous interactions (even highly transactional ones) arise from nothing more than a layered set of capabilities on top of a basic messaging paradigm. This is a universal architecture regardless of the technology chosen. It is as true of TCP/IP as it is of SOAP/WS-*. Synchronous is Asynchronous.

But surely "Real-time is Batch" is a bit of a stretch?

I remember learning about the phenomenon of Diffraction in Optics many years ago. If you close a door or window so that just a tiny crack is visible, you may find that the light from the other side forms a regular, fan-shaped pattern on the floor or windowsill, with alternate dark and bright areas. My Physics textbook said this happens when the crack is "small", then asked the thought-provoking question, "Small in relation to what?"

The answer, of course, is the wavelength of light. If the width of the crack in the door or window is of a similar order of magnitude to the wavelength of light, you see a nice diffraction pattern.

Now what is "real-time"? Think of the mainframe-based batch payroll process in an organisation that must complete by 3 a.m. on Monday. How is this any less time-critical than the ABS (Anti-lock Braking System) on a car that must kick in within milliseconds whenever the wheels start to skid? Like with the diffraction example, we have to ask, "Real-time compared to what?" Again, the answer is the degree of precision with which we measure the deadline in question.

Strictly speaking, "real-time" is a requirement or a constraint. "Batch" is a processing model. But there's nothing to prevent us from using a batch processing model as long as it meets the required "real-time" deadline within the degree of precision with which the deadline is measured. Developers who work with message queues will know what I'm talking about. In large organisations, interfaces to many systems are through message queues. Place a request message on one queue and pick up the response message from another queue. And these queues are fast. They're fast enough to handle critical, "real-time" applications like Internet Banking, for example. The customer interacting with their bank using a browser and obtaining satisfactory response times is unaware that their messages are being put on queues and retrieved from queues, - effectively in batches of one. In many implementations therefore, Real-time is Batch.

What has all this to do with SOA?

Well, the SOA mindset is based upon such Inversion of Conventional Wisdom. Indeed, that's what makes SOA unique and that's why it works.

In a later entry, I'll expand on these concepts and present some more Orwellian truths.