Orwellian Truths of SOA - 2

OK, so this second post may not have the immediate shock value of a "War is Peace" kind of statement that I had in my first, but it still talks about things that people don't normally acknowledge.

Here's today's Orwellian quote:

"SOA subsumes BPM. SOA subsumes ECM."

First of all, why should this be a shocking statement? Because in many organisations, SOA, BPM and ECM are viewed as different things altogether. SOA is considered to be new-age stuff. It's supposed to be all about SOAP, Web Services, BPEL, ESB and other new-fangled technologies. BPM and ECM, on the other hand, having evolved from earlier workflow and document management products, are seen as more "legacy".

Is BPM (Business Process Management) part of SOA? Many people I talk to make a distinction between "workflow" (by which they mean business processes involving people) and process orchestration (by which they mean the automated invocation of computer-based services). The latter is typically implemented as BPEL-defined processes coordinating SOAP-based Web Services. And while BPM products have begun to support BPEL today, they're still used for only workflow tasks. Even more interestingly, large vendors who sell both BPM and process orchestration tools discourage customer organisations from using BPM tools for the latter task even if they're BPEL-capable.

I'd like to challenge this distinction. And I'm going to do this by going back to SOA's first principles, and not falling into the trap of thinking that SOA means Web Services technology.

The SOA approach requires us to analyse business processes as first-class entities. What is the business function that is being performed? The analyst breaks down the process into logical steps, perhaps rationalises it as well in a BPR kind of exercise, then defines the boundaries between process steps through contracts that define the information flow that occurs across those boundaries.

That's why SOA is often referred to as "Contract-Oriented Architecture".

Take the example of a loan approval process. One of the initial steps is a credit check. The process needs to call upon a "service" that performs this function. It needs to pass this service the necessary identifying data about the applicant so that the service can return a credit rating. At a later stage in the process, especially if the loan application is a borderline case, a loan officer may be required to assess the application and provide an approval/rejection decision.

From a SOA perspective, I would argue that these two examples are no different. In each case, the process involves a step where some data needs to be sent off to some specialised service provider in order to receive a specialised piece of information, which is then used in subsequent steps of the process. If we define the boundaries of the process steps in terms of contracts, there is nothing whatever at the logical level to distinguish one from the other on the basis that the first is automated and the second is manual. They're both identical from an interaction perspective. The first may take mere seconds while the second may take minutes, hours or days, but logically, the interaction model is the same.

This ties back to the earlier point about synchronous being asynchronous and real-time being batch. There is a standard interaction model within SOA. The process interacts with an automated credit check service asynchronously by placing applicant data (a batch of one) in a queue, and picks up credit rating data from another queue when triggered by the arrival of that data. In exactly analogous fashion, the process places application data in a queue (the loan officer's inbox) and picks up the approval/rejection decision from another queue (the loan officer's outbox).

It's a sort of Turing test within SOA. How can a process tell that a particular service step is being performed by a human and not an automated system? The answer is that it can't. In fact, it shouldn't, or else it's no longer SOA, because we violate the contract if we use implied knowledge! In that sense, it's an unfair Turing test, because we're not allowed to use any information outside of the contract to guess at the implementation. Hiding implementation behind contracts is the beauty of SOA, because it allows replacement of manual systems by automated ones, or the outsourcing of certain process steps, without breaking the process itself and impacting business continuity.

So that proves my point. SOA subsumes BPM.

ECM (Enterprise Content Management) seems to be another beast altogether. We're not talking services here. We're talking data. Surely that doesn't have much to do with SOA unless services are built on top of it?

Surprise. If we adopt the REST approach to SOA, we find that what we're doing is perfunctorily dismissing the "service" or "verb" side of the design by employing a uniform, polymorphic CRUDish interface and concentrating mainly on the modelling of the "resources" involved. The distinctive "services" that make up the application from a service consumer's point of view automatically follow from the type of resource and the polymorphic nature of the RESTian verbs. REST is effectively based on a very audacious premise that no matter what your application, you can always model it as a set of resources in such a way that the standard four RESTian verbs will let you perform all the kinds of manipulation that you could possibly hope to perform on them.

I haven't yet seen a convincing repudiation of that approach. The REST approach seems to scale to arbitrary levels of application complexity. It's almost as if service-orientation appears as a byproduct of the organisation of the application domain as a structured collection of "content". An approach to Content Management is therefore a valid approach to SOA! And that proves my second point. SOA subsumes ECM.

I'm sure these views will gradually gain currency, and that bolder organisations will start to bring all of these functions under a common umbrella. But for the near future, I'm afraid we will continue to see a rather fragmented approach to SOA, BPM and ECM.

Orwellian Truths of SOA - 1

"War is Peace. Freedom is Slavery. Ignorance is Strength."

George Orwell made those words famous in his novel Nineteen Eighty-Four.

In a discussion with a colleague today, I realised that to attain the Zen-like wisdom of SOA, we need to accept similarly ridiculous-sounding Orwellian propositions. By Orwellian, I don't imply something sinister or Big Brother-like. I'm just talking about opposites being considered the same.

Try my version:

"Local is Remote. Synchronous is Asynchronous. Real-time is Batch."

Sounds ridiculous, right? But it's all true, believe me.

Take "Local is Remote". I've already blogged here and here about why the older concept of "distributed objects" and remote procedure calls (RPC) was wrong-headed. In a nutshell, it's because copies of objects can be passed either locally or across a network but memory references can only be passed around locally and cannot be passed over a network. So working with copies of objects is a more universally applicable paradigm than working with memory references to objects. In other words, treating everything as a remote object and passing around copies of it always works (even in the local case), but treating everything as a local object and passing around references to its memory location will not work in the remote case. The SOA approach, whether using the SOAP-based messaging model or the REST-based representation-transfer model, deals with passing copies of objects around rather than passing (or pretending to pass!) memory references around. In other words, the SOA approach treats local objects as if they are remote. Local is Remote.

What about "Synchronous is Asynchronous"? When we delve into the specifics of inter-system communication, we realise that all systems fundamentally communicate through messages, and messages are asynchronous. Then what causes "synchronous" behaviour? Simply put, synchronous behaviour arises when two systems agree on certain conventions when they exchange messages. Conceptually, we can layer the capability to correlate messages on top of a pure messaging system (Give every message a unique ID that travels with it. Then we can say that this message is in response to that one). Once we have the ability to correlate messages, we have a foundation on which we can then layer the capability to coordinate actions ("I'll do an action when you send me a response to my message, and I'll then respond to your message telling you I've done it, based on which you can perform some action"). Synchronous behaviour is a kind of coordination ("I won't do anything else until you respond to my message"). Another kind of coordination is the notion of transactions. Transactions can either be of the classic all-or-nothing variety or a looser style based on compensating actions. Both of them are types of coordination between systems. So peeling back the layers, synchronous interactions (even highly transactional ones) arise from nothing more than a layered set of capabilities on top of a basic messaging paradigm. This is a universal architecture regardless of the technology chosen. It is as true of TCP/IP as it is of SOAP/WS-*. Synchronous is Asynchronous.

But surely "Real-time is Batch" is a bit of a stretch?

I remember learning about the phenomenon of Diffraction in Optics many years ago. If you close a door or window so that just a tiny crack is visible, you may find that the light from the other side forms a regular, fan-shaped pattern on the floor or windowsill, with alternate dark and bright areas. My Physics textbook said this happens when the crack is "small", then asked the thought-provoking question, "Small in relation to what?"

The answer, of course, is the wavelength of light. If the width of the crack in the door or window is of a similar order of magnitude to the wavelength of light, you see a nice diffraction pattern.

Now what is "real-time"? Think of the mainframe-based batch payroll process in an organisation that must complete by 3 a.m. on Monday. How is this any less time-critical than the ABS (Anti-lock Braking System) on a car that must kick in within milliseconds whenever the wheels start to skid? Like with the diffraction example, we have to ask, "Real-time compared to what?" Again, the answer is the degree of precision with which we measure the deadline in question.

Strictly speaking, "real-time" is a requirement or a constraint. "Batch" is a processing model. But there's nothing to prevent us from using a batch processing model as long as it meets the required "real-time" deadline within the degree of precision with which the deadline is measured. Developers who work with message queues will know what I'm talking about. In large organisations, interfaces to many systems are through message queues. Place a request message on one queue and pick up the response message from another queue. And these queues are fast. They're fast enough to handle critical, "real-time" applications like Internet Banking, for example. The customer interacting with their bank using a browser and obtaining satisfactory response times is unaware that their messages are being put on queues and retrieved from queues, - effectively in batches of one. In many implementations therefore, Real-time is Batch.

What has all this to do with SOA?

Well, the SOA mindset is based upon such Inversion of Conventional Wisdom. Indeed, that's what makes SOA unique and that's why it works.

In a later entry, I'll expand on these concepts and present some more Orwellian truths.

SpringSource App Server Released

Ben Alex of SpringSource announced this last night at the Sydney Java Users Group meeting.

As big as the Spring framework has been, I think this product is going to have a pretty profound impact on the Java world. I have always wondered why we couldn't use OSGi for server-side deployment when it seemed such a natural model for dealing with dynamic modules. Apparently, while the model has been working for client-side apps for years, there have been issues with connections and threads on the server side, and these issues have only just been solved by the SpringSource "platform", which enhances the basic OSGi container.

OSGi bundles have long been a good way to build applications with flexibly-definable dependencies. Multiple versions of the same library can be deployed into a container without conflicts, allowing other modules that depend on them to run without problems. One can also deploy urgent patches to production servers without having to bring them down. All this is huge, because it allows application modules to be developed in a loosely-coupled manner, by specifying dependency ranges and following standards for version numbering. This is a Service-Oriented approach to deployment.

Going one step further is the ability to build the App Server itself in the form of bundles rather than as a monolithic executable. (JBoss does this too, but not with OSGi). The SpringSource App Server (S2AP) is the first of this new breed of beast. By way of analogy, S2AP is as different from earlier generations of app servers as the T-1000 of Terminator 2 was different from the earlier T-800 (think liquid metal). It means app servers need be no larger than necessary for the task at hand.

Well, the S2AP product is still in beta, so a few more months are going to pass while the industry kicks its tyres. But ultimately, I think there will be a shift to this more modular, version-friendly style of application deployment.

Is the Domain an Aspect?

We know about aspects - those cross-cutting concerns that we can layer on top of an application, the functions like logging that need to be done in a number of different places.

A curious thought struck me for the first time since I was introduced to Grails. Here's a complete framework for any generic application. It has persistence, a web interface, a testing framework, etc., etc. All you need to do to build your own custom application is to add your domain classes and a set of constraints governing them, and the system essentially generates whatever is required for the application to run.

It's almost as if the developer applied a "domain aspect" to a completely generic application and turned it into a bespoke one.

It's a fascinating thought, and I need to explore this idea some more. By Jove, this could be as profound an idea as Inversion of Control! Before Martin Fowler can jump in to do his mischief, let me lay claim to a number of possible pattern names:

1. Inversion of Aspect
2. Domain as Aspect
3. Domain Injection

There, that should do it until I gain more insights on this.

New Home for SOFEA, Thin Server Architecture

Peter Svensson has set up a website where like-minded people can discuss the brave new world of applications whose common characteristic is that no aspect of presentation logic resides on the server side. I admit that's an overly broad-brush generalisation, and it will be necessary to read what the various authors of this camp have to say.

In addition to the SOFEA paper that Rajat Taneja, Vikrant Todankar and I co-wrote last year, the site has Peter's articles on The End of Web Frameworks I and II and Mario Valente's four articles 1, 2, 3 and 4.

The site is admittedly a bit rough round the edges at the moment, but we will be polishing it as the days go by, and we would welcome community feedback. For the moment, since the site itself has no comment facility, you may comment here, or send me a mail (g dot c dot prasad at gmail dot com).

Context Diagrams as a Tool for SOA

Pursuing the theme of reverting to a form of procedural thinking to implement SOA, I remember how we used to draw Context Diagrams and Data Flow Diagrams (DFDs) as part of system design in the eighties. Context Diagrams were also called DFD Level 0. We used to draw the system being designed as a single bubble, and show all interactions (dataflows) with external parties (drawn as rectangles) as a set of arrows going back and forth between these external parties and the system.

To draw the Level 1 and Level 2 DFDs, we used to "explode" the single bubble into "modules". The dataflows shown in the Context Diagram would still need to be shown, but this time, (1) the arrows would terminate at one of the smaller bubbles representing a particular module of the system, (2) the external parties themselves would no longer be shown and (3) there would be new arrows showing dataflows between modules of the system and not involving external parties.

I now think the Context Diagram is the modelling tool for SOA. The dataflows going back and forth between the system and external parties that deal with it form the "service contract". We can express this in SOAP or REST terms. That really doesn't matter.

When we begin to "explode" the single bubble representing the system as a whole into smaller "modules", we are entering the world of domain design. This is no longer SOA. In the old days, the inter-module dataflows were also procedural. Today, they're method calls on objects.

I wrote earlier that when a student embarks on the study of Zen, the mountains are nothing more than mountains. Partway through the training, the mountains are no longer mountains. But finally, when the student has mastered Zen, the mountains are once again mountains, but they're somehow not the same as what they were.

Here's the SOA analogy:

Back in the eighties and early nineties, when systems and modules were purely procedural, we used to draw Context Diagrams and consider them to be Level 0 Data Flow Diagrams. Then we used to use the same paradigm to further explode the Level 0 DFD into Level 1 and Level 2 DFDs.

In the mid-to-late nineties, when OO thinking predominated, we stopped using Context Diagrams and DFDs altogether. We started using UML's Class Diagrams and Sequence/Interaction diagrams and began to look down upon procedural thinking.

In the new millennium, with SOA thinking in vogue, we need to start using Context Diagrams once again to define our service contracts, but we must no longer use the same paradigm to "explode" the single system bubble into modules. Leaving the Context Diagram in place, we must dive below that level and independently model the underlying domain using OO thinking. Alternatively, we build the domain model first, and then independently draw a Context Diagram around it, with a single bubble completely encompassing the domain model and terminating dataflows to and from external parties. Either way, we must then reconcile the two layers by creating a layer of "services" that sit within the Context Diagram's bubble and outside the domain model and translate between the external-facing procedural view and the internal-facing OO view. This service layer would be responsible for the specific transformation that I call The Viewpoint Flip.

A diagram would illustrate this better.

So you see, grasshopper, the mountains are once again mountains, are they not? But they're not the same as when you started on this journey. The Buddha-ness pervades all things, but the SOA-ness and Object-ness live in different worlds.

Sun Tech Days Sydney - Day 3 (Community Day, 6 March 2008)

(I've already blogged about the first and second days.) There were 5 tracks on this (third) day, called "Community Day": GlassFish, NetBeans, Mobility, Solaris and Sun University. I attended 3 sessions each from the GlassFish and Mobility tracks.

Alexis Moussine-Pouchkine provided an introduction to the GlassFish project. GlassFish is almost exactly the same product as the Sun Java Application Server. The latter has a couple of extra proprietary bits added, and also provides more frequent updates.

A few tidbits that I noted:

The admin console looks pretty professional, unlike Tomcat's minimal one, and unlike the ugly raw JMX that I've seen in the past with JBoss (I haven't used JBoss in a while, thanks to Spring and Tomcat). GlassFish's admin console is even better than Geronimo's, which I thought was pretty good at the time. It's task-oriented, so users will find it easier to find what they need.

GlassFish logs to a bundled JavaDB database (Cloudscape/Derby).

Apparently, GlassFish has a third container apart from the web and EJB containers. It's called the ACC (Application Client Container). Developers can build a Swing client and bundle it with the ear file. Java Web Start is used to deliver it. Dependency Injection can be used even with the client.

In cluster mode, GlassFish keeps Applications, Configurations and Resources consistent across the cluster. GlassFish uses a dynamic clustering feature called Shoal that is built on top of JXTA.

There's a project called SailFin which is meant to bridge HTTP and SIP. This is useful not only for the telecom industry but also generally, to provide rich media capabilities to enterprise apps.

I haven't explored Sun's Portal offering before (OpenPortal). That's another thing on my growing to-do list.

GlassFish has a number of components:

The GlassFish App Server itself
OpenPortal (portal server)
OpenESB (Enterprise Service Bus, of which more later)
OpenSSO (single sign-on solution)
OpenDS (directory server)
Jersey (for RESTful services)
Hudson (continuous build)
jMaki (aggregates Ajax widgets and provides a common API wrapper)
jRubyOnRails
OpenMQ (message queue)
WoodStock (JavaServer Faces components)
JavaDB (Apache Derby pure Java database)
Apache Roller (blog software, one of the "Enterprise 2.0" offerings)
Slynkr (social tagging software, another of the "Enterprise 2.0" offerings)
SailFin (SIP-HTTP bridging software)

Here's the GlassFish project wiki that provides ongoing detail about the project.

David Pulkrabek from Sun's labs in Prague spoke about the future of JavaME development tools, and his demos were very impressive.

He showed a demo of how to record and replay a set of actions, that could be used in automated testing. Then he showed a demo of how a Sun SPOT device could be used to capture sensory inputs like orientation and 3-dimensional acceleration to control a game on a mobile device. Could this power the next Wii?

Next he showed how real devices and emulators are treated the same by toolkits. A mobile phone plugged into the laptop with a USB cable is detected by the tools, and the debugger can execute applications on the mobile phone, with breakpoints and variable editing, just as in a local emulator. The output of the mobile phone can be redirected back into the toolkit's console.

Very impressive. I think JavaME developers have a pretty good set of tools at their disposal, thanks to the work done at Sun's Prague labs.

But even JavaME pales into insignificance when compared to JavaFX Mobile. This session was from Noel Poore. JavaFX Mobile is a rather unfortunately named piece of technology, and will continue to cause Sun grief because of the needless confusion with JavaFX Script. Let me clarify. JavaFX Mobile is the name of a mobile device platform that Sun acquired when it took over SavaJe in May 2007. JavaFX Script is a new (as yet unreleased) scripting language for the UI, which essentially reduces Swing programming to a declarative style. Yes, JavaFX Mobile supports JavaFX Script, but that's the only (tenuous) connection. Sun should stop these silly and confusing names (The Java Desktop System was another one. That was just a Linux (or Solaris) desktop that could run Java).

JavaFX Mobile is based on the Linux kernel and Java SE. Note that it's Java Standard Edition, not JavaME, the Mobile Edition that the previous session dealt with. This has two major implications. One is that the full range of Java features is now available to mobile devices, not just the subset that JavaME used to provide. Second, from a market viewpoint, it removes the fragmentation that the various JavaME implementations necessarily suffered on account of their having to be tailored to devices. JavaFX Mobile is now a full-fledged platform that can take on the iPhone or Google's Android. It can still run MIDP Midlets, but Sun would encourage developers to build full-featured JavaFX Script-based UIs and applets rather than the comparatively anaemic Midlets, now that the full capabilities of a JavaSE platform are available.

The platform has three components: Core, Frameworks and User Experience (Applications).

Core: Kernel abstraction, Linux kernel 2.6, Core graphics, Core networking, Media engine, Device drivers, Filesystems and a C library (not glibc)

Frameworks: Content management, Java, Web content, Provisioning, Graphics, Connectivity, Multimedia, Networking, Messaging, Telephony

User Experience (Applications): Calendar, Browser, Phone, Camera, App Selector, Personal Utilities, Media Player, Content Viewer, System Utilities, Contacts, Messaging Apps, Home Screen, Themes and third party apps

Messaging includes IM and Presence (OMA IMPS and Jabber).

Jim Weaver (who has written many books on Java, including J2EE 1.4 and JEE 5) spoke next about JavaFX Script and showed a few demos. I'll just sum up my impressions on JavaFX Script.

First, JavaFX Script is to Swing as Groovy is to Java. It's a lightweight scripting approach to building GUIs, and it uses a declarative style.

Second, although JavaFX script is far less verbose than the equivalent Swing code, it still unfortunately looks like the dog's breakfast (compared to Flex, for example). I think it will only become popular when a graphical editor is layered on top of it.

I always like systems that are based on textual files that can be edited by hand if required, compared to systems that store data in binary format. But that doesn't mean I want to use the text editing approach exclusively. "Build graphically and tweak textually" is my motto for GUI development.

Michael Czapski from Sun's Sydney office spoke about JBI (Java Business Integration) and OpenESB.

Historically, approaches to integration have ranged from file transfer through shared databases and EAI to messaging. Messaging is the model currently in vogue. Models have progressed from point-to-point, through hub-and-spokes (brokered) to the "bus", which is a distributed, logically decoupled, anywhere-to-anywhere model.

ESB's facilitate a messaging-based bus model of integration.

JSR 208 deals with Java Business Integration, another unfortunately-named standard that doesn't immediately explain what it does.

This is a standard that governs how ESB components plug into the bus, allowing integrators to mix and match components from different sources to build up a "bus". I think it's more relevant to those who write components like adapters, BPEL engines, XSL transformation engines and the like rather than end-user organisations (like mine) that want to use a bus to integrate their applications. The latter don't care how the "bus" was put together, whether it is a monolithic piece of technology or a pluggable architecture that has components from different sources. [Well, we should care from a vendor lock-in perspective, but functionally, it makes no difference to us.]

JBI standardises component interfaces, component lifecycle (start, stop, deploy, undeploy, etc.), deployment model, monitoring and management, among other things.

It fosters loose coupling through WSDL interfaces, and implements WSDL 2.0 MEPs (Message Exchange Patterns).

The components that plug into a bus are either Service Engines or Binding Components. Examples of Service Engines are Aspect engines, BPEL engines, mashup processors, encoding, ETL, scripting, XSL transformation engines, etc. Examples of Binding Components are BCs for SAP, Siebel, CICS, CORBA, FTP, HTTP, Queues, etc.

OpenESB is Sun's JBI implementation. It uses NetBeans 6 for development and deploys naturally to GlassFish.

Michael demonstated a BPEL process that used two Binding Components (an input queue and an output file), with a Service Engine written on the spot using an EJB with Web Service annotations. The process read in a string from a queue (which was manually input), called the web service to turn it to uppercase, then wrote it into the file. Sure enough, when the file was opened, the string was there, in uppercase. Though it was a simple enough demo, it was fairly impressive.

NetBeans 6.0 is shaping up to be a good development platform for SOA. It has editors for BPEL and XSLT, among other things.

The last session I attended was by Alexis Moussine-Pouchkine again. This was on "Practical GlassFish". He talked about how GlassFish is built, the release cycles, etc.

There's reportedly a graphical installer from a third party (Izpack) for those who don't want to wrestle with the command line.

Alexis also demonstrated the Update Center, which works like Eclipse's.

He showed us some of the asadmin commands for command-line administration. These can obviously be used in scripts as well, which administrators would like.

From a competitive perspective, Sun has a migration tool that helps users automatically (to a large extent) migrate their current deployments on Tomcat, JBoss, WebLogic, WebSphere and older versions of Sun App Server onto GlassFish/Sun App Server. It's reportedly not foolproof, but it could help.

Documentation can be obtained by typing 'asadmin --help' on the command line or online.

That was the end of the three-day Sun Tech Days event. Very useful and informative.

My main takeaway, ironically, was nothing from Sun. It was Grails. I've started doing the InfoQ tutorial, and I'm mightily impressed. I'll blog more about Grails as I learn more.