Annotations and the Servlet 3 Specification

Now I'm seriously beginning to wonder about the authors of the Java Servlet 3 specification. This time, it's not their architectural wisdom (or the lack of it) regarding session state. It's about something even more basic to the Java language - the nature of annotations.

Chapter 8 deals with annotations that developers may use to mark their classes. Anything about the following strike you as crazy?

Classes annotated with @WebServlet class (sic) MUST extend the javax.servlet.http.HttpServlet class.
[...]
Classes annotated with @WebFilter MUST implement javax.servlet.Filter.

If we must extend a base class anyway, I wonder what the annotation is for. Just to avoid putting a few lines of config code into the web.xml file?

I would have thought an annotation like @WebServlet would be capable of turning any POJO into a servlet class, not just subclasses of HttpServlet! And we could have annotations like @GetMethod, @PostMethod, @PutMethod and @DeleteMethod to annotate any arbitrary methods in the class. We shouldn't have to rely on overriding the doGet(), doPost(), doPut() and doDelete() methods.

The same applies with @WebFilter. It could be used to annotate any arbitrary class, and @FilterMethod could then annotate any arbitrary method in the class.

Look at the way JSR 311 and Spring REST work.

I'm disappointed in the Servlet spec committee. If you're going to use annotations, then use them smartly.

It wouldn't be out of place here to comment on the horrific class hierarchy of the Servlet spec. It certainly shows the era from which it began, an era where interfaces were underappreciated and inheritance hierarchies featured classes themselves. Naming conventions hadn't matured yet, either.

E.g., my application's concrete class "MyServlet" must either extend abstract class "GenericServlet", which in turn partially implements interface "Servlet", or implement "Servlet" directly. This by itself isn't so bad, but go ahead a bit.

My application's concrete class "MyHttpServlet" must only extend abstract class "HttpServlet" which extends abstract class "GenericServlet", which in turn partially implements interface "Servlet". There is no interface to implement.

And why GenericServlet should also implement ServletConfig is something I don't understand. There's a HAS-A relationship between a servlet and its configuration. It's not an IS-A relationship.

HttpServlet should have been an interface extending Servlet.

The abstract class GenericServlet (that partially implements the Servlet interface) should have been called AbstractServlet instead, and there could have been a concrete convenience class called SimpleServlet or BasicServlet that extended AbstractServlet and provided a default implementation that subclasses could override.

Similarly, there should have been an abstract class called AbstractHttpServlet that partially implemented the HttpServlet interface and only provided a concrete service() method, dispatching requests to doXXX() methods that remained unimplemented. There could have been a concrete convenience class called SimpleHttpServlet or BasicHttpServlet that extended the AbstractHttpServlet class and provided a default implementation that subclasses could override.

My application's concrete classes should have had the option to implement one of the interfaces directly or to extend one of the abstract or convenience classes.

Oh well, too late now.

REST and the Servlet 3 Specification

I've been going through the Java Servlet 3 specification, and I just came across this gem at the start of the chapter on Sessions:

The Hypertext Transfer Protocol (HTTP) is by design a stateless protocol. To build effective Web applications, it is imperative that requests from a particular client be associated with each other. [...] This specification defines a simple HttpSession interface that allows a servlet container to use any of several approaches to track a user’s session [...]

I don't think the spec authors have been adequately exposed to the REST philosophy, or they wouldn't be talking so casually about how "imperative" sessions are to build "effective" Web applications. A few years ago, I would have read this without batting an eyelid. Now, I had to keep myself from falling off my chair in shock. One would think spec writers of advanced technology would know a bit better. At the very least, they could have written something like this:

The Hypertext Transfer Protocol (HTTP) is by design a stateless protocol, and it is strongly recommended that Web applications be built in a stateless manner to be effective, with all state management delegated to the persistence tier. If, for legacy or other reasons, it is unavoidable to maintain in-memory state in a web application, the servlet specification defines a simple HttpSession interface that provides a relatively painless way to manage it. Application developers should however be aware of the severe scalability and recoverability issues that will accompany the use of this feature.

There! Now I feel much better.

Does REST Need Versioning?

In my ongoing conversations with JJ Dubray, he has often made the point that "REST couples identity and access together in a terrible way". When pressed to explain, he provided the following example.

Assume that there is a Resource identified by "/customers/1234". Updating the state of this customer requires a PUT. JJ asks how REST can handle a change to the business logic implied by the PUT.

Since we cannot say

   PUTv2 /customers/1234

implying a change to the logic of PUT, he believes we have no option but to say

   PUT /customers/v2/1234

but this is different from the identity of the customer, which remains at

   /customers/1234

Hence REST "couples identity with access".

Well, I disagree. First of all, it's a mistake to think there are only two places where the version of business logic can be exposed - the Verb and the Resource. The data submitted is an implicit third, which I'll come to in a moment. But this example only makes me question the whole basis for versioning.

Does REST need versioning? For that matter, does any service need versioning? What is versioning in the context of SOA?

I would say service versioning is a mechanism that allows us to simultaneously maintain two or more sets of business logic in a consumer-visible way.

Why does it have to be consumer-visible as opposed to just consumer-specific? After all, if the service implementation can distinguish between two classes of consumer, it can apply two different business rules to them in a completely opaque manner. The consumer doesn't even have to know that two (or more) different sets of business rules are being weighed and applied under the covers.

Let's ask the more interesting question: Why do we need to maintain two or more sets of business logic simultaneously? The interesting (and circular) answer is often that business logic happens to be consumer-visible, hence a new version of business logic also needs to be distinguished from the old in a consumer-visible way. This is often stated as the need to support legacy consumers, i.e., consumers dependent in some way upon the previous version of business logic. But why do we have to support legacy consumers? Because existing contracts break when services are silently upgraded.

This argument leads to an interesting train of thought. Perhaps the answer lies in the opposite direction to what JJ believes, i.e., not versioning of services but abstraction of detail. Are our service contracts too specific and therefore too brittle? Service versioning is perhaps a "smell" that says we are going about SOA all wrong. Let us see.

I want to take up a more real-world example than the customer access example that JJ talked about. After all, that's more of a "data service" than a business service. Let's look at a real "business service".

Let's take the case of the insurance industry where a customer asks for a quote for an insurance product. The client-side application has to submit a set of data to the service and get a quote (a dollar value for the premium) in return.

In REST, here's how it could work.

Request:

   POST    /quotes
   <insurable-details>
   ...
   </insurable-details>

Response:

   201 Created
   Location: /quotes/06fb633b-fec4-4fb6-ae32-f298b8f499c1

The client is referred to the location of the newly-created quote Resource, which is at /quotes/06fb633b-fec4-4fb6-ae32-f298b8f499c1. When the client does a GET on this URI, the quote details are transferred.

So far, so good. Now let's say the business logic changes. Premiums are now calculated using very different logic. The first question is, can this new business logic be applied to all customers, or do we need to keep track of "old" customers and keep applying the old business logic to them? If we can "upgrade" all customers to the new business logic, there is, of course, no problem at all. The interface remains the same. The client application POSTs data to the same URI, and they are redirected in the same way to the location of the newly-created quote Resource. The business logic applied is all new, but customers don't see the change in their interface (only in the dollar values they are quoted!)

However, if we do need to maintain two sets of business logic, it could be for three reasons. One, the data that the client app needs to submit has changed, so the change is unavoidably visible to the customer and has to be communicated as a new and distinct contract. Two, there is another business reason to tell two types of customers apart, perhaps to reward longstanding customers with better rates, and this difference between customers is not obvious from the data they submit. Third, the client app somehow "knows" the behaviour of the old version and is dependent on it. In this case, we need a new version just to keep legacy clients from breaking.

We can readily see that the third reason is an artificial case for versioning. It's in fact a case to break implicit dependencies that have crept in.

In contrast, the first and second reasons provide their own resolution. If the type of data submitted by the client changes, that is itself a way to distinguish new clients from old ones and apply different business logic to them. In other words, we only need to tell newer customers about the change in the data they need to POST. Older customers don't need to do a thing. Also, if we can somehow derive that the customer is an existing one, even if this is not explicit in the data submitted, we can still apply different business logic transparently.

JJ may consider this a messy and unstructured approach to versioning. Business stakeholders may have the opposite view. It's less disruptive. The less clients are exposed to the way services are implemented, the better.

Service versions are not really an interface detail. They're an implementation detail that often leaks into the interface.

That means version numbers are a problem, not a solution.

None of these arguments may satisfy someone like JJ. In that case, if service versioning is absolutely essential, there is a simple way to include it, after all. Include the version number in the message body accompanying a POST or PUT request. In fact, message bodies are allowed even for GET and DELETE requests (anything except a TRACE), so versioning of any type of service is possible. REST does not enforce versioning, (that would be a bad thing considering that versions are often a smell), but doesn't impede it either.

With this approach, neither Verbs (e.g., POST) nor URIs (e.g., /quotes) are affected by versions and the "terrible" coupling of identity and access is avoided.

It seems to me that the problem is not with REST, it's with looking at REST through WS-* eyes.

Is REST Another Variant of Distributed Objects?

In my discussions with JJ on REST, he's often made the observation that REST is nothing but Distributed Objects and is therefore a bad style. But is it really?

I have two different arguments why I believe it isn't.

1. Let me first talk about my experience as a migrant from India to Australia many years ago. Working in Indian companies had acclimatised me to a rather direct management style. If my manager ever wanted me to do something, he or she would say so in so many words, "Do this!"

When I arrived in Australia and began working, I was struck by the very different style of Australian managers. I would hear things like, "You may want to do this," or "You may like to do this." It took me a while to realise that they were essentially saying the same thing, only less directly. Let me coin a term for this style, because I'm going to use it to explain a concept with Distributed Objects. Let me call this a polite command.

Let's now turn to methods that set an object's attributes. We may see setter methods like these:

 widget.setSomeAttribute( someValue );
 widget.setAnotherAttribute( anotherValue );

Now consider another style of doing the same thing.

 widget.updateSelf( widgetDTO );

where widgetDTO is a structure that holds the new values of someAttribute and anotherAttribute.

Let's call the direct setter methods commands. "Remoting" these commands leads to a tightly-coupled, RPC mechanism. This is the Distributed Objects style. I would be the first to agree with JJ that this is a bad approach.

But the second style is a polite command. It's requesting the object to update itself based on values held in a Data Transfer Object (i.e., a representation). Now this is a style that can be remoted without problems, because it's not really RPC.

The REST style of updating resources follows the latter model.

PUT   /widgets/1234
<attributes>
<some-attribute>some value</some-attribute>
<another-attribute>another value</another-attribute>
</attributes>

In other words, this is a polite command. It can be safely remoted. It's not Distributed Objects.

2. JJ laughs at the approach of annotating object methods to turn them into REST resources, the way JAX-RS does. This is another reason why he considers REST to be Distributed Objects. The annotations seem to be doing nothing more than remoting object methods. Therefore, REST = Distributed Objects and consequently a horrible way to design systems.

Not so fast.

Let's not forget the concept of Service Objects, which are not really Domain Objects.

Let's look at a simplistic domain model for a banking system. The major Domain Object in this model is an Account. An Account object has the following methods:

class Account()
{
  public double getBalance() {...};
  public void credit( double amount ) {...};
  public void debit( double amount ) {...};
}

The Domain Model is internally-focused. Nothing here understands the concept of a transfer. Indeed, a transfer cannot be elegantly modelled using domain objects.

That's because a transfer is an aspect of a Service. Service verbs are really free-standing verbs. They don't belong to classes the way domain methods do. The methods getBalance(), credit() and debit() don't make any sense by themselves. It always has to be account.getBalance(), account.credit() and account.debit().

In contrast, transfer() can be a free-standing verb. In fact, it seems downright clumsy to push transfer() into a class, because it really doesn't belong inside any class. It's just that languages like Java are unyieldingly object-oriented and don't tolerate free-standing verbs. So in practice, designers create an awkwardly-named class like AccountService and stick transfer() inside it, like this:

class AccountService()
{
   public void transfer( Account fromAccount, Account toAccount, double amount )
   throws InsufficientFundsException
   {
       if ( fromAccount.getBalance() >= amount )
       {
           fromAccount.debit( amount );
           toAccount.credit( amount );
       }
       else
       {
           throw new InsufficientFundsException();
       }
   }
}

[If J2EE designers feel a sense of déjà vu on seeing this, it's the old Session Façade pattern all over again, with Stateless Session Beans acting as a service façade for domain objects represented by Entity Beans. Stateless Session Beans were not part of the Domain Model at all. They were an explicit service layer that lent itself to being remoted through (what else?) a Remote Interface.]

Now, if we tried to remote a method like credit() or debit(), it would be a classic case of RPC (RMI, strictly speaking) and therefore Distributed Objects hell.

But a "method" like transfer() readily lends itself to being remoted! That's because an instance of AccountService isn't a Domain Object but a Service Object.

If designers took care to annotate only the methods of Service Objects and avoided doing so with methods of Domain Objects, then they neatly avoid being trapped into the Distributed Objects paradigm.

All that leads to a certain insight. And that is that although REST is an architectural style, it isn't prescriptive enough. We need to tell designers what types of domain objects can be modelled as resources and what cannot.

With a nod to the term polite command, perhaps it's not enough for systems to be RESTful. They should also be RESPECTful :-).

I realise I have blogged extensively on this over 2 years ago, here and here.

The Real and Imagined Limitations of REST

One of the things that struck me about my discussion with JJ Dubray on a previous blog posting was how closely we agreed on fundamental architectural principles (decoupling of interface from implementation, avoiding a hub-and-spokes architecture, etc.), yet how diametrically opposed our views were on REST.

For example, I think REST does a great job of decoupling interface from implementation. JJ feels the exact opposite. Why?

Analysing the problem more closely, I guess the common examples of RESTful interface design are partly to blame.

1. A URI like

http://myapp.mycompany.com/customers/1234

where 1234 is the actual id of the customer in my database, would be a horrible example of tight coupling, in my opinion. I believe URIs should be opaque and meaningless. I think designers should take care to create a mapping layer that decouples their implementation data model from their externally exposed service data model.

For example, I would prefer this to be exposed:

http://myapp.mycompany.com/customers/cb77380-7425-11df-93f2-0800200c9a66

There should be a mapping within the service implementation that relates the opaque identifier "cb77380-7425-11df-93f2-0800200c9a66" to the customer's id within the domain data model, i.e., 1234. That would be true decoupling.

Mind you, the earlier example of tight coupling is not a limitation of REST, merely bad application design.

[Incidentally, I think UUIDs have given the world a wonderful gift in the form of an inexhaustible, federated, opaque identification scheme for resource representations. Decoupling identity is a key part of decoupling domain data models (Pat Helland's Data on the Inside) from service data models (Data on the Outside).]

2. Another bad example is the use of "meaningful" URIs. Even though the following two URIs may seem obviously related, client applications must not make any assumptions about their relationship:

http://myapp.mycompany.com/customers
http://myapp.mycompany.com/customers/1234

In other words, a client application must not assume that it can derive the resource representation for the set of customers by merely stripping off the last token "/1234" from the URI of an individual customer.

And this is not a limitation of REST either. The HatEoAS (Hypermedia as the Engine of Application State) principle says that client applications must only rely on fully-formed URIs provided by the service to perform further manipulations of resources. In other words, URIs are to be treated as opaque and client applications must not attempt to reverse-engineer them by making assumptions about their structure.

Examples like these are bad from the perspective of architects who understand the evils of tight coupling, and the ones who produce these examples understand it too, but these naive examples have the benefit of being easy to understand.

A RESTful system would work perfectly well with URIs that looked like these:

http://myapp.mycompany.com/51bb2db7-b51c-4fd9-b235-2b310a644a84
http://myapp.mycompany.com/a7ac1f23-6bc7-44be-af9b-4b68fd31879f
http://myapp.mycompany.com/208657ef-1e86-47ba-9c5f-3f177c018293

The first is a URI for a customer, the second is a URI for a bank account and the third is a URI for an insurance policy. How about that? This would be architecturally clean, but most REST newbies would go, "Huh? Weird!" and turn away with a shudder.

Sometimes, the desire for understandability of a design by humans introduces semantic coupling that is anti-SOA. I guess there's a trade-off that we need to be aware of, but it's not a limitation of REST itself. It's an application designer's choice.

3. In another context, JJ has expressed his opinion that SOA is about asynchronous peer-to-peer interaction, and I completely agree. Where I think he has misunderstood REST is in its superficial characteristic of synchronous request-response. There are several well-known design patterns for asynchronous interaction using REST, so in practice, this is not a limitation at all. The HTTP status code of "202 Accepted" is meant for precisely this condition - "I've received your message but have not yet acted on it". At one stroke, it's also a model for reliable message delivery. Combine a POST with a one-time token to ensure idempotence, then keep (blindly) trying until you get a 202 Accepted response. Voila! Reliable message delivery.

I find I am able to look beyond the superficial characteristics of REST that seem like showstoppers to JJ. I can see simple workarounds and Best Practice guidelines that can make a RESTful application fully compliant with SOA principles. JJ stops at the characteristics as given and concludes that REST is horribly antithetical to SOA.

The Need for SOA in the Real World

To those who think SOA is just hype, this should be a sobering piece of news from the real world:

ANZ Bank hits a wall on financial software rollout

Sources have told The Australian that the bank is having second thoughts about going ahead with installing a Teradata solution made up of enterprise data-warehousing and database software into a predominantly Oracle-run house.
[...]
"It just doesn't marry well with the rest of the organisation as it's an Oracle house," one source said.

Anyone see the problem here?

Why should the brand of software being introduced have anything to do with a decision to implement? We in IT tend to treat this sort of argument as perfectly reasonable. Of course software from one vendor won't play nicely with software from another! We've internalised the obscenity of the situation. That's what makes this more horror than farce. We don't see the lack of interoperability between vendors as a problem, merely as a constraint.

Well, SOA thinking would have challenged this right off the bat. The brand of existing software and new software just doesn't matter. What matters is business functionality. As long as it's all wrapped up in contract-based services, they should play well together.

But we obviously don't live in that world. We live in a world where a strange kind of Stockholm Syndrome grips customers, preventing them from asserting their rights and causing them to acquiesce in a patently unfair seller's market. The "pragmatic" decisions that then follow create vendor lock-in and much higher outflows in the long run. So much for pragmatism.

What this news item tells me is that not only is the software vendor world not SOA-friendly, the business user community doesn't think in SOA terms either.

More's the pity.

SOAP, REST and the "Uniform Interface"

When REST folk compare their approach with SOAP-based Web Services (which they often mistakenly call "SOA", not realising that REST is an equally valid way to do SOA), they often refer to a "uniform interface" that REST provides that SOAP does not.

The response of the SOAP crowd (when there is one), is "What's a uniform interface?" (They'd be surprised to hear that they have one too, which is the topic of this post.)

The uniform interface refers to a standard way to do something regardless of what the specific activity is.

As an example, if I asked what the SOAP interface would be like for a service, and I refuse to say what the actual business function is, the SOAP people would have to stop after describing the service at a very high level. There's a SOAP message with a header and a body, and what goes into them depends entirely on what I want to do. The general semantics of a SOAP message are "Process this". That's actually too general to be useful. Any further detail they could give me would depend on what I tell them about the actual operation I'm trying to perform.

In contrast, the REST guys can actually tell me much more about what my service is going to look like based only on the rough contours of my business function.

They can tell me that what I operate on is a resource that will look like this:

"http://domain/resource"

They can tell me that I will be using one of four verbs (GET, POST, PUT or DELETE).

They can tell me that my response status codes will be one of some 50-odd pre-defined status codes.

Plus they can give me other general tips that will apply, such as a GET operation will have no side-effects, a deferred processing request will return a "202 Accepted" status when successful, a POST request will be to a resource collection, and its response status code will be "201 Created" when successful, and the URI of the newly created resource will be found in a response header called "Location:", etc., etc.

That's actually quite a bit of information considering I haven't said anything yet about the business function I'm trying to implement.

That's what is called a "uniform interface". It's a structured framework within which application designers have to work. Far from being restrictive and limiting, these constraints actually make things easier for the designer because they provide standard patterns that can be reused in similar situations and deliver predictable results.

So far, so good. What the REST side doesn't understand is that SOAP-based Web Services offer a uniform interface too. Obviously not for the application itself, but for qualities of service.

Ask a SOAP service designer how they plan to implement reliable messaging, and they can show you very detailed WS-ReliableExchange headers that are virtually the same regardless of the business service. Ask them about security (encryption and signing/authentication) and they can show you the detailed WS-Security/SecureConversation/Trust headers required, and these too are standard and unchanging.

Security and Reliability are "Qualities of Services". SOAP-based Web Services have standardised these, and are also on track to standardising Transactions through WS-AtomicTransaction and WS-BusinessActivity (the latter deals with real-world transactions that require compensating actions to "roll back"). [Some may argue that these have in fact been standardised, but there isn't yet a WS-I Basic XXX Profile for these, like there is for Security and Reliability, which means there is no benchmark yet for interoperability in these areas.]

This isn't something to be sneezed at, because a uniform interface leads to improved interoperability, which in turn reduces the cost of integration. From my own experience in IT, I know that integration cost forms a very high proportion of the cost of most projects. Indeed, Project Tango proved the interoperability of Java and .NET-based Web Services using declarative statements in policy files to have these QoS headers automatically generated. Now that's cool.

REST promises to reduce integration cost and thereby overall cost, mainly though the mechanism of the "uniform interface". However, REST does not have a standard model for Security or Reliability, let alone Transactions or Process Coordination. SSL is a rigid security mechanism that is adequate for most, but not all situations. And while it's true that Reliability is probably better implemented through patterns like Idempotence than through the TCP-like model followed by WS-ReliableExchange, it requires an application designer to consciously build it. Transactions and Process Coordination are also missing from the REST lineup.

That's the challenge. Both SOAP and REST have standardised a part of the service interface, but not all of it. REST has standardised the interface of the application's core logic. SOAP has standardised the interface for Qualities of Service. What we need is a model that provides us with both.

Spring Integration - Enabling the Enterprise Service Cloud

Inversion of Control is a great concept, and the Spring framework excels at it.

Little did I think that Spring could also effect an Inversion of Architecture! I just recently blogged about the need for organisations to move from a hub-and-spokes ESB (Enterprise Service Bus) to its architectural opposite, a federated ESC (Enterprise Service Cloud).

In the hub-and-spokes model, you run an application (a Service) on the ESB. To 'invert' that architecture and make it federated, you need to embed an ESB into your application! Spring Integration allows us to do just that.

This manual provides a number of examples that explain how the tiny, deconstructed pieces of ESB functionality can be injected into an application as required.

That neatly solves the problem of having to deploy multiple heavyweight ESB servers at every endpoint to front-end each application, which would have been the only way to get federation with that heavyweight approach, - a clearly unacceptable solution even if one ignores licence costs.

I used to think that a federated ESB was the answer and that JBI (Java Business Integration) provided the most flexibility to deploy tailored endpoints. (JBI allows one to plug in specific ESB modules into a common "backplane", but Spring Integration goes a step further and uses the application itself as the backplane.)

The future seems to be here already.

Wanted: An ESC, not an ESB

Here's my fundamental beef with an ESB: It follows a hub-and-spokes model that is more suited to an EAI initiative of the 1990s. In the year 2010, we should be doing SOA, not EAI.

SOA is federated where EAI is brokered. It's the architectural antithesis of EAI. The smarts are pushed out from the middle to the endpoints until there's no "middle" anymore. I wrote about the difference between "federated" and bad old "point-to-point" before. Federated is a standardised smartening of the endpoints.

It doesn't matter whether we do SOAP or REST. Either way, it's SOA, and it's inherently federated.

Both SOAP's Web Service Endpoints and REST's Resources are represented by URIs. URIs offer a "flat" address space. They're completely opaque. DNS can hide where the services physically sit, so any node can talk to any other without the need for an intermediary.

Look at a SOAP message. Look at the headers - WS-Addressing, WS-Security, WS-ReliableExchange. They're all end-to-end protocols. The SOAP engines at the endpoints negotiate and handshake every Quality of Service they need without depending on anything in between.

WS-ReliableExchange uses endpoint-based logic such as acknowledgements, timeouts and retries to provide a measure of reliability even with unreliable transports, much like TCP provides connection-oriented communications over a plain packet routing protocol like IP. There is no need for a "guaranteed delivery" transport like a message queue.

WS-Trust does key exchange, WS-SecureConversation sets up a session key and WS-Security encrypts messages, all in an end-to-end fashion. There is no need for a secure transport.

Any node that talks SOAP and WS-* and conforms to the WS-I Basic Profile (and Basic Security Profile and Basic Reliability Profile) can participate. That's federation.

This isn't an academic discussion. The main drawbacks of a hub-and-spokes model are Availability and Scalability. The ESB is a single point of failure and a performance bottleneck. The normal "solution" is to beef up the ESB by providing for redundancy and "High Availability", but this costs a fair bit and only postpones the inevitable. That's why I said this isn't an academic discussion. There are serious dollars involved. Organisations would do well to think about the implications of an ESB approach and take early steps to avoid the inevitable costs that lie along that path. The correct solution would be to do away with the centralised ESB altogether and embrace the inherently federated model of SOAP (or REST).

Otherwise, we're forcing inherently federated protocols into a hub-and-spokes model, - a needless throwback to an earlier era , - and inviting problems that need to be "solved" at great expense. (Of course, it's a different matter if we use multiple instances of the ESB as brokers close to endpoints, but economics militates against such a topology, so I suspect federated ESBs will not be feasible.)

In the year 2010, when clouds are all the rage, we should recognise that what SOAP and REST give us are "service clouds".

So that's my buzzword for today - ESC (Enterprise Service Cloud)!

The ESB is dead, long live the ESC!

Mediation Doesn't Require Something "In The Middle"

I blogged earlier about the subtle difference between federation and point-to-point connectivity, but didn't spell out how systems suffering from point-to-point connectivity ("the Dark Ages") could be effectively federated to bring them into the Modern Age without having to interpose a broker between them ("the Middle Ages").

Here's the problem - a point-to-point connection between two systems.
Here's the obvious solution - a broker that mediates all communication between the two systems. This is the ESB model, of course.
Is there another decoupling model for a federated system that can (by definition) have no centralised broker?

Consider an application that has the following hyperlink:

http://192.168.2.15/policy-page.html

Now look at this one:

http://intranet.mycompany.com/policy-page.html

If we assume that both links are in fact referring to the same page, what's the difference between them? Yes, we know that in the second case, a DNS lookup is required to resolve the domain name to an IP address before the page can be accessed, but the actual HTTP call to GET the page is exactly the same (it uses the IP address) and the end result of the call is also the same. The HTTP GET seems to be point-to-point in both cases with no intermediaries.

So what's the real difference?

The first is truly an example of a point-to-point link. How do we know? Test it to see how brittle it is. If the intranet is now hosted on a different server with IP address 192.168.2.16, the link breaks immediately.

However, in the second example, if we change the server hosting the intranet but also update the relevant DNS entry so that "intranet.mycompany.com" points to "192.168.2.16" instead of "192.168.2.15", then the link continues to work and the application that embeds it is none the wiser. This is no longer a tight, point-to-point connection. It has been effectively intermediated by the introduction of a DNS lookup before the actual call is made.

This is what lookup-mediation looks like.
As we can see, we don't need to physically interpose a component between two systems to break the point-to-point link between them. We can use a lookup mechanism to achieve the same result.

Incidentally, web-style interaction with domain names in URIs instead of IP addresses is inherently loosely-coupled and therefore the REST style of integration that follows this approach is not an example of point-to-point connectivity. Any node can make HTTP calls to any other, yet nodes are not tightly coupled to each other. This is a practical example of federation.

"Federated" is not "Point-to-Point"

One of the interesting topics in the "to ESB or not to ESB" discussion is the value of avoiding point-to-point interactions between systems. Somewhere along the way, I suspect that mainstream IT has been indoctrinated with the message that point-to-point connections between systems are bad, and in "four legs good, two legs bad" fashion, has begun to parrot the phrase dumbly without understanding what it means.

I realise that most of the goodwill towards centralised brokers comes from that mindless Orwellian mantra. If someone (like me) argues for a federated ecosystem that does not require a component "in the middle" and where any node can talk to any other, the common assumption is that I'm advocating bad old point-to-point connectivity. A lot of the arguments against point-to-point connectivity then get trotted out.

That's not what I'm advocating, and it's very difficult to get people to understand the subtle but important difference between point-to-point connectivity and a federated system. A system that relies on point-to-point connectivity is in the Dark Ages. A system that uses a centralised broker to break those point-to-point connections has progressed to the Middle Ages. But to evolve to the Modern Age, we need to get rid of the centralised broker and move to a federated model, and this does NOT mean a backward step to the point-to-point model, even though communication once again occurs directly between endpoints! How is that possible?

The answer lies in a standardised smartening of the endpoints.

Perhaps an analogy would help.

I'm at work, and I find I need to talk to another person in my organisation to understand some specialised information about Application X. I've never met this person before. What do I do?

I'm not totally lost. There is a common protocol governing what I need to do. I look up the (rough) spelling of this person's name on the company intranet. I see a few names there, and I select the one that I think I need. A page comes up, listing the person's name, job title, department, email address and phone number.

Next I need to decide if the information I need requires an email exchange or a simple phone call. Note that phone and email have given me two communication options - one synchronous and one asynchronous. Even the synchronous option has an asynchronous fallback called voicemail.

Lo and behold! I already have a phone and a laptop on my desk. I look around and see that every employee does. Every employee has the basic capability to make phone calls and send emails to every other employee.

Let's say I decide to pick up the phone and call the person. I know the mechanics of the protocol. I pick up the handset and punch in numbers corresponding to what I see on the person's intranet page. When the person picks it up, I know the etiquette I'm expected to follow.

"Hi, I'm so-and-so. Is this a good time to talk? Thank you. I work in such-and-such an area and I'm currently looking at such-and-such an issue. I find I need to understand a bit about how Application X works, and I'm told you're the best person to talk to."

If it's a simple question, I may just ask it then and there. If the issue is going to require some discussion, I may request a meeting and then send a meeting invite through email after checking the person's calendar to see when they're likely to be free.

If no one picks up the phone, I leave a voicemail with roughly the same information, but also my contact details so the person can get back to me.

What a lot of smarts that took! Yet it's something we take for granted. We call it "business training". We don't expect that a primary school student placed in our position would be able to do all that, yet all of us do it unconsciously. There's a lot of smarts at the endpoints in this model, and no one argues for doing away with it.

Oh, and all communication is in English, of course (I'm in Australia :-). No matter what language I speak at home, I'm expected to speak English when I talk to someone at work. And so are they. It's not considered an unacceptable condition.

Now consider a very different situation. There's no corporate intranet. I have no means of looking up a person. Not everyone has phone and email. Some only have a phone number. Some only have email. Others have neither and I will need to walk across to their desks to talk to them face to face. And I have no way of knowing this up front. If someone isn't available when I try to talk to them, I have no means of leaving a message for them. I have a phone, but I have no idea if they will be able to call me. Oh, and I don't speak English, and neither do they. We speak completely different languages. And no one taught either of us how to use a phone or email or the etiquette of a business conversation! And if by chance I manage to surmount these challenges and discover a way to talk to a certain person, I need to carry that information around with me all the time. I need to do this for the dozen or so people I talk to regularly. Everyone has to do this for all their contacts, - carry individualised communication information around with them. If one of my contacts has their phone number changed, there's no standard way for them to let me know. I simply can't talk to them anymore until I discover their new phone number. This is the work environment from hell (even if the people are the nicest). This is a point-to-point world.

Let's call this the Dark Ages. How could we solve these seemingly intractable communication problems?

Here's the solution! Appoint a liaison officer. If I need to talk to anyone in the organisation, rather than keep in my head the idiosyncrasies of the mode of communication I need to employ with that person, I simply approach the liaison officer with my request, spoken or written out in my own mothertongue. I'm guaranteed a response in two business days. The liaison officer takes care of talking to the other person in the mode of communication best suited to them, with suitable translation to their language. All my communication goes through the liaison officer, and I don't have to know anything about the person I'm dealing with at the other end, - where they're located, whether they use a phone, email or carrier pigeon, what language they speak, - nothing at all! Isn't this progress? Of course it is. We've progressed from the Dark Ages to the Middle Ages. This is a brokered world.

We know this is not the Modern Age because we know what the Modern Age looks like. We live in it - the world of the Standard Operating Environment (SOE) with phones and laptops on every desk, the corporate intranet, the English-only work environment, assumed business training on the part of everyone in the game, etc., etc. In this world, anyone can call anyone at will and do business instantaneously, in an almost effortless manner, with no liaison officers required to mediate our conversations. This is a federated world.

However, here's a crucial issue. If we look back at the work environment that we have called the Middle Ages, and ask ourselves how things could be made better, we're unlikely to get an answer that remotely resembles the Modern Age. We can't get here from there. We're much more likely to be told of problems that still exist in the Middle Ages and of incremental patches required to make them work better.

For example, the liaison officer falls ill once in a while and can't come to work. On those days, work comes to a standstill because people can't communicate anymore. It's an Availability issue.

Some days, there are twenty people huddled around the poor harried liaison officer, all talking at the same time and insisting that their work is very urgent. Everyone is delayed and everyone is unhappy. That's a Scalability problem.

The Middle Ages solution is simple and obvious, of course. Instead of a single liaison officer, institute a Liaison Office staffed with more than one officer. This will provide both redundancy for greater availability as well as scalability. Of course, there will be problems with this model as well. We never seem to have sufficient budget to hire as many officers as we need, so some problems of Availability and Scalability will always remain. Also, if the particular liaison officer I spoke to yesterday is on leave, I need to explain the context of my requirement to another officer in painstaking detail all over again. The Liaison Office really needs to be made more "stateful". So we need to have elaborate procedures for officers to document their interactions with people and do proper handovers before going on leave. The Liaison Office becomes more and more complex and acquires a bureaucracy of its own. But nothing can be done about it because this is "best practice", isn't it?

This is indeed where we are today in the SOA world. Smack in the Middle Ages with a great big honking ESB in the middle that we need to beef up with High Availability at great cost. But to many SOA practitioners (and indeed, to the mass of regular IT folk out there), this is Best Practice. This is the only way.

It seems too hard to imbue all endpoints with the smarts to function in a federated fashion. At the very least, we need to equip every node with a metaphorical phone and email to make it physically possible for any node to call another. Well, the model of endpoint-based SOAP messaging is meant to do just that, with WS-* headers used to provide end-to-end Qualities of Service and with absolutely no need for any intermediary. But that's still just the plumbing. How do we provide business training and teach common business etiquette? In other words, how do we standardise the "application protocol"? Well, that's been done as well, using a model entirely independent of SOAP. It's called REST. REST is simultaneously low in technological sophistication (all we need is a web server and a database) and high in conceptual sophistication (we need to learn to think in a certain way). We have to understand these federated models if we're going to have any hope of progressing from the Middle Ages to the Modern Age.

And that's where we are today. I know that "inherent interoperability" is possible and far superior to having explicit "integration" projects, yet in my day-to-day discussions with colleagues on the way forward, I find that any suggestion that we do away with the ESB in favour of smart endpoints is often greeted with dismay, because many people don't really understand the word "federated". Oh no, we don't want to go back to the Dark Ages and their point-to-point connections!

I hope the 'point' is made.

Heavyweight Services Have Let IT Practitioners Down

Call it coincidence, but I had two meetings in the same day when different IT folk complained that a services approach was too cumbersome for them and argued for their applications to be able to connect directly to the databases of other applications to perform queries, joins, etc.

My involuntary shudder at this suggestion betrayed my SOA leanings, of course. I don't think these people realised how much future pain they would expose the organisation to by building implicit dependencies ("tight coupling") between their independent applications. But I could also empathise with their frustration.

The first generation of services in organisations has been well-meaning in intent, but expensive in practical terms. People ranted to me about the sheer effort overheads involved in trying to access data from elsewhere - setting up queues and queue clients, formalising XML messages to drop onto queues and pick from queues, parsing and generating XML, etc., etc., - all to do something that was essentially a SQL select or a join across two tables (theirs and another app's). I also heard a fair bit about the performance overheads of calling services to do simple things - operations that take seconds to do because they involve 2 or more separate service calls. "Decoupling at what cost?" was the refrain.

I'm forced to the realisation that our collective enthusiasm for SOA, while entirely correct and justified, has not provided a uniformly satisfactory solution to everyday IT practitioners. Bluntly put, we've let down the average developer by making their job unnecessarily harder, to the extent that even experienced designers who know the benefits of loose coupling are beginning to argue for a return to the Bad Old Ways.

A reader's comment on my previous post has made me think that "Data Services", especially of the RESTian CRUD variety, could be the answer. We still have a service interface that decouples applications from each other and their gory data structures and implementation details, but now we can set them up with minimal effort and call them with minimal overhead. Data Services could be the SOA SQL (That's a pretty apt pun, actually, if you pronounce SQL as "sequel").

More and more, I'm leaning to the view that most technical problems within an enterprise can be solved with "a web server, a database and some brains". This is a rich topic for a future post or even a whitepaper, but the unsurprising insight is that the third ingredient is probably the one in short supply at most IT shops ;-). (No insult intended, just that designers and architects tend not to step back and look for solutions from first principles, but rely on precedent and product-based approaches.)

Want Reusability? Sell Off Your ESB and Hire a Data Architect

I'm not being facetious. I'm sometimes subjected to entirely earnest arguments about how services must be implemented within an ESB (Enterprise Service Bus) and not at various endpoints, - in the interests of reusability! Apparently it's OK to host non-reusable services at arbitrary endpoints, but reusable ones need to be wrapped up in centralised wrapping paper and hosted on the magic box in the middle.

The words hogwash, baloney and fiddlesticks come to mind.

I've always been a fan of scalable, federated "endpointware", because I believe that services naturally form a "flat" address namespace that enables them to be hosted in a location-transparent manner, so centralising them within a physical component achieves nothing more than provide the organisation with a nice little performance bottleneck and a single point of failure. Oh, and a fat revenue stream for the vendor who can then address those limitations with an expensive HA (High-Availability) option. When was the last time the Internet was down for maintenance while someone upgraded the central server?

I'm not entirely joking when I suggest that organisations can achieve service reusability more effectively by selling their ESB (to a more gullible one) and using the money to hire a good data architect. After all, IT budgets are always tight and there are usually headcount restrictions, so some sort of swap seems the most feasible way to acquire these skills :-).

I believe most organisations have very poor data modelling skills in XML. Most organisations probably have data modellers who are skilled in relational database technology and can draw a mean ERD (Entity-Relationship Diagram). But I haven't seen too much evidence that any thought is given to XML Schemas as enterprise-reusable resources.

Banking experts! In what way is a savings account similar to a loan account? In what way is it different?

Insurance gurus! In what way is a workers compensation claim similar to a home & contents claim? In what way is it different?

A good data architect should be able to state these similarities and differences - in terms of an XML schema!

A good data architect would be able to define base types for common, abstract concepts and derived subtypes for each specialised variant. This would automatically guide the design of the service interface by constraining the elements and even the structure of message documents. Then regardless of whether the services are hosted on a single node or distributed across multiple disparate nodes and implementation technologies, the organisation will achieve service reuse. Over time, fewer versions of services will need to be maintained (another common bane), because variants have been catered for through insightful data modelling.

I'm not sure this point is appreciated enough. IT professionals are human beings, after all, and tend to understand tangible products better than abstract concepts. That's why most of them believe in magic product-based solutions rather than intellectual and conceptual disciplines. But the architects within IT must rise higher, and make their voices heard.

The alternative is a bunch of non-reusable, non-scalable non-services.

Architectural Requirements for a Line-of-Business Product

I was recently asked for my views as an architect on what factors to consider when procuring a line-of-business product for an organisation. Keeping in mind that business SMEs (Subject Matter Experts) were likely to be covering the functional aspects of the product quite adequately, I came up with this list:

1. Coarse-grained business functions (e.g., underwriting, claims and reinsurance in the insurance industry) should be modular with well-defined service-based interfaces. There should be no implicit dependencies between modules (e.g., no shared tables or other knowledge of another module's internals). This will make it easier for a module to be swapped out for a better one from another source, if required. [Obviously, product vendors will fight to prevent this from happening, but the customer organisation must skew their evaluation rules in such a way that the vendor is forced to see a competitive negative as a positive.]

2. Data models of the interfaces (not the internal domain models) should conform to standards as far as possible (If a data model exists for that industry domain, that's to be preferred). The IP relating to the external data model (the universal metadata model governing the data transfer objects/XML documents) should be revealed to the customer in any case, and guaranteed to be stable across product versions, with a well-defined version strategy for service enhancements. The internal data model can remain the vendor's IP. In fact, the more opaque that is, the better. It will prevent unwitting dependencies from being built by applications external to this one. If a compact and standard vocabulary is established, it will make it much easier for other applications to integrate with this one by "learning" to speak the same language.

3. Technology platform dependencies should be minimal – commodity infrastructure is preferred (e.g., Intel, Linux, plain Java-based web servers). The product should offer choices, not constraints. Ideally, it should be able to run on any platform from mainframes to Intel-based systems, and the customer organisation should choose the most commoditised platform that will suit its needs. If the product is Java-based, it should not require an EJB container. The Spring framework is preferred. Database dependencies should be minimised (wherever possible) through an appropriate ORM layer. Again, the customer organisation should be free to choose the most commoditised database that meets their requirements.

4. Services support should include both SOAP and REST in order to support the widest variety of external service consumers (and producers). SOAP services should be true contract-first services and not automated remoting of object methods. REST services should exhibit support for enterprise requirements through simple patterns for security (SSL), reliability/idempotence, asynchronous processing with callback options, horizontal scalability through statelessness, etc.

5. Technical skills requirements should be as basic as possible (e.g., commonly available skills like Java, XML, HTML, JavaScript, etc.). There should be minimal requirement for specialised skills. Very importantly, the product should not require vendor consultancy on an ongoing basis to make modifications to extend the product's use. Extensibility of the design and flexibility of the service interface are key enablers here.

I'm sure there are other good requirements, but these should provide reasonable guidance, I think. Working through my thought process, I guess these were my guiding principles:

1. Ensure high cohesion and low coupling between functional units, keep interfaces as small as possible (this includes the shared vocabulary).
2. Be conservative in what you produce, be liberal in what you accept.
3. Aim for simplicity (but beware of expediency masquerading as simplicity).
4. Seek to use commodity building blocks as far as possible - try to source technology from competitive markets rather than from "market leaders".

Idempotence - Reliability that Works

In many cases of messaging where uncertainty is undesirable (e.g., a timeout on a critical transaction request), the usual solution sought is "reliable message delivery". Apart from its theoretical impossibility, "reliable messaging" is expensive. One needs "strong queuing" products, which cost a lot of money.

I've been a fan of a far simpler approach - idempotence. We're really not after reliable message delivery in most cases. We'd often be happy to settle for certainty, i.e., we don't mind whether a requested operation succeeded, failed (or indeed, timed out before it could be attempted), as long as we know what happened. What we don't want is to be stuck, not knowing what happened and afraid to retry the operation for fear of unwitting duplication.

Idempotence is, of course, the property that attempting something more than once has exactly the same effect as attempting it once. Idempotent operations can be blindly retried in a situation of uncertainty, because one is guaranteed that the operation will never be duplicated.

The trick is to identify every transaction request with a unique identifier. Provided the receiver of the message is set up to check the identifier against previously-used ones, duplication of transactions can be avoided even if requests are sent multiple times. This is extremely powerful because it allows a requesting application to simply retry the request message in situations of uncertainty until a definite response is eventually received. There is no danger of the request being acted upon more than once.

Reliability is then reduced to an endpoint-based protocol. It does not require any special capabilities on the part of the transport. In fact, the transport can afford to be quite unreliable. Idempotence allows reliable messaging solutions to be built (and quite cheaply at that) on top of unreliable components!

Here's a one-page document that illustrates the concept.


Hopefully this should make it very clear that we don't need strong queuing or "reliable message delivery" to eliminate uncertainty. A plain web server, a database and a system of one-time tokens (UUIDs?) can solve the problem.

The Coming Overthrow of XML - Orderly Makes Further Strides

My feeling that XML is due to be dethroned grows stronger by the month.

A quick recap of recent history:

First, JSON offered a simpler data structure than the angle bracketed format of Unicode XML. But that still lacked rigour around data definition, so even though XML suffered the confusion of having at least three competing schema definition languages (XML Schema, RelaxNG and Schematron), the world did have a way to specify data types, formats and constraints with XML that JSON could not match.

Then, thanks to Kris Zyp, JSON Schema appeared on the scene and plugged the rigour gap. JSON Schema parsers now exist for a number of languages including Java. One of the design decisions of JSON Schema was for a schema document itself to be valid JSON, much as XML Schema is itself valid XML. Unfortunately, this meant that brevity wasn't JSON Schema's strong point because the JSON way of expressing properties is necessarily long-winded.

The third shoe has dropped now (never mind the grotesque image that conjures up of the wearer). Orderly is a new schema language developed by Lloyd Hilaiel that is far more compact than JSON Schema and yet round-trips to JSON Schema quite effectively. [There's a really cool Ajax-y screen that converts back and forth between Orderly and JSON Schema before your eyes, so you can tweak either code to see how it looks in the other representation.]

Bottomline: SOA architects can recommend the use of the simpler JSON data format instead of XML without having to worry about the lack of rigour in data definition. Data architects, designers and developers can use Orderly to design schemas without bothering about JSON Schema's cumbersome syntax. JSON parsers can work with the equivalent JSON Schema to validate a piece of JSON data without the need to understand two different syntaxes.

A great solution, and it's all come together quite nicely in time for Christmas. Thanks to Lloyd (and Kris before him) for a wonderful Christmas present to all SOA practitioners, and ultimately, everyone wrestling with XML in any capacity.

The Meaning of Open - By Google

A friend sent me this link to a piece written by Jonathan Rosenberg of Google on the meaning of the term "open". This is old hat to those of us who have already seen the light, of course ;-). [Rosenberg's calisthenics when he then tries to justify the closed bits of the Google ecosystem are quite amusing.] But to people who have not given much thought to openness and tend to follow the herd on technology (the bigger the brand name, the better), this open letter may hold many eye-opening insights (all puns intended).

I would perhaps have said what Rosenberg did in half the length, but brevity isn't a necessary quality of openness, so I'll forgive him :-). The main danger of the unnecessary length is it may just cause some of the audience to stop reading before the end, when Rosenberg delivers his most inspired paragraph:

Open will win. It will win on the Internet and will then cascade across many walks of life: The future of government is transparency. The future of commerce is information symmetry. The future of culture is freedom. The future of science and medicine is collaboration. The future of entertainment is participation. Each of these futures depends on an open Internet.

Amen to that. In fact, that's worth repeating in a more structured form:

The future of government is transparency.
The future of commerce is information symmetry.
The future of culture is freedom.
The future of science and medicine is collaboration.
The future of entertainment is participation.

I would like to analyse these in greater detail and add to/modify the list, because I'm sure this is incomplete.

For now, this is a document that is worth circulating to our brand name-dazzled colleagues. After all, Google is one of the biggest brands out there, so if Google is endorsing openness, there must be something in it ;-).

Now if only IBM would come out with an Open^TM line of products, we would be willing to write a cheque...

Is Canonical Trying to Purge Ubuntu of the L-word?

I don't think much of revisionist history, and biting the hand that feeds isn't an endearing trait.

Visiting the Ubuntu site today after a while, I was unpleasantly surprised that I couldn't see the word "Linux" anywhere. After trawling the site exhaustively, I did find two or three references, and I leave it as an exercise for the reader to find more. Warning: You'll have to search really hard.

Under "What is Ubuntu?", the site says, "Ubuntu is a community developed operating system that is perfect for laptops, desktops and servers".

What's up, guys? Does it hurt a lot to use the phrase "based on Linux" somewhere in that sentence?

Canonical and Ubuntu, great as their contributions have been, would be nowhere without Linux, especially the Debian distribution. So why not acknowledge that debt? Why try to pass Ubuntu off to newbies as a completely original operating system with no ties to Linux?

On the same page, right at the bottom, there's a section titled "What does Ubuntu mean?" and it goes on to explain, "Ubuntu is an African word meaning 'Humanity to others', or 'I am what I am because of who we all are'."

How apt. Dear Canonical, why not show some Ubuntu (humanity to others) and acknowledge that you are who you are because of what Linux is?

Advice To Organisations Embarking On SOA Today

I have been involved with SOA in various roles over the last two to three years, and my thinking has evolved a fair bit over this period. If I was asked to advise an organisation embarking on a major SOA initiative today, I would probably say this to them:

1. The End Goal: Remember that SOA is not about integration but about inherent interoperability. Think health, not medication. SOA is about raising the capability of your systems such that they can easily and inexpensively integrate with others, not about introducing a new, slick technology that will connect your systems together more easily. Simplifying the components of your enterprise and making them easy to understand and connect to will give you SOA. So keep simplicity at the back of your mind all the time, and don't confuse it with expediency, which is the path of least resistance. Simplification could take some effort.

2. Domain Models: Don't waste too much time searching for "the" canonical data model. Most off-the-shelf ones are too high-level and abstract to be useful. And building your own comprehensive dictionary is wasteful and time-consuming. Instead, identify logical owners of different elements of data and let them own the data dictionary for those elements. All services exposed out of these logical domains should use these definitions and it is the responsibility of service consumers from other domains to understand them. Processes that combine services crossing such domains should perform their own mapping between similar data elements. It's necessarily messy and plenty of out-of-band communication will be required at design time. After all, even similarly-named elements may suffer from subtle interpretation errors, so manual discussion and clarification will always be part of a service or process design.

This is not as bad as it sounds because only a subset of data elements managed by a domain is exposed through its service interface, and it's only these that may need translation to the context of their consumers. Don't look to do away with this manual effort through a single canonical data model. That's a wild goose chase, so don't even start.

3. Infrastructure and Connectivity: Try and avoid using message queues unless you're looking at low latency situations like a trading floor, or if there's simply no other way to interface to a legacy system. The queuing paradigm introduces various coordination issues into application design, and implementing message queues requires establishing more complex patterns to solve these gratuitous problems. [I have a larger, philosophical argument about the need to innovate an application protocol on top of an asynchronous, peer-to-peer transport, but let's not confuse the current set of recommendations with that idea.]

In today's world, HTTP-based communication patterns backed up by databases will often do the trick more simply than expensive message queues. Look beyond the apparent need for reliable message delivery. Often, an idempotent operation will suffice to meet the real requirement, and this is quite a standard pattern to implement. Often, queues are used in synchronous (blocking) patterns anyway (to avoid the coordination problems I talked about), so nothing is being gained in an architectural sense by the use of queues. And even asynchronous communications, where required, can be implemented in standard ways over HTTP, so HTTP is quite a universal protocol to use as the logical infrastructural element for your SOA.

ESBs, Service Directories and other "governance" components are often only required to manage the complexity that they themselves introduce. It's amazing what you can achieve with a farm of simple web servers and a database, and still keep things simple and understandable.

4. Service Enablement: Try and avoid the entire SOAP/WS-* stack of technologies if you can. There is a significant complexity overhead associated with this set of technologies, and you will need an expensive toolset to (partially) simplify its use. Look seriously at REST instead. Even though REST advocates don't make the case strongly enough (and sometimes see SOA as an antithetical philosophy), REST is in fact a valid way to do SOA and can usually help to deliver solutions at much lower cost and complexity. The hard part about doing REST is finding good people who can think that way. REST is subtly different from the SOAP/WS-* approach, even though they may just look like different kinds of plumbing to move XML documents around (and I confess that's the way I initially sell REST to corporate skeptics brought up on a diet of vendor-provided Web Services propaganda).

5. Data Contract: Consider alternatives to XML for the data contract. Though this sounds like heresy, XML is heavyweight and cumbersome, and XML manipulation tools in high-level languages (with the possible exception of E4X in JavaScript) are clumsy to use and suffer major impedance mismatches. You will spend more time wrestling with XML than on the service itself. Although many in the web world will immediately recommend JSON, raw JSON is not sufficient to ensure data integrity, because it has hitherto lacked a strong schema definition capability. Maintain a watching brief on the JSON Schema proposal, submitted for approval as an IETF standard. Already, there are JSON Schema libraries in many high-level languages such as Java. It should be possible to define data contracts with as much rigour as with XML, but at a much lower level of complexity. A newer and more compact JSON Schema representation called Orderly is also maturing, which makes this approach simple as well as easy.

So instead of going down the XML rabbit-hole, start with JSON anyway, and incorporate JSON Schema/Orderly as it matures. You will find this works especially well in combination with REST. A quick Proof-of-Concept may convince the skeptics in your organisation (although the opposite result may also occur, with many going away convinced by the speed of this approach that it's either simplistic or too good to be true!)

6. Web Service Implementation: If you're trapped by circumstances into an XML-and-SOAP/WS-* approach, look at the WSO2 suite of commercially-supported Open Source products. Especially look at the WSO2 Mashup Server. Don't be fooled by the name. It's more than just a mashup server. It's a service orchestration engine that (curiously) uses server-side JavaScript as its programming language. The major advantage of JavaScript is the ability to use the E4X library to perform extremely straightforward XML manipulation. Once you use E4X, you will never go back to JAXB or any other XML-processing library. WSO2 Mashup Server allows SOAP or REST services to be consumed, combined and orchestrated, and in turn exposes SOAP or REST services. It's a good way to hedge your bets if you're only half-convinced about REST. The WSO2 suite is also much less expensive than its proprietary rivals, although the real expense is in the heavyweight approach that it unfortunately shares with them.

7. The Paradox: SOA is really all about simplicity, but it's hard to find SOA architects who seek to simplify systems. Conventional SOA practice seems to be about making integration complex through heavyweight approaches, then introducing tools to manage that complexity, tools that require specialist skills to use properly. If done the conventional way as most SOA consultants seem to agree, your SOA initiative will only leave you with additional complexity to manage.

Of course, if you're politically inclined, you will bask in the prestige of a hefty budget and a large team, and can declare victory anyway on the basis of the number of services and processes you have delivered. But if you want to be really successful at delivering SOA (i.e., making your business more agile and able to operate on a sustainably lower cost basis) while keeping your burn rate low along that journey, you would do well to look at boring, unimpressive and even anticlimactic approaches and technologies such as the ones I've listed above. Give the big vendors a wide berth. You don't need to buy technology (beyond the web servers and databases you already have). You certainly don't need to buy complex technology, which is what the vendors all want to sell you.

And don't let the lack of grandeur in this approach worry you. Complexity impresses the novice, but results are what ultimately impress all.

Postscript: Vendor REST is coming. Beware.

Spring Roo Makes a Quiet Debut

This has always been the Holy Grail (pun definitely intended!): To design an application using just object-orientation as the paradigm, and to have persistence, a service interface and a sensible user interface all generated for you automatically. In other words, just define the core behaviour of the application's objects, and you have a working web application with non-visual service interfaces to boot. This is the Domain-Driven Design dream (DDDD?)

Spring Roo has been talked about for a while, but Grails beat it to market. However, it's better late than never for the product from the SpringSource stables, and Release Candidate 3 is now available.

For a teaser on what Roo can do, see this video. It's a third-party video based on RC2 (and you'll see some syntax differences when you start to play with RC3), but the spirit is the same.

Exciting stuff!

Update 17/11/2009: I sent the video link to a friend and colleague for his feedback. He 's a senior architect and based on his experience, he believes Roo needs to provide an Eclipse-based graphical interface rather than (in addition to?) a command-line interface. With that, he believes it's a winner.

Cisco Buys Jabber - What Does It Mean?

Cisco has bought Jabber (another of those purchases of Open Source companies that make no sense to me - what are they really buying when the IP is public?)

What it does indicate to me is that the Presence and Instant Messaging protocol pioneered by Jabber (XMPP) is now becoming a more widely adopted standard. GoogleTalk and Avaya have been using XMPP for a while now, and Cisco has just joined the bandwagon.

My idea of a Unified Communications architecture is something that leverages SIP/SIMPLE, XMPP, HTTP and the open email protocols, rather than a vendor-proprietary stack. Microsoft and IBM have interesting stories in the UC space, but I would argue that this would be precisely the wrong time for an organisation to go with a proprietary suite of products. There's no such thing as a free pair of handcuffs.

I had drawn up this architecture diagram on UC some time ago but didn't publicise it because it seemed a bit theoretical at the time. But now it seems to be more relevant with the strengthening of XMPP and SIP/SIMPLE.

FST Media Annual Technology and Innovation Seminar Day One (05/11/2009)

I attended FST Media's 4th annual seminar on technology and innovation themed "The Future of Banking and Financial Services" at the Hilton in Sydney.

Some quick impressions:

Don Koch, CEO of ING Direct, delivered the opening keynote and spoke about what customers will want next (answer: who knows?). (Incidentally, every single talk was billed as a "keynote", which leads one to question if FST Media knows what a keynote really is.) He made a number of points but none stuck in my head as anything particularly insightful.

Rocky Scopelliti, General Manager, Financial Services, Industry Development, Telstra Enterprise & Government, spoke next about Gen Y and made some interesting points. Essentially, he was reporting on research by noted Australian sociologist Hugh Mackay. Although Gen Y-ers have a reputation for being difficult to please and notoriously non-loyal, the research suggests that Gen Y-ers are in fact influenced by their parents, the Baby Boomers, who in turn grew up influenced by two opposing forces - on the one hand, abundant and growing prosperity that promised a better tomorrow, and on the other, by the threat of the Cold War and the possibility that there may be no tomorrow. The unique solution to this was the demand for instant gratification - enjoy life like there's no tomorrow. So Gen Y is not really demanding and disloyal; they're just keeping their options open for as long as possible.

I forget whether Koch or Scopelliti made this point, but it was interesting that nowadays, all you need to establish a person's identity is their date of birth and mobile phone number. People tend not to change their mobile numbers, even if they switch providers. More on this later.

David Boyle, CIO, International Financial Services, Commonwealth Bank next spoke about cloud computing at CBA. The interesting aspect of his talk was his call for fellow customers to get together with him and his team to define some basic standards for cloud computing, because he's afraid of the same thing that I am, i.e., that some vendor-proprietary technology will become the de facto standard for cloud computing which will make it unnecessarily pricey.

There were a few eminently forgettable panel discussions (with the only exception being the one on which Dhiren Kulkarni, joint CIO of St George Bank, was being needlessly competitive and obnoxious towards his fellow panelists. He earned a rejoinder at one stage from Pravir Vohra, CTO of ICICI Bank, which was roundly applauded.)

The next talk was billed as an "International Keynote" and was delivered by the aforementioned Pravir Vohra, who provided a few interesting insights into the growth of ICICI Bank. It's been almost 15 years since I left India, and I didn't know ICICI Bank had become the second largest Indian Bank. I guess it would still be a distant second to State Bank of India, though. SBI has more branches than some banks have customers :-).

Vohra would be the envy of CTOs everywhere, because he says he has 25% of the technology budget to play with. I.e., he's free to invest in various technologies and initiatives. If they work out, he "transfer prices" the benefits to the business units that he can serve with it. I have always felt that a similar system is required for Enterprise Utilities. Neither the "first project pays" model nor a Big Bang enterprise rollout in a single financial year is the right one. We need an accounting treatment for Utilities akin to R&D, with carryovers possible across financial years.

I was very pleased to hear that ICICI Bank only uses OpenOffice, not Microsoft Office. Vohra estimates that this choice saves his bank more than $15 million a year, even more than the projected savings from cloud computing! Who said the Third World lags behind the First? Sometimes they lead the way.

One thing Vohra said that didn't seem quite right was his contrast of consumer behaviour in India with that in Australia - he said Indians change their mobile numbers every six months! Apart from the fact that the Indians I know haven't changed their mobile numbers in quite a while, it seems very counter-intuitive behaviour. Why would anyone make life more difficult for themselves in this manner?

The three sessions after lunch and before tea were uniformly missable.

David Cartwright, Group Managing Director, Operations, Technology and Shared Services, ANZ was probably the best of the lot. He spoke about ANZ Bank's "Journey to a Super-Regional Bank". An interesting aspect was the emphasis ANZ Bank places on its new corporate headquarters in Melbourne that has a 6 star green rating. It resembles a skyscraper on its side and is deliberately designed to have a huge floor area on a limited number of floors so as to maximise interaction between people. Another interesting technology they have is "telepresence", which is very high quality videoconferencing, which gives the impression that remote participants are actually in the room on the opposite side of the table.

The aforementioned Dhiren Kulkarni spoke next, and repeatedly parrotted his boss Gail Kelly's favourite phrase "delighting the customer" ad nauseam. You don't get more patronising, defensive, competitive and humourless than this bloke. Not very good PR for Westpac-St George to trot him out at industry seminars.

Rounding out the boring lot was Darren Covington, Vice President, Asia Pacific Japan, Enterprise Software Applications, Hewlett Packard. He spoke about churn and the desirability of retaining customers because it's far cheaper than acquiring new ones (a well-worn concept), but added no new insights. My cynical reaction was that with the Australian banking sector being as oligopolistic as it is, a customer who walks out the door is readily replaced by another bank's customer who walks in. There are only four banks in the game, so where will customers go anyway? It's not so much churn as a nice merry-go-round.

After tea, we had Randy Fennel, General Manager, Engineering and Sustainability, Technology, Westpac, talking about sustainability, which again was a bit ho-hum.

The last session of the day was by far the best. Phil Hopper, founder of iGrin spoke about peer-to-peer lending. This is an extremely interesting concept that I hadn't heard of before. There have been players like Zopa, LendingClub, etc., who have all fallen on hard times now, both because of the Global Financial Crisis and because of some increased regulatory requirements.

Peer-to-peer lending exploits the big spread enjoyed by the large institutional lenders. Banks borrow from individuals at 5% (the deposit interest rate) and offer personal loans to individuals at anywhere over 12%, sometimes going up to 18% in the case of some credit cards. This means that there is a spread of at least 7% here to be exploited if individuals lend direct to each other. A P2P lending marketplace can take 2%, leaving 5% to be split between lender and borrower, with both consequently better off than by going through banks. Hopper described P2P lending as "eBay for money".

He also spoke about Australia currently operating on a model of negative credit data only, which makes it hard to assess the creditworthiness of most people. But there is apparently some privacy-related legislation due shortly that will remove impediments to acquiring (positive) credit data.

Other points were the subtle differences between microfinance, lending to the underbanked, and prime lending. Hopper pointed out that by ceding a degree of control to customers, P2P lending agencies allowed innovation to happen, such as "blenders" (arbitrage players who use their superior credit rating to borrow at lower rates and lend to others at a slightly higher rate), and people who run Self-managed Super Funds who invest (lend) from the fund into the P2P market and then borrow the money right back. Some of this sounds dodgy to me (and to the Tax Office, I'm sure), but it certainly is innovative.

In spite of the temporary problems faced by P2P players including iGrin, Hopper sees a good future for the concept. There are niches (after the Online Auction itself) that can be independently filled by various players, such as Identity Verification, Credit Assessment, Fulfillment/Settlement (both contracts and statements) and Collections. A bill likely to be passed by parliament soon (the Private Properties Securities Bill) will also allow people to securitise their own assets like their property, enabling lending to become more "secured". Once a secondary market develops, it will be possible for the P2P segment to expand from its personal lending base (with a typical term between 12 months and 5 years) into the mortgage market (with terms of upto 30 years). This is because although the typical small lender would not be willing to enter into 30-year loans, the emergence of a secondary market will make this possible, since debt obligations like these can themselves be traded. Wholesale funds are another step along the development of P2P lending.

All in all, a most fascinating concept, and Phil Hopper did an excellent job of presenting it. It seemed a little too slick, though. As moderator and host Michael Pascoe asked him, I wonder if this is a case of amateurs attempting to take on the professionals. To a more pointed question, Hopper defended the eventual viability of the "amateur" P2P market even against the eventual entry of the "professionals", and Pascoe insightfully remarked that he would rather have had Hopper express a desire to be bought out by a bank.

That was the end of the first day.

Rare Home Truths about Windows

I never expected to actually read such news. A senior police officer spoke to members of parliament and candidly told them that they should use Linux if they expect secure Internet banking.

I guess the truth always comes out in the end. Microsoft has lies, hush money and non-tech savvy users on its side, but as Lincoln said, you can't fool all the people all the time.

Of course, I could also tell that we still have a long way to go.

The collection of MPs listening to van der Graaf were very enthusiastic about his suggestion but didn't understand what he meant and asked for clarification.

"You may need to explain further for us," said one MP, while another responded, "yes, we need to understand that".

On the brighter side, knowledge comes from asking questions and finding out more. When lay users find out more about Ubuntu Linux, I doubt if many will stay with Windows. The success of Firefox proves that people are not wedded to Microsoft's products.

JSON Schema becomes more Orderly

I have been convinced for a while that just like REST will gradually displace its more heavyweight SOAP/WS-* equivalent, JSON will slowly displace the mighty XML in its various strongholds (today Web Services, tomorrow the world :-). But to do that, JSON first needs to incorporate some rigour into its definition, using an equivalent to XML Schema, Relax NG or Schematron.

The JSON Schema proposal seemed to fit the bill quite nicely, but I was always vaguely uneasy that it was so verbose. There was probably no escape from that, since one of the requirements was that JSON Schema should itself be valid JSON (otherwise two parsers would be needed to consume a snippet of schema-compliant JSON).

Now along comes another schema syntax for JSON called Orderly, which has the twin advantages of being succinct and being able to round-trip to JSON Schema. The syntax has already been revised with inputs from commenters, and is looking much better in its second version.

Orderly's main advantage is its human-readability and -composability. Its simplicity (with no loss of rigour) will give JSON (and JSON Schema) the impetus they need to challenge XML. If Orderly catches fire, I believe it will accelerate the adoption of JSON for serious service-oriented work.

It's overdue.

REST is Polymorphic CRUD

I've been trying to evangelise REST a bit lately and have had mixed success. There is cautious interest. But there are also huge conceptual hurdles to be overcome. Pete Lacey said it best about enterprise IT folk when it comes to REST: They Can't Hear You.

One architect looked at the definition of a service interface I proposed and thought it a bit "bland". Perhaps it just needed a big WSDL file, lots of XML and SOAP faults!

Another common reaction to REST when it's presented is that "it's just CRUD", with the implication that it's just too fine-grained to be used to create good business services. I've been struggling to explain that just because REST uses four HTTP verbs that correspond roughly to CRUD operations, it doesn't necessarily mean that REST is a CRUD approach to manage data at a very low and detailed level. The resources on which the verbs operate can be arbitrarily coarse-grained. But what has eluded me so far is a succinct term that can drive the point home.

I think I've finally found it - "Polymorphic CRUD".

IT folk in the enterprise understand both polymorphism and CRUD, so the combined term should make sense. I want to drive home the point that a verb itself is neither coarse- nor fine-grained, it's how each resource interprets it. Fine-grained resources will interpret the REST verbs as CRUD operations. But more coarse-grained resources can interpret the verbs as any arbitrary business operation.

Accompanied by the appropriate payload, POSTing to the resource "/applications" is nothing but submitting an application. There's no need for a specific "submitApplication" method.

I've also realised that one can clear a process inbox by DELETEing a "/pending" resource, with a standard WebDAV status code in response (207 Multi-Status), indicating that different items encountered different status codes during the batch process.

It's the way the verb is interpreted by each resource that gives it its meaning in that context. Therefore the REST approach is to manipulate business objects of arbitrary size and complexity through polymorphic CRUD operations.

I hope that gets people to go "Aha!"

The Answer to Complexity

This sounds trite, but long arguments with colleagues in IT have convinced me that this is anything but obvious and therefore needs to be explicitly stated.

Guys, the answer to complexity is not better tooling.

It's simplicity.