Mediation Doesn't Require Something "In The Middle"

I blogged earlier about the subtle difference between federation and point-to-point connectivity, but didn't spell out how systems suffering from point-to-point connectivity ("the Dark Ages") could be effectively federated to bring them into the Modern Age without having to interpose a broker between them ("the Middle Ages").

Here's the problem - a point-to-point connection between two systems.
Here's the obvious solution - a broker that mediates all communication between the two systems. This is the ESB model, of course.
Is there another decoupling model for a federated system that can (by definition) have no centralised broker?

Consider an application that has the following hyperlink:

http://192.168.2.15/policy-page.html

Now look at this one:

http://intranet.mycompany.com/policy-page.html

If we assume that both links are in fact referring to the same page, what's the difference between them? Yes, we know that in the second case, a DNS lookup is required to resolve the domain name to an IP address before the page can be accessed, but the actual HTTP call to GET the page is exactly the same (it uses the IP address) and the end result of the call is also the same. The HTTP GET seems to be point-to-point in both cases with no intermediaries.

So what's the real difference?

The first is truly an example of a point-to-point link. How do we know? Test it to see how brittle it is. If the intranet is now hosted on a different server with IP address 192.168.2.16, the link breaks immediately.

However, in the second example, if we change the server hosting the intranet but also update the relevant DNS entry so that "intranet.mycompany.com" points to "192.168.2.16" instead of "192.168.2.15", then the link continues to work and the application that embeds it is none the wiser. This is no longer a tight, point-to-point connection. It has been effectively intermediated by the introduction of a DNS lookup before the actual call is made.

This is what lookup-mediation looks like.
As we can see, we don't need to physically interpose a component between two systems to break the point-to-point link between them. We can use a lookup mechanism to achieve the same result.

Incidentally, web-style interaction with domain names in URIs instead of IP addresses is inherently loosely-coupled and therefore the REST style of integration that follows this approach is not an example of point-to-point connectivity. Any node can make HTTP calls to any other, yet nodes are not tightly coupled to each other. This is a practical example of federation.

"Federated" is not "Point-to-Point"

One of the interesting topics in the "to ESB or not to ESB" discussion is the value of avoiding point-to-point interactions between systems. Somewhere along the way, I suspect that mainstream IT has been indoctrinated with the message that point-to-point connections between systems are bad, and in "four legs good, two legs bad" fashion, has begun to parrot the phrase dumbly without understanding what it means.

I realise that most of the goodwill towards centralised brokers comes from that mindless Orwellian mantra. If someone (like me) argues for a federated ecosystem that does not require a component "in the middle" and where any node can talk to any other, the common assumption is that I'm advocating bad old point-to-point connectivity. A lot of the arguments against point-to-point connectivity then get trotted out.

That's not what I'm advocating, and it's very difficult to get people to understand the subtle but important difference between point-to-point connectivity and a federated system. A system that relies on point-to-point connectivity is in the Dark Ages. A system that uses a centralised broker to break those point-to-point connections has progressed to the Middle Ages. But to evolve to the Modern Age, we need to get rid of the centralised broker and move to a federated model, and this does NOT mean a backward step to the point-to-point model, even though communication once again occurs directly between endpoints! How is that possible?

The answer lies in a standardised smartening of the endpoints.

Perhaps an analogy would help.

I'm at work, and I find I need to talk to another person in my organisation to understand some specialised information about Application X. I've never met this person before. What do I do?

I'm not totally lost. There is a common protocol governing what I need to do. I look up the (rough) spelling of this person's name on the company intranet. I see a few names there, and I select the one that I think I need. A page comes up, listing the person's name, job title, department, email address and phone number.

Next I need to decide if the information I need requires an email exchange or a simple phone call. Note that phone and email have given me two communication options - one synchronous and one asynchronous. Even the synchronous option has an asynchronous fallback called voicemail.

Lo and behold! I already have a phone and a laptop on my desk. I look around and see that every employee does. Every employee has the basic capability to make phone calls and send emails to every other employee.

Let's say I decide to pick up the phone and call the person. I know the mechanics of the protocol. I pick up the handset and punch in numbers corresponding to what I see on the person's intranet page. When the person picks it up, I know the etiquette I'm expected to follow.

"Hi, I'm so-and-so. Is this a good time to talk? Thank you. I work in such-and-such an area and I'm currently looking at such-and-such an issue. I find I need to understand a bit about how Application X works, and I'm told you're the best person to talk to."

If it's a simple question, I may just ask it then and there. If the issue is going to require some discussion, I may request a meeting and then send a meeting invite through email after checking the person's calendar to see when they're likely to be free.

If no one picks up the phone, I leave a voicemail with roughly the same information, but also my contact details so the person can get back to me.

What a lot of smarts that took! Yet it's something we take for granted. We call it "business training". We don't expect that a primary school student placed in our position would be able to do all that, yet all of us do it unconsciously. There's a lot of smarts at the endpoints in this model, and no one argues for doing away with it.

Oh, and all communication is in English, of course (I'm in Australia :-). No matter what language I speak at home, I'm expected to speak English when I talk to someone at work. And so are they. It's not considered an unacceptable condition.

Now consider a very different situation. There's no corporate intranet. I have no means of looking up a person. Not everyone has phone and email. Some only have a phone number. Some only have email. Others have neither and I will need to walk across to their desks to talk to them face to face. And I have no way of knowing this up front. If someone isn't available when I try to talk to them, I have no means of leaving a message for them. I have a phone, but I have no idea if they will be able to call me. Oh, and I don't speak English, and neither do they. We speak completely different languages. And no one taught either of us how to use a phone or email or the etiquette of a business conversation! And if by chance I manage to surmount these challenges and discover a way to talk to a certain person, I need to carry that information around with me all the time. I need to do this for the dozen or so people I talk to regularly. Everyone has to do this for all their contacts, - carry individualised communication information around with them. If one of my contacts has their phone number changed, there's no standard way for them to let me know. I simply can't talk to them anymore until I discover their new phone number. This is the work environment from hell (even if the people are the nicest). This is a point-to-point world.

Let's call this the Dark Ages. How could we solve these seemingly intractable communication problems?

Here's the solution! Appoint a liaison officer. If I need to talk to anyone in the organisation, rather than keep in my head the idiosyncrasies of the mode of communication I need to employ with that person, I simply approach the liaison officer with my request, spoken or written out in my own mothertongue. I'm guaranteed a response in two business days. The liaison officer takes care of talking to the other person in the mode of communication best suited to them, with suitable translation to their language. All my communication goes through the liaison officer, and I don't have to know anything about the person I'm dealing with at the other end, - where they're located, whether they use a phone, email or carrier pigeon, what language they speak, - nothing at all! Isn't this progress? Of course it is. We've progressed from the Dark Ages to the Middle Ages. This is a brokered world.

We know this is not the Modern Age because we know what the Modern Age looks like. We live in it - the world of the Standard Operating Environment (SOE) with phones and laptops on every desk, the corporate intranet, the English-only work environment, assumed business training on the part of everyone in the game, etc., etc. In this world, anyone can call anyone at will and do business instantaneously, in an almost effortless manner, with no liaison officers required to mediate our conversations. This is a federated world.

However, here's a crucial issue. If we look back at the work environment that we have called the Middle Ages, and ask ourselves how things could be made better, we're unlikely to get an answer that remotely resembles the Modern Age. We can't get here from there. We're much more likely to be told of problems that still exist in the Middle Ages and of incremental patches required to make them work better.

For example, the liaison officer falls ill once in a while and can't come to work. On those days, work comes to a standstill because people can't communicate anymore. It's an Availability issue.

Some days, there are twenty people huddled around the poor harried liaison officer, all talking at the same time and insisting that their work is very urgent. Everyone is delayed and everyone is unhappy. That's a Scalability problem.

The Middle Ages solution is simple and obvious, of course. Instead of a single liaison officer, institute a Liaison Office staffed with more than one officer. This will provide both redundancy for greater availability as well as scalability. Of course, there will be problems with this model as well. We never seem to have sufficient budget to hire as many officers as we need, so some problems of Availability and Scalability will always remain. Also, if the particular liaison officer I spoke to yesterday is on leave, I need to explain the context of my requirement to another officer in painstaking detail all over again. The Liaison Office really needs to be made more "stateful". So we need to have elaborate procedures for officers to document their interactions with people and do proper handovers before going on leave. The Liaison Office becomes more and more complex and acquires a bureaucracy of its own. But nothing can be done about it because this is "best practice", isn't it?

This is indeed where we are today in the SOA world. Smack in the Middle Ages with a great big honking ESB in the middle that we need to beef up with High Availability at great cost. But to many SOA practitioners (and indeed, to the mass of regular IT folk out there), this is Best Practice. This is the only way.

It seems too hard to imbue all endpoints with the smarts to function in a federated fashion. At the very least, we need to equip every node with a metaphorical phone and email to make it physically possible for any node to call another. Well, the model of endpoint-based SOAP messaging is meant to do just that, with WS-* headers used to provide end-to-end Qualities of Service and with absolutely no need for any intermediary. But that's still just the plumbing. How do we provide business training and teach common business etiquette? In other words, how do we standardise the "application protocol"? Well, that's been done as well, using a model entirely independent of SOAP. It's called REST. REST is simultaneously low in technological sophistication (all we need is a web server and a database) and high in conceptual sophistication (we need to learn to think in a certain way). We have to understand these federated models if we're going to have any hope of progressing from the Middle Ages to the Modern Age.

And that's where we are today. I know that "inherent interoperability" is possible and far superior to having explicit "integration" projects, yet in my day-to-day discussions with colleagues on the way forward, I find that any suggestion that we do away with the ESB in favour of smart endpoints is often greeted with dismay, because many people don't really understand the word "federated". Oh no, we don't want to go back to the Dark Ages and their point-to-point connections!

I hope the 'point' is made.

Heavyweight Services Have Let IT Practitioners Down

Call it coincidence, but I had two meetings in the same day when different IT folk complained that a services approach was too cumbersome for them and argued for their applications to be able to connect directly to the databases of other applications to perform queries, joins, etc.

My involuntary shudder at this suggestion betrayed my SOA leanings, of course. I don't think these people realised how much future pain they would expose the organisation to by building implicit dependencies ("tight coupling") between their independent applications. But I could also empathise with their frustration.

The first generation of services in organisations has been well-meaning in intent, but expensive in practical terms. People ranted to me about the sheer effort overheads involved in trying to access data from elsewhere - setting up queues and queue clients, formalising XML messages to drop onto queues and pick from queues, parsing and generating XML, etc., etc., - all to do something that was essentially a SQL select or a join across two tables (theirs and another app's). I also heard a fair bit about the performance overheads of calling services to do simple things - operations that take seconds to do because they involve 2 or more separate service calls. "Decoupling at what cost?" was the refrain.

I'm forced to the realisation that our collective enthusiasm for SOA, while entirely correct and justified, has not provided a uniformly satisfactory solution to everyday IT practitioners. Bluntly put, we've let down the average developer by making their job unnecessarily harder, to the extent that even experienced designers who know the benefits of loose coupling are beginning to argue for a return to the Bad Old Ways.

A reader's comment on my previous post has made me think that "Data Services", especially of the RESTian CRUD variety, could be the answer. We still have a service interface that decouples applications from each other and their gory data structures and implementation details, but now we can set them up with minimal effort and call them with minimal overhead. Data Services could be the SOA SQL (That's a pretty apt pun, actually, if you pronounce SQL as "sequel").

More and more, I'm leaning to the view that most technical problems within an enterprise can be solved with "a web server, a database and some brains". This is a rich topic for a future post or even a whitepaper, but the unsurprising insight is that the third ingredient is probably the one in short supply at most IT shops ;-). (No insult intended, just that designers and architects tend not to step back and look for solutions from first principles, but rely on precedent and product-based approaches.)

Want Reusability? Sell Off Your ESB and Hire a Data Architect

I'm not being facetious. I'm sometimes subjected to entirely earnest arguments about how services must be implemented within an ESB (Enterprise Service Bus) and not at various endpoints, - in the interests of reusability! Apparently it's OK to host non-reusable services at arbitrary endpoints, but reusable ones need to be wrapped up in centralised wrapping paper and hosted on the magic box in the middle.

The words hogwash, baloney and fiddlesticks come to mind.

I've always been a fan of scalable, federated "endpointware", because I believe that services naturally form a "flat" address namespace that enables them to be hosted in a location-transparent manner, so centralising them within a physical component achieves nothing more than provide the organisation with a nice little performance bottleneck and a single point of failure. Oh, and a fat revenue stream for the vendor who can then address those limitations with an expensive HA (High-Availability) option. When was the last time the Internet was down for maintenance while someone upgraded the central server?

I'm not entirely joking when I suggest that organisations can achieve service reusability more effectively by selling their ESB (to a more gullible one) and using the money to hire a good data architect. After all, IT budgets are always tight and there are usually headcount restrictions, so some sort of swap seems the most feasible way to acquire these skills :-).

I believe most organisations have very poor data modelling skills in XML. Most organisations probably have data modellers who are skilled in relational database technology and can draw a mean ERD (Entity-Relationship Diagram). But I haven't seen too much evidence that any thought is given to XML Schemas as enterprise-reusable resources.

Banking experts! In what way is a savings account similar to a loan account? In what way is it different?

Insurance gurus! In what way is a workers compensation claim similar to a home & contents claim? In what way is it different?

A good data architect should be able to state these similarities and differences - in terms of an XML schema!

A good data architect would be able to define base types for common, abstract concepts and derived subtypes for each specialised variant. This would automatically guide the design of the service interface by constraining the elements and even the structure of message documents. Then regardless of whether the services are hosted on a single node or distributed across multiple disparate nodes and implementation technologies, the organisation will achieve service reuse. Over time, fewer versions of services will need to be maintained (another common bane), because variants have been catered for through insightful data modelling.

I'm not sure this point is appreciated enough. IT professionals are human beings, after all, and tend to understand tangible products better than abstract concepts. That's why most of them believe in magic product-based solutions rather than intellectual and conceptual disciplines. But the architects within IT must rise higher, and make their voices heard.

The alternative is a bunch of non-reusable, non-scalable non-services.