"Federated" is not "Point-to-Point"

One of the interesting topics in the "to ESB or not to ESB" discussion is the value of avoiding point-to-point interactions between systems. Somewhere along the way, I suspect that mainstream IT has been indoctrinated with the message that point-to-point connections between systems are bad, and in "four legs good, two legs bad" fashion, has begun to parrot the phrase dumbly without understanding what it means.

I realise that most of the goodwill towards centralised brokers comes from that mindless Orwellian mantra. If someone (like me) argues for a federated ecosystem that does not require a component "in the middle" and where any node can talk to any other, the common assumption is that I'm advocating bad old point-to-point connectivity. A lot of the arguments against point-to-point connectivity then get trotted out.

That's not what I'm advocating, and it's very difficult to get people to understand the subtle but important difference between point-to-point connectivity and a federated system. A system that relies on point-to-point connectivity is in the Dark Ages. A system that uses a centralised broker to break those point-to-point connections has progressed to the Middle Ages. But to evolve to the Modern Age, we need to get rid of the centralised broker and move to a federated model, and this does NOT mean a backward step to the point-to-point model, even though communication once again occurs directly between endpoints! How is that possible?

The answer lies in a standardised smartening of the endpoints.

Perhaps an analogy would help.

I'm at work, and I find I need to talk to another person in my organisation to understand some specialised information about Application X. I've never met this person before. What do I do?

I'm not totally lost. There is a common protocol governing what I need to do. I look up the (rough) spelling of this person's name on the company intranet. I see a few names there, and I select the one that I think I need. A page comes up, listing the person's name, job title, department, email address and phone number.

Next I need to decide if the information I need requires an email exchange or a simple phone call. Note that phone and email have given me two communication options - one synchronous and one asynchronous. Even the synchronous option has an asynchronous fallback called voicemail.

Lo and behold! I already have a phone and a laptop on my desk. I look around and see that every employee does. Every employee has the basic capability to make phone calls and send emails to every other employee.

Let's say I decide to pick up the phone and call the person. I know the mechanics of the protocol. I pick up the handset and punch in numbers corresponding to what I see on the person's intranet page. When the person picks it up, I know the etiquette I'm expected to follow.

"Hi, I'm so-and-so. Is this a good time to talk? Thank you. I work in such-and-such an area and I'm currently looking at such-and-such an issue. I find I need to understand a bit about how Application X works, and I'm told you're the best person to talk to."

If it's a simple question, I may just ask it then and there. If the issue is going to require some discussion, I may request a meeting and then send a meeting invite through email after checking the person's calendar to see when they're likely to be free.

If no one picks up the phone, I leave a voicemail with roughly the same information, but also my contact details so the person can get back to me.

What a lot of smarts that took! Yet it's something we take for granted. We call it "business training". We don't expect that a primary school student placed in our position would be able to do all that, yet all of us do it unconsciously. There's a lot of smarts at the endpoints in this model, and no one argues for doing away with it.

Oh, and all communication is in English, of course (I'm in Australia :-). No matter what language I speak at home, I'm expected to speak English when I talk to someone at work. And so are they. It's not considered an unacceptable condition.

Now consider a very different situation. There's no corporate intranet. I have no means of looking up a person. Not everyone has phone and email. Some only have a phone number. Some only have email. Others have neither and I will need to walk across to their desks to talk to them face to face. And I have no way of knowing this up front. If someone isn't available when I try to talk to them, I have no means of leaving a message for them. I have a phone, but I have no idea if they will be able to call me. Oh, and I don't speak English, and neither do they. We speak completely different languages. And no one taught either of us how to use a phone or email or the etiquette of a business conversation! And if by chance I manage to surmount these challenges and discover a way to talk to a certain person, I need to carry that information around with me all the time. I need to do this for the dozen or so people I talk to regularly. Everyone has to do this for all their contacts, - carry individualised communication information around with them. If one of my contacts has their phone number changed, there's no standard way for them to let me know. I simply can't talk to them anymore until I discover their new phone number. This is the work environment from hell (even if the people are the nicest). This is a point-to-point world.

Let's call this the Dark Ages. How could we solve these seemingly intractable communication problems?

Here's the solution! Appoint a liaison officer. If I need to talk to anyone in the organisation, rather than keep in my head the idiosyncrasies of the mode of communication I need to employ with that person, I simply approach the liaison officer with my request, spoken or written out in my own mothertongue. I'm guaranteed a response in two business days. The liaison officer takes care of talking to the other person in the mode of communication best suited to them, with suitable translation to their language. All my communication goes through the liaison officer, and I don't have to know anything about the person I'm dealing with at the other end, - where they're located, whether they use a phone, email or carrier pigeon, what language they speak, - nothing at all! Isn't this progress? Of course it is. We've progressed from the Dark Ages to the Middle Ages. This is a brokered world.

We know this is not the Modern Age because we know what the Modern Age looks like. We live in it - the world of the Standard Operating Environment (SOE) with phones and laptops on every desk, the corporate intranet, the English-only work environment, assumed business training on the part of everyone in the game, etc., etc. In this world, anyone can call anyone at will and do business instantaneously, in an almost effortless manner, with no liaison officers required to mediate our conversations. This is a federated world.

However, here's a crucial issue. If we look back at the work environment that we have called the Middle Ages, and ask ourselves how things could be made better, we're unlikely to get an answer that remotely resembles the Modern Age. We can't get here from there. We're much more likely to be told of problems that still exist in the Middle Ages and of incremental patches required to make them work better.

For example, the liaison officer falls ill once in a while and can't come to work. On those days, work comes to a standstill because people can't communicate anymore. It's an Availability issue.

Some days, there are twenty people huddled around the poor harried liaison officer, all talking at the same time and insisting that their work is very urgent. Everyone is delayed and everyone is unhappy. That's a Scalability problem.

The Middle Ages solution is simple and obvious, of course. Instead of a single liaison officer, institute a Liaison Office staffed with more than one officer. This will provide both redundancy for greater availability as well as scalability. Of course, there will be problems with this model as well. We never seem to have sufficient budget to hire as many officers as we need, so some problems of Availability and Scalability will always remain. Also, if the particular liaison officer I spoke to yesterday is on leave, I need to explain the context of my requirement to another officer in painstaking detail all over again. The Liaison Office really needs to be made more "stateful". So we need to have elaborate procedures for officers to document their interactions with people and do proper handovers before going on leave. The Liaison Office becomes more and more complex and acquires a bureaucracy of its own. But nothing can be done about it because this is "best practice", isn't it?

This is indeed where we are today in the SOA world. Smack in the Middle Ages with a great big honking ESB in the middle that we need to beef up with High Availability at great cost. But to many SOA practitioners (and indeed, to the mass of regular IT folk out there), this is Best Practice. This is the only way.

It seems too hard to imbue all endpoints with the smarts to function in a federated fashion. At the very least, we need to equip every node with a metaphorical phone and email to make it physically possible for any node to call another. Well, the model of endpoint-based SOAP messaging is meant to do just that, with WS-* headers used to provide end-to-end Qualities of Service and with absolutely no need for any intermediary. But that's still just the plumbing. How do we provide business training and teach common business etiquette? In other words, how do we standardise the "application protocol"? Well, that's been done as well, using a model entirely independent of SOAP. It's called REST. REST is simultaneously low in technological sophistication (all we need is a web server and a database) and high in conceptual sophistication (we need to learn to think in a certain way). We have to understand these federated models if we're going to have any hope of progressing from the Middle Ages to the Modern Age.

And that's where we are today. I know that "inherent interoperability" is possible and far superior to having explicit "integration" projects, yet in my day-to-day discussions with colleagues on the way forward, I find that any suggestion that we do away with the ESB in favour of smart endpoints is often greeted with dismay, because many people don't really understand the word "federated". Oh no, we don't want to go back to the Dark Ages and their point-to-point connections!

I hope the 'point' is made.