SOA and MDM - Friends or Enemies?

SOA (Service-Oriented Architecture) and MDM (Master Data Management) are two terms that are often encountered in Enterprise IT. I've found a fair amount of confusion over how these two disciplines overlap, if at all, and whether their principles are compatible with or antithetical to each other.

One of the comments I heard went like this

MDM subverts SOA principles because it connects implementation to implementation. It bypasses the business layer with all its rules and validation, and could create a massively coupled mess.

I have a more optimistic view. I think the very opposite is the case. Business rules should only apply when updating the source of truth for any data item. That is emphatically not being compromised by MDM, because MDM explicitly identifies sources of truth and replicas, and only controls the update of replicas whenever the corresponding sources of truth change.

In fact, MDM could help to clean up the mess that may exist with incumbent systems. Many current systems apply business logic redundantly wherever the same data is updated. But common sense will tell us that updating replicas should be a no-no, and putting a duplicate layer of business logic over it is not a solution. It's far better to remove the business logic altogether from around the update of replicas (shocking as it may sound), because MDM can provide a cleaner logic to govern the update, i.e., that replicas are never independently updated but only refreshed when the source of truth is updated. With MDM, business logic around updating a data item only needs to be in a single place - where the source of truth is updated.

I drew this diagram to try and show that SOA and MDM are complementary organising principles that an enterprise could use to advantage.

[There is sometimes the situation that two data stores hold different horizontal subsets of the same data, such as a database that holds details of Sydney customers and another that holds similar details of Melbourne customers. If these two data stores are mastered by different systems, it may be acceptable to duplicate the business logic that governs updates. However, MDM doesn't come into the picture unless the data is actually being replicated, such as when the Sydney database also holds copies of Melbourne customer data and vice-versa. There would be no need to apply business logic to this replicated data because it has already run that gauntlet once at the source of truth.]

Tight Coupling in the TCP/IP Stack!

Scandal and embarrassment!

The TCP/IP stack of Internet protocols, the poster child for a layered architecture with well-defined responsibilities and interfaces that abstract out needless dependencies, has a dirty little secret that I just stumbled upon since I now work for a telco. But before I tell you what it is, a quick recap of how the technology works.

IP stands for Internet Protocol. Every device has an IP address. [IP version 4 has been the most common so far, and IPv4 addresses look like this: 192.168.1.1 (or in hex, C0.A8.01.01). IP version 6 (IPv6)  addresses look like this: 3ffe:1900:4545:3:200:f8ff:fe21:67cf.]

The way the Internet works is by routing packets of information, hop-by-hop, from a source to a destination. Each node along the way knows, by looking at the destination IP address in a packet, how to forward that packet so it gets one step closer to its destination. So all that an IP network really has are routing smarts. It's the destination's IP address in a message that holds all the information required for it to reach its intended audience.

That's great when the sources and destinations of messages are fixed in location. They have a certain IP address assigned to them when they start up, and from then on, that IP address typically doesn't change until they next start up.

Mobile data devices (which include 3G mobile phones and later devices that use the packet-switched data network) have introduced a problem. Their IP addresses need to keep changing because they connect to different nodes (or cells, or towers) as they move, and it would play havoc with routing if they carried their original IP addresses around when connecting to new nodes. So fine, the technology allows for their IP addresses to change dynamically. However, the logical data connections that the devices establish need to remain for the duration of the session. There could be a download going on, for example, and an interruption of the connection will abort the download. Innovations such as "fast mobile IP" were introduced to mitigate the visible effects of the problem, but did not address its root cause.

The root cause lies in a rather ugly fact about IP addresses. An IP address confuses a device's identity with its location. A device's location keeps changing as it moves, but its identity does not change. A location is important to know where packets are to be delivered. But logical concepts like sessions need to be tied to a device's identity, not to its location. These are two different concepts, but a single mechanism (the IP address) has been chosen to implement both of them. As long as the location and identity did not independently change, the design flaw remained hidden. Now with data-enabled mobile devices, device location and device identity show themselves very clearly as two different things, and the conceptual limitation of the IP address has therefore been exposed.

That's the rationale behind the new protocol specification called HIP (Host Identity Protocol). HIP is meant to sit between TCP and IP. Normally, a TCP-level domain name is resolved by DNS to an IP address. A whole generation of IT professionals has come of age with this principle internalised as an axiom of How Things Work. HIP is a Copernicus or a Galileo challenging an established view. The Sun does not go round the Earth, after all. It's the Earth that goes round the Sun! That's going to take some getting used to. For a networking professional or a web architect, discovering that the venerable TCP/IP stack should actually be the TCP/HIP/IP stack is a bit like discovering that they're an adopted child. But however painful the realisation and readjustment, it's better that the truth be known.

Under the new proposal, a TCP-level domain name needs to be resolved by DNS to a logical HIP name, which then gets further resolved to an IP address! Now, if a device is moving, its IP address can keep changing, but its HIP name will remain the same. Therefore TCP connections need not be torn down and re-established. Sessions need not be dropped and re-created.

RFC 4423 (HIP Architecture) says:

In the current Internet, the transport layers are coupled to the IP addresses. Neither can evolve separately from the other.
[...]
There are three critical deficiencies with the current namespaces. First, dynamic readdressing cannot be directly managed. Second, anonymity is not provided in a consistent, trustable manner. Finally, authentication for systems and datagrams is not provided. All of these deficiencies arise because computing platforms are not well named with the current namespaces.

It goes on to say:

An independent namespace for computing platforms could be used in end-to-end operations independent of the evolution of the internetworking layer and across the many internetworking layers. This could support rapid readdressing of the internetworking layer because of mobility, rehoming, or renumbering.

Amazing, isn't it? We've been nursing a tightly-coupled serpent in our collective bosom for over 3 decades, and we didn't even know...

It's going to take a while for HIP to become part of the Internet ecosystem (if it ever will!) The power of entrenched ways of thinking could prove too powerful to allow a much-needed rationalisation.

The lesson for me personally is that if we don't architect a system right, we will live with its negative implications for a long, long time. Even the founding fathers of the Internet, geniuses as they were, were not perfect, and we can clearly see in hindsight how a conceptual blunder (a conflation of location with identity) has impacted us.

I do believe though, that even the current HIP proposal is making a blunder of its own by confusing identifiers with identity credentials. RFC 4423 says:

In theory, any name that can claim to be 'statistically globally unique' may serve as a Host Identifier. However, in the authors' opinion, a public key of a 'public key pair' makes the best Host Identifier. As will be specified in the Host Identity Protocol specification, a public-key-based HI can authenticate the HIP packets and protect them from man-in-the-middle attacks. 

From my own work on Identity Management, I have come to realise that multiple sets of credentials can be used to arrive at, or establish, an identity. The establishment of an identity within a given context requires an identifier. This identifier may be the credentials themselves, or something else. It's important to realise that the "may be" should not be taken as a "must be". For the purpose of security, the authors of the HIP specification are proposing that verifiable credentials be used as the identifier in all situations. I fear that will result in a similar problem later on when the requirements of authentication and identity establishment diverge in some context. I'll write to the committee explaining my concerns.

The Unholy Alliance of Analyst and Big Vendor

I was browsing IT news as usual and came across a commentary page that discussed Progress Software's recent decision to divest itself of several core products, including Sonic ESB, Savvion BPM and Actional Service Management. To tell you the truth, I wasn't paying attention to the website where this page was hosted. But then I came upon this statement:

The situation reminds us of a key benefit of selecting a top-tier enterprise vendor: IBM and SAP rarely kill established products, and the same can be said of Oracle in recent years.

I've been in the industry long enough to detect a "this-message-brought-to-you-by-your-friendly-neighbourhood-commercial-sponsor" marketing insert when I see one. So I glanced up at the website, and sure enough, it was one of the big analysts, Forrester to be precise.

The funny thing is, I would have drawn a very different inference from the news about Progress.

This news confirms a few things I already know:

1. Commoditisation continues its onward march in product category after product category. This is good news for customers because it means lower prices as well as standardisation of features (which in turn leads to greater interoperability and lower operating costs).

2. Commoditisation is dreaded by vendors because it erodes their profit margins. The large vendors have enough of a lock on their customers through network externalities that they can often maintain their profit margins in spite of commoditisation. But smaller commercial vendors must either exit these market segments or accept lower margins.

3. The unspoken trend in any modern-day commoditisation story is the rise of Open Source. After all, Progress is not quitting these middleware markets because of competition from the big vendors. The pressure on them is from below, from Open Source. And there is a thriving market here in the "supported Open Source" category, as this gleeful blog post from WSO2's VP of Technology Evangelism shows.

I'm disappointed (but not surprised) that a big analyst is spinning what is really a welcome story of commoditisation into a warning to customers to buy the big expensive brands, or else. I guess there's no money in it for them to recommend that customers choose Open Source alternatives to start with.

But really, chee, how low can they go?

Mentoring NoSQL From Adolescence to Maturity

NoSQL is more than just the flavour of the month. There's no doubt that it's here to stay. But the movement is now experiencing growing pains.

In short, most variants of NoSQL have established a niche for themselves by creatively dropping one or more of the ACID constraints of traditional relational databases. Consequently, what they've gained on the swings in terms of features, they've lost on the roundabouts. Today, some of those shortcomings are becoming pain points, and the respective projects are attempting to layer the missing features on top of their existing products.

An expert view is that this is the wrong approach to take. Databases are complex beasts, and their features cannot be layered on, but engineered concurrently. It's a non-intuitive insight, but one that the greybeards of the industry have learnt through hard experience over decades.

It's very timely that one of the most respected names in the database field, IBM Fellow C Mohan, has stepped up to provide much-needed leadership and guidance to the NoSQL movement. His initial analysis and critique of NoSQL is on his blog (Part 1, Part 2, Part 3 and Part 4).

Mohan has promised to study individual NoSQL databases in more detail so as to understand their design nuances better. If he can then propose ways for these projects to enhance their capabilities in the most effective way, he would have succeeded in enabling a whole new wave of applications.

Here's wishing him all success!

An Optimistic Approach to Identity

I've been working in the Identity Management area for a few years now, and I've seen three different industries up close (banking, insurance and telecom). What I'm struck by in all these industries is that none of them has historically been customer-centric in their business approach. For decades, banks have always looked at their customers through the prism of accounts, insurance companies through policies, and telecom companies through billing accounts and sometimes carriage services (broadband or mobile services). And everywhere, the holy grail is the same - "single view of customer". Identity and Access Management (IAM) is the way these organisations aim to achieve single view of customer as well as other benefits.

However, IAM initiatives at organisations in all these industries have generally floundered. Why?

I believe that IAM is simple but subtle. That's why although it's not hard to design and deliver an IAM system, it's also treacherously easy to get it wrong.

Some of the major reasons why organisations struggle with IAM are these:

1. Rather than bite the bullet and create a top-level data entity called "customer" with its own unique identifier, organisations choose what they consider a cheaper compromise because of a misplaced belief that using a surrogate for customer (i.e., account, policy, billing account) would somehow do the job. Reality check: it doesn't, and it's more expensive in the long run.

2. Even where identifiers are created for customers, these are not carefully designed. The result is that many identifiers that are chosen have business meaning. It's quite funny at one level to see a system designed with a person's email address as their identifier, and where the major business pain point is that it's very hard to handle the situation where a customer changes their email address. (Why are we not surprised?) Quite often in such cases, there is no other way around the problem but to delete and re-create the customer record.

3. Even where organisations avoid the first two mistakes and embark on an IAM initiative to tie customer data across multiple systems to a new, unique and meaning-free customer ID, they run into logistical problems relating to the existing user base. They struggle to "marry" records across systems to the appropriate customer entity because of the sheer volume of data involved, the cost of changing existing systems, the unreliability of matching algorithms and the need to replace engines while the plane is flying, so to speak. The two problems with matching algorithms are false positives (two or more customers being assigned the same identifier) and false negatives (a customer being assigned two or more identifiers).

I have some suggestions that can make life easier.

1. Create a database external to all existing systems that will maintain mappings. [Resist the temptation to migrate customer attributes from other systems to this one. This is just a mapping database, not a customer master. Use Master Data Management (MDM) principles instead to keep data in source systems in sync.]

2. Use a universally unique and meaning-free identifier for customers. Version 4 (random) UUIDs are a great scheme to use.

3. Adopt an optimistic model of "eventual consistency". I.e., generate a new customer UUID corresponding to every system record, in effect assuming (in the case of a bank) that each account belongs to a different customer, then pare them down to reflect known relationships. 

a) You can generate UUIDs for a system in an optimistic way because the probability of two UUIDs conflicting is infinitesimally low, even if you have hundreds of millions of customers. You can check for duplicates out of band if you're paranoid.

b) Similarly, you can optimistically generate UUIDs in a federated way (i.e., each system generates its own UUIDs corresponding to its surrogate records). The probability of conflict is so low it's worth doing this and checking for duplicates out of band.

c) You can afford to start with a system with a large number of false negatives (but no false positives) because this corresponds to a siloed organisation with no "single view of customer". False positives are a greater danger, and we avoid that with this scheme.

d) You can use the existing intelligence in your systems (i.e., the knowledge of which records belong to the same customer) to merge customer UUIDs relating to the same physical customer by eliminating all but one of them at random. Since UUIDs are meaningless, it doesn't matter which one you keep and which ones you remove.

Now you're no worse off than you were before in terms of data quality (i.e., your data is just as clean in terms of known relationships). But structurally, you're far better off because you now have a customer data entity for the first time. As your data quality improves with more reliable mappings, the siloes effectively disappear and you get to a "single view of customer" with no more changes to data structures or processes.

In the case of a telecom company, your mapping database will now consist of three parts. The first part will map customer UUIDs to billing accounts. The second part will map customer UUIDs to product holdings (mobile, broadband and other carriage services, media products, etc.) The third part will map customer UUIDs to other customer UUIDs to reflect corporate organisational structures and household relationships. With this model, the many problems that telecom companies currently face will simply melt away.

- We can see all the product holdings of a customer to determine what else to sell them. We can see this at an individual customer level as well as at the level of a household or organisational unit.

- We can sell media products even to customers who haven't purchased an underlying carriage service

- We can group billing accounts independently of product holdings. In a household, the kids use various products but mum or dad alone may pay the bill.

As you can see, this kind of design isn't hard. But it requires conceptual clarity around the nature of Identity. As I said before, IAM is simple but subtle. It isn't hard to design and deliver an IAM system, but it's treacherously easy to get it wrong.

Dimensions of Decoupling

I was in a meeting at work discussing deployment strategies for various SOA components. Let me take a subset of the problem as today's topic.

The issue was how to deploy a bunch of ESB instances and App Server instances onto server boxes. One of the infrastructure guys said he preferred to have one ESB instance and one App Server instance on each box for ease of administration. Now, since we were going to run the instances on virtual servers, I suggested that we not worry about it at a SOA topology level. We would only talk about virtual servers running ESB instances and virtual servers running App Server instances. If the infrastructure guy wanted, he could always run a virtual server of each type on each physical box and would then get what he wanted. He had the ability to tune the configuration, allocating 0.5 CPUs to each virtual server, etc.

He objected to that idea, saying the performance of the virtual servers was bad if he deployed them like that. He wanted to deploy one instance of the ESB and one instance of the App Server on each virtual server.

I said, "OK, then deploy the two instances on the same virtual server, but don't assume that they're on the same virtual server."

This statement was a bit too cryptic for the others in the room, and they asked me to explain. This is how I explained it:

Assume that you have to write a deployment script that installs one instance of the ESB and one instance of the App Server on a virtual server. There are at least two ways you could do it.

Script option 1:

DEPLOY_ADDR=192.168.1.2

# Deploy ESB to $DEPLOY_ADDR

# Deploy App Server to $DEPLOY_ADDR

Script option 2:

ESB_DEPLOY_ADDR=192.168.1.2

AS_DEPLOY_ADDR=192.168.1.2

# Deploy ESB to $ESB_DEPLOY_ADDR

# Deploy App Server to $AS_DEPLOY_ADDR

In both cases, the scripts will deploy one instance of the ESB and one instance of the App Server onto a single server.

However, in the first script, the two servers are assumed to be the same. In the second, they happen to be the same. That is the difference between tight coupling and loose coupling.

In the second script, a simple change in the value of (say) AS_DEPLOY_ADDR to 192.168.1.3 will see the two instances running on different servers. This is not possible with the first script. Changing DEPLOY_ADDR to 192.168.1.3 will move *both* instances to the new server address but will not separate the two instances.

I was therefore recommending the approach exemplified by the second script - deploy the two instances on the same server if you must, but don't hardcode the assumption that they are on the same server into your scripts.

It took a while for this concept to sink in, but the idea was finally accepted.

I guess a decade of SOA experience has sensitised me to looking out for needless dependencies, but most people in IT still don't think this way. I wonder how much rigidity and subsequent operational inefficiency is in IT systems all over the world because people are not sensitised towards the elimination of needless dependencies.

Identity Management on a Shoestring - eBook Published by InfoQ Today

I've written before about the white paper on Identity Management that I co-authored with Umesh Rajbhandari. We have been in discussions with InfoQ about publishing it as an eBook, and after a long edit process (mainly due to delays from my side), this has finally made its appearance.

Anyone interested can download the PDF here after (free) registration.

According to the publishing agreement, we have to point back to the InfoQ site for the soft copy, so I have removed all links from my blog to the original site where I had hosted the white paper. The latest version has some additional material that wasn't in the original white paper, so it makes more sense to go to the InfoQ site in any case.

I hope this book will prove useful to practitioners embarking on implementing an Identity and Access Management (IAM) system for their organisations. We believe we have pioneered a loosely-coupled approach to IAM that is unique and delivers significant benefits compared to off-the-shelf commercial alternatives, and we call this LIMA (Lightweight/Low-cost/Loosely-coupled Identity Management Architecture).

My "Goldilocks Logic Quadrant" as a Superior Alternative to Gartner's Magic Quadrant

I must confess I don't have a great opinion of Gartner, Inc., or indeed, most technology analyst groups. It is my belief that analysts overly complicate technology to appear expert, and collude with vendors to move kit by convincing customers that they need the useless bells and whistles that vendor products feature. [As a particular pet peeve, I will never forgive Gartner for overcomplicating and fatally crippling SOA, with their reference models, the baroque cathedral of "SOA Governance" and execrable offshoots like SOBA and SOMA that nobody can understand or explain. Which is why I wrote the Practical SOA white paper in collaboration with WSO2. Update 28/11/2012: I've also written a white paper on SOA Governance.]

Lest this appear too broad a criticism, here is a more nuanced picture of what I think of Gartner's contributions overall:

1. Their categorisation and classification of players and sub-domains within a technology domain is usually very comprehensive, and useful in acquiring a quick mental picture of what the domain represents. It would be churlish on my part not to thank them for aiding my understanding on numerous occasions.

2. Their Hype Cycle is moderately useful, because (in addition to the concept of a Hype Cycle itself being valid and very insightful) it helps to position current and future technologies, even if the specific placements of some of them on the curve are often questionable.

3. Their predictions about the future (along with the ridiculous probability figures attached) are often plain wrong. Their crystal ball is in fact a rear-view mirror. [In the previous decade, Linux and Open Source left them with egg dripping from their face year after year, and they were obviously playing catch-up with reality.]

4. Their Magic Quadrant is positively harmful to customers.

It's the last of these that I want to talk about today.

The Gartner Magic Quadrant is an article of faith among corporations (including most of my employers over the past decade). It is the first port of call when evaluating products for purchase in a given category. There are many reasons why thinking architects hate the malign influence that the MQ has over corporate drones. The Gartner MQ is a convenient and backside-covering way to outsource one's brains when embarking on something as risky as sourcing a new technology. Just like nobody ever got fired for buying IBM, it appears nobody ever got fired for selecting a vendor from the Leader quadrant of the Gartner MQ.

It wouldn't be so bad if the MQ actually did a good job of analysing and categorising the vendors in a given market segment. It doesn't. It's horribly biased and reactionary. Many good technology options never get on the diagram. It doesn't consider hybrid end-to-end solutions, which are often the best answer for a customer. And I won't even go into the industry whispers about vendors buying their way onto the diagram, though that's a fairly common complaint.

For those who aren't already familiar with it, the Gartner Magic Quadrant has two axes, as shown below.

On the x-axis is "Completeness of Vision". Call me cynical, but I see this as a euphemism for "Overly complex and a checklist-ticker's dream".

On the y-axis is "Ability to Execute", which I see as a euphemism for "Big vendor".

The four quadrants represent Leaders, Visionaries, Challengers and Niche Players.

The Leadership quadrant, which is supposedly the unadulterated goodness of the IT industry crammed into one little corner, is nothing but the collection of the most bloated and expensive technology sold by the most rapacious corporations. That's what I mean by the unholy alliance of analyst and vendor.

I propose a more useful variant on the Gartner Magic Quadrant. I call it the Goldilocks Logic Quadrant, with tongue firmly in cheek. But though the name is whimsical, I'm deadly serious about the tool itself. I use the name "Goldilocks" because of its focus on what is "just right", as with all of Baby Bear's things. Qualities on either extreme ("Papa Bear" and "Mama Bear"  characteristics) are both undesirable in comparison.

In place of "Completeness of Vision", I would use "Capability Fit". Unlike with Gartner's MQ, more features aren't necessarily better in the Goldilocks Logic Quadrant. What we prize is the minimal capability that will meet our needs. Anything more increases the complexity of our ecosystem with no additional benefit, so it earns negative points. This is about "just enough IT", as a good friend of mine puts it.

Similarly, in place of "Ability to Execute", I would use "Solution Viability". I don't care if a vendor can do this or that. What I'm interested in is whether the technology when deployed within my environment can sustain itself without giving me headaches. In other words, is it supported now and into the foreseeable future, and is such support inexpensive? Unlike with Gartner's Magic Quadrant, a popular Open Source technology with a thriving user community and lots of third-party organisations willing to support it commercially at competitive rates would score highly on this axis of the Goldilocks Logic Quadrant.

If a technology solution is minimally adequate (it does the job with no frills) and is supportable at reasonable cost, then it represents Value for Money.

If it is minimally adequate but inadequately supported, then it could be considered a Quick-and-Dirty solution. If minimally adequate but overly expensive, then it's an embarrassment to justify to the bean-counters. Such solutions are therefore either Risky or a Rip-off.

If a technology solution is viable and inexpensive, it tends to remain in place because there is no economic incentive to remove it. If it is functionally inadequate, it's viewed as a Tactical Stopgap. If it is overly complex, then it tends to be cursed - a Whipping Boy, Either way, it's an Irritant because the business case to replace it is hard to make on account of its favourable economics.

There are many ways in which the worst aspects of Capability Fit and Solution Viability intersect. A functionally inadequate solution that is inadequately supported is little more than Demoware. If support for such an inadequate solution is overly expensive, then such criminal waste could even be viewed as a Sackable Offence. On the other hand, an overly complex feature set if inadequately supported is a ticking Time Bomb, because the understanding required to maintain it simply isn't there. If an overly complex technology is being maintained at great expense, then it's a White Elephant. In all these cases, this combination of bad features can be considered a Train Wreck.

I see the GLQ as a useful tool for thinking IT folk who are willing to stick their necks out by doing independent research into the options they have, and choosing one that represents the best value for money for their organisations.

And as you can see, the Goldilocks Logic Quadrant offers value to customer organisations but tends to disadvantage big vendors. That's OK, because I don't have Silver Lake as my holding company, nor do I call Bill Gates, Michael Dell and Larry Ellison "limited partners". The best argument in favour of the Goldilocks Logic Quadrant over Gartner's Magic Quadrant is that it has no axe to grind.

Stay Tight on Loose Coupling

My Guest View was published on SD Times. This is based on the points covered in the white paper Practical SOA for the Solution Architect.

Cool Innovations by WSO2

I've long held that the WSO2 suite of middleware products is one of the industry's best-kept secrets. But just what is it that makes these products so special?

Well, if the mere fact that there exists a functionally comprehensive, 100% Open Source middleware suite isn't remarkable enough, there are plenty of other reasons why IT practitioners should take a good look at WSO2's offerings.

I have had the opportunity for a few months now to work closely with the company's engineers and play with their products, using them to build (demo) distributed systems, and this is what I have found.

There is significant innovation here that is really cool. There may be even more that I haven't discovered yet.

1. OSGi bundles, functionality "features", and the fluid definition of a "product"

(WSO2 has no rigid products, although their brochures list about 12. The truth is that they have hundreds of capability bundles that can be combined with a common core to create tailored products at will.)

2. A "Middleware Anywhere" architecture that spans cloud and terrestrial servers

(When cloud-native features like multi-tenancy and elasticity are baked into the common core of the middleware product suite, there is no need for applications to be written differently for cloud and terrestrial deployments.)

3. Not just an ESB - the right tool for every job

(As in my writings on "Practical SOA for the Solution Architect", there are three core technology components required for SOA - the Service Container, the Broker and the Process Coordinator. They do different things and are not mutual substitutes. There are also eight supporting aspects at the technology layer. The ESB, being just the Broker component, cannot do all of these functions. WSO2 has products corresponding to all of them.)

4. Making a federated ESB an economically viable architecture

(Economics forces many organisations to deploy their expensive ESB product in a centralised, hub-and-spokes architecture, which leads to a single point of failure and a performance bottleneck. But Brokers are best deployed in a federated manner close to service provider and service consumer endpoints. WSO2's affordable pricing model makes it easy for organisations to do the right thing, architecturally speaking.)

5. The "Server Role" concept - enabling a logical treatment of SOA topology

(Architects don't think in terms of products but in terms of functional capability. One would rather earmark an artifact for deployment to a "mainframe proxy" and another to a "customer complaints process node" than to an "ESB" or a "Business Process Server" instance. Tagging both servers and artifacts with a user-defined "Server Role" makes it possible to speak in these convenient abstractions.)

6. The CAR file as a version snapshot of a working and tested distributed system

(In distributed systems, upgrading the version of software X can break its interoperability with software Y. That's why version change in distributed systems is such a nightmare involving expensive regression testing and a higher probability of outage after an upgrade. But what if a related set of changes can be tested together, certified as interoperable, labelled as a single version, and deployed to multiple systems through a common mechanism? We've just described the Carbon Archive.)

7. The CAR file and Carbon Studio as a unifier of diverse developer skillsets

(Developing distributed systems is challenging from a skills perspective. Writing business logic in Java and exposing it as a web service requires different skills from writing data transformation in XSLT, which is again different from specifying process logic in WS-BPEL. And then there are specialised languages to codify business rules, etc. WSO2 has a single IDE to support all these diverse developer needs - Carbon Studio. It also provides a single package that can hold all these types of artifacts - the Carbon Archive.)

8. The Admin console and support for configuration over coding

(Not every artifact needs to be "developed" through code. Every WSO2 server product has an Admin console that looks largely alike, yet tailored to the peculiarities of the artifacts deployed on that server. Following an 80-20 rule, the bulk of the (simple) artifacts that need to be deployed on a server can be created through configuration using the admin console.)

9. Port offsets and the ability to run multiple servers on the same machine

(Owing to their common core, all WSO2 server products listen on the same ports - 9763 (HTTP) and 9443 (HTTPS). Obviously, port conflicts will result when attempting to run two servers (or even two instances of the same server product) on a single machine. But with a simple configuration change (a "port offset"), a server can be nudged away from its default port to a non-conflicting one. A port offset of 1 will have a server listening on ports 9764 (HTTP) and 9444 (HTTPS), for example.)

10. The dark horse - Mashup Server and server-side JavaScript

(Nodejs has refocused industry attention on server-side JavaScript and the power that brings. But Nodejs has no support for E4X! If you're doing XML manipulation of data from multiple sources, which is what mashups often entail, WSO2's Mashup Server is worth a serious look.)

11. Carbon Studio - A light-touch IDE for middleware developers

(The best GUIs are those built on top of non-visual scripting. Every artifact used by the WSO2 development process is text-based, including the build process that relies on Maven. Command-line junkies can coexist peacefully with those who prefer a GUI, because the IDE imposes no additional requirements on developers. It's purely an option, not an obligation. You can even use another IDE like IntelliJ's IDEA with no loss of capability.)

12. Governance Registry - for control both mundane and subtle

(Governance or plain management? People often use the former term when they mean the latter. In any case, for the mechanics of what you want to hold to support either of these functions, the WSO2 registry and repository tool is simple, flexible and powerful enough to support you. The registry is embedded inside every server product as well as being a standalone server, so the way one can choose to store and share configuration settings as well as policy files is pleasurably versatile.)

If you work with middleware, you should be seriously checking out the offerings of WSO2.

Nutshell Definitions

When conducting the Practical SOA workshops in different Australian cities last month, I felt the need to explain a few concepts for my audience in very simple and memorable terms. I realised I have already been doing this for myself for a long time. I try and distil a concept into a single word, or at most a short phrase, in order to understand it. This has been very useful to me in evaluating the merits of technologies and comparing them to others. Call these my trade secrets which I'm now sharing with you :-).

OK, so here are some of my definitions of popular terms, each in a nutshell:

Service-Oriented Architecture (SOA):

1. In one word, dependencies. More precisely, it is the science of analysing and managing dependencies - making implicit dependencies explicit and eliminating unnecessary dependencies between systems. That's what it's all about.

2. Alternative definition: Lego-isation of the Enterprise, i.e., refactoring the various application silos in an enterprise into reusable building blocks based on their core functions.

Middleware:

That which converts organisational silos into Lego blocks.

Integration:

This is a tricky one. When you point your browser at a website and the page loads, no one thinks of it as integration. For something to be recognised as "integration", it seems it cannot afford to appear effortless! For this reason, I steer clear of trying to define integration in terms of technology. The best integration is seamless and not seen as such. It's an art that involves dependency management (see the definition of SOA above). Minimalism is a virtue here, and appropriate data design is an unacknowledged part of integration.

Governance, as opposed to plain old Management:

Governance has suddenly become a very fashionable word, and not just in IT. "Corporate governance" is a phrase that is parrotted by commentators when they often mean just management. So what's the difference?

In a nutshell,
Governance is about doing the right thing, the "what", the "goal".
Management is about doing things right, the "how", the "task".

Cloud computing as opposed to conventional "terrestrial" computing:

The leasing of IT capability as opposed to ownership. "IT capability" is a very loose term, and its nature varies based on the type of cloud (below). Leasing has several benefits as opposed to ownership - no upfront costs, pay-as-you-go scalability that is easy on startup operations, i.e., practically limitless capacity without having to provision it upfront, etc.

Infrastructure as a Service (IaaS):

Leasing infrastructure (storage, compute power and networking), and owning all applications above it.

Platform as a Service (PaaS):

Leasing infrastructure as well as some application frameworks and common utilities, and owning all applications above it. The frameworks and utilities make it easier to build the applications above.

Software as a Service (SaaS):

Leasing all the application functionality required, and owning nothing.

Virtualisation:

Imagine separating your "mind" from your "brain". We assume exactly one mind per brain and vice-versa. But what if you can have multiple minds within a single brain, a kind of benign schizophrenia? Or more eerily, if a mind can span multiple brains and think more powerful thoughts as a result? That's virtualisation - turning a one-to-one relationship between mind and brain into a many-to-many one.

Cloud computing as opposed to virtualised servers in "terrestrial" computing:

In virtualisation, you still own the brains. In cloud computing, you lease them.

Private cloud:

Leasing brains to yourself to run minds on. This may make sense in certain cases, like with transfer pricing between business units of the same enterprise. But because you haven't rid yourself of the responsibilities of ownership, it may not be as attractive as a public cloud. Nevertheless, it's attractive because sometimes legislation prevents minds from running on strange brains, and you may yourself be testing the concept of mind-brain separation before trusting someone else's brains with the sensitive job of running (and therefore knowing) your minds.

Building RESTful applications using the WSO2 platform

Someone taking a casual look at the WSO2 middleware platform might be forgiven for thinking this is exclusively about SOAP and WS-*. But there is in fact strong support for building RESTful applications with this platform using the JAX-RS framework library, and Prabath Siriwardena (one of WSO2's experts on Identity Management) has blogged about the recommended component architecture to achieve this. There is also a WSO2 workshop on this topic conducted by Asanka Abeysinghe, who has many years of experience on the customer side of the fence and understands both the vendor (technology) and customer perspectives. This workshop is on Dec 8th and should be worth attending for people living close to Palo Alto.

An interesting sidelight: In Practical SOA for the Solution Architect, I re-introduce the practitioner to SOA principles by talking about three core technology components - the Service Container, the Broker and the Process Coordinator. According to this view of SOA, a service can be exposed through any of these components and a service consumer will be none the wiser as to the nature of its implementation. In other words, the Practical SOA approach does not make any prescriptions about which component should be the consumer-facing one. All of them are equally valid candidates, and the only criterion for choosing one over the other is the nature of the service enablement mechanism (bespoke, brokered or orchestrated).

In Prabath's architecture diagram below, the reader will notice that runtime clients must all consume services through the ESB (the Broker), even though these services are hosted on the App Server (the Service Container). Why can't services be exposed directly from the App Server where they are hosted?

Prabath explains that using a Broker (ESB) instance as the front-end for services is recommended practice because the ESB can provide security features like authentication and authorisation, as well as throttling capabilities to guard against Denial of Service (DoS) attacks.

So in the real world, we may often need to front-end Service Containers and Process Coordinators with an instance of the Broker that is dedicated to providing these security and traffic shaping features. This could (and should!) be a different instance of the Broker from those used to mediate access to backend systems. Such an architecture will work well because ESBs are better deployed in a federated topology than in a centralised hub-and-spokes fashion. [The unnatural hub-and-spokes topology for ESB deployment, which the high cost of most commercial ESBs forces on customer organisations, then results in a performance bottleneck and a single point of failure. Fortunately, the more favourable economics of WSO2's Commercial Open Source model makes it feasible for customer organisations to implement a more flexible federated architecture for the ESB.]

PaaS by Lineage

If you've ever wondered about what "Platform as a Service" (PaaS) really means, then you may find this analysis of mine useful.

The traditional NIST model of Cloud Computing shows three layers from a consumer's perspective:

Infrastructure as a Service refers to (shared and scalable) infrastructural capabilities such as compute power, storage and networking, as provided by a cloud vendor. The attributes "shared" (or "multi-tenanted") and "scalable" (or "elastic") distinguish cloud solutions from "terrestrial" alternatives. Amazon's EC2 (Elastic Compute Cloud) and S3 (Simple Storage Service) are examples of IaaS.

Software as a Service refers to applications that you don't have to install on your own computers but can consume through your browser. For an individual, Gmail is the best example of SaaS, while companies can relate to SalesForce.com.

But PaaS has always been a bit of a mystery. What is a "platform", exactly? This is a rather nebulous (pardon the cloudy adjective) area between IaaS and SaaS, and appears to be defined by what the other two are not. The market segment is also still in flux and yet to mature, and there are many players here, each staking out a piece of turf and attempting to define the segment to its own advantage.

I have a high-level, vendor-neutral view of this. [Disclosure: I'm currently working for one of the PaaS vendors, WSO2.]

My preferred categorisation of the PaaS landscape is by lineage. In other words, where did the various PaaS offerings evolve from? I think this is a useful way to look at PaaS because it indicates the traditional strengths of a vendor and therefore where their PaaS offering is likely to be stronger than its competitors. Potential consumers who are looking for a particular emphasis in their PaaS solution will know which vendors are likely to meet their requirements better.

In short, I think PaaS offerings have evolved from one of three directions.

IaaS vendors like VMWare (with their vCloud IaaS) have added DevOps capability to their traditional strength and are targeting organisations that want to develop and run generic applications on the cloud. Their version of PaaS is CloudFoundry.

SaaS vendors like SalesForce.com have made their application more generic and supportive of customisation. Organisations that want to build employee-oriented applications can do so through configuration rather than coding. Their version of PaaS, which is more oriented towards a vertical segment (employee-oriented applications), is called Force.com.

A third direction from which PaaS has evolved is from traditional "terrestrial" middleware (also known as Integration or SOA products). WSO2, which has a full stack of SOA middleware products collectively referred to as Carbon, has added cloud-native features (multi-tenancy, elasticity, etc.) to its suite of products to turn them into yet another version of horizontal PaaS called Stratos. Their DevOps tools have also been upgraded to be able to deploy applications to the cloud just like they do to terrestrial servers.

The diagram below that loosely follows the NIST layering, illustrates my analysis. There may be more versions of PaaS depending on where a vendor has traditionally been based, and I will update this model as more such examples appear.

Glamorous Tech and Workhorse Tech

IT people are biased towards the new, the cool and the "sexy". Perhaps Node.js and Cloud Computing fall into this category. Glamorous technology is very attractive, because it promises to be "the next big thing", but it's usually not usable today. Like Benjamin Franklin's proverbial new-born baby, it's full of potential, but does nothing for us in the here and now.

Contrast this with "workhorse technology", which is mature and relatively boring, and used to build working real-world systems that make or save money today. Not many people get excited about workhorse technology, but the majority of them work with it anyway, because it pays the bills.

I want to talk about one particular category of workhorse technology that I'm discovering can be quite glamorous as well. I have gradually become more familiar with this over the last few months of working with WSO2, writing the "Practical SOA for the Solution Architect" white paper, conducting a webinar and then hitting the road to conduct workshops in three Australian cities. I've realised that the combination of a lightweight SOA methodology in combination with a lightweight SOA product suite can be a very effective workhorse technology that is usable today and saves real dollars. In addition, it's actually pretty cool because of how quickly and easily it can help practitioners integrate diverse and distributed systems into an end-to-end business solution.

In a nutshell, in the white paper, webinar and workshops, I evangelise the message that SOA is not an esoteric and complex black art but simple commonsense that can be readily applied. I talk about the technology layer of SOA, of course, but also cover the equally important data layer that is often neglected and that contributes to the "tight coupling" that plagues so many solution designs and prevents them from realising the benefits of SOA. I talk about three core technology components to use (the Service Container, the Broker and the Process Coordinator) and when to use them, the inefficiencies that result from using the wrong tool for the job, and the dangers of treating the Broker as a singleton, centralised component and deploying it in a hub-and-spokes architecture rather than a federated one. I also cover four simple Data layer principles (make implicit dependencies explicit, remove unnecessary dependencies, loosely couple internal domain data with externally-visible message data, and settle on an intermediate granularity for domain data model(s) rather than a single overarching Canonical Data Model for the entire enterprise).

To drive home these concepts, we work through the real-world example of a well-known banking process, i.e., opening an account. Using the lightweight Practical SOA methodology, participants are encouraged to try their hand at producing an outline solution design to a described requirement in about 15 minutes, using a few standard Lego-style components from the SOA technology layer. Then we demonstrate how that solution design can actually be implemented, substituting each of the conceptual Lego blocks with an actual platform product that hosts the corresponding logic, and get the entire system to work.

Here's a view of the conceptual building blocks that form the outline solution design, once the lightweight SOA methodology is applied to the business problem:

In the next step, the conceptual components are replaced by actual WSO2 SOA products that perform each of those functions, and the physical version of the above diagram looks like this:

The Customer Master Database, the Mainframe and the Card System are all mock objects. A Data Services Server exposes the Customer Master as a set of CRUD services. A Broker (ESB) component exposes the mainframe as a set of Account services, and a second Broker exposes the Card System as a set of Card Services. Since the mainframe can usually only be accessed over IBM MQ in real-life, we simulate that through a JMS connection over ActiveMQ. A Process Coordinator (Business Process Server) coordinates all these services into a BPEL process that performs the account opening business function.

It was gratifying to see that participants at all the workshops we conducted were visibly impressed by this demonstration. In the space of 45 minutes, we had moved from a problem, through a conceptual solution design, to a working implementation. And since we had their active participation through the design exercise, they had an emotional investment in the solution and were able to appreciate it all the more. [To be fair, the demo was developed earlier over 6 person-days of effort, and in the workshop, we stepped quickly through the development by copying code that was written earlier.]

Traditional SOA vendors and the big-name analysts have done the industry a disservice by complicating SOA and scaring off practitioners. In our workshops, we've shown that SOA is just commonsense and relies on just a few simple principles that practitioners can readily apply. Once they know the right tools to use in the solution, it's a simple matter to gather those required components and hook them together to create the end-to-end solution.

This is obviously workhorse technology, because it's mature enough to solve bread-and-butter business problems today. But the lightweight methodology and product suite also make it glamorous.

I hope this attracts more practitioners towards the practice of lightweight SOA and the use of simple and cost-effective products like the WSO2 product suite.

Enterprise Shared Services and the Cloud

I've worked in the area of Enterprise Shared Services (or Enterprise Utilities) for many years, so when InfoQ approached me asking if I would be interested in writing an article on cloud computing, this was one of the angles I thought of. Of course, I'm also working with WSO2 at present, and WSO2 has a distinct type of PaaS (Platform as a Service) offering called Stratos. PaaS usually either evolves up from IaaS (Infrastructure as a Service) with the addition of support for DevOps (the development-operations continuum) or evolves down from SaaS (Software as a Service) with the addition of customisation support for the software application. Stratos is unique because it has evolved "sideways" from enterprise middleware with the addition of cloud-native features. The 12 SOA products of WSO2 are all available as cloud-native middleware on the Stratos PaaS.

In any case, since InfoQ wanted vendor-neutral content, I couldn't write about Stratos (which I will write about in some context because I find it fascinating). So I fell back to my old favourite - Enterprise Shared Services.

The long and short of it is that when we factor in Enterprise Shared Services, the old monikers of SaaS and PaaS are no longer enough. We have to deal with "vertical" and "horizontal" variants of these, and the distinction is important because in an organisational context, they have unique characteristics around how they are requisitioned, evaluated for feasibility, funded and charged back.

I'll let you read about it here.

A Good Primer on NoSQL

NoSQL databases are all the rage, but the array of choices before us is bewildering. I must confess I'm still confused about the features and differences between BigTable, GAE DataStore, GemFire, SimpleDB, SQLFire, CouchDB, MongoDB, RavenDB, Redis, Cassandra, Riak, HBase, Neo4j and so many other names that I have only recently begun to hear about. I'm sure many others would be in the same situation.

I was therefore happy to see that my colleague at WSO2, Dr Srinath Perera, has analysed the NoSQL landscape in depth, zeroed in on the characteristics of NoSQL databases that are relevant, and summarised this for our common understanding in an InfoQ article that provides a simple overview of the choices that designers and developers have today, choices that go beyond the traditional relational databases that we're familiar with.

I've often wondered about why NoSQL should be so popular in the first place. Srinath explains:

A few years ago, most systems were small and relational databases could handle [their requirements] without any trouble. Therefore, the storage choices for architects and programmers were simple. However, the size and scale of these systems have grown significantly over the last few years. High tech companies like Amazon and Google faced the challenge of scale before others. They soon observed that relational databases could not scale to handle those use cases.

In other words, this demanding new requirements wave has probably not hit most of us yet, but with the jump in the number of connected devices (smartphones, tablets and the coming "Internet of Things"), applications dealing with huge volumes of data are probably not going to be as rare as in the past. And when we say "huge", we're not even talking Gigabytes anymore. It's Terabytes and larger. As we learnt from Godzilla, size does matter. And drastic situations call for drastic measures, hence the NoSQL revolution.

Srinath refers to Eric Brewer's CAP theorem, which states that a distributed system can only have two of the three properties - Consistency, Availability, and Partition Tolerance. The NoSQL databases aim to break through the limitations imposed on traditional relational databases by loosening the fundamental principles on which these have been based, dropping one or more constraints as appropriate, to obtain a desired behaviour.

Depending on the constraints dropped, the resulting solution falls into one of several new categories:

• Local memory
• Distributed cache
• Column Family Storage
• Document storage
• Name-value pairs
• Graph DB
• Service Registry
• Tuple Space

...in addition to the traditional filesystems, relational databases and message queues that are familiar to IT practitioners today.

Perhaps the most important contribution of Srinath's article is his distilling of the four primary characteristics that are important from a usage point of view - data structure, the level of scalability required, the nature of data retrieval and the level of consistency required. He then puts these characteristics together in various combinations to show which of the above-listed categories of data store would be the most appropriate solution to use.

He's certainly succeeded in demystifying NoSQL for me, although I suspect I'll need to go back and read the article a few times till I've fully internalised the concepts in it. This is an overview article that I'd recommend to anyone trying to make sense of NoSQL and wanting to decide on the appropriate product category that would be right for their needs.

I can see the demand for a follow-up article from Srinath drilling down into each of these data storage categories and providing recommendations about actual products (e.g., Cassandra, Redis, CouchDB, etc.) While the sands shift more rapidly in the product space, it's also a more practically urgent decision for a developer or architect to make. So while such an article might need to be updated quite frequently, the advice in it would be more practical than this one, which provides the necessary initial understanding of the NoSQL landscape.

Strange Creature on the Mozilla Firefox Download Page

I've never seen this creature before (circled in red). Do you know who or what it is? It looks friendly enough, but it also reminds me of the Morlocks in H.G. Wells's The Time Machine (shudder).

I Hate HatEoAS

For something that's supposed to be THE defining characteristic of REST, it could have done with better naming.

I would have been happy with the term HatEoAS if it had stood for "Hypermedia as the Envelope of Application State" rather than "Hypermedia as the Engine of Application State".

An Engine actively drives things. E.g., A process engine is well named, because it drives a process.

A constraint doesn't drive anything. It constrains. It provides an envelope around the range of possibilities.

And so they really should have called this an envelope rather than an engine of application state.

There, I've said it. Because the expansion of HatEoAS has been driving me up the wall.

Fortunately, it's being referred to as "Hypermedia Constraint" now, which is both more elegant and more accurate.

Oneiric Ocelot Not Quite the Stuff of Dreams

One of the nicest things about Linux distributions like Ubuntu is that you don't have to spend the night standing in a queue just to get the latest version of an operating system on the day of its release.

Unlike some other (closed) systems where supply is deliberately constrained
to create an impression of even greater demand, Ubuntu is upfront and
relaxed about new versions.

I sat down at my Linux desktop yesterday and was pleasantly surprised to see a popup informing me that the next version of Ubuntu Linux (version 11.10 a.k.a. "Oneiric Ocelot") was now available and would I like to upgrade?

With a smile of anticipation, I clicked Yes, and the upgrade began. My ADSL link showed a steady bandwidth of around 400 kBps throughout. An uneventful hour and a half later, I rebooted into Oneiric Ocelot. That was the kind of experience I've got used to with Ubuntu over so many online upgrades.

Most of the time, upgrades to Ubuntu are boringly anticlimactic. That's a good thing, by the way, because users hate surprises, and there are really very few nice surprises possible on an upgrade.

The dictionary says "oneiric" means "pertaining to dreams", but there was something almost nightmarish with this upgrade that was even worse than the upgrade to 11.04 ("Natty Narwhal").

Someone at Canonical has taken it into their head that a completely re-imagined user interface would be a good thing. The same someone has also arrogantly assumed that there's no need to give users a choice when changing a fundamental aspect of the user interface that they will use all the time.

My unpleasant surprise after both upgrades was the horror they call the Unity Desktop. I tried to be fair to them. I believe I gave Unity an hour of my time both times. In the end, I gave up. I just hated it. Sorry Canonical, I recognise you're trying, but Unity really doesn't work for me. From the number of similar comments I read on the web, I'm hardly alone.

What was far worse than installing Unity by default (without asking me if I wanted it) was not providing me a quick way to get back to the default Gnome desktop of earlier versions. I was actually forced to download the Gnome desktop, then re-login to select it as my default desktop.

1. Why didn't I have the choice to say no to Unity at the time of the upgrade?
2. Why wasn't it a straightforward option to return to the "classic" desktop?

For a distribution that is supposed to be the friendliest desktop Linux, this is very poor showing indeed.

My other major whinges are that the "Show desktop" button on the taskbar has disappeared, as has the "System" menu on the menu bar. I now have to minimise every window manually, and have no way to set several preferences. Cosmetically as well, the taskbar at the bottom and the menu bar at the top now sport a ghastly dark grey colour, and the desktop theme that I used to use has disappeared from the list of options. Since I've forgotten what it was called, I don't think I can get it back.

This upgrade experience has been anything but oneiric. I feel like I've been mauled by an ocelot.

Practical SOA for the Solution Architect

This is a story that has had a fairly long history, so stay with me till the end.

I've worked as an architect in the shared services space for almost a decade now, at some of Australia's biggest, richest and technologically diverse financial services organisations. During that time, I first heard about Service-Oriented Architecture (SOA), learnt what it was about, bought into its philosophy and attempted to implement it at work.

And while I have seen a few successful examples of SOA projects (mostly individual services, truth be told), by and large, I did not see SOA having an impact at all at these large and reputed organisations. Many hundreds of thousands of dollars were spent on acquiring SOA tools from vendors as reputed as IBM and TIBCO, and many millions more were spent on integration projects using these tools, but somehow, the results failed to live up to the SOA promise. [I guess organisations need a terrorising CEO like Amazon's Jeff Bezos to achieve the benefits of SOA. Read the fascinating story of how an online bookstore built a platform that it now rents out to others.]

For a while, I lost faith. Influenced by a few cynical colleagues, I too began to think SOA was marketing hype and nothing more. But then, as I continued to work in the shared services domain and had the opportunity to review more solution designs, I had an epiphany. Most of the designs I was seeing used SOA products (usually an ESB) or were implemented as Web Services, yet I could see they were still tightly-coupled at the level of the application design, i.e., the data. Also, it was very common for solution designers to use the wrong tool for the job, simply because it was the one they were most familiar with. They even seemed to lack a conceptual ability to tell which tool would be right for a given requirement.

With that epiphany, I revisited my understanding of SOA. I realised that solution architects needed to be educated to produce SOA-compliant application designs, otherwise all the investment made by their organisations in SOA tool suites would be a waste. Worse still, SOA itself would be unfairly blamed for the waste of resources, when it in fact remains the best hope to reduce waste, improve business agility and reduce operational risk.

I've been tossing this idea around in my head for at least a couple of years now, the idea that a lightweight method is required to get solution architects up to speed with the required concepts quickly. Unfortunately, most solution architects would bristle if it was suggested to them that they don't really understand SOA. "Can't you see we're using an ESB?" would be the defensive response. That the response is a non sequitur would be lost on them. One can use an ESB and still come up with a design that is not SOA-compliant. How can we educate solution architects when they don't know what they don't know?

Fortuitously, I had a chat about this in August this year with Sanjiva Weerawarana, the CEO of the innovative Open Source middleware company, WSO2. Our needs meshed perfectly. WSO2 has a full suite of middleware products based on SOA concepts and Web Service standards. They are compact and not bloated, fairly straightforward to install and use, fully Open Source under the Apache Software Licence, and for which WSO2 offers a very attractively priced support model as well as other professional services. For such an innovative range of value-for-money products, the level of awareness in the customer community has been surprisingly low. Sanjiva has been trying to raise the level of awareness in the industry about his company's products for some time now. For too long WSO2's pitch had been aimed at developers (techies talking to techies), but for a real breakthrough, they needed to target decision-makers and decision-influencers. They were looking for a way to reach the solution architect with a compelling message.

And here I was, trying to educate the same solution architect about SOA using a new, lightweight approach. So Sanjiva and I came to an understanding. I would do a paid consulting assignment with WSO2 for a few months and turn out a few white papers to present their offerings to an audience that was higher up the corporate food chain than the developers. In return (in addition to helping me pay the bills), they would provide me a vehicle to popularise some of my ideas on SOA, especially the lightweight methodology I came to call "Practical SOA". I guess this approach is the good cop to Jeff Bezos's bad cop :-).

The first of those white papers ("Practical SOA for the Solution Architect") has now been completed and is available on WSO2's website. It's my audacious hope that a Solution Architect can read it in half an hour and be immediately effective on their project thanks to a simpler and more powerful mental model of SOA. A short summary of the paper is available for you to skim through, but I would encourage you to download the full paper (a free registration is required), since it has much more detailed descriptions, extensive explanations for the final conclusions and a couple of industry examples to drive home the concepts. So please have a read and, if you like it, recommend it to all your architect- and designer-type friends. (At my own request, my name doesn't appear on this document because I don't want to dilute the appeal of the method by causing it to look like a mere individual's opinion. WSO2's cachet is better than mine!)

A second white paper that is currently in the works will describe the full suite of WSO2's products and how they map to the framework established in the first. Stay tuned for that too. The first white paper will equip you with concepts. The second will equip you with know-how about a comprehensive set of reasonably-priced tools. Together, they should provide a customer organisation with excellent value for money and the long sought-for return on their investment in SOA.