My Books On Dependency-Oriented Thinking - Why They Should Count

InfoQ has published both volumes of my book "Dependency-Oriented Thinking". The links are below.

Dependency-Oriented Thinking: Volume 1 - Analysis and Design (Service-Oriented Architecture Made Simple)

Dependency-Oriented Thinking: Volume 2 - Governance and Management (Control Systems Made Simple)

I'll admit it feels somewhat anticlimactic for me to see these books finally published, because I finished writing them in December 2013 after about two years of intermittent work. They have been available as white papers on Slideshare since Christmas 2013. The last seven months have gone by in reviews, revisions and the various other necessary steps in the publication process. And they have made their appearance on InfoQ's site with scarcely a splash. Is that all?, I feel like asking myself. But I guess I shouldn't feel blasé. These two books are a major personal achievement for me and represent a significant milestone for the industry, and I say this entirely without vanity.

You see, the IT industry has been misled for over 15 years by a distorted and heavyweight philosophy that has gone by the name "Service-Oriented Architecture" (SOA). It has cost organisations billions of dollars of unnecessary spend, and has fallen far short of the benefits that it promised. I too fell victim to the hype around SOA in its early days, and like many other converted faithful, tried hard to practise my new religion. Finally, like many others who turned apostate, I grew disillusioned with the lies, and what disillusioned me the most was the heavyhandedness of the "Church of SOA", a ponderous cathedral of orthodox practice that promised salvation, yet delivered nothing but daily guilt.

But unlike others who turned atheist and denounced SOA itself, I realised that I had to found a new church. Because I realised that there was a divine truth to SOA after all. It was just not to be found in the anointed bible of the SOA church, for that was a cynical document designed to suit the greed of the cardinals of the church rather than the needs of the millions of churchgoers. The actual truth was much, much simpler. It was not easy, because "simple" and "easy" are not the same thing. (If you find this hard to understand, think about the simple principle "Don't tell lies", and tell me whether it is easy to follow.)

I stumbled upon this simple truth through a series of learnings. I thought I had hit upon it when I wrote my white paper "Practical SOA for the Solution Architect" under the aegis of WSO2. But later, I realised there was more. The WSO2 white paper identified three core components at the technology layer. It also recognised that there was something above the technology layer that had to be considered during design. What was that something? Apart from a recognition of the importance of data, the paper did not manage to pierce the veil.

The remaining pieces of the puzzle fell into place as I began to consider the notion of dependencies as a common principle across the technology and data layers. The more I thought about dependencies, the more things started to make sense at layers even above data, and the more logical design at all these layers followed from requirements and constraints.

In parallel, there was another train of thought to which I once again owe a debt of gratitude to WSO2. While I was employed with the company, I was asked to write another white paper on SOA governance. A lot of the material I got from company sources hewed to the established industry line on SOA governance, but as with SOA design, the accepted industry notion of SOA governance made me deeply uncomfortable. Fortunately, I'm not the kind to suppress my misgivings to please my paymasters, and so at some point, I had to tell them that my own views on SOA governance were very different. To WSO2's credit, they encouraged me to write up my thoughts without the pressure to conform to any expected models. And although the end result was something so alien to establishment thought that they could not endorse it as a company, they made no criticism.

So at the end of 2011, I found myself with two related but half-baked notions of SOA design and SOA governance, and as 2012 wore on, my thoughts began to crystallise. The notion of dependencies, I saw, played a central role in every formulation. The concept of dependencies also suggested how analysis, design, governance and management had to be approached. It had a clear, compelling logic.

I followed my instincts and resisted all temptation to cut corners. Gradually, the model of "Dependency-Oriented Thinking" began to take shape. I conducted a workshop where I presented the model to some practising architects, and received heartening validation and encouragement. The gradual evolution of the model mainly came about through my own ruminations upon past experiences, but I also received significant help from a few friends. Sushil Gajwani and Ravish Juneja are two personal friends who gave me examples from their own (non-IT) experience. These examples confirmed to me that dependencies underpin every interaction in the world. Another friend and colleague, Awadhesh Kumar, provided an input that elegantly closed a gaping hole in my model of the application layer. He pointed out that grouping operations according to shared interface data models and according to shared internal data models would lead to services and to products, respectively. Kalyan Kumar, another friend who attended one of my workshops, suggested that I split my governance whitepaper into two to address the needs of two different audiences - designers and managers.

And so, sometime in 2013, the model crystallised. All I then had to do was write it down. On December 24th, I completed the two whitepapers and uploaded them to Slideshare. There has been a steady trickle of downloads since then, but it was only after their publication by InfoQ that the documents have gained more visibility.

These are not timid, establishment-aligned documents. They are audacious and iconoclastic. I believe the IT industry has been badly misled by a wrongheaded notion of SOA, and that I have discovered (or re-discovered, if you will) the core principle that makes SOA practice dazzlingly simple and blindingly obvious. I have not just criticised an existing model. I have been constructive in proposing an alternative - a model that I have developed rigorously from first principles, validated against my decades of experience, and delineated in painstaking detail. This is not an edifice that can be lightly dismissed. Again, these are not statements of vanity, just honest conviction.

I believe that if an organisation adopts the method of "Dependency-Oriented Thinking" that I have laid out in these two books (after testing the concepts and being satisfied that they are sound), then it will obtain the many benefits of SOA that have been promised for years - business agility, sustainably lower operating costs, and reduced operational risk.

It takes an arc of enormous radius to turn around a gigantic oil tanker cruising at top speed, and I have no illusions about the time it will take to bring the industry around to my way of thinking. It may be 5-10 years before the industry adopts Dependency-Oriented Thinking as a matter of course, but I'm confident it will happen. This is an idea whose time has come.

Glamorous Tech and Workhorse Tech

IT people are biased towards the new, the cool and the "sexy". Perhaps Node.js and Cloud Computing fall into this category. Glamorous technology is very attractive, because it promises to be "the next big thing", but it's usually not usable today. Like Benjamin Franklin's proverbial new-born baby, it's full of potential, but does nothing for us in the here and now.

Contrast this with "workhorse technology", which is mature and relatively boring, and used to build working real-world systems that make or save money today. Not many people get excited about workhorse technology, but the majority of them work with it anyway, because it pays the bills.

I want to talk about one particular category of workhorse technology that I'm discovering can be quite glamorous as well. I have gradually become more familiar with this over the last few months of working with WSO2, writing the "Practical SOA for the Solution Architect" white paper, conducting a webinar and then hitting the road to conduct workshops in three Australian cities. I've realised that the combination of a lightweight SOA methodology in combination with a lightweight SOA product suite can be a very effective workhorse technology that is usable today and saves real dollars. In addition, it's actually pretty cool because of how quickly and easily it can help practitioners integrate diverse and distributed systems into an end-to-end business solution.

In a nutshell, in the white paper, webinar and workshops, I evangelise the message that SOA is not an esoteric and complex black art but simple commonsense that can be readily applied. I talk about the technology layer of SOA, of course, but also cover the equally important data layer that is often neglected and that contributes to the "tight coupling" that plagues so many solution designs and prevents them from realising the benefits of SOA. I talk about three core technology components to use (the Service Container, the Broker and the Process Coordinator) and when to use them, the inefficiencies that result from using the wrong tool for the job, and the dangers of treating the Broker as a singleton, centralised component and deploying it in a hub-and-spokes architecture rather than a federated one. I also cover four simple Data layer principles (make implicit dependencies explicit, remove unnecessary dependencies, loosely couple internal domain data with externally-visible message data, and settle on an intermediate granularity for domain data model(s) rather than a single overarching Canonical Data Model for the entire enterprise).

To drive home these concepts, we work through the real-world example of a well-known banking process, i.e., opening an account. Using the lightweight Practical SOA methodology, participants are encouraged to try their hand at producing an outline solution design to a described requirement in about 15 minutes, using a few standard Lego-style components from the SOA technology layer. Then we demonstrate how that solution design can actually be implemented, substituting each of the conceptual Lego blocks with an actual platform product that hosts the corresponding logic, and get the entire system to work.

Here's a view of the conceptual building blocks that form the outline solution design, once the lightweight SOA methodology is applied to the business problem:

In the next step, the conceptual components are replaced by actual WSO2 SOA products that perform each of those functions, and the physical version of the above diagram looks like this:

The Customer Master Database, the Mainframe and the Card System are all mock objects. A Data Services Server exposes the Customer Master as a set of CRUD services. A Broker (ESB) component exposes the mainframe as a set of Account services, and a second Broker exposes the Card System as a set of Card Services. Since the mainframe can usually only be accessed over IBM MQ in real-life, we simulate that through a JMS connection over ActiveMQ. A Process Coordinator (Business Process Server) coordinates all these services into a BPEL process that performs the account opening business function.

It was gratifying to see that participants at all the workshops we conducted were visibly impressed by this demonstration. In the space of 45 minutes, we had moved from a problem, through a conceptual solution design, to a working implementation. And since we had their active participation through the design exercise, they had an emotional investment in the solution and were able to appreciate it all the more. [To be fair, the demo was developed earlier over 6 person-days of effort, and in the workshop, we stepped quickly through the development by copying code that was written earlier.]

Traditional SOA vendors and the big-name analysts have done the industry a disservice by complicating SOA and scaring off practitioners. In our workshops, we've shown that SOA is just commonsense and relies on just a few simple principles that practitioners can readily apply. Once they know the right tools to use in the solution, it's a simple matter to gather those required components and hook them together to create the end-to-end solution.

This is obviously workhorse technology, because it's mature enough to solve bread-and-butter business problems today. But the lightweight methodology and product suite also make it glamorous.

I hope this attracts more practitioners towards the practice of lightweight SOA and the use of simple and cost-effective products like the WSO2 product suite.

Practical SOA for the Solution Architect

This is a story that has had a fairly long history, so stay with me till the end.

I've worked as an architect in the shared services space for almost a decade now, at some of Australia's biggest, richest and technologically diverse financial services organisations. During that time, I first heard about Service-Oriented Architecture (SOA), learnt what it was about, bought into its philosophy and attempted to implement it at work.

And while I have seen a few successful examples of SOA projects (mostly individual services, truth be told), by and large, I did not see SOA having an impact at all at these large and reputed organisations. Many hundreds of thousands of dollars were spent on acquiring SOA tools from vendors as reputed as IBM and TIBCO, and many millions more were spent on integration projects using these tools, but somehow, the results failed to live up to the SOA promise. [I guess organisations need a terrorising CEO like Amazon's Jeff Bezos to achieve the benefits of SOA. Read the fascinating story of how an online bookstore built a platform that it now rents out to others.]

For a while, I lost faith. Influenced by a few cynical colleagues, I too began to think SOA was marketing hype and nothing more. But then, as I continued to work in the shared services domain and had the opportunity to review more solution designs, I had an epiphany. Most of the designs I was seeing used SOA products (usually an ESB) or were implemented as Web Services, yet I could see they were still tightly-coupled at the level of the application design, i.e., the data. Also, it was very common for solution designers to use the wrong tool for the job, simply because it was the one they were most familiar with. They even seemed to lack a conceptual ability to tell which tool would be right for a given requirement.

With that epiphany, I revisited my understanding of SOA. I realised that solution architects needed to be educated to produce SOA-compliant application designs, otherwise all the investment made by their organisations in SOA tool suites would be a waste. Worse still, SOA itself would be unfairly blamed for the waste of resources, when it in fact remains the best hope to reduce waste, improve business agility and reduce operational risk.

I've been tossing this idea around in my head for at least a couple of years now, the idea that a lightweight method is required to get solution architects up to speed with the required concepts quickly. Unfortunately, most solution architects would bristle if it was suggested to them that they don't really understand SOA. "Can't you see we're using an ESB?" would be the defensive response. That the response is a non sequitur would be lost on them. One can use an ESB and still come up with a design that is not SOA-compliant. How can we educate solution architects when they don't know what they don't know?

Fortuitously, I had a chat about this in August this year with Sanjiva Weerawarana, the CEO of the innovative Open Source middleware company, WSO2. Our needs meshed perfectly. WSO2 has a full suite of middleware products based on SOA concepts and Web Service standards. They are compact and not bloated, fairly straightforward to install and use, fully Open Source under the Apache Software Licence, and for which WSO2 offers a very attractively priced support model as well as other professional services. For such an innovative range of value-for-money products, the level of awareness in the customer community has been surprisingly low. Sanjiva has been trying to raise the level of awareness in the industry about his company's products for some time now. For too long WSO2's pitch had been aimed at developers (techies talking to techies), but for a real breakthrough, they needed to target decision-makers and decision-influencers. They were looking for a way to reach the solution architect with a compelling message.

And here I was, trying to educate the same solution architect about SOA using a new, lightweight approach. So Sanjiva and I came to an understanding. I would do a paid consulting assignment with WSO2 for a few months and turn out a few white papers to present their offerings to an audience that was higher up the corporate food chain than the developers. In return (in addition to helping me pay the bills), they would provide me a vehicle to popularise some of my ideas on SOA, especially the lightweight methodology I came to call "Practical SOA". I guess this approach is the good cop to Jeff Bezos's bad cop :-).

The first of those white papers ("Practical SOA for the Solution Architect") has now been completed and is available on WSO2's website. It's my audacious hope that a Solution Architect can read it in half an hour and be immediately effective on their project thanks to a simpler and more powerful mental model of SOA. A short summary of the paper is available for you to skim through, but I would encourage you to download the full paper (a free registration is required), since it has much more detailed descriptions, extensive explanations for the final conclusions and a couple of industry examples to drive home the concepts. So please have a read and, if you like it, recommend it to all your architect- and designer-type friends. (At my own request, my name doesn't appear on this document because I don't want to dilute the appeal of the method by causing it to look like a mere individual's opinion. WSO2's cachet is better than mine!)

A second white paper that is currently in the works will describe the full suite of WSO2's products and how they map to the framework established in the first. Stay tuned for that too. The first white paper will equip you with concepts. The second will equip you with know-how about a comprehensive set of reasonably-priced tools. Together, they should provide a customer organisation with excellent value for money and the long sought-for return on their investment in SOA.