Showing posts with label Dependency-Oriented Thinking. Show all posts
Showing posts with label Dependency-Oriented Thinking. Show all posts

Tuesday, August 05, 2014

My Books On Dependency-Oriented Thinking - Why They Should Count

InfoQ has published both volumes of my book "Dependency-Oriented Thinking". The links are below.


I'll admit it feels somewhat anticlimactic for me to see these books finally published, because I finished writing them in December 2013 after about two years of intermittent work. They have been available as white papers on Slideshare since Christmas 2013. The last seven months have gone by in reviews, revisions and the various other necessary steps in the publication process. And they have made their appearance on InfoQ's site with scarcely a splash. Is that all?, I feel like asking myself. But I guess I shouldn't feel blasé. These two books are a major personal achievement for me and represent a significant milestone for the industry, and I say this entirely without vanity.

You see, the IT industry has been misled for over 15 years by a distorted and heavyweight philosophy that has gone by the name "Service-Oriented Architecture" (SOA). It has cost organisations billions of dollars of unnecessary spend, and has fallen far short of the benefits that it promised. I too fell victim to the hype around SOA in its early days, and like many other converted faithful, tried hard to practise my new religion. Finally, like many others who turned apostate, I grew disillusioned with the lies, and what disillusioned me the most was the heavyhandedness of the "Church of SOA", a ponderous cathedral of orthodox practice that promised salvation, yet delivered nothing but daily guilt.

But unlike others who turned atheist and denounced SOA itself, I realised that I had to found a new church. Because I realised that there was a divine truth to SOA after all. It was just not to be found in the anointed bible of the SOA church, for that was a cynical document designed to suit the greed of the cardinals of the church rather than the needs of the millions of churchgoers. The actual truth was much, much simpler. It was not easy, because "simple" and "easy" are not the same thing. (If you find this hard to understand, think about the simple principle "Don't tell lies", and tell me whether it is easy to follow.)

I stumbled upon this simple truth through a series of learnings. I thought I had hit upon it when I wrote my white paper "Practical SOA for the Solution Architect" under the aegis of WSO2. But later, I realised there was more. The WSO2 white paper identified three core components at the technology layer. It also recognised that there was something above the technology layer that had to be considered during design. What was that something? Apart from a recognition of the importance of data, the paper did not manage to pierce the veil.

The remaining pieces of the puzzle fell into place as I began to consider the notion of dependencies as a common principle across the technology and data layers. The more I thought about dependencies, the more things started to make sense at layers even above data, and the more logical design at all these layers followed from requirements and constraints.

In parallel, there was another train of thought to which I once again owe a debt of gratitude to WSO2. While I was employed with the company, I was asked to write another white paper on SOA governance. A lot of the material I got from company sources hewed to the established industry line on SOA governance, but as with SOA design, the accepted industry notion of SOA governance made me deeply uncomfortable. Fortunately, I'm not the kind to suppress my misgivings to please my paymasters, and so at some point, I had to tell them that my own views on SOA governance were very different. To WSO2's credit, they encouraged me to write up my thoughts without the pressure to conform to any expected models. And although the end result was something so alien to establishment thought that they could not endorse it as a company, they made no criticism.

So at the end of 2011, I found myself with two related but half-baked notions of SOA design and SOA governance, and as 2012 wore on, my thoughts began to crystallise. The notion of dependencies, I saw, played a central role in every formulation. The concept of dependencies also suggested how analysis, design, governance and management had to be approached. It had a clear, compelling logic.

I followed my instincts and resisted all temptation to cut corners. Gradually, the model of "Dependency-Oriented Thinking" began to take shape. I conducted a workshop where I presented the model to some practising architects, and received heartening validation and encouragement. The gradual evolution of the model mainly came about through my own ruminations upon past experiences, but I also received significant help from a few friends. Sushil Gajwani and Ravish Juneja are two personal friends who gave me examples from their own (non-IT) experience. These examples confirmed to me that dependencies underpin every interaction in the world. Another friend and colleague, Awadhesh Kumar, provided an input that elegantly closed a gaping hole in my model of the application layer. He pointed out that grouping operations according to shared interface data models and according to shared internal data models would lead to services and to products, respectively. Kalyan Kumar, another friend who attended one of my workshops, suggested that I split my governance whitepaper into two to address the needs of two different audiences - designers and managers.

And so, sometime in 2013, the model crystallised. All I then had to do was write it down. On December 24th, I completed the two whitepapers and uploaded them to Slideshare. There has been a steady trickle of downloads since then, but it was only after their publication by InfoQ that the documents have gained more visibility.

These are not timid, establishment-aligned documents. They are audacious and iconoclastic. I believe the IT industry has been badly misled by a wrongheaded notion of SOA, and that I have discovered (or re-discovered, if you will) the core principle that makes SOA practice dazzlingly simple and blindingly obvious. I have not just criticised an existing model. I have been constructive in proposing an alternative - a model that I have developed rigorously from first principles, validated against my decades of experience, and delineated in painstaking detail. This is not an edifice that can be lightly dismissed. Again, these are not statements of vanity, just honest conviction.

I believe that if an organisation adopts the method of "Dependency-Oriented Thinking" that I have laid out in these two books (after testing the concepts and being satisfied that they are sound), then it will obtain the many benefits of SOA that have been promised for years - business agility, sustainably lower operating costs, and reduced operational risk.

It takes an arc of enormous radius to turn around a gigantic oil tanker cruising at top speed, and I have no illusions about the time it will take to bring the industry around to my way of thinking. It may be 5-10 years before the industry adopts Dependency-Oriented Thinking as a matter of course, but I'm confident it will happen. This is an idea whose time has come.

Saturday, February 15, 2014

Sydney Workshop "Introduction to Dependency-Oriented Thinking" Held

After weeks of hectic preparation, Rahul Singh and I held our long-awaited workshop today on "Dependency-Oriented Thinking", comprising all-new material from my recent document "Dependency-Oriented Thinking: Vol 1 - Analysis and Design".

Somewhat disappointingly, we only had three signed-up participants, but the numbers only tell half the story. Only one of the three was from Sydney. One flew in from Melbourne the night before, and another got up at 4 am to make the 3+ hour drive from Canberra to Sydney. With such determination on their part, I just had to do my best, and I hope they were happy with the workshop. They certainly expressed satisfaction on their feedback forms :-).

The slides are now available on Slideshare. If anyone was put off from reading the original document on account of its size (264 pages), they could go through this slide pack instead (only 220 slides, heh).

Tuesday, January 21, 2014

Sydney Workshop On Dependency-Oriented Thinking (Saturday Feb 15, 2014)

Well, it's time to road-test my SOA method "Dependency-Oriented Thinking". I released the two DOT documents before Christmas, Volume 1 on Analysis and Design and Volume 2 on Governance and Management. Now, Rahul Singh and I are conducting an all-day workshop to provide a more interactive instruction into the Analysis and Design aspects of the method. It'll be on Saturday the 15th of February, 2014.

The course fees are $450 plus GST for the all-day program. There's an early bird price of $400 plus GST for those who register on or before the 3rd of Feb.

You can download the workshop brochure and schedule here: http://slidesha.re/1j59nqK

If you're a Sydney-based business analyst, solution architect or senior designer, you may be interested in this workshop. My objective is to make IT professionals more effective by helping them think more powerfully about complex, inter-connected systems. Dependency-Oriented Thinking is just structured common sense, but it doesn't come naturally. It needs to be taught, and it requires practice before it can become second nature.

Wednesday, December 25, 2013

Dependency-Oriented Thinking - Documents Released!

I know I haven't been active on this blog for almost 6 months, but I haven't been idle. Ever since Rahul Singh and I conducted our first workshop on "Dead Simple SOA" in Sydney last year, I have been working on the feedback we received, namely that we needed to split "Analysis & Design" from "Governance & Management" because these two topics are of interest to two different audiences.

Splitting out just Governance & Management from the original document "Slicing the Gordian Knot of SOA Governance" was the easy part. I realised that I didn't have much material on Analysis and Design, so I had to write that from scratch. That task ultimately took the better part of a year.

Well, that's completed now, and the two documents are now ready to download from SlideShare:


I have believed for a while now that the core concept behind SOA ought to be "dependencies". I could explain more here, but I'm exhausted after the marathon effort of the last few months. Besides, you'll find a very detailed argument in Volume 1.

Please do download these documents and have a read. More importantly, give me your feedback. My email address is in the "About the Author" section in both documents.

Tuesday, July 02, 2013

Dependency Principles

I've been used to thinking of SOA as "Dependency-Oriented Thinking" for such a long time that it suddenly struck me that I should readily be able to postulate "dependency principles" that can be applied at each of the BAIT layers. As it turned out, I was fortunate enough to have some actual case studies to verify my proposed set, and these cases were selected for being representative of different tiers, so they were quite comprehensive.

It turns out that the lesson to be learnt after a post-mortem of each of those case studies was that one or more dependency principles had been ignored or violated, resulting in the problem that was being faced. The corollary is that an organisation that scrupulously adheres to these principles will end up achieving SOA.

So, without further ado, here are the dependency principles (updated on 25/12/2013):

Business Layer Principles

1. Traceability – Enforce core governance; ensure that everything that is done makes sense with respect to the organisation's Vision
2. Minimalism – Challenge assumptions, reject unwarranted constraints, do no more than required, reuse functional building blocks
3. Domain Insight – Understand the true nature of the business; don't be misled by superficialities or conventional wisdom; re-imagine the business from first principles
4. Business Process Coordination Style – Choose a coordination style (orchestration or choreography) based on whether tight control is possible and/or necessary. This influences the choice of technology standard (SOAP or REST), but may also be influenced by a prior choice of standard.

Application Layer Principles

5. High Cohesion (“Belonging”) – What belongs together should go together, with minimal links between distinct systems; question instances of a single logical function split across multiple systems, or multiple logical functions combined within a single system. Group operations that share a Domain Data Model and business logic into Products, those that share an Interface Data Model into Services.
6. Decoupling of Interfaces from Internals – Ensure that no external system develops dependencies on the internal aspects of a system; create an interface to isolate the systems from each other.
7. “Goldilocks” Signatures (Stability versus Precision) – Identify multiple concurrent flavours of each logical operation; establish a way to express the business intent common to all of them; make sure that the interface doesn't change for minor variations in logic but does change with the business intent.
8. Shared Semantics (Negotiation of Intent and Content) – In choreographed processes, ensure that the service provider and service consumer understand the meaning of every Operation as well as the mechanics of how each Operation should be invoked.

Information (Data) Layer Principles

9. Decoupling of Interface Data from Internal Data – Distinguish the Interface Data Model from the Domain Data Model; use this to guide the grouping of Operations into both Products and Services
10. Isolation of Context from Content – Separate the core data elements of the Interface Data Model from its qualifying elements; establish a separate Type Hierarchy for Context; use Context to categorise Variants and minimise Versions
11. Low External Coupling (“Need to Know”) – Expose the minimum possible set of data to service consumers; make the data as generic as possible but exposing the business intent (i.e., conforming to the “Goldilocks” signature).
12. Type Hierarchy – Create a data type hierarchy for both Content and Context elements; use this to expose multiple Variants as a single logical operation
13. Identity Association – Ensure that entity identifiers do not leak out through the interface; provide a mapping between external and internal identifiers.

Technology Layer Principles

14. Extraneous Constraints – Avoid introducing fresh dependencies between unrelated components in the process of implementing business logic
15. Logic Bundling – Avoid combining unrelated pieces of data or logic
16. State (“Stickiness”) – Avoid tight and unwarranted associations between instances of data/logic and physical components
17. Topology Hotspots – Avoid associating physical components together in specific layouts and hard-wired connections unless warranted
18. Late Binding – Delay unavoidable dependencies till the last responsible juncture

When you put these principles together with the entities at each layer, it forms a simple and mutually reinforcing set of techniques, and the complete approach is what is "Dependency-Oriented Thinking". This is an extract from the document "Dependency-Oriented Thinking: Volume 1 - Analysis and Design":


There is a companion document on Governance and Management. Feel free to download both documents:

Dependency-Oriented Thinking: Volume 1 - Analysis and Design
Dependency-Oriented Thinking: Volume 2 - Governance and Management

Monday, May 20, 2013

SOA As Dependency-Oriented Thinking - One Diagram That Explains It All


I've been talking and writing about SOA as "Dependency-Oriented Thinking" for a while now, and have even conducted a couple of workshops on the theme. The feedback after the interactive sessions has always been positive, and it surprises me that such a simple concept is not already widely known.

I'm in the process of splitting (slicing?) my white paper "Slicing the Gordian Knot of SOA Governance" into two, one dealing with SOA ("Dependency-Oriented Thinking" or DOT) and the other dealing with Governance and Management ("Dependency-Oriented Governance and Management" or DOGMa).

Partway through the DOT document, I realised that one of the diagrams in it explains the entire approach at a glance.

Here it is (updated after the release of the Dependency-Oriented Thinking documents). Click to expand.



This is of course the BAIT model of an organisation, with a specific focus on Dependencies. BAIT refers to Business, Application, Information (Data) and Technology, the four "layers" through which we can trace the journey of application logic from business intent to implementation.

[Basic definitions: SOA is the science of analysing and managing dependencies between systems, and "managing dependencies" means eliminating needless dependencies and formalising legitimate dependencies into readily-understood contracts.]

At the Business layer, the focus on dependencies forces us to rationalise processes and make them leaner. Business processes need to be traceable back to the organisation's vision (its idea of Utopia), its mission (its strategy to bring about that Utopia) and the broad functions it needs to have in place to execute those strategies (Product Management, Engineering, Marketing, Sales, etc.). Within each function, there will need to be a set of processes, each made up of process steps. Here is where potential reuse of business logic is first identified.

At the end of this phase, we know the basic process steps (operations) required, and how to string them together into processes that run the business. But we can't just have these operations floating around in an organisational "soup". We need to organise them better.

At the Application layer, we try to group operations. Note that the Business Layer has already defined the run-time grouping of operations into Processes. At the application layer, we need to group them more statically. Which operations belong together and which do not? That's the dependency question that needs to be asked at this layer.

The answer though, is to be found only in the Information layer below, because operations only "belong" together if they share a data model. As it turns out, there are two groups of data models, those on the "outside" and those on the "inside". The data models on the "inside" of any system are also known as "domain data models", and these are never visible to other systems. In contrast, a data model on the "outside" of a system, known as an "interface data model", is always exposed and shared with other systems. In SOA, data on the outside is at least an order of magnitude more important than data on the inside because it impacts the integration of systems with one another, whereas data on the inside is only seen by a single system.

Version churn is a potential problem at the Information Layer, because changing business requirements could result in changed interfaces. With a well-designed type hierarchy that only exposes generic super-types, the interface data model can remain stable even as newer implementations pop up to handle specialised sub-types. Most changes to interfaces are then compatible with older clients, and incompatible changes are minimised.

Once we have our data models, we can go back up one level to the Application layer and start to group our operations in two different ways, depending on whether they share an internal (domain) data model or an interface data model. Operations sharing a domain data model form Products. Operations sharing an interface data model form Services. (And that's where the "Service" in "Service-Oriented Architecture" comes from.) Products are "black boxes" meant to be used as standalone applications. Services are "glass boxes" with no other function than to loosely bundle together related operations.

Finally, we have to implement our Services. The description and deployment bundles that are used need not correspond one-to-one with the Services themselves. They should in general be smaller, so that the volatility (rate of change) of any single operation does not needlessly impact others merely because they share a description bundle (e.g., a WSDL file) or a deployment bundle (e.g., a JAR file). If we also pay attention to the right types of components to use to host logic, transform and adapt data, and coordinate logic, we will be implementing business intent in the most efficient and agile way possible.

This, in fact, is all there is to SOA. This is Dependency-Oriented Thinking in practice.

The white paper will explain all this in much greater detail and with examples, but this post is an early appetiser. [Update: you can fall to and help yourselves, since the main course is now ready. Grab the two documents below.]


Dependency-Oriented Thinking: Volume 1 - Analysis and Design
Dependency-Oriented Thinking: Volume 2 - Governance and Management