Targeting A Better Life Above The Service Tier

In 2007, Rajat Taneja, Vikrant Todankar and I co-wrote a white paper called Life above the Service Tier. This was back when web applications were built using server-side frameworks. Our paper decried these frameworks as representing anti-patterns, and recommended that web applications ought to run independently within client devices (such as mobile phones and desktops), and only make service calls to the back-end.

We called this architecture SOFEA, for Service-Oriented Front-End Architecture.

The core principle of SOFEA was to decouple the three orthogonal concerns of Application Download, Presentation Flow, and Data Interchange, which were tightly coupled in server-based frameworks.

This recommendation seemed revolutionary at the time, as witnessed by the many blog posts and comments that it received. However, when we fast-forward 16 years to today, SOFEA is in fact the way the vast majority of web applications are being built. All the popular web frameworks, such as Angular, React, Vue, and Svelte, are SOFEA-based.

Everyone should have lived happily thereafter, but the main problem being faced by development shops today is that these web frameworks are too complex and heavyweight. It takes months or years to acquire true expertise in any of them. Full Stack developers are hard to come by, because both the back-end and the front-end require in-depth skills. This has made it costly and time-consuming not only to build applications, but also to maintain them over their lifetime.

Rajat, Vikrant and I discussed this once again, and we were joined by another friend, Subbu Balakrishnan.

The time seemed ripe to release an update to the SOFEA paper, this time recommending a simpler architecture for the front-end. We have been tracking developments in the industry over the past 16 years, and have detected some paradigm shifts taking place, which the most popular web frameworks have not taken advantage of. Other frameworks have leapfrogged them, but aren't as well-known, or as comprehensive.

Here's our high-level view of industry trends.

What we have come up with is a new architectural style, a successor to SOFEA that we call BUCAREST (for Browser-based Unified Client Architecture (using) REST). This is not just a theoretical concept, but practically implementable using frameworks that are available today. While no single framework today can implement BUCAREST in its entirety, a combination of frameworks used in alignment with the patterns we describe will do the job just fine.

The two frameworks we have identified are Alpine.js and HTMX.

The diagram describing this architecture is here:

This new white paper, which describes BUCAREST in detail and explains how to use it to build web applications in a much simpler way, is called Targeting a Better Life above the Service Tier.

We hope the BUCAREST style will greatly simplify web development in the coming years. Thanks to its simplicity, it should also be possible for those who have considered themselves back-end developers to venture into front-end developent too. We daresay there will be more "Full Stack" developers in the years to come.

Download our white paper, have a read and tell us what you think!

How To Understand Systems - My Secret Superpower

In Poul Anderson's SF story "The Problem of Pain", there is the following quote:

The primitive faiths see God or the gods, as power; the higher ones see Him as justice; the highest see Him as love.

There is also the quote attributed to Eleanor Roosevelt:

Great minds discuss ideas; average minds discuss events; small minds discuss people.

The common thread through both of these quotes is that there is a lower and a higher way to think, with some intermediate level too.

Over my 32-year IT career, I have come across three ways by which people tend to understand systems, and I would express my findings in an analogous manner:

The shallowest understanding of systems comes from a study of user interfaces; an intermediate level of understanding comes from a study of processes; the deepest level of understanding comes from a study of data.

Let me elaborate.

There are many people who work in the IT industry, and not all of them have undergone rigorous study or qualification in IT disciplines. Many have just drifted into it from other functions, and they often believe they can make a contribution based on their understanding of the field and of the software systems they are familiar with. Yet a lot of them exaggerate their competence, because one cannot understand IT systems by a superficial understanding of how they seem to work. There is a lot more to it than this.

The most common misunderstanding is seen in clerical-level operations staff who have only experienced IT systems as users. All they know of a system is its user interface. They see the system as a series of screens, with fields and buttons on each. If asked to describe the system, they will proceed to describe what a user will do on each screen - "Fill up this field, check this box, select a value from this dropdown, then click 'Next'. On the next screen, select this value, select from one of the radio buttons, then click 'Submit'."

Clearly, this level of understanding is superficial, and often misleading, when it comes to determining how to improve or extend a system to do something more.

Senior operations professionals and business people tend to understand systems a lot better than the clerical-level staff who have only experienced a system through its screens. They know what processes and process steps are being represented by those screens. This insight could result in suggestions for improvement of the system in various ways, since processes could be optimised over time. But processes too are limited in the insight they can provide into a system's design.

Many developers too are stuck at the level of understanding systems through process logic alone.

The ultimate level of insight into a system's design comes from an understanding of its data model. What entities are represented in the system? How are they related to each other? What constraints govern those relationships? In formal practice, this is known as Entity-Relationship Modelling.

The actual data stored in these data structures forms a secondary source of important information. The relative volumes of data in different data structures provide valuable information of their own, and usage patterns provide another dimension.

I identify most as a data analyst. When people describe systems to me, I mentally draw Entity-Relationship Diagrams. This has now become second nature to me. Unless I understand the entities in a system and their relationships, I'm not comfortable that I understand anything at all. As processes are explained to me, my understanding of them is formed in terms of the data entities being populated, updated, relationships established, etc. In other words, I see processes in terms of their effects on data. And when I see screens, I can see them as manifestations of process steps, just as business people can.

Three ways of understanding systems, from the lowest (most superficial and least effective) to the highest (most insightful and effective) - Click to expand

My focus on data is my secret superpower. It has helped me whenever I have been called in to consult on a problem. I have been able to diagnose problems within hours, sometimes within the first hour, where others have been stumped for days. I have been able to propose innovative and elegant approaches to enhance the capability of systems, without the elaborate and expensive projects proposed by others. When you can see data first and foremost, a process as something that changes data, and a screen as simply representing a step in a process, you can understand a system in its entirety. Your depth of vision is then so great, and your insights so profound, as to amaze coworkers who have not been similarly trained.

Yet it is not magic. It is very simple science. The focus on data - data modelling, data design - is fundamentally important to an IT professional. Alas, it is not more widely taught in the IT industry.

The Difference Between Knowledge And Wisdom

I realised today that I have become wise. I'm writing this out of a sense of amusement rather than vanity.

I had a discussion with a client about some changes she wanted made on an application. I realised even as she was speaking what changes I would need to make to the underlying database in order to display the results she wanted on the screen. After that conversation, I sat down to write out the SQL script that would make the required changes to the database.

But I didn't run it.

And at that moment, I realised the difference between knowledge and wisdom.

Knowledge refers to the capacity to understand how to implement something.

Wisdom is the meta-knowledge, born of years of experience, that it's rarely that straightforward.

It's the experience of receiving another email or phone call from the client within a few hours saying, "Hey Ganesh, I just discussed this with the team, and we realised that we need the system to do something more. So when you implement your change, can you please ensure that it also does X?"

It's the experience of looking at a script I wrote earlier and thinking, "Hey, why am I doing it like this? I can do it much more simply in this other way!"

And then I have to change the script I initially wrote, and undo all the changes I had made earlier.

That's really what's changed in me. I knew SQL even 25 years ago, and I could possibly have come up with the same script even back then.

That's knowledge, and I haven't significantly added to that knowledge in so many years.

The difference is that 25 years ago, I would have implemented that change immediately.

Today, I have a voice inside me that says, "Just wait. Save this script in a file and look at it again tomorrow."

That's wisdom.

Why The Aadhar Data Breach Is A Very Big Deal

A sting operation by the newspaper The Tribune has reported that with a very nominal payment of 500 rupees, their team was able to get access to an Aadhar portal that was intended for use only by authorised officials responsible for helping citizens retrieve lost or forgotten data.

Aadhar is a relatively new unique identifier for all residents of India

When the news broke, the organisation in charge of India's unique ID database, the Unique Identification Authority of India (UIDAI), played down suggestions that a data breach had taken place. Their main contention was that the biometric data (fingerprints and iris scans) of residents was stored in a secure and encrypted manner, and that it was not exposed through the mechanism used by the Tribune.

Let us analyse what happened, and why it is in fact a big deal.

There are four primary security threats that organisations have to guard against:

• Disclosure (unauthorised persons gaining access to sensitive information)
• Deception (the system being presented with false data and made to accept it)
• Disruption (interruption in normal operations and loss of service)
• Usurpation (unauthorised persons gaining control of the system)

What the UIDAI is saying is that there has been no Disclosure of authentication tokens (biometric data) that could lead to a future Deception or Usurpation attack. But importantly, they have not denied that a breach has occurred which could have given an unauthorised user access to certain kinds of information. In fact, some of their officials have confirmed this:

Sanjay Jindal, Additional Director-General, UIDAI Regional Centre, Chandigarh, accepting that this was a lapse, told The Tribune: "Except the Director-General and I, no third person in Punjab should have a login access to our official portal. Anyone else having access is illegal, and is a major national security breach."

In other words, an attack could have successfully taken place resulting in the Disclosure of certain kinds of information.

What kind of information? According to the Tribune article, they were able to retrieve the following data on provision of an Aadhar number:

• Name
• Address
• Post code (PIN)
• Photo
• Phone number
• Email address

In fact, this is probably the minimal set of data that is accessible through the portal.

It took just Rs 500, paid through Paytm, and 10 minutes in which an “agent” of the group running the racket created a "gateway" for this correspondent and gave a login ID and password. Lo and behold, you could enter any Aadhaar number in the portal, and instantly get all particulars that an individual may have submitted to the UIDAI (Unique Identification Authority of India), including name, address, postal code (PIN), photo, phone number and email.

Reading between the lines, it doesn't appear that an API (a non-visual means by which a software program could retrieve data) was provided. It looks like login credentials were provided into an administrator's portal, in other words, access to a secure web page.

What is the worst that could happen if such access were granted to a non-authorised user? How much data could they steal? In other words, how severe is the Disclosure exposure?

To a layperson, this may not seem like a huge exposure. How many people's data can a person steal by sitting at a portal and entering Aadhar numbers on a screen? A few hundred, perhaps a couple of thousand records. Unfortunately, a potential data thief is much more efficient.

I'm not a security expert, but even I would potentially be able to steal the entire Aadhar database (i.e., not biometric data but the set of data listed above) in a matter of hours or days, without much personal effort on my part. Hackers may have much more efficient means at their disposal, but I would probably use a web application testing tool like Selenium, which I use for user testing of software as part of my day job. Selenium is built on top of a basic browser "engine", and has a number of programmable features built around it. It can be "trained" by a user to follow a certain sequence of steps, and it can then repeat that sequence of steps ad nauseam, using different data values based on its controlling script. Best of all, Selenium looks just like a regular browser to any web server, so the server would have no idea that an automated tool is logging in and moving around the site, and not a human user.

An article from DeveloperFusion explains how Selenium works

Assuming I have been granted login credentials into the Aadhar administrator's portal (on payment of the going rate of 500 rupees), I would first put Selenium into "learning mode", where it would record my actions to be replayed later. I would use it just as I would use a regular browser, except that this browser is recording my every action, including the data values I am entering. Based on its observation of my actions, it would then know which URL to navigate to, what username and password to enter on the login form, which menu item to select to get to the search screen, etc. Then it would learn about the repeatable part of the task, which is the entry of an Aadhar number in a certain field, and a click on a Search or Retrieve button. When the system responded with the data of the resident (name, address, etc.), I would stop the learning mode. (In practice, I would train Selenium to deal with invalid Aadhar numbers also, since it should be able to recognise when a query was unsuccessful.)

I would then look into the script that Selenium generated to describe the sequence of actions that it had learnt. I would modify this script to make Selenium repeat its search in a loop.

Aadhar numbers are 12-digit numeric strings, which means they can theoretically be any number in the range "000000000000" to "999999999999", a trillion (10¹²) numbers in all. I would modify the script to loop through all trillion numbers, perhaps in lots of a million at a time, and I would get it to extract the data from within specific HTML tags on the result screen. If these tags had helpful "id" attributes, it would make my job much easier, otherwise I would have to rely on the relative position of each field's tag within the returned page (known as a DOM search). This is how I would get Selenium to "read" the returned name, address, postcode, phone number, email address,  etc.

The last thing I would do within the loop is to record the Aadhar number and all the retrieved fields into a local database on my machine, provided the query returned a value. I expect that only about a billion of the trillion numbers I use in my loop will return valid data, since there are just about a billion Indian residents.

And this is how I would collect the basic personal and contact details of every single resident of India. Unless UIDAI has a throttling or choking mechanism to prevent such a rapid-fire query of data, it's possible that I will be able to get away with this over a couple of days at most.

More sophisticated hackers would hide behind multiple IP addresses, and use multiple sets of user credentials, over a longer time period, so as to fool any auditing system on the UIDAI side from realising that a bulk theft of records was underway.

Since the Aadhar numbers are permanent identifiers for residents, the data in this database is likely to remain useful for decades to whoever steals it. It can form the foundation of a database to which other data can be added.

And that brings us to the even bigger threat that this breach enables.

The Indian government has also committed another cardinal sin from a privacy angle. It has mandated the linking of Aadhar numbers to a number of other important data, for example, bank account numbers and mobile SIM cards. In fact, although the courts have ruled that such linking is to be voluntary, the government, banks and telcos are not endorsing that liberal message at all. Residents are being virtually threatened into providing their Aadhar numbers to their telecom providers and their banks. Aadhar is also being used across various arms and services of the government, such as the Tax department and the Public Distribution System.

Now, the databases of such organisations are generally a lot less secure than the biometric data stored by UIDAI. It is unlikely that banks and telcos are being diligent enough to encrypt their data. Besides, with just a few large banks and telcos operating in the country, it is relatively easy for a malicious organisation, such as the secret service agency of a foreign power, to perform the necessary "social engineering" required to access this data. No conventional "hacking" is even necessary for such simple data theft. The same goes for government departments holding Aadhar-linked data.

Now, if one had a list of bank account numbers with Aadhar numbers against them, it would be a trivial matter to map these to the Aadhar database stolen from UIDAI. One would then have the personal and contact details of every resident of India, -- plus their bank account numbers!

Repeat this for mobile SIM cards, ration cards, PAN cards (tax), etc.

Now one has put together a very lucrative set of data that a number of hostile powers would be very interested in. The cost of acquiring this data, as I have shown, is well within their budgets.

The Tribune's report also claims

Spotting an opportunity to make a quick buck, more than one lakh VLEs (Village-Level Enterprises) are now suspected to have gained this illegal access to UIDAI data to provide “Aadhaar services” to common people for a charge, including the printing of Aadhaar cards. However, in wrong hands, this access could provide an opportunity for gross misuse of the data.

Indeed, if the vulnerability has been around for the last few months, as suspected, it would not be unreasonable to assume that sensitive information on every Indian resident is now sitting in a Big Data lab in more than one foreign country, subject to sophisticated analysis and insight mining. In fact, if organisations like the NSA have not already acquired this data, my opinion of their competence has plummeted.

It's nothing short of a national security disaster.

Still, from the muted press around it, it appears that no one has a realistic handle on how grave the breach is. Critics of the government have seized on The Tribune's initial exposé to claim that Aadhar has been a massive security failure, but without understanding the nuances of what we have seen in this analysis. Supporters of the government have seized on the UIDAI's reassurances to downplay the significance of the breach, again without a sophisticated analysis of the potential exposure. Both sides are being irresponsible.

My conclusion is that a serious breach of data security has been proven by The Tribune. This vulnerability has probably been around for a few months, enough time for a competent organisation or set of individuals to steal the master data (about a billion records), and create a foundation to add more useful data as it is acquired, since the Aadhar number is a permanent identifier that is likely to be associated with every significant product or service that residents may own or use.

A security assessment of the implications is imperative. At the very least, linking of Aadhar numbers to services must be halted.

And it would not be unreasonable to demand that someone somewhere should resign.

Update 24/03/2018: It appears that nothing has been done to fix the Aadhaar breach. If anything, even more breaches have been discovered, as described in this ZDNet article.

Designing A Tamper-Proof Electronic Voting System

India's Electronic Voting Machine (EVM)

India's Electronic Voting Machines (EVMs) have been in the news a lot lately, and not always for the right reasons. There have been complaints by some candidates that when voters pressed the button in their favour, another party's symbol lit up. Faced with accusations that the EVMs may have been hacked (especially with the string of electoral successes of the ruling Bharatiya Janata Party), India's Election Commission has begun to conduct elections with a Voter-Verifiable Paper Audit Trail (VVPAT), which prints out the details of the party/candidate that a voter selected, so they can verify that their vote was registered correctly.

An EVM unit with a printer to provide immediate paper verification to voters

As a systems architect, I'm afraid I have to say that may not be good enough. Let me explain why, and then suggest a more foolproof alternative.

First of all, my familiarity with IT security tells me never to believe that a hardware device has in-built safeguards. We have all heard about how backdoors can be built into hardware, with whispers about the Russian mafia or Chinese government having control of fabrication plants that produce integrated circuits, so we know it's at least theoretically possible for criminal elements to inject malicious logic right into the hardware of an electronic device.

At the same time, I believe it's being Luddite to advocate a return to entirely paper-based ballots. It's true that many Western countries stick to paper ballots for the sheer auditability of a poll (which electronic voting makes opaque), but India has had bad experiences with paper-based polls in the past, with uniquely Indian subversions of the system such as "booth-capturing", as well as more conventional forms of fraud like "ballot-stuffing".

No, there's no going back to purely paper-based ballots, but there are serious vulnerabilities with the electronic voting system, even with VVPAT.

Let me illustrate.

The basic EVM logic is as follows. The voter presses a button corresponding to their preferred party or candidate, and the machine confirms their selection by lighting up the corresponding election symbol (because many voters are illiterate and can't read). The choice is also recorded in the unit's memory. After the polls close, all the voting units are collected and connected to a central unit that tallies the votes in each. Once all units have uploaded their votes to the central unit, the results of that election can be announced, with the tallies of all parties and candidates available.

Now, based on voter complaints about the wrong symbol lighting up, here's what many people suspect happened. Somehow (never mind how) a hack was introduced into some of the units that recorded a selection of party A as a selection of party B.

This is actually a pretty amateurish hack, as I'll explain shortly. It's readily detectable by an alert voter. What the Election Commission is attempting to do with the Voter-Verifiable Paper Audit Trail (VVPAT) is to make the voter's selection more explicit, in the hope that more of them will be forced to verify that their choice was correctly recorded. It does not make the system more secure in the sense of being able to trap more subtle hacks.

Here's the schematic of the basic logic when things go well.

(Click to expand)

When faced with a simple hack like the suspected one above, the system will respond as below.

(Click to expand)

However, any hacker with a little more smarts will realise that their subversion will have to be less readily detectable. In other words, the hack would have to be placed in a slightly different place.

(Click to expand)

With this kind of hack, any mischief would be virtually undetectable. Both the lighted symbol and the paper printout would confirm to the voter that their choice was faithfully recorded, yet their vote would have been subtly hijacked in favour of another party.

The logic of the hack could be designed to be extremely subtle indeed. Instead of switching every single vote from party A to party B, it could be designed to apply a random function so that, on average, only 1 in N votes was switched across. In many marginal constituencies, even a small skimming of votes would be enough to tip the balance, so desired results could be achieved without any suspiciously large vote swings. There could even be a threshold below which the logic would not start to kick in, say a few thousand votes. That way, if the Election Commission conducted a few test runs to ensure that a unit was working correctly, it would not arouse suspicions.

Now all this seems depressing. Is there any way to combat this?

Yes, there is, but it's not purely in hardware and software. If it were, this post would have been titled "Designing A Tamper-Proof Electronic Voting Machine". The system that we design needs to incorporate electronic and manual elements.

What we need are not one but two printouts for every vote. One copy is for the voter's own records. The other is for the Election Commission. The voter must verify that both match their selection, then place the EC copy into a ballot box before they leave the booth, just like in a paper-based poll. However, this paper ballot will only be used for verification, not for the actual vote tally on counting day, otherwise we may as well go back to a purely manual vote count.

(Click to expand)

A number of statistical techniques may be used to sample and test the performance of voting machine units in various constituencies.

Under the most pessimistic scenario, the ballot boxes of every single booth will be tallied offline, and the counting may continue for weeks after the official results. Elections will only be rescinded if the manual tally grossly contradicts the electronic one (there will always be minor discrepancies due to voter or official error).

Under less pessimistic scenarios, a random sample of booths may be chosen for such manual verification. If gross discrepancies are detected in any booth, then all of the ballot boxes in that constituency will have to be manually tallied. If more than a certain number of constituencies show suspicious results, then the tally may be expanded to cover an entire state, and so on.

There can be further refinements, such as ensuring that the random sample of booths to be verified is drawn publicly, after the voting is completed, so as to afford no opportunity for malicious elements to know in advance which booths are "safe" from being audited.

In general, the design of the overall process is meant to detect subversions after the fact, so the technically accurate term is tamper-evident rather than tamper-proof. However, advertising the fact that such audits will be taking place may deter malicious elements from attempting these hacks in the first place. Hence, in a larger sense, the system consisting of the combined electronic and manual process, plus a widespread foreknowledge of an inevitable audit, may result in a tamper-proof system after all.

Democracy works because citizens have faith that their will is reflected in the results of elections. If citizens lose faith in the electoral process, it could cause a breakdown in society, with violent revolution in the worst case. That's why it's important to act quickly to restore faith in the process, even if this makes the process costlier.

As the quote commonly attributed to Thomas Jefferson goes, "Eternal vigilance is the price of freedom."

My Books On Dependency-Oriented Thinking - Why They Should Count

InfoQ has published both volumes of my book "Dependency-Oriented Thinking". The links are below.

Dependency-Oriented Thinking: Volume 1 - Analysis and Design (Service-Oriented Architecture Made Simple)

Dependency-Oriented Thinking: Volume 2 - Governance and Management (Control Systems Made Simple)

I'll admit it feels somewhat anticlimactic for me to see these books finally published, because I finished writing them in December 2013 after about two years of intermittent work. They have been available as white papers on Slideshare since Christmas 2013. The last seven months have gone by in reviews, revisions and the various other necessary steps in the publication process. And they have made their appearance on InfoQ's site with scarcely a splash. Is that all?, I feel like asking myself. But I guess I shouldn't feel blasé. These two books are a major personal achievement for me and represent a significant milestone for the industry, and I say this entirely without vanity.

You see, the IT industry has been misled for over 15 years by a distorted and heavyweight philosophy that has gone by the name "Service-Oriented Architecture" (SOA). It has cost organisations billions of dollars of unnecessary spend, and has fallen far short of the benefits that it promised. I too fell victim to the hype around SOA in its early days, and like many other converted faithful, tried hard to practise my new religion. Finally, like many others who turned apostate, I grew disillusioned with the lies, and what disillusioned me the most was the heavyhandedness of the "Church of SOA", a ponderous cathedral of orthodox practice that promised salvation, yet delivered nothing but daily guilt.

But unlike others who turned atheist and denounced SOA itself, I realised that I had to found a new church. Because I realised that there was a divine truth to SOA after all. It was just not to be found in the anointed bible of the SOA church, for that was a cynical document designed to suit the greed of the cardinals of the church rather than the needs of the millions of churchgoers. The actual truth was much, much simpler. It was not easy, because "simple" and "easy" are not the same thing. (If you find this hard to understand, think about the simple principle "Don't tell lies", and tell me whether it is easy to follow.)

I stumbled upon this simple truth through a series of learnings. I thought I had hit upon it when I wrote my white paper "Practical SOA for the Solution Architect" under the aegis of WSO2. But later, I realised there was more. The WSO2 white paper identified three core components at the technology layer. It also recognised that there was something above the technology layer that had to be considered during design. What was that something? Apart from a recognition of the importance of data, the paper did not manage to pierce the veil.

The remaining pieces of the puzzle fell into place as I began to consider the notion of dependencies as a common principle across the technology and data layers. The more I thought about dependencies, the more things started to make sense at layers even above data, and the more logical design at all these layers followed from requirements and constraints.

In parallel, there was another train of thought to which I once again owe a debt of gratitude to WSO2. While I was employed with the company, I was asked to write another white paper on SOA governance. A lot of the material I got from company sources hewed to the established industry line on SOA governance, but as with SOA design, the accepted industry notion of SOA governance made me deeply uncomfortable. Fortunately, I'm not the kind to suppress my misgivings to please my paymasters, and so at some point, I had to tell them that my own views on SOA governance were very different. To WSO2's credit, they encouraged me to write up my thoughts without the pressure to conform to any expected models. And although the end result was something so alien to establishment thought that they could not endorse it as a company, they made no criticism.

So at the end of 2011, I found myself with two related but half-baked notions of SOA design and SOA governance, and as 2012 wore on, my thoughts began to crystallise. The notion of dependencies, I saw, played a central role in every formulation. The concept of dependencies also suggested how analysis, design, governance and management had to be approached. It had a clear, compelling logic.

I followed my instincts and resisted all temptation to cut corners. Gradually, the model of "Dependency-Oriented Thinking" began to take shape. I conducted a workshop where I presented the model to some practising architects, and received heartening validation and encouragement. The gradual evolution of the model mainly came about through my own ruminations upon past experiences, but I also received significant help from a few friends. Sushil Gajwani and Ravish Juneja are two personal friends who gave me examples from their own (non-IT) experience. These examples confirmed to me that dependencies underpin every interaction in the world. Another friend and colleague, Awadhesh Kumar, provided an input that elegantly closed a gaping hole in my model of the application layer. He pointed out that grouping operations according to shared interface data models and according to shared internal data models would lead to services and to products, respectively. Kalyan Kumar, another friend who attended one of my workshops, suggested that I split my governance whitepaper into two to address the needs of two different audiences - designers and managers.

And so, sometime in 2013, the model crystallised. All I then had to do was write it down. On December 24th, I completed the two whitepapers and uploaded them to Slideshare. There has been a steady trickle of downloads since then, but it was only after their publication by InfoQ that the documents have gained more visibility.

These are not timid, establishment-aligned documents. They are audacious and iconoclastic. I believe the IT industry has been badly misled by a wrongheaded notion of SOA, and that I have discovered (or re-discovered, if you will) the core principle that makes SOA practice dazzlingly simple and blindingly obvious. I have not just criticised an existing model. I have been constructive in proposing an alternative - a model that I have developed rigorously from first principles, validated against my decades of experience, and delineated in painstaking detail. This is not an edifice that can be lightly dismissed. Again, these are not statements of vanity, just honest conviction.

I believe that if an organisation adopts the method of "Dependency-Oriented Thinking" that I have laid out in these two books (after testing the concepts and being satisfied that they are sound), then it will obtain the many benefits of SOA that have been promised for years - business agility, sustainably lower operating costs, and reduced operational risk.

It takes an arc of enormous radius to turn around a gigantic oil tanker cruising at top speed, and I have no illusions about the time it will take to bring the industry around to my way of thinking. It may be 5-10 years before the industry adopts Dependency-Oriented Thinking as a matter of course, but I'm confident it will happen. This is an idea whose time has come.

An Example Of Public Money Used For The Public Good

I've always held that Free and Open Source Software (FOSS) is one of the best aspects of the modern IT landscape. But like all software, FOSS needs constant effort to keep up to date, and this effort costs money. A variety of funding models have sprung up, where for-profit companies try to sell a variety of peripheral services while keeping software free.

However, one of the most obvious ways to fund the development of FOSS is government funding. Government funding is public money, and if it isn't used to fund the development of software that is freely available to the public but spent on proprietary software instead, then it's an unjustifiable waste of taxpayers' money.

It was therefore good to read that the Dutch government recently paid to develop better support for the WS-ReliableMessaging standard in the popular Open Source Apache CXF services framework. I was also gratified to read that the developer who was commissioned to make these improvements was Dennis Sosnoski, with whom I have been acquainted for many years, thanks mainly to his work on the JiBX framework for mapping Java to XML and vice-versa. It's good to know that talented developers can earn a decent dime while doing what they love and contributing to the world, all at the same time.

Here's to more such examples of publicly funded public software!

A Neat Tool To Manage Sys V Services in Linux

I was trying to get PostgreSQL's "pgagent" process (written to run as a daemon) to run on startup like other Linux services, and came upon this nice visual (i.e., curses) tool to manage services.

It's called "sysv-rc-conf" (install with "sudo apt-get install sysv-rc-conf"), and when run with "sudo sysv-rc-conf", brings up a screen like this:

It's not really "graphics", but to a command-line user, this is as graphical as it gets

All services listed in /etc/init.d appear in this table. The columns are different Unix runlevels. Most regular services need to be running in runlevels 2, 3, 4 and 5, and stopped in the others. Simply move the cursor to the desired cells and press Tab to toggle it on or off. The 'K' (stop) and 'S' (start) symbolic links are automatically written into the respective rc.d directories. Press 'q' to quit the tool and satisfy yourself that the symbolic links are all correctly set up.

You can manually start and stop as usual:

/etc/init.d$ sudo ./myservice start

/etc/init.d$ sudo ./myservice stop

Plus, your service will be automatically started and stopped when the system enters the appropriate runlevels.

Enjoy.

The End Of Ubuntu One - What It Means

Although a big fan of Ubuntu Linux as a desktop OS, I've never been interested in their cloud storage platform Ubuntu One, and found it a bit of a nuisance when asked to sign up for it every time I installed the OS.

Now Ubuntu One is being shut down. I'm 'meh' but still a bit surprised.

The linked article talks about mobile, and how new mobiles such as the Ubuntu-powered ones need cloud storage to succeed. If so, isn't it really bad timing for Canonical to walk away from a fully operational cloud platform just when its mobile devices are entering the market?

Ubuntu-powered smartphones
(Do you know what the time on the middle phone refers to?)

I think it's about economics.

Ubuntu's statement says:

If we offer a service, we want it to compete on a global scale, and for Ubuntu One to continue to do that would require more investment than we are willing to make. We choose instead to invest in making the absolute best, open platform and to highlight the best of our partners’ services and content.

Hmm. I read this as Canonical trying to build a partner ecosystem that will substitute for having a big cloud-and-mobile story like Google does, without the investment that such a proprietary ecosystem will require. Let's see if they succeed.

The other side-story in the linked article is about telcos and their role. Having worked at a telco over the last two years, I can confirm that the major fear in the telco industry is being reduced to commodity carriers by "over the top" services. The telcos are fighting to offer content, and will want willing mobile wannabe partners like Mozilla and Canonical to offer smartphone platforms that will work with networking infrastructure and make the telcos more attractive (through content that both players source from content providers). It will be interesting to see how this four-way, federated partnership (between multiple telcos, independent smartphone platform vendors like Mozilla and Canonical, smartphone device OEMs and content providers) will play out. Many of these companies will think of themselves as the centre of the Universe and the others as partners.

"Nothing runs like a fox" - Well, let's see if the Firefox Smartphone has legs

In the meantime, some good news for startup cloud providers ("startup" only with respect to the cloud, since they will still need deep pockets to set up the infrastructure!): Canonical is open-sourcing its Ubuntu One storage code “to give others an opportunity to build on this code to create an open source file syncing platform.” This should be interesting.

Tools for HTML Table and Browser-Side Database Manipulation

I've been having a lot of fun building a realistic-looking demo app that is meant to showcase the features of a coming product. A big challenge in such cases is obviously dynamic data, since hard-coded data won't cut it. It needs to behave like a web app, providing the illusion of a server-side database, but without a server-side database.

HTML5 provides built-in client-side persistence features called sessionStorage and localStorage, but when the data requirements are complex, as in my application, these just aren't adequate. What I need is a client-side database like SQLite. As it turns out, different browsers have taken different routes to providing client-side databases, and it made my head hurt to read about Web SQL and SQLite and which browser incorporated which database and which browser abandoned which database. Finally, I found a JavaScript library called HTML5SQL.js that promised to abstract all those implementation issues from me, giving me a standard API to work with.

So far, it's worked great for me and seems very powerful and flexible. So that's the first tool I'd recommend, for client-side database manipulation - HTML5SQL.js.

Then I had the challenge of building HTML tables that would display this data, allow the user to manipulate it visually and save changes. After a lot of searching, I found another nice JavaScript library called DataTables. DataTables provides lots of powerful features, including sortable columns and pagination. These work even in the default settings with no configuration.

So that's the second tool I'd recommend, for HTML table manipulation - DataTables.js.

By blending HTML5SQL and DataTables functions, I could create HTML tables from a client-side  database. Even better, I could open multiple tabs on the browser, each representing a different part of the application, and they could all see the same data, because the database is a global resource to all tabs in the browser. Best of all, the data stayed persistent across browser restarts. It's a true database, with persistence, relational structure and SQL smarts.

Other tools:

Needless to say, jQuery has now become required technology, and a web developer simply cannot leave home without it. It's so powerful I'm still looking for the right categorisation to describe it.

Many of the new JavaScript libraries are built in an asynchronous style, so a rather naive usage assuming synchronous behaviour (that a function invoked first will complete before the next invoked function) will lead to some surprises. If you need to ensure that two asynchronous functions are executed in strict sequence, you will need to resort to callbacks, as explained on StackOverflow.

For visual designers, the old method of using HTML tables to lay out components is now passé. Using CSS layouts is the in thing. A popular CSS layout framework is called the "960 Grid System", or 960.gs. It's named for the fact that most modern screens have a width of at least 960 pixels, and this number lends itself to being divided into 12 or 16 columns with spaces or gutters in-between. A newer one is unsemantic, which is said to be better for "responsive" UIs (those that adapt to different screen sizes, such as desktop browsers, tablets and mobile phones), but is a bit more complex to use. In both these frameworks, HTML components such as "div" and "table" elements just need to be given special class names, and the CSS then uses these to lay them out. It's quite neat and powerful, but I'm not working in that area because I'm not a graphic or layout designer. I've just seen the nice effects you can get with a CSS layout framework.

Sydney Workshop "Introduction to Dependency-Oriented Thinking" Held

After weeks of hectic preparation, Rahul Singh and I held our long-awaited workshop today on "Dependency-Oriented Thinking", comprising all-new material from my recent document "Dependency-Oriented Thinking: Vol 1 - Analysis and Design".

Somewhat disappointingly, we only had three signed-up participants, but the numbers only tell half the story. Only one of the three was from Sydney. One flew in from Melbourne the night before, and another got up at 4 am to make the 3+ hour drive from Canberra to Sydney. With such determination on their part, I just had to do my best, and I hope they were happy with the workshop. They certainly expressed satisfaction on their feedback forms :-).

The slides are now available on Slideshare. If anyone was put off from reading the original document on account of its size (264 pages), they could go through this slide pack instead (only 220 slides, heh).

Sydney Workshop On Dependency-Oriented Thinking (Saturday Feb 15, 2014)

Well, it's time to road-test my SOA method "Dependency-Oriented Thinking". I released the two DOT documents before Christmas, Volume 1 on Analysis and Design and Volume 2 on Governance and Management. Now, Rahul Singh and I are conducting an all-day workshop to provide a more interactive instruction into the Analysis and Design aspects of the method. It'll be on Saturday the 15th of February, 2014.

The course fees are $450 plus GST for the all-day program. There's an early bird price of $400 plus GST for those who register on or before the 3rd of Feb.

You can download the workshop brochure and schedule here: http://slidesha.re/1j59nqK

If you're a Sydney-based business analyst, solution architect or senior designer, you may be interested in this workshop. My objective is to make IT professionals more effective by helping them think more powerfully about complex, inter-connected systems. Dependency-Oriented Thinking is just structured common sense, but it doesn't come naturally. It needs to be taught, and it requires practice before it can become second nature.

Dependency-Oriented Thinking - Documents Released!

I know I haven't been active on this blog for almost 6 months, but I haven't been idle. Ever since Rahul Singh and I conducted our first workshop on "Dead Simple SOA" in Sydney last year, I have been working on the feedback we received, namely that we needed to split "Analysis & Design" from "Governance & Management" because these two topics are of interest to two different audiences.

Splitting out just Governance & Management from the original document "Slicing the Gordian Knot of SOA Governance" was the easy part. I realised that I didn't have much material on Analysis and Design, so I had to write that from scratch. That task ultimately took the better part of a year.

Well, that's completed now, and the two documents are now ready to download from SlideShare:

Dependency-Oriented Thinking: Volume 1 - Analysis and Design

Dependency-Oriented Thinking: Volume 2 - Governance and Management

I have believed for a while now that the core concept behind SOA ought to be "dependencies". I could explain more here, but I'm exhausted after the marathon effort of the last few months. Besides, you'll find a very detailed argument in Volume 1.

Please do download these documents and have a read. More importantly, give me your feedback. My email address is in the "About the Author" section in both documents.

Dependency Principles

I've been used to thinking of SOA as "Dependency-Oriented Thinking" for such a long time that it suddenly struck me that I should readily be able to postulate "dependency principles" that can be applied at each of the BAIT layers. As it turned out, I was fortunate enough to have some actual case studies to verify my proposed set, and these cases were selected for being representative of different tiers, so they were quite comprehensive.

It turns out that the lesson to be learnt after a post-mortem of each of those case studies was that one or more dependency principles had been ignored or violated, resulting in the problem that was being faced. The corollary is that an organisation that scrupulously adheres to these principles will end up achieving SOA.

So, without further ado, here are the dependency principles (updated on 25/12/2013):

Business Layer Principles

1. Traceability – Enforce core governance; ensure that everything that is done makes sense with respect to the organisation's Vision
2. Minimalism – Challenge assumptions, reject unwarranted constraints, do no more than required, reuse functional building blocks
3. Domain Insight – Understand the true nature of the business; don't be misled by superficialities or conventional wisdom; re-imagine the business from first principles
4. Business Process Coordination Style – Choose a coordination style (orchestration or choreography) based on whether tight control is possible and/or necessary. This influences the choice of technology standard (SOAP or REST), but may also be influenced by a prior choice of standard.

Application Layer Principles

5. High Cohesion (“Belonging”) – What belongs together should go together, with minimal links between distinct systems; question instances of a single logical function split across multiple systems, or multiple logical functions combined within a single system. Group operations that share a Domain Data Model and business logic into Products, those that share an Interface Data Model into Services.
6. Decoupling of Interfaces from Internals – Ensure that no external system develops dependencies on the internal aspects of a system; create an interface to isolate the systems from each other.
7. “Goldilocks” Signatures (Stability versus Precision) – Identify multiple concurrent flavours of each logical operation; establish a way to express the business intent common to all of them; make sure that the interface doesn't change for minor variations in logic but does change with the business intent.
8. Shared Semantics (Negotiation of Intent and Content) – In choreographed processes, ensure that the service provider and service consumer understand the meaning of every Operation as well as the mechanics of how each Operation should be invoked.

Information (Data) Layer Principles

9. Decoupling of Interface Data from Internal Data – Distinguish the Interface Data Model from the Domain Data Model; use this to guide the grouping of Operations into both Products and Services
10. Isolation of Context from Content – Separate the core data elements of the Interface Data Model from its qualifying elements; establish a separate Type Hierarchy for Context; use Context to categorise Variants and minimise Versions
11. Low External Coupling (“Need to Know”) – Expose the minimum possible set of data to service consumers; make the data as generic as possible but exposing the business intent (i.e., conforming to the “Goldilocks” signature).
12. Type Hierarchy – Create a data type hierarchy for both Content and Context elements; use this to expose multiple Variants as a single logical operation
13. Identity Association – Ensure that entity identifiers do not leak out through the interface; provide a mapping between external and internal identifiers.

Technology Layer Principles

14. Extraneous Constraints – Avoid introducing fresh dependencies between unrelated components in the process of implementing business logic
15. Logic Bundling – Avoid combining unrelated pieces of data or logic
16. State (“Stickiness”) – Avoid tight and unwarranted associations between instances of data/logic and physical components
17. Topology Hotspots – Avoid associating physical components together in specific layouts and hard-wired connections unless warranted
18. Late Binding – Delay unavoidable dependencies till the last responsible juncture

When you put these principles together with the entities at each layer, it forms a simple and mutually reinforcing set of techniques, and the complete approach is what is "Dependency-Oriented Thinking". This is an extract from the document "Dependency-Oriented Thinking: Volume 1 - Analysis and Design":

There is a companion document on Governance and Management. Feel free to download both documents:

Dependency-Oriented Thinking: Volume 1 - Analysis and Design
Dependency-Oriented Thinking: Volume 2 - Governance and Management

Red Hat's Competition Is From...Amazon

I was talking to a colleague about Red Hat yesterday, and it struck me that a new competitor has emerged that threatens Red Hat's business model in a fundamental way.

Red Hat has two main lines of business, Linux and JBoss. Yes, they also have a cloud platform called Redshift, but I'm guessing it's not yet of the scale of the other two.

Both Linux and JBoss are Open Source, so the revenues to Red Hat are through support subscriptions. Put very bluntly, Red Hat's business model is based on fear, rather like that of insurance companies. Customer organisations are generally afraid to run unsupported software in their data centres, so they will gladly pay the insurance premiums to ensure that someone stands behind the software they run.

But consider this subtle change in the model of reliability that comes about when we move to a cloud-based platform like Amazon. Reliability is achieved not so much by keeping servers running and getting them back online when there's a problem. The new model of reliability is to simply spin up new instances to replace failed servers. We can create images (AMIs) of our entire software stack, and spin up new instances either in response to increased volumes, or to replace instances that have died. This is actually done automatically, as part of the "elasticity" that cloud platforms deliver.

Mind you, Amazon itself suffers in adoption right now because of its own version of the fear factor.  The thought of placing one's business-critical data and processes in the cloud keeps many CIOs awake at night. But that fear is gradually easing as more and more organisations are seen to be migrating to that cloud platform with no adverse effects.

When Amazon becomes the norm for production infrastructure, the requirement for supported versions of operating systems may well reduce. The main difference between Red Hat Enterprise Linux and Centos (a RHEL clone) is that the former comes with paid support. If the reliability problem can be automatically addressed through elasticity, then Centos will do just as well.

That's why I think Red Hat needs to be afraid of Amazon.

Resources Are Four-Dimensional

The term ROA (Resource-Oriented Architecture) is misleading. It should ideally stand for "Resource/Representation-Oriented Architecture", even though that's quite a mouthful.

I've found in my discussions with people who work on REST-based systems that lots of them are very fuzzy about the notions of "resource" and "representation", even though these are the fundamental concepts underlying REST (and my forthcoming messaging-oriented variant, ROMA).

Let me try and explain this. Let's say I found a space capsule that had fallen out of the sky, and by deciphering the alien glyphs on its surface, I understood that it contained a 4-dimensional object from the planet Royf. Unfortunately, I couldn't take the object out, because ours is a 3-dimensional world and such an operation is impossible. However, I found that it was possible to get the capsule to "project" a 3-dimensional image of the object it contained, so I could "see" it in a way that made sense to my limited mind. I found that I could also ask for the object to be manipulated in 3-dimensional terms. I knew, of course, that the object itself was 4-dimensional and so my instructions had to be somehow translated into terms that made sense in the 4D world. But I found to my satisfaction that the 3D image that resulted from my instructions was exactly what I wanted.

I realised then that my interactions with the space capsule were RESTian. The 4-dimensional object was the resource, an unseeable thing that I had no way of even comprehending and which was therefore mercifully shielded from my vision. What I could ask for (through a GET) was a 3D "representation" of the object, and this was something I could understand. I could also manipulate the object in several ways. I could show the capsule other 3D objects and say, "Change its shape to resemble this", or "Make its colour more like this", and it would happen! Obviously, the objects I was holding up were not the same as the object inside the capsule. They were representations of what I wanted the object to look like, when I saw it in 3D terms.

That's really what REST is. The only aspect of the resource itself that we can directly deal with is its name, or URI. The actual resource is completely unseen, indeed unseeable.  Everything that is actually seen is a representation, whether it's a representation of what the object is like right now, or a representation of what we want the object to be like. Everything that goes "over the wire" in REST is a representation.

Nerds can readily understand what a 3D projection is

[See also that REST is the very opposite of "Distributed Objects", although some industry personalities continue to insist that REST is DO! (JJ, I'm talking to you.) Distributed Objects tries to bring about an illusion that remote objects are local, allowing you to grasp them using virtual reality gloves. REST tries to bring about a discipline that says even local objects like the one inside the space capsule should be treated as remote, and we shouldn't try to grasp them directly, only deal with them indirectly through representations. Distributed Objects works well when it does and fails horribly when it doesn't. REST always works.]

Hopefully, this should set to REST some of the confusion around resources and representations.

SOA As Dependency-Oriented Thinking - One Diagram That Explains It All

I've been talking and writing about SOA as "Dependency-Oriented Thinking" for a while now, and have even conducted a couple of workshops on the theme. The feedback after the interactive sessions has always been positive, and it surprises me that such a simple concept is not already widely known.

I'm in the process of splitting (slicing?) my white paper "Slicing the Gordian Knot of SOA Governance" into two, one dealing with SOA ("Dependency-Oriented Thinking" or DOT) and the other dealing with Governance and Management ("Dependency-Oriented Governance and Management" or DOGMa).

Partway through the DOT document, I realised that one of the diagrams in it explains the entire approach at a glance.

Here it is (updated after the release of the Dependency-Oriented Thinking documents). Click to expand.

This is of course the BAIT model of an organisation, with a specific focus on Dependencies. BAIT refers to Business, Application, Information (Data) and Technology, the four "layers" through which we can trace the journey of application logic from business intent to implementation.

[Basic definitions: SOA is the science of analysing and managing dependencies between systems, and "managing dependencies" means eliminating needless dependencies and formalising legitimate dependencies into readily-understood contracts.]

At the Business layer, the focus on dependencies forces us to rationalise processes and make them leaner. Business processes need to be traceable back to the organisation's vision (its idea of Utopia), its mission (its strategy to bring about that Utopia) and the broad functions it needs to have in place to execute those strategies (Product Management, Engineering, Marketing, Sales, etc.). Within each function, there will need to be a set of processes, each made up of process steps. Here is where potential reuse of business logic is first identified.

At the end of this phase, we know the basic process steps (operations) required, and how to string them together into processes that run the business. But we can't just have these operations floating around in an organisational "soup". We need to organise them better.

At the Application layer, we try to group operations. Note that the Business Layer has already defined the run-time grouping of operations into Processes. At the application layer, we need to group them more statically. Which operations belong together and which do not? That's the dependency question that needs to be asked at this layer.

The answer though, is to be found only in the Information layer below, because operations only "belong" together if they share a data model. As it turns out, there are two groups of data models, those on the "outside" and those on the "inside". The data models on the "inside" of any system are also known as "domain data models", and these are never visible to other systems. In contrast, a data model on the "outside" of a system, known as an "interface data model", is always exposed and shared with other systems. In SOA, data on the outside is at least an order of magnitude more important than data on the inside because it impacts the integration of systems with one another, whereas data on the inside is only seen by a single system.

Version churn is a potential problem at the Information Layer, because changing business requirements could result in changed interfaces. With a well-designed type hierarchy that only exposes generic super-types, the interface data model can remain stable even as newer implementations pop up to handle specialised sub-types. Most changes to interfaces are then compatible with older clients, and incompatible changes are minimised.

Once we have our data models, we can go back up one level to the Application layer and start to group our operations in two different ways, depending on whether they share an internal (domain) data model or an interface data model. Operations sharing a domain data model form Products. Operations sharing an interface data model form Services. (And that's where the "Service" in "Service-Oriented Architecture" comes from.) Products are "black boxes" meant to be used as standalone applications. Services are "glass boxes" with no other function than to loosely bundle together related operations.

Finally, we have to implement our Services. The description and deployment bundles that are used need not correspond one-to-one with the Services themselves. They should in general be smaller, so that the volatility (rate of change) of any single operation does not needlessly impact others merely because they share a description bundle (e.g., a WSDL file) or a deployment bundle (e.g., a JAR file). If we also pay attention to the right types of components to use to host logic, transform and adapt data, and coordinate logic, we will be implementing business intent in the most efficient and agile way possible.

This, in fact, is all there is to SOA. This is Dependency-Oriented Thinking in practice.

The white paper will explain all this in much greater detail and with examples, but this post is an early appetiser. [Update: you can fall to and help yourselves, since the main course is now ready. Grab the two documents below.]


Dependency-Oriented Thinking: Volume 1 - Analysis and Design
Dependency-Oriented Thinking: Volume 2 - Governance and Management