The Promise And The Potential Of WSO2

(I write this piece with a bit of trepidation as I don't know how it will be taken, but I mean it with the very best of intentions and wishes for my ex-colleagues.)

Looking over the list of my FaceBook friends, I am struck by how many of those I connected with during my brief assignment with WSO2 (Aug-Dec 2011) have moved on to pursue graduate programs of study at US universities. Just like Macquarie Bank in Australia was once called "The Millionaire Factory" for making so many employees rich through bonuses, this little company is rapidly becoming an Intellectuals' Factory. If they succeed in staying in business over the next decade, they will have seeded some very powerful and influential links in academia and the higher levels of the technology world.

Which brings me to my main point - does WSO2 have what it takes to survive the next decade? I believe they do, and not just survive but also move up to the next higher level. But to do that requires a kind of thinking I have not seen from the company's leadership so far.

For at least the last five years, I have been searching for a "next-generation SOA company". I thought I had found it with WSO2, but was quickly disappointed when I realised that this was a company with traditional SOA thinking and only a next-generation business model (free software and paid-for support). That's not quite what I was looking for, and to me, it explains why WSO2 hasn't yet hit the big time in spite of a decent product suite and some very smart, sincere and hard-working people. They're just not revolutionary enough for the market to take notice.

OK, so what do I have in mind?

The view of SOA as being ultimately about SOAP-based Web Services in its implementation (OK, and some REST as well) is so last-decade. That view of SOA, I am convinced, is actually toxic to organisations. The impacts of that kind of SOA to agility and cost are entirely negative. (I have numbers to prove it but I could be sued by more than one ex-employer for breach of confidentiality, so you'll just have to take my word for it.)

If you sell traditional SOA with a different business model, you're not even half revolutionary. You're an interesting sideshow to the main game, and when the traditional SOA model gets discredited, as it largely has already, you get washed away in a way the IBMs and Oracles don't because of their size. (Which is a pity because I simply hate the IBMs and Oracles for what they have done to the industry - milked customers while providing them no benefit.)

My view of SOA is simple - it's "dependency-oriented thinking", and it applies to every layer of the organisation - Business, Applications, Information and Technology. Unfortunately, every SOA guru, after paying ritual obeisance to the notion that "SOA is not technology", proceeds to insult our collective intelligence by discussing Web Services technology (or lately, REST). It seems we just need to look away for a second after someone says SOA is not technology, and we find they're talking technology when we next look!

Similarly, I have a very jaundiced view of the term "SOA governance" as it is popularly applied. I suspect most SOA experts wouldn't be able to define the term in a readily comprehensible manner if they were tied to a chair and threatened with a copy of Lotus Notes. I also suspect most of them use the term "governance" for effect when all they mean is plain old "management".

My definitions of governance versus management are also simple - "doing the right thing" versus "doing things right" - in other words, the "what" versus the "how".

And so, "SOA Governance" is nothing but the process of determining the right dependencies that ought to exist at all layers (Business, Applications, Information and Technology) and identifying the dependencies that do exist. "SOA Management" is about using these as a starting point and eliminating the dependencies that should not exist, formalising the ones that should exist into contracts, and ensuring that fresh dependencies do not creep in. That's all there is to SOA governance and practice, but the benefits are significant and will not escape attention.

I talk about all this at some length in my InfoQ interview. [Update 29/11/2012: I've now also written a white paper on it, which interested folk can download from box.com.]

There is so much potential for a truly SOA-enlightened consultant to enter a client organisation and point out all the areas of tight coupling that are driving up their costs and risks and driving down their agility. Heck, if all that the consultant did was concentrate on tight coupling at the data layer, so much wasted potential could be unleashed. That's the power of SOA in the right hands.

What I want to see in a next-generation SOA company is an organisation that is not fixated on technology but on education and consultancy, because it's SOA thinking that is so badly in need of a reboot. WSO2 is full of smart people, but they're focused on the wrong thing. It's not technology they need to be tinkering with. They need to apply their screwdrivers to the minds of IT and business folk at all levels. They need to hire fewer nerds and more business-savvy consultant types trained to think about dependencies. People with backgrounds in risk management, project management and contract law are especially good at thinking about every kind of dependency or "fine print" that could trip up an undertaking, and these are the kinds of people needed for SOA consultancy.

Admittedly, that's a bit hard to do when you do technology too well. Perhaps WSO2 ought to start by setting up a consultancy arm (which isn't focused on selling their own technology support services), getting that division to drive SOA thinking from the business layer down, and then watching it outgrow its older sibling as customers start to see its dramatic impact.

Can they rise to the challenge?

A Simple and Nice Open Source Tool To Draw Sequence Diagrams

I was looking for a very simple tool to help me put together some sequence diagrams in a hurry, and a Google search turned up sdedit.

This is an Open Source Java-based tool. You download the jar file and run it with the command

java  -jar  sdedit-4.0-RC1.jar

It looks like a standard visual tool, but you don't actually interact with it visually. You type in statements in the bottom right window. Now that may sound like a retrograde feature, but it's actually very elegant, and you need to try it to realise just how simple it is.

It took me less than 5 minutes to create this PNG image file (generating a PDF or image file is just a couple of button clicks, although the PDF version does suffer some distortion):

The format of the text is really simple, and here are the rules.

It's like a simple program. You first declare all the entities that will be interacting, leave a blank line, then specify all the interactions.

There's one icon for the Actor type, and a standard rectangle for every other. The tool imposes a few simple rules. Undeclared entities are flagged. Only Actors are allowed to initiate interactions, and their interactions are always asynchronous.

It's also possible to mark off sections that are loops or that represent alternate flows. This is one tool that is really worth trying out.

You can use this code to get started:

#![eCommerce Example]
customer:Actor "Customer"
portal:Process "Sales Portal"
productdb:Database "Product Database"
paymentgw:Process "Payment Gateway"

customer:portal.login
portal:productdb.get product list
portal:customer.list of products
[c:loop for selected products]
  customer:portal.add to shopping cart
[/c]
customer:portal.check out
portal:customer.redirect to payment gateway
customer:paymentgw.make payment
paymentgw:customer.paid - redirect to portal
customer:portal.back to portal
portal:customer.confirmation of order

My Interview at QCon, Where I Mostly Rant About SOA

I attended QCon New York in June this year and spoke about loosely-coupled Identity Management. I was also interviewed after my talk by Eric Newcomer, and the video of this interview has now been put up by InfoQ (the organisers of the conference).

The video is here. The transcript of the entire conversation is right below the video, so click on the "plus" sign to the right of each question to see the full text of the question and my answer.

I'm glad I got the opportunity to rant about SOA and how badly it has been misunderstood and misapplied by the industry.

I also managed to get a couple of sentences in about Identity Management, which I what I went there to speak on :-).

The Luxury of Instant Information Gratification



Have you ever thought about how easy it is to scratch an information itch these days?

I was looking at a presentation on a technical tool that is used in emergency warning systems. The system works by locating all mobile cell towers in an affected area, then sending SMS messages to subscribers within range of those cells.

One of the slides in the presentation illustrated the concept using this map:

Mobile cell tower coverage superimposed on a map - but which city is this?

I was distracted by an entirely irrelevant angle to the point being made. I noticed that all the street names on the map looked Spanish, and wondered which city it was from.

Well, Google and Google Maps were just a click away, so I did a quick search on "Calle de Alcala" (one of the street names I could see at the top of the map) and was immediately informed that this was in Madrid, Spain. [In fact, I wrongly spelt it "Alcaia" and Google helpfully corrected me.] I then switched to Google's Map view and zoomed in till I could see the "Sevila" metro station that was in the original, and positioned the map so I could compare it with the one in the presentation.

Yup, it's Madrid all right.

It took less than a minute for my idle curiosity (for it was nothing but that) to be satisfied.

And thinking about it, surely this is an amazing aspect of progress in the modern world! We didn't have this capability even 20 years ago. Trillions of trivial questions must have gone unanswered because there was no easy way to answer them. I don't know how I would have found the answer to my question in the pre-Google days.

It's nothing short of a luxury that we can afford to scratch our information itches almost without thinking. What wondrous innovations and advances will build on the shoulders of this capability, I wonder.

98% Compression For Diagrams - Unbelievable?

I didn't believe it was possible for compression technology to get any more efficient than it is today.

Take a PNG diagram like the one below:

How small do you think a file containing this diagram could be? The challenge is to get it under a kilobyte. Your choice of file formats and compression algorithms!

The uploaded version of the diagram on this blog may be smaller, but the one I have on disk is a PNG file that is 40,468 bytes in size. Compressing it using zip results in a file 38,394 bytes in size, - a 5% reduction.

That's not too surprising, because PNG is already a pretty efficient file format. All said, that's still way off our target size of a kilobyte. We're asking for an impossible 97.5% compression ratio.

Now, what if I told you there was a way to get there with no loss of information?

Unbelievable?

The above diagram is the standard example on the ditaa.org website. The website hosts the eponymous ditaa tool, which is Free Software under the GNU GPL. Ditaa converts ASCII art into image files. Aha!

The above diagram is generated from an ASCII art text file that weighs in at 1,569 bytes, which is still 50% above our target, but since ASCII text is very space-inefficient, it can be further zipped into a file that is just 713 bytes in size. Compared to the size of the original PNG image that it captures in a lossless representation, that's a compression ratio of 98%!

Here's what the ASCII art version looks like (this is a PNG screenshot, by the way ;-).

[How does one draw ASCII art in the first place? Isn't that a pain? Not really. There's another wonderful tool over at www.asciiflow.com that provides a simple UI to create ASCII art diagrams (doesn't work with IE). Use the tool and follow the conventions specified by ditaa, and you can create really compact diagrams that can be expanded whenever required.]

To me, it's not the compression ratio that's so cool. It's the whole approach to representing technical diagrams. The efficiency is in being able to capture the basics of representation in such a minimalistic way, yet sacrifice nothing in terms of the output's visual appeal.

Full marks to asciiflow and ditaa!

Apple's UDID Leak and the LIMA Solution

A million UDIDs (Unique Device IDs) that identified Apple devices were leaked this week, and it was hinted that this was just part of a set of 12 million UDIDs that had been stolen.

From an Identity Management perspective, the questions that arise are:

• Is this serious?
• Could this have been prevented?
• And (if anyone is perspicacious enough to ask), is there an architectural approach that will render such leaks harmless?

The answer to the first question so far appears to be no, because the UDIDs themselves are hopefully meaningless. They tend to be associated with meaningful data, and it is these associations that are privacy-sensitive. As long as privacy-sensitive data indexed by UDIDs is not also leaked, this leakage of IDs need not be a disaster.

Being cynical, I believe that the answer to the second question is always no. To quote Dr Ian Malcolm of Jurassic Park, "Life will find a way". I don't believe any secret is perpetually or provably safe. Think Wikileaks.

Which brings us to the third question, which hardly anyone even thinks of asking. Note that there is a big difference between asking the obvious but naive question (which was our second one), "Is there a way to prevent such leaks from taking place?" and asking the smarter one, "Is there a way to prevent (inevitable) leaks from having an impact?"

I believe there is such a way, based on my experience of designing loosely-coupled IAM systems. Loose coupling not only makes housekeeping tasks such as splitting and merging identities easier, it can help to recover relatively painlessly from events like the leakage of Apple's UDIDs. Now, I'm not an Apple fanboy and have studiously avoided being sucked into that closed ecosystem, so I'm not quite sure how exactly it operates. They may already be using a variant of the scheme I am about to describe, and if so, good for them.

LIMA (Lightweight/Low-cost/Loosely-coupled Identity Management Architecture) has the notion of multiple identifiers for an entity. There is a meaning-free internal identifier that is private to a system and not meant to be shared with third parties or systems. And then there could be multiple external identifiers, which may be either meaningful or meaning-free, that may be shared with third parties. The privacy of the internal identifier is not so much for confidentiality as it is for flexibility, as we will see. (Confidentiality is an orthogonal concern that is ensured through other means.)

Under the LIMA scheme, Apple's UDID would be classified as an external identifier because it is shared between third parties such as app developers and customers themselves. If Apple and its partners followed LIMA, they would all use separate internal identifiers in their systems that are distinct from the UDID. They would each map these internal identifiers to the shared UDID and associate their data elements with these internal identifiers, not with the UDID itself.

UDID <===> Internal Identifier <===> All other data elements

Once a set of UDIDs was (inevitably) leaked, it could be replaced with a minimum of fuss. Anyone concerned about the leakage of their UDID could log into Apple and request another one. They would first need to be authenticated, of course. Since we are not talking about a leakage of passwords here, the authentication would still be relatively foolproof. Once the identity of the user (device) was established (i.e., the internal identifier was determined from the UDID), another UDID could be easily generated and associated with the internal identifier. The new UDID could then be shared with all authorised partners that used the old one. The old UDID would be marked "retired" and de-linked immediately from the internal identifier, and thereby from all privacy-sensitive data. Thus the leakage of UDIDs could be rendered harmless.

What if the internal identifiers used by Apple or one of its partners were themselves leaked? Well, though this appears at first glance to be much more serious, we did say that the private nature of the internal identifier was not for confidentiality but for flexibility. It's in fact even simpler to replace internal identifiers because they're not shared with any external party. Simply generate new meaning-free identifiers to replace the old ones wherever they are used within the organisation. There should be systems to do this as a matter of routine, because any dependence upon the specific value of an identifier is a source of rigidity and brittleness.

Loose coupling through two levels of identifiers provides flexibility. If the UDID is the only identifier (internal as well as external) used in the Apple ecosystem, then they're ninnies.

NDAP - Because User Provisioning Is Too Important To Be Left To LDAP

As I discovered during my foray into SCIM-land, the proposed user provisioning API is extremely clumsy and non-intuitive (check out these examples for yourself), and this is mainly because of multi-valued attributes. By converting multi-valued attributes (arrays) into "key-per-value" attributes (dictionaries), I argued that a much simpler and more elegant API could be developed. While some members accepted that my API was more elegant, the SCIM spec committee overall rejected my proposal as unimplementable. And this was not bloody-mindedness on their part. It is genuinely difficult for existing cloud providers to adopt a sensible and elegant API for user provisioning because they are stuck with legacy data stores that have to support multi-valued attributes, and LDAP directories are the prime culprit. Relational databases are perhaps a tad more flexible than LDAP directories, but it comes down to whether Service Providers would be willing to make changes to their relational schemas. We can pretty much guess the answer to that!

However, it seems a shame to abandon efforts to simplify and standardise the user provisioning API, which is what the SCIM effort should have been all about. Enterprise clients don't care about the legacy constraints of Service Providers. They want a simple and standard API for user provisioning. There may also be emerging Service Providers who are not encumbered by legacy constraints, and who would be willing to explore a better option than SCIM. [There are 3 Gartner people on the SCIM spec committee, but predictably enough, none of them chose to speak up on behalf of their paying customers (enterprise clients) and the one that did speak up supported the interests of incumbent service providers. That sort of behaviour is what lies behind the saying "Gartner spelt backwards is rent-rag."]

I last blogged about the need for "NoLDAP", which I described as stores of JSON documents that permitted no arrays, only dictionaries. Actually, this doesn't have to be restricted to JSON. Any data store that outlaws multi-valued attributes would do as well. In other words, all documents have to be nested dictionaries if they are to allow simple manipulation. The abbreviation "ND" then led to the initially flippant name "NDAP Directory" as an alternative to LDAP directories. But in retrospect, "NDAP Directory" isn't a bad term at all. The "access protocol" part of NDAP is just the RESTful API which includes the "Best Practice" use of PATCH that I suggested earlier.

Here's a high-level picture of how a Service Provider could configure their infrastructure to provide support to conventional authentication mechanisms as always, yet simplify the user provisioning process that has to happen behind the scenes.

The NDAP Directory is the source of truth, and the LDAP Directory is a read-only replica that is used for authentication and retrieval of common user attributes.

Perhaps such a hybrid architecture will be more practical. Will any next-generation SPs rise to the challenge? 

After NoSQL, It's Time For NoLDAP

I've been in discussions with the IETF SCIM working group (formerly Simple Cloud Identity Management and now called System for Cross-Domain Identity Management) about the three suggestions for improvement that I wrote about in my InfoQ article.

One of the things I've learnt from this discussion is that, while I sense a bit of an NIH syndrome in some of the pushback I have received, there are also some genuine technical challenges involved in implementing my ideas. In my article, I stated that the fundamental problem with manipulating resources was multi-valued attributes that lacked a unique key per value. Over the past two weeks of discussion, I'm convinced that I got that absolutely right. Multi-valued attributes are the problem. And the biggest culprit in keeping them entrenched with little hope of change is that legacy beast known as an LDAP directory.

You can see for yourself how clumsy it is in LDAP to update multi-valued attributes. When I googled "multi-valued attributes" and "LDAP" together, I found this prescient piece written by Mark Wahl in 2005. I was amazed to see how accurately he had anticipated the problem:

Unfortunately, some of the emerging protocols which also intend to represent and transfer personal identity information have perhaps taken a step backwards by not even considering these issues [problems with multi-valued attributes], perhaps sweeping them under the rug in the guise of simplicity, XMLification, or "fix in the next version", which only postpone finding interoperable solutions to allowing applications to express the identity entries they want to express.

SCIM is one of these "emerging protocols", and sure enough, the arguments that I have been facing have explicitly cited "simplicity" as the reason not to accept my solution proposal, exactly as Mark Wahl had predicted. However, the real reason for resistance is not about "simple" but about "easy".

It is not easy for SCIM to move to a simple model, and that is the problem. It is more expedient to maintain complexity.

The irony is that the SCIM spec is anything but simple, thanks to multi-valued attributes. This is how the spec proposes to deal with multi-valued attributes:

Multi-valued attributes: An attribute value in the PATCH request body is added to the value collection if the value does not exist and merged if a matching value is present. Values are matched by comparing the value Sub-Attribute from the PATCH request body to the value Sub-Attribute of the Resource. Attributes that do not have a value Sub-Attribute; e.g., addresses, or do not have unique value Sub-Attributes cannot be matched and must instead be deleted then added. Specific values can be removed from a Resource by adding an "operation" Sub-Attribute with the value "delete" to the attribute in the PATCH request body. As with adding/updating attribute value collections, the value to delete is determined by comparing the value Sub-Attribute from the PATCH request body to the value Sub-Attribute of the Resource. Attributes that do not have a value Sub-Attribute or that have a non-unique value Sub- Attribute are matched by comparing all Sub-Attribute values from the PATCH request body to the Sub-Attribute values of the Resource. A delete operation is ignored if the attribute's name is in the meta.attributes list. If the requested value to delete does not match a unique value on the Resource the server MAY return a HTTP 400 error.

Sounds like quite the dog's breakfast, doesn't it?

To be fair, that's the implementation, which is usually hidden from sight. Let's see if the visible API looks any better.

Here's how to delete a single member from a Group, as per the current spec:

   PATCH /Groups/acbf3ae7-8463-4692-b4fd-9b4da3f908ce
   Host: example.com
   Accept: application/json
   Authorization: Bearer h480djs93hd8
   ETag: W/"a330bc54f0671c9"

   {
     "schemas": ["urn:scim:schemas:core:1.0"],
     "members": [
       {
         "display": "Babs Jensen",
         "value": "2819c223-7f76-453a-919d-413861904646"
         "operation": "delete"
       }
     ]
   }

Here's how to delete ALL members from a group according to the current spec:

   PATCH /Groups/acbf3ae7-8463-4692-b4fd-9b4da3f908ce
   Host: example.com
   Accept: application/json
   Authorization: Bearer h480djs93hd8
   ETag: W/"a330bc54f0671c9"

   {
     "schemas": ["urn:scim:schemas:core:1.0"],
     "meta": {
       "attributes": [
         "members"
       ]
     }
   }

For two functions that do very similar things, the syntax is wildly different. It's hardly what one would call simple. But these represent the easy way out from an implementation perspective, and so the SCIM Working Group is extremely reluctant to tamper with this implementation.

With my suggestion (based on "best practice" around the use of PATCH), here's how to delete a single member from a group:

   PATCH /Groups/acbf3ae7-8463-4692-b4fd-9b4da3f908ce
   Host: example.com
   Accept: application/json
   Authorization: Bearer h480djs93hd8
   ETag: W/"a330bc54f0671c9"

   {
     "operations" : [
       {
         "RETIRE" : {
           "key" : "members.2819c223-7f76-453a-919d-413861904646"
         }
       }
     ]
   }

Here's how I suggest deleting ALL members from a group:

   PATCH /Groups/acbf3ae7-8463-4692-b4fd-9b4da3f908ce
   Host: example.com
   Accept: application/json
   Authorization: Bearer h480djs93hd8
   ETag: W/"a330bc54f0671c9"

   {
     "operations" : [
       {
         "RETIRE" : {
           "key" : "members"
         }
       }
     ]
   }

That's a darn sight more elegant, if I may say so myself. In both cases, a reader can tell exactly what is being attempted. What's more, the syntax for both "delete" functions is identical.

This is arguably simple as an API. But I'm told it cannot be implemented, because it's not simple easy. I'm told that cloud service providers will not be willing to make the changes required to make this work, and this will hinder adoption of the spec. And what do incumbent cloud providers use to store identity information? Why, LDAP of course!

That's why I'm coming around to the view that no incremental change is possible. Human society progresses in its views not because people gradually change their minds, but because generations die, and new generations with different ideas take their place. Ideas only die when people die. People never change their minds. It's the people who have to be replaced. A cynical observation, but probably true. The Augean stables can only be cleaned with a flood. 

So we need a new spec, a new set of cloud service providers and a new type of directory, if we are to simplify cross-domain identity management.

It's the last of these that I want to talk about here.

I want a directory that has the advantage of LDAP (fast reads) without the disadvantages (a rigid tree structure and awful treatment of multi-valued attributes). I think one of the NoSQL document databases that can hold JSON documents may fit the bill, -- with one important constraint. JSON supports two types of data structures, the dictionary and the array. The dictionary forces every value to have a unique key. That's goodness. The array allows multiple values per key. As we now know, that's the root of all evil. Therefore, our document database must only allow dictionaries and not arrays. Clients are still free to upload data to the directory in array form, but the directory will only store these arrays after converting them into dictionaries. It will generate its own random keys for them. (If the order of elements is to be preserved, the generated keys can be in sequence, with sufficient gaps between them to permit insertions of other values later.)

Every element of such a directory can be addressed in a fully-qualified way through "dot notation". Take this example of part of a Telecom company's customer database. The logical customer record shown has two addresses, and different telecom "carriage" services are available at the two addresses. The first address has carriage services "cable" and "ADSL", while the second has "ADSL2+" and "Wi-fi". As we can see, that's two nested arrays. The outer array holds addresses, and within each address, there's an inner array holding available services.

{
  ...
  addresses: [
    {
      "type" : "home",
      "street_number" : "35",
      "street_name" : "High Road",
      ...
      "country" : "Australia",
      "available_services" : ["cable", "ADSL"]
    },
    {
      "type" : "office",
      "street_number" : "213",
      "street_name" : "Main Street",
      ...
      "country" : "Australia",
      "available_services: ["ADSL2+", "Wi-fi"]
    }
  ]
}

Let's say we need to update the customer record, because the first address now has "Wi-fi" too, and it no longer has "cable". How do we do this?

We want to delete
"customerX.addresses[0].available_services[0]"

and insert "Wi-fi" into the array
"customerX.addresses[0].available_services".

However, we can't reliably use positional indexes because these can change with every operation. We need stable identifiers. (I will not even attempt to formulate the SCIM syntax required for these operations!)

Here's how I suggest it should be done:

PATCH /Users/2819c223-7f76-453a-919d-413861904646

{
  "operations" : [
    {
      "INCLUDE" : {
        "key" : "addresses.d6ea365462f5.available_services",
        "value" : "Wi-fi"
      },
      "RETIRE" : {
        "key" : "addresses.d6ea365462f5.available_services.9be6378dc303"
      }
    }
  ]
}

Again, if may say so myself, this is elegance. And that's only possible because my directory has converted these two arrays into dictionaries, like this:

{
  ...
  addresses: {
    "d6ea365462f5" :
    {
      "type" : "home",
      "street-number" : "35",
      "street-name" : "High Road",
      ...
      "country" : "Australia",
      "available_services" : {
        "9be6378dc303" : "cable", 
        "6aa1429eba34" : "ADSL"
      }
    },
    "3cbaaff8e84e" :
    {
      "type" : "office",
      "street-number" : "213",
      "street-name" : "Main Street",
      ...
      "country" : "Australia",
      "available_services: {
        "2beca1fdf3e5" : "ADSL2+", 
        "8c3dcc204a33" : "Wi-fi"
      }
    }
  }
}

(But how does the client know these generated keys? They're returned as part of the record creation response, and also provided as part of the resource representation returned with every GET request.)

This model is easy to support if a service provider implements its identity store as a database of JSON documents rather than in LDAP. One can have both "simple" and "easy", without compromises. But for that, LDAP has to go. "NoLDAP" is what we need in its place.

My definition of a "NoLDAP Directory" is simply this:
A document database that holds dictionary-only JSON documents.

I think the next generation of cloud service providers will emerge based on this architecture, offering a simple API that is also easy for them to implement.

As for LDAP and SCIM, I guess the best TLA is RIP.

Don't SCIM Over Your Data Model

My article critiquing the SCIM (System for Cross-Domain Identity Management, formerly Simple Cloud Identity Management) is now published on InfoQ.

I had sent the suggestions to the SCIM mailing list a week earlier, but the discussion there only started after the InfoQ article came out...

Best Practice For The Use of PATCH in RESTful API Design

This topic deserves standalone treatment. I wrote before in the context of SCIM that PATCH is a relatively recent HTTP verb that is being recommended for use in situations where a resource is to be partially modified. In comparison, PUT is a bit of a sledgehammer, because it requires the entire resource to be replaced in toto using a fresh representation.

So the advent of PATCH is a welcome development no doubt, but one immediate problem is posed by current web infrastructure. Not all infrastructure components understand PATCH. Firewalls are suspicious, proxies could mangle or drop messages containing the unfamiliar verb, not all web servers are geared to handle it, and so on. Heck, even the ultramodern HTML 5 spec has no support for the PUT and DELETE verbs, so asking for PATCH support is a bit ambitious at this point.

However, I'm not too worried about those problems because Time is a great infrastructure deployer. (Besides, there's nothing I can do to help on that front.)

The bigger problem I see is that we as API designers don't really know how to use PATCH. There is no guidance around its use other than to "use it for partial updates". Everyone agrees that we need more fine-grained control over updating resources, but this level of granularity needs detailed specification. We simply don't have that anywhere.

Well, we've got it now, and remember that you read it here first :-).

First off, let's take a JSON model of a resource because it's the most popular and everyone understands it. (Yes, I know, I'm one of the SOFEA authors, but I don't like XML either.)

You should know that the JSON family actually comprises two siblings, one good and one evil. Abel and Cain. Thor and Loki. Edwin Booth and John Wilkes Booth. Take your pick.

I'm going to call them JSON the Argonaut and JSON Voorhees (of Friday the Thirteenth notoriety).

JSON the Argonaut is the good sibling. He buys a ticket for every member of his family when he goes to see a show.

JSON Voorhees is the bad one. He buys a single ticket for himself, and gets all his kids to slip under the turnstile.

And you wonder why it's so hard to get good seats at the show.

You have no idea what I'm talking about, do you?

Check out the JSON website:

JSON is built on two structures:

• A collection of name/value pairs. In various languages, this is realized as an object, record, struct, dictionary, hash table, keyed list, or associative array.

• An ordered list of values. In most languages, this is realized as an array, vector, list, or sequence.

See the connection? The first structure describes JSON the Argonaut, who ensures that every member of his family has a ticket (a label or key). The second structure describes JSON Voorhees, who gets away with a single ticket (a label or key) for his entire family. The first thing we need to do therefore, before we even start looking at PATCH, is to fix JSON Voorhees. By which I mean, get him to buy tickets for all his kids.

This is what JSON Voorhees's family looks like when he slips them past with a single ticket:

This is what we want his family to look like when he buys all the tickets he needs to:

You see? One label per value.

In other words, when working with JSON objects, convert all arrays to dictionaries.

[If the sequence of items in the array is important and you don't want to lose that when turning it into a dictionary, generate keys that are in sequence, only taking care to see that there are enough "gaps" between keys so that new items can be inserted between them later, if required. Something like a B-Tree algorithm will be useful, I think.]

When we POST a resource containing arrays to a server, the server should convert them into dictionaries and return the new representation of the resource to us. All the generated keys should be in the new representation. And this is Goodness with a capital G, because now, all of a sudden, PATCH has a much easier job to do.

[Why can't we just use positional indexes, e.g., email-addrs[i]? Think about what will happen when one client does a GET and decides it needs to delete email-addrs[1]. In the meantime, another client deletes email-addrs[1], resulting in email-addrs[2] becoming the new email-addrs[1]. Then the first client will end up deleting the wrong attribute. Yes, you can check the Etags and the if-not-modified headers before an update to stop this from happening, but you will then end up with a large number of failed updates under high concurrency, which can be avoided with unique and stable identifiers.]

Now PATCH has a unique handle on every attribute value within a resource, no matter how complex or how deeply nested it is. Every value has a unique key that can be expressed as

first.second.third.fourth.fifth

if it's (say) five levels down from the top level resource URI.

Now we can use the same REST concepts that we had before PATCH, to operate on each of these. The coarse-grained verbs used to operate on resources can now be used to operate on parts of a resource as well.

POST to add a new attribute to a collection
PUT to replace the value of an attribute
DELETE to make an attribute inaccessible (and not necessarily delete it physically, of course)

In place of a URI, we specify the key of the attribute within the resource, using the dot-separated notation we just showed.

There's just one small detail. Actually two.

1. PATCH is turning out to be an operation container rather than an operation in itself. This means we're going to have to nest other verbs within it to operate on different parts of a resource. There's nothing to stop us from bundling operations to add, modify and delete various parts of a resource whose top-level URI the PATCH request targets. In fact, that's probably the kind of elegance we're after. But then we wouldn't want any confusion between these nested operations and the coarse-grained HTTP verbs used to operate on resources, so we should perhaps look for equivalent verbs to POST, PUT and DELETE to use inside of PATCH.

2. When we think about it, the semantics of the HTTP PUT verb are a bit unclear (at least to me). Do we intend to replace a resource that is already there, or do we intend to create a new resource with a URI of our own choosing (as opposed to a server-decided URI)? And what constitutes an error in either case (because they're the exact opposite conditions)? We probably need to split PUT into its subtypes to reflect these separate "create" and "update" semantics, and for good measure define a third subtype to cover the "create-or-update" case, which is also very likely to be required. While we're at it, we may want to narrow the scope of POST to only mean "add to a collection", because POST as it's used today is a bit of a catch-all command in REST when none of the others quite applies.

So with these two drivers in mind, let's look at what new verbs we could use as nested operations within a PATCH request.

1. INCLUDE (equivalent to POST): Add the given attribute value to a collection and return the unique key generated for it.
2. PLACE (equivalent to one form of PUT): Add the given attribute key-value pair at the location implied by the key. (If there’s already an attribute with this key, return an error status.)
3. REPLACE (equivalent to another form of PUT): Replace the current value of the given key with this new value. (If there’s no such key, return an error status.)
4. FORCE (equivalent to a third form of PUT): This means PLACE or REPLACE. (At the end of this operation, we want the specified key to refer to the given value whether the key already existed or not.)
5. RETIRE (equivalent to DELETE): Delete, deactivate or otherwise render inaccessible the attribute with the given key.
6. AMEND (equivalent to PATCH): (This verb is just listed for completeness. We probably don’t need a nested PATCH since PATCH cascades to every level of the tree.)

And here's a picture that illustrates far more effectively than a thousand words how PATCH will then work. Click to see the enlarged version. (There's a minor syntax error in the PATCH command. Each element in the "operations" array needs to be surrounded with curly braces.)

The final piece of the puzzle is the PATCH response.

If PATCH is to be used as an operation container with multiple nested operations, then the most appropriate status code is the WebDAV-defined "207 Multi-Status". Each nested operation must have its own status code, because some may succeed while others fail. Even with successful operations, there is a difference between "200 OK" and "201 Created", because with "201 Created", we expect to find the server-generated key accompanying the response.

Here's what the response to the above PATCH command could look like.

(Like with the PATCH request, there's a minor syntax error in the PATCH response. Each element in the "results" array needs to be surrounded with curly braces.)

To sum up,

1. Ensure that when resources are created, all their array-type elements are replaced by dictionaries. Every value should be independently addressable using its own unique key.
2. Use one of 5 different operations (INCLUDE, PLACE, REPLACE, FORCE and RETIRE) within PATCH to refer to various keys and specify new values. There is no possible confusion between the scope of these verbs and those of the HTTP verbs POST, PUT and DELETE, and their meanings are also unambiguous.
3. Use the "207 Multi-Status" code in the response to the PATCH itself, and nest the status codes for each individual operation inside the response.

And that, in a nutshell, is what I hope will be considered Best Practice around the use of PATCH!

OAuth2 - Whom to Believe?

In the field of Identity Management, a new dawn has begun, if one is to believe authorities such as Eve Maler of Forrester. The old, complex era of XML-based standards (SAML, SPML, XACML) is being replaced as we speak by a more lightweight, web-scale set of standards which are loosely called "the OAuth2 family".

OAuth2 is the one that underpins the others. Eve Maler refers to it as "a means of enabling constrained delegated authorisation of access." In other words, if you need to allow a photo printing service to print some of the photos that you have hosted on Picasa, then you don't have to give the service your Picasa username and password. The OAuth2 protocol allows you to give the service an "access token" for a limited period of time that allows them to access your photos, without having to reveal your security credentials. It takes an interesting 3-way handshake to accomplish this, but once this pattern is established, it can be used to protect any resource, most importantly APIs or service invocations.

The other members of the "OAuth2 family" use this feature so that they no longer need to concern themselves with the authorisation part of security, since OAuth2 will ensure that only a client with the right authorisation can invoke a service. These other APIs are "OpenID Connect" for Single Sign-On and SCIM for user provisioning. [Eve Maler talks up UMA (User Managed Access) as the third piece of the puzzle instead of SCIM, but I view UMA as just OAuth2 at industrial scale. Authentication, Authorisation and Provisioning are the three major aspects of IAM (audit is a fourth), and these are rightfully represented by OpenID Connect, OAuth2/UMA and SCIM, respectively.]

However, just when it looks like things are finally coming together in a simple and sane way, the party has been rudely interrupted.

Just 3 days ago, Eran Hammer, one of the leading figures in the OAuth2 standardisation effort, resigned from the working group and withdrew his name from the effort. The reasons he provides, at least initially, are dismaying and lead one to believe that OAuth2 is doomed as a standard and that we are going to see some catastrophic security breaches very soon.

His reasons for this seem plausible.

• When the standardisation effort behind OAuth moved from the WHAT-WG to the venerable IETF (Internet Engineering Task Force), the lightweight-and-agile web developer crowd was replaced almost overnight by the  enterprise-y crowd. They squashed the agility out of the standard and replaced it with the dead weight of Enterprise thinking. The OAuth2 standard is the camel that the committee came up with when it set out to design a horse.

• The specification fails to deliver on its two main goals of security and interoperability. As evidence, he says it has been renamed from a protocol to a framework, and the spec explicitly warns that implementations are unlikely to be interoperable.

• The main point that the "enterprise crowd" fought for and won was flexibility and extensibility. In Eran Hammer's view, that's a bad thing, because "it is this extensibility and required flexibility that destroyed the protocol. With very little effort, pretty much anything can be called OAuth 2.0 compliant."

Fortunately, I read through the comments following his blog post, especially this very different viewpoint expressed by Leif Johansson.

I now have a more nuanced picture of where OAuth2 stands, and it isn't a picture of doom and despair. I've been in this industry for over 25 years, and I've seen this sort of thing before. Eran Hammer is young and idealistic, and design-by-committee always frustrates the young and idealistic.

The IT industry knows three types of innovators:

1. The young, smart and agile, who come up with a new way of doing something, which then suffers growing pains because its inventors didn't think of all the possible situations in which it would be used. The NoSQL movement exhibits several examples of this.
2. The older, smarter and experienced, who have learnt from years of mistakes (their own and those of other people), and who now know how things are to be done and will brook no questions. They're insufferable and nobody can work with them. (The much-respected and avoided Marshall T Rose comes to mind.)
3. The committees of people with vested interests, who design the ugly camel that no one likes.

I think the most successful products come from the first group. The most successful standards come from the third. (Probably the second group can write good books and tell war stories.)

What Eran Hammer describes about the IETF and what they have done to OAuth is unsurprising and hardly dismaying. He's describing the sausage machine at work, and nobody really wants to see this in operation.

It's important that he's qualifying his rant with these two statements:

• "OAuth 2.0 [in the hands] of a developer with deep understanding of web security will likely result [in] a secure implementation"
• "It is true that [OAuth] 2.0 can be profiled into a simple, straightforward protocol. But every single one of these profiles is slightly different."

What this tells me is that OAuth2 is entering the second phase of its standardisation, when interoperability profiles will need to be defined. The standard isn't inherently broken. We can and do have secure implementations today. We just need to make them interoperable.

As an industry, we know how to do this. We've done it before with Web Services. That was called the WS-I Basic Profile, where "I" stood for Interoperability. Indeed, when Eran began his rant, he called the OAuth2 protocol "a bad protocol. WS-* bad". In other words, not that bad.

In fact, compared to the WS-I Basic Profile, OAuth2 is more of an orthogonal concern. The only aspect of it that an API developer will see is the "bearer token" header. So once the OAuth2 infrastructure is set up, it'll be an unobtrusive part of the IAM fabric. I know people who shudder at the term "profiles", but I don't believe profiles in OAuth2 are going to be visible in the sense of WS-I Basic Profile. So no course corrections required, folks. It's still full steam ahead with OAuth2. We'll give the angry young man some time and space to settle down.

[I have a rant of my own, a different one altogether, about the whole Identity-Management-as-Technology mindset, and will shortly be writing about the neglected stepchild of IAM - The Data Model. Watch this space.]

A RESTful Protocol to Add/Modify/Delete Elements of a Collection

I've been reading the SCIM specification recently.

[What does SCIM stand for? There's a moral-of-the-story here, by the way. SCIM originally stood for Simple Cloud Identity Management. Now it stands for System for Cross-domain Identity Management. The moral of the story is, if you're swayed by the buzzword of the day, you will be forced into embarrassing retreat at some point. Cross-domain Identity Management was what it was always about. What a shame someone felt the need to jump onto the cloud bandwagon.]

Anyway, coming back to the topic of this post, I realised that one of the aspects dealt with by the SCIM specification was a much more general problem that would benefit from a standard approach.

I was struck by the similarity of 3 types of operations in the SCIM protocol, though not impressed by any of them.

One set of operations deals with adding, modifying and deleting multi-valued attributes of a user, e.g., the set of addresses or email IDs.

A second set deals with adding, modifying and deleting users as members of a group.

SCIM uses a RESTful API, so for these sorts of partial updates to a larger entity, it uses PATCH. A single PATCH command can be used to simultaneously add, modify and delete attributes of an entity, or to add, modify and delete members of a group. [PATCH when used to update one or more single-valued attributes is fairly straightforward, so I'm not talking about that here.] I suspect that PATCH is so new that best practice around its use has yet to emerge, which accounts for some of the ugliness I'm about to describe.

A third set of operations is called 'bulk operations', and deals with a collection of operations on multiple entities, which could include add, modify and delete operations. This is physically formatted as a POST with nested POST, PATCH and DELETE operations as attributes within the document.

To my mind, these three situations are conceptually the same (a set of fine-grained operations grouped into a single composite operation), therefore the same mechanism should ideally be used for all of them.

As it turns out, SCIM uses many different mechanisms, all ugly.

1. Multi-valued attributes of a user:

(In the spec document linked above, scroll up and down a bit to search for these examples)

Some of the ugliness comes from passing 'operation:"delete"' as an attribute of the document. After arguing for years that a GET request with URL parameter "?operation=delete" is not RESTful, we can't now barefacedly offer a "RESTful" API that does the same thing. True, this is not a GET, so it's probably not as awful, but this is a really bad "smell" that suggests poor design at some level.

The syntax to replace a user's primary email address with another one and simultaneously demote the previous primary email address to the status of just another email address is more than just ugly, it's horrific. Do have a look for yourself, if you can keep from gagging.

2. Members of a group:

(In the spec document linked above, scroll up and down a bit to search for this example)

Deleting all members from a group is even uglier. There is nothing in the message to suggest that a deletion is taking place! There's a "meta" attribute that has a value equal to the set of attribute names whose (multiple) values are being deleted. If that confuses you, you now know what I mean.

3. Bulk operations:

The main operation is a POST to an arbitrary URI which could be called "/bulk" or something similar.

There are nested operations within it that reflect operations on various entities.

As you can see, these are 3 entirely different mechanisms for essentially the same desired behaviour. Can't we use the REST philosophy of polymorphic verbs to make this interface more uniform?

I would like to propose a simpler scheme that uses explicit identifiers for every sub-entity that may need to be independently manipulated (updated or deleted). This is similar to having resource identifiers, but at finer levels of granularity.

All three sets of operations should have a common structure like this:

PATCH /main-entities/29d70c13-8651-41ce-97f8-c82ccdfcd6d6

(or POST /bulk-operations)

Host: example.com

Accept: application/json

Authorization: Bearer h480djs93hd8

Content-Length: ...

{

  operations:

  [

    {

      operation: "POST",

      URI: " /sub-entities",

      sub-entity:

      {

      ...sub-entity attributes

      }

    },

    {

      operation: "PUT",

      URI: "/sub-entities/dced46ca-b96d-44dd-b494-41e5de9405b6",

      sub-entity:

      {

      ...sub-entity attributes

      }

    },

    {

      operation: "DELETE",

      URI: "/sub-entities/12feb1db-8031-4a23-8810-bccfffb13f56"

    }

  ]

}

i.e., the nested operations should be RESTful operations in their own right, with standard HTTP verbs and standard URIs using the enveloping entity's URI as their base URI. How do we know what the URIs of the sub-entities are? The server should generate one for every array element that is created (multi-valued attributes of an entity), and return those URIs with the "201 Created" response, as well as in response to any GET on the main entity. That will then provide a handle for the client to specify fine-grained operations on the sub-entities.

I also don't like the SCIM convention of returning a "200 OK" even for a bulk request where different sub-operations may have had different outcomes. This is as bad as HTTP tunnelling in SOAP where the HTTP response could be a cheery "200 OK", but the message inside contains a SOAP fault.

There is a WebDAV status code for exactly this kind of situation, - "207 Multi-Status". That's what should be used.

HTTP 1.1/207 Multi-Status

Content-type: application/json

{

  operation-results:

  [

    {

      operation: "POST",

      URI: "/sub-entities",

      status: "200 OK",

      Location: "/sub-entities/c44fcc7b-8881-4f0f-b5fb-b2344615cc34",

      request:

      {

      ...sub-entity attributes repeated for verification

      }

    },

    {

      operation: "PUT",

      URI: "/sub-entities/dced46ca-b96d-44dd-b494-41e5de9405b6",

      status: "409 Conflict"

    },

    {

      operation: "DELETE",

      URI: "/sub-entities/12feb1db-8031-4a23-8810-bccfffb13f56",

      status: "404 Not found"

    }

  ]

}

This is a clean syntax, easily understood, and uniformly applicable when dealing with attributes or sub-entities of an entity, or with bulk operations. All it requires is for resources and resource collections to be identifiable at finer levels of granularity as well, i.e., attributes that are array elements.

I don't believe that's too much of an ask. UUIDs are cheap and plentiful, and the alternative is an ugly mess.

Architectural Lessons in Identity from the Higgs Boson

As I read the details of last week's virtual confirmation of the existence of the Higgs boson, I smacked myself on the head with a "D'oh!"

As an architect, I've been preaching against tight coupling for years, and I utterly failed to realise that there could be examples of tight coupling in the way we think about the real world.

What the discovery of the Higgs boson does is confirm what is called the Standard Model of Particle Physics, and one of the fundamental, though unintuitive, aspects of that model is that mass is not an inherent property of a particle. A particle may acquire mass through interaction with the Higgs field, but it's not necessary. That's why it's possible for particles like photons to have no mass.

It's ironical that in the months just prior to this discovery in the world of Physics, I made a modest discovery of my own that nevertheless shook the foundations of my conceptual world. In the realm of identity management, I realised that any given entity has no inherent properties (attributes) at all! All attributes that an entity may be deemed to have are only by association (just as a particle acquires mass independently of its existence), and therefore the only necessary aspect of an entity is a unique and meaning-free identifier to set it apart from other similar entities. Attributes can then be assigned and de-assigned to it at will through the mechanism of the identifier, and gradually a more sophisticated model can be built up. In creating such sophisticated models, it's important to remember that nothing is inherent to an entity, - not first name/last name, not primary email address, not social security number, - nothing! Just a unique identifier that is internal to the domain and kept invisible to the world outside the domain.

I've found this minimalist model of identity to be amazingly powerful and flexible. In addition to having arbitrary groups of attributes associated with an entity's identifier to form its properties, multiple external identifiers can also be independently associated with the (internal) identifier. The usual surrogate for identity, the username, is then just one such external identifier. Much of the conceptual confusion in identity management comes from mistaking external identifiers like username for the entity itself, leading to extremely clumsy and costly implementations. By creating an explicit internal identifier and treating any external identifier as an attribute associated with the internal identifier, such confusion and costly error can be avoided.

Cross-domain entity references can also be managed very cleanly by mapping both domains' internal identifiers to a shared external identifier, so that neither domain is coupled to the other domain's internal identifier for an entity. That's goodness!

Federated identity, such as when using one's FaceBook ID to enter a website, is nothing but treating the FaceBook ID as an external identifier and associating it with the website's internal identifier to establish the entity's identity. So multiple domains can independently maintain models about the same entity, and for most of the time, they can behave as though they are dealing with different entities. When it becomes necessary for the two domains to recognise these entities as the same, they then create a shared identifier that is external to both of them, and they each map this external identifier to their respective internal identifier.

This is a model of federation that does not make the assumption that there is a centre of the universe. No domain is the centre of the universe. Every domain is independent, and when they need to share their model of an entity, they loosely associate their entity references through a shared external identifier. There is now traceability across domains, but no implied control. I've found in practice that this is the only model that works, - technically, logistically and politically.

It's amazing that I should receive indirect confirmation of my identity model from as unlikely a source as Particle Physics, but I guess that's just a reflection of the fundamental nature of all things.

We have nothing inherent to ourselves but an identity. That's a pretty humbling thought that verges on the religious.

Report from QCon New York 2012

I'm back in Sydney after a week in New York attending QCon, the conference for developers organised by InfoQ. I was invited to speak on the topic of "loosely-coupled identity management", since the eBook co-authored by me was published by InfoQ recently.

It's always a nice experience attending tech conferences. There's an exciting buzz in the air with so many "new, improved" ways of doing things on display. Some are marketing hype by vendors, others genuine technology advancements. QCon is not a vendor-focused conference though. Most of the technologies being discussed were Open Source.

One of the keynote speakers was Cameron Purdy, who is currently a senior executive at Oracle. I remember when Cameron was CEO of his own company Tangosol, and their product Coherence was one of the leading persistence technologies for Java. I used to enjoy the sparring matches between Cameron and a bunch of other people (usually JBoss employees) on the now all-but-defunct site TheServerSide.com (I mean the site is still there, but the party has since moved on to other places, notably StackOverflow and InfoQ). "Peace" was always his sign-off line. Looks like he's moved up quite a bit in life now, with Oracle's purchase of Tangosol.

Cameron's keynote had one major takeaway for me. I agree with his assessment that we as an industry are once again in the midst of a once-in-fifteen-years technology paradigm shift, and this one involves three elements:

1. The Cloud

2. HTML 5

3. Mobile devices

I'm going to take his advice very seriously and focus my attention on these three areas. I strongly feel the combination is going to be formidable.

My old friend and colleague Jim Webber was there. Jim is now Chief Scientist at Neo Technology, the maker of the Open Source Neo4j graph database. Graph databases are quite possibly a much more powerful way of modelling relations between entities than relational databases are, and the performance seems in no way compromised. On the contrary, if Jim is to be believed, the performance of a graph database can be orders of magnitude better than that of a relational database when the paradigm fits. (To be fair to Jim, he doesn't sell graph databases as the solution to every problem, but there does seem to be a large and growing set of applications that can benefit from them.) I confess I haven't got a firm handle on exactly how graph databases work and exactly what use cases can benefit from them, but they seem to work very well for data points that have a rich set of relationships with one another. According to Neo Technology's site, graph databases can find answers to questions like “what music do my friends like that I don’t yet own,” or “if this power supply goes down, what web services are affected?”

I missed Jim's tutorial on Neo4j because some competing (and more basic!) tutorials caught my attention, but I did attend his other tutorial on REST, about which I blogged earlier.

I attended a number of other talks and tutorials. The most memorable for me were the ones by Kaazing on HTML5 WebSockets. Richard Clark's half-day tutorial was quite good, but since the contents were compressed into 3 hours from a full-day workshop, we couldn't do justice to all the examples. But WebSocket technology is definitely something I need to play with and understand.

Subbu Allamaraju's talk was very interesting too. I was intrigued when he started off by saying the client side of REST had been neglected so far, but his thinking proved to be very different from mine. I've been thinking about a client-side resource model applied to a peer-to-peer messaging protocol (I know it sounds silly to talk about clients in a P2P world, but what I mean by "client" is "initiating peer"), but Subbu's idea is about a structured way to manage application state held on the client across "service invocations", a sort of client-side orchestration capability. The capability seems quite well advanced judging from his demo. He calls it ql.io, and it's well worth a dekko.

I met some of the REST greats at the conference, notably Mike Amundsen and Leonard Richardson. I didn't realise until later that Leonard was one of the authors of my favourite REST book, "RESTful Web Services". Leonard has also recently published a Science Fiction novel called "Constellation Games", which makes him a multi-dimensional nerd.

My own talk went off OK. (The slides are here.)

 "Faster, Cheaper Identity Management Through Loose Coupling - The LIMA Approach"

I had only 16 attendees, which was a bit disappointing, but the organisers told me it's always hard to get interest in the security track because developers are so much more interested in the mainstream technology tracks. However, I did receive good feedback about my talk. They have a system of gauging audience response by letting attendees place different coloured cards in a bowl after the talk (green for 'good', yellow for 'OK', red for 'not good'). I got 12 green and 2 yellow cards (no reds), which I take as a pretty good endorsement :-).

It's a heck of a journey from Sydney to New York (13 hours to LA, a 3 hour layover, and then a further 4.5 hours to NY) and back again, but I enjoyed myself. Qantas economy is pretty comfortable and I didn't feel the strain much.

IANA and the Last Layer of REST's "Uniform Interface"

[Before you read any further, go and read Martin Fowler's simple description of Leonard Richardson's REST Maturity Model. What I'm writing about here will make a lot more sense after that. Trust me.]

One of the things I've liked about REST as compared to SOAP (other than the fact that SOAP-based Web Services introduce fresh dependencies in the form of schema and WSDL files, negating the dependency-reducing spirit of SOA) is that REST standardises vocabulary, allowing designers to construct solutions out of standard terminological "components" rather than having to invent their own.

For example, instead of the arbitrarily-named and "open set" of SOAP verbs CreateQuote, SubmitQuote, SearchOrders, AmendOrder, CancelOrder, AddItem, RemoveItem, etc., one merely creates resources like Quote, Order and Item (and others for convenience) and applies a "closed set" of standard verbs to them (GET, PUT, POST and DELETE).

This much has been clear, and I like this. But I confess I have never fully understood how exactly REST manages to shift the coordination model away from centralised orchestration (à la WS-BPEL) to one of decentralised choreography (which is what the HatEoAS principle is all about).

Yes, the server provides a context-dependent, constrained set of links with each response, and the client is forced to follow one of these links, and is therefore shepherded in certain well-defined directions so that a business process completes in a well-defined way. [In the past, I've even referred to HatEoAS as "Hypermedia as the Envelope (not the Engine) of Application State" in recognition of its function as a set of constraints rather than an active driver.] This is choreography all right, where it takes two to tango, rather than orchestration, where services march to a single drum (pardon the discordant musical references). What I didn't understand was how these links could be made intelligible to an automated client that wasn't primed to understand them a priori.

A link looks like this:

<link rel="logical-action" uri="/some-resource"/>

I didn't understand how the descriptive "logical-action" in the link relation above was made meaningful to the client. After all, URIs are meant to be opaque in the sense that clients are not expected to understand their internal patterns and "hack" them to make future requests. Clients need to be guided purely by link relations (the "rel" attribute).

This finally became clear to me yesterday when I attended Jim Webber's excellent full-day tutorial on REST at QCon New York. [Attendees at Jim's talks can be assured of three things: (1) an unending spate of wry Brit jokes, here mainly directed at the Yanks ("tax dodgers who drive on the wrong side of the road"), (2) lots of Doctor Who references, and (3) some absolutely top-class technical insight and instruction.]

Much as I enjoyed the Brit humour (even as some of it came at the expense of Australians ("convicts who speak degenerate English")) and the Doctor Who references, I think the major value for me from this talk was the clarification that link relations are standardised too! I learnt that IANA has published a standard list of link relations that servers can use to describe the "next steps" that a client can perform, and since these are by definition "well-understood" descriptions, clients do not need prior negotiation with the server to learn what they mean!

The last piece of the jigsaw puzzle has now fallen into place in my head.

Some of the most important IANA-defined links, in my opinion, are:

• "first", "last", "next" and "prev" (alias "previous"): useful for chunked navigation of lists
• "self", "up", "start", "index", "contents", "glossary", "appendix": useful for orientation, repositioning and meta-information about the content itself
• "item" and "collection": useful to navigate master-detail relationships in both directions
• "alternate", "duplicate", "canonical", "current": helpful to refine the ability to link to a resource more accurately
• "edit" and "edit-media": suggest how to change content
• "help": needs no explanation
• "search", "tag": useful for research and taxonomy
• "working-copy", "working-copy-of", "version-history", "successor-version", "predecessor-version": helpful when editing documents collaboratively
• "payment": extremely useful in e-commerce applications by providing a standardised label to nudge customers towards the checkout

[If an application requires link relations that are more specialised, then they need to be namespaced, e.g., "mycorp:specialised-function", with the term before the colon being the namespace. The client will have to understand terms in the specialised namespace of course, but that's only reasonable.]

I prefer the more verbose form of the link tag, which looks like this:

<link rel="mycorp:apply" uri="/applications" method="POST" 

         type="application/vnd.mycorp-application-v1_1+xml"/>

Now the client also knows what HTTP method to use, and what schema the submitted document should conform to.

I can now see that REST has done a lot to standardise the development of applications in a SOA-compliant way. My last major query about how REST communicates with clients to bring about choreographed processes has now been answered.

This is great stuff. Thanks once again to Jim Webber for a very instructive tutorial! I guess we have a more powerful weapon than Torchwood now, Jim :-).