50 Data Principles For Loosely-Coupled Identity Management

It's been a while since our eBook on Loosely-Coupled IAM (Identity and Access Management) came out. In it, my co-author Umesh Rajbhandari and I had described a radically simpler and more elegant architecture for a corporate identity management system, an architecture we called LIMA (Lightweight/Low-cost/Loosely-coupled Identity Management Architecture).

Looking at developments since then, it looks like that book isn't going to be my last word on the subject.

IAM has quickly moved from within the confines of a corporate firewall to encompass players over the web. New technology standards have emerged that are in general more lightweight and scalable than anything the corporation has seen before. The "cloud" has infected IAM like everything else, and it appears that IAM in the age of the cloud is a completely different beast.

And yet, some things have remained the same.

I saw this for myself when reviewing the SCIM specification. This is a provisioning API that is meant to work across generic domains, not just "on the cloud". It's part of the OAuth 2.0 family of specifications, and OAuth 2.0 is an excellent, layerable protocol that can be applied as a cross-cutting concern to protect other APIs. SCIM too is OAuth 2.0-protected, but that's probably where the elegance ends.

The biggest problem with SCIM is its clumsy data model, which then impacts the intuitiveness and friendliness of its API. I critiqued SCIM on InfoQ, and in response to a "put up or shut up" challenge from some of the members of the SCIM working group, I began working on an Internet Draft to propose a new distributed computing protocol, no less. That's a separate piece of work that should see the light of day in a couple of months.

In the meantime, I began to work on IAM at another organisation, a telco this time. My experiences with IAM at a bank, an insurance company and then a telco, had by then given me a much better understanding of Identity as a concept, and I began to see that many pervasive ideas about Identity were either limiting or just plain wrong. Funnily enough, most of these poor ideas had more to do with the Identity data model than with technology. I also observed that practitioners tended to focus more on the "sexy" technology bits of IAM and less on the "boring" data bits, and that explained to me, very convincingly, why systems were so clumsy.

I then consciously began to set down some data-specific tips and recommendations that I saw being ignored or violated. The irony is that it doesn't cost much to follow these tips. All it costs is a change of mindset, but perhaps that's too high a price to pay for many! In dollar terms, the business benefits of IAM can be had for a song. Expensive technology is simply not required.

So that's the lesson I learnt once more, and the lesson I want to share. No matter what changes we think are occurring in technology, the fundamental concepts of Identity have not changed. The data model underlying Identity has not changed. Collectively, we have a very poor understanding of this data model and how we need to design our systems to work with this data model.

So here are 50 data principles for you, the architect of your organisation's Identity Management solution. I hope these will be useful.

The presentation on Slideshare:
http://slidesha.re/14uo3YY

The document hosted on mesfichiers.org:
http://atarj9.mesfichiers.org/en/

The Happy Confluence of IAM, SOA and Cloud

Someone pointed me to this Gartner blog post on IAM, and I was once again reminded why Gartner doesn't get it, (or when they do, they get it much after everyone else).

The Gartner analyst in his presentation makes a big deal of the fact that LDAP, being a hierarchical data structure, is incapable of modelling the various complex relationships between entities in an IAM system. This is one of the reasons he believes we need to "kill IAM in order to save it". But is this limitation in traditional IAM systems really new? I'm no fan of LDAP, and it has been known in IAM circles for at least 5 years that LDAP directories are suited for nothing other than the storage of authentication credentials (login names and passwords)! Everything else should go into a relational database, which is much better at modelling complex relationships. A meaning-free identifier links an LDAP entry with its corresponding record in the relational database. I describe this hybrid design in a fair amount of detail in my book "Identity Management on a Shoestring". And this wasn't even my original idea. It was one of the pieces of advice my team got from a consultant (Stan Levine) that my employer hired to review our IAM plans.

Seriously, where has Gartner been?

Another big point made by the Gartner analyst was that IAM should not be "apart from" the rest of an organisation's systems but become "a part of" them. Joining the dots with my cynical knowledge of where Gartner tends to go with this kind of argument, I can see them making the case for big vendors that do everything including IAM. The cash registers at SAP, Oracle and Salesforce.com must have started ringing already, since Gartner has given those vendors' product strategies their all-important blessing.

Um, no. If there's anything we've learnt in the last few years (especially from SOA thinking), it's the great benefits that are gained from loose coupling. IAM should neither be "apart from" (decoupled) nor "a part of" (tightly coupled) with respect to an organisation's other, business-related systems. IAM needs to be loosely-coupled with respect to them.

What does this mean in practical terms? It means IAM needs to be a cross-cutting concern that can be transparently layered onto business systems to enforce access policies, but without disrupting those systems with IAM-related logic.

That's really what the latest IAM technology, OAuth 2, brings to the table. But the Gartner analyst, while dwelling for quite a while on how great OAuth is, completely omits to define its true contribution.

Eve Maler of Forrester says it much better in her presentations. She defines OAuth as a way to delegate authorisation, and positions it as a way to protect APIs. Can you see the confluence of IAM, SOA and the Cloud in that simple characterisation?

Let's take those two aspects one by one and have a closer look.

OAuth as a way to delegate authorisation:

The traditional model of authorisation works like this. There is an entity that owns as well as physically controls access to a resource. When a client requests access to that resource, the owning entity does three things:

1. Authenticates the client (i.e., establishes that they are who they claim to be)

2. Checks the authorisation of the authenticated client to access the resource (i.e., acts as a Policy Decision Point)

3. Allows (or denies) the client access to the resource (i.e., acts as a Policy Enforcement Point)

What OAuth does is recognise that the Policy Decision Point and the Policy Enforcement Point may be two very different organisational entities, not just two systems within the same organisational entity. The PDP role is typically performed by the owner of the resource. The PEP role is performed by the custodian of the resource. The owner need not be the custodian.

Under the OAuth model, there is a three-way handshake between the owner of a resource, the custodian of the resource and a client. Three separate trust relationships are established between the three pairs of entities in this model, and authentication is obviously required in setting these up (owner-to-client, owner-to-custodian and client-to-custodian-through-owner). Once the owner's permission to access the resource for a certain window of time is recorded in the form of an access token that the client stores, the owner's presence is no longer required when such access takes place. The custodian is able to verify the token and allow access in accordance with the owner's wishes even in the owner's absence. This is delegated authorisation.

And since the resource doesn't even know it's being protected, this is loose coupling. IAM is neither "apart from" nor "a part of" the business system with OAuth.

OAuth as a way to protect APIs:

The delegated authorisation model can be used to protect resources that are not just "things" but also "actions". In other words, OAuth can be used to control who can invoke what logic, and do so in a delegated manner. In other words, owners of business logic can grant access to clients to invoke business logic, and custodians that host such business logic can validate the access tokens presented by clients and allow or deny access in accordance with the wishes of the owners.

Now why does this development in the IAM world bring it into confluence with the SOA and cloud worlds?

The SOA bit is easy to understand. We did mention that an API is a form of resource. If all business logic can be reduced to service operations exposed through endpoints, then these form an API. Endpoints can be protected by OAuth as we saw, so OAuth can be an effective security mechanism for SOA.

The cloud bit isn't hard to understand either. If business logic can be abstracted behind APIs, then does it matter where that logic sits? Bingo - cloud! The cloud also forces separation of owner and custodian roles, with the cloud platform performing the role of custodian, and the cloud customer performing the role of resource owner or API owner. With OAuth as the authorisation mechanism, the cloud model becomes viable from an access control perspective as well.

So that's really what OAuth signifies. It's not just a development in IAM. It has profound implications for SOA security and the viability of the cloud model.

Watch for Gartner to break this news to their clients in 3 to 5 years' time...

(Meanwhile, someone at Gartner or elsewhere ought to tell that analyst that "staid" is not spelled "stayed". This presentation has irritated me on so many levels - spiritually, ecumenically, grammatically, as Captain Jack Sparrow said.)

NDAP - Because User Provisioning Is Too Important To Be Left To LDAP

As I discovered during my foray into SCIM-land, the proposed user provisioning API is extremely clumsy and non-intuitive (check out these examples for yourself), and this is mainly because of multi-valued attributes. By converting multi-valued attributes (arrays) into "key-per-value" attributes (dictionaries), I argued that a much simpler and more elegant API could be developed. While some members accepted that my API was more elegant, the SCIM spec committee overall rejected my proposal as unimplementable. And this was not bloody-mindedness on their part. It is genuinely difficult for existing cloud providers to adopt a sensible and elegant API for user provisioning because they are stuck with legacy data stores that have to support multi-valued attributes, and LDAP directories are the prime culprit. Relational databases are perhaps a tad more flexible than LDAP directories, but it comes down to whether Service Providers would be willing to make changes to their relational schemas. We can pretty much guess the answer to that!

However, it seems a shame to abandon efforts to simplify and standardise the user provisioning API, which is what the SCIM effort should have been all about. Enterprise clients don't care about the legacy constraints of Service Providers. They want a simple and standard API for user provisioning. There may also be emerging Service Providers who are not encumbered by legacy constraints, and who would be willing to explore a better option than SCIM. [There are 3 Gartner people on the SCIM spec committee, but predictably enough, none of them chose to speak up on behalf of their paying customers (enterprise clients) and the one that did speak up supported the interests of incumbent service providers. That sort of behaviour is what lies behind the saying "Gartner spelt backwards is rent-rag."]

I last blogged about the need for "NoLDAP", which I described as stores of JSON documents that permitted no arrays, only dictionaries. Actually, this doesn't have to be restricted to JSON. Any data store that outlaws multi-valued attributes would do as well. In other words, all documents have to be nested dictionaries if they are to allow simple manipulation. The abbreviation "ND" then led to the initially flippant name "NDAP Directory" as an alternative to LDAP directories. But in retrospect, "NDAP Directory" isn't a bad term at all. The "access protocol" part of NDAP is just the RESTful API which includes the "Best Practice" use of PATCH that I suggested earlier.

Here's a high-level picture of how a Service Provider could configure their infrastructure to provide support to conventional authentication mechanisms as always, yet simplify the user provisioning process that has to happen behind the scenes.

The NDAP Directory is the source of truth, and the LDAP Directory is a read-only replica that is used for authentication and retrieval of common user attributes.

Perhaps such a hybrid architecture will be more practical. Will any next-generation SPs rise to the challenge? 

OAuth2 - Whom to Believe?

In the field of Identity Management, a new dawn has begun, if one is to believe authorities such as Eve Maler of Forrester. The old, complex era of XML-based standards (SAML, SPML, XACML) is being replaced as we speak by a more lightweight, web-scale set of standards which are loosely called "the OAuth2 family".

OAuth2 is the one that underpins the others. Eve Maler refers to it as "a means of enabling constrained delegated authorisation of access." In other words, if you need to allow a photo printing service to print some of the photos that you have hosted on Picasa, then you don't have to give the service your Picasa username and password. The OAuth2 protocol allows you to give the service an "access token" for a limited period of time that allows them to access your photos, without having to reveal your security credentials. It takes an interesting 3-way handshake to accomplish this, but once this pattern is established, it can be used to protect any resource, most importantly APIs or service invocations.

The other members of the "OAuth2 family" use this feature so that they no longer need to concern themselves with the authorisation part of security, since OAuth2 will ensure that only a client with the right authorisation can invoke a service. These other APIs are "OpenID Connect" for Single Sign-On and SCIM for user provisioning. [Eve Maler talks up UMA (User Managed Access) as the third piece of the puzzle instead of SCIM, but I view UMA as just OAuth2 at industrial scale. Authentication, Authorisation and Provisioning are the three major aspects of IAM (audit is a fourth), and these are rightfully represented by OpenID Connect, OAuth2/UMA and SCIM, respectively.]

However, just when it looks like things are finally coming together in a simple and sane way, the party has been rudely interrupted.

Just 3 days ago, Eran Hammer, one of the leading figures in the OAuth2 standardisation effort, resigned from the working group and withdrew his name from the effort. The reasons he provides, at least initially, are dismaying and lead one to believe that OAuth2 is doomed as a standard and that we are going to see some catastrophic security breaches very soon.

His reasons for this seem plausible.

• When the standardisation effort behind OAuth moved from the WHAT-WG to the venerable IETF (Internet Engineering Task Force), the lightweight-and-agile web developer crowd was replaced almost overnight by the  enterprise-y crowd. They squashed the agility out of the standard and replaced it with the dead weight of Enterprise thinking. The OAuth2 standard is the camel that the committee came up with when it set out to design a horse.

• The specification fails to deliver on its two main goals of security and interoperability. As evidence, he says it has been renamed from a protocol to a framework, and the spec explicitly warns that implementations are unlikely to be interoperable.

• The main point that the "enterprise crowd" fought for and won was flexibility and extensibility. In Eran Hammer's view, that's a bad thing, because "it is this extensibility and required flexibility that destroyed the protocol. With very little effort, pretty much anything can be called OAuth 2.0 compliant."

Fortunately, I read through the comments following his blog post, especially this very different viewpoint expressed by Leif Johansson.

I now have a more nuanced picture of where OAuth2 stands, and it isn't a picture of doom and despair. I've been in this industry for over 25 years, and I've seen this sort of thing before. Eran Hammer is young and idealistic, and design-by-committee always frustrates the young and idealistic.

The IT industry knows three types of innovators:

1. The young, smart and agile, who come up with a new way of doing something, which then suffers growing pains because its inventors didn't think of all the possible situations in which it would be used. The NoSQL movement exhibits several examples of this.
2. The older, smarter and experienced, who have learnt from years of mistakes (their own and those of other people), and who now know how things are to be done and will brook no questions. They're insufferable and nobody can work with them. (The much-respected and avoided Marshall T Rose comes to mind.)
3. The committees of people with vested interests, who design the ugly camel that no one likes.

I think the most successful products come from the first group. The most successful standards come from the third. (Probably the second group can write good books and tell war stories.)

What Eran Hammer describes about the IETF and what they have done to OAuth is unsurprising and hardly dismaying. He's describing the sausage machine at work, and nobody really wants to see this in operation.

It's important that he's qualifying his rant with these two statements:

• "OAuth 2.0 [in the hands] of a developer with deep understanding of web security will likely result [in] a secure implementation"
• "It is true that [OAuth] 2.0 can be profiled into a simple, straightforward protocol. But every single one of these profiles is slightly different."

What this tells me is that OAuth2 is entering the second phase of its standardisation, when interoperability profiles will need to be defined. The standard isn't inherently broken. We can and do have secure implementations today. We just need to make them interoperable.

As an industry, we know how to do this. We've done it before with Web Services. That was called the WS-I Basic Profile, where "I" stood for Interoperability. Indeed, when Eran began his rant, he called the OAuth2 protocol "a bad protocol. WS-* bad". In other words, not that bad.

In fact, compared to the WS-I Basic Profile, OAuth2 is more of an orthogonal concern. The only aspect of it that an API developer will see is the "bearer token" header. So once the OAuth2 infrastructure is set up, it'll be an unobtrusive part of the IAM fabric. I know people who shudder at the term "profiles", but I don't believe profiles in OAuth2 are going to be visible in the sense of WS-I Basic Profile. So no course corrections required, folks. It's still full steam ahead with OAuth2. We'll give the angry young man some time and space to settle down.

[I have a rant of my own, a different one altogether, about the whole Identity-Management-as-Technology mindset, and will shortly be writing about the neglected stepchild of IAM - The Data Model. Watch this space.]

A RESTful Protocol to Add/Modify/Delete Elements of a Collection

I've been reading the SCIM specification recently.

[What does SCIM stand for? There's a moral-of-the-story here, by the way. SCIM originally stood for Simple Cloud Identity Management. Now it stands for System for Cross-domain Identity Management. The moral of the story is, if you're swayed by the buzzword of the day, you will be forced into embarrassing retreat at some point. Cross-domain Identity Management was what it was always about. What a shame someone felt the need to jump onto the cloud bandwagon.]

Anyway, coming back to the topic of this post, I realised that one of the aspects dealt with by the SCIM specification was a much more general problem that would benefit from a standard approach.

I was struck by the similarity of 3 types of operations in the SCIM protocol, though not impressed by any of them.

One set of operations deals with adding, modifying and deleting multi-valued attributes of a user, e.g., the set of addresses or email IDs.

A second set deals with adding, modifying and deleting users as members of a group.

SCIM uses a RESTful API, so for these sorts of partial updates to a larger entity, it uses PATCH. A single PATCH command can be used to simultaneously add, modify and delete attributes of an entity, or to add, modify and delete members of a group. [PATCH when used to update one or more single-valued attributes is fairly straightforward, so I'm not talking about that here.] I suspect that PATCH is so new that best practice around its use has yet to emerge, which accounts for some of the ugliness I'm about to describe.

A third set of operations is called 'bulk operations', and deals with a collection of operations on multiple entities, which could include add, modify and delete operations. This is physically formatted as a POST with nested POST, PATCH and DELETE operations as attributes within the document.

To my mind, these three situations are conceptually the same (a set of fine-grained operations grouped into a single composite operation), therefore the same mechanism should ideally be used for all of them.

As it turns out, SCIM uses many different mechanisms, all ugly.

1. Multi-valued attributes of a user:

(In the spec document linked above, scroll up and down a bit to search for these examples)

Some of the ugliness comes from passing 'operation:"delete"' as an attribute of the document. After arguing for years that a GET request with URL parameter "?operation=delete" is not RESTful, we can't now barefacedly offer a "RESTful" API that does the same thing. True, this is not a GET, so it's probably not as awful, but this is a really bad "smell" that suggests poor design at some level.

The syntax to replace a user's primary email address with another one and simultaneously demote the previous primary email address to the status of just another email address is more than just ugly, it's horrific. Do have a look for yourself, if you can keep from gagging.

2. Members of a group:

(In the spec document linked above, scroll up and down a bit to search for this example)

Deleting all members from a group is even uglier. There is nothing in the message to suggest that a deletion is taking place! There's a "meta" attribute that has a value equal to the set of attribute names whose (multiple) values are being deleted. If that confuses you, you now know what I mean.

3. Bulk operations:

The main operation is a POST to an arbitrary URI which could be called "/bulk" or something similar.

There are nested operations within it that reflect operations on various entities.

As you can see, these are 3 entirely different mechanisms for essentially the same desired behaviour. Can't we use the REST philosophy of polymorphic verbs to make this interface more uniform?

I would like to propose a simpler scheme that uses explicit identifiers for every sub-entity that may need to be independently manipulated (updated or deleted). This is similar to having resource identifiers, but at finer levels of granularity.

All three sets of operations should have a common structure like this:

PATCH /main-entities/29d70c13-8651-41ce-97f8-c82ccdfcd6d6

(or POST /bulk-operations)

Host: example.com

Accept: application/json

Authorization: Bearer h480djs93hd8

Content-Length: ...

{

  operations:

  [

    {

      operation: "POST",

      URI: " /sub-entities",

      sub-entity:

      {

      ...sub-entity attributes

      }

    },

    {

      operation: "PUT",

      URI: "/sub-entities/dced46ca-b96d-44dd-b494-41e5de9405b6",

      sub-entity:

      {

      ...sub-entity attributes

      }

    },

    {

      operation: "DELETE",

      URI: "/sub-entities/12feb1db-8031-4a23-8810-bccfffb13f56"

    }

  ]

}

i.e., the nested operations should be RESTful operations in their own right, with standard HTTP verbs and standard URIs using the enveloping entity's URI as their base URI. How do we know what the URIs of the sub-entities are? The server should generate one for every array element that is created (multi-valued attributes of an entity), and return those URIs with the "201 Created" response, as well as in response to any GET on the main entity. That will then provide a handle for the client to specify fine-grained operations on the sub-entities.

I also don't like the SCIM convention of returning a "200 OK" even for a bulk request where different sub-operations may have had different outcomes. This is as bad as HTTP tunnelling in SOAP where the HTTP response could be a cheery "200 OK", but the message inside contains a SOAP fault.

There is a WebDAV status code for exactly this kind of situation, - "207 Multi-Status". That's what should be used.

HTTP 1.1/207 Multi-Status

Content-type: application/json

{

  operation-results:

  [

    {

      operation: "POST",

      URI: "/sub-entities",

      status: "200 OK",

      Location: "/sub-entities/c44fcc7b-8881-4f0f-b5fb-b2344615cc34",

      request:

      {

      ...sub-entity attributes repeated for verification

      }

    },

    {

      operation: "PUT",

      URI: "/sub-entities/dced46ca-b96d-44dd-b494-41e5de9405b6",

      status: "409 Conflict"

    },

    {

      operation: "DELETE",

      URI: "/sub-entities/12feb1db-8031-4a23-8810-bccfffb13f56",

      status: "404 Not found"

    }

  ]

}

This is a clean syntax, easily understood, and uniformly applicable when dealing with attributes or sub-entities of an entity, or with bulk operations. All it requires is for resources and resource collections to be identifiable at finer levels of granularity as well, i.e., attributes that are array elements.

I don't believe that's too much of an ask. UUIDs are cheap and plentiful, and the alternative is an ugly mess.

Architectural Lessons in Identity from the Higgs Boson

As I read the details of last week's virtual confirmation of the existence of the Higgs boson, I smacked myself on the head with a "D'oh!"

As an architect, I've been preaching against tight coupling for years, and I utterly failed to realise that there could be examples of tight coupling in the way we think about the real world.

What the discovery of the Higgs boson does is confirm what is called the Standard Model of Particle Physics, and one of the fundamental, though unintuitive, aspects of that model is that mass is not an inherent property of a particle. A particle may acquire mass through interaction with the Higgs field, but it's not necessary. That's why it's possible for particles like photons to have no mass.

It's ironical that in the months just prior to this discovery in the world of Physics, I made a modest discovery of my own that nevertheless shook the foundations of my conceptual world. In the realm of identity management, I realised that any given entity has no inherent properties (attributes) at all! All attributes that an entity may be deemed to have are only by association (just as a particle acquires mass independently of its existence), and therefore the only necessary aspect of an entity is a unique and meaning-free identifier to set it apart from other similar entities. Attributes can then be assigned and de-assigned to it at will through the mechanism of the identifier, and gradually a more sophisticated model can be built up. In creating such sophisticated models, it's important to remember that nothing is inherent to an entity, - not first name/last name, not primary email address, not social security number, - nothing! Just a unique identifier that is internal to the domain and kept invisible to the world outside the domain.

I've found this minimalist model of identity to be amazingly powerful and flexible. In addition to having arbitrary groups of attributes associated with an entity's identifier to form its properties, multiple external identifiers can also be independently associated with the (internal) identifier. The usual surrogate for identity, the username, is then just one such external identifier. Much of the conceptual confusion in identity management comes from mistaking external identifiers like username for the entity itself, leading to extremely clumsy and costly implementations. By creating an explicit internal identifier and treating any external identifier as an attribute associated with the internal identifier, such confusion and costly error can be avoided.

Cross-domain entity references can also be managed very cleanly by mapping both domains' internal identifiers to a shared external identifier, so that neither domain is coupled to the other domain's internal identifier for an entity. That's goodness!

Federated identity, such as when using one's FaceBook ID to enter a website, is nothing but treating the FaceBook ID as an external identifier and associating it with the website's internal identifier to establish the entity's identity. So multiple domains can independently maintain models about the same entity, and for most of the time, they can behave as though they are dealing with different entities. When it becomes necessary for the two domains to recognise these entities as the same, they then create a shared identifier that is external to both of them, and they each map this external identifier to their respective internal identifier.

This is a model of federation that does not make the assumption that there is a centre of the universe. No domain is the centre of the universe. Every domain is independent, and when they need to share their model of an entity, they loosely associate their entity references through a shared external identifier. There is now traceability across domains, but no implied control. I've found in practice that this is the only model that works, - technically, logistically and politically.

It's amazing that I should receive indirect confirmation of my identity model from as unlikely a source as Particle Physics, but I guess that's just a reflection of the fundamental nature of all things.

We have nothing inherent to ourselves but an identity. That's a pretty humbling thought that verges on the religious.

Report from QCon New York 2012

I'm back in Sydney after a week in New York attending QCon, the conference for developers organised by InfoQ. I was invited to speak on the topic of "loosely-coupled identity management", since the eBook co-authored by me was published by InfoQ recently.

It's always a nice experience attending tech conferences. There's an exciting buzz in the air with so many "new, improved" ways of doing things on display. Some are marketing hype by vendors, others genuine technology advancements. QCon is not a vendor-focused conference though. Most of the technologies being discussed were Open Source.

One of the keynote speakers was Cameron Purdy, who is currently a senior executive at Oracle. I remember when Cameron was CEO of his own company Tangosol, and their product Coherence was one of the leading persistence technologies for Java. I used to enjoy the sparring matches between Cameron and a bunch of other people (usually JBoss employees) on the now all-but-defunct site TheServerSide.com (I mean the site is still there, but the party has since moved on to other places, notably StackOverflow and InfoQ). "Peace" was always his sign-off line. Looks like he's moved up quite a bit in life now, with Oracle's purchase of Tangosol.

Cameron's keynote had one major takeaway for me. I agree with his assessment that we as an industry are once again in the midst of a once-in-fifteen-years technology paradigm shift, and this one involves three elements:

1. The Cloud

2. HTML 5

3. Mobile devices

I'm going to take his advice very seriously and focus my attention on these three areas. I strongly feel the combination is going to be formidable.

My old friend and colleague Jim Webber was there. Jim is now Chief Scientist at Neo Technology, the maker of the Open Source Neo4j graph database. Graph databases are quite possibly a much more powerful way of modelling relations between entities than relational databases are, and the performance seems in no way compromised. On the contrary, if Jim is to be believed, the performance of a graph database can be orders of magnitude better than that of a relational database when the paradigm fits. (To be fair to Jim, he doesn't sell graph databases as the solution to every problem, but there does seem to be a large and growing set of applications that can benefit from them.) I confess I haven't got a firm handle on exactly how graph databases work and exactly what use cases can benefit from them, but they seem to work very well for data points that have a rich set of relationships with one another. According to Neo Technology's site, graph databases can find answers to questions like “what music do my friends like that I don’t yet own,” or “if this power supply goes down, what web services are affected?”

I missed Jim's tutorial on Neo4j because some competing (and more basic!) tutorials caught my attention, but I did attend his other tutorial on REST, about which I blogged earlier.

I attended a number of other talks and tutorials. The most memorable for me were the ones by Kaazing on HTML5 WebSockets. Richard Clark's half-day tutorial was quite good, but since the contents were compressed into 3 hours from a full-day workshop, we couldn't do justice to all the examples. But WebSocket technology is definitely something I need to play with and understand.

Subbu Allamaraju's talk was very interesting too. I was intrigued when he started off by saying the client side of REST had been neglected so far, but his thinking proved to be very different from mine. I've been thinking about a client-side resource model applied to a peer-to-peer messaging protocol (I know it sounds silly to talk about clients in a P2P world, but what I mean by "client" is "initiating peer"), but Subbu's idea is about a structured way to manage application state held on the client across "service invocations", a sort of client-side orchestration capability. The capability seems quite well advanced judging from his demo. He calls it ql.io, and it's well worth a dekko.

I met some of the REST greats at the conference, notably Mike Amundsen and Leonard Richardson. I didn't realise until later that Leonard was one of the authors of my favourite REST book, "RESTful Web Services". Leonard has also recently published a Science Fiction novel called "Constellation Games", which makes him a multi-dimensional nerd.

My own talk went off OK. (The slides are here.)

 "Faster, Cheaper Identity Management Through Loose Coupling - The LIMA Approach"

I had only 16 attendees, which was a bit disappointing, but the organisers told me it's always hard to get interest in the security track because developers are so much more interested in the mainstream technology tracks. However, I did receive good feedback about my talk. They have a system of gauging audience response by letting attendees place different coloured cards in a bowl after the talk (green for 'good', yellow for 'OK', red for 'not good'). I got 12 green and 2 yellow cards (no reds), which I take as a pretty good endorsement :-).

It's a heck of a journey from Sydney to New York (13 hours to LA, a 3 hour layover, and then a further 4.5 hours to NY) and back again, but I enjoyed myself. Qantas economy is pretty comfortable and I didn't feel the strain much.

An Optimistic Approach to Identity

I've been working in the Identity Management area for a few years now, and I've seen three different industries up close (banking, insurance and telecom). What I'm struck by in all these industries is that none of them has historically been customer-centric in their business approach. For decades, banks have always looked at their customers through the prism of accounts, insurance companies through policies, and telecom companies through billing accounts and sometimes carriage services (broadband or mobile services). And everywhere, the holy grail is the same - "single view of customer". Identity and Access Management (IAM) is the way these organisations aim to achieve single view of customer as well as other benefits.

However, IAM initiatives at organisations in all these industries have generally floundered. Why?

I believe that IAM is simple but subtle. That's why although it's not hard to design and deliver an IAM system, it's also treacherously easy to get it wrong.

Some of the major reasons why organisations struggle with IAM are these:

1. Rather than bite the bullet and create a top-level data entity called "customer" with its own unique identifier, organisations choose what they consider a cheaper compromise because of a misplaced belief that using a surrogate for customer (i.e., account, policy, billing account) would somehow do the job. Reality check: it doesn't, and it's more expensive in the long run.

2. Even where identifiers are created for customers, these are not carefully designed. The result is that many identifiers that are chosen have business meaning. It's quite funny at one level to see a system designed with a person's email address as their identifier, and where the major business pain point is that it's very hard to handle the situation where a customer changes their email address. (Why are we not surprised?) Quite often in such cases, there is no other way around the problem but to delete and re-create the customer record.

3. Even where organisations avoid the first two mistakes and embark on an IAM initiative to tie customer data across multiple systems to a new, unique and meaning-free customer ID, they run into logistical problems relating to the existing user base. They struggle to "marry" records across systems to the appropriate customer entity because of the sheer volume of data involved, the cost of changing existing systems, the unreliability of matching algorithms and the need to replace engines while the plane is flying, so to speak. The two problems with matching algorithms are false positives (two or more customers being assigned the same identifier) and false negatives (a customer being assigned two or more identifiers).

I have some suggestions that can make life easier.

1. Create a database external to all existing systems that will maintain mappings. [Resist the temptation to migrate customer attributes from other systems to this one. This is just a mapping database, not a customer master. Use Master Data Management (MDM) principles instead to keep data in source systems in sync.]

2. Use a universally unique and meaning-free identifier for customers. Version 4 (random) UUIDs are a great scheme to use.

3. Adopt an optimistic model of "eventual consistency". I.e., generate a new customer UUID corresponding to every system record, in effect assuming (in the case of a bank) that each account belongs to a different customer, then pare them down to reflect known relationships. 

a) You can generate UUIDs for a system in an optimistic way because the probability of two UUIDs conflicting is infinitesimally low, even if you have hundreds of millions of customers. You can check for duplicates out of band if you're paranoid.

b) Similarly, you can optimistically generate UUIDs in a federated way (i.e., each system generates its own UUIDs corresponding to its surrogate records). The probability of conflict is so low it's worth doing this and checking for duplicates out of band.

c) You can afford to start with a system with a large number of false negatives (but no false positives) because this corresponds to a siloed organisation with no "single view of customer". False positives are a greater danger, and we avoid that with this scheme.

d) You can use the existing intelligence in your systems (i.e., the knowledge of which records belong to the same customer) to merge customer UUIDs relating to the same physical customer by eliminating all but one of them at random. Since UUIDs are meaningless, it doesn't matter which one you keep and which ones you remove.

Now you're no worse off than you were before in terms of data quality (i.e., your data is just as clean in terms of known relationships). But structurally, you're far better off because you now have a customer data entity for the first time. As your data quality improves with more reliable mappings, the siloes effectively disappear and you get to a "single view of customer" with no more changes to data structures or processes.

In the case of a telecom company, your mapping database will now consist of three parts. The first part will map customer UUIDs to billing accounts. The second part will map customer UUIDs to product holdings (mobile, broadband and other carriage services, media products, etc.) The third part will map customer UUIDs to other customer UUIDs to reflect corporate organisational structures and household relationships. With this model, the many problems that telecom companies currently face will simply melt away.

- We can see all the product holdings of a customer to determine what else to sell them. We can see this at an individual customer level as well as at the level of a household or organisational unit.

- We can sell media products even to customers who haven't purchased an underlying carriage service

- We can group billing accounts independently of product holdings. In a household, the kids use various products but mum or dad alone may pay the bill.

As you can see, this kind of design isn't hard. But it requires conceptual clarity around the nature of Identity. As I said before, IAM is simple but subtle. It isn't hard to design and deliver an IAM system, but it's treacherously easy to get it wrong.

Identity Management on a Shoestring - eBook Published by InfoQ Today

I've written before about the white paper on Identity Management that I co-authored with Umesh Rajbhandari. We have been in discussions with InfoQ about publishing it as an eBook, and after a long edit process (mainly due to delays from my side), this has finally made its appearance.

Anyone interested can download the PDF here after (free) registration.

According to the publishing agreement, we have to point back to the InfoQ site for the soft copy, so I have removed all links from my blog to the original site where I had hosted the white paper. The latest version has some additional material that wasn't in the original white paper, so it makes more sense to go to the InfoQ site in any case.

I hope this book will prove useful to practitioners embarking on implementing an Identity and Access Management (IAM) system for their organisations. We believe we have pioneered a loosely-coupled approach to IAM that is unique and delivers significant benefits compared to off-the-shelf commercial alternatives, and we call this LIMA (Lightweight/Low-cost/Loosely-coupled Identity Management Architecture).

Identity Management on a Shoestring - The LIMA Approach

Umesh Rajbhandari and I have documented the design of an Identity and Access Management (IAM) system that we successfully built for a large Australian corporation. We call our unique architectural approach "LIMA" (Lightweight/Low-cost/Loosely-coupled Identity Management Architecture).

The document describing LIMA is here. [Update 20/03/2012: I have changed the link to point to the InfoQ site instead of my own hosted copy because that was one of the conditions of publication.]