The End Of Ubuntu One - What It Means

Although a big fan of Ubuntu Linux as a desktop OS, I've never been interested in their cloud storage platform Ubuntu One, and found it a bit of a nuisance when asked to sign up for it every time I installed the OS.

Now Ubuntu One is being shut down. I'm 'meh' but still a bit surprised.

The linked article talks about mobile, and how new mobiles such as the Ubuntu-powered ones need cloud storage to succeed. If so, isn't it really bad timing for Canonical to walk away from a fully operational cloud platform just when its mobile devices are entering the market?

Ubuntu-powered smartphones
(Do you know what the time on the middle phone refers to?)

I think it's about economics.

Ubuntu's statement says:

If we offer a service, we want it to compete on a global scale, and for Ubuntu One to continue to do that would require more investment than we are willing to make. We choose instead to invest in making the absolute best, open platform and to highlight the best of our partners’ services and content.

Hmm. I read this as Canonical trying to build a partner ecosystem that will substitute for having a big cloud-and-mobile story like Google does, without the investment that such a proprietary ecosystem will require. Let's see if they succeed.

The other side-story in the linked article is about telcos and their role. Having worked at a telco over the last two years, I can confirm that the major fear in the telco industry is being reduced to commodity carriers by "over the top" services. The telcos are fighting to offer content, and will want willing mobile wannabe partners like Mozilla and Canonical to offer smartphone platforms that will work with networking infrastructure and make the telcos more attractive (through content that both players source from content providers). It will be interesting to see how this four-way, federated partnership (between multiple telcos, independent smartphone platform vendors like Mozilla and Canonical, smartphone device OEMs and content providers) will play out. Many of these companies will think of themselves as the centre of the Universe and the others as partners.

"Nothing runs like a fox" - Well, let's see if the Firefox Smartphone has legs

In the meantime, some good news for startup cloud providers ("startup" only with respect to the cloud, since they will still need deep pockets to set up the infrastructure!): Canonical is open-sourcing its Ubuntu One storage code “to give others an opportunity to build on this code to create an open source file syncing platform.” This should be interesting.

After NoSQL, It's Time For NoLDAP

I've been in discussions with the IETF SCIM working group (formerly Simple Cloud Identity Management and now called System for Cross-Domain Identity Management) about the three suggestions for improvement that I wrote about in my InfoQ article.

One of the things I've learnt from this discussion is that, while I sense a bit of an NIH syndrome in some of the pushback I have received, there are also some genuine technical challenges involved in implementing my ideas. In my article, I stated that the fundamental problem with manipulating resources was multi-valued attributes that lacked a unique key per value. Over the past two weeks of discussion, I'm convinced that I got that absolutely right. Multi-valued attributes are the problem. And the biggest culprit in keeping them entrenched with little hope of change is that legacy beast known as an LDAP directory.

You can see for yourself how clumsy it is in LDAP to update multi-valued attributes. When I googled "multi-valued attributes" and "LDAP" together, I found this prescient piece written by Mark Wahl in 2005. I was amazed to see how accurately he had anticipated the problem:

Unfortunately, some of the emerging protocols which also intend to represent and transfer personal identity information have perhaps taken a step backwards by not even considering these issues [problems with multi-valued attributes], perhaps sweeping them under the rug in the guise of simplicity, XMLification, or "fix in the next version", which only postpone finding interoperable solutions to allowing applications to express the identity entries they want to express.

SCIM is one of these "emerging protocols", and sure enough, the arguments that I have been facing have explicitly cited "simplicity" as the reason not to accept my solution proposal, exactly as Mark Wahl had predicted. However, the real reason for resistance is not about "simple" but about "easy".

It is not easy for SCIM to move to a simple model, and that is the problem. It is more expedient to maintain complexity.

The irony is that the SCIM spec is anything but simple, thanks to multi-valued attributes. This is how the spec proposes to deal with multi-valued attributes:

Multi-valued attributes: An attribute value in the PATCH request body is added to the value collection if the value does not exist and merged if a matching value is present. Values are matched by comparing the value Sub-Attribute from the PATCH request body to the value Sub-Attribute of the Resource. Attributes that do not have a value Sub-Attribute; e.g., addresses, or do not have unique value Sub-Attributes cannot be matched and must instead be deleted then added. Specific values can be removed from a Resource by adding an "operation" Sub-Attribute with the value "delete" to the attribute in the PATCH request body. As with adding/updating attribute value collections, the value to delete is determined by comparing the value Sub-Attribute from the PATCH request body to the value Sub-Attribute of the Resource. Attributes that do not have a value Sub-Attribute or that have a non-unique value Sub- Attribute are matched by comparing all Sub-Attribute values from the PATCH request body to the Sub-Attribute values of the Resource. A delete operation is ignored if the attribute's name is in the meta.attributes list. If the requested value to delete does not match a unique value on the Resource the server MAY return a HTTP 400 error.

Sounds like quite the dog's breakfast, doesn't it?

To be fair, that's the implementation, which is usually hidden from sight. Let's see if the visible API looks any better.

Here's how to delete a single member from a Group, as per the current spec:

   PATCH /Groups/acbf3ae7-8463-4692-b4fd-9b4da3f908ce
   Host: example.com
   Accept: application/json
   Authorization: Bearer h480djs93hd8
   ETag: W/"a330bc54f0671c9"

   {
     "schemas": ["urn:scim:schemas:core:1.0"],
     "members": [
       {
         "display": "Babs Jensen",
         "value": "2819c223-7f76-453a-919d-413861904646"
         "operation": "delete"
       }
     ]
   }

Here's how to delete ALL members from a group according to the current spec:

   PATCH /Groups/acbf3ae7-8463-4692-b4fd-9b4da3f908ce
   Host: example.com
   Accept: application/json
   Authorization: Bearer h480djs93hd8
   ETag: W/"a330bc54f0671c9"

   {
     "schemas": ["urn:scim:schemas:core:1.0"],
     "meta": {
       "attributes": [
         "members"
       ]
     }
   }

For two functions that do very similar things, the syntax is wildly different. It's hardly what one would call simple. But these represent the easy way out from an implementation perspective, and so the SCIM Working Group is extremely reluctant to tamper with this implementation.

With my suggestion (based on "best practice" around the use of PATCH), here's how to delete a single member from a group:

   PATCH /Groups/acbf3ae7-8463-4692-b4fd-9b4da3f908ce
   Host: example.com
   Accept: application/json
   Authorization: Bearer h480djs93hd8
   ETag: W/"a330bc54f0671c9"

   {
     "operations" : [
       {
         "RETIRE" : {
           "key" : "members.2819c223-7f76-453a-919d-413861904646"
         }
       }
     ]
   }

Here's how I suggest deleting ALL members from a group:

   PATCH /Groups/acbf3ae7-8463-4692-b4fd-9b4da3f908ce
   Host: example.com
   Accept: application/json
   Authorization: Bearer h480djs93hd8
   ETag: W/"a330bc54f0671c9"

   {
     "operations" : [
       {
         "RETIRE" : {
           "key" : "members"
         }
       }
     ]
   }

That's a darn sight more elegant, if I may say so myself. In both cases, a reader can tell exactly what is being attempted. What's more, the syntax for both "delete" functions is identical.

This is arguably simple as an API. But I'm told it cannot be implemented, because it's not simple easy. I'm told that cloud service providers will not be willing to make the changes required to make this work, and this will hinder adoption of the spec. And what do incumbent cloud providers use to store identity information? Why, LDAP of course!

That's why I'm coming around to the view that no incremental change is possible. Human society progresses in its views not because people gradually change their minds, but because generations die, and new generations with different ideas take their place. Ideas only die when people die. People never change their minds. It's the people who have to be replaced. A cynical observation, but probably true. The Augean stables can only be cleaned with a flood. 

So we need a new spec, a new set of cloud service providers and a new type of directory, if we are to simplify cross-domain identity management.

It's the last of these that I want to talk about here.

I want a directory that has the advantage of LDAP (fast reads) without the disadvantages (a rigid tree structure and awful treatment of multi-valued attributes). I think one of the NoSQL document databases that can hold JSON documents may fit the bill, -- with one important constraint. JSON supports two types of data structures, the dictionary and the array. The dictionary forces every value to have a unique key. That's goodness. The array allows multiple values per key. As we now know, that's the root of all evil. Therefore, our document database must only allow dictionaries and not arrays. Clients are still free to upload data to the directory in array form, but the directory will only store these arrays after converting them into dictionaries. It will generate its own random keys for them. (If the order of elements is to be preserved, the generated keys can be in sequence, with sufficient gaps between them to permit insertions of other values later.)

Every element of such a directory can be addressed in a fully-qualified way through "dot notation". Take this example of part of a Telecom company's customer database. The logical customer record shown has two addresses, and different telecom "carriage" services are available at the two addresses. The first address has carriage services "cable" and "ADSL", while the second has "ADSL2+" and "Wi-fi". As we can see, that's two nested arrays. The outer array holds addresses, and within each address, there's an inner array holding available services.

{
  ...
  addresses: [
    {
      "type" : "home",
      "street_number" : "35",
      "street_name" : "High Road",
      ...
      "country" : "Australia",
      "available_services" : ["cable", "ADSL"]
    },
    {
      "type" : "office",
      "street_number" : "213",
      "street_name" : "Main Street",
      ...
      "country" : "Australia",
      "available_services: ["ADSL2+", "Wi-fi"]
    }
  ]
}

Let's say we need to update the customer record, because the first address now has "Wi-fi" too, and it no longer has "cable". How do we do this?

We want to delete
"customerX.addresses[0].available_services[0]"

and insert "Wi-fi" into the array
"customerX.addresses[0].available_services".

However, we can't reliably use positional indexes because these can change with every operation. We need stable identifiers. (I will not even attempt to formulate the SCIM syntax required for these operations!)

Here's how I suggest it should be done:

PATCH /Users/2819c223-7f76-453a-919d-413861904646

{
  "operations" : [
    {
      "INCLUDE" : {
        "key" : "addresses.d6ea365462f5.available_services",
        "value" : "Wi-fi"
      },
      "RETIRE" : {
        "key" : "addresses.d6ea365462f5.available_services.9be6378dc303"
      }
    }
  ]
}

Again, if may say so myself, this is elegance. And that's only possible because my directory has converted these two arrays into dictionaries, like this:

{
  ...
  addresses: {
    "d6ea365462f5" :
    {
      "type" : "home",
      "street-number" : "35",
      "street-name" : "High Road",
      ...
      "country" : "Australia",
      "available_services" : {
        "9be6378dc303" : "cable", 
        "6aa1429eba34" : "ADSL"
      }
    },
    "3cbaaff8e84e" :
    {
      "type" : "office",
      "street-number" : "213",
      "street-name" : "Main Street",
      ...
      "country" : "Australia",
      "available_services: {
        "2beca1fdf3e5" : "ADSL2+", 
        "8c3dcc204a33" : "Wi-fi"
      }
    }
  }
}

(But how does the client know these generated keys? They're returned as part of the record creation response, and also provided as part of the resource representation returned with every GET request.)

This model is easy to support if a service provider implements its identity store as a database of JSON documents rather than in LDAP. One can have both "simple" and "easy", without compromises. But for that, LDAP has to go. "NoLDAP" is what we need in its place.

My definition of a "NoLDAP Directory" is simply this:
A document database that holds dictionary-only JSON documents.

I think the next generation of cloud service providers will emerge based on this architecture, offering a simple API that is also easy for them to implement.

As for LDAP and SCIM, I guess the best TLA is RIP.

Report from QCon New York 2012

I'm back in Sydney after a week in New York attending QCon, the conference for developers organised by InfoQ. I was invited to speak on the topic of "loosely-coupled identity management", since the eBook co-authored by me was published by InfoQ recently.

It's always a nice experience attending tech conferences. There's an exciting buzz in the air with so many "new, improved" ways of doing things on display. Some are marketing hype by vendors, others genuine technology advancements. QCon is not a vendor-focused conference though. Most of the technologies being discussed were Open Source.

One of the keynote speakers was Cameron Purdy, who is currently a senior executive at Oracle. I remember when Cameron was CEO of his own company Tangosol, and their product Coherence was one of the leading persistence technologies for Java. I used to enjoy the sparring matches between Cameron and a bunch of other people (usually JBoss employees) on the now all-but-defunct site TheServerSide.com (I mean the site is still there, but the party has since moved on to other places, notably StackOverflow and InfoQ). "Peace" was always his sign-off line. Looks like he's moved up quite a bit in life now, with Oracle's purchase of Tangosol.

Cameron's keynote had one major takeaway for me. I agree with his assessment that we as an industry are once again in the midst of a once-in-fifteen-years technology paradigm shift, and this one involves three elements:

1. The Cloud

2. HTML 5

3. Mobile devices

I'm going to take his advice very seriously and focus my attention on these three areas. I strongly feel the combination is going to be formidable.

My old friend and colleague Jim Webber was there. Jim is now Chief Scientist at Neo Technology, the maker of the Open Source Neo4j graph database. Graph databases are quite possibly a much more powerful way of modelling relations between entities than relational databases are, and the performance seems in no way compromised. On the contrary, if Jim is to be believed, the performance of a graph database can be orders of magnitude better than that of a relational database when the paradigm fits. (To be fair to Jim, he doesn't sell graph databases as the solution to every problem, but there does seem to be a large and growing set of applications that can benefit from them.) I confess I haven't got a firm handle on exactly how graph databases work and exactly what use cases can benefit from them, but they seem to work very well for data points that have a rich set of relationships with one another. According to Neo Technology's site, graph databases can find answers to questions like “what music do my friends like that I don’t yet own,” or “if this power supply goes down, what web services are affected?”

I missed Jim's tutorial on Neo4j because some competing (and more basic!) tutorials caught my attention, but I did attend his other tutorial on REST, about which I blogged earlier.

I attended a number of other talks and tutorials. The most memorable for me were the ones by Kaazing on HTML5 WebSockets. Richard Clark's half-day tutorial was quite good, but since the contents were compressed into 3 hours from a full-day workshop, we couldn't do justice to all the examples. But WebSocket technology is definitely something I need to play with and understand.

Subbu Allamaraju's talk was very interesting too. I was intrigued when he started off by saying the client side of REST had been neglected so far, but his thinking proved to be very different from mine. I've been thinking about a client-side resource model applied to a peer-to-peer messaging protocol (I know it sounds silly to talk about clients in a P2P world, but what I mean by "client" is "initiating peer"), but Subbu's idea is about a structured way to manage application state held on the client across "service invocations", a sort of client-side orchestration capability. The capability seems quite well advanced judging from his demo. He calls it ql.io, and it's well worth a dekko.

I met some of the REST greats at the conference, notably Mike Amundsen and Leonard Richardson. I didn't realise until later that Leonard was one of the authors of my favourite REST book, "RESTful Web Services". Leonard has also recently published a Science Fiction novel called "Constellation Games", which makes him a multi-dimensional nerd.

My own talk went off OK. (The slides are here.)

 "Faster, Cheaper Identity Management Through Loose Coupling - The LIMA Approach"

I had only 16 attendees, which was a bit disappointing, but the organisers told me it's always hard to get interest in the security track because developers are so much more interested in the mainstream technology tracks. However, I did receive good feedback about my talk. They have a system of gauging audience response by letting attendees place different coloured cards in a bowl after the talk (green for 'good', yellow for 'OK', red for 'not good'). I got 12 green and 2 yellow cards (no reds), which I take as a pretty good endorsement :-).

It's a heck of a journey from Sydney to New York (13 hours to LA, a 3 hour layover, and then a further 4.5 hours to NY) and back again, but I enjoyed myself. Qantas economy is pretty comfortable and I didn't feel the strain much.

Wanted: An ESC, not an ESB

Here's my fundamental beef with an ESB: It follows a hub-and-spokes model that is more suited to an EAI initiative of the 1990s. In the year 2010, we should be doing SOA, not EAI.

SOA is federated where EAI is brokered. It's the architectural antithesis of EAI. The smarts are pushed out from the middle to the endpoints until there's no "middle" anymore. I wrote about the difference between "federated" and bad old "point-to-point" before. Federated is a standardised smartening of the endpoints.

It doesn't matter whether we do SOAP or REST. Either way, it's SOA, and it's inherently federated.

Both SOAP's Web Service Endpoints and REST's Resources are represented by URIs. URIs offer a "flat" address space. They're completely opaque. DNS can hide where the services physically sit, so any node can talk to any other without the need for an intermediary.

Look at a SOAP message. Look at the headers - WS-Addressing, WS-Security, WS-ReliableExchange. They're all end-to-end protocols. The SOAP engines at the endpoints negotiate and handshake every Quality of Service they need without depending on anything in between.

WS-ReliableExchange uses endpoint-based logic such as acknowledgements, timeouts and retries to provide a measure of reliability even with unreliable transports, much like TCP provides connection-oriented communications over a plain packet routing protocol like IP. There is no need for a "guaranteed delivery" transport like a message queue.

WS-Trust does key exchange, WS-SecureConversation sets up a session key and WS-Security encrypts messages, all in an end-to-end fashion. There is no need for a secure transport.

Any node that talks SOAP and WS-* and conforms to the WS-I Basic Profile (and Basic Security Profile and Basic Reliability Profile) can participate. That's federation.

This isn't an academic discussion. The main drawbacks of a hub-and-spokes model are Availability and Scalability. The ESB is a single point of failure and a performance bottleneck. The normal "solution" is to beef up the ESB by providing for redundancy and "High Availability", but this costs a fair bit and only postpones the inevitable. That's why I said this isn't an academic discussion. There are serious dollars involved. Organisations would do well to think about the implications of an ESB approach and take early steps to avoid the inevitable costs that lie along that path. The correct solution would be to do away with the centralised ESB altogether and embrace the inherently federated model of SOAP (or REST).

Otherwise, we're forcing inherently federated protocols into a hub-and-spokes model, - a needless throwback to an earlier era , - and inviting problems that need to be "solved" at great expense. (Of course, it's a different matter if we use multiple instances of the ESB as brokers close to endpoints, but economics militates against such a topology, so I suspect federated ESBs will not be feasible.)

In the year 2010, when clouds are all the rage, we should recognise that what SOAP and REST give us are "service clouds".

So that's my buzzword for today - ESC (Enterprise Service Cloud)!

The ESB is dead, long live the ESC!