Resources Are Four-Dimensional

The term ROA (Resource-Oriented Architecture) is misleading. It should ideally stand for "Resource/Representation-Oriented Architecture", even though that's quite a mouthful.

I've found in my discussions with people who work on REST-based systems that lots of them are very fuzzy about the notions of "resource" and "representation", even though these are the fundamental concepts underlying REST (and my forthcoming messaging-oriented variant, ROMA).

Let me try and explain this. Let's say I found a space capsule that had fallen out of the sky, and by deciphering the alien glyphs on its surface, I understood that it contained a 4-dimensional object from the planet Royf. Unfortunately, I couldn't take the object out, because ours is a 3-dimensional world and such an operation is impossible. However, I found that it was possible to get the capsule to "project" a 3-dimensional image of the object it contained, so I could "see" it in a way that made sense to my limited mind. I found that I could also ask for the object to be manipulated in 3-dimensional terms. I knew, of course, that the object itself was 4-dimensional and so my instructions had to be somehow translated into terms that made sense in the 4D world. But I found to my satisfaction that the 3D image that resulted from my instructions was exactly what I wanted.

I realised then that my interactions with the space capsule were RESTian. The 4-dimensional object was the resource, an unseeable thing that I had no way of even comprehending and which was therefore mercifully shielded from my vision. What I could ask for (through a GET) was a 3D "representation" of the object, and this was something I could understand. I could also manipulate the object in several ways. I could show the capsule other 3D objects and say, "Change its shape to resemble this", or "Make its colour more like this", and it would happen! Obviously, the objects I was holding up were not the same as the object inside the capsule. They were representations of what I wanted the object to look like, when I saw it in 3D terms.

That's really what REST is. The only aspect of the resource itself that we can directly deal with is its name, or URI. The actual resource is completely unseen, indeed unseeable.  Everything that is actually seen is a representation, whether it's a representation of what the object is like right now, or a representation of what we want the object to be like. Everything that goes "over the wire" in REST is a representation.

Nerds can readily understand what a 3D projection is

[See also that REST is the very opposite of "Distributed Objects", although some industry personalities continue to insist that REST is DO! (JJ, I'm talking to you.) Distributed Objects tries to bring about an illusion that remote objects are local, allowing you to grasp them using virtual reality gloves. REST tries to bring about a discipline that says even local objects like the one inside the space capsule should be treated as remote, and we shouldn't try to grasp them directly, only deal with them indirectly through representations. Distributed Objects works well when it does and fails horribly when it doesn't. REST always works.]

Hopefully, this should set to REST some of the confusion around resources and representations.

The Case for Locally Unique IDs (LUIDs)

It should be no surprise to regular readers of this blog that I am in love with UUIDs. As I have said before, they are an inexhaustible source of identifiers that are meaningless (not a pejorative term!) and whose generation can be distributed/federated without danger of duplication. As a result, they are an extremely powerful means of providing a uniform and federated identity scheme.

As a SOA-indoctrinated IT practitioner, I am loathe to expose domain-specific entity identifiers to the larger world because such leakage leads to tight coupling and brittle integration. Yet identifiers of "resources" must often be exposed. How do we do this? Even "best practice" guidelines fail to adequately admonish designers against exposing domain-specific identifiers. [e.g., Subbu Allamaraju's otherwise excellent RESTful Web Services Cookbook talks about "opaque" URIs in recipe 4.2 but ends up recommending the use of internal entity identifiers in recipe 3.10 :-(.]

My point is simple. If the 'employee' table in my application's database looks like this

+------+--------------+---------------+-------------+
| id   | first_name   | last_name     | dob         |
+------+--------------+---------------+-------------+
| 1122 | John         | Doe           | 12-Jan-1960 |
+------+--------------+---------------+-------------+
| 3476 | Jane         | Doe           | 08-Sep-1964 |
+------+--------------+---------------+-------------+
| 6529 | Joe          | Bloggs        | 15-Jun-1970 |
+------+--------------+---------------+-------------+

I do not want to be exposing resources that look like these

/employees/1122
/employees/3476
/employees/6529

I don't want to expose my application's local primary keys to the entire world. They may be "meaningless" but they're still coupled to my domain's internal data structures. I need something else.

My standard solution so far has been the magnificent UUID. I add a candidate key column to my table, like so

+------+--------------+---------------+-------------+--------------------------------------+
| id   | first_name   | last_name     | dob         | UUID                                 |
+------+--------------+---------------+-------------+--------------------------------------+
| 1122 | John         | Doe           | 12-Jan-1960 | 4885c205-8248-4e5b-9c45-4d042e7cc992 |
+------+--------------+---------------+-------------+--------------------------------------+
| 3476 | Jane         | Doe           | 08-Sep-1964 | cdbf87dd-93cb-4c53-9c5d-718c596b0a00 |
+------+--------------+---------------+-------------+--------------------------------------+
| 6529 | Joe          | Bloggs        | 15-Jun-1970 | 73feb1bf-e687-4d58-9750-5bf98ca7b9fa |
+------+--------------+---------------+-------------+--------------------------------------+

or I maintain a separate mapping table, like so

+------+--------------------------------------+
| id   | UUID                                 |
+------+--------------------------------------+
| 1122 | 4885c205-8248-4e5b-9c45-4d042e7cc992 |
+------+--------------------------------------+
| 3476 | cdbf87dd-93cb-4c53-9c5d-718c596b0a00 |
+------+--------------------------------------+
| 6529 | 73feb1bf-e687-4d58-9750-5bf98ca7b9fa |
+------+--------------------------------------+

and I expose my resources like so

/employees/4885c205-8248-4e5b-9c45-4d042e7cc992
/employees/cdbf87dd-93cb-4c53-9c5d-718c596b0a00
/employees/73feb1bf-e687-4d58-9750-5bf98ca7b9fa

I've still got unique identifiers, they're guaranteed not to conflict with anything else in time and space, and more importantly, my domain-specific identifiers remain hidden. I can now even change my entire domain model, including identifiers, and still preserve my external contracts. That's SOA.

But the sad fact of the matter is that many legacy systems and packaged software do not readily support UUID datatypes or even char(36) columns for various reasons. I have recently heard of a far-sighted software vendor that has provided for a "Public ID" field in their database tables for this precise reason, i.e., to allow an externally visible identifier to be specified for their entities. But alas, the column is defined to be varchar(20), much too small to hold a UUID.

It occurred to me that there is nothing sacrosanct about a 128-bit UUID (expressed as a 36-character string). It's just that the larger a random number gets, the more remote the probability of conflict with another such random number. 128 bits is a nice, safe length. But smaller lengths also have the same property, only with a lower degree of confidence.

The constraints of vendor packages like the one I described above led me to postulate the concept of the LUID (Locally Unique ID). This is just a string that is smaller than 32 hex digits (a UUID has 32 hex digits and 4 hyphens in-between). I call this a Locally Unique ID because the smaller it gets, the lower the confidence with which we can claim it to be universally unique. But we may still be able to rely on its uniqueness within a local system. If I'm only holding details of a few thousand employees (or even a few million customers) in my database, an LUID may still be expected to provide unique identifiers with a reasonable degree of confidence.

That vendor package definitely cannot hold a UUID such as "2607881a-fec1-4e5d-a7fc-f87527c93e2d" in its "Public ID" field, but a 20-character substring such as "4e5da7fcf87527c93e2d" is definitely possible.

Accordingly, I've written a Java class called LUID with static methods

String randomLUID( int _length ) and
String getLUID( String _uuidString, int _length )

The first generates a random hex string of the desired length (without hyphens). The second chops a regular UUID down to a hex string of the required length, again without hyphens.

You can download the class from here. Just running the compiled class with "java LUID" will result in some test output which should illustrate how it works. Feel free to incorporate it into your own projects, but be warned that there is no warranty ;-).

Of course, there is a limit to how small an LUID can become before it loses its utility, but I'm not going to draw an arbitrary line in the sand over this. The class above represents mechanism, not policy. Application designers need to think about what makes sense in their context. An LUID of the appropriate length can enable them to implement SOA Best Practice by decoupling externally visible resource identifiers from internal entity identifiers (another example of the difference between Pat Helland's Data on the Outside and Data on the Inside) when a standard UUID cannot be used.

Bottomline: If you can use a standard UUID, do so. If you can't, consider using an LUID of the kind I've described. But always hide the specifics of your application domain (which include entity identifiers) when exposing interfaces to the outside. That's non-negotiable if you want to be SOA-compliant.

Update 27/06/2010:
I should perhaps make it clear what my proposal is really about, because judging from a reader comment, I think I may have created the impression that all I want is for entity identifiers to be "meaningless" in order to be opaque. That's actually not what I mean.

To be blunt, I believe that any entity/resource that is to be uniquely identified from the outside needs _two_ identifiers (i.e., two candidate keys). One of them is the "natural" primary key of the entity within the domain application. The other is a new and independent identifier that is required to support the exposure of this entity through a service interface (whether SOAP or REST). There should be no way to derive this new key from the old one. The two keys should be independently unique, and the only way to derive one from the other should be through column-to-column mapping, either within the same table or through a separate mapping table as I showed above.

To repeat what I wrote in the comments section in reply to this reader:

There was a content management system that generated (meaningless) IDs for all documents stored in it, and returned the document ID to a client application that requested storage, as part of a URI. At one stage, it became necessary to move some of the documents (grouped by a certain logical category) to another instance of the CMS, and all the document IDs obviously changed when reloaded onto the other instance. The client application unfortunately had references to the old IDs. Even if we had managed to switch the host name through some smart content-based routing (there was enough metadata to know which set of documents was being referred to), the actual document ids were all mixed up.

If we had instead maintained two sets of IDs and _mapped_ the automatically generated internal ID of each document to a special externally-visible ID and returned the latter to the client, we could have simply changed the mapping table when moving documents to the new CMS instance and the clients would have been insulated from the change. As it turned out, the operations team had to sweat a lot to update references on the calling system's databases also.

I hope it's clear now.

1. Entity only seen within the domain => single primary key is sufficient
2. Entity visible outside the domain => two candidate keys are required, as well as a mapping (not an automated translation) between the two.
3. UUID feasible for the new candidate key => use a UUID
4. UUID not possible for some reason => use a Locally Unique ID or LUID of appropriate length (code included)

Modelling Resources from First Principles

I've been providing architectural advice to a group of colleagues who are building a set of services. Without going into too much detail, they need to uniquely identify some entities. Clients of the services use these identifiers as references when they return to make related queries on these entities. They've proposed using UUIDs as the unique identifiers, and while I liked the idea, I thought it was too simplistic. There was more to the requirement than just unique identifiers.

They're actually dealing with two types of entities - widgets (say) and requests for widgets. These are different because a request can pertain to a set of widgets, so it may be necessary to model them distinctly. The service interfaces dealing with requests and widgets may need to be distinct as well.

Mind you, these services are not going to be REST services. But having been exposed to the RESTian way of thinking, I immediately thought of resource representations for the two types of entities. Rather than plain old UUIDs, I thought there should be a degree of structure around them (but not so much detail as to make the scheme brittle and inflexible).

Something like these, in other words:

http://www.mycompany.com/widgets/4f138ff2-362f-4e35-8f9e-173290fe86d7

http://www.mycompany.com/widget-requests/eabf5bdb-9800-4af5-9ad7-32c3b95fc48a

However, this suggestion proved to be a surprisingly hard sell. The cross-examination was withering.

Why all the extra information?
Why http://?
Why www.mycompany.com?

Why not a simpler scheme like these examples show:

widget:4f138ff2-362f-4e35-8f9e-173290fe86d7

widget-request:eabf5bdb-9800-4af5-9ad7-32c3b95fc48a

I found I had to retrace my steps and work through my reasoning from first principles. In the process, I learnt a great deal about naming.

My initial response was to point my colleagues to points 7 and 8 of "Common REST Mistakes", where we are admonished not to try and invent our own proprietary object identifiers, and not to try for "protocol independence" (i.e., avoid HTTP URIs). But this wasn't too convincing.

I made a bit of progress by getting agreement on the following:

1. It probably made sense to distinguish between widget identifiers and widget-request identifiers, so some sort of prefix to distinguish between them was necessary. UUIDs alone were probably not enough.
2. It also probably made sense to specify the "domain" within which these resources were being identified, so the "mycompany" string probably belonged somewhere as well.

But then, why not just these:

mycompany:widget:4f138ff2-362f-4e35-8f9e-173290fe86d7

mycompany:widget-request:eabf5bdb-9800-4af5-9ad7-32c3b95fc48a

Frankly, I hated this. My point was that such a format, even though "simple", would have to be explained to anyone looking at it. The structure wasn't immediately obvious. Worse, it was ambiguous and could be extended by later designers in ways that violated the original designers' intent. To this, the counterargument was that the knowledge of the format was only required on the server side. To the client, the whole name was just going to be an opaque string, - a reference ID.

I wrestled with this objection for a while. Then I proposed a guiding principle that given a choice between two naming conventions, a universally understood one was preferable to one that we made up ourselves, provided it wasn't unnecessarily complex.

My research led me to the definition of a URN (Universal Resource Name). What I learnt from this was that in order to name something, we first need to specify a "scheme" that then defines what the rest of the name denotes according to the predefined format for that scheme. The name of the scheme is followed by a colon, then the rest of the name is something that can only be interpreted according to the rules specified by that scheme.

In other words, a standard name (URN) looks like this:

<scheme name>:<some scheme-specific format>

A common example is

"http://www.mycompany.com/widgets/4f138ff2-362f-4e35-8f9e-173290fe86d7" (Heh!)

It's important to point out that "http" in the string above does not refer to the HTTP protocol! It's the name of a "scheme". What does this mean?

Well, in the URN "file:///home/ganesh", the string "file" is not a protocol, because more than one protocol may be used to get to the file.

Similarly, in the URN "mailto:ganesh@mycompany.com", the string "mailto" is not a protocol. SMTP is the actual mail protocol.

[For those familiar with XML namespaces, when we say "xmlns='http://www.example.com/schema'", the URN being referred to here is not necessarily a web page that one can point a browser at. It just needs to be a unique string.]

So we're not necessarily modelling our resources as web resources. All that the "http" scheme defines is that after the colon (":"), there is a scheme-specific structure that specifies a few things.

There are two slashes, then there's a dot-separated domain name, then a slash, then a "resource path" which is itself slash-separated. So that's what a URN conforming to the "http" scheme looks like:

"http" (the name of the scheme)
":" (the colon separating the name of the scheme from the scheme specific structure. This is from the basic definition of a URN)
"//" (the "http" scheme just specifies this, OK?)
"www.mycompany.com" (this is the dot-separated domain name)
"/" (this is the first slash that signifies that the domain name is terminated)
"widgets/4f138ff2-362f-4e35-8f9e-173290fe86d7" (all of this is the "resource path" , and internal slashes are possible, as we can see)

So now going back to our guiding principle (using a well-understood format is preferable to rolling our own) as well as the two points on which there was agreement (i.e., that we may need to qualify the resource's UUID with the type of resource as well as the organisational domain), it looks like the "http" scheme of the URN naming standard fits the bill. This is a well-understood way to include both a domain and a resource path to provide some structure around an already unique ID.

I concede that the "www" prefix of the domain could confuse. All we really need to identify the domain is "mycompany.com".

And so, a unique, standards-based and minimal way to name resources in this business domain would be

http://mycompany.com/widgets/4f138ff2-362f-4e35-8f9e-173290fe86d7

http://mycompany.com/widget-requests/eabf5bdb-9800-4af5-9ad7-32c3b95fc48a

Namespace-Time: Why REST and SOAP Composition Models are so Different

There's a mini-controversy going on over the importance of hyperlinks, and whether they're only usable by humans. Jonathan Marsh seems to think that hyperlinks are less useful for machine-to-machine interaction. The REST viewpoint seems to be that services without hyperlinks "dead-end" the Web, hence the absence of solid support for links in SOAP is a further argument against its usefulness.

I have (as usual) a different viewpoint on this altogether. I think SOAP and REST are duals. They both model the Universe of Discourse, but in different ways. With apologies to Einstein, I think the universe being modelled exists in a Namespace-Time continuum. REST lies on the Namespace axis where hyperlinks play (URIs form a namespace). SOAP lies on the Time axis, which is not surprising considering its verb-orientation. That also explains why there is much more emphasis on process orchestration in the SOAP/WS-* world ("Do this, then do that" is time-sequenced). Even within the SOAP/WS-* world, while services are verb-oriented, metadata on services is noun-oriented. No wonder many recent registries for SOAP-based services are based on RESTful APIs (WSO2 Registry and Mule Galaxy).

It would be silly to disparage REST for being silent on "process", just as it would be silly to accuse SOAP of "dead-ending" services from a namespace perspective.

REST resources are composable in Namespace (to form content). SOAP services are composable in Time (to form process). Decide how you want to model your Universe, then take your pick.