Resources Are Four-Dimensional

The term ROA (Resource-Oriented Architecture) is misleading. It should ideally stand for "Resource/Representation-Oriented Architecture", even though that's quite a mouthful.

I've found in my discussions with people who work on REST-based systems that lots of them are very fuzzy about the notions of "resource" and "representation", even though these are the fundamental concepts underlying REST (and my forthcoming messaging-oriented variant, ROMA).

Let me try and explain this. Let's say I found a space capsule that had fallen out of the sky, and by deciphering the alien glyphs on its surface, I understood that it contained a 4-dimensional object from the planet Royf. Unfortunately, I couldn't take the object out, because ours is a 3-dimensional world and such an operation is impossible. However, I found that it was possible to get the capsule to "project" a 3-dimensional image of the object it contained, so I could "see" it in a way that made sense to my limited mind. I found that I could also ask for the object to be manipulated in 3-dimensional terms. I knew, of course, that the object itself was 4-dimensional and so my instructions had to be somehow translated into terms that made sense in the 4D world. But I found to my satisfaction that the 3D image that resulted from my instructions was exactly what I wanted.

I realised then that my interactions with the space capsule were RESTian. The 4-dimensional object was the resource, an unseeable thing that I had no way of even comprehending and which was therefore mercifully shielded from my vision. What I could ask for (through a GET) was a 3D "representation" of the object, and this was something I could understand. I could also manipulate the object in several ways. I could show the capsule other 3D objects and say, "Change its shape to resemble this", or "Make its colour more like this", and it would happen! Obviously, the objects I was holding up were not the same as the object inside the capsule. They were representations of what I wanted the object to look like, when I saw it in 3D terms.

That's really what REST is. The only aspect of the resource itself that we can directly deal with is its name, or URI. The actual resource is completely unseen, indeed unseeable.  Everything that is actually seen is a representation, whether it's a representation of what the object is like right now, or a representation of what we want the object to be like. Everything that goes "over the wire" in REST is a representation.

Nerds can readily understand what a 3D projection is

[See also that REST is the very opposite of "Distributed Objects", although some industry personalities continue to insist that REST is DO! (JJ, I'm talking to you.) Distributed Objects tries to bring about an illusion that remote objects are local, allowing you to grasp them using virtual reality gloves. REST tries to bring about a discipline that says even local objects like the one inside the space capsule should be treated as remote, and we shouldn't try to grasp them directly, only deal with them indirectly through representations. Distributed Objects works well when it does and fails horribly when it doesn't. REST always works.]

Hopefully, this should set to REST some of the confusion around resources and representations.

"What Are The Drawbacks Of REST?"

It seems the season for me to post comments in response to provocative topics on LinkedIn. 

A few days ago, Pradeep Bhat posed the question, "What Are The Drawbacks Of REST?" on the REST Architects LinkedIn Group. As before, I had an opinion on this too, which I reproduce below:

"I wouldn't say REST has "drawbacks" as such. It does what it says on the tin, and does that very well. But remember that the only implementation of the REST architecture uses the HTTP protocol. We can surely think of a future RESTian implementation that uses another transport protocol, and that is where some improvements could be made. 

1. HTTP is a synchronous, request/response protocol. This means the protocol does not inherently support server-initiated notifications (peer-to-peer), which are often required. That's why callbacks in RESTian applications require the use of application-level design patterns like Webhooks. Now that we have a bidirectional transport protocol in the form of WebSockets, perhaps the industry should be looking at layering a new application protocol on top of it that follows RESTian principles. 

2. The much-reviled WS-* suite of protocols has at least one very elegant feature. These are all end-to-end protocols layered on top of the core SOAP+WS-Addressing "messaging" capability. They resemble the TCP stack in that the basic protocol is IP, which only knows how to route packets. SOAP messages with WS-Addressing headers are analogous to IP packets. In the TCP world, end-to-end reliability is implemented through TCP over IP, and the SOAP world's analogy is WS-ReliableMessaging headers in SOAP messages. In the TCP stack, IPSec is the end-to-end security protocol (not TLS, which is point-to-point). The SOAP equivalent is WS-SecureConversation. Such Qualities of Service (QoS - reliability, security, transactions) can be specified by policy declaration (WS-PolicyFramework) and SOAP endpoints can apply them like an "aspect" to regular SOAP traffic. 

The REST world has nothing like this. Yes, an argument could be made that idempotence at the application level is a better form of reliability than automated timeouts and retries at the transport level. Similarly, we could argue that an application-level Try-Confirm/Cancel pattern is better than distributed transactions. But what remains is security. WS-SecureConversation with WS-Security is routable, unlike SSL/TLS, which is the only security mechanism in REST. With WS-Sec*, messages can also be partially encrypted, leaving some content in the clear to aid in content-based routing or switching. This is something REST does not have an elegant equivalent for. SSL is point-to-point, cannot be inspected by proxies and violates RESTian principles. It is just tolerated. 

The reason behind REST's inability to support such QoS in general is that all of these require *conversation state* to be maintained. Statefulness has known drawbacks (i.e., impacts to scalability and failure recovery), but with the advent of NoSQL datastores like Redis that claim constant-time, i.e., O(1), performance, it may be possible to delegate conversation state from memory to this datastore and thereby support shared sessions for multiple nodes for the purposes of QoS alone. I don't mean to use this for application-level session objects like shopping carts. If nodes can routinely use shared NoSQL datastores to maintain sessions, then the argument against statefulness weakens, and Qualities of Service can be more readily supported *as part of the protocol*. In RESTian terms, we can have a "uniform interface" for QoS.

3. While REST postulates a "limited" set of verbs, HTTP's verbs are too few! 

POST (add to a resource collection), PUT (replace in toto), PATCH (partially update), DELETE (remove from accessibility) and GET. These are actually not sufficient and they are frequently overloaded, resulting in ambiguity. 

I would postulate a more finely-defined set of verbs if defining a RESTian application protocol over a new peer-to-peer transport: 

INCLUDE (add to a resource collection and return a server-determined URI), PLACE (add to a resource collection with client-specified URI), REPLACE (in toto), FORCE (PLACE or REPLACE), AMEND (partial update, a container verb specifying one or more other verbs to specify operations on a resource subset), MERGE (populate parts of the resource with the supplied representation), RETIRE (a better word than DELETE) and SOLICIT (a GET replacement that is also a container verb, to tell the responding peer what to do to the initiator's own resource(s), because this is a peer-to-peer world now). Think of GET as a SOLICIT-POST to understand the peer-to-peer model. We also need a verb of last resort, a catch-all verb, APPLY, which caters to conditions not covered by any of the others. 

4. HTTP combines application-level and transport-level status codes (e.g., 304 Not Modified and 400 Bad Request vs 407 Proxy Authentication Required and 502 Bad Gateway). The next implementation of REST on another transport should design for a cleaner separation between the application protocol and the transport protocol. HTTP does double-duty and the results are often a trifle inelegant. 

So that's what I think could be done as an improvement to REST-over-HTTP. Apply the principles (which are very good) to a more capable peer-to-peer transport protocol, and design the combination more elegantly."

I'm in the process of writing an Internet Draft for a new application protocol that can be bound to any transport (Pub/Sub, Asynchronous Point-to-Point or Synchronous Request/Response). The protocol is part of a new distributed computing architecture that I call ROMA (Resource/Representation-Oriented Messaging Architecture) and covers not just the data model and message API but also higher levels (QoS, description and process coordination). It's been 5 years in the making and has reached 170 pages so far. It may take another couple of months to get to a reviewable state. Stay tuned.

How to Implement An Atomic "Get And Set" Operation In REST

This question came up yesterday at work, and it's probably a common requirement.

You need to retrieve the value of a record (if it exists), or else create it with a default value. An example would be when you're mapping identifiers between an external domain and your own. If the external domain is passing in a reference to an existing entity in your domain, you need to look up the local identifier for that entity. If the entity doesn't yet exist in your domain, you need to create (i.e., auto-provision) it and insert a record in the mapping table associating the two identifiers. The two operations have to be atomic because you can't allow two processes to both check for the existence of the mapping record, find out it doesn't exist, then create two new entity instances. Only one of the processes should win the race.

(Let's ignore for a moment the possibility that you can rely on a uniqueness constraint in a relational database to prevent this situation from occurring. We're talking about a general pattern here.)

Normally, you would be tempted to create an atomic operation called "Get or Create". But if this is to be a RESTian service operation, there is no verb that combines the effects of GET and POST, nor would it be advisable to invent one, because it would in effect be a GET with side-effects - never a good idea.

One solution is as follows (and there could be others):

Step 1:

GET /records/{external-id}

If a record exists, you receive a "200 OK" status and the mapping record containing the internal ID.

Body:

{

  "external-id" :  ...

  "internal-id" :  ...

}

If the record does not exist, you get a "404 Not found" and a one-time URI in the "Location" header.

Location: /newrecords/84c5d65a-2198-42eb-8537-b16f58733791

(The server will also use the header "Cache-control: no-cache" to ensure that intermediate proxies do not cache this time-sensitive response but defer to the origin server on every request.)

Step 2 (Required only if you receive a "404 Not found"):

2a) Generate an internal ID.

2b) Create a new entity with this internal ID and also create a mapping record that associates this internal ID with the external ID passed in. This can be done with a single POST to the one-time URI.

POST /newrecords/84c5d65a-2198-42eb-8537-b16f58733791

Body:

{

  "external-id" :  ...

  "internal-id" :  ... (what you just generated)
  "other-entity-attributes" : ...

}

The implementation of the POST will create a new local entity instance as well as insert a new record in the mapping table - in one atomic operation (which is easy enough to ensure on the server side).

If you win the race, you receive a "201 Created" and the mapping record as a confirmation.

Body:

{

  "external-id" :  ...

  "internal-id" :  ... (what you generated)

}

If you lose the race, you receive a "409 Conflict" and the mapping record that was created by the previous (successful) process.

Body:

{

  "external-id" :  ...

  "internal-id" :  ... (what the winning process generated)

}

Either way, the local system now has an entity instance with a local (internal) identifier, and a mapping from the external domain's identifier to this one. Subsequent GETs will return this mapping along with a "200 OK". The operation is guaranteeably consistent, without having to rely on an atomic "Get or Create" verb.

One could quibble that a GET that fails to retrieve a representation of a resource does have a side-effect - the creation of a one-time URI with the value "84c5d65a-2198-42eb-8537-b16f58733791" being inserted somewhere. This is strictly true, but the operation is idempotent, which mitigates its impact. The next process to do an unsuccessful GET on the same value must receive the same one-time URI.

It's a bit of work on the server side, but it results in an elegant RESTian solution.

Building RESTful applications using the WSO2 platform

Someone taking a casual look at the WSO2 middleware platform might be forgiven for thinking this is exclusively about SOAP and WS-*. But there is in fact strong support for building RESTful applications with this platform using the JAX-RS framework library, and Prabath Siriwardena (one of WSO2's experts on Identity Management) has blogged about the recommended component architecture to achieve this. There is also a WSO2 workshop on this topic conducted by Asanka Abeysinghe, who has many years of experience on the customer side of the fence and understands both the vendor (technology) and customer perspectives. This workshop is on Dec 8th and should be worth attending for people living close to Palo Alto.

An interesting sidelight: In Practical SOA for the Solution Architect, I re-introduce the practitioner to SOA principles by talking about three core technology components - the Service Container, the Broker and the Process Coordinator. According to this view of SOA, a service can be exposed through any of these components and a service consumer will be none the wiser as to the nature of its implementation. In other words, the Practical SOA approach does not make any prescriptions about which component should be the consumer-facing one. All of them are equally valid candidates, and the only criterion for choosing one over the other is the nature of the service enablement mechanism (bespoke, brokered or orchestrated).

In Prabath's architecture diagram below, the reader will notice that runtime clients must all consume services through the ESB (the Broker), even though these services are hosted on the App Server (the Service Container). Why can't services be exposed directly from the App Server where they are hosted?

Prabath explains that using a Broker (ESB) instance as the front-end for services is recommended practice because the ESB can provide security features like authentication and authorisation, as well as throttling capabilities to guard against Denial of Service (DoS) attacks.

So in the real world, we may often need to front-end Service Containers and Process Coordinators with an instance of the Broker that is dedicated to providing these security and traffic shaping features. This could (and should!) be a different instance of the Broker from those used to mediate access to backend systems. Such an architecture will work well because ESBs are better deployed in a federated topology than in a centralised hub-and-spokes fashion. [The unnatural hub-and-spokes topology for ESB deployment, which the high cost of most commercial ESBs forces on customer organisations, then results in a performance bottleneck and a single point of failure. Fortunately, the more favourable economics of WSO2's Commercial Open Source model makes it feasible for customer organisations to implement a more flexible federated architecture for the ESB.]

I Hate HatEoAS

For something that's supposed to be THE defining characteristic of REST, it could have done with better naming.

I would have been happy with the term HatEoAS if it had stood for "Hypermedia as the Envelope of Application State" rather than "Hypermedia as the Engine of Application State".

An Engine actively drives things. E.g., A process engine is well named, because it drives a process.

A constraint doesn't drive anything. It constrains. It provides an envelope around the range of possibilities.

And so they really should have called this an envelope rather than an engine of application state.

There, I've said it. Because the expansion of HatEoAS has been driving me up the wall.

Fortunately, it's being referred to as "Hypermedia Constraint" now, which is both more elegant and more accurate.

Does Redis Undermine a Key REST Tenet?

While I have always admired the elegance of REST's resource model and its standardised verbs and status codes, there are two things about the RESTful style that I think are drawbacks when building complex applications. One is its client/server model rather than a peer-to-peer model, which I think would have been more generic and useful, especially when unsolicited notifications need to be supported. The second, which is what I want to talk about now, is the insistence on statelessness (or alternatively forcing anything stateful to be either managed by the client or modelled as a resource). This is an acceptable approach for application domain-specific elements, but there is also a domain-neutral class of session-stateful elements that are useful for providing Qualities of Service (such as a sliding window of message sequence numbers for reliability, or session keys for security). These are common requirements that any application may require. This kind of session state is fundamentally different from domain-specific state, e.g., a shopping cart in an e-commerce application which would not be relevant to any other type of application.

The lack of support for this application domain-neutral subset of session state within the RESTful style of design costs us the ability to provide Qualities of Service in a uniform way (which the much-disdained WS-* is able to do using standards like WS-SecureConversation/WS-Security and WS-ReliableMessaging). [Many will point out that RESTful applications can use SSL for security, but the SSL protocol is not itself RESTful, and it breaks the cacheability and routability of REST operations.] So Qualities of Service end up becoming an application responsibility in RESTful applications, which has always struck me as a bit of a cop-out on the part of the protocol.

REST has standardised many other aspects of interaction and turned them into a uniform protocol that interacting systems can use. I would like to see QoS-related session management also similarly standardised and "pulled into the protocol", even if that protocol is not HTTP but something else (WebSockets?).

I have often wondered exactly why statefulness is considered "evil". There are two reasons, as far as I can tell - scalability and failure recovery. Session state occupies memory, so holding session state for multiple clients impacts the scalability of a server. Also, holding session state within a server makes it less easy for clients to switch to another if the one holding its session objects goes down. We'll need to have session-aware clusters rather than stateless farms of servers to provide failover, which again doesn't scale linearly.

Two considerations make these arguments weaker.

The first is that we are not talking about application-specific session state like shopping carts and the like. We're talking about relatively tiny data elements that are domain-neutral and dedicated to providing Qualities of Service, such as sliding windows of sequence numbers for reliability, and security tokens for message encryption. These aren't huge, perhaps a few hundred bytes at most.

The second is that the advent of NoSQL databases has given us a way to delegate the storage of session state. It's now becoming recommended best practice to store session state in a NoSQL database like Redis rather than in memory. Delegating session storage is an alternative to session-aware clusters, since the servers can now be a simpler stateless farm and access the NoSQL database for shared session state. What's impressive about Redis in particular is that the complexity of its GET and SET operations is claimed to be of Order(1), i.e., constant time. This means that (in theory at least) one can increase the number of servers using a Redis datastore indefinitely with no impact on performance.

Now, a horizontally scalable farm of servers can share a Redis datastore and use it to hold the individual sliding windows of message sequence numbers for a number of clients, and also their security tokens. Scalability is no longer a problem, because the data elements are small and the storage/retrieval operations are of constant-time complexity. Failover is also not a problem since the servers themselves are stateless and switchable at run-time.

It would be good if a standard and application domain-agnostic mechanism could be evolved to provide security and reliability to REST-based interactions, using Redis-style scalable session storage. This may be the long-awaited successor to REST, which Rohit Khare's ARRESTED style attempted to do (but which has no practical implementation to date).

That will then leave client/server as my only bugbear with the RESTful style. We'll need another extension to handle that.

REST and the Servlet 3 Specification

I've been going through the Java Servlet 3 specification, and I just came across this gem at the start of the chapter on Sessions:

The Hypertext Transfer Protocol (HTTP) is by design a stateless protocol. To build effective Web applications, it is imperative that requests from a particular client be associated with each other. [...] This specification defines a simple HttpSession interface that allows a servlet container to use any of several approaches to track a user’s session [...]

I don't think the spec authors have been adequately exposed to the REST philosophy, or they wouldn't be talking so casually about how "imperative" sessions are to build "effective" Web applications. A few years ago, I would have read this without batting an eyelid. Now, I had to keep myself from falling off my chair in shock. One would think spec writers of advanced technology would know a bit better. At the very least, they could have written something like this:

The Hypertext Transfer Protocol (HTTP) is by design a stateless protocol, and it is strongly recommended that Web applications be built in a stateless manner to be effective, with all state management delegated to the persistence tier. If, for legacy or other reasons, it is unavoidable to maintain in-memory state in a web application, the servlet specification defines a simple HttpSession interface that provides a relatively painless way to manage it. Application developers should however be aware of the severe scalability and recoverability issues that will accompany the use of this feature.

There! Now I feel much better.

Does REST Need Versioning?

In my ongoing conversations with JJ Dubray, he has often made the point that "REST couples identity and access together in a terrible way". When pressed to explain, he provided the following example.

Assume that there is a Resource identified by "/customers/1234". Updating the state of this customer requires a PUT. JJ asks how REST can handle a change to the business logic implied by the PUT.

Since we cannot say

   PUTv2 /customers/1234

implying a change to the logic of PUT, he believes we have no option but to say

   PUT /customers/v2/1234

but this is different from the identity of the customer, which remains at

   /customers/1234

Hence REST "couples identity with access".

Well, I disagree. First of all, it's a mistake to think there are only two places where the version of business logic can be exposed - the Verb and the Resource. The data submitted is an implicit third, which I'll come to in a moment. But this example only makes me question the whole basis for versioning.

Does REST need versioning? For that matter, does any service need versioning? What is versioning in the context of SOA?

I would say service versioning is a mechanism that allows us to simultaneously maintain two or more sets of business logic in a consumer-visible way.

Why does it have to be consumer-visible as opposed to just consumer-specific? After all, if the service implementation can distinguish between two classes of consumer, it can apply two different business rules to them in a completely opaque manner. The consumer doesn't even have to know that two (or more) different sets of business rules are being weighed and applied under the covers.

Let's ask the more interesting question: Why do we need to maintain two or more sets of business logic simultaneously? The interesting (and circular) answer is often that business logic happens to be consumer-visible, hence a new version of business logic also needs to be distinguished from the old in a consumer-visible way. This is often stated as the need to support legacy consumers, i.e., consumers dependent in some way upon the previous version of business logic. But why do we have to support legacy consumers? Because existing contracts break when services are silently upgraded.

This argument leads to an interesting train of thought. Perhaps the answer lies in the opposite direction to what JJ believes, i.e., not versioning of services but abstraction of detail. Are our service contracts too specific and therefore too brittle? Service versioning is perhaps a "smell" that says we are going about SOA all wrong. Let us see.

I want to take up a more real-world example than the customer access example that JJ talked about. After all, that's more of a "data service" than a business service. Let's look at a real "business service".

Let's take the case of the insurance industry where a customer asks for a quote for an insurance product. The client-side application has to submit a set of data to the service and get a quote (a dollar value for the premium) in return.

In REST, here's how it could work.

Request:

   POST    /quotes
   <insurable-details>
   ...
   </insurable-details>

Response:

   201 Created
   Location: /quotes/06fb633b-fec4-4fb6-ae32-f298b8f499c1

The client is referred to the location of the newly-created quote Resource, which is at /quotes/06fb633b-fec4-4fb6-ae32-f298b8f499c1. When the client does a GET on this URI, the quote details are transferred.

So far, so good. Now let's say the business logic changes. Premiums are now calculated using very different logic. The first question is, can this new business logic be applied to all customers, or do we need to keep track of "old" customers and keep applying the old business logic to them? If we can "upgrade" all customers to the new business logic, there is, of course, no problem at all. The interface remains the same. The client application POSTs data to the same URI, and they are redirected in the same way to the location of the newly-created quote Resource. The business logic applied is all new, but customers don't see the change in their interface (only in the dollar values they are quoted!)

However, if we do need to maintain two sets of business logic, it could be for three reasons. One, the data that the client app needs to submit has changed, so the change is unavoidably visible to the customer and has to be communicated as a new and distinct contract. Two, there is another business reason to tell two types of customers apart, perhaps to reward longstanding customers with better rates, and this difference between customers is not obvious from the data they submit. Third, the client app somehow "knows" the behaviour of the old version and is dependent on it. In this case, we need a new version just to keep legacy clients from breaking.

We can readily see that the third reason is an artificial case for versioning. It's in fact a case to break implicit dependencies that have crept in.

In contrast, the first and second reasons provide their own resolution. If the type of data submitted by the client changes, that is itself a way to distinguish new clients from old ones and apply different business logic to them. In other words, we only need to tell newer customers about the change in the data they need to POST. Older customers don't need to do a thing. Also, if we can somehow derive that the customer is an existing one, even if this is not explicit in the data submitted, we can still apply different business logic transparently.

JJ may consider this a messy and unstructured approach to versioning. Business stakeholders may have the opposite view. It's less disruptive. The less clients are exposed to the way services are implemented, the better.

Service versions are not really an interface detail. They're an implementation detail that often leaks into the interface.

That means version numbers are a problem, not a solution.

None of these arguments may satisfy someone like JJ. In that case, if service versioning is absolutely essential, there is a simple way to include it, after all. Include the version number in the message body accompanying a POST or PUT request. In fact, message bodies are allowed even for GET and DELETE requests (anything except a TRACE), so versioning of any type of service is possible. REST does not enforce versioning, (that would be a bad thing considering that versions are often a smell), but doesn't impede it either.

With this approach, neither Verbs (e.g., POST) nor URIs (e.g., /quotes) are affected by versions and the "terrible" coupling of identity and access is avoided.

It seems to me that the problem is not with REST, it's with looking at REST through WS-* eyes.

Is REST Another Variant of Distributed Objects?

In my discussions with JJ on REST, he's often made the observation that REST is nothing but Distributed Objects and is therefore a bad style. But is it really?

I have two different arguments why I believe it isn't.

1. Let me first talk about my experience as a migrant from India to Australia many years ago. Working in Indian companies had acclimatised me to a rather direct management style. If my manager ever wanted me to do something, he or she would say so in so many words, "Do this!"

When I arrived in Australia and began working, I was struck by the very different style of Australian managers. I would hear things like, "You may want to do this," or "You may like to do this." It took me a while to realise that they were essentially saying the same thing, only less directly. Let me coin a term for this style, because I'm going to use it to explain a concept with Distributed Objects. Let me call this a polite command.

Let's now turn to methods that set an object's attributes. We may see setter methods like these:

 widget.setSomeAttribute( someValue );
 widget.setAnotherAttribute( anotherValue );

Now consider another style of doing the same thing.

 widget.updateSelf( widgetDTO );

where widgetDTO is a structure that holds the new values of someAttribute and anotherAttribute.

Let's call the direct setter methods commands. "Remoting" these commands leads to a tightly-coupled, RPC mechanism. This is the Distributed Objects style. I would be the first to agree with JJ that this is a bad approach.

But the second style is a polite command. It's requesting the object to update itself based on values held in a Data Transfer Object (i.e., a representation). Now this is a style that can be remoted without problems, because it's not really RPC.

The REST style of updating resources follows the latter model.

PUT   /widgets/1234
<attributes>
<some-attribute>some value</some-attribute>
<another-attribute>another value</another-attribute>
</attributes>

In other words, this is a polite command. It can be safely remoted. It's not Distributed Objects.

2. JJ laughs at the approach of annotating object methods to turn them into REST resources, the way JAX-RS does. This is another reason why he considers REST to be Distributed Objects. The annotations seem to be doing nothing more than remoting object methods. Therefore, REST = Distributed Objects and consequently a horrible way to design systems.

Not so fast.

Let's not forget the concept of Service Objects, which are not really Domain Objects.

Let's look at a simplistic domain model for a banking system. The major Domain Object in this model is an Account. An Account object has the following methods:

class Account()
{
  public double getBalance() {...};
  public void credit( double amount ) {...};
  public void debit( double amount ) {...};
}

The Domain Model is internally-focused. Nothing here understands the concept of a transfer. Indeed, a transfer cannot be elegantly modelled using domain objects.

That's because a transfer is an aspect of a Service. Service verbs are really free-standing verbs. They don't belong to classes the way domain methods do. The methods getBalance(), credit() and debit() don't make any sense by themselves. It always has to be account.getBalance(), account.credit() and account.debit().

In contrast, transfer() can be a free-standing verb. In fact, it seems downright clumsy to push transfer() into a class, because it really doesn't belong inside any class. It's just that languages like Java are unyieldingly object-oriented and don't tolerate free-standing verbs. So in practice, designers create an awkwardly-named class like AccountService and stick transfer() inside it, like this:

class AccountService()
{
   public void transfer( Account fromAccount, Account toAccount, double amount )
   throws InsufficientFundsException
   {
       if ( fromAccount.getBalance() >= amount )
       {
           fromAccount.debit( amount );
           toAccount.credit( amount );
       }
       else
       {
           throw new InsufficientFundsException();
       }
   }
}

[If J2EE designers feel a sense of déjà vu on seeing this, it's the old Session Façade pattern all over again, with Stateless Session Beans acting as a service façade for domain objects represented by Entity Beans. Stateless Session Beans were not part of the Domain Model at all. They were an explicit service layer that lent itself to being remoted through (what else?) a Remote Interface.]

Now, if we tried to remote a method like credit() or debit(), it would be a classic case of RPC (RMI, strictly speaking) and therefore Distributed Objects hell.

But a "method" like transfer() readily lends itself to being remoted! That's because an instance of AccountService isn't a Domain Object but a Service Object.

If designers took care to annotate only the methods of Service Objects and avoided doing so with methods of Domain Objects, then they neatly avoid being trapped into the Distributed Objects paradigm.

All that leads to a certain insight. And that is that although REST is an architectural style, it isn't prescriptive enough. We need to tell designers what types of domain objects can be modelled as resources and what cannot.

With a nod to the term polite command, perhaps it's not enough for systems to be RESTful. They should also be RESPECTful :-).

I realise I have blogged extensively on this over 2 years ago, here and here.

The Real and Imagined Limitations of REST

One of the things that struck me about my discussion with JJ Dubray on a previous blog posting was how closely we agreed on fundamental architectural principles (decoupling of interface from implementation, avoiding a hub-and-spokes architecture, etc.), yet how diametrically opposed our views were on REST.

For example, I think REST does a great job of decoupling interface from implementation. JJ feels the exact opposite. Why?

Analysing the problem more closely, I guess the common examples of RESTful interface design are partly to blame.

1. A URI like

http://myapp.mycompany.com/customers/1234

where 1234 is the actual id of the customer in my database, would be a horrible example of tight coupling, in my opinion. I believe URIs should be opaque and meaningless. I think designers should take care to create a mapping layer that decouples their implementation data model from their externally exposed service data model.

For example, I would prefer this to be exposed:

http://myapp.mycompany.com/customers/cb77380-7425-11df-93f2-0800200c9a66

There should be a mapping within the service implementation that relates the opaque identifier "cb77380-7425-11df-93f2-0800200c9a66" to the customer's id within the domain data model, i.e., 1234. That would be true decoupling.

Mind you, the earlier example of tight coupling is not a limitation of REST, merely bad application design.

[Incidentally, I think UUIDs have given the world a wonderful gift in the form of an inexhaustible, federated, opaque identification scheme for resource representations. Decoupling identity is a key part of decoupling domain data models (Pat Helland's Data on the Inside) from service data models (Data on the Outside).]

2. Another bad example is the use of "meaningful" URIs. Even though the following two URIs may seem obviously related, client applications must not make any assumptions about their relationship:

http://myapp.mycompany.com/customers
http://myapp.mycompany.com/customers/1234

In other words, a client application must not assume that it can derive the resource representation for the set of customers by merely stripping off the last token "/1234" from the URI of an individual customer.

And this is not a limitation of REST either. The HatEoAS (Hypermedia as the Engine of Application State) principle says that client applications must only rely on fully-formed URIs provided by the service to perform further manipulations of resources. In other words, URIs are to be treated as opaque and client applications must not attempt to reverse-engineer them by making assumptions about their structure.

Examples like these are bad from the perspective of architects who understand the evils of tight coupling, and the ones who produce these examples understand it too, but these naive examples have the benefit of being easy to understand.

A RESTful system would work perfectly well with URIs that looked like these:

http://myapp.mycompany.com/51bb2db7-b51c-4fd9-b235-2b310a644a84
http://myapp.mycompany.com/a7ac1f23-6bc7-44be-af9b-4b68fd31879f
http://myapp.mycompany.com/208657ef-1e86-47ba-9c5f-3f177c018293

The first is a URI for a customer, the second is a URI for a bank account and the third is a URI for an insurance policy. How about that? This would be architecturally clean, but most REST newbies would go, "Huh? Weird!" and turn away with a shudder.

Sometimes, the desire for understandability of a design by humans introduces semantic coupling that is anti-SOA. I guess there's a trade-off that we need to be aware of, but it's not a limitation of REST itself. It's an application designer's choice.

3. In another context, JJ has expressed his opinion that SOA is about asynchronous peer-to-peer interaction, and I completely agree. Where I think he has misunderstood REST is in its superficial characteristic of synchronous request-response. There are several well-known design patterns for asynchronous interaction using REST, so in practice, this is not a limitation at all. The HTTP status code of "202 Accepted" is meant for precisely this condition - "I've received your message but have not yet acted on it". At one stroke, it's also a model for reliable message delivery. Combine a POST with a one-time token to ensure idempotence, then keep (blindly) trying until you get a 202 Accepted response. Voila! Reliable message delivery.

I find I am able to look beyond the superficial characteristics of REST that seem like showstoppers to JJ. I can see simple workarounds and Best Practice guidelines that can make a RESTful application fully compliant with SOA principles. JJ stops at the characteristics as given and concludes that REST is horribly antithetical to SOA.

SOAP, REST and the "Uniform Interface"

When REST folk compare their approach with SOAP-based Web Services (which they often mistakenly call "SOA", not realising that REST is an equally valid way to do SOA), they often refer to a "uniform interface" that REST provides that SOAP does not.

The response of the SOAP crowd (when there is one), is "What's a uniform interface?" (They'd be surprised to hear that they have one too, which is the topic of this post.)

The uniform interface refers to a standard way to do something regardless of what the specific activity is.

As an example, if I asked what the SOAP interface would be like for a service, and I refuse to say what the actual business function is, the SOAP people would have to stop after describing the service at a very high level. There's a SOAP message with a header and a body, and what goes into them depends entirely on what I want to do. The general semantics of a SOAP message are "Process this". That's actually too general to be useful. Any further detail they could give me would depend on what I tell them about the actual operation I'm trying to perform.

In contrast, the REST guys can actually tell me much more about what my service is going to look like based only on the rough contours of my business function.

They can tell me that what I operate on is a resource that will look like this:

"http://domain/resource"

They can tell me that I will be using one of four verbs (GET, POST, PUT or DELETE).

They can tell me that my response status codes will be one of some 50-odd pre-defined status codes.

Plus they can give me other general tips that will apply, such as a GET operation will have no side-effects, a deferred processing request will return a "202 Accepted" status when successful, a POST request will be to a resource collection, and its response status code will be "201 Created" when successful, and the URI of the newly created resource will be found in a response header called "Location:", etc., etc.

That's actually quite a bit of information considering I haven't said anything yet about the business function I'm trying to implement.

That's what is called a "uniform interface". It's a structured framework within which application designers have to work. Far from being restrictive and limiting, these constraints actually make things easier for the designer because they provide standard patterns that can be reused in similar situations and deliver predictable results.

So far, so good. What the REST side doesn't understand is that SOAP-based Web Services offer a uniform interface too. Obviously not for the application itself, but for qualities of service.

Ask a SOAP service designer how they plan to implement reliable messaging, and they can show you very detailed WS-ReliableExchange headers that are virtually the same regardless of the business service. Ask them about security (encryption and signing/authentication) and they can show you the detailed WS-Security/SecureConversation/Trust headers required, and these too are standard and unchanging.

Security and Reliability are "Qualities of Services". SOAP-based Web Services have standardised these, and are also on track to standardising Transactions through WS-AtomicTransaction and WS-BusinessActivity (the latter deals with real-world transactions that require compensating actions to "roll back"). [Some may argue that these have in fact been standardised, but there isn't yet a WS-I Basic XXX Profile for these, like there is for Security and Reliability, which means there is no benchmark yet for interoperability in these areas.]

This isn't something to be sneezed at, because a uniform interface leads to improved interoperability, which in turn reduces the cost of integration. From my own experience in IT, I know that integration cost forms a very high proportion of the cost of most projects. Indeed, Project Tango proved the interoperability of Java and .NET-based Web Services using declarative statements in policy files to have these QoS headers automatically generated. Now that's cool.

REST promises to reduce integration cost and thereby overall cost, mainly though the mechanism of the "uniform interface". However, REST does not have a standard model for Security or Reliability, let alone Transactions or Process Coordination. SSL is a rigid security mechanism that is adequate for most, but not all situations. And while it's true that Reliability is probably better implemented through patterns like Idempotence than through the TCP-like model followed by WS-ReliableExchange, it requires an application designer to consciously build it. Transactions and Process Coordination are also missing from the REST lineup.

That's the challenge. Both SOAP and REST have standardised a part of the service interface, but not all of it. REST has standardised the interface of the application's core logic. SOAP has standardised the interface for Qualities of Service. What we need is a model that provides us with both.

Heavyweight Services Have Let IT Practitioners Down

Call it coincidence, but I had two meetings in the same day when different IT folk complained that a services approach was too cumbersome for them and argued for their applications to be able to connect directly to the databases of other applications to perform queries, joins, etc.

My involuntary shudder at this suggestion betrayed my SOA leanings, of course. I don't think these people realised how much future pain they would expose the organisation to by building implicit dependencies ("tight coupling") between their independent applications. But I could also empathise with their frustration.

The first generation of services in organisations has been well-meaning in intent, but expensive in practical terms. People ranted to me about the sheer effort overheads involved in trying to access data from elsewhere - setting up queues and queue clients, formalising XML messages to drop onto queues and pick from queues, parsing and generating XML, etc., etc., - all to do something that was essentially a SQL select or a join across two tables (theirs and another app's). I also heard a fair bit about the performance overheads of calling services to do simple things - operations that take seconds to do because they involve 2 or more separate service calls. "Decoupling at what cost?" was the refrain.

I'm forced to the realisation that our collective enthusiasm for SOA, while entirely correct and justified, has not provided a uniformly satisfactory solution to everyday IT practitioners. Bluntly put, we've let down the average developer by making their job unnecessarily harder, to the extent that even experienced designers who know the benefits of loose coupling are beginning to argue for a return to the Bad Old Ways.

A reader's comment on my previous post has made me think that "Data Services", especially of the RESTian CRUD variety, could be the answer. We still have a service interface that decouples applications from each other and their gory data structures and implementation details, but now we can set them up with minimal effort and call them with minimal overhead. Data Services could be the SOA SQL (That's a pretty apt pun, actually, if you pronounce SQL as "sequel").

More and more, I'm leaning to the view that most technical problems within an enterprise can be solved with "a web server, a database and some brains". This is a rich topic for a future post or even a whitepaper, but the unsurprising insight is that the third ingredient is probably the one in short supply at most IT shops ;-). (No insult intended, just that designers and architects tend not to step back and look for solutions from first principles, but rely on precedent and product-based approaches.)

Advice To Organisations Embarking On SOA Today

I have been involved with SOA in various roles over the last two to three years, and my thinking has evolved a fair bit over this period. If I was asked to advise an organisation embarking on a major SOA initiative today, I would probably say this to them:

1. The End Goal: Remember that SOA is not about integration but about inherent interoperability. Think health, not medication. SOA is about raising the capability of your systems such that they can easily and inexpensively integrate with others, not about introducing a new, slick technology that will connect your systems together more easily. Simplifying the components of your enterprise and making them easy to understand and connect to will give you SOA. So keep simplicity at the back of your mind all the time, and don't confuse it with expediency, which is the path of least resistance. Simplification could take some effort.

2. Domain Models: Don't waste too much time searching for "the" canonical data model. Most off-the-shelf ones are too high-level and abstract to be useful. And building your own comprehensive dictionary is wasteful and time-consuming. Instead, identify logical owners of different elements of data and let them own the data dictionary for those elements. All services exposed out of these logical domains should use these definitions and it is the responsibility of service consumers from other domains to understand them. Processes that combine services crossing such domains should perform their own mapping between similar data elements. It's necessarily messy and plenty of out-of-band communication will be required at design time. After all, even similarly-named elements may suffer from subtle interpretation errors, so manual discussion and clarification will always be part of a service or process design.

This is not as bad as it sounds because only a subset of data elements managed by a domain is exposed through its service interface, and it's only these that may need translation to the context of their consumers. Don't look to do away with this manual effort through a single canonical data model. That's a wild goose chase, so don't even start.

3. Infrastructure and Connectivity: Try and avoid using message queues unless you're looking at low latency situations like a trading floor, or if there's simply no other way to interface to a legacy system. The queuing paradigm introduces various coordination issues into application design, and implementing message queues requires establishing more complex patterns to solve these gratuitous problems. [I have a larger, philosophical argument about the need to innovate an application protocol on top of an asynchronous, peer-to-peer transport, but let's not confuse the current set of recommendations with that idea.]

In today's world, HTTP-based communication patterns backed up by databases will often do the trick more simply than expensive message queues. Look beyond the apparent need for reliable message delivery. Often, an idempotent operation will suffice to meet the real requirement, and this is quite a standard pattern to implement. Often, queues are used in synchronous (blocking) patterns anyway (to avoid the coordination problems I talked about), so nothing is being gained in an architectural sense by the use of queues. And even asynchronous communications, where required, can be implemented in standard ways over HTTP, so HTTP is quite a universal protocol to use as the logical infrastructural element for your SOA.

ESBs, Service Directories and other "governance" components are often only required to manage the complexity that they themselves introduce. It's amazing what you can achieve with a farm of simple web servers and a database, and still keep things simple and understandable.

4. Service Enablement: Try and avoid the entire SOAP/WS-* stack of technologies if you can. There is a significant complexity overhead associated with this set of technologies, and you will need an expensive toolset to (partially) simplify its use. Look seriously at REST instead. Even though REST advocates don't make the case strongly enough (and sometimes see SOA as an antithetical philosophy), REST is in fact a valid way to do SOA and can usually help to deliver solutions at much lower cost and complexity. The hard part about doing REST is finding good people who can think that way. REST is subtly different from the SOAP/WS-* approach, even though they may just look like different kinds of plumbing to move XML documents around (and I confess that's the way I initially sell REST to corporate skeptics brought up on a diet of vendor-provided Web Services propaganda).

5. Data Contract: Consider alternatives to XML for the data contract. Though this sounds like heresy, XML is heavyweight and cumbersome, and XML manipulation tools in high-level languages (with the possible exception of E4X in JavaScript) are clumsy to use and suffer major impedance mismatches. You will spend more time wrestling with XML than on the service itself. Although many in the web world will immediately recommend JSON, raw JSON is not sufficient to ensure data integrity, because it has hitherto lacked a strong schema definition capability. Maintain a watching brief on the JSON Schema proposal, submitted for approval as an IETF standard. Already, there are JSON Schema libraries in many high-level languages such as Java. It should be possible to define data contracts with as much rigour as with XML, but at a much lower level of complexity. A newer and more compact JSON Schema representation called Orderly is also maturing, which makes this approach simple as well as easy.

So instead of going down the XML rabbit-hole, start with JSON anyway, and incorporate JSON Schema/Orderly as it matures. You will find this works especially well in combination with REST. A quick Proof-of-Concept may convince the skeptics in your organisation (although the opposite result may also occur, with many going away convinced by the speed of this approach that it's either simplistic or too good to be true!)

6. Web Service Implementation: If you're trapped by circumstances into an XML-and-SOAP/WS-* approach, look at the WSO2 suite of commercially-supported Open Source products. Especially look at the WSO2 Mashup Server. Don't be fooled by the name. It's more than just a mashup server. It's a service orchestration engine that (curiously) uses server-side JavaScript as its programming language. The major advantage of JavaScript is the ability to use the E4X library to perform extremely straightforward XML manipulation. Once you use E4X, you will never go back to JAXB or any other XML-processing library. WSO2 Mashup Server allows SOAP or REST services to be consumed, combined and orchestrated, and in turn exposes SOAP or REST services. It's a good way to hedge your bets if you're only half-convinced about REST. The WSO2 suite is also much less expensive than its proprietary rivals, although the real expense is in the heavyweight approach that it unfortunately shares with them.

7. The Paradox: SOA is really all about simplicity, but it's hard to find SOA architects who seek to simplify systems. Conventional SOA practice seems to be about making integration complex through heavyweight approaches, then introducing tools to manage that complexity, tools that require specialist skills to use properly. If done the conventional way as most SOA consultants seem to agree, your SOA initiative will only leave you with additional complexity to manage.

Of course, if you're politically inclined, you will bask in the prestige of a hefty budget and a large team, and can declare victory anyway on the basis of the number of services and processes you have delivered. But if you want to be really successful at delivering SOA (i.e., making your business more agile and able to operate on a sustainably lower cost basis) while keeping your burn rate low along that journey, you would do well to look at boring, unimpressive and even anticlimactic approaches and technologies such as the ones I've listed above. Give the big vendors a wide berth. You don't need to buy technology (beyond the web servers and databases you already have). You certainly don't need to buy complex technology, which is what the vendors all want to sell you.

And don't let the lack of grandeur in this approach worry you. Complexity impresses the novice, but results are what ultimately impress all.

Postscript: Vendor REST is coming. Beware.

REST is Polymorphic CRUD

I've been trying to evangelise REST a bit lately and have had mixed success. There is cautious interest. But there are also huge conceptual hurdles to be overcome. Pete Lacey said it best about enterprise IT folk when it comes to REST: They Can't Hear You.

One architect looked at the definition of a service interface I proposed and thought it a bit "bland". Perhaps it just needed a big WSDL file, lots of XML and SOAP faults!

Another common reaction to REST when it's presented is that "it's just CRUD", with the implication that it's just too fine-grained to be used to create good business services. I've been struggling to explain that just because REST uses four HTTP verbs that correspond roughly to CRUD operations, it doesn't necessarily mean that REST is a CRUD approach to manage data at a very low and detailed level. The resources on which the verbs operate can be arbitrarily coarse-grained. But what has eluded me so far is a succinct term that can drive the point home.

I think I've finally found it - "Polymorphic CRUD".

IT folk in the enterprise understand both polymorphism and CRUD, so the combined term should make sense. I want to drive home the point that a verb itself is neither coarse- nor fine-grained, it's how each resource interprets it. Fine-grained resources will interpret the REST verbs as CRUD operations. But more coarse-grained resources can interpret the verbs as any arbitrary business operation.

Accompanied by the appropriate payload, POSTing to the resource "/applications" is nothing but submitting an application. There's no need for a specific "submitApplication" method.

I've also realised that one can clear a process inbox by DELETEing a "/pending" resource, with a standard WebDAV status code in response (207 Multi-Status), indicating that different items encountered different status codes during the batch process.

It's the way the verb is interpreted by each resource that gives it its meaning in that context. Therefore the REST approach is to manipulate business objects of arbitrary size and complexity through polymorphic CRUD operations.

I hope that gets people to go "Aha!"

AMQP Lurches Towards Completion

I really mean the word "lurches" that I used in the title of this post. I've been following the AMQP specification since the 0.8 version, and there have been more dramatic twists and turns in this than in an old Perry Mason novel. (In the latest twist, they seem to have dropped the very concept of an Exchange, a mainstay of previous versions).

I don't believe the latest version of the spec is available (I would dearly love to have a read), but the working group met recently in San Diego, and the materials presented are here.

My overall reactions are these:

1. We really do need an industry-standard messaging protocol to match the capabilities of proprietary products like IBM's MQ and TIBCO's EMS. These two vendors have built their hugely profitable businesses at the expense of customers who have nowhere else to go. The commoditisation that has lowered prices in other areas of distributed computing (TCP/IP for networking, HTTP for web, SMTP for email, etc.) has simply not occurred in enterprise messaging. AMQP aims to rectify that. All I can say is, "It's about time".

2. I note with amusement that Microsoft has joined the AMQP bandwagon. Nothing like a lack of market share to elicit good behaviour from Microsoft ;-)

3. As I said, I don't have access to the latest spec, but I know that an intermediate version had dropped support for streaming communications and file transfer. If these are still not part of the spec, I believe that's a huge mistake. An enterprise messaging protocol must natively support these significant aspects of distributed computing, otherwise the protocol as a whole will fail to pass muster in spite of its excellence in other areas.

4. I don't know what the AMQP Working Group is thinking, but I believe that the main competition for AMQP is not IBM and TIBCO but HTTP and REST. Whenever I talk to REST afficionados about AMQP, the response is a big ho-hum. "What can AMQP do that we can't already do with HTTP?" is the response. Asynchronous notification? Just use XMPP for that. Security? SSL is good enough. And so on. The AMQP Working Group should co-opt a particular breed of REST expert - one who understands the value of resource-orientation without being wedded to HTTP. I believe that if we can define an application protocol on top of the transport protocol that AMQP is (by usefully constraining it with special verbs, headers and status codes), we will have a more capable architectural style than REST, one which includes native support for event notification, end-to-end security, reliable message delivery, transactions, file transfer, streaming communications and process coordination. Rohit Khare's ARRESTED style can become a reality.

Is anyone listening?

How SOFEA fits into a SOA Ecosystem

How do user interface front-ends fit into a landscape that is increasingly service-oriented? A brief discussion yesterday with Peter Svensson sparked off a flurry of activity, and this diagram is the result. Thanks, Peter. Thanks also to Justin Meyer for his feedback and suggestions.

I won't waste words here, except to say that this diagram and its accompanying text should be sufficient to show how a SOFEA-based client application fits into a Service-Oriented ecosystem and works seamlessly with services (both SOAP and REST) and with processes (both the orchestrated and the choreographed variety).

[For those who are wondering what SOFEA is, read this gentle introduction. The original detailed paper is here and my original blog posting on it is here. There is also the original ServerSide article, but be warned that it points to an earlier version of the paper. Then there's the InfoQ article co-authored with Peter Svensson, which is not just about SOFEA but a family of similar architectures that rationalise the Presentation Tier. And finally, a Google Groups community that any interested party can join.]

A Rant about GlassFish and Jersey

So you want to play around with Web Services, both SOAP and REST? Maybe you've had your fill of WSO2's Mashup Server, drooled over its strengths (XML processing and service exposure) and understood its limitations (legacy resource connectivity including persistence - no, Data Services didn't cut it for me). Now you want to look at a more conventional, Java-based (as opposed to JavaScript-based) server.

I had attended a Sun Tech Day many months ago when they demonstrated the GlassFish JEE app server and its REST implementation called Jersey. I had also heard a lot about Sun's advanced SOAP/WS-* stack, Project Tango. One of my consultant colleagues (a REST advocate, no less) showed me a Tango demo he had put together and was himself impressed with the policy-driven interoperability that it made possible.

So it was with fond visions of being able to build a Domain Model using Java objects and then expose the domain as services using both SOAP/WS-* and REST paradigms from the same box that I downloaded GlassFish.

The GlassFish Quickstart Guide gave me my first indications of trouble.

Why would you have instructions labelled Step 1 and Step 2 if there's something you need to do before either of them? (Shouldn't that at least be labelled Step 0?)

Steps

1. Add the install-dir/bin/ directory to the PATH environment variable.
2. Start the server by entering this command: asadmin start-domain domain1.

After searching in vain for the "bin" directory under the directory where I had installed (well, unzipped) GlassFish, I found this line above the two steps listed:

Once you install using "ant -f setup.xml", you start the server by starting the domain.

Casually, just in passing. Why they didn't just say

Step 0: Install GlassFish using "ant -f setup.xml"

beats me.

So I finally installed GlassFish and started up a "domain" from the command line. (All right, I'll go with your idiosyncrasies, although why I should know about "domains" and why you couldn't just give me a Linux "startup.sh" script and a Windows "startup.bat" script beats me once again.)

Oh, then I found that Jersey doesn't come bundled with GlassFish. I'd have to get that separately.

No problem, GlassFish has a tool called "Update Center", which lets you download modules like Jersey automatically.

Uh-oh. Update Center doesn't work through proxies. (You idiot, configure the proxy settings in the tool.) I did that. The "doesn't work" comment refers to after I configured my proxy settings in Update Center. And I didn't get a meaningful error message either, so I had no clue how to fix the problem, if indeed the problem was my fault. (These guys are starting to bug me.)

OK, good news, bad news. It appears from someone else's experience that Update Center only shows you older versions of Jersey, not the very latest one, so you can't use it anyway. You'll need to download and install Jersey manually.

All right, so I downloaded Jersey and ran the recommended script

$ ant -f jersey-on-glassfish.xml
-Dgf.home=$HOME/bin/glassfish install

Got the predicted error message, edited "jersey-on-glassfish.xml" and commented out the offending line. (Seriously Sun, even teeny-tiny Open Source projects provide working code. Why can't you?)

Finally got the ant script to work.

Hold on! Turns out there's something more. A lot more. To quote Drop By,

Be aware! This procedures will only install the examples and won’t add auto-magically the jersey jars to Glassfish’s runtime classpath. So
you have to take care by yourself to include the jars to the build target, see the provided examples:

Okay-doke. I ran

ant -f build-impl.xml library-inclusion-in-archive

I got an error with a TaskDef of all things. Some class (I forget what it was now) wasn't found, although it should have been. That's when I finally gave up.

There's simply no excuse for the abysmal state of readiness of GlassFish and Jersey. I know Jersey isn't yet at version 1.0, but GlassFish is at version 2.1, for goodness sake! In contrast, WSO2's Mashup Server worked, and worked pretty well, way back in the days of version 0.2. It was trivial to install and get running, the examples worked, and all my test examples worked as well. There were some problems that I faced, but those were when I had gone well past the basic stuff and started pushing the envelope.

I don't know if I have the time and patience to try GlassFish and/or Jersey again. It's just too hard.

The Deciding Characteristic of SOA - The Viewpoint Flip

I used to tell the curious that SOA's distinguishing characteristic was loose coupling. That definition probably says it all who those who already "get" it, but probably fails to communicate anything meaningful to those who don't already know what it is.

These days, I try and emphasise one particular kind of loose coupling that I think epitomises what SOA is, and I call it The Viewpoint Flip. The Viewpoint Flip is a more tangible concept than "loose coupling", and therefore more practical advice to give to aspiring SOA architects. The Viewpoint Flip is more important than breaking the synchronous request-response paradigm and replacing it with asynchronous messaging - that's transport sugar. The more important decoupling happens in the application domain, and I'll talk about it again in this post, although I've covered it in the past. A recent discussion with a colleague has convinced me that this aspect of SOA is often underappreciated even by SOA practitioners. That's why I'm talking about it once again.

The flip in viewpoint that I'm talking about is from the application domain's own navel-gazing view of itself to the view of an external party who wishes to interact with the domain for their own purposes. It's not for nothing that SOA is called "service-oriented". You can't be service-oriented, even in the English sense of the term, if you're focused on your own view of the world. You need to think about what your customers want. Ironically, object-oriented systems (supposedly the pinnacle of modelling paradigms) may require the most wrenching viewpoint change, because their model is, by design, internally-focused. Older, procedural languages are often more naturally oriented to the viewpoint of an external party.

Two examples, one of which I've touched on before:

Take a drum (the musical instrument, not the vessel that stores water). Model its (State and) Behaviour using the OO paradigm. What methods can you think of that would model the drum's behaviour? If you said "beat()", you're still thinking procedurally. Beating is something you do to the drum, not something that the drum does. What the drum does in response to the beating is make a sound. Therefore "makeSound()" is an appropriate method for the Drum class. That's how you would model the domain. (If you had a Drummer class, then "beat()" could be a legitimate method of that class.)

But "makeSound()" is not a service-oriented verb! It's not what a prospective external user of the drum would do. It's too internally-focused. Funnily enough, "beat()" is a good service-oriented verb. Procedural thinking is alive and well in the SOA world, it would seem.

Another example is the common one of funds transfer between bank accounts. For some reason, this simple example seems to tie everyone up in knots. I've seen examples like this:

fromAccount.transferTo( toAccount, amount )

or alternatively,

toAccount.transferFrom( fromAccount, amount )

Neither of them is aesthetically satisfying. Neither of them models the inherent symmetry of the "transfer funds" operation.

What we want is an unashamedly procedural syntax

transfer( fromAccount, toAccount, amount )

But we're too tied up in OO thinking to be able to do this elegantly. But that, if I may point out, is precisely what we need to be able to say in SOA-speak.

Insight 1: SOA thinking requires a fallback to procedural logic. This is one of the fundamental drivers for the Viewpoint Flip.

[The Domain Model for the banking Account class would probably have methods like "acceptDebit()" and "acceptCredit()". The "transfer" verb would be modelled from the start as a Service, if one goes by the principles of Domain-Driven Design, because "transfer()" (whether to or from) is not part of the behaviour of the Account class. It's this non-Object-Oriented Service that would acquire the two account entities from a repository based on the identifiers passed to it, and then (transactionally) invoke the "acceptDebit()" method of one and the "acceptCredit()" method of the other.]

By the way, REST is beautifully elegant, partly because it achieves the Viewpoint Flip without anyone even noticing. GET, PUT, POST, DELETE - these are all things that you do. They're not verbs that model the behaviour of resources. They're verbs that model what service consumers do to resources. When you adopt the REST model, you are unconsciously forced into service-oriented thinking and away from object-oriented thinking. With SOAP/WS-*, there's a degree of conceptual sophistication that's demanded of the designer, without which the whole thing degenerates into a bunch of remote procedure calls in XML syntax, needlessly complex and brittle beyond belief.

If I may fire my pet arrow at SOAP in the eternal REST-vs-SOAP debate, the approach of many Web Services tool vendors is to generate SOAP interfaces and WSDL files by annotating methods on objects. This is so naive I don't know whether to laugh or cry. There is no annotation in the world that can cause the "makeSound()" method on a Drum class to transform itself into the "beat()" service that should be exposed to external systems. The Viewpoint Flip is a sophisticated conceptual exercise that only a human designer can engage in. If you rely on tooling to expose services from your domain objects, and you end up with a SOAP operation called "makeSound()", that's not SOA, that's a horrible Distributed Objects train wreck. Distributed Objects don't work.

In contrast, so many good things come together thanks to the singular concept of the Viewpoint Flip.

RPC goes out of the window, because we're not doing remote method calls. We're fundamentally transforming the nature of the verb. So it can't be about taking a local method on a class and making it remotely invocable. People have railed against SOAP-RPC for years (and I have as well), but the most damning argument against SOAP-RPC, I now realise, is that it doesn't implement the essential Viewpoint Flip that's required to turn a Domain Model into a Service Model.

Contract-First Design is another approach that emerges naturally out of accepting the need for the Viewpoint Flip. If you can't automatically generate a service interface from methods on objects, then it follows that it must be independently designed. In other words, the Service Interface and the Domain Model are independent, first-class entities. Neither can be generated from the other. They must be mapped to each other.

Did you notice that that's loose coupling? Change either the Domain Model or the Service Interface, and the other doesn't necessarily have to change. You may just have to change the mapping logic between them and hence preserve the other.

Nowadays, when someone asks me what SOA is all about, I don't glibly say "loose coupling." I say very slowly, "It's about being s-e-r-v-i-c-e o-r-i-e-n-t-e-d, being oriented to the viewpoint of the service consumer. SOA thinking means looking at a domain from the outside in, from the perspective of a consumer and what they wish to achieve, rather than how the domain sees itself, i.e., as a set of inter-related objects." [As Kirstan observes, it's also the most natural way in which a business analyst looks at the domain, i.e., as a set of service "steps" that can be composed into a business process.]

A bit of philosophy: I read somewhere that when a student embarks on the study of Zen, the mountains are nothing more than mountains. Partway through the training, the mountains are no longer mountains. But finally, when the student has mastered Zen, the mountains are once again mountains, but they're somehow not the same as what they were.

So let it be with models of the world. From procedural thinking to object-orientation and back again to procedural thinking. But this time around, procedural thinking isn't quite the same thing. It's service-oriented.

REST Eye for the Relational Guy - The "Uniform Interface" Explained

My thanks go to Rhys Frederick for the analogy I'm about to share with you. Just the day after my last post, in which I'd said, "On the 'uniform interface' issue, I'm hearing the words, but I'm not seeing the REST vision", Rhys asked me to remind him of the name for a certain pattern used in relational database design, and casually mentioned that it was analogous to the uniform interface of REST. I did a double-take. A few Google searches later, I finally grokked the REST Uniform Interface concept, thanks to my experience with RDBMS in a past life. Thanks again, Rhys. The coincidences in my life never cease to amaze.

It's funny how many times I've had to change my way of thinking in the last twenty years of my career. I learned programming with BASIC and FORTRAN, complete with that wonderful "feature" called GOTO. I feared GOSUB. I couldn't understand it. GOTO was easy to understand. Then I learned structured programming, and GOSUB suddenly became the most natural construct in the world. GOTO became (cough) a four-letter word. As relational databases became popular, data for me became a set of tables, and with SQL came the shift from procedural to declarative thinking. Then came the difficult shift to object-oriented thinking (although I'm grateful it didn't have to be through the torture of C++!) A combination of PowerBuilder (yes!) and Java got me over that line. The latest change in my thinking has been thanks to SOA. I've now learnt to think "between domains" in addition to peering "inwards" at a domain with pure OO eyes. And while I've been able to fit REST fairly naturally into my generic SOA worldview, I have struggled to see the world through RESTian eyes, the way the REST folk seem to see it.

Until now.

Now at last, one major facet of REST lies exposed before me, and the irony is that I had to fall back to relational thinking in order to understand it.

Let me launch into the analogy. It's variously called (as I learned) the Value-Pairs Pattern, the Vertical Schema and the Object-Attribute-Value Pattern.

Consider some arbitrary domain modelled using an Entity-Relationship Diagram. The model looks something like this:

Note that each entity has an ID (its primary key), some attributes, and also some foreign keys that refer to the IDs of other entities. This is a specialised model, tailored to this particular application. But there may be a common pattern here. If you find that the attributes (names and types) are similar for all the entities, it may suggest a generalisation of the form shown below:

What we've done here is create a separate table (ATTRIBUTE) listing the names of the different attributes that an entity may have. Then we create mapping tables (ENTITYn_ATTRIBUTE) that create an association between each entity and the generic attribute table. Each row here holds the value that an attribute has for a particular entity. The entity tables themselves are now much smaller. They only hold foreign keys. This model is no longer tied so strongly to a particular domain, because we've taken part of what makes it unique to a domain and genericised it. If we next assume that each entity ID is globally unique, we can apply some generic thinking to this aspect of the model as well.

Here's a completely generic schema.

Look at the names of the tables now! They're all completely generic (ENTITY, ENTITY_RELATIONSHIP, ATTRIBUTE, ENTITY_ATTRIBUTE). The new ENTITY_RELATIONSHIP table even takes the foreign key constraints out of the various ENTITYn tables and holds it as an external mapping. Now all the different ENTITYn tables can be collapsed into a single generic one called simply ENTITY. This model is completely generic. We can add and remove entities, attributes and foreign key relationships at will. We don't have to change the schema at all. Contrast this with the specialised model we started with. Every such change there would have required a change to the schema.

What's more, the generic schema is simple and can be universally communicated, even turned into assumed knowledge. Then all you're concerned with when you talk about a particular domain is the data. The schema ceases to be a differentiator. It's universal. With specialised schemas, on the other hand, you need to communicate the schema as well as the data. Now I understand why the REST guys say REST doesn't need a WSDL equivalent. When the schema is generic and well-known (the "uniform interface") why would you need to describe it?

More analogies creep out of the woodwork as I gaze at the model. See the IDs? They're URIs. It was the assumption of global uniqueness that allowed us to mix the IDs of completely different entities in the one table. URIs are globally unique too. Nobody is going to confuse http://something.com/customers/1234 with http://something.com/orders/1234.

See the foreign keys? They're hyperlinks between entities.

And because we're assuming a limited number of attributes (but not restricting it to a particular number), it vaguely corresponds to REST's limited set of verbs.

URIs, hyperlinks, a limited set of verbs - there's the relational analogy for REST!

Message to the REST folk: I finally get your model :-).

</euphoria>

There is something, though, that we lose by going down the generic path, and that is understandability. I can understand a domain pretty well by studying a specialised Entity-Relationship Diagram, but a generic schema does not speak to me at all. If all domains look the same, what makes them unique? Infinite plasticity makes me very uncomfortable. I can't "see" the domain from the data. I need a schema. When we talk about automated reasoning, perhaps schemas are more important than ever...?

To turn an old REST argument around, constraints empower. If I make my schemas more rigid, I gain something - obviousness of purpose. So the vertical schema pattern isn't a constraint, it's the absence of one. Anything will fit into it, and so it describes nothing.

I must say I'm enjoying this journey. Each day brings fresh insights, and fresh questions...

Decoupling Interface from Implementation - Is REST Really Better Than SOAP/WS-*?

(New diagrams - download from here)

In a previous post, I tried to argue that the REST and SOAP/WS-* models were duals, and that they could be seen as two views into the same domain. Mark Baker disagreed, and his point centred around the decoupling of interface from implementation.

Now, as an architect, I am rather sensitive to issues of tight coupling, and have often railed against examples of this, such as the SOAP-RPC style itself and the generation of WSDL files from Java implementation classes. But Mark goes much further. He would like to change the implementation of a service from a Stock Quote service to a Weather Report, and he would like to see his interface unchanged! To my mind, this goes beyond the reasonable. It's like changing from pasta to soup, and expecting to continue using a fork.

The decoupling I expect is far more modest, and dare I say, more of the kind expected to be encountered in practice? A change in an algorithm, an extra parameter or two, changes to qualities of service or a change of business domain owner. All of these except the second (extra parameters), I would expect to support without breaking the service interface. For the second, I would expect to be able to create a new version of the service while supporting the older version in parallel for a time until deprecated.

I don't know if it's a Zen thing. On the "uniform interface" issue, I'm hearing the words, but I'm not seeing the REST vision. Maybe one day it'll come to me in a blinding flash, and I'll be a RESTafarian myself, but for now, I'm in the "both models are valid" camp.

So with my current state of understanding, I've drawn a new set of diagrams which shows how one can implement services using the SOAP/WS-* and REST models, and the degree of decoupling between the Service Contract (the interface) and the Domain Model (the implementation) in each case. I see no difference in concept between the two approaches. In fact, the two are extremely similar. I would agree with Mark on one point, though. The SOAP/WS-* world today does not practise the kind of decoupling I describe in this diagram. I guess I'm proposing what I think to be "best practice" in this space. Let's see what responses I get.