Does Redis Undermine a Key REST Tenet?

While I have always admired the elegance of REST's resource model and its standardised verbs and status codes, there are two things about the RESTful style that I think are drawbacks when building complex applications. One is its client/server model rather than a peer-to-peer model, which I think would have been more generic and useful, especially when unsolicited notifications need to be supported. The second, which is what I want to talk about now, is the insistence on statelessness (or alternatively forcing anything stateful to be either managed by the client or modelled as a resource). This is an acceptable approach for application domain-specific elements, but there is also a domain-neutral class of session-stateful elements that are useful for providing Qualities of Service (such as a sliding window of message sequence numbers for reliability, or session keys for security). These are common requirements that any application may require. This kind of session state is fundamentally different from domain-specific state, e.g., a shopping cart in an e-commerce application which would not be relevant to any other type of application.

The lack of support for this application domain-neutral subset of session state within the RESTful style of design costs us the ability to provide Qualities of Service in a uniform way (which the much-disdained WS-* is able to do using standards like WS-SecureConversation/WS-Security and WS-ReliableMessaging). [Many will point out that RESTful applications can use SSL for security, but the SSL protocol is not itself RESTful, and it breaks the cacheability and routability of REST operations.] So Qualities of Service end up becoming an application responsibility in RESTful applications, which has always struck me as a bit of a cop-out on the part of the protocol.

REST has standardised many other aspects of interaction and turned them into a uniform protocol that interacting systems can use. I would like to see QoS-related session management also similarly standardised and "pulled into the protocol", even if that protocol is not HTTP but something else (WebSockets?).

I have often wondered exactly why statefulness is considered "evil". There are two reasons, as far as I can tell - scalability and failure recovery. Session state occupies memory, so holding session state for multiple clients impacts the scalability of a server. Also, holding session state within a server makes it less easy for clients to switch to another if the one holding its session objects goes down. We'll need to have session-aware clusters rather than stateless farms of servers to provide failover, which again doesn't scale linearly.

Two considerations make these arguments weaker.

The first is that we are not talking about application-specific session state like shopping carts and the like. We're talking about relatively tiny data elements that are domain-neutral and dedicated to providing Qualities of Service, such as sliding windows of sequence numbers for reliability, and security tokens for message encryption. These aren't huge, perhaps a few hundred bytes at most.

The second is that the advent of NoSQL databases has given us a way to delegate the storage of session state. It's now becoming recommended best practice to store session state in a NoSQL database like Redis rather than in memory. Delegating session storage is an alternative to session-aware clusters, since the servers can now be a simpler stateless farm and access the NoSQL database for shared session state. What's impressive about Redis in particular is that the complexity of its GET and SET operations is claimed to be of Order(1), i.e., constant time. This means that (in theory at least) one can increase the number of servers using a Redis datastore indefinitely with no impact on performance.

Now, a horizontally scalable farm of servers can share a Redis datastore and use it to hold the individual sliding windows of message sequence numbers for a number of clients, and also their security tokens. Scalability is no longer a problem, because the data elements are small and the storage/retrieval operations are of constant-time complexity. Failover is also not a problem since the servers themselves are stateless and switchable at run-time.

It would be good if a standard and application domain-agnostic mechanism could be evolved to provide security and reliability to REST-based interactions, using Redis-style scalable session storage. This may be the long-awaited successor to REST, which Rohit Khare's ARRESTED style attempted to do (but which has no practical implementation to date).

That will then leave client/server as my only bugbear with the RESTful style. We'll need another extension to handle that.

Roy Fielding's Fundamental Omission

I think there is a fundamental omission at the heart of the argument that leads to the REST architectural style. It doesn't invalidate the style itself, but it does lead one to question REST's pretensions to being "the only way".

To understand this omission, it's important to study Roy Fielding's doctoral thesis, because that is the Bible of REST. The architectural principles of the Web were enunciated here for the first time, as an architecture. Before this, for many people, the Web just was. After Fielding's work, the Web began to be seen as something consciously designed and built according to well thought-out principles.

In Chapter 5, Fielding develops the REST model systematically and step-by-step, starting with a "null style" architecture with no constraints and then layering constraints upon it one by one in a controlled and logical manner. Indeed, this seems to be one of the core RESTian philosophies, that "constraints empower". With each set of restrictions that you place on an architecture, you actually get new benefits. When you're finished with all the constraints (capabilities) you need, the theory goes, you end up with the ideal architecture.

Indeed, on first walking through the analysis, the REST architecture seems to evolve naturally, inexorably and inevitably. With each new constraint (e.g., statelessness, caching, a uniform interface), the model moves closer to REST. The diagram towards the end of section 5.1 (which I'll call "All Roads Lead to REST" :-) reinforces this impression. No matter in what order we choose to layer our constraints (capabilities), we will always end up with REST.

Diagram: "All Roads Lead to REST"

At this point, I will invite my readers to pause and pop over to this page in Fielding's thesis to see for themselves if they're convinced, or if they think there's something fishy there. Don't read my post any further until you've formed your own opinion.

My fundamental issue with Fielding's argument is how he leaps straight from the null style architecture to the client-server model. When I first read this, I went, "Whoa, not so fast!" Because isn't there something between the null style and the client-server style? How about peer-to-peer?

I would argue that peer-to-peer is a more basic model than client-server, because we can always layer an additional constraint on the peer-to-peer model to convert it into client-server. The constraint is simply that only one peer can initiate an interaction and the other peer can only respond. In other words, request-response semantics.

Now here's my second objection to the REST philosophy. We don't have to glorify all constraints as unqualified positives for an architecture. Sometimes constraints are nothing more than just irritating, chafing restrictions. What does the request-response constraint buy us except the immediate loss of event notification capability (including callbacks)? (Using firewalls as a justification to ban unsolicited communications is circular reasoning. Firewall behaviour is informed by the web's request-response model.)

Besides, even if request-response wasn't a crippling limitation, client-server doesn't model the way things work in the real world. The real world is not client-server, it's peer-to-peer. Between organisations, there is a relationship of equals ("peers"). No organisation likes to see itself as controlled by another. All organisations are proudly autonomous. We need a peer-to-peer model to govern interactions between autonomous organisations, because that most closely models reality.

Even within a corporate setting, the various departments and product systems behave like autonomous units. My employer's HR system isn't sitting there passively, just waiting to answer queries and accept updates to employee data. It must actively remind me to fill in my weekly timesheet and remind my boss that it's time for my annual performance appraisal. But with a web architecture, that's needlessly hard to implement. With only browser-based interfaces to the HR system, we require users to log in to the system before they can receive any events. And that's just on login, or when they actually do something. They can't receive event notifications when they're just sitting there staring at the screen. So we resort to polling the system through page refreshes, or we go right around the limitations of the web architecture and get the HR system to send us e-mails instead. (Or we use AJAX clients, which hide the polling. AJAX is clever lipstick on the request-response pig.)

[I know that at this point, many REST advocates will point out that REST is not about one-way client-server. Each node can be both a client and a server. But two pairs of client-server nodes facing in opposite directions are not the same as a simple peer-to-peer system. That's trying to put Humpty-Dumpty together again.]

So now go back and read Roy Fielding's thesis again. Why, pray why, did Roy curse us with client-server with one perfunctory stroke of his pen, when he could have spent a few minutes exploring the peer-to-peer model? Mind you, there are enough constraints built into the peer-to-peer model when you assume autonomous peers. In fact, the autonomous peer-to-peer model itself can lead to the specification of important concepts like high cohesion (within a peer's domain) and loose coupling (between peers), the messaging paradigm, stateless interactions, discovery, logical addressing (which in turn leads to late binding, substitutability, routability, proxyability), metadata (generic policies as well as domain models), etc. Proxyability in turn leads to the notion of intermediaries (which can do caching, encryption, authentication and a whole heap of domain-aware functions as well). And then there's the whole hierarchy of capabilities that comes out of defining context and refining it:

Context -> Correlation -> Coordination -> Transactions

My, my, so many of the concepts we're wrestling with today, and we haven't had to move a step away from peer-to-peer! It appears that we can build a complete architectural style for SOA based on messaging between autonomous peers without taking on the needless additional constraints of client-server (which, as we argued before, buys us nothing but a loss of event notification capability). What was Roy thinking?

I can't help having mischievous thoughts at this stage. If Roy Fielding's analysis had been more rigorous, and he had properly explored the peer-to-peer model instead of jumping straight into the client-server model, would he have been known today as the father of SOAP messaging?