Though I like REST and consider it a very elegant model for SOA, it's a little tiresome to hear day in and day out that it's so much more elegant than the SOAP-based Web Services model. In fact, I'm getting so tired of this shrill posturing that I'm going to stick it to the RESTafarians right now, in their own style. Watch.
REST exploits the features of that most scalable application there is - the web, right? REST doesn't fight the features of the web, it works with them, right? HTTP is not just a transport protocol, it's an application protocol, right? URIs are the most natural way to identify resources, HTTP's standard verbs are the most natural way to expose what can be done with resources, and hyperlinks are the most natural way to tie all these together to expose the whole system as a State Machine, right??
Great. I'd like to introduce you to something far older, more basic and more extensible than your web with its URIs, standard verbs and hyperlinks. You may have heard of the Internet. That's right, The Internet. Do you consider the Internet to be scalable, resilient, supportive of innovation? Yes? Good.
This may sound familiar, but the trick, dear friends, is not to fight the Internet model but to work with it. And what is this Internet model? In a sentence, it's about message passing between nodes using a network that is smart enough to provide resilience, but nothing more. This means that every higher-order piece of logic must be implemented by the nodes themselves. In other words, the Internet philosophy is "Dumb network, smart endpoints" (remembering of course, that the network isn't really "dumb", just deliberately constrained in its smartness to do nothing more than resilient packet routing). And the protocol used by the Internet for packet routing is, of course, Internet Protocol (IP).
Voila! We've decentralised innovation! Need a new capability? Just create your own end-to-end protocol and communicate between nodes (endpoints) using IP packets to carry your message payload. You don't need to consult or negotiate with anyone "in the middle". There is no middle. The Internet is not middleware, it's endpointware. (Thanks to Jim Webber for that term :-)
What's TCP? A connection-oriented protocol that ensures reliable delivery of packets in the right order. Did the Internet's designers put extra smarts into the network to provide this higher quality of service? Heck no, they designed an end-to-end protocol and called it TCP. What does TCP look like on the wire? Dunno - they're just IP packets whizzing around like always. TCP packets are wrapped inside IP packets. TCP is interpreted at the endpoints. That's why the networking software on computers is called the TCP stack. Each computer connected to an IP network is an endpoint, identified by nothing more than - that's right - an IP address. Processing of what's inside an IP message takes place at these endpoints. The Internet (that is, the IP network) is completely unaware of concepts such as sockets and port numbers. Those are concepts created and used by a protocol above it. The extensibility and layered architecture of the Internet enable the operation of a more sophisticated networking abstraction without in any way tampering with the fundamental resilient packet-routing capability of the core Internet.
Time for a more dramatic example. What's IPSec? An encrypted end-to-end channel between two nodes that may be routed through any number of intermediary nodes. Wow, that must have required an overhaul to the network, right? Nope, they just created another end-to-end protocol called ESP (Encapsulating Security Payload) and endpoints that understood it. Then they "layered" it between TCP and IP. Can they do that? Of course! The Internet's architecture doesn't in any way force TCP to sit directly atop IP. The "next protocol" attribute of every packet in the "TCP/IP suite" allows virtually any protocol to be layered on top of any other. So TCP packets get wrapped inside ESP packets that get wrapped inside IP packets. At the receiving endpoint, when the IP envelope is examined, there's a little note there to say that the "next protocol" is ESP, not TCP. So an ESP processing layer is given the payload of the IP packet. When ESP is through with its decryption, it hands its payload to the "next protocol", TCP. And life goes on. Any higher-level protocol that runs above TCP will be none the wiser. Is this cool or what? And with all this smart processing going on at the endpoints, what do we see on the wire? Just IP packets whizzing around as usual.
Since you RESTafarians are so forthcoming about how the Web is superior to other forms of distributed computing, let me tell you a little bit about how the Internet is superior to other distributed computing platforms. Look at the old Telco network, called POTS (Plain Old Telephone System), the philosophical antithesis of the Internet - dumb endpoints, smart network. The telephone handsets provided by the Telcos are traditionally dumb, just dialtone-capable. All the smarts are in the network (the "cloud"). All the messagebanks, the call waiting, call forwarding, teleconferencing, everything is taken care of by the smart network. The Telcos claim that the model is least disruptive because they can upgrade capability without having to upgrade millions of handsets.
Is that so? Then how come the Internet has beaten the Telco network even in telephony? Made an overseas call using Skype lately? How much did it cost you? Did you use a webcam? Do the Telcos offer anything equivalent? I rest my case.
The Internet, with its "smart endpoints, dumb network" approach, has comprehensively beaten the Telco network with its "smart network, dumb endpoints" philosophy. It's a fact in plain view that cannot be denied. When innovation is decentralised, it flourishes. The Internet is a platform for decentralised innovation. (Heck, the very web that you RESTafarians wave in people's faces is an example of the innovation that the Internet enables. HTTP is an end-to-end protocol, remember?) You don't fight such a model. You work with it.
Now what does all this have to do with SOAP and Web Services? Everything.
Remember that when we talk about SOAP today, we're talking about SOAP messaging, not SOAP-RPC. Forget that SOAP-RPC ever existed. It was something invented by evil axe murderers and decent SOA practitioners had nothing to do with it ;-).
And another subtle point to remember is that when we say "SOAP message", we always mean "SOAP message with WS-Addressing headers". Why is this important? Because a SOAP message with WS-Addressing headers is an independently-routable unit, just like that other independently-routable unit, the IP packet. Begin to see the parallels?
Now imagine a messaging infrastructure that knows how to route SOAP messages and nothing more. We will deliberately stult the IQ of this infrastructure and prevent it from growing beyond this capability. How do we innovate and build up higher levels of capability? Why, follow the Internet model, of course. Create protocols for each of them and embed all their messages into the SOAP message itself (specifically into the SOAP headers). There's no need to layer them as in the TCP stack, because reliable delivery, security, transactions, etc., are all orthogonal concerns. They can all co-exist at the same level within the SOAP header block - headers like WS-ReliableMessaging, WS-Trust, WS-SecureConversation, WS-Security, WS-AtomicTransaction, WS-BusinessActivity, etc.
Now we have the plumbing required for a loosely-coupled service ecosystem. This isn't by any means the totality of what we require for SOA. At the very least, there's still the domain layer that sits on top of all this elegant message-oriented plumbing. So now define document contracts using an XML schema definition language and a vocabulary of your choice (for Banking, Insurance or Airlines), stick those conforming XML documents into the body of the SOAP messages we've been talking about, and ensure that your message producers and consumers have dependencies only on those document contracts. Ta-da! Loosely coupled components! Service-Oriented Architecture! (Note that just one language - XML - is being used to define both a core application wire protocol (SOAP/WS-*) and one or more application domain protocols (the document payloads). What does 'X' stand for in XML again?)
Now all we need is a bit of metadata to describe and guide access to these services (WSDL and WS-*Policy - some might say SSDL) and a recursive technique to create composite services (WS-BPEL - some might say SSDL again) and we're more or less done.
Sounds good, right? In fact, it sounds more than just good. This is revolutionary! But why haven't we seen this wonderful vision unfold?
A reality check. (And this is no apology or reason for RESTafarians to crow. SOAP-based Web Services technology is not a tired model about to be pushed aside by a simpler and more elegant rival. It's a sleeping tiger that is now stirring.) Why we don't enjoy the brave new world of SOAP-based Web Services today is because:
(1) Those axe murderers we talked about earlier misled the world with SOAP-RPC for a few years, and SOA practitioners are still being detoxified. (There's a school of thought that says the use of WSDL implies RPC even today, but that has more to do with the entrenched (and entirely unjustified) exposure of domain object methods as "services". See my later post on the Viewpoint Flip which is my recommended technique to service-enable a domain model. At a syntactic level, the wrapped document/literal style also distances SOAP usage from blatant RPC encoding.)
(2) Another group of petty criminals talked up SOAP's transport-independence but only wrote HTTP bindings for it, needlessly making the SOAP visionaries look silly. (I think when AMQP arrives - in its fittingly asynchronous way 20-odd years after TCP and IP, we'll begin to see TCP and AMQP as the synchronous and asynchronous transports that synchronous and asynchronous SOAP message exchange patterns should most naturally bind to. There's also the view that the sync/async dichotomy is artificial, and it is correlation which is key. Be that as it may, I don't see a need for SOAP to bind to HTTP at all, except for the unrelated constraint described next.)
(3) There's the pesky firewall issue that causes weak people to surrender to HTTP expediency, but I think that's another thing we will sort out in time. I think of SOAP as standing for "SOA Protocol". It's an application protocol in its own right, and it sits above infrastructure protocols like TCP, UDP and the coming AMQP. We need to define a port for SOAP itself, and get the firewalls to open up that port. No more flying HTTP Airlines just because we like the Port 80 decor. We are SOAPerman and we can ourselves fly ;-).
(4) Vendors of centralised message broker software have been pretending to embrace the SOAP/WS-* vision ("We're on the standards committees!") but in reality they're still peddling middleware, not endpointware. There's a whole flurry of vendor-driven "marketectures" intended purely to protect their centralised broker cash cows from being slaughtered by a slew (pun intended) of cheap, decentralised SOAP-capable nodes. Another round of detoxification needs to happen before people begin to understand that ESB is not an expensive broker you plonk in the middle of your network but a set of capabilities that magically becomes available when you have fully WS-* capable SOAP endpoints. This understanding will dawn, and soon.
(5) As long as commercial licensing of SOA products remains in vogue, deployments will tend to be centralised and will fail to exploit the fundamental Internet-style architecture of SOAP/WS-*. Who wants to pay for multiple instances of software when a single one will do? SOAP and WS-* have only made endpointware architecturally feasible. Open Source implementations will make it economically viable. Three projects to watch: ServiceMix, Mule and WSO2.
(6) The WS-* standards that define security and reliable messaging only recently arrived, so implementations are still being rolled out. But hey, it's here now. Take a look at any Tango-.NET interop demo and you'll be wowed by the policy-driven magic that happens.
I've put together a couple of diagrams that explain these basic concepts. This one illustrates the parallels between the SOAP/WS-* stack and the TCP/IP stack. This one shows my classification of the various SOA approaches in vogue today, with my biases clearly visible :-).
In short, I think there are only two valid approaches to building Service-Oriented Architectures today - (1) REST and (2) SOAP-messaging-using-Smart-Endpoints (and I hate both the SOAP-RPC axe murderers and the Centralised ESB marketing sharks for forcing me to use such a long qualifier to explain exactly what I mean).
And if all this didn't finally turn out to be the poke in the snoot of the RESTafarians that I intended, that's because I like REST. I just don't accept that REST is in any way superior to SOAP-messaging-using-Smart-Endpoints. As I hope I've explained in this lo-o-ong post, the latter's based on an equally elegant model - the Internet itself. As we address the essentially implementation-related deficiencies of the technology, SOAP-messaging-using-Smart-Endpoints will be seen as an equally elegant and lightweight approach to SOA.
Most importantly perhaps, the REST crowd will finally shut up.
REST exploits the features of that most scalable application there is - the web, right? REST doesn't fight the features of the web, it works with them, right? HTTP is not just a transport protocol, it's an application protocol, right? URIs are the most natural way to identify resources, HTTP's standard verbs are the most natural way to expose what can be done with resources, and hyperlinks are the most natural way to tie all these together to expose the whole system as a State Machine, right??
Great. I'd like to introduce you to something far older, more basic and more extensible than your web with its URIs, standard verbs and hyperlinks. You may have heard of the Internet. That's right, The Internet. Do you consider the Internet to be scalable, resilient, supportive of innovation? Yes? Good.
This may sound familiar, but the trick, dear friends, is not to fight the Internet model but to work with it. And what is this Internet model? In a sentence, it's about message passing between nodes using a network that is smart enough to provide resilience, but nothing more. This means that every higher-order piece of logic must be implemented by the nodes themselves. In other words, the Internet philosophy is "Dumb network, smart endpoints" (remembering of course, that the network isn't really "dumb", just deliberately constrained in its smartness to do nothing more than resilient packet routing). And the protocol used by the Internet for packet routing is, of course, Internet Protocol (IP).
Voila! We've decentralised innovation! Need a new capability? Just create your own end-to-end protocol and communicate between nodes (endpoints) using IP packets to carry your message payload. You don't need to consult or negotiate with anyone "in the middle". There is no middle. The Internet is not middleware, it's endpointware. (Thanks to Jim Webber for that term :-)
What's TCP? A connection-oriented protocol that ensures reliable delivery of packets in the right order. Did the Internet's designers put extra smarts into the network to provide this higher quality of service? Heck no, they designed an end-to-end protocol and called it TCP. What does TCP look like on the wire? Dunno - they're just IP packets whizzing around like always. TCP packets are wrapped inside IP packets. TCP is interpreted at the endpoints. That's why the networking software on computers is called the TCP stack. Each computer connected to an IP network is an endpoint, identified by nothing more than - that's right - an IP address. Processing of what's inside an IP message takes place at these endpoints. The Internet (that is, the IP network) is completely unaware of concepts such as sockets and port numbers. Those are concepts created and used by a protocol above it. The extensibility and layered architecture of the Internet enable the operation of a more sophisticated networking abstraction without in any way tampering with the fundamental resilient packet-routing capability of the core Internet.
Time for a more dramatic example. What's IPSec? An encrypted end-to-end channel between two nodes that may be routed through any number of intermediary nodes. Wow, that must have required an overhaul to the network, right? Nope, they just created another end-to-end protocol called ESP (Encapsulating Security Payload) and endpoints that understood it. Then they "layered" it between TCP and IP. Can they do that? Of course! The Internet's architecture doesn't in any way force TCP to sit directly atop IP. The "next protocol" attribute of every packet in the "TCP/IP suite" allows virtually any protocol to be layered on top of any other. So TCP packets get wrapped inside ESP packets that get wrapped inside IP packets. At the receiving endpoint, when the IP envelope is examined, there's a little note there to say that the "next protocol" is ESP, not TCP. So an ESP processing layer is given the payload of the IP packet. When ESP is through with its decryption, it hands its payload to the "next protocol", TCP. And life goes on. Any higher-level protocol that runs above TCP will be none the wiser. Is this cool or what? And with all this smart processing going on at the endpoints, what do we see on the wire? Just IP packets whizzing around as usual.
Since you RESTafarians are so forthcoming about how the Web is superior to other forms of distributed computing, let me tell you a little bit about how the Internet is superior to other distributed computing platforms. Look at the old Telco network, called POTS (Plain Old Telephone System), the philosophical antithesis of the Internet - dumb endpoints, smart network. The telephone handsets provided by the Telcos are traditionally dumb, just dialtone-capable. All the smarts are in the network (the "cloud"). All the messagebanks, the call waiting, call forwarding, teleconferencing, everything is taken care of by the smart network. The Telcos claim that the model is least disruptive because they can upgrade capability without having to upgrade millions of handsets.
Is that so? Then how come the Internet has beaten the Telco network even in telephony? Made an overseas call using Skype lately? How much did it cost you? Did you use a webcam? Do the Telcos offer anything equivalent? I rest my case.
The Internet, with its "smart endpoints, dumb network" approach, has comprehensively beaten the Telco network with its "smart network, dumb endpoints" philosophy. It's a fact in plain view that cannot be denied. When innovation is decentralised, it flourishes. The Internet is a platform for decentralised innovation. (Heck, the very web that you RESTafarians wave in people's faces is an example of the innovation that the Internet enables. HTTP is an end-to-end protocol, remember?) You don't fight such a model. You work with it.
Now what does all this have to do with SOAP and Web Services? Everything.
Remember that when we talk about SOAP today, we're talking about SOAP messaging, not SOAP-RPC. Forget that SOAP-RPC ever existed. It was something invented by evil axe murderers and decent SOA practitioners had nothing to do with it ;-).
And another subtle point to remember is that when we say "SOAP message", we always mean "SOAP message with WS-Addressing headers". Why is this important? Because a SOAP message with WS-Addressing headers is an independently-routable unit, just like that other independently-routable unit, the IP packet. Begin to see the parallels?
Now imagine a messaging infrastructure that knows how to route SOAP messages and nothing more. We will deliberately stult the IQ of this infrastructure and prevent it from growing beyond this capability. How do we innovate and build up higher levels of capability? Why, follow the Internet model, of course. Create protocols for each of them and embed all their messages into the SOAP message itself (specifically into the SOAP headers). There's no need to layer them as in the TCP stack, because reliable delivery, security, transactions, etc., are all orthogonal concerns. They can all co-exist at the same level within the SOAP header block - headers like WS-ReliableMessaging, WS-Trust, WS-SecureConversation, WS-Security, WS-AtomicTransaction, WS-BusinessActivity, etc.
Now we have the plumbing required for a loosely-coupled service ecosystem. This isn't by any means the totality of what we require for SOA. At the very least, there's still the domain layer that sits on top of all this elegant message-oriented plumbing. So now define document contracts using an XML schema definition language and a vocabulary of your choice (for Banking, Insurance or Airlines), stick those conforming XML documents into the body of the SOAP messages we've been talking about, and ensure that your message producers and consumers have dependencies only on those document contracts. Ta-da! Loosely coupled components! Service-Oriented Architecture! (Note that just one language - XML - is being used to define both a core application wire protocol (SOAP/WS-*) and one or more application domain protocols (the document payloads). What does 'X' stand for in XML again?)
Now all we need is a bit of metadata to describe and guide access to these services (WSDL and WS-*Policy - some might say SSDL) and a recursive technique to create composite services (WS-BPEL - some might say SSDL again) and we're more or less done.
Sounds good, right? In fact, it sounds more than just good. This is revolutionary! But why haven't we seen this wonderful vision unfold?
A reality check. (And this is no apology or reason for RESTafarians to crow. SOAP-based Web Services technology is not a tired model about to be pushed aside by a simpler and more elegant rival. It's a sleeping tiger that is now stirring.) Why we don't enjoy the brave new world of SOAP-based Web Services today is because:
(1) Those axe murderers we talked about earlier misled the world with SOAP-RPC for a few years, and SOA practitioners are still being detoxified. (There's a school of thought that says the use of WSDL implies RPC even today, but that has more to do with the entrenched (and entirely unjustified) exposure of domain object methods as "services". See my later post on the Viewpoint Flip which is my recommended technique to service-enable a domain model. At a syntactic level, the wrapped document/literal style also distances SOAP usage from blatant RPC encoding.)
(2) Another group of petty criminals talked up SOAP's transport-independence but only wrote HTTP bindings for it, needlessly making the SOAP visionaries look silly. (I think when AMQP arrives - in its fittingly asynchronous way 20-odd years after TCP and IP, we'll begin to see TCP and AMQP as the synchronous and asynchronous transports that synchronous and asynchronous SOAP message exchange patterns should most naturally bind to. There's also the view that the sync/async dichotomy is artificial, and it is correlation which is key. Be that as it may, I don't see a need for SOAP to bind to HTTP at all, except for the unrelated constraint described next.)
(3) There's the pesky firewall issue that causes weak people to surrender to HTTP expediency, but I think that's another thing we will sort out in time. I think of SOAP as standing for "SOA Protocol". It's an application protocol in its own right, and it sits above infrastructure protocols like TCP, UDP and the coming AMQP. We need to define a port for SOAP itself, and get the firewalls to open up that port. No more flying HTTP Airlines just because we like the Port 80 decor. We are SOAPerman and we can ourselves fly ;-).
(4) Vendors of centralised message broker software have been pretending to embrace the SOAP/WS-* vision ("We're on the standards committees!") but in reality they're still peddling middleware, not endpointware. There's a whole flurry of vendor-driven "marketectures" intended purely to protect their centralised broker cash cows from being slaughtered by a slew (pun intended) of cheap, decentralised SOAP-capable nodes. Another round of detoxification needs to happen before people begin to understand that ESB is not an expensive broker you plonk in the middle of your network but a set of capabilities that magically becomes available when you have fully WS-* capable SOAP endpoints. This understanding will dawn, and soon.
(5) As long as commercial licensing of SOA products remains in vogue, deployments will tend to be centralised and will fail to exploit the fundamental Internet-style architecture of SOAP/WS-*. Who wants to pay for multiple instances of software when a single one will do? SOAP and WS-* have only made endpointware architecturally feasible. Open Source implementations will make it economically viable. Three projects to watch: ServiceMix, Mule and WSO2.
(6) The WS-* standards that define security and reliable messaging only recently arrived, so implementations are still being rolled out. But hey, it's here now. Take a look at any Tango-.NET interop demo and you'll be wowed by the policy-driven magic that happens.
I've put together a couple of diagrams that explain these basic concepts. This one illustrates the parallels between the SOAP/WS-* stack and the TCP/IP stack. This one shows my classification of the various SOA approaches in vogue today, with my biases clearly visible :-).
In short, I think there are only two valid approaches to building Service-Oriented Architectures today - (1) REST and (2) SOAP-messaging-using-Smart-Endpoints (and I hate both the SOAP-RPC axe murderers and the Centralised ESB marketing sharks for forcing me to use such a long qualifier to explain exactly what I mean).
And if all this didn't finally turn out to be the poke in the snoot of the RESTafarians that I intended, that's because I like REST. I just don't accept that REST is in any way superior to SOAP-messaging-using-Smart-Endpoints. As I hope I've explained in this lo-o-ong post, the latter's based on an equally elegant model - the Internet itself. As we address the essentially implementation-related deficiencies of the technology, SOAP-messaging-using-Smart-Endpoints will be seen as an equally elegant and lightweight approach to SOA.
Most importantly perhaps, the REST crowd will finally shut up.
15 comments:
Four days since I posted, and not a single comment on such a deliberately provocative article !?
Reminds me of the scene from Last Action Hero, where the movie villain crosses over into the real world and conducts an experiment by killing someone. No one appears even after he loudly announces, "I have committed a murder and I wish to confess."
Maybe I'm in the real world :-(.
Great Blog Post! Thanks, Paul
Thanks Ganesh,
excellent read. Learned quite a bit going throught the article, and not just about REST. Well explained.
Anyway, added you to my reader list.
Thanks,
Stijn.
The problem with referring to WS-Security as a security model for SOAP messages is that you shoot yourself in the foot.
WS-Security does not scale since it requires DOM for parsing. You can do the same thing with SAX but you still have to construct DOM-like structures to validate the security tokens.
Since both DOM and DOM-like structures are CPU hogs WS-Security just is not an option for high-volume traffic.
Which is an attribute of IP networks that I think you forgot to mention. It's not only resilient, it's also capable of high-volume traffic without imposing intensive resource usage on endpoints.
> WS-Security just is not an option for high-volume traffic.
Nonsense, there are known ways to deal with XML processing overheads. For example, there are dedicated hardware devices that can front-end software-based endpoints and handle some of the heavier-duty XML processing. That's the beauty of the SOAP header approach. SOAP intermediaries (or proxies) can selectively process a set of tags, allowing others to pass through unchanged. Thus, a hardware proxy can terminate WS-Security connections while allowing (say) WS-ReliableMessaging tags to go through to the actual endpoint. As long as the "last mile" between the hardware device and the actual endpoint is in a trusted zone (as it usually is within corporate firewalls), it's not a major issue.
And of course, in the longer term, implementations will themselves improve.
I find such comments needlessly defeatist.
Ganesh
> I find such comments needlessly defeatist.
We're all entitled to our opinions. At which cost are these hardware solutions available? Do they work with key stores?
How will they identify security tokens after processing security headers? Can their behavior be modified?
Steven, no offense meant. I hope you get what I'm saying (the "big picture"): The fundamental SOAP/WS-* model is based on sound principles and is as elegant as REST. Any implementation-related deficiencies can be sorted out as time goes by. Transient issues such as XML processing overheads are not dead-ends or showstoppers for SOAP/WS-*.
To address your questions, given that XML appliances are commodities and it is a competitive market segment, prices will tend towards commodity prices in time. As for your remaining questions, it depends on the vendor/implementation. I don't generally endorse vendors on my blog, but these links may help:
http://publib.boulder.ibm.com/infocenter/soasdbox/v1r0m0/index.jsp?topic=/com.ibm.soln.ContentSecAndMgmt.doc_1.0.0/topics/content_installingITFIMDP.html
http://www.informationweek.com/story/showArticle.jhtml?articleID=172301718
http://www.infoworld.com/article/03/10/17/41TCxmlfire_1.html
Regards,
Ganesh
Oh, and SSL suffers from pretty heavy overheads too (compared to plain HTTP), plus it's not cacheable or routable. Whenever you hit a proxy, you either terminate the SSL connection and start another one (exposing the message contents in the clear in the proxy server's memory) or you pass the entire contents through opaquely (which has its own security risks). But no one uses these arguments to "prove" that REST is unviable.
WS-Security is proxyable and routable, plus you can selectively encrypt only a few critical data items in your message, leaving other parts free to be inspected and even used for content-based routing.
I consider SSL to be a blunt instrument, and WS-Security + WS-Trust + WS-SecureConversation to be "SSL done right".
Ganesh
Been there, done that, pointed this out to the WS-* partisans close to three and a half years ago:
http://bitworking.org/news/SOAP_over_Beep
Go there. Stay there. Make it your happy place.
You're right, BEEP was a wonderful opportunity lost.
I don't know, I keep wondering about whether it will make a comeback if some killer Open Source app starts using it...
Ganesh
There is a lot to read here, I'm not quite sure how to respond.
Firstly, I really have to wonder why you feel you have to poke at the snoot of RESTafarians. I mean, it's just a few geeks with blogs. Some happen to be fairly influential, a couple have written a book, but they don't nearly have the mounds of capital that have been thrown down this WS-* hole. I don't think writing a long blog tirade is going to get "the REST crowd to finally shut up", particularly with an entry like this. And given the current market trend.
To some of your points:
The first half of your article talks about the Internet and end-to-end protocols. I'm not sure what point you're trying to make: REST was built was an assumption that it was an architectural style to build a global information space on top of the Internet. It starts with the Internet's design assumptions and adds a few more. All it really adds on top of the internet is a uniform references and uniform operational semantics, so that people can communicate pretty much *anything* with a hypermedia model, at the cost of reduced performance compared to more specialized styles.
You'll notice that very popular new Internet protocols embrace the web, even if they sit on the Internet. BitTorrent relies on the web for trackers and directories, for example. The alternative would be to build their own information system on top of TCP/IP. Why bother?
"SOAP-based Web Services technology is not a tired model about to be pushed aside by a simpler and more elegant rival. It's a sleeping tiger that is now stirring"
On the Internet, I seriously doubt it. Within the enterprise, it has been stirring for some time, though WS-Addressing and friends are still rare. WS-Security being the only exception.
"Remember that when we talk about SOAP today, we're talking about SOAP messaging, not SOAP-RPC. Forget that SOAP-RPC ever existed. It was something invented by evil axe murderers and decent SOA practitioners had nothing to do with it."
This is revisionist history. SOAP was designed by Microsoft and others as a replacement for Dave Winer's XML-RPC. WSDL was designed to provide RPC-like bindings to SOAP as a messaging protocol. JAX-RPC flourished while JAXM failed.... Almost all toolkits that use SOAP today, even document/literal SOAP, ..still... bind... to... procedure ... calls.
RPC isn't just about what the network protocol looks like, it's about how the programmer interacts with the network.
Can you use SOAP asynchronously? Absolutely! Is it what the majority of people do, or will do? Will it be interoperable?
When you design or model an application with SOA, what do you think of? A messaging network? Or components with interfaces and operations and return values?
I agree with you that there is a place for "a better MOM", and the WS-* suite is it, but it seems most frameworks & tools are really just trying to make CORBA and COM+ easier and more extensible, because that's what paying customers want.
"And another subtle point to remember is that when we say "SOAP message", we always mean "SOAP message with WS-Addressing headers". Why is this important? Because a SOAP message with WS-Addressing headers is an independently-routable unit, just like that other independently-routable unit, the IP packet. Begin to see the parallels?"
Not really. Almost every application protocol has to have some form of application-layer addressing in it.
" There's no need to layer them as in the TCP stack, because reliable delivery, security, transactions, etc., are all orthogonal concerns. "
In practice, this is questionable, and not at all how the WS stack is composed.
"They can all co-exist at the same level within the SOAP header block - headers like WS-ReliableMessaging, WS-Trust, WS-SecureConversation, WS-Security, WS-AtomicTransaction, WS-BusinessActivity, etc."
You do realize that for a fan of the end-to-end principle, you're talking about protocols like application-layer reliable messaging that are actually *counter-examples* in the original Saltzer, Reed, and Clark paper?
So now define document contracts using an XML schema definition language and a vocabulary of your choice (for Banking, Insurance or Airlines), stick those conforming XML documents into the body of the SOAP messages we've been talking about, and ensure that your message producers and consumers have dependencies only on those document contracts. Ta-da! Loosely coupled components! Service-Oriented Architecture!
Excuse me, but where's the Ta-da? Who defines the various actions these messages can be used within?
Leaving aside data interoperability issues, how can one semantic domain (Marketing) successfully communicate actions or requests to another semantic domain (Finance) without clunky transformation and yet-another-integration project?
Let's understand end-to-end here:
- When I send a TCP connection to any host/port, will it be able to accept it without me knowing anything about that host in advance? Yes.
- When I send an HTTP GET to any host, will it be able to process it without me knowing anything about what is represented by that URI in advance? Yes!
- When I send a SOAP message to any host, will it be able to process it without me knowing anythign about what is represented by that endpoint in advance? Uhm, not really.
Unless I'm using WS-Transfer or one of the WS-Trust exchange protocols, or WS-MetadataExchange (on supported platforms), or I've looked the WSDL up before hand, which usually occurs through... .:gasp:, an HTTP GET.
Beyond this, where's my data? How do you manage primary keys, relationships, integrity, and provenance of data in this world? If I have a data identifier, how do I know how to dereference it? Is the solution WS-Addressing's EndpointReference? Why is that somehow superior to just a URI?
"(Note that just one language - XML - is being used to define both a core application wire protocol (SOAP/WS-*) and one or more application domain protocols (the document payloads). What does 'X' stand for in XML again?)"
Extensible, but you'll not find it a lot harder to extend in practice if you use XSD and WSDL.
" I think of SOAP as standing for "SOA Protocol". It's an application protocol in its own right, and it sits above infrastructure protocols like TCP, UDP and the coming AMQP. We need to define a port for SOAP itself, and get the firewalls to open up that port. "
I think there is merit to discussing this point. My first thought is, "Good luck with that." WSDL 2.0 was defined with many variations of MEPs because HTTP has its own application semantics. *any* application protocol you layer SOAP over is going to have its quirks and weirdness that makes clear separation impossible. As an example, you'll have SOAP 1.2/WSDL 2.0 "In - Only", except it is actually In/Out with an HTTP response code. SOAP/WSDL bindings over AMQP is going to have its own quirks.
The solution, of course, is to ensure SOAP just sits on TCP or UDP. But there are no specs for this yet, and there are a number of things that SOAP already uses that will need to be reconsidered (HTTP URIs, HTTP response codes & redirection, HTTP GET to discover a ?WSDL, etc.).
Stu,
Thanks for your detailed comments. I don't see anything in them that invalidates what I've said, but I'll respond to your points in a day or so.
Ganesh
Ganesh, thanks so much. I've long thought that the composable services model in SOAP deserved a lot more respect than it gets, but was never able to write it down clearly. The analogy with IP is a great way to do this.
Stu, the somewhat acerbic pushback on REST lately is due to the rather triumphalist and dogmatic tone of all too many posts in this ongoing debate. People on the WS side are typically quick to acknowledge the power of the Web and the elegance of REST, but poke on what are corner cases for public web applications but are mission critical to enterprises, e.g. efficiency, security and reliability. People on the REST side tend to sneer at these use cases ("enterprisey") and assert that the success of the Web somehow proves that adherence to HTTP, MIME, URI, etc. fundamentals is the proper way to build applications.
One example: the pushback on Amazon's SimpleDB's "REST" API for tunneling everything through GET. Now I'm happy to agree that this is puzzling, not consistent with what is prescribed by the HTTP standard, and a misuse of the term "REST". The REST advocate bloggers didn't stop there, or puzzle over whether there might be a method behind the apparent madness, they poured out abuse. "SOAPY" seems to be the favored epiphet, even though (as Ganesh's post should make clear) this approach has nothing in common with SOAP (at least SOAP tunnels HTTP through POST!).
Stu,
A response to your points below. First, a general comment. I think many people (yourself included) haven't read my post carefully enough. I think the title blinded people to what I was actually saying.
> Firstly, I really have to wonder why you feel you have to poke at the snoot of RESTafarians. I mean, it's just a few geeks with blogs. Some happen to be fairly influential, a couple have written a book, but they don't nearly have the mounds of capital that have been thrown down this WS-* hole.
If you'll read my piece carefully, you'll see that I'm not the greatest fan of the centralised ESB thesis backed by those mounds of capital. I'm also not against REST. It's the smugness of REST advocates that's my target. I don't like religions that claim a monopoly on the truth, especially when there's another model that could also legitimately claim a degree of elegance.
> I don't think writing a long blog tirade is going to get "the REST crowd to finally shut up", particularly with an entry like this.
Again, if you read my entry carefully, I listed out the things that needed to be set right in the SOAP/WS-* camp which would then cause REST advocates to stop their crowing. My blog entry was not meant to shut anyone up, rather to get people to understand that there's more to the SOAP model than the REST side has been willing to acknowledge.
> And given the current market trend.
What market trend? The one where people in corporations still go "REST? What's that?" or the Gartner Hype Cycle one that says REST is at the Peak of Inflated Expectations while SOAP/WS-* is at the Trough of Disillusionment? Note that this latter analysis is implying that REST is heading for a fall in popularity very soon (as its usage increases and its limitations are discovered the hard way), while SOAP/WS-* is at last going to be put to useful work - the Plateau of Productivity. I don't usually agree with Gartner, but this time I do.
> The first half of your article talks about the Internet and end-to-end protocols. I'm not sure what point you're trying to make
Simply this: A messaging paradigm that uses a simple routable "packet" like IP (or like SOAP), with all higher-level functions (higher than routing) being built into the smarts of the network's endpoints and buried within the basic packet when on the "wire" is an elegant model that is resilient and supports innovation.
> REST was built was an assumption that it was an architectural style to build a global information space on top of the Internet. It starts with the Internet's design assumptions and adds a few more. All it really adds on top of the internet is a uniform references and uniform operational semantics, so that people can communicate pretty much *anything* with a hypermedia model, at the cost of reduced performance compared to more specialized styles.
No defence of REST is required. Nowhere in my piece will you find a criticism of REST itself. The entire article was about "Agreed, REST is elegant. Now acknowledge the elegance of the SOAP messaging model."
> You'll notice that very popular new Internet protocols embrace the web, even if they sit on the Internet. BitTorrent relies on the web for trackers and directories, for example. The alternative would be to build their own information system on top of TCP/IP. Why bother?
This is a non sequitur. My discussion of IP was at two levels - (1) as an analogy to illustrate the power of the messaging paradigm and (2) as a possible substrate to SOAP. My belief (expressed later on in the article) is that SOAP with WS-* headers is itself an application protocol, just like HTTP. I agree with REST advocates that SOAP should not be tunnelled through HTTP. Your argument is that HTTP is being "abused". My argument is that we don't need HTTP to make the messaging paradigm work. SOAP began to ride on HTTP because of firewalls. What needs to happen is for SOAP to be declared a protocol with a set of assigned port numbers (corresponding to TCP, UDP and AMQP), and then a process of education needs to happen to get people to open up firewall ports to let SOAP traffic through. That's what I said in the post.
>> "SOAP-based Web Services technology is not a tired model about to be pushed aside by a simpler and more elegant rival. It's a sleeping tiger that is now stirring"
> On the Internet, I seriously doubt it. Within the enterprise, it has been stirring for some time, though WS-Addressing and friends are still rare. WS-Security being the only exception.
It is now starting to happen. Look at Tango. Look at WSO2. In spite of all the hype about "too many" WS-* standards, only a handful are really required to do serious work with. And they're here now.
>> "Remember that when we talk about SOAP today, we're talking about SOAP messaging, not SOAP-RPC. Forget that SOAP-RPC ever existed. It was something invented by evil axe murderers and decent SOA practitioners had nothing to do with it."
> This is revisionist history. SOAP was designed by Microsoft and others as a replacement for Dave Winer's XML-RPC. WSDL was designed to provide RPC-like bindings to SOAP as a messaging protocol. JAX-RPC flourished while JAXM failed.... Almost all toolkits that use SOAP today, even document/literal SOAP, ..still... bind... to... procedure ... calls.
Lighten up. When I say SOAP-RPC was invented by axe murderers and not "decent" SOA practitioners, I'm not trying to rewrite history. It's a way of saying that the industry made a mistake earlier by going down the SOAP-RPC route. Nowadays, we hear more about SOAP messaging and see a fair amount of support for it as well, although regrettably, the RPC mindset still holds sway.
> RPC isn't just about what the network protocol looks like, it's about how the programmer interacts with the network. Can you use SOAP asynchronously? Absolutely! Is it what the majority of people do, or will do? Will it be interoperable? When you design or model an application with SOA, what do you think of? A messaging network? Or components with interfaces and operations and return values?
You're not disagreeing with me there. I'm also railing against the culture of RPC within the SOAP community, which has still not been entirely rooted out.
> I agree with you that there is a place for "a better MOM", and the WS-* suite is it, but it seems most frameworks & tools are really just trying to make CORBA and COM+ easier and more extensible, because that's what paying customers want.
It may seem that way, but the decentralised ESB model is slowly sinking into the consciousness of corporate architects and decision-makers, at about the same rate as REST is seeping into their consciousness :-).
>> "And another subtle point to remember is that when we say "SOAP message", we always mean "SOAP message with WS-Addressing headers". Why is this important? Because a SOAP message with WS-Addressing headers is an independently-routable unit, just like that other independently-routable unit, the IP packet. Begin to see the parallels?"
> Not really. Almost every application protocol has to have some form of application-layer addressing in it.
Not sure what you mean by "application-layer addressing", because by definition, the application starts where the transport ends, and the transport is the one that deals with addresses. If this is any help, WS-Addressing addresses are (gasp!) URIs. Of course, not all URIs are HTTP URIs, and I'm hoping that HTTP won't be used for SOAP. For example, I've seen Microsoft documentation (http://msdn2.microsoft.com/en-us/library/bb608213.aspx#LinkTarget_445) that describes WS-Addressing address formats for SOAP-over-UDP:
soap.udp://<host>[:<port>][/<rel_path>][?<query>]
We need standardisation around SOAP-over-TCP, SOAP-over-UDP and SOAP-over-AMQP.
>> "There's no need to layer them as in the TCP stack, because reliable delivery, security, transactions, etc., are all orthogonal concerns."
> In practice, this is questionable, and not at all how the WS stack is composed.
I don't understand your comment. The WS-* tags for security, reliable messaging, etc., are all placed together within the SOAP header block. It's not like an HTTP packet is wrapped inside a TCP packet which is then wrapped inside an IP packet.
>>"They can all co-exist at the same level within the SOAP header block - headers like WS-ReliableMessaging, WS-Trust, WS-SecureConversation, WS-Security, WS-AtomicTransaction, WS-BusinessActivity, etc."
> You do realize that for a fan of the end-to-end principle, you're talking about protocols like application-layer reliable messaging that are actually *counter-examples* in the original Saltzer, Reed, and Clark paper?
Two points:
1. Saltzer, Reed and Clark simply argue that no layer can provide perfect reliability, and that higher layers can/must take advantage of whatever level of reliability lower layers offer. This is by no means a counter-example.
2. "End-to-end" is an unfortunate term that could mean two different things depending on the context. I used it as the antonym to "centralised" when arguing against a "smart bus". But "end-to-end" could also be used to mean "application logic-to-application logic". This is clearly not the case with SOAP headers. They are communicated between "SOAP endpoints", and the actual applications (business logic) lie behind them on either side. It's well-understood that the end-to-end reliability offered by WS-ReliableMessaging is only between the two SOAP endpoints. It cannot guarantee application-to-application reliability. That's constrained by the two-army problem anyway (http://simplanet.org/content/view/77/1/).
>> So now define document contracts using an XML schema definition language and a vocabulary of your choice (for Banking, Insurance or Airlines), stick those conforming XML documents into the body of the SOAP messages we've been talking about, and ensure that your message producers and consumers have dependencies only on those document contracts. Ta-da! Loosely coupled components! Service-Oriented Architecture!
> Excuse me, but where's the Ta-da? Who defines the various actions these messages can be used within?
The "vocabulary" I talked about includes nouns and verbs. The high-level verbs for a domain typically correspond to "processes" (WS-BPEL processes, for example). These when broken down into "process steps", correspond to SOAP operations. Each SOAP operation defines XML documents in and out, which correspond to a set of related nouns. So the answer to your question is, the enterprise architects define these actions (SOAP operations) and the messages (XML documents) that are used within them.
> Leaving aside data interoperability issues, how can one semantic domain (Marketing) successfully communicate actions or requests to another semantic domain (Finance) without clunky transformation and yet-another-integration project?
They can't, that's what the SOA exercise is all about. When you have independent semantic domains, there has to be some kind of "translation" when they communicate, and it can't happen on the fly. Humans need to sit down together and discuss exactly what each term means and how it must be used. What is your point here? Are you suggesting there is a magical way for this to happen?
> Let's understand end-to-end here:
> - When I send a TCP connection to any host/port, will it be able to accept it without me knowing anything about that host in advance? Yes.
> - When I send an HTTP GET to any host, will it be able to process it without me knowing anything about what is represented by that URI in advance? Yes!
> - When I send a SOAP message to any host, will it be able to process it without me knowing anythign about what is represented by that endpoint in advance? Uhm, not really.
> Unless I'm using WS-Transfer or one of the WS-Trust exchange protocols, or WS-MetadataExchange (on supported platforms), or I've looked the WSDL up before hand, which usually occurs through... .:gasp:, an HTTP GET.
Yes, and this invalidates the SOAP model how? This is not the Cola wars where a Coke employee can be fired for drinking Pepsi and vice-versa. The WSO2 Registry uses AtomPub. So?
The service consumer must understand the service. That's what WS-MetadataExchange was designed for, but the model won't fall over if WS-MetadataExchange isn't available. In current practice, this transfer occurs out of band, between groups of people responsible for the service consumer and producer. Are you suggesting that REST makes this transfer of metadata unnecessary? I think such a view is either naive or dishonest. Are you really implying that in the REST model, if I have a URI and know that POST is one of the supported verbs, I will automatically know what XML document I can POST to that URI?
> Beyond this, where's my data? How do you manage primary keys, relationships, integrity, and provenance of data in this world?
Aha! You're confusing the Domain Model with its Representation, and perhaps that's key to understanding why REST people don't see the similarities REST has to SOAP. Ultimately, all of SOA is about loose coupling, which means you don't want service consumers to see the actual domain model used by the service producer, but only a representation of it. The _entire REST state machine_ is only about representations, not the underlying domain model, as the R in REST should tell us. Even in the SOAP model, the actual SOAP operations exposed are _not_ intended to correspond one-to-one with any method call on any object in the domain. That would be RPC, as you yourself said when you described RPC as a programming style. Both SOAP and REST decouple the actual domain model by providing a separate "representational layer" that service consumers interact with. Primary keys and relationships are aspects of the Domain Model. They are not required to be translated to the representation layer. In fact, if they are, that would suggest tight coupling!
By the way, SOAP tooling that allows (say) POJOs to be automatically exposed as SOAP services with all their methods as SOAP operations are _evil_. They are tightly coupling the domain model to its representation and are therefore antithetical to SOA principles.
> If I have a data identifier, how do I know how to dereference it?
Your service _implementation_ is the one that must dereference it. Think opaque URIs in the context of REST. Opaque URIs are deliberately meant to decouple domain model from representation. Who does the dereferencing in that case? The service developer, within the service implementation. The service developer needs to provide the mapping between the domain model and the representation anyway. You can't just generate one from the other - that's tight coupling.
> Is the solution WS-Addressing's EndpointReference? Why is that somehow superior to just a URI?
The WS-Addressing EndpointReference is not superior to a URI. It _is_ a URI, according to the spec. (Note that URI does not imply HTTP URI, and I don't believe SOAP should ride on HTTP.)
>>"(Note that just one language - XML - is being used to define both a core application wire protocol (SOAP/WS-*) and one or more application domain protocols (the document payloads). What does 'X' stand for in XML again?)"
> Extensible, but you'll not find it a lot harder to extend in practice if you use XSD and WSDL.
That's an orthogonal argument. I'm trying to show that in spite of the uniform look of a SOAP message, there are actually two different things that it represents. The set of headers represents an "application wire protocol" which can support any domain, analogous to the four HTTP verbs that can support any domain. The body contains an XML document that is domain-specific. I could visualise two encrypted, reliably-delivered, transactional SOAP messages from the Insurance and Airline industry domains whose header sections were virtually identical. Their body sections (payload) would however be dramatically different.
>> "I think of SOAP as standing for "SOA Protocol". It's an application protocol in its own right, and it sits above infrastructure protocols like TCP, UDP and the coming AMQP. We need to define a port for SOAP itself, and get the firewalls to open up that port. "
> I think there is merit to discussing this point.
At last, some agreement! :-)
> My first thought is, "Good luck with that." WSDL 2.0 was defined with many variations of MEPs because HTTP has its own application semantics. *any* application protocol you layer SOAP over is going to have its quirks and weirdness that makes clear separation impossible. As an example, you'll have SOAP 1.2/WSDL 2.0 "In - Only", except it is actually In/Out with an HTTP response code. SOAP/WSDL bindings over AMQP is going to have its own quirks.
> The solution, of course, is to ensure SOAP just sits on TCP or UDP. But there are no specs for this yet, and there are a number of things that SOAP already uses that will need to be reconsidered (HTTP URIs, HTTP response codes & redirection, HTTP GET to discover a ?WSDL, etc.).
Granted, but as you said, this merits discussion.
Conclusion:
1. If it hasn't become clear by now, my post was not to attack REST, which I think is a great model. It's to attack the blind attitude that anything to do with SOAP is wrong.
2. I don't claim that everything on the SOAP/WS-* side of the fence is perfect. In fact, I've pointed out heaps of things that need to be done to improve matters. What I'm saying is, SOAP/WS-* isn't all trash either. There are very elegant ideas here. They just need to be fully developed.
What did I end my post with again?
"In short, I think there are only two valid approaches to building Service-Oriented Architectures today - (1) REST and (2) SOAP-messaging-using-Smart-Endpoints"
I haven't found any arguments in your comments that refuted what I've said in my post.
Genius, man.
Post a Comment