The Wisdom of Ganesh

Stay Tight on Loose Coupling

2012-01-28T09:05:00.001-08:00

My Guest View was published on SD Times. This is based on the points covered in the white paper Practical SOA for the Solution Architect.

Cool Innovations by WSO2

2011-12-12T06:39:00.000-08:00

I've long held that the WSO2 suite of middleware products is one of the industry's best-kept secrets. But just what is it that makes these products so special?

Well, if the mere fact that there exists a functionally comprehensive, 100% Open Source middleware suite isn't remarkable enough, there are plenty of other reasons why IT practitioners should take a good look at WSO2's offerings.

I have had the opportunity for a few months now to work closely with the company's engineers and play with their products, using them to build (demo) distributed systems, and this is what I have found.

There is significant innovation here that is really cool. There may be even more that I haven't discovered yet.

1. OSGi bundles, functionality "features", and the fluid definition of a "product"

(WSO2 has no rigid products, although their brochures list about 12. The truth is that they have hundreds of capability bundles that can be combined with a common core to create tailored products at will.)

2. A "Middleware Anywhere" architecture that spans cloud and terrestrial servers

(When cloud-native features like multi-tenancy and elasticity are baked into the common core of the middleware product suite, there is no need for applications to be written differently for cloud and terrestrial deployments.)

3. Not just an ESB - the right tool for every job

(As in my writings on "Practical SOA for the Solution Architect", there are three core technology components required for SOA - the Service Container, the Broker and the Process Coordinator. They do different things and are not mutual substitutes. There are also eight supporting aspects at the technology layer. The ESB, being just the Broker component, cannot do all of these functions. WSO2 has products corresponding to all of them.)

4. Making a federated ESB an economically viable architecture

(Economics forces many organisations to deploy their expensive ESB product in a centralised, hub-and-spokes architecture, which leads to a single point of failure and a performance bottleneck. But Brokers are best deployed in a federated manner close to service provider and service consumer endpoints. WSO2's affordable pricing model makes it easy for organisations to do the right thing, architecturally speaking.)

5. The "Server Role" concept - enabling a logical treatment of SOA topology

(Architects don't think in terms of products but in terms of functional capability. One would rather earmark an artifact for deployment to a "mainframe proxy" and another to a "customer complaints process node" than to an "ESB" or a "Business Process Server" instance. Tagging both servers and artifacts with a user-defined "Server Role" makes it possible to speak in these convenient abstractions.)

6. The CAR file as a version snapshot of a working and tested distributed system

(In distributed systems, upgrading the version of software X can break its interoperability with software Y. That's why version change in distributed systems is such a nightmare involving expensive regression testing and a higher probability of outage after an upgrade. But what if a related set of changes can be tested together, certified as interoperable, labelled as a single version, and deployed to multiple systems through a common mechanism? We've just described the Carbon Archive.)

7. The CAR file and Carbon Studio as a unifier of diverse developer skillsets

(Developing distributed systems is challenging from a skills perspective. Writing business logic in Java and exposing it as a web service requires different skills from writing data transformation in XSLT, which is again different from specifying process logic in WS-BPEL. And then there are specialised languages to codify business rules, etc. WSO2 has a single IDE to support all these diverse developer needs - Carbon Studio. It also provides a single package that can hold all these types of artifacts - the Carbon Archive.)

8. The Admin console and support for configuration over coding

(Not every artifact needs to be "developed" through code. Every WSO2 server product has an Admin console that looks largely alike, yet tailored to the peculiarities of the artifacts deployed on that server. Following an 80-20 rule, the bulk of the (simple) artifacts that need to be deployed on a server can be created through configuration using the admin console.)

9. Port offsets and the ability to run multiple servers on the same machine

(Owing to their common core, all WSO2 server products listen on the same ports - 9763 (HTTP) and 9443 (HTTPS). Obviously, port conflicts will result when attempting to run two servers (or even two instances of the same server product) on a single machine. But with a simple configuration change (a "port offset"), a server can be nudged away from its default port to a non-conflicting one. A port offset of 1 will have a server listening on ports 9764 (HTTP) and 9444 (HTTPS), for example.)

10. The dark horse - Mashup Server and server-side JavaScript

(Nodejs has refocused industry attention on server-side JavaScript and the power that brings. But Nodejs has no support for E4X! If you're doing XML manipulation of data from multiple sources, which is what mashups often entail, WSO2's Mashup Server is worth a serious look.)

11. Carbon Studio - A light-touch IDE for middleware developers

(The best GUIs are those built on top of non-visual scripting. Every artifact used by the WSO2 development process is text-based, including the build process that relies on Maven. Command-line junkies can coexist peacefully with those who prefer a GUI, because the IDE imposes no additional requirements on developers. It's purely an option, not an obligation. You can even use another IDE like IntelliJ's IDEA with no loss of capability.)

12. Governance Registry - for control both mundane and subtle

(Governance or plain management? People often use the former term when they mean the latter. But never mind. Regardless of what you want to do, the WSO2 registry and repository tool is simple, flexible and powerful enough to support you. The registry is embedded inside every server product as well as being a standalone server, so the way one can choose to store and share configuration settings as well as policy files is pleasurably versatile.)

As these short descriptions hint, there is something unique and innovative about each of the above, and I'm going to write in greater detail about all of them over the next few days. Stay tuned.

Nutshell Definitions

2011-12-09T02:05:00.000-08:00

When conducting the Practical SOA workshops in different Australian cities last month, I felt the need to explain a few concepts for my audience in very simple and memorable terms. I realised I have already been doing this for myself for a long time. I try and distil a concept into a single word, or at most a short phrase, in order to understand it. This has been very useful to me in evaluating the merits of technologies and comparing them to others. Call these my trade secrets which I'm now sharing with you :-).

OK, so here are some of my definitions of popular terms, each in a nutshell:

Service-Oriented Architecture (SOA):

1. In one word, dependencies. More precisely, the management of dependencies - making implicit dependencies explicit and eliminating unnecessary dependencies between systems. That's what it's all about.

2. Alternative definition: Lego-isation of the Enterprise, i.e., refactoring the various application silos in an enterprise into reusable building blocks based on their core functions.

Middleware:

That which converts organisational silos into Lego blocks.

Integration:

This is a tricky one. When you point your browser at a website and the page loads, no one thinks of it as integration. For something to be recognised as "integration", it seems it cannot afford to appear effortless! For this reason, I steer clear of trying to define integration in terms of technology. The best integration is seamless and not seen as such. It's an art that involves dependency management (see the definition of SOA above). Minimalism is a virtue here, and appropriate data design is an unacknowledged part of integration.

Governance, as opposed to plain old Management:

Governance has suddenly become a very fashionable word, and not just in IT. "Corporate governance" is a phrase that is parrotted by commentators when they often mean just management. So what's the difference?

In a nutshell,

Management is about ensuring adherence to operational parameters.

Governance is about ensuring adherence to policy.

Sometimes, an event can be both a Management incident as well as a Governance incident. If an organisation has a policy around risk appetite, for example, and they quantify that as a number, then operational systems can monitor and enforce adherence to that parameter. Breaches of that limit are simultaneously an operational management issue and a governance issue that need to be reported to the appropriate bodies.

Cloud computing as opposed to conventional "terrestrial" computing:

The leasing of IT capability as opposed to ownership. "IT capability" is a very loose term, and its nature varies based on the type of cloud (below). Leasing has several benefits as opposed to ownership - no upfront costs, pay-as-you-go scalability that is easy on startup operations, i.e., practically limitless capacity without having to provision it upfront, etc.

Infrastructure as a Service (IaaS):

Leasing infrastructure (storage, compute power and networking), and owning all applications above it.

Platform as a Service (PaaS):

Leasing infrastructure as well as some application frameworks and common utilities, and owning all applications above it. The frameworks and utilities make it easier to build the applications above.

Software as a Service (SaaS):

Leasing all the application functionality required, and owning nothing.

Virtualisation:

Imagine separating your "mind" from your "brain". We assume exactly one mind per brain and vice-versa. But what if you can have multiple minds within a single brain, a kind of benign schizophrenia? Or more eerily, if a mind can span multiple brains and think more powerful thoughts as a result? That's virtualisation - turning a one-to-one relationship between mind and brain into a many-to-many one.

Cloud computing as opposed to virtualised servers in "terrestrial" computing:

In virtualisation, you still own the brains. In cloud computing, you lease them.

Private cloud:

Leasing brains to yourself to run minds on. This may make sense in certain cases, like with transfer pricing between business units of the same enterprise. But because you haven't rid yourself of the responsibilities of ownership, it may not be as attractive as a public cloud. Nevertheless, it's attractive because sometimes legislation prevents minds from running on strange brains, and you may yourself be testing the concept of mind-brain separation before trusting someone else's brains with the sensitive job of running (and therefore knowing) your minds.

Building RESTful applications using the WSO2 platform

2011-12-04T00:25:00.000-08:00

Someone taking a casual look at the WSO2 middleware platform might be forgiven for thinking this is exclusively about SOAP and WS-*. But there is in fact strong support for building RESTful applications with this platform using the JAX-RS framework library, and Prabath Siriwardena (one of WSO2's experts on Identity Management) has blogged about the recommended component architecture to achieve this. There is also a WSO2 workshop on this topic conducted by Asanka Abeysinghe, who has many years of experience on the customer side of the fence and understands both the vendor (technology) and customer perspectives. This workshop is on Dec 8th and should be worth attending for people living close to Palo Alto.

An interesting sidelight: In Practical SOA for the Solution Architect, I re-introduce the practitioner to SOA principles by talking about three core technology components - the Service Container, the Broker and the Process Coordinator. According to this view of SOA, a service can be exposed through any of these components and a service consumer will be none the wiser as to the nature of its implementation. In other words, the Practical SOA approach does not make any prescriptions about which component should be the consumer-facing one. All of them are equally valid candidates, and the only criterion for choosing one over the other is the nature of the service enablement mechanism (bespoke, brokered or orchestrated).

In Prabath's architecture diagram below, the reader will notice that runtime clients must all consume services through the ESB (the Broker), even though these services are hosted on the App Server (the Service Container). Why can't services be exposed directly from the App Server where they are hosted?

Prabath explains that using a Broker (ESB) instance as the front-end for services is recommended practice because the ESB can provide security features like authentication and authorisation, as well as throttling capabilities to guard against Denial of Service (DoS) attacks.

So in the real world, we may often need to front-end Service Containers and Process Coordinators with an instance of the Broker that is dedicated to providing these security and traffic shaping features. This could (and should!) be a different instance of the Broker from those used to mediate access to backend systems. Such an architecture will work well because ESBs are better deployed in a federated topology than in a centralised hub-and-spokes fashion. [The unnatural hub-and-spokes topology for ESB deployment, which the high cost of most commercial ESBs forces on customer organisations, then results in a performance bottleneck and a single point of failure. Fortunately, the more favourable economics of WSO2's Commercial Open Source model makes it feasible for customer organisations to implement a more flexible federated architecture for the ESB.]

PaaS by Lineage

2011-11-29T01:20:00.000-08:00

If you've ever wondered about what "Platform as a Service" (PaaS) really means, then you may find this analysis of mine useful.

The traditional NIST model of Cloud Computing shows three layers from a consumer's perspective:

Infrastructure as a Service refers to (shared and scalable) infrastructural capabilities such as compute power, storage and networking, as provided by a cloud vendor. The attributes "shared" (or "multi-tenanted") and "scalable" (or "elastic") distinguish cloud solutions from "terrestrial" alternatives. Amazon's EC2 (Elastic Compute Cloud) and S3 (Simple Storage Service) are examples of IaaS.

Software as a Service refers to applications that you don't have to install on your own computers but can consume through your browser. For an individual, Gmail is the best example of SaaS, while companies can relate to SalesForce.com.

But PaaS has always been a bit of a mystery. What is a "platform", exactly? This is a rather nebulous (pardon the cloudy adjective) area between IaaS and SaaS, and appears to be defined by what the other two are not. The market segment is also still in flux and yet to mature, and there are many players here, each staking out a piece of turf and attempting to define the segment to its own advantage.

I have a high-level, vendor-neutral view of this. [Disclosure: I'm currently working for one of the PaaS vendors, WSO2.]

My preferred categorisation of the PaaS landscape is by lineage. In other words, where did the various PaaS offerings evolve from? I think this is a useful way to look at PaaS because it indicates the traditional strengths of a vendor and therefore where their PaaS offering is likely to be stronger than its competitors. Potential consumers who are looking for a particular emphasis in their PaaS solution will know which vendors are likely to meet their requirements better.

In short, I think PaaS offerings have evolved from one of three directions.

IaaS vendors like VMWare (with their vCloud IaaS) have added DevOps capability to their traditional strength and are targeting organisations that want to develop and run generic applications on the cloud. Their version of PaaS is CloudFoundry.

SaaS vendors like SalesForce.com have made their application more generic and supportive of customisation. Organisations that want to build employee-oriented applications can do so through configuration rather than coding. Their version of PaaS, which is more oriented towards a vertical segment (employee-oriented applications), is called Force.com.

A third direction from which PaaS has evolved is from traditional "terrestrial" middleware (also known as Integration or SOA products). WSO2, which has a full stack of SOA middleware products collectively referred to as Carbon, has added cloud-native features (multi-tenancy, elasticity, etc.) to its suite of products to turn them into yet another version of horizontal PaaS called Stratos. Their DevOps tools have also been upgraded to be able to deploy applications to the cloud just like they do to terrestrial servers.

The diagram below that loosely follows the NIST layering, illustrates my analysis. There may be more versions of PaaS depending on where a vendor has traditionally been based, and I will update this model as more such examples appear.

Glamorous Tech and Workhorse Tech

2011-11-25T05:22:00.000-08:00

IT people are biased towards the new, the cool and the "sexy". Perhaps Node.js and Cloud Computing fall into this category. Glamorous technology is very attractive, because it promises to be "the next big thing", but it's usually not usable today. Like Benjamin Franklin's proverbial new-born baby, it's full of potential, but does nothing for us in the here and now.

Contrast this with "workhorse technology", which is mature and relatively boring, and used to build working real-world systems that make or save money today. Not many people get excited about workhorse technology, but the majority of them work with it anyway, because it pays the bills.

I want to talk about one particular category of workhorse technology that I'm discovering can be quite glamorous as well. I have gradually become more familiar with this over the last few months of working with WSO2, writing the "Practical SOA for the Solution Architect" white paper, conducting a webinar and then hitting the road to conduct workshops in three Australian cities. I've realised that the combination of a lightweight SOA methodology in combination with a lightweight SOA product suite can be a very effective workhorse technology that is usable today and saves real dollars. In addition, it's actually pretty cool because of how quickly and easily it can help practitioners integrate diverse and distributed systems into an end-to-end business solution.

In a nutshell, in the white paper, webinar and workshops, I evangelise the message that SOA is not an esoteric and complex black art but simple commonsense that can be readily applied. I talk about the technology layer of SOA, of course, but also cover the equally important data layer that is often neglected and that contributes to the "tight coupling" that plagues so many solution designs and prevents them from realising the benefits of SOA. I talk about three core technology components to use (the Service Container, the Broker and the Process Coordinator) and when to use them, the inefficiencies that result from using the wrong tool for the job, and the dangers of treating the Broker as a singleton, centralised component and deploying it in a hub-and-spokes architecture rather than a federated one. I also cover four simple Data layer principles (make implicit dependencies explicit, remove unnecessary dependencies, loosely couple internal domain data with externally-visible message data, and settle on an intermediate granularity for domain data model(s) rather than a single overarching Canonical Data Model for the entire enterprise).

To drive home these concepts, we work through the real-world example of a well-known banking process, i.e., opening an account. Using the lightweight Practical SOA methodology, participants are encouraged to try their hand at producing an outline solution design to a described requirement in about 15 minutes, using a few standard Lego-style components from the SOA technology layer. Then we demonstrate how that solution design can actually be implemented, substituting each of the conceptual Lego blocks with an actual platform product that hosts the corresponding logic, and get the entire system to work.

Here's a view of the conceptual building blocks that form the outline solution design, once the lightweight SOA methodology is applied to the business problem:

In the next step, the conceptual components are replaced by actual WSO2 SOA products that perform each of those functions, and the physical version of the above diagram looks like this:

The Customer Master Database, the Mainframe and the Card System are all mock objects. A Data Services Server exposes the Customer Master as a set of CRUD services. A Broker (ESB) component exposes the mainframe as a set of Account services, and a second Broker exposes the Card System as a set of Card Services. Since the mainframe can usually only be accessed over IBM MQ in real-life, we simulate that through a JMS connection over ActiveMQ. A Process Coordinator (Business Process Server) coordinates all these services into a BPEL process that performs the account opening business function.

It was gratifying to see that participants at all the workshops we conducted were visibly impressed by this demonstration. In the space of 45 minutes, we had moved from a problem, through a conceptual solution design, to a working implementation. And since we had their active participation through the design exercise, they had an emotional investment in the solution and were able to appreciate it all the more. [To be fair, the demo was developed earlier over 6 person-days of effort, and in the workshop, we stepped quickly through the development by copying code that was written earlier.]

Traditional SOA vendors and the big-name analysts have done the industry a disservice by complicating SOA and scaring off practitioners. In our workshops, we've shown that SOA is just commonsense and relies on just a few simple principles that practitioners can readily apply. Once they know the right tools to use in the solution, it's a simple matter to gather those required components and hook them together to create the end-to-end solution.

This is obviously workhorse technology, because it's mature enough to solve bread-and-butter business problems today. But the lightweight methodology and product suite also make it glamorous.

I hope this attracts more practitioners towards the practice of lightweight SOA and the use of simple and cost-effective products like the WSO2 product suite.

Enterprise Shared Services and the Cloud

2011-11-18T20:09:00.000-08:00

I've worked in the area of Enterprise Shared Services (or Enterprise Utilities) for many years, so when InfoQ approached me asking if I would be interested in writing an article on cloud computing, this was one of the angles I thought of. Of course, I'm also working with WSO2 at present, and WSO2 has a distinct type of PaaS (Platform as a Service) offering called Stratos. PaaS usually either evolves up from IaaS (Infrastructure as a Service) with the addition of support for DevOps (the development-operations continuum) or evolves down from SaaS (Software as a Service) with the addition of customisation support for the software application. Stratos is unique because it has evolved "sideways" from enterprise middleware with the addition of cloud-native features. The 12 SOA products of WSO2 are all available as cloud-native middleware on the Stratos PaaS.

In any case, since InfoQ wanted vendor-neutral content, I couldn't write about Stratos (which I will write about in some context because I find it fascinating). So I fell back to my old favourite - Enterprise Shared Services.

The long and short of it is that when we factor in Enterprise Shared Services, the old monikers of SaaS and PaaS are no longer enough. We have to deal with "vertical" and "horizontal" variants of these, and the distinction is important because in an organisational context, they have unique characteristics around how they are requisitioned, evaluated for feasibility, funded and charged back.

I'll let you read about it here.

A Good Primer on NoSQL

2011-10-28T16:59:00.000-07:00

NoSQL databases are all the rage, but the array of choices before us is bewildering. I must confess I'm still confused about the features and differences between BigTable, GAE DataStore, GemFire, SimpleDB, SQLFire, CouchDB, MongoDB, RavenDB, Redis, Cassandra, Riak, HBase, Neo4j and so many other names that I have only recently begun to hear about. I'm sure many others would be in the same situation.

I was therefore happy to see that my colleague at WSO2, Dr Srinath Perera, has analysed the NoSQL landscape in depth, zeroed in on the characteristics of NoSQL databases that are relevant, and summarised this for our common understanding in an InfoQ article that provides a simple overview of the choices that designers and developers have today, choices that go beyond the traditional relational databases that we're familiar with.

I've often wondered about why NoSQL should be so popular in the first place. Srinath explains:

A few years ago, most systems were small and relational databases could handle [their requirements] without any trouble. Therefore, the storage choices for architects and programmers were simple. However, the size and scale of these systems have grown significantly over the last few years. High tech companies like Amazon and Google faced the challenge of scale before others. They soon observed that relational databases could not scale to handle those use cases.

In other words, this demanding new requirements wave has probably not hit most of us yet, but with the jump in the number of connected devices (smartphones, tablets and the coming "Internet of Things"), applications dealing with huge volumes of data are probably not going to be as rare as in the past. And when we say "huge", we're not even talking Gigabytes anymore. It's Terabytes and larger. As we learnt from Godzilla, size does matter. And drastic situations call for drastic measures, hence the NoSQL revolution.

Srinath refers to Eric Brewer's CAP theorem, which states that a distributed system can only have two of the three properties - Consistency, Availability, and Partition Tolerance. The NoSQL databases aim to break through the limitations imposed on traditional relational databases by loosening the fundamental principles on which these have been based, dropping one or more constraints as appropriate, to obtain a desired behaviour.

Depending on the constraints dropped, the resulting solution falls into one of several new categories:

Local memory
Distributed cache
Column Family Storage
Document storage
Name-value pairs
Graph DB
Service Registry
Tuple Space

...in addition to the traditional filesystems, relational databases and message queues that are familiar to IT practitioners today.

Perhaps the most important contribution of Srinath's article is his distilling of the four primary characteristics that are important from a usage point of view - data structure, the level of scalability required, the nature of data retrieval and the level of consistency required. He then puts these characteristics together in various combinations to show which of the above-listed categories of data store would be the most appropriate solution to use.

He's certainly succeeded in demystifying NoSQL for me, although I suspect I'll need to go back and read the article a few times till I've fully internalised the concepts in it. This is an overview article that I'd recommend to anyone trying to make sense of NoSQL and wanting to decide on the appropriate product category that would be right for their needs.

I can see the demand for a follow-up article from Srinath drilling down into each of these data storage categories and providing recommendations about actual products (e.g., Cassandra, Redis, CouchDB, etc.) While the sands shift more rapidly in the product space, it's also a more practically urgent decision for a developer or architect to make. So while such an article might need to be updated quite frequently, the advice in it would be more practical than this one, which provides the necessary initial understanding of the NoSQL landscape.

Strange Creature on the Mozilla Firefox Download Page

2011-10-19T03:08:00.001-07:00

I've never seen this creature before (circled in red). Do you know who or what it is? It looks friendly enough, but it also reminds me of the Morlocks in H.G. Wells's The Time Machine (shudder).

I Hate HatEoAS

2011-10-18T16:31:00.000-07:00

For something that's supposed to be THE defining characteristic of REST, it could have done with better naming.

I would have been happy with the term HatEoAS if it had stood for "Hypermedia as the Envelope of Application State" rather than "Hypermedia as the Engine of Application State".

An Engine actively drives things. E.g., A process engine is well named, because it drives a process.

A constraint doesn't drive anything. It constrains. It provides an envelope around the range of possibilities.

And so they really should have called this an envelope rather than an engine of application state.

There, I've said it. Because the expansion of HatEoAS has been driving me up the wall.

Fortunately, it's being referred to as "Hypermedia Constraint" now, which is both more elegant and more accurate.

Oneiric Ocelot Not Quite the Stuff of Dreams

2011-10-16T09:29:00.000-07:00

One of the nicest things about Linux distributions like Ubuntu is that you don't have to spend the night standing in a queue just to get the latest version of an operating system on the day of its release.

Unlike some other (closed) systems where supply is deliberately constrained
to create an impression of even greater demand, Ubuntu is upfront and
relaxed about new versions.

I sat down at my Linux desktop yesterday and was pleasantly surprised to see a popup informing me that the next version of Ubuntu Linux (version 11.10 a.k.a. "Oneiric Ocelot") was now available and would I like to upgrade?

With a smile of anticipation, I clicked Yes, and the upgrade began. My ADSL link showed a steady bandwidth of around 400 kBps throughout. An uneventful hour and a half later, I rebooted into Oneiric Ocelot. That was the kind of experience I've got used to with Ubuntu over so many online upgrades.

Most of the time, upgrades to Ubuntu are boringly anticlimactic. That's a good thing, by the way, because users hate surprises, and there are really very few nice surprises possible on an upgrade.

The dictionary says "oneiric" means "pertaining to dreams", but there was something almost nightmarish with this upgrade that was even worse than the upgrade to 11.04 ("Natty Narwhal").

Someone at Canonical has taken it into their head that a completely re-imagined user interface would be a good thing. The same someone has also arrogantly assumed that there's no need to give users a choice when changing a fundamental aspect of the user interface that they will use all the time.

My unpleasant surprise after both upgrades was the horror they call the Unity Desktop. I tried to be fair to them. I believe I gave Unity an hour of my time both times. In the end, I gave up. I just hated it. Sorry Canonical, I recognise you're trying, but Unity really doesn't work for me. From the number of similar comments I read on the web, I'm hardly alone.

What was far worse than installing Unity by default (without asking me if I wanted it) was not providing me a quick way to get back to the default Gnome desktop of earlier versions. I was actually forced to download the Gnome desktop, then re-login to select it as my default desktop.

1. Why didn't I have the choice to say no to Unity at the time of the upgrade?
2. Why wasn't it a straightforward option to return to the "classic" desktop?

For a distribution that is supposed to be the friendliest desktop Linux, this is very poor showing indeed.

My other major whinges are that the "Show desktop" button on the taskbar has disappeared, as has the "System" menu on the menu bar. I now have to minimise every window manually, and have no way to set several preferences. Cosmetically as well, the taskbar at the bottom and the menu bar at the top now sport a ghastly dark grey colour, and the desktop theme that I used to use has disappeared from the list of options. Since I've forgotten what it was called, I don't think I can get it back.

This upgrade experience has been anything but oneiric. I feel like I've been mauled by an ocelot.

Practical SOA for the Solution Architect

2011-10-14T05:06:00.000-07:00

This is a story that has had a fairly long history, so stay with me till the end.

I've worked as an architect in the shared services space for almost a decade now, at some of Australia's biggest, richest and technologically diverse financial services organisations. During that time, I first heard about Service-Oriented Architecture (SOA), learnt what it was about, bought into its philosophy and attempted to implement it at work.

And while I have seen a few successful examples of SOA projects (mostly individual services, truth be told), by and large, I did not see SOA having an impact at all at these large and reputed organisations. Many hundreds of thousands of dollars were spent on acquiring SOA tools from vendors as reputed as IBM and TIBCO, and many millions more were spent on integration projects using these tools, but somehow, the results failed to live up to the SOA promise. [I guess organisations need a terrorising CEO like Amazon's Jeff Bezos to achieve the benefits of SOA. Read the fascinating story of how an online bookstore built a platform that it now rents out to others.]

For a while, I lost faith. Influenced by a few cynical colleagues, I too began to think SOA was marketing hype and nothing more. But then, as I continued to work in the shared services domain and had the opportunity to review more solution designs, I had an epiphany. Most of the designs I was seeing used SOA products (usually an ESB) or were implemented as Web Services, yet I could see they were still tightly-coupled at the level of the application design, i.e., the data. Also, it was very common for solution designers to use the wrong tool for the job, simply because it was the one they were most familiar with. They even seemed to lack a conceptual ability to tell which tool would be right for a given requirement.

With that epiphany, I revisited my understanding of SOA. I realised that solution architects needed to be educated to produce SOA-compliant application designs, otherwise all the investment made by their organisations in SOA tool suites would be a waste. Worse still, SOA itself would be unfairly blamed for the waste of resources, when it in fact remains the best hope to reduce waste, improve business agility and reduce operational risk.

I've been tossing this idea around in my head for at least a couple of years now, the idea that a lightweight method is required to get solution architects up to speed with the required concepts quickly. Unfortunately, most solution architects would bristle if it was suggested to them that they don't really understand SOA. "Can't you see we're using an ESB?" would be the defensive response. That the response is a non sequitur would be lost on them. One can use an ESB and still come up with a design that is not SOA-compliant. How can we educate solution architects when they don't know what they don't know?

Fortuitously, I had a chat about this in August this year with Sanjiva Weerawarana, the CEO of the innovative Open Source middleware company, WSO2. Our needs meshed perfectly. WSO2 has a full suite of middleware products based on SOA concepts and Web Service standards. They are compact and not bloated, fairly straightforward to install and use, fully Open Source under the Apache Software Licence, and for which WSO2 offers a very attractively priced support model as well as other professional services. For such an innovative range of value-for-money products, the level of awareness in the customer community has been surprisingly low. Sanjiva has been trying to raise the level of awareness in the industry about his company's products for some time now. For too long WSO2's pitch had been aimed at developers (techies talking to techies), but for a real breakthrough, they needed to target decision-makers and decision-influencers. They were looking for a way to reach the solution architect with a compelling message.

And here I was, trying to educate the same solution architect about SOA using a new, lightweight approach. So Sanjiva and I came to an understanding. I would do a paid consulting assignment with WSO2 for a few months and turn out a few white papers to present their offerings to an audience that was higher up the corporate food chain than the developers. In return (in addition to helping me pay the bills), they would provide me a vehicle to popularise some of my ideas on SOA, especially the lightweight methodology I came to call "Practical SOA". I guess this approach is the good cop to Jeff Bezos's bad cop :-).

The first of those white papers ("Practical SOA for the Solution Architect") has now been completed and is available on WSO2's website. It's my audacious hope that a Solution Architect can read it in half an hour and be immediately effective on their project thanks to a simpler and more powerful mental model of SOA. A short summary of the paper is available for you to skim through, but I would encourage you to download the full paper (a free registration is required), since it has much more detailed descriptions, extensive explanations for the final conclusions and a couple of industry examples to drive home the concepts. So please have a read and, if you like it, recommend it to all your architect- and designer-type friends. (At my own request, my name doesn't appear on this document because I don't want to dilute the appeal of the method by causing it to look like a mere individual's opinion. WSO2's cachet is better than mine!)

A second white paper that is currently in the works will describe the full suite of WSO2's products and how they map to the framework established in the first. Stay tuned for that too. The first white paper will equip you with concepts. The second will equip you with know-how about a comprehensive set of reasonably-priced tools. Together, they should provide a customer organisation with excellent value for money and the long sought-for return on their investment in SOA.

Steve Jobs and Dennis Ritchie

2011-10-13T05:05:00.000-07:00

This hasn't been a great week to be a computer pioneer. Dennis Ritchie has now followed Steve Jobs off the stage and into that great big computer industry in the sky. Two giants in a single week.

Steve Jobs

Dennis Ritchie

I guess, unlike most people, I have been touched by Ritchie much more than by Jobs. I have never owned a Mac or an iPhone, and regretted buying an iPod as soon as I realised there was no way to bypass the iTunes straitjacket. iPod generation 4 worked with the gtkPod application on my Ubuntu desktop, but generation 5 corrected that shocking oversight, and the Apple empire, with a sigh of relief, regained its pristine purity, shutting out the great unwashed once more. That ended my dalliance with Jobs and his closed system. I don't fancy handcuffs even when they're haute mode.

I feel sorry about the death of Steve Jobs the human being. I have no sympathy for the worldview that he represented, of closed systems, slimy lawyers and patent lawsuits.

Dennis Ritchie, of course, was the polar opposite of Jobs. Those who have read Asimov's science fiction trilogy Foundation may remember that there were two Foundations, a well-known one at the periphery of the Galactic empire, and the other, a secret one, located "at the opposite end of the galaxy". Many characters in the novel tried searching for the Second Foundation along the opposite edge of the galaxy where they thought it would be, but its actual location was right at the centre of the empire! The term "opposite" was meant in a sociological sense, not a physical one.

And in true Foundation-esque fashion, Ritchie's contribution to mankind, while in a sense the opposite of Jobs's, was not a rival closed system but an open one. Along with Ken Thompson, he wrote the most open operating system of its time, -- Unix.

Ken Thompson

The popular web article "The Last Dinosaur and The Tarpits of Doom" has a matchless passage describing the world at the time of Unix's birth.

In 1970, primitive proprietary operating systems bestrode the landscape like mighty dinosaurs: Prime's PrimeOS, DEC's RSTS, RT-11, etc. (with VAX/VMS soon to come), IBM's innumerable offerings, CDC's Scope and of course dominating the scientific workstation market, Apollo's Domain.
Who would then have dared to predict the fall of such giants?
What force could topple such entrenched operating systems, backed by massive industry investment, hacker culture and customer loyalty?
Today, of course, we all know the answer:
In 1975 Bell Labs released Unix.
Unix had no support from its creator, AT&T: Buy the magtape and don't call us. (AT&T was legally barred from entering the operating system market.)
Unix had no support from any existing vendor: None had the slightest interest in backing, supporting or developing an alternative to its proprietary operating systems offerings.
Unix had zero customer base: Nobody had ever heard of it, nobody was requesting it.
Unix had zero marketing: Nobody had any reason to spend money building mindshare for it.
A one-sided competition?
Decidedly: Unix wiped all workstation competition off the map in less than fifteen years.
On April 12, 1989, HP bought up Apollo at a fire-sale price, putting out of its misery the last remaining proprietary operating system vendor in the workstation world, and the workstation proprietary OS era was over: Unix was left alone in the workstation market.
In fifteen years, a [magnetic] tape and an idea had effectively destroyed all opposition: Every workstation vendor was either supporting Unix or out of business.

Let me add one more point to that. At the heart of Apple's operating systems is a version of Unix (BSD Unix). Steve Jobs's business empire took a freely available operating system, layered a user-friendly graphical interface over it, and without a word of thanks, proceeded to build a proprietary edifice that was as closed as its enabling technology was open.

So thank you, Dennis Ritchie, for giving us today's Mac.

Ritchie was an inventor second to none. People today forget one of the main reasons Unix is considered "open". Before Unix, an operating system was written for a specific processor chip, in the assembly language corresponding to that chip. One of the key factors that made Unix open was the fact that it could be ported to any chip at all. More than 90% of an operating system's logic is in fact independent of the underlying hardware architecture. Less than 10% is specific to the chip. That's why only very low-level code in Unix is written in assembly language. That's the only part that needs to be re-written when porting Unix to a different processor architecture.

Once the operating system was liberated from its ties to hardware, any hardware manufacturer could port Unix to their computers. That's the openness that destroyed the proprietary dinosaurs and created the world we see today. We have Thompson's and Ritchie's genius to thank for that. In the next generation, Linux proceeded to wipe out proprietary Unix variants to take over the server room.

Today, Google's Android has Linux at its core. So now Ritchie's invention has taken over the server, a significant part of the desktop (through the Mac) and an increasingly dominant part of the smartphone and tablet markets (through Android and Apple's iOS). Not bad for a simple and open operating system!

Now we know that 90% of Unix is written in a higher-level language, and therein hangs another tale. At the time Thompson and Ritchie wrote Unix, there was no suitable high-level language to write an operating system in. It had to have the higher-level constructs of most modern, structured, procedural programming languages. Yet it also had to provide sufficient control over low-level constructs like memory addresses and file structures. This was a challenge that may have stumped other people and caused them to compromise in some way. Not Ritchie. Necessity for him was the mother of invention - the invention of the C programming language. Together with Brian Kernighan, Dennis Ritchie created the first C compiler, and it is astonishing that the language has hardly had to change since their version to the present day. The standardised version of their language, ANSI C, is largely the same as their original one, with just minor changes. Now that's vision for you.

Brian Kernighan

C inspired C++, Java, JavaScript, Perl, C# and a whole bunch of other languages. Any language with curly braces and semicolons owes an intellectual debt to Ritchie and Kernighan.

The laptop that I'm composing this on runs Ubuntu Linux, another variant of Unix. Most of Linux is written in C. I'm probably not fully aware of the extent to which I owe Ritchie a debt of gratitude, as the one common factor in the creation of both Unix and C.

By the way, if you think Unix has an ugly user interface because of its command line, there are two rebuttals to that argument. The trivial one is that modern Unix variants like Linux have very sophisticated and friendly user interfaces indeed. The deeper rebuttal is that there is beauty and power in the Unix command line that MacOS has eagerly embraced as an offering to the "power user".

User Interface experts Don Gentner and Jakob Nielsen write in their classic paper The Anti-Mac Interface:

The see-and-point principle states that users interact with the computer by pointing at the objects they can see on the screen. It's as if we have thrown away a million years of evolution, lost our facility with expressive language, and been reduced to pointing at objects in the immediate environment. Mouse buttons and modifier keys give us a vocabulary equivalent to a few different grunts. We have lost all the power of language, and can no longer talk about objects that are not immediately visible (all files more than one week old), objects that don't exist yet (future messages from my boss), or unknown objects (any guides to restaurants in Boston).

Like they said. As an advocate of the power of the Unix command line, I rest my case.

Unix is such a unique phenomenon in the world of computing that noted academic Prof Martin Vermeer believes it should be treated as a basic element of literacy, alongside the three Rs.

And so a tumultuous week has gone by, and the computer industry mourns its two luminaries. Among computer pioneers, Steve Jobs was the shiny user interface, slick and popular. Ritchie was the kernel, unseen and unknown to the masses, yet the workhorse that made everything else possible, including the user interface. He may be less widely mourned, but no less mourned.

And I like to think Ritchie rushed after Jobs to make sure the Pearly Gates stayed open to all!

Google's Aimless Darting

2011-10-12T01:47:00.000-07:00

Search Engine vendors should be in the clarification business, not the muddying business. All the more frustrating to read the news about Dart, Google's new web programming language to replace JavaScript.

My only reaction is - WHY???

Yes, JavaScript is not exactly a perfect language. It has warts, huge ones. That's been known for a long time. Is that sufficient reason to throw the whole language overboard and try and popularise a new one? Especially after CoffeeScript has already done the job? JavaScript has good parts as well as bad, and I'm not talking about just its technical aspects. Its ubiquity is a major strength. Replacing it is an exercise with very doubtful prospects. I don't think even Google can pull it off.

CoffeeScript has taken the right approach, in my opinion. CoffeeScript is JavaScript with lead shielding over the reactor core. And that was all that was required. CoffeeScript on the server side uses Nodejs to run scripts anyway, so one can write server-side code in CoffeeScript and run it with "coffee" instead of with "node", and the ugliness and danger of JavaScript can be neatly sidestepped. Even on the client side, a single line such as


<script type="text/JavaScript" src="coffeescript.js"></script>

will allow you to write the rest of your client-side code in CoffeeScript, because coffeescript.js is a minimised library that will let your browser interpret CoffeeScript natively. Your application code will look like this:


<script type="text/CoffeeScript">
# CoffeeScript code
</script>

What is Dart going to do for us beyond that, functionality-wise? Technically, Dart won't even replace JavaScript, because it will compile to JavaScript. Does that sound familiar? Because that's just what CoffeeScript does as well! Was it so hard for Google to get behind CoffeeScript? Some NIH at work, I think.

Now the frustrating thing is that Dart will waste at least some developer mindshare and bandwidth when the world should have been just getting on with the job - using CoffeeScript plus jQuery on the client side, CoffeeScript plus Nodejs on the server side.

What a waste!

Life above the Service Tier (Change of Links)

2011-10-10T16:37:00.000-07:00

Google Groups no longer allows file uploads. Worse, they went back on a commitment to keep the files that were already uploaded accessible. Now all the files I uploaded to the "wisdomofganesh" group have to find alternate homes (if I still have a local copy, that is), and I have to update all the links from my blog. Totally uncool of Google. I'm very annoyed.

Anyway, since the following docs seem to be the most frequently accessed, here are the new links:

1. Life above the Service Tier (white paper)

2. Life above the Service Tier (presentation)

3. SOFEA in a SOA Ecosystem (diagram)

Amazon Cloud Drive Tech Talk - Sydney, 10-10-2011

2011-10-10T05:07:00.000-07:00

I attended the Amazon Cloud Drive Tech Talk today in Sydney. This was held on the 39th floor of the Citigroup building at 2 Park Street in the CBD. The views from the window are awesome, by the way.

That's Park Street stretching away towards King's Cross

Those red slugs are the new Metro double-buses

I reached the venue just after 1800, when registration was scheduled to begin. There were eats, bottles of beer and canned carbonated drinks in the room, but none suitable for a vegetarian teetotaller trying to lose a few calories, so I drank a glass of water instead and waited for the proceedings to begin.

The event kicked off promptly at 1830. John Scott of the Android Australia User Group made a short speech introducing Piragash Velummylum of Amazon, who had flown in from the US.

Then Piragash took the stage and spoke. I must say the Amazon Cloud Drive turned out to be more Amazon Recruitment Drive than anything else! Piragash and a colleague Brad (who spoke for half a minute) made no secret of the fact that they were in town to recruit developers for their Seattle office. The tech talk was like a campus recruitment talk - just enough data to pique the interest of developers wanting to work on cool technology.

The unofficial motto of the Amazon Cloud Drive service is "anything digital, securely stored, accessible anywhere". Piragash spoke a bit about Amazon CEO Jeff Bezos, who he projected to be a likeable nerd who listens to everybody and does the right thing. [Piragash must have a weird sense of humour. Just read this piece and the comments that follow. I wouldn't want to work for Jeff Bezos in a hundred years.]

Amazon Cloud Drive is meant to address three customer pain points:

Multiple music downloads
Moving files from one store to another
Data loss

He said something about Amazon MP3, which seems to be similar to Apple's iTunes (Piragash sidestepped a question from the audience on a comparison with iTunes, saying they had a policy not to talk about competitors, but mentioned that the Amazon version allowed upload of customer content). I think the story might have begun with Amazon MP3 which catered to music files. Then Amazon Cloud Drive came along which was more general-purpose and catered to videos and other kinds of files as well. That's what I gathered.

He emphasised a few times that Amazon Cloud Drive is for customer content, not purchased content (also called studio content or catalog content). It's like DropBox, I guess. It's said to be free, but there was no mention about storage limits. Unlike DropBox, they don't do de-duplication of files (yet), and certainly not across users. They would need to sort out licensing before they did that sort of thing, and he made a statement to the effect that Amazon is DMCA-compliant.

They obviously leverage off their other technologies, i.e., AWS (Amazon Web Services). They currently use S3 (Scalable Storage Solution), not EC2 (Elastic Cloud Computing), but they may use that in future if required.

Amazon Cloud Drive has three layers. Storage is provided by S3. They've defined a hierarchical file system on top of that (which has long been a demand of S3 users). Finally, they have metadata on top that defines relationships and queries, making the file system much more useful.

These are some of the operations supported at each of those levels:

S3: Upload, download

Filesystem: Copy, move, delete, list, recycle, restore, create

Metadata: Select, get, put, delete (Aha, REST! But Select seems to have replaced Post. Puzzling.)

Piragash also talked about some of their technical challenges.

Scaling was the major one (and continues to be a major focus of research and innovation). The principles Amazon follows to ensure scalability are:

1. "Don't get in the way", i.e., let AWS and Akamai do the job they know best, don't interpose Amazon Cloud Drive between those systems and the user. Rather, allow Amazon Cloud Drive to be proxied by CDNs (Content Delivery Networks) like Akamai.

2. "Be flexible", i.e., be forward-thinking and don't prevent services like allowing a zipped set of files to be downloaded.

Security is another major concern. Content is meant to be private and personal. As part of their PaaS offering, Amazon provides an Identity and Access Management (IAM) system, and Amazon Cloud Drive uses IAM and AWS to control the generation of time-bound tokens. Delegated S3 access has an extra security token in the URL that expires after a certain period, so people can't pass content URLs around and have them accessible indefinitely to anyone who gets the URL. Can customers share their content with others, or is this purely private? Piragash's response: "Stay tuned".

Then Piragash briefly talked about the Kindle Fire, due for release in about a month. Incidentally, Kindle for iPad is said to use HTML5 throughout and to look like a native app.

There's also a new business called Amazon Fresh, currently only rolled out to the Seattle area. Order your groceries online at midnight, and have them delivered to your door by 6 am!

There was a little talk about version synchronisation strategies and rules for conflict resolution. They support multiple schemes for different domains. Sometimes they use automated algorithms, and at other times they let the customer resolve conflicts.

Asked about an API for Cloud Drive, Piragash would only say, "Stay tuned".

He mentioned that it was important to weed out "phantom requirements" and to concentrate on solving the customer's real problems.

A member of the audience remarked that Australia really needed an edge server located here to reduce latency. Piragash merely smiled acknowledgement. Someone else asked about the number of servers used by Amazon, but Piragash could not talk about that.

That was the end of the talk, and Piragash invited people to stay and talk to him and his colleagues. He called for CVs and said they'd be in Sydney the whole week, recruiting people to be based out of Seattle.

So if you like rain throughout the year and don't mind sharing a city with Microsoft, do apply to Amazon for a job.

[What was even more interesting than the technology talk was a nugget of cultural trivia that I've written about on my other blog.]

Aakash - The Sky's the Limit

2011-10-06T00:55:00.000-07:00

Android tablet prices went down even further with the announcement of the Indian-made tablet "Aakash" (Sanskrit for "sky", pronounced "aakaash"). The price to students (admittedly with a government subsidy) is projected to be $35. Even without the subsidy, the retail price should still be a groundbreaking $60. The aim is to bridge the "digital divide" and allow less affluent sections of society to participate in the digital economy.

I like free markets, but given the tendency for cartels to form in most market segments (negating the freedom that true liquidity would deliver) and the distortions of patent law that entrench the power of large corporations, I also think governments need to step in from time to time to ensure equity. Untrammelled capitalism isn't going to bridge the digital divide. There has to be a deus ex machina that kicks in at critical junctures.

Stepping back a bit to take a historical view, the primary difference between the 19th and 20th centuries was not in scientific knowledge or even technological invention (because many 20th century inventions were known even in the 19th), but in the mass production and mass consumption of such technology. In analogous fashion, I can see the potential for India to become a technology "enabler" for the world's poorer half (or two-thirds), democratising technology usage across a geographical span rather than a historical one.

I remember the dire warnings a decade ago about an AIDS epidemic that was poised to devastate Africa, and the utterly shameful behaviour of the big Western pharma companies in refusing to lower the prices of AIDS medication to save millions of lives. They tried to use patent law to block any attempt by other parties to provide cheaper medication. If the intellectual wherewithal had been entirely lacking in the Third World, they may well have got away with it. But Indian pharma companies were able to produce AIDS medication, and more importantly, produce it at a much lower price point that African countries could afford, and the Third World as a whole managed to vote their way around the IP laws that the US (and other Western countries) have forced all others to sign. A humanitarian disaster was averted that would have dwarfed the Holocaust, and India had a large (though also largely unsung) role to play in averting it.

More recently, Tata Motors established a radically lower price point for cars ($3000). The Nano is a revolutionary breakthrough, and it isn't a toy either. If a car can survive on Indian roads, it will positively thrive anywhere else ;-). The Nano is going through a few teething problems right now, but I have a sense that in a decade or two, it will be one of the world's iconic car brands, perhaps the most ubiquitous. The Nano could change the image of car ownership as an indication of wealth.

And now, with Aakash, India is once again bringing technology within the reach of ordinary people, at Third World prices. The term "reasonably priced" means something very different in Western countries. Even middle class people in Third World countries cannot afford these "reasonably priced" products. In Marxist terminology, as long as the "means of production" were concentrated in Western hands, there wasn't much the rest of the world could do about it. They either paid those prices (exorbitant by their standards) or simply did without. Now they have a choice. Incongruous as it may seem, India is swooping in in shining armour to save the day.

I do have some reservations, though. Indian ingenuity has never been in doubt. What's in doubt is India's institutional ability to follow through, to execute, to deliver. India has always been a muddle-through country rather than a reliably-achieve country. Even the Nano is a case in point. The political shenanigans that preceded its launch very nearly canned the project. If I was someone big in the Indian government, I would not look at this as just a private sector enterprise that is none of the government's business. I would see it as a national enterprise where India has a chance to put its stamp on the world and change it for the better, and I would provide Tata Motors with all the support needed to manufacture and sell the Nano in volume, worldwide.

[I'm not going to argue about whether an invention that serves to consume more fossil fuels is going to change the world for the better. The environmentalists don't seem to have an easy answer to the developmental issue of stagnation-versus-pollution either.]

Again on the topic of the Indian character, I remember my student days at IIT Madras when a new Siemens mainframe computer was delivered to the institute. This was in the mid-eighties. The box was too big to go up the stairs of the computer centre. Many students and professors watched as the workmen, mostly uneducated, rigged up a makeshift pulley and with the help of ropes, winched the box up the side of the building, and others leaned over the parapet wall on the first floor and hauled it in. Mission accomplished with a minimum of technology (and no insurance!)

Watching this, one of my professors who had done his PhD in the US and worked there for a while, remarked wisely, "We Indians are great at improvisation. The danger is that we will be satisfied with our ability to improvise, and fail to develop real systems."

I see no evidence that India has improved significantly on the systems front. Flashes of Indian brilliance, like the Nano and the Aakash, will remain just flashes in the pan unless the country learns to be more disciplined about delivery. Virtually every developed country emphasises delivery discipline. Only when India masters that can the world look forward to a steady stream of dirt-cheap high technology that will change billions, not just millions, of lives.

Meeting of Android Australia User Group - Sydney 05/10/2011

2011-10-05T04:48:00.000-07:00

I attended my first meeting of the Android Australia User Group - Sydney this evening.

The organiser, John Scott, started with some general announcements, which should be of interest to many.

- There is an Amazon Cloud Drive Developer Tech Talk in Sydney on Monday October 10, 2011.

- Google Developer Day will be on November 8, 2011 in Sydney.

Those interested should register for these events. They're both free, as far as I know.

John then kicked off the first talk of the evening, which was a brief overview of Motorola's IDE for Android application development - MOTODEV Studio. According to John, this IDE is better than the Google Android plugin for Eclipse, which seems to be the de facto IDE for Android, because it has a few extra features like the generation of boilerplate code to make the developer's life easier, and also graphical tools to manage SQLite databases. I found a couple of other discussions on MOTODEV Studio vs Eclipse here and here, but they seem somewhat dated.

The second talk was by Gianpaolo De Biase, a developer with AppCast. Gianpaolo talked about some real-life applications that his company has built (one of which is JustStartWalking, an app for an initiative of the same name from the Chiropractors' Association of Australia). He also discussed two important architectural decisions that his company made when developing these products.

1. When dealing with local databases, Gianpaolo recommends using an Object-Relational Mapping (ORM) tool rather than the raw SQLite API provided by Google. The tool he used is OrmLite, which is Open Source like Hibernate but more lightweight and adequate for the simpler data structures needed for local storage on mobile devices.

2. When making REST calls to remote servers, he recommends using the Spring Android module with its REST Template in preference to raw HttpClient. He did encounter some bugs with Spring Android in the areas of cookie management and SSL certificates, but believes the product is rapidly maturing.

As a side benefit of both OrmLite and Spring Android being annotation-based, Gianpaolo was able to have a single set of domain objects. Each object had two sets of annotations applied, one for persistence and one for interfacing with REST services. [I'm a bit suspicious of the latter annotation, since it looks like the design combines domain objects and message documents into a common entity, a form of tight coupling I've long warned against.]

After a short break, we had our third talk of the evening by James Zaki, a freelance developer with goCatch. This was a high-level description of the goCatch app, which brings together cab drivers and prospective passengers. There was general satisfaction expressed around the room in favour of this cartel-breaking app, since the taxi companies and CabCharge engage in significant rent-seeking behaviour at the expense of both drivers and passengers. goCatch allows the two to bypass the middlemen. I had some reservations about one of the aspects of the goCatch design as described by James, i.e., its statefulness, which led to problems of synchronisation of state held on devices with that on the server. Perhaps a suitable set of messages based on idempotence could solve the problem. I didn't have time to discuss this offline with James.

Our fourth and final talk of the evening was by Darren Younger, CTO of IPScape, in whose offices the meeting was held. IPScape is a provider of cloud-based contact centre solutions catering to both voice and web. One of their interesting applications allows mobile device users to make phone calls not through the device's native telephony capabilities, but through the IPScape app. The server then initiates regular (teleconference-style) phone contact with both the caller and the receiver. The advantage of this is that the server can record the call. Many financial service providers are required by law to record all customer conversations, and it is easier for them to use this app rather than approach the telcos for a voice recording service. A developer API may be coming in a few months.

I also met another attendee, Nanik Tolaram, an amateur Android enthusiast with his own Android-related website.

I picked up a few useful tidbits of information over the course of the evening.

Samsung sent a couple of people to the meeting. They seem keen to understand the size and strength of the Android developer community. Samsung wants to carve out a unique niche even within the Android ecosystem and they have their own app store separate from the generic Android one.

DroidDraw is a testing tool for Android User Interfaces.

Google has a cloud-to-device API.

developer.android.com is Google's portal for Android developers.

Balsamiq is a commercial tool to mock up UIs, including mobile UIs. Pencil seems to be a good Open Source equivalent, and Lumzy is a free one.

Two of the presenters talked about their negative experiences with outsourcing. Although the outsourcing countries were as varied as Israel, India and Singapore, there seem to be some common problems caused by distance as well as the seeming cultural inability of some developers to look beyond the literal specification and to understand the higher abstraction that an application is trying to implement. Errors in the documentation of some specs led to literal implementation of those features even though they patently made no sense. Outsourcing sites like Freelancer.com seem very cost-effective, but the elapsed time to obtain a working solution negates those benefits. Examples: $200 and 4.5 months to develop an app, $800 and 9 months to develop another. The moral of the story seems to be to hire good local developers so communication problems are reduced and results are achieved quickly.

The Android Australia User Group is a good place for developers to hang out. The organisations that some of the speakers represented are looking for developers, and this may be a good way to get introduced.

Daylight Saving Time - Ubuntu Passes the Test

2011-10-01T09:05:00.000-07:00

60 seconds after 0159 on October 2nd is when New South Wales, Australia mysteriously loses an hour to Daylight Saving Time and the time becomes 0300.

I was awake to record what our computers did. My wife's Windows 7 machine showed 0300 all right. I would have been surprised if it hadn't. Microsoft needs to show some degree of competence for all that market share!

What was personally gratifying to me was that my Ubuntu Linux machines also neatly skipped the hour and showed 0300.

I'm going to bed now with a smile.

Facebook's Secret Sauce

2011-09-30T06:19:00.000-07:00

I would say there have been Three Ages of the Internet.

The first was the Internet era proper, where only the US government and universities (and maybe a few other organisations) had it. There was email of course, and really cool stuff like Archie and Gopher (whatever they were) that you could access on text-based terminals.

Then there was the Web, which was nice and graphical and coincidentally came out at roughly the same time as the extremely popular Windows 95. The Web through Windows made the Internet something for regular folks, not just geeks. There was surfing and search, pleasurable as well as useful. The Web soon gobbled up email too (webmail changed the user-facing protocol from SMTP and POP to HTTP). And blogs became a way for the little guy to communicate his own views to the world instead of just consuming the output of established media. That was the first step towards mass participation.

Then there came Facebook. More than Picasa and on par with Skype, Facebook has suddenly made the Internet a must-have for everyone. I'm betting Facebook and Skype have driven Internet usage to new highs, both in terms of bandwidth consumed and in terms of market penetration. And along the way, Facebook has sucked the oxygen out of blogs. Ask me. I should know.

That deserves the title of 'third generation Internet'.

Skype's appeal is easily understood. It's the videophone of science fiction that the Telco monopolies never gave us. (Thanks, guys.) But what is it with Facebook? If it's just a place where friends hang out and exchange news and funny stories, would it really have become all that big?

I've been thinking a lot about this, because I'm a latecomer to Facebook, having resisted it for a while. Now I find I'm thinking of it as 'Wastehook', an addictive way to spend time that I later regret. What made Facebook so addictive to a person who resisted it so much to start with?

Yes, it's cool that we can keep tabs on all the people we've ever known. But there's more to it than that.

I think the one thing of absolute genius that Facebook has pioneered is the 'friends of friends' concept. 'Friends of friends' gives you visibility just beyond the horizon of your circle of contacts. You get to know of people who are somewhat interesting because of the friends you share. Sometimes, they turn out to be people you know too! Bonus points for the thrill in such cases.

'Friends of friends' are people our friends like and trust, and since we like and trust our friends, their friends are people we're already favourably disposed towards. We don't mind reading the things they say. Sometimes, they say witty and wise things. Facebook is a lot more interesting because of these people. It's not just our boring and predictable friends. It's these people, unknown but not quite irrelevant, who bring a bit of variety to the experience. It's somewhat interesting when a friend puts up photos of themselves, but a lot more interesting to read what their friends have to say about it, even if we don't know many of them. We can join in the conversation and politely add to what they're saying, and nobody minds. A nice polite party with decent people we could be friends with. That's Facebook.

'Friends of friends' prevents Facebook from becoming a stale backwater, a stagnant pond, an eddy in a stream. It's not quite the wide blue ocean, though. It's just a safe little harbour. 'Friends of friends' expands our circle just enough to make Facebook an interesting place to hang out, but not so much that it becomes overwhelming or irrelevant.

I'd say that's Facebook's secret sauce.

[What about Google+? Well, Google+ pioneeered categories of friends, but Facebook quickly neutralised that advantage. An authoritative source informs me that Google+ has a rough equivalent to 'friends of friends' based on followers and following, so the effect could be similar. But my prize for pioneering the next generation of the Internet goes to Facebook in any case, not Google+.]

Does Redis Undermine a Key REST Tenet?

2011-09-23T09:18:00.000-07:00

While I have always admired the elegance of REST's resource model and its standardised verbs and status codes, there are two things about the RESTful style that I think are drawbacks when building complex applications. One is its client/server model rather than a peer-to-peer model, which I think would have been more generic and useful, especially when unsolicited notifications need to be supported. The second, which is what I want to talk about now, is the insistence on statelessness (or alternatively forcing anything stateful to be either managed by the client or modelled as a resource). This is an acceptable approach for application domain-specific elements, but there is also a domain-neutral class of session-stateful elements that are useful for providing Qualities of Service (such as a sliding window of message sequence numbers for reliability, or session keys for security). These are common requirements that any application may require. This kind of session state is fundamentally different from domain-specific state, e.g., a shopping cart in an e-commerce application which would not be relevant to any other type of application.

The lack of support for this application domain-neutral subset of session state within the RESTful style of design costs us the ability to provide Qualities of Service in a uniform way (which the much-disdained WS-* is able to do using standards like WS-SecureConversation/WS-Security and WS-ReliableMessaging). [Many will point out that RESTful applications can use SSL for security, but the SSL protocol is not itself RESTful, and it breaks the cacheability and routability of REST operations.] So Qualities of Service end up becoming an application responsibility in RESTful applications, which has always struck me as a bit of a cop-out on the part of the protocol.

REST has standardised many other aspects of interaction and turned them into a uniform protocol that interacting systems can use. I would like to see QoS-related session management also similarly standardised and "pulled into the protocol", even if that protocol is not HTTP but something else (WebSockets?).

I have often wondered exactly why statefulness is considered "evil". There are two reasons, as far as I can tell - scalability and failure recovery. Session state occupies memory, so holding session state for multiple clients impacts the scalability of a server. Also, holding session state within a server makes it less easy for clients to switch to another if the one holding its session objects goes down. We'll need to have session-aware clusters rather than stateless farms of servers to provide failover, which again doesn't scale linearly.

Two considerations make these arguments weaker.

The first is that we are not talking about application-specific session state like shopping carts and the like. We're talking about relatively tiny data elements that are domain-neutral and dedicated to providing Qualities of Service, such as sliding windows of sequence numbers for reliability, and security tokens for message encryption. These aren't huge, perhaps a few hundred bytes at most.

The second is that the advent of NoSQL databases has given us a way to delegate the storage of session state. It's now becoming recommended best practice to store session state in a NoSQL database like Redis rather than in memory. Delegating session storage is an alternative to session-aware clusters, since the servers can now be a simpler stateless farm and access the NoSQL database for shared session state. What's impressive about Redis in particular is that the complexity of its GET and SET operations is claimed to be of Order(1), i.e., constant time. This means that (in theory at least) one can increase the number of servers using a Redis datastore indefinitely with no impact on performance.

Now, a horizontally scalable farm of servers can share a Redis datastore and use it to hold the individual sliding windows of message sequence numbers for a number of clients, and also their security tokens. Scalability is no longer a problem, because the data elements are small and the storage/retrieval operations are of constant-time complexity. Failover is also not a problem since the servers themselves are stateless and switchable at run-time.

It would be good if a standard and application domain-agnostic mechanism could be evolved to provide security and reliability to REST-based interactions, using Redis-style scalable session storage. This may be the long-awaited successor to REST, which Rohit Khare's ARRESTED style attempted to do (but which has no practical implementation to date).

That will then leave client/server as my only bugbear with the RESTful style. We'll need another extension to handle that.

The Return of JavaScript - My Thoughts Crystallise

2011-09-21T20:06:00.000-07:00

I wrote some months ago that JavaScript was back. I've been watching developments in that space with close interest, and a few glimmerings of "the right way forward" have begun to suggest themselves to me.

The first aspect pertains to the language itself. The JavaScript Renaissance has brought with it a desire to do things right the second time around. JavaScript doyen Douglas Crockford has lectured extensively (and even written a book) on the need to use only "the good parts" of JavaScript and avoid the bad ones. It's excellent advice, but to a new generation of JavaScript programmers, I suspect it may not be enough to post warning signs around bad features. We need a way to make it physically impossible to use JavaScript's "bad parts".

Enter CoffeeScript. While I haven't played with it too much, I like what I see here. [Update 24/09/2011 - I *have* played with it for a few hours now, and I really like!] There has been some skepticism about CoffeeScript from a section of the JavaScript old guard, but that's to be expected. The best answer to such skepticism is from ThoughtWorks's latest Technology Radar document, which is well worth a read in itself.

First, even if it seems a trifle off-topic, read what Thoughtworks thinks about GWT, which is a way to write visual components in Java and have JavaScript-based widgets automatically generated for a web application that runs within a browser:

GWT is a reasonable implementation of a poor architectural choice. GWT attempts to hide many of the details of the web as a platform by creating desktop metaphors in Java and generating JavaScript code to implement them. First, in many ways, JavaScript is more powerful and expressive than Java, so we suspect that the generation is going in the wrong direction. Secondly, it is impossible to hide a complex abstraction difference like that from event-driven desktop to stateless-web without leaky abstraction headaches eventually popping up. Third, it suffers from the same shortcomings of many elaborate frameworks, where building simple, aligned applications is quick and easy, building more sophisticated but not supported functionality is possible but difficult, and building the level of sophistication required by any non-trivial application becomes either impossible or so difficult it isn’t reasonable.

Now read what Thoughtworks says about CoffeeScript:

Some readers may be confused by our advocacy of Coffeescript given our general dislike for GWT, because on the surface they seem similar: tools that generate JavaScript. However, it is the level of abstraction that differs. GWT has an elaborate component model, which tries to hide details about the underlying language (JavaScript) and platform (the web). Coffeescript tries to make it easier to write proper JavaScript, avoiding pathological but default “features” of JavaScript, and does not build a layer that tries to insulate you from the platform.

I think they've nailed it. I have been ambivalent about GWT for a while, but couldn't quite put my finger on why (i.e., it's a "reasonable implementation of a poor architectural choice"). They've also very cogently argued why CoffeeScript doesn't fall into that category.

So my first takeway from the JavaScript Renaissance is that today's developers should code in CoffeeScript and not in JavaScript itself. If Java and Scala are two languages that compile to Java bytecode and run on the JVM, then think of JavaScript and CoffeeScript as two languages that "compile to JavaScript" and run within a JavaScript engine. The runtime behaviour is unchanged, but you use the language that works better for you as a developer.

CoffeeScript is JavaScript with the sharp edges removed. I'm sold. [There is an issue with E4X, which I'll cover when I talk about Nodejs, but it applies equally to CoffeeScript. In short, there's no support for E4X, and I want it!]

Beyond the choice of language, the direction is not so straightforwardly clear. It's tempting to say that the future of JavaScript on the client is CoffeeScript plus jQuery, and the future of JavaScript on the server is CoffeeScript plus Nodejs. But there are some issues.

To paraphrase Thoughtworks's critique of GWT, jQuery is an elegant way of working with a horrible object model (the DOM). I don't know if there is any way to replace the DOM with a better client-side abstraction, but until then, the best of client-side goodness will remain somewhat unsatisfactory.

Nodejs is the server-side angel in white from all accounts, but I've heard some valid grumbling about Node as well. The Spanish outfit ASPL has a BEEP implementation called jsVortex. This JavaScript-based server pre-dates Nodejs, and ASPL was reportedly keen to explore Node as the underlying platform for the next version of jsVortex. However, they quickly realised that Node had no support for TLS at its core. This is not the same as supporting HTTPS, because HTTPS support obviously exists in Node. Think of TLS as HTTPS-like capability for any protocol, and since BEEP is a framework to create protocols à la carte, and since security is a ubiquitous and pretty strong requirement for most protocols, the lack of TLS support within Node is a serious deficiency for them and for anyone who wants BEEP on the Nodejs platform.

[Most people might go "BEEP who?", but in my opinion, this is a technology that was sadly delayed by over a decade, perhaps because of personality politics caused by creator Marshall Rose's less-than-smooth interactions with powerful members of standards bodies, who then blocked BEEP from becoming a standard until very recently. If we'd had BEEP 15 years ago, I suspect SOAP and WS-* would have looked very different, and perhaps REST would have looked different too. That's how powerful it is, and I still think it will change our lives one day.]

Well, so there's a piece of work to be done to enable TLS within the core of Nodejs, but that's not all.

WSO2's Mashup Server was another under-appreciated piece of server-side JavaScript from many years ago, and one of its best features was its support for E4X. It's understandable why WSO2 would be keen on E4X, because their enterprise middleware suite is based on SOAP/XML, and a JSON-only approach would hinder interoperability between Mashup Server and the rest of their product suite. Like everyone else in the server-side JavaScript space, WSO2 is looking at Nodejs as a possible underlying platform for the next generation of Mashup Server, but the leading lights of Node are reportedly not keen on supporting E4X at all. I wonder what the problem is. Surely supporting XML (through E4X) in Node wouldn't come at the cost of supporting JSON! At least the CoffeeScript guys have an excuse for not supporting E4X (not that I'm letting them off the hook for that reason!) because not all browsers support E4X and they don't want to be browser-specific. But Nodejs? I think the throat to choke is V8. If V8 supports E4X, then so will Node, and then there'll be pressure on CoffeeScript to support it too, since both Mozilla and V8 do. I asked them for this 3 years ago, but no dice. Ahem, Google...?

Then there are the various flavours of web frameworks for Nodejs (Backbone, Express, Geddy, etc.), but where is SpringFramework.js? Where is the Grails, Roo or PlayFramework equivalent? I certainly hope server-side JavaScript revs up to an equivalent level of maturity as server-side Java double quick. [Server-side Java meandered a fair bit and had many mis-steps along its journey, so server-side JavaScript could get there far faster by learning from Java's mistakes.] Judging by the level of activity in the Nodejs ecosystem, we need not fret on this account. It's just a matter of time before the current effervescence and flux give way to a stable yet rich multi-layered platform.

So that's my current view in a nutshell:

* Use CoffeeScript and not raw JavaScript from now on

* jQuery is great on the client side, but will no one rid me of this turbulent DOM?

* Nodejs is The One on the server side, but lacks some critical features (e.g., TLS and E4X). I don't know if they know about broken TLS, but they don't seem to think E4X even belongs in the to-do list, which is a pity if true. I'd like to be proved wrong.

* We need better server-side frameworks, and these are already on the way. We just need to wait till the dust settles to pick a winner.

I'll post updates as my view evolves.

The Zen of Tight Coupling and Loose Coupling

2011-09-20T23:28:00.000-07:00

After my talk on SOA at WSO2Con 2011, I was inspired to create this fun, Zen-style instructive presentation on what really constitutes tight and loose coupling.

Enjoy!

A Geek's Dream Address in Colombo, Sri Lanka

2011-09-18T06:54:00.000-07:00

With apologies to Freddy of My Fair Lady:

I have often walked down this street before;
But the pavement always stayed beneath my feet before.
All at once am I several stories high.
Knowing I'm on the street where you live.
[...]
People stop and stare. They don't bother me.
For there's nowhere else on earth that I would rather be.
Let the time go by, I won't care if I
Can be here on the street where you live.

This road was very close to the Cinnamon Lakeside Hotel where I stayed when I was in Colombo to attend WSO2Con 2011:

That's the address I'd like to have if I ever relocate to Colombo :-)

Back from WSO2Con 2011

2011-09-18T01:03:00.000-07:00

I'm back home in Sydney after a very enjoyable week in Colombo attending WSO2's 3-day conference as well as the pre-conference and post-conference tutorial sessions flanking it.

Over the next few days, I'll be posting a detailed trip report with my impressions of the highlights of the conference. In this post, I want to record my overall impressions.

I was highly impressed by a few things.

One, I could not but be impressed by the rapid rise to prominence of a visionary software company. The "Open Source middleware" niche has only one serious player with a complete range of offerings. That's WSO2, a five year old company. This relative newcomer is giving established players like IBM, TIBCO and Oracle a good run for their money. I have no doubt that middleware prices will come down in a few years from their current extortionate levels to something much more reasonable, and the credit will go in large measure to WSO2's disruptive impact on the market.

Two, I also had to marvel at the achievements of a country. I spoke to a few Indian delegates at the conference, and we all agreed that in spite of the huge number of Indian IT professionals and some very large Indian IT companies, no one in India has achieved (or even attempted) what a relatively small Sri Lankan company (WSO2) has pulled off. In hindsight, it shouldn't have been too hard. Take some software products from the Apache stable, enhance them to make them usable by corporate customers, release the lot again under an Apache licence, and charge customers for just support and professional services. It takes some vision of course, but a lot more courage and conviction, and WSO2's CEO Sanjiva Weerawarana has that in spades. It has been his determination that Sri Lanka should be a leading Open Source nation, not just in terms of usage but in terms of contribution, that has led to the success of WSO2. Today, I'm told Sri Lanka ranks third in the world in terms of Open Source contribution, behind the US and Germany. It would be good if Indian companies and professionals took inspiration from their smaller neighbour and started contributing to the world through Open Source.

I was also very glad to see the number of Sri Lankans (not just WSO2 employees but also others from Universities and the like) using Ubuntu Linux on their laptops. I felt right at home after a long time of appearing like a curiosity. Indian and Australian professionals, on the other hand, seem to have sold their souls to Microsoft (or else to Apple). Any argument that Linux is "not there yet" is just an excuse. These people have been using Linux exclusively and are none the worse for the experience. They also have a bit more cash left in their pockets as a result of their choice :-).

Three, I was very impressed with the way the conference was organised. Everything went like clockwork. Events started and ended (largely) on time with very few overruns. Sri Lanka follows Indian Standard Time, but fortunately, the other connotations of the term have not been infectious ;-).

There were distinguished speakers from around the world, giving the conference an international flavour. This was not just a Sri Lankan affair.

IBM Fellow C Mohan giving the main keynote speech of the conference

Gregor Hohpe, author of Enterprise Integration Patterns, giving another keynote speech on the final day of the conference

eBay's Distinguished Architect Sastry Malladi gave a talk and participated in a "fireside chat" to talk about, among other topics, eBay's use of WSO2's ESB to process a billion transactions a day

David Schumm of the Institute of Architecture of Application Systems (IAAS), Stuttgart, talking about BPM research at his institute based on WSO2 Business Process Server and similar products

Dmitry Lukyanov, Head of Integration Solutions, Alfa Bank, Ukraine, talking about his bank's positive experience with the WSO2 middleware suite

Cognizant's Principal Architect Dipanjan Sengupta's presentation demonstrating the interoperability between IBM's WebSphere Service Registry & Repository (WSRR) and WSO2's Governance Registry, was very well received

Catering was top-class, though some of the Western delegates found the food excessively spicy. (I had no complaints, except that I have probably put on some weight ;-). The hotel accommodation and local transport were great and there were no snafus of any kind. It was testimony to the conscientiousness and attention to detail that the people involved displayed.

Four, I realised that the conference was a showcase not just for WSO2's technology but also for Sri Lankan culture. Every evening, after the main technical events were over, there was some cultural program or the other that was fascinating. Day One had an elephant ride (with people able to feed the elephant by hand before the rides got started).

There was also a very interesting talk and slideshow by a conservationist on "human-elephant conflict" and possible solutions, with very moving and often tragic stories about individual elephants. Then there was a set of folk dances that I can hardly do justice to with mere words or even pictures.

Day Two's cultural event was a Jam Session, where WSO2's employees enthralled the crowd with their musical talent.

That, and the musical talents of CTO Paul Fremantle and VP (Business Development & Product Design) Jonathan Marsh, made me think one needs to have musical ability as well as software competency in order to work at WSO2!

Fremantle and Marsh present a musical mashup (In India, it would be called a Jugalbandi)

On Day Three, a well-known local group "Bathiya and Santhush" (rather like Shaan or Hariharan in India) performed and the whole crowd danced to their music. The lyrics were in Sinhala, but the beat had universal appeal.

That was WSO2Con at a glance, and I'm sure next year's event will be even bigger and better. Those who missed this year's experience should try not to miss the next!

(Disclosure: I am in advanced negotiations with WSO2 to join them in a suitable role reporting to the CTO. While this obviously makes me an interested party, I promise not to make any statements or representations about the company and its products that I do not sincerely believe in. My decision to consider a position with WSO2 is itself driven by my conviction that (1) their product suite is fit for purpose and very good value for money, (2) the company operates in accordance with the ideals of Open Source - there is no proprietary version of any of their products, and (3) there is a good cultural fit because I like the people and the company culture (they never use the word "resource" for people) and found myself getting along very well with people across the organisation. Regardless of my relationship with the company, I'm determined to remain a voice that readers of this blog can trust, because I'm acutely conscious that my independent stance on technology matters has always been my biggest source of credibility as a commentator.)

Gregor Hohpe's Talk at WSO2Con 2011 on Enterprise Integration Patterns

2011-09-15T17:46:00.000-07:00

I had the pleasure of attending Gregor Hohpe's keynote speech on the third and final day of WSO2Con in Colombo. (The conference was a hugely successful affair, by the way, judging by the quality of the talks and the fact that the attendance far exceeded expectations.)

Gregor's talk was a reference to his famous book "Enterprise Integration Patterns", and was titled "Enterprise Integration Patterns - Past, Present and Future". (For all his achievements and fame, Gregor is an endearingly unassuming man, as I found when I spoke to him person-to-person. He's also a very humorous speaker, and kept the audience in titters during his keynote.)

When he came to the Future part of the talk, one of the many gems he let slip was his insight that his famous book was really about Messaging Patterns, in which case it would be the first of four volumes (the other three devoted to Conversations, Processes and Events). He specifically called for two things from the audience and the community in general - some ideas on notation for these more complex patterns, and for exerting guilt-inducing pressure on him so he would actually write the books!

I have been interested in the Distributed Computing area for a long time (I would in fact have called these "Distributed Computing Patterns", because they are not specific to enterprise contexts, and the purpose of the interactions isn't always integration).

In any case, this is how I see the four volumes of the book that Gregor described, and how they would relate to each other. I would like to mail him on these ideas and I'm sure I'll have an interesting conversation.

"Messaging" is really the most generic description of a class of distributed computing systems. It also takes a top-down, system-wide focus (a "God" view). One can think of specific subsets of these generic interactions, and one can also shift focus from the overall system to a particular node. So, as we add a statefulness aspect to messaging, we begin to enable conversations. And as the purpose of the conversations becomes coordination (one participant controlling the others), we have (process) orchestration. Somewhere along the way, we can explicitly consider multi-way messaging, i.e., between multiple peers.

When we shift focus to a single node, we have an event-driven system because incoming messages at any node are "events". And when we take a set of event-driven nodes and consider them in aggregate, we have a choreographed system, where each node is ignorant of the existence of other nodes and only responds to events, and the system as a whole "works" because of these inter-related events. The design of a choreographed system is really the design of appropriate events, since the previous step (event-driven systems) has already designed the behaviour of the individual nodes.

So that's my simple view of the Distributed Computing universe. I also have a more complex diagram that I created a few years ago to explain how the various facets of distributed computing are related.

Preparing for WSO2Con 2011

2011-08-30T02:56:00.000-07:00

Well, it looks like I'm going to be attending WSO2's annual technical conference WSO2Con in Colombo this September (12th-16th). It'll be good to hear speakers from around the world speak about SOA-related topics and the WSO2 product suite in particular. I believe the latter is one of the SOA industry's best-kept secrets. Here is a comprehensive suite of middleware products that is completely Open Source, and for which commercial support is available at very reasonable rates. I'm surprised more organisations aren't using it. Perhaps the awareness wave is just breaking and we'll see a surge in adoption in the next couple of years. Let's see.

I'm also slated to conduct an introductory tutorial on SOA on the first day of the conference, and I'm excited about that. Over the last few years, I've been developing a few ideas on "Practical SOA for Solution Architects" that I'd like to present, and I'm eager to get feedback on that approach.

What I've seen to my disappointment over the past several years is that in spite of extensive coverage of SOA in the IT industry, the actual payoff from the application of SOA is embarrassingly low. In fact, I hear from sources in large IT shops that have rolled out multi-million dollar SOA initiatives that the cost of building services today is higher than in the past when older EAI techniques were used.

I've been puzzled at this, and have discussed it at length with colleagues and peers. It looks like three different factors are at work.

Cynicism: There is a view among many IT professionals that SOA is yet another buzzword that has come and gone, and it's no longer relevant to the industry. So they don't even bother to learn what it's about.
The theory-practice gap: Lots of IT folk have been exposed to core SOA concepts ("Loose coupling"), but they don't seem to be able to apply it in real-world solution designs. SOA remains in the realm of theory, which is a pity, because it can be a real money-saver.
Products as surrogates for principles: Organisations have collectively spent a packet on SOA products from IBM, TIBCO, Oracle and the like, but the actual solution designs being produced in-house for business projects are as tightly-coupled as ever, suggesting that IT practitioners seem to think the use of SOA products is an adequate substitute for Service-Oriented application design.

I have come to the conclusion that what's required is a new approach to educating people about SOA. It has to be less theoretical and more focused on real-world situations. And it has to be lightweight, because people are busy and don't have the attention span required for a heavyweight methodology. Above all, it has to be sound, because we don't want to spread incorrect practices in the name of simplicity.

I'm preparing a whitepaper on this, and it will probably be released through WSO2, since a lot of the encouragement for it has come from Sanjiva Weerawarana, WSO2's CEO. It's only fitting that WSO2 get the credit for bringing this educational tool to a wider audience.

I'll also be posting my slides from the conference on this blog when I'm done. I'm sure they'll also be available from the WSO2Con conference website.

Stay tuned.

Identity Management on a Shoestring - The LIMA Approach

2011-07-31T08:20:00.001-07:00

Umesh Rajbhandari and I have documented the design of an Identity and Access Management (IAM) system that we successfully built for a large Australian corporation. We call our unique architectural approach "LIMA" (Lightweight/Low-cost/Loosely-coupled Identity Management Architecture).

The document describing LIMA is here.

Practical SOA Building Blocks

2011-05-23T23:29:00.001-07:00

I'll make this short. This post is not a theoretical discussion of what SOA is. This is a practical guide to the core physical components required to build a "traditional" SOA-based ecosystem.

I believe there are three basic components required - a Service Container, a Broker and a Process Engine.

A Service Container is used when implementing business logic to expose as a service.

A Broker is used to connect to legacy business logic through mediation, routing or transformation.
A Process Engine is used to coordinate two or more services into a process and expose that as a composite service.

We put these together in different combinations to build out the specific business processes we need. Needless to say, what we run on these components should be reusable, so their interfaces need to be generic, yet extensible. Good data design is required for this.

The Return of JavaScript

2011-05-18T07:30:00.000-07:00

Hardly had I begun to play with Node.js when a friend pointed me to this news item about JavaScript being used to run Linux.

Déjà vu all over again.

Some months ago, after the perceptible pall that descended on Java with the Oracle takeover of Sun, a group of tech enthusiast friends of mine opined that the time was ripe for a new technology. Java was now legacy. It would continue to grow and influence, of course, but its glory days were over. Java had been a shooting star under Sun. Under Oracle, it was now a cash cow. The next paradigm shift would therefore have to come from a new technology altogether. Much as we might pride ourselves as technology futurists, we only struggled to pierce the veil. We spoke about Android, cloud computing, NVIDIA processors and Web 3.0, but none of us imagined that the revolution would come from an established and very familiar technology.

JavaScript is as old as the web, and born close on the heels of Java, which explains its confusingly similar name. (Well-meaning managers at leading technology companies were known to discourage programmers from writing JavaScript because they heard that the corporate firewalls were configured to block Java applets. Microsoft evangelist Stephen Jeffries even ruined a perfectly credible sales pitch for Microsoft technology in 1999 by claiming to have written "some Java scripts", which were inferior to his company's offerings, of course.)

Naming confusion aside, not many will remember that in the early days of the web (circa 1995), Netscape not only had a browser (the legendary Navigator), but also a web server, a directory server, a certificate server and many other components of a web-based ecosystem. JavaScript was a Netscape invention, and the Netscape web server could run Java servlets as well as JavaScript code. There was an attribute that could be used with the "script" tag in HTML, which was "runAt = 'server' ". JavaScript running on the server - imagine that!

But that initial innovation died out shortly after. JavaScript became a purely client-side phenomenon. Other languages, notably Java and PHP (and ASP in the Microsoft world) took over the server side.

The next time I heard of JavaScript running on the server side, it was in the context of WSO2's highly innovative Mashup Server.

Server-side JavaScript was still a rare phenomenon, and had the faint aroma of a toy system to boot. After all, serious software was written in Java. JavaScript was at best useful to demonstrate clever ideas. It wasn't production strength.

Now along comes Node.js with its innovative thesis about how to do I/O right. Ryan Dahl explains the philosophy in a way that appears obvious in hindsight. When you see the numbers he puts up comparing the CPU cycles for different levels of I/O, it becomes extremely clear that it's criminally wasteful of computing resources to wait on disk and network I/O.

L1 cache: 3 cycles
L2 cache: 14 cycles
RAM: 250 cycles
Disk: 41,000,000 cycles
Network: 240,000,000 cycles

Node.js boasts a single event loop which, combined with its absolute refusal to block on any external I/O operation, allows it to scale up to ridiculous levels of concurrency. Add to that the simplicity, flexibility and familiarity of the JavaScript language, and the runaway popularity of Node.js becomes understandable. I'm personally blown away.

Even Ryan Dahl, its inventor, struggles to explain what Node.js is (beyond the obvious fact that it's a bunch of libraries on top of Google's V8 JavaScript virtual machine). That's because Node.js is nothing less than a new platform for innovation. It's clay in the hands of a new generation of developers, and fascinating new products (Node.js modules and frameworks) are emerging.

I think one of these will be a rich GUI library to rival the offerings from Adobe. It will be possible to build extremely flexible and lightweight client apps with rich user interfaces very soon. Web forms will soon seem clunky, because we can soon have web applications without web forms.

Another phenomenon that Node.js will enable is peer-to-peer systems. I believe we've outgrown the synchronous request-response behaviour of the traditional web. Something new and big is coming soon.

So move over, Java. Here comes JavaScript.

The Magic of Node.js

2011-05-03T06:56:00.000-07:00

A while ago, Google came out with a high-performance JavaScript engine called V8. This is what powers the Chrome browser.

Now along comes an Open Source project called Node.js (or simply Node) that wraps V8 with networking capabilities, enabling some very cool JavaScript-based servers (HTTP or TCP) to be written with just a few lines of code. Servers written for Node are non-blocking and multi-tasking without having to be multi-threaded (the demo below explains more), which also makes them very fast and scalable. This is server-side JavaScript at its best.

This hour-long demo is worth watching. I haven't been so amazed since I saw Scott Davis demonstrate Groovy at JavaOne 2009.

The only downside (not to me, though ;-) is that Node currently only works on Unix-like systems (Linux, Mac) and not yet on Windows. You also have to compile from source (no binaries yet), which is not a big deal because you only have to type "./configure". "make" and "sudo make install" one after the other in the main directory once you've unzipped the tar.gz file using "tar zxvf node-VERSION.tar.gz".

Refcards has a reference card for Node.js.

There are some great possibilities here, such as when you combine this with E4X.

How Disruptive Innovations Can Upset Monopolies

2010-10-19T04:10:00.000-07:00

Monopolies give me heartburn, and what warms my heart is the prospect of monopolies being overthrown.

Two Australian examples of the latter were in the news recently.

Monopolist Telstra is feeling the heat from mobile phones. The exodus of customers giving up their fixed lines is turning into a stampede. In a few years, landlines will seem an anachronism. Apparently, there is higher demand for speed and data that is "difficult to monetise". Translation: It's impossible to charge monopoly rents on commodities. Hasta la vista, Telstra.

Australia Post is being squeezed by email. It's amazing that a 4.2 percent fall in postal volume can result in a 65 percent fall in profit. Of course, there were other costs as well, but this is dramatic. With more and more communications going electronic, it's hard to see how Australia Post is going to stay relevant. There'll be electronic media, and there'll be courier services, with nothing left over for the postal monopoly. We live in a post-post world.

The takeaway for me is that monopolies may stifle innovation up to a point, but a wave of innovation of a higher order will eventually overthrow them.

My cup of joy runneth over.

HereDo - An Idea for a Mobile App

2010-08-02T02:48:00.000-07:00

I don't know if this has been thought of before, but it's an app I would appreciate.

I enter a list of tasks into my mobile device, along with a hint as to where I need to do them (e.g., I need to buy a bicycle pump the next time I pass by the cycle shop near my iridologist's clinic). When I next go to my iridologist, my GPS-smart device gives me a friendly reminder to buy the pump.

Places need not be exclusive. "Outside" could be a place that includes all places outside. My device can then show me a list of all the things I can do when I step outside the house.

Urgency and radius may also be parameters. When I step outside, I would appreciate being reminded to do the urgent things first. If I'm driving past a friend's suburb, I would appreciate a reminder to pick up that DVD he'd promised to lend me, if it's not too much of a detour.

I think of HereDo as a short form for "What can I do now that I'm here?"

The number of times I've cursed myself thinking, "I was there just this morning! I should have done XYZ when I was there!" means this will be a very useful app.

The Case for Watermarked UUIDs

2010-06-30T03:05:00.001-07:00

UUIDs are my latest toy
They fill my little world with joy

Forgive me for waxing lyrical. The more I play with UUIDs, the more wonderful uses I find for them.

Take one-time tokens used for idempotence, for instance.

Joe Gregorio's nugget on how to use UUIDs with hashes to prevent spoofing got me thinking. What if the hash could be hidden in the UUID itself? UUIDs are so roomy and spacious, they can tuck a lot of things inside without losing their core property of universal uniqueness.

But first, a summary of Joe's piece for those too impatient to go over and read it.

Idempotence is (or should be!) a basic concept that all IT folk are familiar with. Performing an idempotent operation more than once has exactly the same effect as doing it once. The way this is commonly implemented is through the use of one-time tokens. [If you do a 'View Source' on your bank's transaction confirmation page (the one that asks "Are you sure?"), you may see a hidden form variable with a large random value. That's the one-time token the bank uses to ensure that you don't end up posting a transaction twice even if you hit the submit button twice. It's a wonderful way to ensure a measure of reliability over an unreliable Internet.]

Joe Gregorio suggests the use of UUIDs as one-time tokens for RESTful idempotent services, but also raises a potential problem with the use of raw UUIDs. Anyone can spoof a token by just generating a UUID. For a server to be able to recognise that a UUID is a genuine one-time token that it itself had handed out earlier, it would normally be expected to store the token somewhere so it can verify it when it's presented later on. But such a naive design would open the server up to a "resource exhaustion attack". A malicious user (or bot) can swamp the server with millions of requests for one-time tokens, and the server's database will rapidly fill up with useless UUIDs that are never going to be used.

To address this, Joe suggests a hash. If the generated UUID is combined with a secret string that only the server knows, and this combination is hashed, then the combination of UUID and hash will let the server validate its genuineness, because no one else can generate a valid hash for a random UUID without knowing the server's secret string. In Joe's model, the one-time token is not the raw UUID, but a combination of the UUID and a hash. With this design, the server doesn't have to store a one-time token when it's handed out, only when it's actually used to perform a transaction. The number of such UUIDs can be controlled, because spoofed UUIDs can be filtered out through a mere computational check.

I think this solution is elegant and ingenious, but I believe it can be made even more "clean".

A UUID looks like this:

e5b0eb4c-aeb9-411c-bf3f-cdd207096d3e

It consists of 5 groups of hex characters separated by hyphens, and the regular expression for it is

[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}

What I want to do is replace the last set of 12 hex characters with another one representing a hash (or at least part of a hash). Then, while the UUID will still look like a UUID (and indeed, will be a valid UUID), it is in effect "watermarked" by the server and can be verified by computation alone.

What do we lose?

Mathematically, we lose a great deal of uniqueness. We're now dealing with just 20 hex characters instead of 32 (It's not 24, because the 4 hyphens don't count). 20 characters are still a lot, and we actually get something back through the 12-character hash, because this is calculated not just on the 20-character UUID prefix but on the combination of the prefix and the server's secret string. So it's an independent 12-character hex string, and while it may be a more sparse range than its length may suggest, it's still something. So I don't believe we lose too much from a uniqueness perspective. UUIDs are so huge you can trim them and still not encounter conflicts.

Is there a danger that some random UUID out there may accidentally be computed as a valid watermarked UUID because its last 12 characters miraculously match the hash? Well, the probability of this is 1 in 16 raised to the power 12, which is about 1 in 3 quadrillion. I'd take my chances.

Architecturally, it would seem that we have introduced meaning into what should have been a meaningless identifier, and that would then open the door to implicit dependencies (tight coupling) and consequent brittleness. However, on closer inspection, there is no way an external system can seek to build any dependency on the structure of a watermarked UUID, because without a knowledge of the server's secret string, the hash cannot be externally calculated. The UUID remains necessarily opaque to all external parties. The implicit dependency of one part of the UUID on another would seem to be a limitation too, but this is by design! The "limitation" serves to exclude spoofed UUIDs.

And so, I believe there is no real downside to the use of watermarked UUIDs. On the contrary, they retain the visual elegance of plain UUIDs and furthermore simplify the design of idempotent services by encapsulating the entire token-validation function within the service with no leakage through to the interface.

I've written a couple of classes in Java that should help developers get started (no warranties, of course). The base class is called TokenWatermarker, and it performs the basic hashing and string concatenating logic on any token string. It can watermark LUIDs, for example. It also performs verification of previously watermarked tokens, of course. Then there's the UUIDWatermarker class that extends TokenWatermarker and provides the same capability for UUIDs.

Running the compiled classes using "java TokenWatermarker" or "java UUIDWatermarker" will print out a sample output that will show you how these classes work.

Enjoy!

The Case for Locally Unique IDs (LUIDs)

2010-06-25T05:16:00.000-07:00

It should be no surprise to regular readers of this blog that I am in love with UUIDs. As I have said before, they are an inexhaustible source of identifiers that are meaningless (not a pejorative term!) and whose generation can be distributed/federated without danger of duplication. As a result, they are an extremely powerful means of providing a uniform and federated identity scheme.

As a SOA-indoctrinated IT practitioner, I am loathe to expose domain-specific entity identifiers to the larger world because such leakage leads to tight coupling and brittle integration. Yet identifiers of "resources" must often be exposed. How do we do this? Even "best practice" guidelines fail to adequately admonish designers against exposing domain-specific identifiers. [e.g., Subbu Allamaraju's otherwise excellent RESTful Web Services Cookbook talks about "opaque" URIs in recipe 4.2 but ends up recommending the use of internal entity identifiers in recipe 3.10 :-(.]

My point is simple. If the 'employee' table in my application's database looks like this


+------+--------------+---------------+-------------+
| id   | first_name   | last_name     | dob         |
+------+--------------+---------------+-------------+
| 1122 | John         | Doe           | 12-Jan-1960 |
+------+--------------+---------------+-------------+
| 3476 | Jane         | Doe           | 08-Sep-1964 |
+------+--------------+---------------+-------------+
| 6529 | Joe          | Bloggs        | 15-Jun-1970 |
+------+--------------+---------------+-------------+

I do not want to be exposing resources that look like these


/employees/1122
/employees/3476
/employees/6529

I don't want to expose my application's local primary keys to the entire world. They may be "meaningless" but they're still coupled to my domain's internal data structures. I need something else.

My standard solution so far has been the magnificent UUID. I add a candidate key column to my table, like so


+------+--------------+---------------+-------------+--------------------------------------+
| id   | first_name   | last_name     | dob         | UUID                                 |
+------+--------------+---------------+-------------+--------------------------------------+
| 1122 | John         | Doe           | 12-Jan-1960 | 4885c205-8248-4e5b-9c45-4d042e7cc992 |
+------+--------------+---------------+-------------+--------------------------------------+
| 3476 | Jane         | Doe           | 08-Sep-1964 | cdbf87dd-93cb-4c53-9c5d-718c596b0a00 |
+------+--------------+---------------+-------------+--------------------------------------+
| 6529 | Joe          | Bloggs        | 15-Jun-1970 | 73feb1bf-e687-4d58-9750-5bf98ca7b9fa |
+------+--------------+---------------+-------------+--------------------------------------+

or I maintain a separate mapping table, like so


+------+--------------------------------------+
| id   | UUID                                 |
+------+--------------------------------------+
| 1122 | 4885c205-8248-4e5b-9c45-4d042e7cc992 |
+------+--------------------------------------+
| 3476 | cdbf87dd-93cb-4c53-9c5d-718c596b0a00 |
+------+--------------------------------------+
| 6529 | 73feb1bf-e687-4d58-9750-5bf98ca7b9fa |
+------+--------------------------------------+

and I expose my resources like so


/employees/4885c205-8248-4e5b-9c45-4d042e7cc992
/employees/cdbf87dd-93cb-4c53-9c5d-718c596b0a00
/employees/73feb1bf-e687-4d58-9750-5bf98ca7b9fa

I've still got unique identifiers, they're guaranteed not to conflict with anything else in time and space, and more importantly, my domain-specific identifiers remain hidden. I can now even change my entire domain model, including identifiers, and still preserve my external contracts. That's SOA.

But the sad fact of the matter is that many legacy systems and packaged software do not readily support UUID datatypes or even char(36) columns for various reasons. I have recently heard of a far-sighted software vendor that has provided for a "Public ID" field in their database tables for this precise reason, i.e., to allow an externally visible identifier to be specified for their entities. But alas, the column is defined to be varchar(20), much too small to hold a UUID.

It occurred to me that there is nothing sacrosanct about a 128-bit UUID (expressed as a 36-character string). It's just that the larger a random number gets, the more remote the probability of conflict with another such random number. 128 bits is a nice, safe length. But smaller lengths also have the same property, only with a lower degree of confidence.

The constraints of vendor packages like the one I described above led me to postulate the concept of the LUID (Locally Unique ID). This is just a string that is smaller than 32 hex digits (a UUID has 32 hex digits and 4 hyphens in-between). I call this a Locally Unique ID because the smaller it gets, the lower the confidence with which we can claim it to be universally unique. But we may still be able to rely on its uniqueness within a local system. If I'm only holding details of a few thousand employees (or even a few million customers) in my database, an LUID may still be expected to provide unique identifiers with a reasonable degree of confidence.

That vendor package definitely cannot hold a UUID such as "2607881a-fec1-4e5d-a7fc-f87527c93e2d" in its "Public ID" field, but a 20-character substring such as "4e5da7fcf87527c93e2d" is definitely possible.

Accordingly, I've written a Java class called LUID with static methods


String randomLUID( int _length ) and
String getLUID( String _uuidString, int _length )

The first generates a random hex string of the desired length (without hyphens). The second chops a regular UUID down to a hex string of the required length, again without hyphens.

You can download the class from here. Just running the compiled class with "java LUID" will result in some test output which should illustrate how it works. Feel free to incorporate it into your own projects, but be warned that there is no warranty ;-).

Of course, there is a limit to how small an LUID can become before it loses its utility, but I'm not going to draw an arbitrary line in the sand over this. The class above represents mechanism, not policy. Application designers need to think about what makes sense in their context. An LUID of the appropriate length can enable them to implement SOA Best Practice by decoupling externally visible resource identifiers from internal entity identifiers (another example of the difference between Pat Helland's Data on the Outside and Data on the Inside) when a standard UUID cannot be used.

Bottomline: If you can use a standard UUID, do so. If you can't, consider using an LUID of the kind I've described. But always hide the specifics of your application domain (which include entity identifiers) when exposing interfaces to the outside. That's non-negotiable if you want to be SOA-compliant.

Update 27/06/2010:
I should perhaps make it clear what my proposal is really about, because judging from a reader comment, I think I may have created the impression that all I want is for entity identifiers to be "meaningless" in order to be opaque. That's actually not what I mean.

To be blunt, I believe that any entity/resource that is to be uniquely identified from the outside needs _two_ identifiers (i.e., two candidate keys). One of them is the "natural" primary key of the entity within the domain application. The other is a new and independent identifier that is required to support the exposure of this entity through a service interface (whether SOAP or REST). There should be no way to derive this new key from the old one. The two keys should be independently unique, and the only way to derive one from the other should be through column-to-column mapping, either within the same table or through a separate mapping table as I showed above.

To repeat what I wrote in the comments section in reply to this reader:

There was a content management system that generated (meaningless) IDs for all documents stored in it, and returned the document ID to a client application that requested storage, as part of a URI. At one stage, it became necessary to move some of the documents (grouped by a certain logical category) to another instance of the CMS, and all the document IDs obviously changed when reloaded onto the other instance. The client application unfortunately had references to the old IDs. Even if we had managed to switch the host name through some smart content-based routing (there was enough metadata to know which set of documents was being referred to), the actual document ids were all mixed up.

If we had instead maintained two sets of IDs and _mapped_ the automatically generated internal ID of each document to a special externally-visible ID and returned the latter to the client, we could have simply changed the mapping table when moving documents to the new CMS instance and the clients would have been insulated from the change. As it turned out, the operations team had to sweat a lot to update references on the calling system's databases also.

I hope it's clear now.

1. Entity only seen within the domain => single primary key is sufficient
2. Entity visible outside the domain => two candidate keys are required, as well as a mapping (not an automated translation) between the two.
3. UUID feasible for the new candidate key => use a UUID
4. UUID not possible for some reason => use a Locally Unique ID or LUID of appropriate length (code included)

Annotations and the Servlet 3 Specification

2010-06-18T04:23:00.000-07:00

Now I'm seriously beginning to wonder about the authors of the Java Servlet 3 specification. This time, it's not their architectural wisdom (or the lack of it) regarding session state. It's about something even more basic to the Java language - the nature of annotations.

Chapter 8 deals with annotations that developers may use to mark their classes. Anything about the following strike you as crazy?

Classes annotated with @WebServlet class (sic) MUST extend the javax.servlet.http.HttpServlet class.
[...]
Classes annotated with @WebFilter MUST implement javax.servlet.Filter.

If we must extend a base class anyway, I wonder what the annotation is for. Just to avoid putting a few lines of config code into the web.xml file?

I would have thought an annotation like @WebServlet would be capable of turning any POJO into a servlet class, not just subclasses of HttpServlet! And we could have annotations like @GetMethod, @PostMethod, @PutMethod and @DeleteMethod to annotate any arbitrary methods in the class. We shouldn't have to rely on overriding the doGet(), doPost(), doPut() and doDelete() methods.

The same applies with @WebFilter. It could be used to annotate any arbitrary class, and @FilterMethod could then annotate any arbitrary method in the class.

Look at the way JSR 311 and Spring REST work.

I'm disappointed in the Servlet spec committee. If you're going to use annotations, then use them smartly.

It wouldn't be out of place here to comment on the horrific class hierarchy of the Servlet spec. It certainly shows the era from which it began, an era where interfaces were underappreciated and inheritance hierarchies featured classes themselves. Naming conventions hadn't matured yet, either.

E.g., my application's concrete class "MyServlet" must either extend abstract class "GenericServlet", which in turn partially implements interface "Servlet", or implement "Servlet" directly. This by itself isn't so bad, but go ahead a bit.

My application's concrete class "MyHttpServlet" must only extend abstract class "HttpServlet" which extends abstract class "GenericServlet", which in turn partially implements interface "Servlet". There is no interface to implement.

And why GenericServlet should also implement ServletConfig is something I don't understand. There's a HAS-A relationship between a servlet and its configuration. It's not an IS-A relationship.

HttpServlet should have been an interface extending Servlet.

The abstract class GenericServlet (that partially implements the Servlet interface) should have been called AbstractServlet instead, and there could have been a concrete convenience class called SimpleServlet or BasicServlet that extended AbstractServlet and provided a default implementation that subclasses could override.

Similarly, there should have been an abstract class called AbstractHttpServlet that partially implemented the HttpServlet interface and only provided a concrete service() method, dispatching requests to doXXX() methods that remained unimplemented. There could have been a concrete convenience class called SimpleHttpServlet or BasicHttpServlet that extended the AbstractHttpServlet class and provided a default implementation that subclasses could override.

My application's concrete classes should have had the option to implement one of the interfaces directly or to extend one of the abstract or convenience classes.

Oh well, too late now.

REST and the Servlet 3 Specification

2010-06-17T02:48:00.000-07:00

I've been going through the Java Servlet 3 specification, and I just came across this gem at the start of the chapter on Sessions:

The Hypertext Transfer Protocol (HTTP) is by design a stateless protocol. To build effective Web applications, it is imperative that requests from a particular client be associated with each other. [...] This specification defines a simple HttpSession interface that allows a servlet container to use any of several approaches to track a user’s session [...]

I don't think the spec authors have been adequately exposed to the REST philosophy, or they wouldn't be talking so casually about how "imperative" sessions are to build "effective" Web applications. A few years ago, I would have read this without batting an eyelid. Now, I had to keep myself from falling off my chair in shock. One would think spec writers of advanced technology would know a bit better. At the very least, they could have written something like this:

The Hypertext Transfer Protocol (HTTP) is by design a stateless protocol, and it is strongly recommended that Web applications be built in a stateless manner to be effective, with all state management delegated to the persistence tier. If, for legacy or other reasons, it is unavoidable to maintain in-memory state in a web application, the servlet specification defines a simple HttpSession interface that provides a relatively painless way to manage it. Application developers should however be aware of the severe scalability and recoverability issues that will accompany the use of this feature.

There! Now I feel much better.

Does REST Need Versioning?

2010-06-12T16:25:00.000-07:00

In my ongoing conversations with JJ Dubray, he has often made the point that "REST couples identity and access together in a terrible way". When pressed to explain, he provided the following example.

Assume that there is a Resource identified by "/customers/1234". Updating the state of this customer requires a PUT. JJ asks how REST can handle a change to the business logic implied by the PUT.

Since we cannot say


   PUTv2 /customers/1234

implying a change to the logic of PUT, he believes we have no option but to say


   PUT /customers/v2/1234

but this is different from the identity of the customer, which remains at


   /customers/1234

Hence REST "couples identity with access".

Well, I disagree. First of all, it's a mistake to think there are only two places where the version of business logic can be exposed - the Verb and the Resource. The data submitted is an implicit third, which I'll come to in a moment. But this example only makes me question the whole basis for versioning.

Does REST need versioning? For that matter, does any service need versioning? What is versioning in the context of SOA?

I would say service versioning is a mechanism that allows us to simultaneously maintain two or more sets of business logic in a consumer-visible way.

Why does it have to be consumer-visible as opposed to just consumer-specific? After all, if the service implementation can distinguish between two classes of consumer, it can apply two different business rules to them in a completely opaque manner. The consumer doesn't even have to know that two (or more) different sets of business rules are being weighed and applied under the covers.

Let's ask the more interesting question: Why do we need to maintain two or more sets of business logic simultaneously? The interesting (and circular) answer is often that business logic happens to be consumer-visible, hence a new version of business logic also needs to be distinguished from the old in a consumer-visible way. This is often stated as the need to support legacy consumers, i.e., consumers dependent in some way upon the previous version of business logic. But why do we have to support legacy consumers? Because existing contracts break when services are silently upgraded.

This argument leads to an interesting train of thought. Perhaps the answer lies in the opposite direction to what JJ believes, i.e., not versioning of services but abstraction of detail. Are our service contracts too specific and therefore too brittle? Service versioning is perhaps a "smell" that says we are going about SOA all wrong. Let us see.

I want to take up a more real-world example than the customer access example that JJ talked about. After all, that's more of a "data service" than a business service. Let's look at a real "business service".

Let's take the case of the insurance industry where a customer asks for a quote for an insurance product. The client-side application has to submit a set of data to the service and get a quote (a dollar value for the premium) in return.

In REST, here's how it could work.

Request:


   POST    /quotes
   <insurable-details>
   ...
   </insurable-details>

Response:


   201 Created
   Location: /quotes/06fb633b-fec4-4fb6-ae32-f298b8f499c1

The client is referred to the location of the newly-created quote Resource, which is at /quotes/06fb633b-fec4-4fb6-ae32-f298b8f499c1. When the client does a GET on this URI, the quote details are transferred.

So far, so good. Now let's say the business logic changes. Premiums are now calculated using very different logic. The first question is, can this new business logic be applied to all customers, or do we need to keep track of "old" customers and keep applying the old business logic to them? If we can "upgrade" all customers to the new business logic, there is, of course, no problem at all. The interface remains the same. The client application POSTs data to the same URI, and they are redirected in the same way to the location of the newly-created quote Resource. The business logic applied is all new, but customers don't see the change in their interface (only in the dollar values they are quoted!)

However, if we do need to maintain two sets of business logic, it could be for three reasons. One, the data that the client app needs to submit has changed, so the change is unavoidably visible to the customer and has to be communicated as a new and distinct contract. Two, there is another business reason to tell two types of customers apart, perhaps to reward longstanding customers with better rates, and this difference between customers is not obvious from the data they submit. Third, the client app somehow "knows" the behaviour of the old version and is dependent on it. In this case, we need a new version just to keep legacy clients from breaking.

We can readily see that the third reason is an artificial case for versioning. It's in fact a case to break implicit dependencies that have crept in.

In contrast, the first and second reasons provide their own resolution. If the type of data submitted by the client changes, that is itself a way to distinguish new clients from old ones and apply different business logic to them. In other words, we only need to tell newer customers about the change in the data they need to POST. Older customers don't need to do a thing. Also, if we can somehow derive that the customer is an existing one, even if this is not explicit in the data submitted, we can still apply different business logic transparently.

JJ may consider this a messy and unstructured approach to versioning. Business stakeholders may have the opposite view. It's less disruptive. The less clients are exposed to the way services are implemented, the better.

Service versions are not really an interface detail. They're an implementation detail that often leaks into the interface.

That means version numbers are a problem, not a solution.

None of these arguments may satisfy someone like JJ. In that case, if service versioning is absolutely essential, there is a simple way to include it, after all. Include the version number in the message body accompanying a POST or PUT request. In fact, message bodies are allowed even for GET and DELETE requests (anything except a TRACE), so versioning of any type of service is possible. REST does not enforce versioning, (that would be a bad thing considering that versions are often a smell), but doesn't impede it either.

With this approach, neither Verbs (e.g., POST) nor URIs (e.g., /quotes) are affected by versions and the "terrible" coupling of identity and access is avoided.

It seems to me that the problem is not with REST, it's with looking at REST through WS-* eyes.

Is REST Another Variant of Distributed Objects?

2010-06-11T02:49:00.000-07:00

In my discussions with JJ on REST, he's often made the observation that REST is nothing but Distributed Objects and is therefore a bad style. But is it really?

I have two different arguments why I believe it isn't.

1. Let me first talk about my experience as a migrant from India to Australia many years ago. Working in Indian companies had acclimatised me to a rather direct management style. If my manager ever wanted me to do something, he or she would say so in so many words, "Do this!"

When I arrived in Australia and began working, I was struck by the very different style of Australian managers. I would hear things like, "You may want to do this," or "You may like to do this." It took me a while to realise that they were essentially saying the same thing, only less directly. Let me coin a term for this style, because I'm going to use it to explain a concept with Distributed Objects. Let me call this a polite command.

Let's now turn to methods that set an object's attributes. We may see setter methods like these:


 widget.setSomeAttribute( someValue );
 widget.setAnotherAttribute( anotherValue );

Now consider another style of doing the same thing.


 widget.updateSelf( widgetDTO );

where widgetDTO is a structure that holds the new values of someAttribute and anotherAttribute.

Let's call the direct setter methods commands. "Remoting" these commands leads to a tightly-coupled, RPC mechanism. This is the Distributed Objects style. I would be the first to agree with JJ that this is a bad approach.

But the second style is a polite command. It's requesting the object to update itself based on values held in a Data Transfer Object (i.e., a representation). Now this is a style that can be remoted without problems, because it's not really RPC.

The REST style of updating resources follows the latter model.


PUT   /widgets/1234
<attributes>
<some-attribute>some value</some-attribute>
<another-attribute>another value</another-attribute>
</attributes>

In other words, this is a polite command. It can be safely remoted. It's not Distributed Objects.

2. JJ laughs at the approach of annotating object methods to turn them into REST resources, the way JAX-RS does. This is another reason why he considers REST to be Distributed Objects. The annotations seem to be doing nothing more than remoting object methods. Therefore, REST = Distributed Objects and consequently a horrible way to design systems.

Not so fast.

Let's not forget the concept of Service Objects, which are not really Domain Objects.

Let's look at a simplistic domain model for a banking system. The major Domain Object in this model is an Account. An Account object has the following methods:


class Account()
{
  public double getBalance() {...};
  public void credit( double amount ) {...};
  public void debit( double amount ) {...};
}

The Domain Model is internally-focused. Nothing here understands the concept of a transfer. Indeed, a transfer cannot be elegantly modelled using domain objects.

That's because a transfer is an aspect of a Service. Service verbs are really free-standing verbs. They don't belong to classes the way domain methods do. The methods getBalance(), credit() and debit() don't make any sense by themselves. It always has to be account.getBalance(), account.credit() and account.debit().

In contrast, transfer() can be a free-standing verb. In fact, it seems downright clumsy to push transfer() into a class, because it really doesn't belong inside any class. It's just that languages like Java are unyieldingly object-oriented and don't tolerate free-standing verbs. So in practice, designers create an awkwardly-named class like AccountService and stick transfer() inside it, like this:


class AccountService()
{
   public void transfer( Account fromAccount, Account toAccount, double amount )
   throws InsufficientFundsException
   {
       if ( fromAccount.getBalance() >= amount )
       {
           fromAccount.debit( amount );
           toAccount.credit( amount );
       }
       else
       {
           throw new InsufficientFundsException();
       }
   }
}

[If J2EE designers feel a sense of déjà vu on seeing this, it's the old Session Façade pattern all over again, with Stateless Session Beans acting as a service façade for domain objects represented by Entity Beans. Stateless Session Beans were not part of the Domain Model at all. They were an explicit service layer that lent itself to being remoted through (what else?) a Remote Interface.]

Now, if we tried to remote a method like credit() or debit(), it would be a classic case of RPC (RMI, strictly speaking) and therefore Distributed Objects hell.

But a "method" like transfer() readily lends itself to being remoted! That's because an instance of AccountService isn't a Domain Object but a Service Object.

If designers took care to annotate only the methods of Service Objects and avoided doing so with methods of Domain Objects, then they neatly avoid being trapped into the Distributed Objects paradigm.

All that leads to a certain insight. And that is that although REST is an architectural style, it isn't prescriptive enough. We need to tell designers what types of domain objects can be modelled as resources and what cannot.

With a nod to the term polite command, perhaps it's not enough for systems to be RESTful. They should also be RESPECTful :-).

I realise I have blogged extensively on this over 2 years ago, here and here.

The Real and Imagined Limitations of REST

2010-06-09T17:10:00.000-07:00

One of the things that struck me about my discussion with JJ Dubray on a previous blog posting was how closely we agreed on fundamental architectural principles (decoupling of interface from implementation, avoiding a hub-and-spokes architecture, etc.), yet how diametrically opposed our views were on REST.

For example, I think REST does a great job of decoupling interface from implementation. JJ feels the exact opposite. Why?

Analysing the problem more closely, I guess the common examples of RESTful interface design are partly to blame.

1. A URI like

http://myapp.mycompany.com/customers/1234

where 1234 is the actual id of the customer in my database, would be a horrible example of tight coupling, in my opinion. I believe URIs should be opaque and meaningless. I think designers should take care to create a mapping layer that decouples their implementation data model from their externally exposed service data model.

For example, I would prefer this to be exposed:

http://myapp.mycompany.com/customers/cb77380-7425-11df-93f2-0800200c9a66

There should be a mapping within the service implementation that relates the opaque identifier "cb77380-7425-11df-93f2-0800200c9a66" to the customer's id within the domain data model, i.e., 1234. That would be true decoupling.

Mind you, the earlier example of tight coupling is not a limitation of REST, merely bad application design.

[Incidentally, I think UUIDs have given the world a wonderful gift in the form of an inexhaustible, federated, opaque identification scheme for resource representations. Decoupling identity is a key part of decoupling domain data models (Pat Helland's Data on the Inside) from service data models (Data on the Outside).]

2. Another bad example is the use of "meaningful" URIs. Even though the following two URIs may seem obviously related, client applications must not make any assumptions about their relationship:

http://myapp.mycompany.com/customers
http://myapp.mycompany.com/customers/1234

In other words, a client application must not assume that it can derive the resource representation for the set of customers by merely stripping off the last token "/1234" from the URI of an individual customer.

And this is not a limitation of REST either. The HatEoAS (Hypermedia as the Engine of Application State) principle says that client applications must only rely on fully-formed URIs provided by the service to perform further manipulations of resources. In other words, URIs are to be treated as opaque and client applications must not attempt to reverse-engineer them by making assumptions about their structure.

Examples like these are bad from the perspective of architects who understand the evils of tight coupling, and the ones who produce these examples understand it too, but these naive examples have the benefit of being easy to understand.

A RESTful system would work perfectly well with URIs that looked like these:

http://myapp.mycompany.com/51bb2db7-b51c-4fd9-b235-2b310a644a84
http://myapp.mycompany.com/a7ac1f23-6bc7-44be-af9b-4b68fd31879f
http://myapp.mycompany.com/208657ef-1e86-47ba-9c5f-3f177c018293

The first is a URI for a customer, the second is a URI for a bank account and the third is a URI for an insurance policy. How about that? This would be architecturally clean, but most REST newbies would go, "Huh? Weird!" and turn away with a shudder.

Sometimes, the desire for understandability of a design by humans introduces semantic coupling that is anti-SOA. I guess there's a trade-off that we need to be aware of, but it's not a limitation of REST itself. It's an application designer's choice.

3. In another context, JJ has expressed his opinion that SOA is about asynchronous peer-to-peer interaction, and I completely agree. Where I think he has misunderstood REST is in its superficial characteristic of synchronous request-response. There are several well-known design patterns for asynchronous interaction using REST, so in practice, this is not a limitation at all. The HTTP status code of "202 Accepted" is meant for precisely this condition - "I've received your message but have not yet acted on it". At one stroke, it's also a model for reliable message delivery. Combine a POST with a one-time token to ensure idempotence, then keep (blindly) trying until you get a 202 Accepted response. Voila! Reliable message delivery.

I find I am able to look beyond the superficial characteristics of REST that seem like showstoppers to JJ. I can see simple workarounds and Best Practice guidelines that can make a RESTful application fully compliant with SOA principles. JJ stops at the characteristics as given and concludes that REST is horribly antithetical to SOA.

The Need for SOA in the Real World

2010-06-09T16:10:00.001-07:00

To those who think SOA is just hype, this should be a sobering piece of news from the real world:

ANZ Bank hits a wall on financial software rollout

Sources have told The Australian that the bank is having second thoughts about going ahead with installing a Teradata solution made up of enterprise data-warehousing and database software into a predominantly Oracle-run house.
[...]
"It just doesn't marry well with the rest of the organisation as it's an Oracle house," one source said.

Anyone see the problem here?

Why should the brand of software being introduced have anything to do with a decision to implement? We in IT tend to treat this sort of argument as perfectly reasonable. Of course software from one vendor won't play nicely with software from another! We've internalised the obscenity of the situation. That's what makes this more horror than farce. We don't see the lack of interoperability between vendors as a problem, merely as a constraint.

Well, SOA thinking would have challenged this right off the bat. The brand of existing software and new software just doesn't matter. What matters is business functionality. As long as it's all wrapped up in contract-based services, they should play well together.

But we obviously don't live in that world. We live in a world where a strange kind of Stockholm Syndrome grips customers, preventing them from asserting their rights and causing them to acquiesce in a patently unfair seller's market. The "pragmatic" decisions that then follow create vendor lock-in and much higher outflows in the long run. So much for pragmatism.

What this news item tells me is that not only is the software vendor world not SOA-friendly, the business user community doesn't think in SOA terms either.

More's the pity.

SOAP, REST and the "Uniform Interface"

2010-06-03T18:53:00.000-07:00

When REST folk compare their approach with SOAP-based Web Services (which they often mistakenly call "SOA", not realising that REST is an equally valid way to do SOA), they often refer to a "uniform interface" that REST provides that SOAP does not.

The response of the SOAP crowd (when there is one), is "What's a uniform interface?" (They'd be surprised to hear that they have one too, which is the topic of this post.)

The uniform interface refers to a standard way to do something regardless of what the specific activity is.

As an example, if I asked what the SOAP interface would be like for a service, and I refuse to say what the actual business function is, the SOAP people would have to stop after describing the service at a very high level. There's a SOAP message with a header and a body, and what goes into them depends entirely on what I want to do. The general semantics of a SOAP message are "Process this". That's actually too general to be useful. Any further detail they could give me would depend on what I tell them about the actual operation I'm trying to perform.

In contrast, the REST guys can actually tell me much more about what my service is going to look like based only on the rough contours of my business function.

They can tell me that what I operate on is a resource that will look like this:

"http://domain/resource"

They can tell me that I will be using one of four verbs (GET, POST, PUT or DELETE).

They can tell me that my response status codes will be one of some 50-odd pre-defined status codes.

Plus they can give me other general tips that will apply, such as a GET operation will have no side-effects, a deferred processing request will return a "202 Accepted" status when successful, a POST request will be to a resource collection, and its response status code will be "201 Created" when successful, and the URI of the newly created resource will be found in a response header called "Location:", etc., etc.

That's actually quite a bit of information considering I haven't said anything yet about the business function I'm trying to implement.

That's what is called a "uniform interface". It's a structured framework within which application designers have to work. Far from being restrictive and limiting, these constraints actually make things easier for the designer because they provide standard patterns that can be reused in similar situations and deliver predictable results.

So far, so good. What the REST side doesn't understand is that SOAP-based Web Services offer a uniform interface too. Obviously not for the application itself, but for qualities of service.

Ask a SOAP service designer how they plan to implement reliable messaging, and they can show you very detailed WS-ReliableExchange headers that are virtually the same regardless of the business service. Ask them about security (encryption and signing/authentication) and they can show you the detailed WS-Security/SecureConversation/Trust headers required, and these too are standard and unchanging.

Security and Reliability are "Qualities of Services". SOAP-based Web Services have standardised these, and are also on track to standardising Transactions through WS-AtomicTransaction and WS-BusinessActivity (the latter deals with real-world transactions that require compensating actions to "roll back"). [Some may argue that these have in fact been standardised, but there isn't yet a WS-I Basic XXX Profile for these, like there is for Security and Reliability, which means there is no benchmark yet for interoperability in these areas.]

This isn't something to be sneezed at, because a uniform interface leads to improved interoperability, which in turn reduces the cost of integration. From my own experience in IT, I know that integration cost forms a very high proportion of the cost of most projects. Indeed, Project Tango proved the interoperability of Java and .NET-based Web Services using declarative statements in policy files to have these QoS headers automatically generated. Now that's cool.

REST promises to reduce integration cost and thereby overall cost, mainly though the mechanism of the "uniform interface". However, REST does not have a standard model for Security or Reliability, let alone Transactions or Process Coordination. SSL is a rigid security mechanism that is adequate for most, but not all situations. And while it's true that Reliability is probably better implemented through patterns like Idempotence than through the TCP-like model followed by WS-ReliableExchange, it requires an application designer to consciously build it. Transactions and Process Coordination are also missing from the REST lineup.

That's the challenge. Both SOAP and REST have standardised a part of the service interface, but not all of it. REST has standardised the interface of the application's core logic. SOAP has standardised the interface for Qualities of Service. What we need is a model that provides us with both.

Spring Integration - Enabling the Enterprise Service Cloud

2010-06-02T20:31:00.000-07:00

Inversion of Control is a great concept, and the Spring framework excels at it.

Little did I think that Spring could also effect an Inversion of Architecture! I just recently blogged about the need for organisations to move from a hub-and-spokes ESB (Enterprise Service Bus) to its architectural opposite, a federated ESC (Enterprise Service Cloud).

In the hub-and-spokes model, you run an application (a Service) on the ESB. To 'invert' that architecture and make it federated, you need to embed an ESB into your application! Spring Integration allows us to do just that.

This manual provides a number of examples that explain how the tiny, deconstructed pieces of ESB functionality can be injected into an application as required.

That neatly solves the problem of having to deploy multiple heavyweight ESB servers at every endpoint to front-end each application, which would have been the only way to get federation with that heavyweight approach, - a clearly unacceptable solution even if one ignores licence costs.

I used to think that a federated ESB was the answer and that JBI (Java Business Integration) provided the most flexibility to deploy tailored endpoints. (JBI allows one to plug in specific ESB modules into a common "backplane", but Spring Integration goes a step further and uses the application itself as the backplane.)

The future seems to be here already.

Wanted: An ESC, not an ESB

2010-06-02T14:09:00.000-07:00

Here's my fundamental beef with an ESB: It follows a hub-and-spokes model that is more suited to an EAI initiative of the 1990s. In the year 2010, we should be doing SOA, not EAI.

SOA is federated where EAI is brokered. It's the architectural antithesis of EAI. The smarts are pushed out from the middle to the endpoints until there's no "middle" anymore. I wrote about the difference between "federated" and bad old "point-to-point" before. Federated is a standardised smartening of the endpoints.

It doesn't matter whether we do SOAP or REST. Either way, it's SOA, and it's inherently federated.

Both SOAP's Web Service Endpoints and REST's Resources are represented by URIs. URIs offer a "flat" address space. They're completely opaque. DNS can hide where the services physically sit, so any node can talk to any other without the need for an intermediary.

Look at a SOAP message. Look at the headers - WS-Addressing, WS-Security, WS-ReliableExchange. They're all end-to-end protocols. The SOAP engines at the endpoints negotiate and handshake every Quality of Service they need without depending on anything in between.

WS-ReliableExchange uses endpoint-based logic such as acknowledgements, timeouts and retries to provide a measure of reliability even with unreliable transports, much like TCP provides connection-oriented communications over a plain packet routing protocol like IP. There is no need for a "guaranteed delivery" transport like a message queue.

WS-Trust does key exchange, WS-SecureConversation sets up a session key and WS-Security encrypts messages, all in an end-to-end fashion. There is no need for a secure transport.

Any node that talks SOAP and WS-* and conforms to the WS-I Basic Profile (and Basic Security Profile and Basic Reliability Profile) can participate. That's federation.

This isn't an academic discussion. The main drawbacks of a hub-and-spokes model are Availability and Scalability. The ESB is a single point of failure and a performance bottleneck. The normal "solution" is to beef up the ESB by providing for redundancy and "High Availability", but this costs a fair bit and only postpones the inevitable. That's why I said this isn't an academic discussion. There are serious dollars involved. Organisations would do well to think about the implications of an ESB approach and take early steps to avoid the inevitable costs that lie along that path. The correct solution would be to do away with the ESB altogether and embrace the inherently federated model of SOAP (or REST).

Otherwise, we're forcing inherently federated protocols into a hub-and-spokes model, - a needless throwback to an earlier era , - and inviting problems that need to be "solved" at great expense.

In the year 2010, when clouds are all the rage, we should recognise that what SOAP and REST give us are "service clouds".

So that's my buzzword for today - ESC (Enterprise Service Cloud)!

The ESB is dead, long live the ESC!

Mediation Doesn't Require Something "In The Middle"

2010-05-31T17:29:00.001-07:00

I blogged earlier about the subtle difference between federation and point-to-point connectivity, but didn't spell out how systems suffering from point-to-point connectivity ("the Dark Ages") could be effectively federated to bring them into the Modern Age without having to interpose a broker between them ("the Middle Ages").

Here's the problem - a point-to-point connection between two systems.
Here's the obvious solution - a broker that mediates all communication between the two systems. This is the ESB model, of course.
Is there another decoupling model for a federated system that can (by definition) have no centralised broker?

Consider an application that has the following hyperlink:

http://192.168.2.15/policy-page.html

Now look at this one:

http://intranet.mycompany.com/policy-page.html

If we assume that both links are in fact referring to the same page, what's the difference between them? Yes, we know that in the second case, a DNS lookup is required to resolve the domain name to an IP address before the page can be accessed, but the actual HTTP call to GET the page is exactly the same (it uses the IP address) and the end result of the call is also the same. The HTTP GET seems to be point-to-point in both cases with no intermediaries.

So what's the real difference?

The first is truly an example of a point-to-point link. How do we know? Test it to see how brittle it is. If the intranet is now hosted on a different server with IP address 192.168.2.16, the link breaks immediately.

However, in the second example, if we change the server hosting the intranet but also update the relevant DNS entry so that "intranet.mycompany.com" points to "192.168.2.16" instead of "192.168.2.15", then the link continues to work and the application that embeds it is none the wiser. This is no longer a tight, point-to-point connection. It has been effectively intermediated by the introduction of a DNS lookup before the actual call is made.

This is what lookup-mediation looks like.
As we can see, we don't need to physically interpose a component between two systems to break the point-to-point link between them. We can use a lookup mechanism to achieve the same result.

Incidentally, web-style interaction with domain names in URIs instead of IP addresses is inherently loosely-coupled and therefore the REST style of integration that follows this approach is not an example of point-to-point connectivity. Any node can make HTTP calls to any other, yet nodes are not tightly coupled to each other. This is a practical example of federation.

"Federated" is not "Point-to-Point"

2010-05-31T08:22:00.000-07:00

One of the interesting topics in the "to ESB or not to ESB" discussion is the value of avoiding point-to-point interactions between systems. Somewhere along the way, I suspect that mainstream IT has been indoctrinated with the message that point-to-point connections between systems are bad, and in "four legs good, two legs bad" fashion, has begun to parrot the phrase dumbly without understanding what it means.

I realise that most of the goodwill towards centralised brokers comes from that mindless Orwellian mantra. If someone (like me) argues for a federated ecosystem that does not require a component "in the middle" and where any node can talk to any other, the common assumption is that I'm advocating bad old point-to-point connectivity. A lot of the arguments against point-to-point connectivity then get trotted out.

That's not what I'm advocating, and it's very difficult to get people to understand the subtle but important difference between point-to-point connectivity and a federated system. A system that relies on point-to-point connectivity is in the Dark Ages. A system that uses a centralised broker to break those point-to-point connections has progressed to the Middle Ages. But to evolve to the Modern Age, we need to get rid of the centralised broker and move to a federated model, and this does NOT mean a backward step to the point-to-point model, even though communication once again occurs directly between endpoints! How is that possible?

The answer lies in a standardised smartening of the endpoints.

Perhaps an analogy would help.

I'm at work, and I find I need to talk to another person in my organisation to understand some specialised information about Application X. I've never met this person before. What do I do?

I'm not totally lost. There is a common protocol governing what I need to do. I look up the (rough) spelling of this person's name on the company intranet. I see a few names there, and I select the one that I think I need. A page comes up, listing the person's name, job title, department, email address and phone number.

Next I need to decide if the information I need requires an email exchange or a simple phone call. Note that phone and email have given me two communication options - one synchronous and one asynchronous. Even the synchronous option has an asynchronous fallback called voicemail.

Lo and behold! I already have a phone and a laptop on my desk. I look around and see that every employee does. Every employee has the basic capability to make phone calls and send emails to every other employee.

Let's say I decide to pick up the phone and call the person. I know the mechanics of the protocol. I pick up the handset and punch in numbers corresponding to what I see on the person's intranet page. When the person picks it up, I know the etiquette I'm expected to follow.

"Hi, I'm so-and-so. Is this a good time to talk? Thank you. I work in such-and-such an area and I'm currently looking at such-and-such an issue. I find I need to understand a bit about how Application X works, and I'm told you're the best person to talk to."

If it's a simple question, I may just ask it then and there. If the issue is going to require some discussion, I may request a meeting and then send a meeting invite through email after checking the person's calendar to see when they're likely to be free.

If no one picks up the phone, I leave a voicemail with roughly the same information, but also my contact details so the person can get back to me.

What a lot of smarts that took! Yet it's something we take for granted. We call it "business training". We don't expect that a primary school student placed in our position would be able to do all that, yet all of us do it unconsciously. There's a lot of smarts at the endpoints in this model, and no one argues for doing away with it.

Oh, and all communication is in English, of course (I'm in Australia :-). No matter what language I speak at home, I'm expected to speak English when I talk to someone at work. And so are they. It's not considered an unacceptable condition.

Now consider a very different situation. There's no corporate intranet. I have no means of looking up a person. Not everyone has phone and email. Some only have a phone number. Some only have email. Others have neither and I will need to walk across to their desks to talk to them face to face. And I have no way of knowing this up front. If someone isn't available when I try to talk to them, I have no means of leaving a message for them. I have a phone, but I have no idea if they will be able to call me. Oh, and I don't speak English, and neither do they. We speak completely different languages. And no one taught either of us how to use a phone or email or the etiquette of a business conversation! And if by chance I manage to surmount these challenges and discover a way to talk to a certain person, I need to carry that information around with me all the time. I need to do this for the dozen or so people I talk to regularly. Everyone has to do this for all their contacts, - carry individualised communication information around with them. If one of my contacts has their phone number changed, there's no standard way for them to let me know. I simply can't talk to them anymore until I discover their new phone number. This is the work environment from hell (even if the people are the nicest). This is a point-to-point world.

Let's call this the Dark Ages. How could we solve these seemingly intractable communication problems?

Here's the solution! Appoint a liaison officer. If I need to talk to anyone in the organisation, rather than keep in my head the idiosyncrasies of the mode of communication I need to employ with that person, I simply approach the liaison officer with my request, spoken or written out in my own mothertongue. I'm guaranteed a response in two business days. The liaison officer takes care of talking to the other person in the mode of communication best suited to them, with suitable translation to their language. All my communication goes through the liaison officer, and I don't have to know anything about the person I'm dealing with at the other end, - where they're located, whether they use a phone, email or carrier pigeon, what language they speak, - nothing at all! Isn't this progress? Of course it is. We've progressed from the Dark Ages to the Middle Ages. This is a brokered world.

We know this is not the Modern Age because we know what the Modern Age looks like. We live in it - the world of the Standard Operating Environment (SOE) with phones and laptops on every desk, the corporate intranet, the English-only work environment, assumed business training on the part of everyone in the game, etc., etc. In this world, anyone can call anyone at will and do business instantaneously, in an almost effortless manner, with no liaison officers required to mediate our conversations. This is a federated world.

However, here's a crucial issue. If we look back at the work environment that we have called the Middle Ages, and ask ourselves how things could be made better, we're unlikely to get an answer that remotely resembles the Modern Age. We can't get here from there. We're much more likely to be told of problems that still exist in the Middle Ages and of incremental patches required to make them work better.

For example, the liaison officer falls ill once in a while and can't come to work. On those days, work comes to a standstill because people can't communicate anymore. It's an Availability issue.

Some days, there are twenty people huddled around the poor harried liaison officer, all talking at the same time and insisting that their work is very urgent. Everyone is delayed and everyone is unhappy. That's a Scalability problem.

The Middle Ages solution is simple and obvious, of course. Instead of a single liaison officer, institute a Liaison Office staffed with more than one officer. This will provide both redundancy for greater availability as well as scalability. Of course, there will be problems with this model as well. We never seem to have sufficient budget to hire as many officers as we need, so some problems of Availability and Scalability will always remain. Also, if the particular liaison officer I spoke to yesterday is on leave, I need to explain the context of my requirement to another officer in painstaking detail all over again. The Liaison Office really needs to be made more "stateful". So we need to have elaborate procedures for officers to document their interactions with people and do proper handovers before going on leave. The Liaison Office becomes more and more complex and acquires a bureaucracy of its own. But nothing can be done about it because this is "best practice", isn't it?

This is indeed where we are today in the SOA world. Smack in the Middle Ages with a great big honking ESB in the middle that we need to beef up with High Availability at great cost. But to many SOA practitioners (and indeed, to the mass of regular IT folk out there), this is Best Practice. This is the only way.

It seems too hard to imbue all endpoints with the smarts to function in a federated fashion. At the very least, we need to equip every node with a metaphorical phone and email to make it physically possible for any node to call another. Well, the model of endpoint-based SOAP messaging is meant to do just that, with WS-* headers used to provide end-to-end Qualities of Service and with absolutely no need for any intermediary. But that's still just the plumbing. How do we provide business training and teach common business etiquette? In other words, how do we standardise the "application protocol"? Well, that's been done as well, using a model entirely independent of SOAP. It's called REST. REST is simultaneously low in technological sophistication (all we need is a web server and a database) and high in conceptual sophistication (we need to learn to think in a certain way). We have to understand these federated models if we're going to have any hope of progressing from the Middle Ages to the Modern Age.

And that's where we are today. I know that "inherent interoperability" is possible and far superior to having explicit "integration" projects, yet in my day-to-day discussions with colleagues on the way forward, I find that any suggestion that we do away with the ESB in favour of smart endpoints is often greeted with dismay, because many people don't really understand the word "federated". Oh no, we don't want to go back to the Dark Ages and their point-to-point connections!

I hope the 'point' is made.

Heavyweight Services Have Let IT Practitioners Down

2010-05-26T07:02:00.000-07:00

Call it coincidence, but I had two meetings in the same day when different IT folk complained that a services approach was too cumbersome for them and argued for their applications to be able to connect directly to the databases of other applications to perform queries, joins, etc.

My involuntary shudder at this suggestion betrayed my SOA leanings, of course. I don't think these people realised how much future pain they would expose the organisation to by building implicit dependencies ("tight coupling") between their independent applications. But I could also empathise with their frustration.

The first generation of services in organisations has been well-meaning in intent, but expensive in practical terms. People ranted to me about the sheer effort overheads involved in trying to access data from elsewhere - setting up queues and queue clients, formalising XML messages to drop onto queues and pick from queues, parsing and generating XML, etc., etc., - all to do something that was essentially a SQL select or a join across two tables (theirs and another app's). I also heard a fair bit about the performance overheads of calling services to do simple things - operations that take seconds to do because they involve 2 or more separate service calls. "Decoupling at what cost?" was the refrain.

I'm forced to the realisation that our collective enthusiasm for SOA, while entirely correct and justified, has not provided a uniformly satisfactory solution to everyday IT practitioners. Bluntly put, we've let down the average developer by making their job unnecessarily harder, to the extent that even experienced designers who know the benefits of loose coupling are beginning to argue for a return to the Bad Old Ways.

A reader's comment on my previous post has made me think that "Data Services", especially of the RESTian CRUD variety, could be the answer. We still have a service interface that decouples applications from each other and their gory data structures and implementation details, but now we can set them up with minimal effort and call them with minimal overhead. Data Services could be the SOA SQL (That's a pretty apt pun, actually, if you pronounce SQL as "sequel").

More and more, I'm leaning to the view that most technical problems within an enterprise can be solved with "a web server, a database and some brains". This is a rich topic for a future post or even a whitepaper, but the unsurprising insight is that the third ingredient is probably the one in short supply at most IT shops ;-). (No insult intended, just that designers and architects tend not to step back and look for solutions from first principles, but rely on precedent and product-based approaches.)

Want Reusability? Sell Off Your ESB and Hire a Data Architect

2010-05-25T01:38:00.000-07:00

I'm not being facetious. I'm sometimes subjected to entirely earnest arguments about how services must be implemented within an ESB (Enterprise Service Bus) and not at various endpoints, - in the interests of reusability! Apparently it's OK to host non-reusable services at arbitrary endpoints, but reusable ones need to be wrapped up in centralised wrapping paper and hosted on the magic box in the middle.

The words hogwash, baloney and fiddlesticks come to mind.

I've always been a fan of scalable, federated "endpointware", because I believe that services naturally form a "flat" address namespace that enables them to be hosted in a location-transparent manner, so centralising them within a physical component achieves nothing more than provide the organisation with a nice little performance bottleneck and a single point of failure. Oh, and a fat revenue stream for the vendor who can then address those limitations with an expensive HA (High-Availability) option. When was the last time the Internet was down for maintenance while someone upgraded the central server?

I'm not entirely joking when I suggest that organisations can achieve service reusability more effectively by selling their ESB (to a more gullible one) and using the money to hire a good data architect. After all, IT budgets are always tight and there are usually headcount restrictions, so some sort of swap seems the most feasible way to acquire these skills :-).

I believe most organisations have very poor data modelling skills in XML. Most organisations probably have data modellers who are skilled in relational database technology and can draw a mean ERD (Entity-Relationship Diagram). But I haven't seen too much evidence that any thought is given to XML Schemas as enterprise-reusable resources.

Banking experts! In what way is a savings account similar to a loan account? In what way is it different?

Insurance gurus! In what way is a workers compensation claim similar to a home & contents claim? In what way is it different?

A good data architect should be able to state these similarities and differences - in terms of an XML schema!

A good data architect would be able to define base types for common, abstract concepts and derived subtypes for each specialised variant. This would automatically guide the design of the service interface by constraining the elements and even the structure of message documents. Then regardless of whether the services are hosted on a single node or distributed across multiple disparate nodes and implementation technologies, the organisation will achieve service reuse. Over time, fewer versions of services will need to be maintained (another common bane), because variants have been catered for through insightful data modelling.

I'm not sure this point is appreciated enough. IT professionals are human beings, after all, and tend to understand tangible products better than abstract concepts. That's why most of them believe in magic product-based solutions rather than intellectual and conceptual disciplines. But the architects within IT must rise higher, and make their voices heard.

The alternative is a bunch of non-reusable, non-scalable non-services.

Architectural Requirements for a Line-of-Business Product

2010-04-19T16:12:00.000-07:00

I was recently asked for my views as an architect on what factors to consider when procuring a line-of-business product for an organisation. Keeping in mind that business SMEs (Subject Matter Experts) were likely to be covering the functional aspects of the product quite adequately, I came up with this list:

1. Coarse-grained business functions (e.g., underwriting, claims and reinsurance in the insurance industry) should be modular with well-defined service-based interfaces. There should be no implicit dependencies between modules (e.g., no shared tables or other knowledge of another module's internals). This will make it easier for a module to be swapped out for a better one from another source, if required. [Obviously, product vendors will fight to prevent this from happening, but the customer organisation must skew their evaluation rules in such a way that the vendor is forced to see a competitive negative as a positive.]

2. Data models of the interfaces (not the internal domain models) should conform to standards as far as possible (If a data model exists for that industry domain, that's to be preferred). The IP relating to the external data model (the universal metadata model governing the data transfer objects/XML documents) should be revealed to the customer in any case, and guaranteed to be stable across product versions, with a well-defined version strategy for service enhancements. The internal data model can remain the vendor's IP. In fact, the more opaque that is, the better. It will prevent unwitting dependencies from being built by applications external to this one. If a compact and standard vocabulary is established, it will make it much easier for other applications to integrate with this one by "learning" to speak the same language.

3. Technology platform dependencies should be minimal – commodity infrastructure is preferred (e.g., Intel, Linux, plain Java-based web servers). The product should offer choices, not constraints. Ideally, it should be able to run on any platform from mainframes to Intel-based systems, and the customer organisation should choose the most commoditised platform that will suit its needs. If the product is Java-based, it should not require an EJB container. The Spring framework is preferred. Database dependencies should be minimised (wherever possible) through an appropriate ORM layer. Again, the customer organisation should be free to choose the most commoditised database that meets their requirements.

4. Services support should include both SOAP and REST in order to support the widest variety of external service consumers (and producers). SOAP services should be true contract-first services and not automated remoting of object methods. REST services should exhibit support for enterprise requirements through simple patterns for security (SSL), reliability/idempotence, asynchronous processing with callback options, horizontal scalability through statelessness, etc.

5. Technical skills requirements should be as basic as possible (e.g., commonly available skills like Java, XML, HTML, JavaScript, etc.). There should be minimal requirement for specialised skills. Very importantly, the product should not require vendor consultancy on an ongoing basis to make modifications to extend the product's use. Extensibility of the design and flexibility of the service interface are key enablers here.

I'm sure there are other good requirements, but these should provide reasonable guidance, I think. Working through my thought process, I guess these were my guiding principles:

1. Ensure high cohesion and low coupling between functional units, keep interfaces as small as possible (this includes the shared vocabulary).
2. Be conservative in what you produce, be liberal in what you accept.
3. Aim for simplicity (but beware of expediency masquerading as simplicity).
4. Seek to use commodity building blocks as far as possible - try to source technology from competitive markets rather than from "market leaders".

Idempotence - Reliability that Works

2010-03-16T05:14:00.000-07:00

In many cases of messaging where uncertainty is undesirable (e.g., a timeout on a critical transaction request), the usual solution sought is "reliable message delivery". Apart from its theoretical impossibility, "reliable messaging" is expensive. One needs "strong queuing" products, which cost a lot of money.

I've been a fan of a far simpler approach - idempotence. We're really not after reliable message delivery in most cases. We'd often be happy to settle for certainty, i.e., we don't mind whether a requested operation succeeded, failed (or indeed, timed out before it could be attempted), as long as we know what happened. What we don't want is to be stuck, not knowing what happened and afraid to retry the operation for fear of unwitting duplication.

Idempotence is, of course, the property that attempting something more than once has exactly the same effect as attempting it once. Idempotent operations can be blindly retried in a situation of uncertainty, because one is guaranteed that the operation will never be duplicated.

The trick is to identify every transaction request with a unique identifier. Provided the receiver of the message is set up to check the identifier against previously-used ones, duplication of transactions can be avoided even if requests are sent multiple times. This is extremely powerful because it allows a requesting application to simply retry the request message in situations of uncertainty until a definite response is eventually received. There is no danger of the request being acted upon more than once.

Reliability is then reduced to an endpoint-based protocol. It does not require any special capabilities on the part of the transport. In fact, the transport can afford to be quite unreliable. Idempotence allows reliable messaging solutions to be built (and quite cheaply at that) on top of unreliable components!

Here's a one-page document that illustrates the concept.

Hopefully this should make it very clear that we don't need strong queuing or "reliable message delivery" to eliminate uncertainty. A plain web server, a database and a system of one-time tokens (UUIDs?) can solve the problem.

The Coming Overthrow of XML - Orderly Makes Further Strides

2009-12-22T20:13:00.000-08:00

My feeling that XML is due to be dethroned grows stronger by the month.

A quick recap of recent history:

First, JSON offered a simpler data structure than the angle bracketed format of Unicode XML. But that still lacked rigour around data definition, so even though XML suffered the confusion of having at least three competing schema definition languages (XML Schema, RelaxNG and Schematron), the world did have a way to specify data types, formats and constraints with XML that JSON could not match.

Then, thanks to Kris Zyp, JSON Schema appeared on the scene and plugged the rigour gap. JSON Schema parsers now exist for a number of languages including Java. One of the design decisions of JSON Schema was for a schema document itself to be valid JSON, much as XML Schema is itself valid XML. Unfortunately, this meant that brevity wasn't JSON Schema's strong point because the JSON way of expressing properties is necessarily long-winded.

The third shoe has dropped now (never mind the grotesque image that conjures up of the wearer). Orderly is a new schema language developed by Lloyd Hilaiel that is far more compact than JSON Schema and yet round-trips to JSON Schema quite effectively. [There's a really cool Ajax-y screen that converts back and forth between Orderly and JSON Schema before your eyes, so you can tweak either code to see how it looks in the other representation.]

Bottomline: SOA architects can recommend the use of the simpler JSON data format instead of XML without having to worry about the lack of rigour in data definition. Data architects, designers and developers can use Orderly to design schemas without bothering about JSON Schema's cumbersome syntax. JSON parsers can work with the equivalent JSON Schema to validate a piece of JSON data without the need to understand two different syntaxes.

A great solution, and it's all come together quite nicely in time for Christmas. Thanks to Lloyd (and Kris before him) for a wonderful Christmas present to all SOA practitioners, and ultimately, everyone wrestling with XML in any capacity.

The Meaning of Open - By Google

2009-12-22T12:37:00.000-08:00

A friend sent me this link to a piece written by Jonathan Rosenberg of Google on the meaning of the term "open". This is old hat to those of us who have already seen the light, of course ;-). [Rosenberg's calisthenics when he then tries to justify the closed bits of the Google ecosystem are quite amusing.] But to people who have not given much thought to openness and tend to follow the herd on technology (the bigger the brand name, the better), this open letter may hold many eye-opening insights (all puns intended).

I would perhaps have said what Rosenberg did in half the length, but brevity isn't a necessary quality of openness, so I'll forgive him :-). The main danger of the unnecessary length is it may just cause some of the audience to stop reading before the end, when Rosenberg delivers his most inspired paragraph:

Open will win. It will win on the Internet and will then cascade across many walks of life: The future of government is transparency. The future of commerce is information symmetry. The future of culture is freedom. The future of science and medicine is collaboration. The future of entertainment is participation. Each of these futures depends on an open Internet.

Amen to that. In fact, that's worth repeating in a more structured form:

The future of government is transparency.
The future of commerce is information symmetry.
The future of culture is freedom.
The future of science and medicine is collaboration.
The future of entertainment is participation.

I would like to analyse these in greater detail and add to/modify the list, because I'm sure this is incomplete.

For now, this is a document that is worth circulating to our brand name-dazzled colleagues. After all, Google is one of the biggest brands out there, so if Google is endorsing openness, there must be something in it ;-).

Now if only IBM would come out with an Open^TM line of products, we would be willing to write a cheque...

Is Canonical Trying to Purge Ubuntu of the L-word?

2009-12-11T02:51:00.000-08:00

I don't think much of revisionist history, and biting the hand that feeds isn't an endearing trait.

Visiting the Ubuntu site today after a while, I was unpleasantly surprised that I couldn't see the word "Linux" anywhere. After trawling the site exhaustively, I did find two or three references, and I leave it as an exercise for the reader to find more. Warning: You'll have to search really hard.

Under "What is Ubuntu?", the site says, "Ubuntu is a community developed operating system that is perfect for laptops, desktops and servers".

What's up, guys? Does it hurt a lot to use the phrase "based on Linux" somewhere in that sentence?

Canonical and Ubuntu, great as their contributions have been, would be nowhere without Linux, especially the Debian distribution. So why not acknowledge that debt? Why try to pass Ubuntu off to newbies as a completely original operating system with no ties to Linux?

On the same page, right at the bottom, there's a section titled "What does Ubuntu mean?" and it goes on to explain, "Ubuntu is an African word meaning 'Humanity to others', or 'I am what I am because of who we all are'."

How apt. Dear Canonical, why not show some Ubuntu (humanity to others) and acknowledge that you are who you are because of what Linux is?

Advice To Organisations Embarking On SOA Today

2009-12-03T19:08:00.000-08:00

I have been involved with SOA in various roles over the last two to three years, and my thinking has evolved a fair bit over this period. If I was asked to advise an organisation embarking on a major SOA initiative today, I would probably say this to them:

1. The End Goal: Remember that SOA is not about integration but about inherent interoperability. Think health, not medication. SOA is about raising the capability of your systems such that they can easily and inexpensively integrate with others, not about introducing a new, slick technology that will connect your systems together more easily. Simplifying the components of your enterprise and making them easy to understand and connect to will give you SOA. So keep simplicity at the back of your mind all the time, and don't confuse it with expediency, which is the path of least resistance. Simplification could take some effort.

2. Domain Models: Don't waste too much time searching for "the" canonical data model. Most off-the-shelf ones are too high-level and abstract to be useful. And building your own comprehensive dictionary is wasteful and time-consuming. Instead, identify logical owners of different elements of data and let them own the data dictionary for those elements. All services exposed out of these logical domains should use these definitions and it is the responsibility of service consumers from other domains to understand them. Processes that combine services crossing such domains should perform their own mapping between similar data elements. It's necessarily messy and plenty of out-of-band communication will be required at design time. After all, even similarly-named elements may suffer from subtle interpretation errors, so manual discussion and clarification will always be part of a service or process design.

This is not as bad as it sounds because only a subset of data elements managed by a domain is exposed through its service interface, and it's only these that may need translation to the context of their consumers. Don't look to do away with this manual effort through a single canonical data model. That's a wild goose chase, so don't even start.

3. Infrastructure and Connectivity: Try and avoid using message queues unless you're looking at low latency situations like a trading floor, or if there's simply no other way to interface to a legacy system. The queuing paradigm introduces various coordination issues into application design, and implementing message queues requires establishing more complex patterns to solve these gratuitous problems. [I have a larger, philosophical argument about the need to innovate an application protocol on top of an asynchronous, peer-to-peer transport, but let's not confuse the current set of recommendations with that idea.]

In today's world, HTTP-based communication patterns backed up by databases will often do the trick more simply than expensive message queues. Look beyond the apparent need for reliable message delivery. Often, an idempotent operation will suffice to meet the real requirement, and this is quite a standard pattern to implement. Often, queues are used in synchronous (blocking) patterns anyway (to avoid the coordination problems I talked about), so nothing is being gained in an architectural sense by the use of queues. And even asynchronous communications, where required, can be implemented in standard ways over HTTP, so HTTP is quite a universal protocol to use as the logical infrastructural element for your SOA.

ESBs, Service Directories and other "governance" components are often only required to manage the complexity that they themselves introduce. It's amazing what you can achieve with a farm of simple web servers and a database, and still keep things simple and understandable.

4. Service Enablement: Try and avoid the entire SOAP/WS-* stack of technologies if you can. There is a significant complexity overhead associated with this set of technologies, and you will need an expensive toolset to (partially) simplify its use. Look seriously at REST instead. Even though REST advocates don't make the case strongly enough (and sometimes see SOA as an antithetical philosophy), REST is in fact a valid way to do SOA and can usually help to deliver solutions at much lower cost and complexity. The hard part about doing REST is finding good people who can think that way. REST is subtly different from the SOAP/WS-* approach, even though they may just look like different kinds of plumbing to move XML documents around (and I confess that's the way I initially sell REST to corporate skeptics brought up on a diet of vendor-provided Web Services propaganda).

5. Data Contract: Consider alternatives to XML for the data contract. Though this sounds like heresy, XML is heavyweight and cumbersome, and XML manipulation tools in high-level languages (with the possible exception of E4X in JavaScript) are clumsy to use and suffer major impedance mismatches. You will spend more time wrestling with XML than on the service itself. Although many in the web world will immediately recommend JSON, raw JSON is not sufficient to ensure data integrity, because it has hitherto lacked a strong schema definition capability. Maintain a watching brief on the JSON Schema proposal, submitted for approva l as an IETF standard. Already, there are JSON Schema libraries in many high-level languages such as Java. It should be possible to define data contracts with as much rigour as with XML, but at a much lower level of complexity. A newer and more compact JSON Schema representation called Orderly is also maturing, which makes this approach simple as well as easy.

So instead of going down the XML rabbit-hole, start with JSON anyway, and incorporate JSON Schema/Orderly as it matures. You will find this works especially well in combination with REST. A quick Proof-of-Concept may convince the skeptics in your organisation (although the opposite result may also occur, with many going away convinced by the speed of this approach that it's either simplistic or too good to be true!)

6. Web Service Implementation: If you're trapped by circumstances into an XML-and-SOAP/WS-* approach, look at the WSO2 suite of commercially-supported Open Source products. Especially look at the WSO2 Mashup Server. Don't be fooled by the name. It's more than just a mashup server. It's a service orchestration engine that (curiously) uses server-side JavaScript as its programming language. The major advantage of JavaScript is the ability to use the E4X library to perform extremely straightforward XML manipulation. Once you use E4X, you will never go back to JAXB or any other XML-processing library. WSO2 Mashup Server allows SOAP or REST services to be consumed, combined and orchestrated, and in turn exposes SOAP or REST services. It's a good way to hedge your bets if you're only half-convinced about REST. The WSO2 suite is also much less expensive than its proprietary rivals, although the real expense is in the heavyweight approach that it unfortunately shares with them.

7. The Paradox: SOA is really all about simplicity, but it's hard to find SOA architects who seek to simplify systems. Conventional SOA practice seems to be about making integration complex through heavyweight approaches, then introducing tools to manage that complexity, tools that require specialist skills to use properly. If done the conventional way as most SOA consultants seem to agree, your SOA initiative will only leave you with additional complexity to manage.

Of course, if you're politically inclined, you will bask in the prestige of a hefty budget and a large team, and can declare victory anyway on the basis of the number of services and processes you have delivered. But if you want to be really successful at delivering SOA (i.e., making your business more agile and able to operate on a sustainably lower cost basis) while keeping your burn rate low along that journey, you would do well to look at boring, unimpressive and even anticlimactic approaches and technologies such as the ones I've listed above. Give the big vendors a wide berth. You don't need to buy technology (beyond the web servers and databases you already have). You certainly don't need to buy complex technology, which is what the vendors all want to sell you.

And don't let the lack of grandeur in this approach worry you. Complexity impresses the novice, but results are what ultimately impress all.

Postscript: Vendor REST is coming. Beware.

Spring Roo Makes a Quiet Debut

2009-11-16T05:46:00.000-08:00

This has always been the Holy Grail (pun definitely intended!): To design an application using just object-orientation as the paradigm, and to have persistence, a service interface and a sensible user interface all generated for you automatically. In other words, just define the core behaviour of the application's objects, and you have a working web application with non-visual service interfaces to boot. This is the Domain-Driven Design dream (DDDD?)

Spring Roo has been talked about for a while, but Grails beat it to market. However, it's better late than never for the product from the SpringSource stables, and Release Candidate 3 is now available.

For a teaser on what Roo can do, see this video. It's a third-party video based on RC2 (and you'll see some syntax differences when you start to play with RC3), but the spirit is the same.

Exciting stuff!

Update 17/11/2009: I sent the video link to a friend and colleague for his feedback. He 's a senior architect and based on his experience, he believes Roo needs to provide an Eclipse-based graphical interface rather than (in addition to?) a command-line interface. With that, he believes it's a winner.

Cisco Buys Jabber - What Does It Mean?

2009-11-11T16:08:00.000-08:00

Cisco has bought Jabber (another of those purchases of Open Source companies that make no sense to me - what are they really buying when the IP is public?)

What it does indicate to me is that the Presence and Instant Messaging protocol pioneered by Jabber (XMPP) is now becoming a more widely adopted standard. GoogleTalk and Avaya have been using XMPP for a while now, and Cisco has just joined the bandwagon.

My idea of a Unified Communications architecture is something that leverages SIP/SIMPLE, XMPP, HTTP and the open email protocols, rather than a vendor-proprietary stack. Microsoft and IBM have interesting stories in the UC space, but I would argue that this would be precisely the wrong time for an organisation to go with a proprietary suite of products. There's no such thing as a free pair of handcuffs.

I had drawn up this architecture diagram on UC some time ago but didn't publicise it because it seemed a bit theoretical at the time. But now it seems to be more relevant with the strengthening of XMPP and SIP/SIMPLE.

FST Media Annual Technology and Innovation Seminar Day One (05/11/2009)

2009-11-05T00:07:00.000-08:00

I attended FST Media's 4th annual seminar on technology and innovation themed "The Future of Banking and Financial Services" at the Hilton in Sydney.

Some quick impressions:

Don Koch, CEO of ING Direct, delivered the opening keynote and spoke about what customers will want next (answer: who knows?). (Incidentally, every single talk was billed as a "keynote", which leads one to question if FST Media knows what a keynote really is.) He made a number of points but none stuck in my head as anything particularly insightful.

Rocky Scopelliti, General Manager, Financial Services, Industry Development, Telstra Enterprise & Government, spoke next about Gen Y and made some interesting points. Essentially, he was reporting on research by noted Australian sociologist Hugh Mackay. Although Gen Y-ers have a reputation for being difficult to please and notoriously non-loyal, the research suggests that Gen Y-ers are in fact influenced by their parents, the Baby Boomers, who in turn grew up influenced by two opposing forces - on the one hand, abundant and growing prosperity that promised a better tomorrow, and on the other, by the threat of the Cold War and the possibility that there may be no tomorrow. The unique solution to this was the demand for instant gratification - enjoy life like there's no tomorrow. So Gen Y is not really demanding and disloyal; they're just keeping their options open for as long as possible.

I forget whether Koch or Scopelliti made this point, but it was interesting that nowadays, all you need to establish a person's identity is their date of birth and mobile phone number. People tend not to change their mobile numbers, even if they switch providers. More on this later.

David Boyle, CIO, International Financial Services, Commonwealth Bank next spoke about cloud computing at CBA. The interesting aspect of his talk was his call for fellow customers to get together with him and his team to define some basic standards for cloud computing, because he's afraid of the same thing that I am, i.e., that some vendor-proprietary technology will become the de facto standard for cloud computing which will make it unnecessarily pricey.

There were a few eminently forgettable panel discussions (with the only exception being the one on which Dhiren Kulkarni, joint CIO of St George Bank, was being needlessly competitive and obnoxious towards his fellow panelists. He earned a rejoinder at one stage from Pravir Vohra, CTO of ICICI Bank, which was roundly applauded.)

The next talk was billed as an "International Keynote" and was delivered by the aforementioned Pravir Vohra, who provided a few interesting insights into the growth of ICICI Bank. It's been almost 15 years since I left India, and I didn't know ICICI Bank had become the second largest Indian Bank. I guess it would still be a distant second to State Bank of India, though. SBI has more branches than some banks have customers :-).

Vohra would be the envy of CTOs everywhere, because he says he has 25% of the technology budget to play with. I.e., he's free to invest in various technologies and initiatives. If they work out, he "transfer prices" the benefits to the business units that he can serve with it. I have always felt that a similar system is required for Enterprise Utilities. Neither the "first project pays" model nor a Big Bang enterprise rollout in a single financial year is the right one. We need an accounting treatment for Utilities akin to R&D, with carryovers possible across financial years.

I was very pleased to hear that ICICI Bank only uses OpenOffice, not Microsoft Office. Vohra estimates that this choice saves his bank more than $15 million a year, even more than the projected savings from cloud computing! Who said the Third World lags behind the First? Sometimes they lead the way.

One thing Vohra said that didn't seem quite right was his contrast of consumer behaviour in India with that in Australia - he said Indians change their mobile numbers every six months! Apart from the fact that the Indians I know haven't changed their mobile numbers in quite a while, it seems very counter-intuitive behaviour. Why would anyone make life more difficult for themselves in this manner?

The three sessions after lunch and before tea were uniformly missable.

David Cartwright, Group Managing Director, Operations, Technology and Shared Services, ANZ was probably the best of the lot. He spoke about ANZ Bank's "Journey to a Super-Regional Bank". An interesting aspect was the emphasis ANZ Bank places on its new corporate headquarters in Melbourne that has a 6 star green rating. It resembles a skyscraper on its side and is deliberately designed to have a huge floor area on a limited number of floors so as to maximise interaction between people. Another interesting technology they have is "telepresence", which is very high quality videoconferencing, which gives the impression that remote participants are actually in the room on the opposite side of the table.

The aforementioned Dhiren Kulkarni spoke next, and repeatedly parrotted his boss Gail Kelly's favourite phrase "delighting the customer" ad nauseam. You don't get more patronising, defensive, competitive and humourless than this bloke. Not very good PR for Westpac-St George to trot him out at industry seminars.

Rounding out the boring lot was Darren Covington, Vice President, Asia Pacific Japan, Enterprise Software Applications, Hewlett Packard. He spoke about churn and the desirability of retaining customers because it's far cheaper than acquiring new ones (a well-worn concept), but added no new insights. My cynical reaction was that with the Australian banking sector being as oligopolistic as it is, a customer who walks out the door is readily replaced by another bank's customer who walks in. There are only four banks in the game, so where will customers go anyway? It's not so much churn as a nice merry-go-round.

After tea, we had Randy Fennel, General Manager, Engineering and Sustainability, Technology, Westpac, talking about sustainability, which again was a bit ho-hum.

The last session of the day was by far the best. Phil Hopper, founder of iGrin spoke about peer-to-peer lending. This is an extremely interesting concept that I hadn't heard of before. There have been players like Zopa, LendingClub, etc., who have all fallen on hard times now, both because of the Global Financial Crisis and because of some increased regulatory requirements.

Peer-to-peer lending exploits the big spread enjoyed by the large institutional lenders. Banks borrow from individuals at 5% (the deposit interest rate) and offer personal loans to individuals at anywhere over 12%, sometimes going up to 18% in the case of some credit cards. This means that there is a spread of at least 7% here to be exploited if individuals lend direct to each other. A P2P lending marketplace can take 2%, leaving 5% to be split between lender and borrower, with both consequently better off than by going through banks. Hopper described P2P lending as "eBay for money".

He also spoke about Australia currently operating on a model of negative credit data only, which makes it hard to assess the creditworthiness of most people. But there is apparently some privacy-related legislation due shortly that will remove impediments to acquiring (positive) credit data.

Other points were the subtle differences between microfinance, lending to the underbanked, and prime lending. Hopper pointed out that by ceding a degree of control to customers, P2P lending agencies allowed innovation to happen, such as "blenders" (arbitrage players who use their superior credit rating to borrow at lower rates and lend to others at a slightly higher rate), and people who run Self-managed Super Funds who invest (lend) from the fund into the P2P market and then borrow the money right back. Some of this sounds dodgy to me (and to the Tax Office, I'm sure), but it certainly is innovative.

In spite of the temporary problems faced by P2P players including iGrin, Hopper sees a good future for the concept. There are niches (after the Online Auction itself) that can be independently filled by various players, such as Identity Verification, Credit Assessment, Fulfillment/Settlement (both contracts and statements) and Collections. A bill likely to be passed by parliament soon (the Private Properties Securities Bill) will also allow people to securitise their own assets like their property, enabling lending to become more "secured". Once a secondary market develops, it will be possible for the P2P segment to expand from its personal lending base (with a typical term between 12 months and 5 years) into the mortgage market (with terms of upto 30 years). This is because although the typical small lender would not be willing to enter into 30-year loans, the emergence of a secondary market will make this possible, since debt obligations like these can themselves be traded. Wholesale funds are another step along the development of P2P lending.

All in all, a most fascinating concept, and Phil Hopper did an excellent job of presenting it. It seemed a little too slick, though. As moderator and host Michael Pascoe asked him, I wonder if this is a case of amateurs attempting to take on the professionals. To a more pointed question, Hopper defended the eventual viability of the "amateur" P2P market even against the eventual entry of the "professionals", and Pascoe insightfully remarked that he would rather have had Hopper express a desire to be bought out by a bank.

That was the end of the first day.

Rare Home Truths about Windows

2009-10-09T06:16:00.000-07:00

I never expected to actually read such news. A senior police officer spoke to members of parliament and candidly told them that they should use Linux if they expect secure Internet banking.

I guess the truth always comes out in the end. Microsoft has lies, hush money and non-tech savvy users on its side, but as Lincoln said, you can't fool all the people all the time.

Of course, I could also tell that we still have a long way to go.

The collection of MPs listening to van der Graaf were very enthusiastic about his suggestion but didn't understand what he meant and asked for clarification.

"You may need to explain further for us," said one MP, while another responded, "yes, we need to understand that".

On the brighter side, knowledge comes from asking questions and finding out more. When lay users find out more about Ubuntu Linux, I doubt if many will stay with Windows. The success of Firefox proves that people are not wedded to Microsoft's products.

JSON Schema becomes more Orderly

2009-10-06T15:13:00.000-07:00

I have been convinced for a while that just like REST will gradually displace its more heavyweight SOAP/WS-* equivalent, JSON will slowly displace the mighty XML in its various strongholds (today Web Services, tomorrow the world :-). But to do that, JSON first needs to incorporate some rigour into its definition, using an equivalent to XML Schema, Relax NG or Schematron.

The JSON Schema proposal seemed to fit the bill quite nicely, but I was always vaguely uneasy that it was so verbose. There was probably no escape from that, since one of the requirements was that JSON Schema should itself be valid JSON (otherwise two parsers would be needed to consume a snippet of schema-compliant JSON).

Now along comes another schema syntax for JSON called Orderly, which has the twin advantages of being succinct and being able to round-trip to JSON Schema. The syntax has already been revised with inputs from commenters, and is looking much better in its second version.

Orderly's main advantage is its human-readability and -composability. Its simplicity (with no loss of rigour) will give JSON (and JSON Schema) the impetus they need to challenge XML. If Orderly catches fire, I believe it will accelerate the adoption of JSON for serious service-oriented work.

It's overdue.

REST is Polymorphic CRUD

2009-09-22T04:05:00.001-07:00

I've been trying to evangelise REST a bit lately and have had mixed success. There is cautious interest. But there are also huge conceptual hurdles to be overcome. Pete Lacey said it best about enterprise IT folk when it comes to REST: They Can't Hear You.

One architect looked at the definition of a service interface I proposed and thought it a bit "bland". Perhaps it just needed a big WSDL file, lots of XML and SOAP faults!

Another common reaction to REST when it's presented is that "it's just CRUD", with the implication that it's just too fine-grained to be used to create good business services. I've been struggling to explain that just because REST uses four HTTP verbs that correspond roughly to CRUD operations, it doesn't necessarily mean that REST is a CRUD approach to manage data at a very low and detailed level. The resources on which the verbs operate can be arbitrarily coarse-grained. But what has eluded me so far is a succinct term that can drive the point home.

I think I've finally found it - "Polymorphic CRUD".

IT folk in the enterprise understand both polymorphism and CRUD, so the combined term should make sense. I want to drive home the point that a verb itself is neither coarse- nor fine-grained, it's how each resource interprets it. Fine-grained resources will interpret the REST verbs as CRUD operations. But more coarse-grained resources can interpret the verbs as any arbitrary business operation.

Accompanied by the appropriate payload, POSTing to the resource "/applications" is nothing but submitting an application. There's no need for a specific "submitApplication" method.

I've also realised that one can clear a process inbox by DELETEing a "/pending" resource, with a standard WebDAV status code in response (207 Multi-Status), indicating that different items encountered different status codes during the batch process.

It's the way the verb is interpreted by each resource that gives it its meaning in that context. Therefore the REST approach is to manipulate business objects of arbitrary size and complexity through polymorphic CRUD operations.

I hope that gets people to go "Aha!"

The Answer to Complexity

2009-09-14T21:04:00.000-07:00

This sounds trite, but long arguments with colleagues in IT have convinced me that this is anything but obvious and therefore needs to be explicitly stated.

Guys, the answer to complexity is not better tooling.

It's simplicity.

An Enterprise Technology Stack for Business Agility and Sustainably Low Cost

2009-08-11T07:08:00.000-07:00

I have been musing about this for a few years now, but some recent discussions with colleagues caused my thinking to finally crystallise. I realise that the two enterprise IT objectives of delivering business agility and sustainably low cost are both realisable through a fairly simple, though disciplined, architectural approach. There is no real trade-off between flexibility and cost-effectiveness, although hard choices need to be made and problematic technologies need to be fearlessly called out.

I've attempted to build this logic out into an argument in this presentation. I'm sure there will be many who will object vigorously to various aspects of this argument, but I'm confident that this logic will appear obvious in hindsight within a decade.

For those who don't have the patience to go through the entire presentation, the final technology stack is illustrated below.

Modelling Resources from First Principles

2009-07-22T05:30:00.000-07:00

I've been providing architectural advice to a group of colleagues who are building a set of services. Without going into too much detail, they need to uniquely identify some entities. Clients of the services use these identifiers as references when they return to make related queries on these entities. They've proposed using UUIDs as the unique identifiers, and while I liked the idea, I thought it was too simplistic. There was more to the requirement than just unique identifiers.

They're actually dealing with two types of entities - widgets (say) and requests for widgets. These are different because a request can pertain to a set of widgets, so it may be necessary to model them distinctly. The service interfaces dealing with requests and widgets may need to be distinct as well.

Mind you, these services are not going to be REST services. But having been exposed to the RESTian way of thinking, I immediately thought of resource representations for the two types of entities. Rather than plain old UUIDs, I thought there should be a degree of structure around them (but not so much detail as to make the scheme brittle and inflexible).

Something like these, in other words:

http://www.mycompany.com/widgets/4f138ff2-362f-4e35-8f9e-173290fe86d7

http://www.mycompany.com/widget-requests/eabf5bdb-9800-4af5-9ad7-32c3b95fc48a

However, this suggestion proved to be a surprisingly hard sell. The cross-examination was withering.

Why all the extra information?
Why http://?
Why www.mycompany.com?

Why not a simpler scheme like these examples show:

widget:4f138ff2-362f-4e35-8f9e-173290fe86d7

widget-request:eabf5bdb-9800-4af5-9ad7-32c3b95fc48a

I found I had to retrace my steps and work through my reasoning from first principles. In the process, I learnt a great deal about naming.

My initial response was to point my colleagues to points 7 and 8 of "Common REST Mistakes", where we are admonished not to try and invent our own proprietary object identifiers, and not to try for "protocol independence" (i.e., avoid HTTP URIs). But this wasn't too convincing.

I made a bit of progress by getting agreement on the following:

1. It probably made sense to distinguish between widget identifiers and widget-request identifiers, so some sort of prefix to distinguish between them was necessary. UUIDs alone were probably not enough.
2. It also probably made sense to specify the "domain" within which these resources were being identified, so the "mycompany" string probably belonged somewhere as well.

But then, why not just these:

mycompany:widget:4f138ff2-362f-4e35-8f9e-173290fe86d7

mycompany:widget-request:eabf5bdb-9800-4af5-9ad7-32c3b95fc48a

Frankly, I hated this. My point was that such a format, even though "simple", would have to be explained to anyone looking at it. The structure wasn't immediately obvious. Worse, it was ambiguous and could be extended by later designers in ways that violated the original designers' intent. To this, the counterargument was that the knowledge of the format was only required on the server side. To the client, the whole name was just going to be an opaque string, - a reference ID.

I wrestled with this objection for a while. Then I proposed a guiding principle that given a choice between two naming conventions, a universally understood one was preferable to one that we made up ourselves, provided it wasn't unnecessarily complex.

My research led me to the definition of a URN (Universal Resource Name). What I learnt from this was that in order to name something, we first need to specify a "scheme" that then defines what the rest of the name denotes according to the predefined format for that scheme. The name of the scheme is followed by a colon, then the rest of the name is something that can only be interpreted according to the rules specified by that scheme.

In other words, a standard name (URN) looks like this:

<scheme name>:<some scheme-specific format>

A common example is

"http://www.mycompany.com/widgets/4f138ff2-362f-4e35-8f9e-173290fe86d7" (Heh!)

It's important to point out that "http" in the string above does not refer to the HTTP protocol! It's the name of a "scheme". What does this mean?

Well, in the URN "file:///home/ganesh", the string "file" is not a protocol, because more than one protocol may be used to get to the file.

Similarly, in the URN "mailto:ganesh@mycompany.com", the string "mailto" is not a protocol. SMTP is the actual mail protocol.

[For those familiar with XML namespaces, when we say "xmlns='http://www.example.com/schema'", the URN being referred to here is not necessarily a web page that one can point a browser at. It just needs to be a unique string.]

So we're not necessarily modelling our resources as web resources. All that the "http" scheme defines is that after the colon (":"), there is a scheme-specific structure that specifies a few things.

There are two slashes, then there's a dot-separated domain name, then a slash, then a "resource path" which is itself slash-separated. So that's what a URN conforming to the "http" scheme looks like:

"http" (the name of the scheme)
":" (the colon separating the name of the scheme from the scheme specific structure. This is from the basic definition of a URN)
"//" (the "http" scheme just specifies this, OK?)
"www.mycompany.com" (this is the dot-separated domain name)
"/" (this is the first slash that signifies that the domain name is terminated)
"widgets/4f138ff2-362f-4e35-8f9e-173290fe86d7" (all of this is the "resource path" , and internal slashes are possible, as we can see)

So now going back to our guiding principle (using a well-understood format is preferable to rolling our own) as well as the two points on which there was agreement (i.e., that we may need to qualify the resource's UUID with the type of resource as well as the organisational domain), it looks like the "http" scheme of the URN naming standard fits the bill. This is a well-understood way to include both a domain and a resource path to provide some structure around an already unique ID.

I concede that the "www" prefix of the domain could confuse. All we really need to identify the domain is "mycompany.com".

And so, a unique, standards-based and minimal way to name resources in this business domain would be

http://mycompany.com/widgets/4f138ff2-362f-4e35-8f9e-173290fe86d7

http://mycompany.com/widget-requests/eabf5bdb-9800-4af5-9ad7-32c3b95fc48a

What's Wrong With Vendor Lock-In?

2009-06-23T02:06:00.000-07:00

A colleague asked me this question today, and I'm really glad it came out in the open, because it's so much easier to deal with ideas when they're plainly stated.

I believe there are at least two strong reasons to actively avoid vendor lock-in.

The first reason, paradoxically, lies in the very justification for agreeing to be locked in - "ease of integration". If an organisation already has a product from a certain vendor and is looking for another product that needs to integrate with this one, then conventional thinking is to go back to the same vendor and buy their offering rather than a competitor's. After all, they're far more likely to be "well integrated". There are typically so many problems integrating products from different vendors that it doesn't seem worth the effort. The best-of-breed approach leads to integration problems, so customer organisations often throw in the towel and go for an "integrated stack" of products from one company.

This approach is antithetical to SOA thinking, however. What they're really saying here is that they don't mind implicit contracts between systems as long as they work. But implicit contracts are a form of tight coupling, and as we know, tight coupling is brittle. Upgrade one product and you'll very likely need to upgrade the other as well. In production systems, we have seen product upgrades delayed for years because the implicit dependencies between "well-integrated" components could cause some of them to break on an upgrade, which is unacceptable in mission-critical systems. As a result, many useful features of later versions are forfeited. That's one of the unseen, downstream costs of the tight coupling that comes from vendor lock-in.

SOA shows us the solution. For robust integration between systems, we need loose coupling between them, which seems a bit paradoxical. Shouldn't tight coupling be more robust? Hard experience has taught us otherwise. But what is loose coupling? It's a hard concept to visualise, so let's define it in terms of tight coupling, which is easier to understand. In practical terms, loose coupling between two systems can be thought of as tight coupling to a mutually agreed, rigid contract rather than to each other. Such contracts are very often nothing more than open standards.

Even though people generally nod their heads when asked about their preference for open standards, the contradiction between that stated preference and their practical choices in favour of "integrated stacks" is hard to understand. If pressed, they might say that creating contracts external to both systems and insisting on adherence to them seems like a waste of time. Why not something that just works "out of the box"? The answer is that this is not an either-or choice. A browser and a web server work together "out of the box", but they do so not because they come from the same company but because they both adhere to the same rigid contract, which is the set of IETF and W3C standards (HTTP, HTML, CSS and JavaScript).

The key is to hold vendors' feet to the fire on adherence to open standards. This isn't that hard. With Open Source products available in the market to keep vendors honest, customers have only themselves to blame if they let themselves get locked in.

The second reason why vendor lock-in is a bad idea is because it often implies vendor lock-out. Very often, customers are held hostage to the politics of vendor competition. Once you start buying into a particular vendor's stack, it will become increasingly hard to look elsewhere. Any third-party component you procure will be less likely to play well with the ones you already have, causing you more integration headaches. My favourite example from just a few years ago is Oracle Portal Server, which claimed to support authentication against LDAP. It turned out later that it wasn't just any LDAP server but just Oracle's own OID. This meant that the corporate directory server which happened to be IBM Tivoli LDAP couldn't be used. The data in it had to be painfully replicated to OID to allow Oracle Portal Server to work.

My solution to incipient vendor lock-in would be to aggressively seek standard interfaces even between products from the same company. I remember asking an IBM rep why they weren't enabling SMTP, IMAP and vCalendar as the mail protocols between Notes client and server. The rep sneered at me as if I was mad. Why would you use these protocols, he wanted to know, when Notes client and server are so "tightly integrated"? Others on my side of the fence agreed with him. Well, the answer came back to bite the company many years later, when they wanted to migrate to Outlook on the client side. Their "tight integration" had resulted in vendor lock-out, preventing them from connecting to Notes server using Outlook (which standard protocols would have allowed) and they were stuck with Notes client indefinitely. By that time, there were too many other dependencies that had been allowed to work their way in, so enabling open protocols at that stage was no longer an option. That was a great outcome for IBM, of course, but to this day, there are people in the customer organisation who don't see that what happened to them was a direct consequence of their neglect of open interfaces in favour of closed, "tight" integration.

Ultimately, vendor lock-in has implications of cost, time and effort, which basically boils down to cost, cost and cost.

As users of technology, we simply cannot afford it.

Microsoft May Lose Some Fair-Weather Friends

2009-06-22T17:37:00.000-07:00

I read the news of the release of Microsoft Security Essentials with some amusement.

One reason for my amusement is the notion that an operating system should require a separate product to ensure its security instead of having security built into its design.

The other reason is the anticipation of the impact this will have on that group of parasites in the Windows ecosystem. I'm talking about the makers of anti-virus software.

For a long time now, these companies have been less-than-honest players in the industry, revelling in the fact that the inherent vulnerabilities in Windows have given them a steady income stream and acting like they have the best interests of the customer at heart, when in fact they have always fought true advances in computer security that would have put them out of business.

The FUD from these players against Linux has been astounding. A common refrain is that "Linux is only secure because no one uses it. When its profile rises, hackers and malware writers will turn their attention to it." Really? How come IIS used to attract a disproportionate share of web server attacks in spite of Open Source Apache having twice its market share at the time? Surely it's badly-designed systems that invite attack.

These folk even try and sell anti-virus software for Linux! This would certainly fool people who don't realise that Linux itself is the best anti-virus software you can install on your computer. I haven't been hit by malware since 1997, when I first installed Slackware Linux on my PC.

So what will Microsoft's announcement of free anti-virus protection do to the likes of McAfee and Symantec? While users will probably be going, "It's about time," I can imagine a very different reaction at these companies.

I'm not shedding any tears, though.

JavaOne 2009 Day Five (05/06/2009)

2009-06-08T04:48:00.000-07:00

The final day of JavaOne 2009 began with a General Session titled "James Gosling's Toy Show". This featured the creator of Java playing host to a long line of people representing organisations that were using Java technology in highly innovative and useful ways. Many of them got Duke's Choice awards.

First up was Ari Zilka (CEO, Terracotta) who was given the award for a tool that makes distributed JVMs appear like one, thereby providing a new and different model of clustering and scalability. Terracotta also allows new servers and JVMs to be added on the fly to scale running Java applications.

Brendan Humphreys ("Chief Code Poet", Atlassian) received the award for Atlassian's tool Clover. [Brendan is from Atlassian's Sydney office, and I've met him there on occasion when Atlassian hosts the Sydney Java User Group meetings.] Clover is about making testing easier by identifying which tests apply to which part of an application. When changing part of an application, Clover helps to run only the tests that apply to that part of the code.

Ian Utting, Poul Henriksen and Darin McCall of BlueJ were recognised (though not with a Duke's Choice award) for their work on Greenfoot, a teaching tool for children. Most of their young users are in India and China, but it's not clear how many there are, because they only interact with the Greenfoot team through their teachers.

Mark Gerhard (CEO of Jagex) was called up to talk about RuneScape. [Mark Gerhard received the Duke's Choice Award on Day 2, as diligent readers of this blog would remember.] This time, the focus was not on the game itself, but on the infrastructure it took to run it. According to Gerhard, Jagex runs the world's second-biggest online forum. There is no firewall in front of the RuneScape servers (!), so they get the full brunt of their user load. The servers haven't been rebooted in years. Jagex had to buid their own toolchain, because a toolchain needs to understand how to package an application for streaming, which off-the-shelf equivalents don't know to do. Jagex runs commodity servers (about 20?) and their support team has just 3 people. Considering that their user base numbers 175 million (10 million of whom are active at any time), this is a stupendous ratio. Of course, Jagex has about 400 other staff, mainly the game developers. Jagex builds their libraries and frameworks in-house, and they maintain feature parity with their commercial competitor, Maya. I found it curious that Gerhard was cagey when asked which version of Java they used. Why would that need to be a secret? All he would say was that we could guess the version from the fact that their servers hadn't been rebooted in 5 years.

By way of vision, Gerhard said that OpenGL would be standard on cell phones in a year, and Jagex's philosophy is that "There's no place a game shouldn't be".

The next people on stage were two researchers from Sun itelf, - Simon Ritter and Angela Caicedo. [Caicedo had been at the Sydney Developer Day only a couple of weeks earlier.] Ritter demonstrated a cool modification of the Wii remote control, especially its infra-red camera. The remote talks Bluetooth to a controller. I didn't grasp the details of how the system was built, although I heard the terms Wii RemoteJ and JavaFX being mentioned. Ritter held a screen in front of him, and a playing card was projected onto it. Nothing hi-tech there. When he rotated the screen by 90 degrees, the projected image rotated as well, which was interesting. But what brought applause was when he flipped the screen around, and the projected image switched to the back of the card! He also showed how a new card could be projected onto the screen by just shaking the screen a bit (shades of the iPod Shuffle there).

Caicedo demonstrated a cool technology that she thought might be useful to people like herself who had young children with a fondness for writing on walls. With a special glove that had a embedded infra-red chip, she could "draw" on a screen with her finger, because a projector would trace out whatever she was drawing based on a detection of the position of her finger at any given time. The application was a regular paint application, allowing the user to select colours from a toolbar and even mix them on a palette.

Tor Norbye (Principal Researcher, Sun) then gave the audience a sneak preview of the JavaFX authoring tool that has not yet been released. Very neat animations can be designed. It's possible to drag an image to various positions and map them to points in time. Then the tool interpolates all positions between them and shows an animation that creates an effect of smooth motion, bouncing images, etc. There are several controls available, like buttons and sliders, and it's possible to visually map between the actions of controls and behaviours of objects. It reminded me of the BeanBox that came with Java 1.1, which showed how JavaBeans could be designed to map events and controls. The lists of events and actions appear in dropdowns through introspection.

There's no edit-compile cycle, which speeds up development. Norbye showed how the same animation could be repurposed to different devices and form factors. There's a master-slave relationship between the main screen and the screens for various devices, such that any change made to the main screen is reflected in the device-specific screens, but any specific overrides made on a device-specific screen remain restricted to that screen alone.

Fritjof Boger-Engelhardtsen of Telenor gave us a demo on a technology I don't pretend to understand. In the mobile world, the SIM card platform is very interesting to operators. The next generation of SIM cards will be TCP/IP connected nodes, with motion sensors, WiFi, etc., embedded within the card. It will be a JavaCard 3 platform. It's possible to use one's own SIM card to authenticate to the network. FB-E gave us a demo of a SunSpot sensor device connected to a mobile phone and being able to control the phone's menu by moving the SunSpot. The phone network itself is oblivious to this manipulation. More details are available at http://playsim.dev.java.net.

Brad Miller (Associate Professor, Worcester Polytechnic Institute Robotics Research Center) and Derek White (Sun Labs) showed some videos of the work done by High School students. Given a kit of parts, the students have to put together robots to participate in the "US First", an annual robotics competition. A large part of the code has been ported across from C/C++ to Java, and the project is always on the lookout for volunteer programmers. Interested people can go to www.usfirst.org. WPI got a Duke's Choice award for this.

Sven Reimers (System Engineer and Software Architect, ND Satcom) received a Duke's Choice award for the use of Java in analysing the input of satellites.

Christopher Boone (President and CEO, Visuvi, Inc) showed off the Visuvi visual search enine. Upload an image and the software analyses it on a variety of aspects and can provide deep insights. Simple examples are uploading images of paintings and finding who the painter was. More useful and complex uses are in the area of cancer diagnosis. Visuvi improves the quality and reduces the cost of diagnosis of prostate cancer. The concordance rate (probablity of agreement) between pathologists is only about 60%, and the software is expected to achieve much better results. The Visuvi software performs colour analysis, feature detection and derives spatial relationships. There's some relationship to Moscow State University that I didn't quite get. At any rate, Visuvi is busy scanning in 400 images a second (at 3000 megapixels and 10 MB each)!

Sam Birney and Van Mikkel-Henkel spoke about Mifos, a web application for institutions that deal in providing microfinance to poor communities. Microfinance is inspired by the work done by Mohammed Yunus of Grameen Bank. This is an Open Source application meant to reduce the barriers to operation of cash-strapped NGOs. The challenge is to scale. Once again, volunteers are wanted: http://www.mifos.org/developer/ , and not just for development but also for translation into many relatively unknown languages. Mifos won a Duke's Choice Award.

Manuel Tijerino (CEO, Check1TWO) told of how many of his musician friends were struggling to find work at diners. So he created a JavaFX based application that allows artistes to upload their work to the Check1TWO site, and it's automatically available on any Check1TWO "jukebox" at any bar or disco. Regular jukeboxes are normally tied up in studio contracts, so the Check1TWO jukeboxes provide a means for struggling artistes to reach over the heads of the studios and connect directly with their potential audiences.

Zoltan Szabo and Balazs Lajer (students at the University of Pannonia, Hungary) showed off their project that won the first prize at RICOH's student competition. Theirs is a Java application that runs on a printer/scanner and is capable of scoring answer sheets.

Marciel Hernandez (Senior Engineer, Volkswagen Electronics Research Lab and Stanford University) and Greg Follella (Distinguished Engineer, Sun) talked about Project Bixby. This is about automating the testing of high-speed vehicles through "drive-by-wire". The core of the system is Java RTS (Run-time System). The primary focus is improving safety. Stanford University is building the control algorithms. It should be possible to control the car when unexpected things happen, which especially happens on dirt racetracks. There's no need to put a test driver at risk. Project Bixby leads to next-generation systems that are faster, such as more advanced ABS (Anti-lock Braking System) and newer stability control systems).

Finally, there was a video clip of the LincVolt car, which turns a classic Lincoln Continental into a green car like the Prius, but with some differences. The Prius has parallel electrical and petrol engines. The LincVolt has batteries driving the wheels all the time, with the petrol engine only serving to top up the battery pack when it starts to run down. What's the connection with Java? The control systems and visual dashboard are all Java.

This concluded the General Session.

I then attended a session titled "Real-World Processes with WS-BPEL" by Murali Pottlapelli and Ron Ten-Hove. The thrust of the whole session was that WS-BPEL as a standard was incomplete and that real-world applications need more capabilities than WS-BPEL 2.0 delivers. A secondary theme of the session was that an extension called BPEL-SE developed for OpenESB is able to address the weaknesses of WS-BPEL.

[My cynical take on this is that every vendor uses the excuse of an anaemic spec to push their own proprietary extensions. If there is consensus that WS-BPEL 2.0 doesn't cut the mustard, why don't the vendors sit down together and produce (say) a WS-BPEL 3.0 spec that addresses the areas missing in 2.0? I'm not holding my breath.]

The structure was fairly innovative. Ten-Hove would talk about the failings of the standard and Pottlapelli would then describe the solution implemented in BPEL-SE. [The session would have been far better if Pottlapelli had been able to communicate more effectively. I'm forced to the painful realisation that many Indians, while being technically competent, fail to impress because their communication skills are lacking.]

These are the shortcomings of WS-BPEL 2.0 that are addressed by BPEL-SE:

- Correlation is hard to use, with multiple artifacts to define and edit. BPEL-SE provides a correlation wizard that hides much of this complexity.
- Reliability is not defined in the context of long-running business processes susceptible to crashes, maintenance downtime, etc. BPEL-SE defines a state machine where the state of an instance's execution is persisted and can be recovered on restart without duplication.
- High Availability is not defined. BPEL-SE on GlassFish ESB can be configured to take advantage of the GlassFish cluster, where instances migrate to available nodes.
- Scalability is an inherent limitation when dealing with long-running processes that hold state even when idle. BPEL-SE defines "dehydrate" and "hydrate" operations to persist and restore state. As the dehydrate/hydrate operation is expensive, two thresholds are defined (the first to dehydrate variables alone, and the next to dehydrate entire instances).
- Retry is clumsy in WS-BPEL, because the <invoke> operation doesn't support retries. Building retry logic into the code obscures the real business logic. BPEL-SE features a configurable number of retries and a configurable delay between retries. It also supports numerous actions on error.
- Throttling is difficult to achieve in WS-BPEL, which makes the system vulnerable to denial of service attacks and bugs that result in runaway processes. BPEL-SE can limit the number of accesses to a "partnerlink" service node, even across processes. This helps to achieve required SLAs.
- Protocol headers are ignored in WS-BPEL by design, as it is meant to be protocol-agnostic. However, many applications place business information in protocol headers such as HTP and JMS. BPEL-SE provides access to protocol-specific headers as an extension to <from> and <to> elements. Protocols such as HTTP, SOAP, JMS (strictly speaking, this isn't a protocol but an API) and file headers.
- Attachments are non-standard in WS-BPEL, with some partners handling document routing inline and some as an attachment. BPEL-SE adds the extensions "inlined" and "attachment" to the <copy> element.
- BPEL extensions fail because it relies on XPath, and XPath 1.0 has a limitation in that false is interpreted as true (boolVar is a non-empty node). BPEL-SE uses JXPath, and is enhanced to be schema-aware. False is not interpreted as true, and integers do not end in ".0".
- XPath limitations hamper WS-BPEL, because XPath has a limited type system, it is impossible to set a unique id on the payload, it's a challenge to extract expensive items in a purchase order document if it spans multiple activities, etc. Any kind of iteration across an XML structure is difficult, and this is partly due to the limitations of XPath and partly due to those of BPEL. With BPEL-SE, Java calls can be embedded in any XPath expression, with syntax adapted from Xalan. It also supports embedded JavaScript (Rhino with E4X), which makes XML parsing much easier.
- Fault handling is problematic in WS-BPEL because standard faults have no error messages, which makes them hard to debug. Standard and non-standard faults can be hard to distinguish. This complicates fault handlers, requiring multiple catches for standard faults. The WSDL 1.1 fault model has been carried too far. BPEL-SE fault handler extensions propagate errors and faults not defined in WSDL as system faults. Associated errors are wrapped in the fault message. Standard faults are associated with a fault message and failure details like activity name are carried along. BPEL-SE catches all standard and system faults.

I asked the presenters if they were making the case that BPEL-SE had addressed all the limitations of WS-BPEL or if they felt there were some shortcomings that still remained. Ten-Hove's answer was that subprocesses were a lacuna that hadn't yet been addressed.

My next session was "Cleaning with Ajax: Building Great Apps that Users Will Love" by Clint Oram of SugarCRM. I regret to say that this session was short on subject matter and ended in just half an hour. The talk seemed to be more a sales pitch for SugarCRM than a discussion of Ajax principles. Even the few design principles that were discussed were fairly commonsensical and did not add much insight.

For what it is worth, let me relate the points that sort of made sense.

Quite often, designers are asked to "make the UI Ajax". This doesn't say what is really expected. Fewer page refreshes? Cool drag-and-drop? Animation?

There are also some downsides of Ajax:
- Users get lost (browser back and forward buttons and bookmarks are broken, the user doesn't always see what has changed on the page, and screen elements sometimes keep resizing constantly, irritating the user)
- There are connection issues (it can be slow, the data can't be viewed offline, and applications often break in IE6)
- There are developer headaches (mysterious server errors (500), inconsistent error handling, PHP errors break Ajax responses)

Some design questions need to be asked before embarking on an Ajax application:
- Does the user really want to see the info?
- Will loading this info be heavy on the server?
- Does the info change rapidly?
- Is there too much info?

The "lessons learned", according to Oram, are:

- Use a library like YUI, Dojo or Prototype instead of working with raw JavaScript.
- Handle your errors, use progress bars to indicate what is happening, don't let the application seem like it's working when there has been an error.
- Ajax is a hammer, and not every problem is a nail.
- Load what you need, when you need it.

As one can see, there was nothing particularly insightful in this talk, and it was a waste of half-an-hour.

One interesting piece of information from this session was that IBM WebSphere sMash (previously Project Zero) is a PHP runtime that runs on the JVM and provides full access to Java libraries.

SugarCRM 5.5 Beta is out now, and adds a REST API to the existing SOAP service API.

I'm not a fan of SugarCRM. The company's "Open Source" strategy is basically to provide crippleware as an Open Source download, and to sell three commercial versions that do the really useful stuff. I don't know how many people are still fooled by that strategy.

The next session I attended was quite possibly the best one I have attended over these five days. It was called "Resource-Oriented Architecture and REST" by Scott Davis (DavisWorld Consulting, Inc).

Unfortunately, the talk was so engrossing that I forgot to take notes in many places, so I cannot do full justice to it here :-(. Plus, Davis whizzed through examples so fast I didn't have time to transcribe them fully, and lost crucial pieces of code.

Davis is the author of the columns "Mastering Grails" and "Practically Groovy" on IBM Developerworks. He's also a fan of David Weinberger's book "Small Pieces Loosely Joined", which describes the architecture of the web. Although ten years old, the book is still relevant today. [David Weinberger is also the author of The Cluetrain Manifesto.]

I knew Groovy was quite powerful and cool, but in Davis's hands it was pure magic. In a couple of unassuming lines, he was addressing URLs on the web and extracting information that would have taken tens of lines with SOAP-based Web Services. I must have transcribed them incorrectly, because I can't get them to work on my computer. I'll post those examples when I manage to get them working.

A really neat example was when he found a website that provided rhyming words to any word entered on a form. He defined a "metaclass" method on the String class to enable it to provide rhymes by invoking the website. Then a simple statement like "print 'java'.rhyme()" resulted in "lava".

As Davis said, REST is the SQL of the Web.

Davis then talked about syndication with the examples of RSS and Atom. Three things draw one in (chronology (newest first), syndication (the data comes to you) and permalinks (which allow you to "pass it on")). He also mentioned the Rome API and called it the Hibernate of syndication.

What's the difference between RSS and Atom? I hadn't heard it put this way before. Davis called RSS "GETful" and Atom "RESTful".

He then did a live grab of a Twitter feed and identified the person in the room who had sent it.

In sum, this session was more a magic show than a talk. It made me determined to learn Groovy and how to use it with RESTful services.

The last session I attended at JavaOne 2009 was "Building Enterprise Java Technology-based Web Apps with Google Open Source Technology" by Dhanji Prasanna of Google.

Prasanna covered Google Guice, GWT (pronounced "gwit") and Web Driver, and provided a sneak preview of SiteBricks.

I'm somewhat familiar with GWT but none of the others, so my descriptions below may make no sense. It's a verbatim transcription of what Prasanna talked about.

Guice helps with testing and testability. Its advantages are:
- Simple, idiomatic AOP
- Modularity
- Separation of Concerns
- Reduction of state-aware code
- Reduction of boilerplate code

It enables applications to horizontally scale. Important precepts are:

- Type safety, leading to the ability to reason about programs
- Good citizenship (modules behave well)
- Focus on core competency
- Modularity

GWT is a Java-to-JavaScript compiler. It supports a hosted mode and a compiled mode. Core Java libraries are emulated. It's also typesafe.

The iGoogle page is an example of what look like "portlet windows", but which are independent modules that are "good citizens".

Unfortunately, social sites like OpenSocial and Google Wave have different contracts, so modules may not be portable across them.

Google Gin is Guice for GWT. It runs as Guice in hosted mode and compiles directly to JavaScript. There is no additional overhead of reflection.

Types are Java's natural currency. Guice and GWT catch errors early, facilitate broad refactorings, prevent unsafe API usage and reason better about programs. They're essential to projects with upwards of ten developers, because these features are impossible with raw JavaScript.

Web Driver is an alternative to the Selenium acceptance testing tool, and apparently the codebases are now merging. Web Driver has a simpler blocking API that is pure Java. It uses a browser plugin instead of JavaScript, features fast DOM interaction and a flexible API and supports native keyboard and mouse emulation. Web Driver supports clustering.

Then Prasanna provided a preview of SiteBricks, which is a RESTful web framework. The focus is on HTTP and lessons have been learned from JAX-RS. There are statically typed templates with rigorous error checking. It's concise and uses type infererence algorithms. It's also fast and raises compilation errors if anything is wrong.

SiteBricks modules are Guice servlet modules. One can ship any module as a widget library. Any page is injectable. It supports the standard web scopes (request, session) and also a "conversation" scope.

SiteBricks has planned Comet ("reverse Ajax") support. The preview release is available on Google Code Blog.

That concludes my notes on JavaOne 2009. Admittedly, a lot of this has been mindless regurgitation in the interests of reporting. If these make sense to readers (even if they make no sense to me), well and good.

In the days to come, I'll ruminate on what I've learned and post my thoughts.

JavaOne 2009 Day Four (04/06/2009)

2009-06-04T23:47:00.000-07:00

The third day of JavaOne proper (the fourth day if including CommunityOne on Monday) started with a General Session on interoperability hosted by (of all companies) Microsoft. It shouldn't be too surprising, actually, because Sun and Microsoft buried the hatchet about 5 years ago and started to work on interoperability. Time will tell if that was an unholy alliance or not.

Dan'l Lewin (Corporate VP, Strategy and Emerging Business Development, Microsoft) took the stage for some opening remarks. What he said resonated quite well, i.e., that users expect things to just work. Data belongs to users and should move freely across systems. The key themes are interoperability, customer choice and collaboration.

Lewin pointed to TCP/IP as the quintessential standard. Antennae and wall plugs may change from country to country, but TCP/IP is the universal standard for connectivity, which is why the Internet just works. [I could add many other standards to this list, which would also have "just worked" but for Microsoft!]

Lewin added that the significant partners that Microsoft is working with in the Java world are Sun, the Eclipse Foundation and the Apache Software Foundation. The key areas where Sun and Microsoft work together are:

- Identity Management, Web SSO and Security Services
- Centralised Systems Management
- Sun/Microsoft Interoperability Center
- Desktop and System Virtualisation
- Java, Windows, .NET

Identity Management interoperability has progressed a great deal with the near-universal adoption of SAML. On virtualisation, where host and guest systems are involved, Lewin put it very well when he said Sun and Microsoft control each other's environment "in a respectful way."

A website on his slide pack was www.interoperabilitybridges.com .

Steven Martin (Senior Director, Developer Platform Productivity Management, Microsoft) took over from Lewin and started off with "we come in peace and want to talk about interoperability".

He introduced Project Stonehenge, a project under Apache, with code available under the Apache Licence. This uses IBM's stock trading application to demonstrate component-level interoperability between the Microsoft and Sun stacks.

Greg Leake of Microsoft and Harold Carr of Sun then provided a live demo of this interoperability.

The stock trading application has four tiers, - the UI, a business logic tier, a further business logic tier housing the order processing service, and the database tier. The reason for splitting the business logic tier into two was to demonstrate not just UI tier to business logic tier connectivity but also service-to-service interop. The Microsoft stack was built on .NET 3.5, with ASP.NET as the UI technology and WCF the services platform. The Sun stack was based on JSP for the UI and the Metro stack for services, running on Glassfish. Both stacks pointed back to a SQLServer database.

The first phase of the demo showed the .NET stack running alone, with Visual Studio breakpoints to show the progress of a transaction through the various tiers. Then the ASP.NET tier was reconfigured to talk to the Metro business logic layer, and the control remained with the Java stack thereafter. In the third phase of the demo, the Metro business service layer called the order processing service in the .NET stack. The application worked identically in all three cases, effectively demonstrating bidirectional service interoperability between .NET and Metro Web Services.

Martin also mentioned a useful principle for interoperability, "Assume that all applications are blended, and that all client requests are non-native". This is analogous, I guess, to that other principle, "Be conservative in what you send, and liberal in what you accept".

He also referred to "the power of choice with the freedom to change your mind", which I thought was a neat summarisation of user benefits.

Aisling MacRunnels (Senior VP, Software Marketing, Sun) joined Steven Martin on stage to talk about the Sun-Microsoft collaboration, which isn't just limited to getting the JRE to run on Windows. Microsoft also cooperates with Sun to get other "Sun products" like MySQL, VirtualBox and OpenOffice to work on the Microsoft platform. The last item must be particularly galling to the monopoly. Microsoft is also working to get Sharepoint authentication happening against OpenSSO using SAML2. Likewise, WebDAV is being supported in Sun's Storage Cloud. In other words, when both parties support open standards, their interoperability improves.

I think it speaks more of the quality and tightness of a standard than of vendor cooperation when systems work together. Sun and Microsoft shouldn't need to talk to each other or have a cozy relationship. Their systems need to just work together in the first place.

The next session I attended was "Metro Web Services Security Usage Scenarios" by Harold Carr and Jiandong Guo of Sun. Carr is Metro Architect and Guo is Metro Security Architect, so we pretty much had the very brains of the project talking to us.

There wasn't very much in the lecture that was specific to Metro. Most of the security usage patterns were general PKI knowledge, but I must say the diagrams that illustrated the logic flow in each pattern were top class. I have seen many documents on PKI, but these are some of the best. My only quibble with them is they tend to use the same terms "encrypt/decrypt" in two separate contexts - "encrypt/decrypt" and "sign/verify".

Some of the interesting points they made were:

- The list of security profiles bundled with Metro will be refactored soon. Some will be dropped and new ones will be added.
- SSL performance is still better than WS-Security, even with optimisations such as derived keys and WS-SecureConversation. [WS-Security uses a fresh ephemeral key for every message, while WS-SecureConversation caches and reuses the same ephemeral key for the whole session.]

- Metro 2.0 is due in September 2009.

I then attended a session called "Pragmatic Identity 2.0: Simple, Open Identity Services using REST" by Pat Patterson and Ron Ten-Hove of Sun. Ten-Hove is also known for his work on Integration and JBI. He was the spec lead for JSR 208.

[As part of their demo, I realised that NetBeans has at least a couple of useful features I didn't know about earlier. There's an option to "create entity classes from tables" (using JPA, I presume), and another one to "create RESTful web services from entity classes".]

It's a bit difficult to describe the demo that Patterson gave. On the surface, it was a standard application that implemented user-based access control. One user saw more functions than another. The trick was in making it RESTful. Nothing had to be explicitly coded for, which was the cool part. The accesses were intercepted and authenticated/authorised without the business application being aware of it. As I said, it's hard to describe without the accompanying code.

The next session was on "The Web on OSGi: Here's How" by Don Brown of Atlassian. Brown is a very polished and articulate speaker and he kept his audience chuckling.

OSGi is something that has fascinated me for a while but I haven't got my head around it completely yet. At a high level, OSGi is a framework to allow applications to be composed dynamically from components and services that may appear and disappear at run-time. Dependencies are reduced or eliminated by having each component "bundle" use its own classloader, so version incompatibilities can be avoided. Different bundles within the same application can use different versions of libraries without conflicts, because they don't share classloaders.

OSGi is cool but complex. As Brown repeatedly pointed out, while it can solve many integration and dependency problems, it is not trivial to learn. Those who want to use OSGi must be prepared to learn many fundamental concepts, especially around the classloader. Also, components may appear and disappear at will in a dynamic manner. How must the application behave when a dependency is suddenly exposed?

There are 3 basic architectures that one can follow with OSGi:

1. OSGi Server
Examples: Spring DM Server, Apache Felix, Equinox
Advantages: Fewer deployment hassles, consistent OSGi environment
Disadvantages: Can't use any existing JEE server

2.Embedded OSGi server via bridge (OSGi container runs within a JEE container using a servlet bridge)
Examples: Equinox servlet bridge
Advantages: Can use JEE server, application is still OSGi
Disadvantages: (I didn't have time to note this down)

3. Embedded OSGi via plugin
Example: Atlassian plugin
Advantages: Can use JEE server, easier migration, fewer library hassles
Disadvantages: More complicated, susceptible to deployment

I learnt a number of new terms and names of software products in this session.
- Spring DM (Dynamic Modules)
- Peaberry using Guice
- Declarative Services (part of the OSGi specification)
- iPOJO
- BND and Spring Bundlor are tools used to create bundles
- Felix OSGi is embedded as part of Atlassian's existing plugin framework.
- Spring DM is used to manage services.
- Automatic bundle transformation is a cool feature that Brown mentioned but did not describe.

There are three types of plugins:
Simple - no OSGi
Moderate - OSGi via a plugin descriptor
Complex - OSGi via Spring XML directly

Brown gave us a demo using JForum and showed that even if a legacy application isn't sophisticated enough to know about new features, modules with such features can be incorporated into it.

I had been under the impression that OSGi was only used by rich client applications on the desktop. This session showed me that it's perhaps even more useful for web applications on the server side.

My last session of the day was a hands-on lab (Bring Your Own Laptop) called "Java Technology Strikes Back on the Client: Easier Development and Deployment", conducted by Joey Shen and a number of others who cruised around and lent a helping hand whenever one got stuck. It looks like Linux support for JavaFX has just landed (Nigel Eke, if you're reading this, you were right after all), and a very quiet landing it has been, too. But it's still only for NetBeans 6.5.1, not NetBeans 6.7 beta. At any rate, I was more interested in just checking to see if it worked. It turned out that there were a couple of syntax errors in the sample app which had to be corrected before the application could run. I was very keen to try the drag-and-drop feature with which one could pull an applet out of the browser window and install it as a desktop icon (Java Web Start application). Unfortunately, this feature requires a workaround in all X11 systems (Unix systems using the X Window System), because the window manager intercepts drag commands and applies it to the window as a whole. There was a workaround described for applets but none for a JavaFX application. As time was up, I had to leave without being able to see drag-and-drop in action. Never mind, I'm sure samples and documentation will only become more widely available as time goes on, and Sun will undoubtedly make JavaFX even easier to use in future.

JavaOne 2009 Day Three (03/06/2009)

2009-06-03T23:17:00.000-07:00

The second day of JavaOne proper (the third day if including CommunityOne) started with a General Session on mobility conducted by Sony Ericsson.

The main host was Christopher David (Head of Developer and Partner Engagement). At about the same time he started his talk, Erik Hellman (Senior Java Developer) got started on a challenge - to develop a mobile application by the end of the session that would display all Tweets originating within a certain radius of the Moscone Center that contained the word 'Java'.

Rikko Sakaguchi (Corporate VP, Head of Creation and Development) and Patrik Olsson (VP, Head of Software Creation and Development) joined David on stage, and between the three of them, kept the Sony Ericsson story going.

One of the demos they attempted failed (controlling a Playstation 3 with a mobile phone), but then it isn't a demo if it doesn't fail.

One of the points made was about the difference between a mobile application and a traditional web application. A traditional web application has its UI on the client device, with business logic and data on a server across the network. A mobile application has the UI, parts of business logic and parts of data and platform access on the device, and the remaining data and business logic across the network. I don't quite buy this distinction. I don't necessarily see a difference between traditional distributed applications and mobile applications. So the device form factor is a bit different and the network is wireless, but that's hardly a paradigm shift. Application architectures like SOFEA are meant to unify all such environments.

The history of Sony Ericsson's technology journey is somewhat familiar. In 2005, they switched from C/C++ to Java. Java became an integral part of the Sony Ericsson platform rather than an add-on. In 2007, they created a unique API on top of the base Java platform. In 2009, the focus is on reducing fragmentation of platforms. The bulk of the APIs are standard, while a few (especially at the top of the stack) are proprietary to SE.

As expected from a company that boasts 200 million customers worldwide and 200 million downloads a year, SE has a marketplace called PlayNow Arena. SE has been selling ringtones, games, music, wallpapers, themes, movies and lately, applications. I'm frankly surprised that it's taken them so long to get to selling applications.

Since time-to-market is important, SE promises software developers a turnaround time of 30 days from submission to appearance in the virtual marketplace, with assistance provided throughout the process.

And yes, Erik Hellman had completed his application with 10 minutes to spare by the time the session ended.

The next session I attended was something completely new to me. This was called "Taking a SIP of Java Technology: Building voice mashups with SIP servlets" by RJ Auburn of Voxeo Corp. The Session Initiation Protocol (SIP) is mainly used in telephony, but can apparently be used to bootstrap any other interaction between systems. SIP has more handshaking than HTTP, with many more exception cases, so it's a more chatty protocol than HTTP. It's also implemented on top of UDP rather than TCP, so SIP itself needs to do much more exception handling than HTTP.

RFC 3261 that describes SIP is reportedly a rather dry document to read. Auburn recommended The Hitchhiker's Guide to SIP, and also some Open Source info at www.voip-info.org and some industry sites (www.sipforum.org and www.sipfoundry.org).

There seem to be two main ways to develop applications with SIP. One uses XML, the other uses programming APIs. The XML approach is a 90% solution, while the API approach provides more options but is more complex.

There are two sister specifications in the XML approach - VoiceXML and CCXML (Call Control XML). VoiceXML supports speech and touchtone, a form-filling model called the FIA, but very limited call control. CCXML, in contrast, manages things like call switching, teleconferencing, etc. The two work in a complementary fashion, with CCXML defining the overall "flow logic", and VoiceXML defining the parameters of a particular "message" (for want of better terms).

The Java API is based on the SIP Servlet API (www.sipservlet.com). JSR 116 was SIP Servlet 1.0, and JSR 289 is SIP Servlet 1.1 (just released). JSR 309 (the Java Media Server API) is based on the CCXML model, but is still in draft.

SIP is complex, so Voxeo has a simpler API called Tropo, available in a number of scripting languages. This is not Open Source, but is free for developers, and the hosting costs about 3 cents a minute. There are also traditional software licensing models available.

There are some phone-enabled games available, and Vooices is a good example.

More information is available on tropo.com and www.voxeo.com/free.

The next session I attended was "What's New in Groovy 1.6?" by Guillaume Laforge himself. The talk was based on an InfoQ article written by him earlier.

In brief, the improvements in Groovy 1.6 are:

- Performance - the compiler is 3x to 5x faster than 1.5, and the runtime is between 150% to 460% faster.
- Syntax changes - multiple assignment (the ability to set more than one variable in a statement), optional return statements
- Ability to use Java 5 annotations
- Ability to specify dependencies
- Swing support (griffon.codehaus.org)
- JSR 223 (scripting engine) support is built-in and can invoke scripting in any language
- OSGi readiness

My next session was on providing REST security. The topic was called "Designing and Building Security into REST Applications" by Paul Bryan, Sean Brydon and Aravindan Ranganathan of Sun. The bulk of the talk focused on OpenSSO and its REST-like interface. But as the presenters confessed, the OpenSSO "REST" API is just a facade over a SOAP API. In OpenSSO 2.0, they visualise a truer RESTian API.

The other term I had never heard of before was the OAuth protocol. Apparently, OAuth is a style of authentication just like HTTP's Basic Authentication and Digest Authentication.

The last session that I attended today was on "Coding REST and SOAP together" by Martin Grebac and Jakub Podlesak of Sun. Although the topic was entirely serious, it felt like cheating at a certain level.

The premise is that we implement SOAP and REST on top of POJOs using annotations defined by JAX-WS and JAX-RS, respectively. So can't we just add both sets of annotations to the same POJO and enable SOAP and REST simultaneously?

I can see one very obvious problem with this approach. REST forces one to think Service-Oriented because the verbs refer to what the service consumer does to resources. It's what I've earlier called the Viewpoint Flip, and I believe it's an essential part of Service-Oriented thinking. But SOAP doesn't enforce any such viewpoint. It's possible to have an RMI flavour to the JAX-WS SOAP methods. So there's no substitute for proper design.