Steve Jobs and Dennis Ritchie

This hasn't been a great week to be a computer pioneer. Dennis Ritchie has now followed Steve Jobs off the stage and into that great big computer industry in the sky. Two giants in a single week.

Steve Jobs

Dennis Ritchie

I guess, unlike most people, I have been touched by Ritchie much more than by Jobs. I have never owned a Mac or an iPhone, and regretted buying an iPod as soon as I realised there was no way to bypass the iTunes straitjacket. iPod generation 4 worked with the gtkPod application on my Ubuntu desktop, but generation 5 corrected that shocking oversight, and the Apple empire, with a sigh of relief, regained its pristine purity, shutting out the great unwashed once more. That ended my dalliance with Jobs and his closed system. I don't fancy handcuffs even when they're haute mode.

I feel sorry about the death of Steve Jobs the human being. I have no sympathy for the worldview that he represented, of closed systems, slimy lawyers and patent lawsuits.

Dennis Ritchie, of course, was the polar opposite of Jobs. Those who have read Asimov's science fiction trilogy Foundation may remember that there were two Foundations, a well-known one at the periphery of the Galactic empire, and the other, a secret one, located "at the opposite end of the galaxy". Many characters in the novel tried searching for the Second Foundation along the opposite edge of the galaxy where they thought it would be, but its actual location was right at the centre of the empire! The term "opposite" was meant in a sociological sense, not a physical one.

And in true Foundation-esque fashion, Ritchie's contribution to mankind, while in a sense the opposite of Jobs's, was not a rival closed system but an open one. Along with Ken Thompson, he wrote the most open operating system of its time, -- Unix.

Ken Thompson

The popular web article "The Last Dinosaur and The Tarpits of Doom" has a matchless passage describing the world at the time of Unix's birth.

In 1970, primitive proprietary operating systems bestrode the landscape like mighty dinosaurs: Prime's PrimeOS, DEC's RSTS, RT-11, etc. (with VAX/VMS soon to come), IBM's innumerable offerings, CDC's Scope and of course dominating the scientific workstation market, Apollo's Domain.

Who would then have dared to predict the fall of such giants?

What force could topple such entrenched operating systems, backed by massive industry investment, hacker culture and customer loyalty?

Today, of course, we all know the answer:

In 1975 Bell Labs released Unix.

• Unix had no support from its creator, AT&T: Buy the magtape and don't call us. (AT&T was legally barred from entering the operating system market.)
  
• Unix had no support from any existing vendor: None had the slightest interest in backing, supporting or developing an alternative to its proprietary operating systems offerings.

• Unix had zero customer base: Nobody had ever heard of it, nobody was requesting it.
  
• Unix had zero marketing: Nobody had any reason to spend money building mindshare for it.

A one-sided competition?

Decidedly: Unix wiped all workstation competition off the map in less than fifteen years.

On April 12, 1989, HP bought up Apollo at a fire-sale price, putting out of its misery the last remaining proprietary operating system vendor in the workstation world, and the workstation proprietary OS era was over: Unix was left alone in the workstation market.

In fifteen years, a [magnetic] tape and an idea had effectively destroyed all opposition: Every workstation vendor was either supporting Unix or out of business.

Let me add one more point to that. At the heart of Apple's operating systems is a version of Unix (BSD Unix). Steve Jobs's business empire took a freely available operating system, layered a user-friendly graphical interface over it, and without a word of thanks, proceeded to build a proprietary edifice that was as closed as its enabling technology was open.

So thank you, Dennis Ritchie, for giving us today's Mac.

Ritchie was an inventor second to none. People today forget one of the main reasons Unix is considered "open". Before Unix, an operating system was written for a specific processor chip, in the assembly language corresponding to that chip. One of the key factors that made Unix open was the fact that it could be ported to any chip at all. More than 90% of an operating system's logic is in fact independent of the underlying hardware architecture. Less than 10% is specific to the chip. That's why only very low-level code in Unix is written in assembly language. That's the only part that needs to be re-written when porting Unix to a different processor architecture.

Once the operating system was liberated from its ties to hardware, any hardware manufacturer could port Unix to their computers. That's the openness that destroyed the proprietary dinosaurs and created the world we see today. We have Thompson's and Ritchie's genius to thank for that. In the next generation, Linux proceeded to wipe out proprietary Unix variants to take over the server room.

Today, Google's Android has Linux at its core. So now Ritchie's invention has taken over the server, a significant part of the desktop (through the Mac) and an increasingly dominant part of the smartphone and tablet markets (through Android and Apple's iOS). Not bad for a simple and open operating system!

Now we know that 90% of Unix is written in a higher-level language, and therein hangs another tale. At the time Thompson and Ritchie wrote Unix, there was no suitable high-level language to write an operating system in. It had to have the higher-level constructs of most modern, structured, procedural programming languages. Yet it also had to provide sufficient control over low-level constructs like memory addresses and file structures. This was a challenge that may have stumped other people and caused them to compromise in some way. Not Ritchie. Necessity for him was the mother of invention - the invention of the C programming language. Together with Brian Kernighan, Dennis Ritchie created the first C compiler, and it is astonishing that the language has hardly had to change since their version to the present day. The standardised version of their language, ANSI C, is largely the same as their original one, with just minor changes. Now that's vision for you.

Brian Kernighan

C inspired C++, Java, JavaScript, Perl, C# and a whole bunch of other languages. Any language with curly braces and semicolons owes an intellectual debt to Ritchie and Kernighan.

The laptop that I'm composing this on runs Ubuntu Linux, another variant of Unix. Most of Linux is written in C. I'm probably not fully aware of the extent to which I owe Ritchie a debt of gratitude, as the one common factor in the creation of both Unix and C.

By the way, if you think Unix has an ugly user interface because of its command line, there are two rebuttals to that argument. The trivial one is that modern Unix variants like Linux have very sophisticated and friendly user interfaces indeed. The deeper rebuttal is that there is beauty and power in the Unix command line that MacOS has eagerly embraced as an offering to the "power user".

User Interface experts Don Gentner and Jakob Nielsen write in their classic paper The Anti-Mac Interface:

The see-and-point principle states that users interact with the computer by pointing at the objects they can see on the screen. It's as if we have thrown away a million years of evolution, lost our facility with expressive language, and been reduced to pointing at objects in the immediate environment. Mouse buttons and modifier keys give us a vocabulary equivalent to a few different grunts. We have lost all the power of language, and can no longer talk about objects that are not immediately visible (all files more than one week old), objects that don't exist yet (future messages from my boss), or unknown objects (any guides to restaurants in Boston).

Like they said. As an advocate of the power of the Unix command line, I rest my case.

Unix is such a unique phenomenon in the world of computing that noted academic Prof Martin Vermeer believes it should be treated as a basic element of literacy, alongside the three Rs.

And so a tumultuous week has gone by, and the computer industry mourns its two luminaries. Among computer pioneers, Steve Jobs was the shiny user interface, slick and popular. Ritchie was the kernel, unseen and unknown to the masses, yet the workhorse that made everything else possible, including the user interface. He may be less widely mourned, but no less mourned.

And I like to think Ritchie rushed after Jobs to make sure the Pearly Gates stayed open to all!

Google's Aimless Darting

Search Engine vendors should be in the clarification business, not the muddying business. All the more frustrating to read the news about Dart, Google's new web programming language to replace JavaScript.

My only reaction is - WHY???

Yes, JavaScript is not exactly a perfect language. It has warts, huge ones. That's been known for a long time. Is that sufficient reason to throw the whole language overboard and try and popularise a new one? Especially after CoffeeScript has already done the job? JavaScript has good parts as well as bad, and I'm not talking about just its technical aspects. Its ubiquity is a major strength. Replacing it is an exercise with very doubtful prospects. I don't think even Google can pull it off.

CoffeeScript has taken the right approach, in my opinion. CoffeeScript is JavaScript with lead shielding over the reactor core. And that was all that was required. CoffeeScript on the server side uses Nodejs to run scripts anyway, so one can write server-side code in CoffeeScript and run it with "coffee" instead of with "node", and the ugliness and danger of JavaScript can be neatly sidestepped. Even on the client side, a single line such as

<script type="text/JavaScript" src="coffeescript.js"></script>

will allow you to write the rest of your client-side code in CoffeeScript, because coffeescript.js is a minimised library that will let your browser interpret CoffeeScript natively. Your application code will look like this:

<script type="text/CoffeeScript">
# CoffeeScript code
</script>


What is Dart going to do for us beyond that, functionality-wise? Technically, Dart won't even replace JavaScript, because it will compile to JavaScript. Does that sound familiar? Because that's just what CoffeeScript does as well! Was it so hard for Google to get behind CoffeeScript? Some NIH at work, I think.

Now the frustrating thing is that Dart will waste at least some developer mindshare and bandwidth when the world should have been just getting on with the job - using CoffeeScript plus jQuery on the client side, CoffeeScript plus Nodejs on the server side.

What a waste!

Life above the Service Tier (Change of Links)

Google Groups no longer allows file uploads. Worse, they went back on a commitment to keep the files that were already uploaded accessible. Now all the files I uploaded to the "wisdomofganesh" group have to find alternate homes (if I still have a local copy, that is), and I have to update all the links from my blog. Totally uncool of Google. I'm very annoyed.

Update 15/11/2012: I've changed again from mesfichiers.org to slideshare, since that seems to be the new place to upload shareable stuff.

Anyway, since the following docs seem to be the most frequently accessed, here are the new links:

1. Life above the Service Tier (white paper)

2. Life above the Service Tier (presentation)

3. SOFEA in a SOA Ecosystem (diagram)

Amazon Cloud Drive Tech Talk - Sydney, 10-10-2011

I attended the Amazon Cloud Drive Tech Talk today in Sydney. This was held on the 39th floor of the Citigroup building at 2 Park Street in the CBD. The views from the window are awesome, by the way.

That's Park Street stretching away towards King's Cross

Those red slugs are the new Metro double-buses

I reached the venue just after 1800, when registration was scheduled to begin. There were eats, bottles of beer and canned carbonated drinks in the room, but none suitable for a vegetarian teetotaller trying to lose a few calories, so I drank a glass of water instead and waited for the proceedings to begin.

The event kicked off promptly at 1830. John Scott of the Android Australia User Group made a short speech introducing Piragash Velummylum of Amazon, who had flown in from the US.


Then Piragash took the stage and spoke. I must say the Amazon Cloud Drive turned out to be more Amazon Recruitment Drive than anything else! Piragash and a colleague Brad (who spoke for half a minute) made no secret of the fact that they were in town to recruit developers for their Seattle office. The tech talk was like a campus recruitment talk - just enough data to pique the interest of developers wanting to work on cool technology.


The unofficial motto of the Amazon Cloud Drive service is "anything digital, securely stored, accessible anywhere". Piragash spoke a bit about Amazon CEO Jeff Bezos, who he projected to be a likeable nerd who listens to everybody and does the right thing. [Piragash must have a weird sense of humour. Just read this piece and the comments that follow. I wouldn't want to work for Jeff Bezos in a hundred years.]

Amazon Cloud Drive is meant to address three customer pain points:

• Multiple music downloads
• Moving files from one store to another
• Data loss

He said something about Amazon MP3, which seems to be similar to Apple's iTunes (Piragash sidestepped a question from the audience on a comparison with iTunes, saying they had a policy not to talk about competitors, but mentioned that the Amazon version allowed upload of customer content). I think the story might have begun with Amazon MP3 which catered to music files. Then Amazon Cloud Drive came along which was more general-purpose and catered to videos and other kinds of files as well. That's what I gathered.

He emphasised a few times that Amazon Cloud Drive is for customer content, not purchased content (also called studio content or catalog content). It's like DropBox, I guess. It's said to be free, but there was no mention about storage limits. Unlike DropBox, they don't do de-duplication of files (yet), and certainly not across users. They would need to sort out licensing before they did that sort of thing, and he made a statement to the effect that Amazon is DMCA-compliant.

They obviously leverage off their other technologies, i.e., AWS (Amazon Web Services). They currently use S3 (Scalable Storage Solution), not EC2 (Elastic Cloud Computing), but they may use that in future if required.

Amazon Cloud Drive has three layers. Storage is provided by S3. They've defined a hierarchical file system on top of that (which has long been a demand of S3 users). Finally, they have metadata on top that defines relationships and queries, making the file system much more useful.

These are some of the operations supported at each of those levels:

S3: Upload, download

Filesystem: Copy, move, delete, list, recycle, restore, create

Metadata: Select, get, put, delete (Aha, REST! But Select seems to have replaced Post. Puzzling.)

Piragash also talked about some of their technical challenges.

Scaling was the major one (and continues to be a major focus of research and innovation). The principles Amazon follows to ensure scalability are:

1. "Don't get in the way", i.e., let AWS and Akamai do the job they know best, don't interpose Amazon Cloud Drive between those systems and the user. Rather, allow Amazon Cloud Drive to be proxied by CDNs (Content Delivery Networks) like Akamai.

2. "Be flexible", i.e., be forward-thinking and don't prevent services like allowing a zipped set of files to be downloaded.

Security is another major concern. Content is meant to be private and personal. As part of their PaaS offering, Amazon provides an Identity and Access Management (IAM) system, and Amazon Cloud Drive uses IAM and AWS to control the generation of time-bound tokens. Delegated S3 access has an extra security token in the URL that expires after a certain period, so people can't pass content URLs around and have them accessible indefinitely to anyone who gets the URL. Can customers share their content with others, or is this purely private? Piragash's response: "Stay tuned".

Then Piragash briefly talked about the Kindle Fire, due for release in about a month. Incidentally, Kindle for iPad is said to use HTML5 throughout and to look like a native app.

There's also a new business called Amazon Fresh, currently only rolled out to the Seattle area. Order your groceries online at midnight, and have them delivered to your door by 6 am!

There was a little talk about version synchronisation strategies and rules for conflict resolution. They support multiple schemes for different domains. Sometimes they use automated algorithms, and at other times they let the customer resolve conflicts.

Asked about an API for Cloud Drive, Piragash would only say, "Stay tuned".

He mentioned that it was important to weed out "phantom requirements" and to concentrate on solving the customer's real problems.

A member of the audience remarked that Australia really needed an edge server located here to reduce latency. Piragash merely smiled acknowledgement. Someone else asked about the number of servers used by Amazon, but Piragash could not talk about that.

That was the end of the talk, and Piragash invited people to stay and talk to him and his colleagues. He called for CVs and said they'd be in Sydney the whole week, recruiting people to be based out of Seattle.

So if you like rain throughout the year and don't mind sharing a city with Microsoft, do apply to Amazon for a job.

[What was even more interesting than the technology talk was a nugget of cultural trivia that I've written about on my other blog.]

Aakash - The Sky's the Limit

Android tablet prices went down even further with the announcement of the Indian-made tablet "Aakash" (Sanskrit for "sky", pronounced "aakaash"). The price to students (admittedly with a government subsidy) is projected to be $35. Even without the subsidy, the retail price should still be a groundbreaking $60. The aim is to bridge the "digital divide" and allow less affluent sections of society to participate in the digital economy.

I like free markets, but given the tendency for cartels to form in most market segments (negating the freedom that true liquidity would deliver) and the distortions of patent law that entrench the power of large corporations, I also think governments need to step in from time to time to ensure equity. Untrammelled capitalism isn't going to bridge the digital divide. There has to be a deus ex machina that kicks in at critical junctures.

Stepping back a bit to take a historical view, the primary difference between the 19th and 20th centuries was not in scientific knowledge or even technological invention (because many 20th century inventions were known even in the 19th), but in the mass production and mass consumption of such technology. In analogous fashion, I can see the potential for India to become a technology "enabler" for the world's poorer half (or two-thirds), democratising technology usage across a geographical span rather than a historical one.

I remember the dire warnings a decade ago about an AIDS epidemic that was poised to devastate Africa, and the utterly shameful behaviour of the big Western pharma companies in refusing to lower the prices of AIDS medication to save millions of lives. They tried to use patent law to block any attempt by other parties to provide cheaper medication. If the intellectual wherewithal had been entirely lacking in the Third World, they may well have got away with it. But Indian pharma companies were able to produce AIDS medication, and more importantly, produce it at a much lower price point that African countries could afford, and the Third World as a whole managed to vote their way around the IP laws that the US (and other Western countries) have forced all others to sign. A humanitarian disaster was averted that would have dwarfed the Holocaust, and India had a large (though also largely unsung) role to play in averting it.

More recently, Tata Motors established a radically lower price point for cars ($3000). The Nano is a revolutionary breakthrough, and it isn't a toy either. If a car can survive on Indian roads, it will positively thrive anywhere else ;-). The Nano is going through a few teething problems right now, but I have a sense that in a decade or two, it will be one of the world's iconic car brands, perhaps the most ubiquitous. The Nano could change the image of car ownership as an indication of wealth.

And now, with Aakash, India is once again bringing technology within the reach of ordinary people, at Third World prices. The term "reasonably priced" means something very different in Western countries. Even middle class people in Third World countries cannot afford these "reasonably priced" products. In Marxist terminology, as long as the "means of production" were concentrated in Western hands, there wasn't much the rest of the world could do about it. They either paid those prices (exorbitant by their standards) or simply did without. Now they have a choice. Incongruous as it may seem, India is swooping in in shining armour to save the day.

I do have some reservations, though. Indian ingenuity has never been in doubt. What's in doubt is India's institutional ability to follow through, to execute, to deliver. India has always been a muddle-through country rather than a reliably-achieve country. Even the Nano is a case in point. The political shenanigans that preceded its launch very nearly canned the project. If I was someone big in the Indian government, I would not look at this as just a private sector enterprise that is none of the government's business. I would see it as a national enterprise where India has a chance to put its stamp on the world and change it for the better, and I would provide Tata Motors with all the support needed to manufacture and sell the Nano in volume, worldwide.

[I'm not going to argue about whether an invention that serves to consume more fossil fuels is going to change the world for the better. The environmentalists don't seem to have an easy answer to the developmental issue of stagnation-versus-pollution either.]

Again on the topic of the Indian character, I remember my student days at IIT Madras when a new Siemens mainframe computer was delivered to the institute. This was in the mid-eighties. The box was too big to go up the stairs of the computer centre. Many students and professors watched as the workmen, mostly uneducated, rigged up a makeshift pulley and with the help of ropes, winched the box up the side of the building, and others leaned over the parapet wall on the first floor and hauled it in. Mission accomplished with a minimum of technology (and no insurance!)

Watching this, one of my professors who had done his PhD in the US and worked there for a while, remarked wisely, "We Indians are great at improvisation. The danger is that we will be satisfied with our ability to improvise, and fail to develop real systems."

I see no evidence that India has improved significantly on the systems front. Flashes of Indian brilliance, like the Nano and the Aakash, will remain just flashes in the pan unless the country learns to be more disciplined about delivery. Virtually every developed country emphasises delivery discipline. Only when India masters that can the world look forward to a steady stream of dirt-cheap high technology that will change billions, not just millions, of lives.

Meeting of Android Australia User Group - Sydney 05/10/2011

I attended my first meeting of the Android Australia User Group - Sydney this evening.

The organiser, John Scott, started with some general announcements, which should be of interest to many.

- There is an Amazon Cloud Drive Developer Tech Talk in Sydney on Monday October 10, 2011.

- Google Developer Day will be on November 8, 2011 in Sydney.

Those interested should register for these events. They're both free, as far as I know.

John then kicked off the first talk of the evening, which was a brief overview of Motorola's IDE for Android application development - MOTODEV Studio. According to John, this IDE is better than the Google Android plugin for Eclipse, which seems to be the de facto IDE for Android, because it has a few extra features like the generation of boilerplate code to make the developer's life easier, and also graphical tools to manage SQLite databases. I found a couple of other discussions on MOTODEV Studio vs Eclipse here and here, but they seem somewhat dated.

The second talk was by Gianpaolo De Biase, a developer with AppCast. Gianpaolo talked about some real-life applications that his company has built (one of which is JustStartWalking, an app for an initiative of the same name from the Chiropractors' Association of Australia). He also discussed two important architectural decisions that his company made when developing these products.

1. When dealing with local databases, Gianpaolo recommends using an Object-Relational Mapping (ORM) tool rather than the raw SQLite API provided by Google. The tool he used is OrmLite, which is Open Source like Hibernate but more lightweight and adequate for the simpler data structures needed for local storage on mobile devices.

2. When making REST calls to remote servers, he recommends using the Spring Android module with its REST Template in preference to raw HttpClient. He did encounter some bugs with Spring Android in the areas of cookie management and SSL certificates, but believes the product is rapidly maturing.

As a side benefit of both OrmLite and Spring Android being annotation-based, Gianpaolo was able to have a single set of domain objects. Each object had two sets of annotations applied, one for persistence and one for interfacing with REST services. [I'm a bit suspicious of the latter annotation, since it looks like the design combines domain objects and message documents into a common entity, a form of tight coupling I've long warned against.]

After a short break, we had our third talk of the evening by James Zaki, a freelance developer with goCatch. This was a high-level description of the goCatch app, which brings together cab drivers and prospective passengers. There was general satisfaction expressed around the room in favour of this cartel-breaking app, since the taxi companies and CabCharge engage in significant rent-seeking behaviour at the expense of both drivers and passengers. goCatch allows the two to bypass the middlemen. I had some reservations about one of the aspects of the goCatch design as described by James, i.e., its statefulness, which led to problems of synchronisation of state held on devices with that on the server. Perhaps a suitable set of messages based on idempotence could solve the problem. I didn't have time to discuss this offline with James.

Our fourth and final talk of the evening was by Darren Younger, CTO of IPScape, in whose offices the meeting was held. IPScape is a provider of cloud-based contact centre solutions catering to both voice and web. One of their interesting applications allows mobile device users to make phone calls not through the device's native telephony capabilities, but through the IPScape app. The server then initiates regular (teleconference-style) phone contact with both the caller and the receiver. The advantage of this is that the server can record the call. Many financial service providers are required by law to record all customer conversations, and it is easier for them to use this app rather than approach the telcos for a voice recording service. A developer API may be coming in a few months.

I also met another attendee, Nanik Tolaram, an amateur Android enthusiast with his own Android-related website.

I picked up a few useful tidbits of information over the course of the evening.

Samsung sent a couple of people to the meeting. They seem keen to understand the size and strength of the Android developer community. Samsung wants to carve out a unique niche even within the Android ecosystem and they have their own app store separate from the generic Android one.

DroidDraw is a testing tool for Android User Interfaces.

Google has a cloud-to-device API.

developer.android.com is Google's portal for Android developers.

Balsamiq is a commercial tool to mock up UIs, including mobile UIs. Pencil seems to be a good Open Source equivalent, and Lumzy is a free one.

Two of the presenters talked about their negative experiences with outsourcing. Although the outsourcing countries were as varied as Israel, India and Singapore, there seem to be some common problems caused by distance as well as the seeming cultural inability of some developers to look beyond the literal specification and to understand the higher abstraction that an application is trying to implement. Errors in the documentation of some specs led to literal implementation of those features even though they patently made no sense. Outsourcing sites like Freelancer.com seem very cost-effective, but the elapsed time to obtain a working solution negates those benefits. Examples: $200 and 4.5 months to develop an app, $800 and 9 months to develop another. The moral of the story seems to be to hire good local developers so communication problems are reduced and results are achieved quickly.

The Android Australia User Group is a good place for developers to hang out. The organisations that some of the speakers represented are looking for developers, and this may be a good way to get introduced.

Daylight Saving Time - Ubuntu Passes the Test

60 seconds after 0159 on October 2nd is when New South Wales, Australia mysteriously loses an hour to Daylight Saving Time and the time becomes 0300.

I was awake to record what our computers did. My wife's Windows 7 machine showed 0300 all right. I would have been surprised if it hadn't. Microsoft needs to show some degree of competence for all that market share!

What was personally gratifying to me was that my Ubuntu Linux machines also neatly skipped the hour and showed 0300.

I'm going to bed now with a smile.

Facebook's Secret Sauce

I would say there have been Three Ages of the Internet.

The first was the Internet era proper, where only the US government and universities (and maybe a few other organisations) had it. There was email of course, and really cool stuff like Archie and Gopher (whatever they were) that you could access on text-based terminals.

Then there was the Web, which was nice and graphical and coincidentally came out at roughly the same time as the extremely popular Windows 95. The Web through Windows made the Internet something for regular folks, not just geeks. There was surfing and search, pleasurable as well as useful. The Web soon gobbled up email too (webmail changed the user-facing protocol from SMTP and POP to HTTP). And blogs became a way for the little guy to communicate his own views to the world instead of just consuming the output of established media. That was the first step towards mass participation.

Then there came Facebook. More than Picasa and on par with Skype, Facebook has suddenly made the Internet a must-have for everyone. I'm betting Facebook and Skype have driven Internet usage to new highs, both in terms of bandwidth consumed and in terms of market penetration. And along the way, Facebook has sucked the oxygen out of blogs. Ask me. I should know.

That deserves the title of 'third generation Internet'.

Skype's appeal is easily understood. It's the videophone of science fiction that the Telco monopolies never gave us. (Thanks, guys.) But what is it with Facebook? If it's just a place where friends hang out and exchange news and funny stories, would it really have become all that big?

I've been thinking a lot about this, because I'm a latecomer to Facebook, having resisted it for a while. Now I find I'm thinking of it as 'Wastehook', an addictive way to spend time that I later regret. What made Facebook so addictive to a person who resisted it so much to start with?

Yes, it's cool that we can keep tabs on all the people we've ever known. But there's more to it than that.

I think the one thing of absolute genius that Facebook has pioneered is the 'friends of friends' concept. 'Friends of friends' gives you visibility just beyond the horizon of your circle of contacts. You get to know of people who are somewhat interesting because of the friends you share. Sometimes, they turn out to be people you know too! Bonus points for the thrill in such cases.

'Friends of friends' are people our friends like and trust, and since we like and trust our friends, their friends are people we're already favourably disposed towards. We don't mind reading the things they say. Sometimes, they say witty and wise things. Facebook is a lot more interesting because of these people. It's not just our boring and predictable friends. It's these people, unknown but not quite irrelevant, who bring a bit of variety to the experience. It's somewhat interesting when a friend puts up photos of themselves, but a lot more interesting to read what their friends have to say about it, even if we don't know many of them. We can join in the conversation and politely add to what they're saying, and nobody minds. A nice polite party with decent people we could be friends with. That's Facebook.

'Friends of friends' prevents Facebook from becoming a stale backwater, a stagnant pond, an eddy in a stream. It's not quite the wide blue ocean, though. It's just a safe little harbour. 'Friends of friends' expands our circle just enough to make Facebook an interesting place to hang out, but not so much that it becomes overwhelming or irrelevant.

I'd say that's Facebook's secret sauce.

[What about Google+? Well, Google+ pioneeered categories of friends, but Facebook quickly neutralised that advantage. An authoritative source informs me that Google+ has a rough equivalent to 'friends of friends' based on followers and following, so the effect could be similar. But my prize for pioneering the next generation of the Internet goes to Facebook in any case, not Google+.]

Does Redis Undermine a Key REST Tenet?

While I have always admired the elegance of REST's resource model and its standardised verbs and status codes, there are two things about the RESTful style that I think are drawbacks when building complex applications. One is its client/server model rather than a peer-to-peer model, which I think would have been more generic and useful, especially when unsolicited notifications need to be supported. The second, which is what I want to talk about now, is the insistence on statelessness (or alternatively forcing anything stateful to be either managed by the client or modelled as a resource). This is an acceptable approach for application domain-specific elements, but there is also a domain-neutral class of session-stateful elements that are useful for providing Qualities of Service (such as a sliding window of message sequence numbers for reliability, or session keys for security). These are common requirements that any application may require. This kind of session state is fundamentally different from domain-specific state, e.g., a shopping cart in an e-commerce application which would not be relevant to any other type of application.

The lack of support for this application domain-neutral subset of session state within the RESTful style of design costs us the ability to provide Qualities of Service in a uniform way (which the much-disdained WS-* is able to do using standards like WS-SecureConversation/WS-Security and WS-ReliableMessaging). [Many will point out that RESTful applications can use SSL for security, but the SSL protocol is not itself RESTful, and it breaks the cacheability and routability of REST operations.] So Qualities of Service end up becoming an application responsibility in RESTful applications, which has always struck me as a bit of a cop-out on the part of the protocol.

REST has standardised many other aspects of interaction and turned them into a uniform protocol that interacting systems can use. I would like to see QoS-related session management also similarly standardised and "pulled into the protocol", even if that protocol is not HTTP but something else (WebSockets?).

I have often wondered exactly why statefulness is considered "evil". There are two reasons, as far as I can tell - scalability and failure recovery. Session state occupies memory, so holding session state for multiple clients impacts the scalability of a server. Also, holding session state within a server makes it less easy for clients to switch to another if the one holding its session objects goes down. We'll need to have session-aware clusters rather than stateless farms of servers to provide failover, which again doesn't scale linearly.

Two considerations make these arguments weaker.

The first is that we are not talking about application-specific session state like shopping carts and the like. We're talking about relatively tiny data elements that are domain-neutral and dedicated to providing Qualities of Service, such as sliding windows of sequence numbers for reliability, and security tokens for message encryption. These aren't huge, perhaps a few hundred bytes at most.

The second is that the advent of NoSQL databases has given us a way to delegate the storage of session state. It's now becoming recommended best practice to store session state in a NoSQL database like Redis rather than in memory. Delegating session storage is an alternative to session-aware clusters, since the servers can now be a simpler stateless farm and access the NoSQL database for shared session state. What's impressive about Redis in particular is that the complexity of its GET and SET operations is claimed to be of Order(1), i.e., constant time. This means that (in theory at least) one can increase the number of servers using a Redis datastore indefinitely with no impact on performance.

Now, a horizontally scalable farm of servers can share a Redis datastore and use it to hold the individual sliding windows of message sequence numbers for a number of clients, and also their security tokens. Scalability is no longer a problem, because the data elements are small and the storage/retrieval operations are of constant-time complexity. Failover is also not a problem since the servers themselves are stateless and switchable at run-time.

It would be good if a standard and application domain-agnostic mechanism could be evolved to provide security and reliability to REST-based interactions, using Redis-style scalable session storage. This may be the long-awaited successor to REST, which Rohit Khare's ARRESTED style attempted to do (but which has no practical implementation to date).

That will then leave client/server as my only bugbear with the RESTful style. We'll need another extension to handle that.

The Return of JavaScript - My Thoughts Crystallise

I wrote some months ago that JavaScript was back. I've been watching developments in that space with close interest, and a few glimmerings of "the right way forward" have begun to suggest themselves to me.

The first aspect pertains to the language itself. The JavaScript Renaissance has brought with it a desire to do things right the second time around. JavaScript doyen Douglas Crockford has lectured extensively (and even written a book) on the need to use only "the good parts" of JavaScript and avoid the bad ones. It's excellent advice, but to a new generation of JavaScript programmers, I suspect it may not be enough to post warning signs around bad features. We need a way to make it physically impossible to use JavaScript's "bad parts".

Enter CoffeeScript. While I haven't played with it too much, I like what I see here. [Update 24/09/2011 - I *have* played with it for a few hours now, and I really like!] There has been some skepticism about CoffeeScript from a section of the JavaScript old guard, but that's to be expected. The best answer to such skepticism is from ThoughtWorks's latest Technology Radar document, which is well worth a read in itself.

First, even if it seems a trifle off-topic, read what Thoughtworks thinks about GWT, which is a way to write visual components in Java and have JavaScript-based widgets automatically generated for a web application that runs within a browser:

GWT is a reasonable implementation of a poor architectural choice. GWT attempts to hide many of the details of the web as a platform by creating desktop metaphors in Java and generating JavaScript code to implement them. First, in many ways, JavaScript is more powerful and expressive than Java, so we suspect that the generation is going in the wrong direction. Secondly, it is impossible to hide a complex abstraction difference like that from event-driven desktop to stateless-web without leaky abstraction headaches eventually popping up. Third, it suffers from the same shortcomings of many elaborate frameworks, where building simple, aligned applications is quick and easy, building more sophisticated but not supported functionality is possible but difficult, and building the level of sophistication required by any non-trivial application becomes either impossible or so difficult it isn’t reasonable.

Now read what Thoughtworks says about CoffeeScript:

Some readers may be confused by our advocacy of Coffeescript given our general dislike for GWT, because on the surface they seem similar: tools that generate JavaScript. However, it is the level of abstraction that differs. GWT has an elaborate component model, which tries to hide details about the underlying language (JavaScript) and platform (the web). Coffeescript tries to make it easier to write proper JavaScript, avoiding pathological but default “features” of JavaScript, and does not build a layer that tries to insulate you from the platform.

I think they've nailed it. I have been ambivalent about GWT for a while, but couldn't quite put my finger on why (i.e., it's a "reasonable implementation of a poor architectural choice"). They've also very cogently argued why CoffeeScript doesn't fall into that category.

So my first takeway from the JavaScript Renaissance is that today's developers should code in CoffeeScript and not in JavaScript itself. If Java and Scala are two languages that compile to Java bytecode and run on the JVM, then think of JavaScript and CoffeeScript as two languages that "compile to JavaScript" and run within a JavaScript engine. The runtime behaviour is unchanged, but you use the language that works better for you as a developer.

CoffeeScript is JavaScript with the sharp edges removed. I'm sold. [There is an issue with E4X, which I'll cover when I talk about Nodejs, but it applies equally to CoffeeScript. In short, there's no support for E4X, and I want it!]

Beyond the choice of language, the direction is not so straightforwardly clear. It's tempting to say that the future of JavaScript on the client is CoffeeScript plus jQuery, and the future of JavaScript on the server is CoffeeScript plus Nodejs. But there are some issues.

To paraphrase Thoughtworks's critique of GWT, jQuery is an elegant way of working with a horrible object model (the DOM). I don't know if there is any way to replace the DOM with a better client-side abstraction, but until then, the best of client-side goodness will remain somewhat unsatisfactory.

Nodejs is the server-side angel in white from all accounts, but I've heard some valid grumbling about Node as well. The Spanish outfit ASPL has a BEEP implementation called jsVortex. This JavaScript-based server pre-dates Nodejs, and ASPL was reportedly keen to explore Node as the underlying platform for the next version of jsVortex. However, they quickly realised that Node had no support for TLS at its core. This is not the same as supporting HTTPS, because HTTPS support obviously exists in Node. Think of TLS as HTTPS-like capability for any protocol, and since BEEP is a framework to create protocols à la carte, and since security is a ubiquitous and pretty strong requirement for most protocols, the lack of TLS support within Node is a serious deficiency for them and for anyone who wants BEEP on the Nodejs platform.

[Most people might go "BEEP who?", but in my opinion, this is a technology that was sadly delayed by over a decade, perhaps because of personality politics caused by creator Marshall Rose's less-than-smooth interactions with powerful members of standards bodies, who then blocked BEEP from becoming a standard until very recently. If we'd had BEEP 15 years ago, I suspect SOAP and WS-* would have looked very different, and perhaps REST would have looked different too. That's how powerful it is, and I still think it will change our lives one day.]

Well, so there's a piece of work to be done to enable TLS within the core of Nodejs, but that's not all.

WSO2's Mashup Server was another under-appreciated piece of server-side JavaScript from many years ago, and one of its best features was its support for E4X. It's understandable why WSO2 would be keen on E4X, because their enterprise middleware suite is based on SOAP/XML, and a JSON-only approach would hinder interoperability between Mashup Server and the rest of their product suite. Like everyone else in the server-side JavaScript space, WSO2 is looking at Nodejs as a possible underlying platform for the next generation of Mashup Server, but the leading lights of Node are reportedly not keen on supporting E4X at all. I wonder what the problem is. Surely supporting XML (through E4X) in Node wouldn't come at the cost of supporting JSON! At least the CoffeeScript guys have an excuse for not supporting E4X (not that I'm letting them off the hook for that reason!) because not all browsers support E4X and they don't want to be browser-specific. But Nodejs? I think the throat to choke is V8. If V8 supports E4X, then so will Node, and then there'll be pressure on CoffeeScript to support it too, since both Mozilla and V8 do. I asked them for this 3 years ago, but no dice. Ahem, Google...?

Then there are the various flavours of web frameworks for Nodejs (Backbone, Express, Geddy, etc.), but where is SpringFramework.js? Where is the Grails, Roo or PlayFramework equivalent? I certainly hope server-side JavaScript revs up to an equivalent level of maturity as server-side Java double quick. [Server-side Java meandered a fair bit and had many mis-steps along its journey, so server-side JavaScript could get there far faster by learning from Java's mistakes.] Judging by the level of activity in the Nodejs ecosystem, we need not fret on this account. It's just a matter of time before the current effervescence and flux give way to a stable yet rich multi-layered platform.

So that's my current view in a nutshell:

* Use CoffeeScript and not raw JavaScript from now on

* jQuery is great on the client side, but will no one rid me of this turbulent DOM?

* Nodejs is The One on the server side, but lacks some critical features (e.g., TLS and E4X). I don't know if they know about broken TLS, but they don't seem to think E4X even belongs in the to-do list, which is a pity if true. I'd like to be proved wrong.

* We need better server-side frameworks, and these are already on the way. We just need to wait till the dust settles to pick a winner.

I'll post updates as my view evolves.

The Zen of Tight Coupling and Loose Coupling

After my talk on SOA at WSO2Con 2011, I was inspired to create this fun, Zen-style instructive presentation on what really constitutes tight and loose coupling.

Enjoy!

A Geek's Dream Address in Colombo, Sri Lanka

With apologies to Freddy of My Fair Lady:

I have often walked down this street before;
But the pavement always stayed beneath my feet before.
All at once am I several stories high.
Knowing I'm on the street where you live.

[...]

People stop and stare. They don't bother me.
For there's nowhere else on earth that I would rather be.
Let the time go by, I won't care if I
Can be here on the street where you live.

This road was very close to the Cinnamon Lakeside Hotel where I stayed when I was in Colombo to attend WSO2Con 2011:

That's the address I'd like to have if I ever relocate to Colombo :-)

Back from WSO2Con 2011

I'm back home in Sydney after a very enjoyable week in Colombo attending WSO2's 3-day conference as well as the pre-conference and post-conference tutorial sessions flanking it.

Over the next few days, I'll be posting a detailed trip report with my impressions of the highlights of the conference. In this post, I want to record my overall impressions.

I was highly impressed by a few things.

One, I could not but be impressed by the rapid rise to prominence of a visionary software company. The "Open Source middleware" niche has only one serious player with a complete range of offerings. That's WSO2, a five year old company. This relative newcomer is giving established players like IBM, TIBCO and Oracle a good run for their money. I have no doubt that middleware prices will come down in a few years from their current extortionate levels to something much more reasonable, and the credit will go in large measure to WSO2's disruptive impact on the market.

Two, I also had to marvel at the achievements of a country. I spoke to a few Indian delegates at the conference, and we all agreed that in spite of the huge number of Indian IT professionals and some very large Indian IT companies, no one in India has achieved (or even attempted) what a relatively small Sri Lankan company (WSO2) has pulled off. In hindsight, it shouldn't have been too hard. Take some software products from the Apache stable, enhance them to make them usable by corporate customers, release the lot again under an Apache licence, and charge customers for just support and professional services. It takes some vision of course, but a lot more courage and conviction, and WSO2's CEO Sanjiva Weerawarana has that in spades. It has been his determination that Sri Lanka should be a leading Open Source nation, not just in terms of usage but in terms of contribution, that has led to the success of WSO2. Today, I'm told Sri Lanka ranks third in the world in terms of Open Source contribution, behind the US and Germany. It would be good if Indian companies and professionals took inspiration from their smaller neighbour and started contributing to the world through Open Source.

I was also very glad to see the number of Sri Lankans (not just WSO2 employees but also others from Universities and the like) using Ubuntu Linux on their laptops. I felt right at home after a long time of appearing like a curiosity. Indian and Australian professionals, on the other hand, seem to have sold their souls to Microsoft (or else to Apple). Any argument that Linux is "not there yet" is just an excuse. These people have been using Linux exclusively and are none the worse for the experience. They also have a bit more cash left in their pockets as a result of their choice :-).

Three, I was very impressed with the way the conference was organised. Everything went like clockwork. Events started and ended (largely) on time with very few overruns. Sri Lanka follows Indian Standard Time, but fortunately, the other connotations of the term have not been infectious ;-).

There were distinguished speakers from around the world, giving the conference an international flavour. This was not just a Sri Lankan affair.

IBM Fellow C Mohan giving the main keynote speech of the conference

Gregor Hohpe, author of Enterprise Integration Patterns, giving another keynote speech on the final day of the conference

eBay's Distinguished Architect Sastry Malladi gave a talk and participated in a "fireside chat" to talk about, among other topics, eBay's use of WSO2's ESB to process a billion transactions a day

David Schumm of the Institute of Architecture of Application Systems (IAAS), Stuttgart, talking about BPM research at his institute based on WSO2 Business Process Server and similar products

Dmitry Lukyanov, Head of Integration Solutions, Alfa Bank, Ukraine, talking about his bank's positive experience with the WSO2 middleware suite

Cognizant's Principal Architect Dipanjan Sengupta's presentation demonstrating the interoperability between IBM's WebSphere Service Registry & Repository (WSRR) and WSO2's Governance Registry, was very well received

Catering was top-class, though some of the Western delegates found the food excessively spicy. (I had no complaints, except that I have probably put on some weight ;-). The hotel accommodation and local transport were great and there were no snafus of any kind. It was testimony to the conscientiousness and attention to detail that the people involved displayed.

Four, I realised that the conference was a showcase not just for WSO2's technology but also for Sri Lankan culture. Every evening, after the main technical events were over, there was some cultural program or the other that was fascinating. Day One had an elephant ride (with people able to feed the elephant by hand before the rides got started).

There was also a very interesting talk and slideshow by a conservationist on "human-elephant conflict" and possible solutions, with very moving and often tragic stories about individual elephants. Then there was a set of folk dances that I can hardly do justice to with mere words or even pictures.

Day Two's cultural event was a Jam Session, where WSO2's employees enthralled the crowd with their musical talent.

That, and the musical talents of CTO Paul Fremantle and VP (Business Development & Product Design) Jonathan Marsh, made me think one needs to have musical ability as well as software competency in order to work at WSO2!

Fremantle and Marsh present a musical mashup (In India, it would be called a Jugalbandi)

On Day Three, a well-known local group "Bathiya and Santhush" (rather like Shaan or Hariharan in India) performed and the whole crowd danced to their music. The lyrics were in Sinhala, but the beat had universal appeal.

That was WSO2Con at a glance, and I'm sure next year's event will be even bigger and better. Those who missed this year's experience should try not to miss the next!

(Disclosure: I am in advanced negotiations with WSO2 to join them in a suitable role reporting to the CTO. While this obviously makes me an interested party, I promise not to make any statements or representations about the company and its products that I do not sincerely believe in. My decision to consider a position with WSO2 is itself driven by my conviction that (1) their product suite is fit for purpose and very good value for money, (2) the company operates in accordance with the ideals of Open Source - there is no proprietary version of any of their products, and (3) there is a good cultural fit because I like the people and the company culture (they never use the word "resource" for people) and found myself getting along very well with people across the organisation. Regardless of my relationship with the company, I'm determined to remain a voice that readers of this blog can trust, because I'm acutely conscious that my independent stance on technology matters has always been my biggest source of credibility as a commentator.)

Gregor Hohpe's Talk at WSO2Con 2011 on Enterprise Integration Patterns

I had the pleasure of attending Gregor Hohpe's keynote speech on the third and final day of WSO2Con in Colombo. (The conference was a hugely successful affair, by the way, judging by the quality of the talks and the fact that the attendance far exceeded expectations.)

Gregor's talk was a reference to his famous book "Enterprise Integration Patterns", and was titled "Enterprise Integration Patterns - Past, Present and Future". (For all his achievements and fame, Gregor is an endearingly unassuming man, as I found when I spoke to him person-to-person. He's also a very humorous speaker, and kept the audience in titters during his keynote.)

When he came to the Future part of the talk, one of the many gems he let slip was his insight that his famous book was really about Messaging Patterns, in which case it would be the first of four volumes (the other three devoted to Conversations, Processes and Events). He specifically called for two things from the audience and the community in general - some ideas on notation for these more complex patterns, and for exerting guilt-inducing pressure on him so he would actually write the books!

I have been interested in the Distributed Computing area for a long time (I would in fact have called these "Distributed Computing Patterns", because they are not specific to enterprise contexts, and the purpose of the interactions isn't always integration).

In any case, this is how I see the four volumes of the book that Gregor described, and how they would relate to each other. I would like to mail him on these ideas and I'm sure I'll have an interesting conversation.

"Messaging" is really the most generic description of a class of distributed computing systems. It also takes a top-down, system-wide focus (a "God" view). One can think of specific subsets of these generic interactions, and one can also shift focus from the overall system to a particular node. So, as we add a statefulness aspect to messaging, we begin to enable conversations. And as the purpose of the conversations becomes coordination (one participant controlling the others), we have (process) orchestration. Somewhere along the way, we can explicitly consider multi-way messaging, i.e., between multiple peers.

When we shift focus to a single node, we have an event-driven system because incoming messages at any node are "events". And when we take a set of event-driven nodes and consider them in aggregate, we have a choreographed system, where each node is ignorant of the existence of other nodes and only responds to events, and the system as a whole "works" because of these inter-related events. The design of a choreographed system is really the design of appropriate events, since the previous step (event-driven systems) has already designed the behaviour of the individual nodes.

So that's my simple view of the Distributed Computing universe. I also have a more complex diagram that I created a few years ago to explain how the various facets of distributed computing are related.

Preparing for WSO2Con 2011

Well, it looks like I'm going to be attending WSO2's annual technical conference WSO2Con in Colombo this September (12th-16th). It'll be good to hear speakers from around the world speak about SOA-related topics and the WSO2 product suite in particular. I believe the latter is one of the SOA industry's best-kept secrets. Here is a comprehensive suite of middleware products that is completely Open Source, and for which commercial support is available at very reasonable rates. I'm surprised more organisations aren't using it. Perhaps the awareness wave is just breaking and we'll see a surge in adoption in the next couple of years. Let's see.

I'm also slated to conduct an introductory tutorial on SOA on the first day of the conference, and I'm excited about that. Over the last few years, I've been developing a few ideas on "Practical SOA for Solution Architects" that I'd like to present, and I'm eager to get feedback on that approach.

What I've seen to my disappointment over the past several years is that in spite of extensive coverage of SOA in the IT industry, the actual payoff from the application of SOA is embarrassingly low. In fact, I hear from sources in large IT shops that have rolled out multi-million dollar SOA initiatives that the cost of building services today is higher than in the past when older EAI techniques were used.

I've been puzzled at this, and have discussed it at length with colleagues and peers. It looks like three different factors are at work.

1. Cynicism: There is a view among many IT professionals that SOA is yet another buzzword that has come and gone, and it's no longer relevant to the industry. So they don't even bother to learn what it's about.
   
2. The theory-practice gap: Lots of IT folk have been exposed to core SOA concepts ("Loose coupling"), but they don't seem to be able to apply it in real-world solution designs. SOA remains in the realm of theory, which is a pity, because it can be a real money-saver.
   
3. Products as surrogates for principles: Organisations have collectively spent a packet on SOA products from IBM, TIBCO, Oracle and the like, but the actual solution designs being produced in-house for business projects are as tightly-coupled as ever, suggesting that IT practitioners seem to think the use of SOA products is an adequate substitute for Service-Oriented application design.
   

I have come to the conclusion that what's required is a new approach to educating people about SOA. It has to be less theoretical and more focused on real-world situations. And it has to be lightweight, because people are busy and don't have the attention span required for a heavyweight methodology. Above all, it has to be sound, because we don't want to spread incorrect practices in the name of simplicity.

I'm preparing a whitepaper on this, and it will probably be released through WSO2, since a lot of the encouragement for it has come from Sanjiva Weerawarana, WSO2's CEO. It's only fitting that WSO2 get the credit for bringing this educational tool to a wider audience.

I'll also be posting my slides from the conference on this blog when I'm done. I'm sure they'll also be available from the WSO2Con conference website.

Stay tuned.

Identity Management on a Shoestring - The LIMA Approach

Umesh Rajbhandari and I have documented the design of an Identity and Access Management (IAM) system that we successfully built for a large Australian corporation. We call our unique architectural approach "LIMA" (Lightweight/Low-cost/Loosely-coupled Identity Management Architecture).

The document describing LIMA is here. [Update 20/03/2012: I have changed the link to point to the InfoQ site instead of my own hosted copy because that was one of the conditions of publication.]

Practical SOA Building Blocks

I'll make this short. This post is not a theoretical discussion of what SOA is. This is a practical guide to the core physical components required to build a "traditional" SOA-based ecosystem.

I believe there are three basic components required - a Service Container, a Broker and a Process Engine.

A Service Container is used when implementing business logic to expose as a service.

A Broker is used to connect to legacy business logic through mediation, routing or transformation.
A Process Engine is used to coordinate two or more services into a process and expose that as a composite service.

We put these together in different combinations to build out the specific business processes we need. Needless to say, what we run on these components should be reusable, so their interfaces need to be generic, yet extensible. Good data design is required for this.