A Good Primer on NoSQL

NoSQL databases are all the rage, but the array of choices before us is bewildering. I must confess I'm still confused about the features and differences between BigTable, GAE DataStore, GemFire, SimpleDB, SQLFire, CouchDB, MongoDB, RavenDB, Redis, Cassandra, Riak, HBase, Neo4j and so many other names that I have only recently begun to hear about. I'm sure many others would be in the same situation.

I was therefore happy to see that my colleague at WSO2, Dr Srinath Perera, has analysed the NoSQL landscape in depth, zeroed in on the characteristics of NoSQL databases that are relevant, and summarised this for our common understanding in an InfoQ article that provides a simple overview of the choices that designers and developers have today, choices that go beyond the traditional relational databases that we're familiar with.

I've often wondered about why NoSQL should be so popular in the first place. Srinath explains:

A few years ago, most systems were small and relational databases could handle [their requirements] without any trouble. Therefore, the storage choices for architects and programmers were simple. However, the size and scale of these systems have grown significantly over the last few years. High tech companies like Amazon and Google faced the challenge of scale before others. They soon observed that relational databases could not scale to handle those use cases.

In other words, this demanding new requirements wave has probably not hit most of us yet, but with the jump in the number of connected devices (smartphones, tablets and the coming "Internet of Things"), applications dealing with huge volumes of data are probably not going to be as rare as in the past. And when we say "huge", we're not even talking Gigabytes anymore. It's Terabytes and larger. As we learnt from Godzilla, size does matter. And drastic situations call for drastic measures, hence the NoSQL revolution.

Srinath refers to Eric Brewer's CAP theorem, which states that a distributed system can only have two of the three properties - Consistency, Availability, and Partition Tolerance. The NoSQL databases aim to break through the limitations imposed on traditional relational databases by loosening the fundamental principles on which these have been based, dropping one or more constraints as appropriate, to obtain a desired behaviour.

Depending on the constraints dropped, the resulting solution falls into one of several new categories:

• Local memory
• Distributed cache
• Column Family Storage
• Document storage
• Name-value pairs
• Graph DB
• Service Registry
• Tuple Space

...in addition to the traditional filesystems, relational databases and message queues that are familiar to IT practitioners today.

Perhaps the most important contribution of Srinath's article is his distilling of the four primary characteristics that are important from a usage point of view - data structure, the level of scalability required, the nature of data retrieval and the level of consistency required. He then puts these characteristics together in various combinations to show which of the above-listed categories of data store would be the most appropriate solution to use.

He's certainly succeeded in demystifying NoSQL for me, although I suspect I'll need to go back and read the article a few times till I've fully internalised the concepts in it. This is an overview article that I'd recommend to anyone trying to make sense of NoSQL and wanting to decide on the appropriate product category that would be right for their needs.

I can see the demand for a follow-up article from Srinath drilling down into each of these data storage categories and providing recommendations about actual products (e.g., Cassandra, Redis, CouchDB, etc.) While the sands shift more rapidly in the product space, it's also a more practically urgent decision for a developer or architect to make. So while such an article might need to be updated quite frequently, the advice in it would be more practical than this one, which provides the necessary initial understanding of the NoSQL landscape.

Strange Creature on the Mozilla Firefox Download Page

I've never seen this creature before (circled in red). Do you know who or what it is? It looks friendly enough, but it also reminds me of the Morlocks in H.G. Wells's The Time Machine (shudder).

I Hate HatEoAS

For something that's supposed to be THE defining characteristic of REST, it could have done with better naming.

I would have been happy with the term HatEoAS if it had stood for "Hypermedia as the Envelope of Application State" rather than "Hypermedia as the Engine of Application State".

An Engine actively drives things. E.g., A process engine is well named, because it drives a process.

A constraint doesn't drive anything. It constrains. It provides an envelope around the range of possibilities.

And so they really should have called this an envelope rather than an engine of application state.

There, I've said it. Because the expansion of HatEoAS has been driving me up the wall.

Fortunately, it's being referred to as "Hypermedia Constraint" now, which is both more elegant and more accurate.

Oneiric Ocelot Not Quite the Stuff of Dreams

One of the nicest things about Linux distributions like Ubuntu is that you don't have to spend the night standing in a queue just to get the latest version of an operating system on the day of its release.

Unlike some other (closed) systems where supply is deliberately constrained
to create an impression of even greater demand, Ubuntu is upfront and
relaxed about new versions.

I sat down at my Linux desktop yesterday and was pleasantly surprised to see a popup informing me that the next version of Ubuntu Linux (version 11.10 a.k.a. "Oneiric Ocelot") was now available and would I like to upgrade?

With a smile of anticipation, I clicked Yes, and the upgrade began. My ADSL link showed a steady bandwidth of around 400 kBps throughout. An uneventful hour and a half later, I rebooted into Oneiric Ocelot. That was the kind of experience I've got used to with Ubuntu over so many online upgrades.

Most of the time, upgrades to Ubuntu are boringly anticlimactic. That's a good thing, by the way, because users hate surprises, and there are really very few nice surprises possible on an upgrade.

The dictionary says "oneiric" means "pertaining to dreams", but there was something almost nightmarish with this upgrade that was even worse than the upgrade to 11.04 ("Natty Narwhal").

Someone at Canonical has taken it into their head that a completely re-imagined user interface would be a good thing. The same someone has also arrogantly assumed that there's no need to give users a choice when changing a fundamental aspect of the user interface that they will use all the time.

My unpleasant surprise after both upgrades was the horror they call the Unity Desktop. I tried to be fair to them. I believe I gave Unity an hour of my time both times. In the end, I gave up. I just hated it. Sorry Canonical, I recognise you're trying, but Unity really doesn't work for me. From the number of similar comments I read on the web, I'm hardly alone.

What was far worse than installing Unity by default (without asking me if I wanted it) was not providing me a quick way to get back to the default Gnome desktop of earlier versions. I was actually forced to download the Gnome desktop, then re-login to select it as my default desktop.

1. Why didn't I have the choice to say no to Unity at the time of the upgrade?
2. Why wasn't it a straightforward option to return to the "classic" desktop?

For a distribution that is supposed to be the friendliest desktop Linux, this is very poor showing indeed.

My other major whinges are that the "Show desktop" button on the taskbar has disappeared, as has the "System" menu on the menu bar. I now have to minimise every window manually, and have no way to set several preferences. Cosmetically as well, the taskbar at the bottom and the menu bar at the top now sport a ghastly dark grey colour, and the desktop theme that I used to use has disappeared from the list of options. Since I've forgotten what it was called, I don't think I can get it back.

This upgrade experience has been anything but oneiric. I feel like I've been mauled by an ocelot.

Practical SOA for the Solution Architect

This is a story that has had a fairly long history, so stay with me till the end.

I've worked as an architect in the shared services space for almost a decade now, at some of Australia's biggest, richest and technologically diverse financial services organisations. During that time, I first heard about Service-Oriented Architecture (SOA), learnt what it was about, bought into its philosophy and attempted to implement it at work.

And while I have seen a few successful examples of SOA projects (mostly individual services, truth be told), by and large, I did not see SOA having an impact at all at these large and reputed organisations. Many hundreds of thousands of dollars were spent on acquiring SOA tools from vendors as reputed as IBM and TIBCO, and many millions more were spent on integration projects using these tools, but somehow, the results failed to live up to the SOA promise. [I guess organisations need a terrorising CEO like Amazon's Jeff Bezos to achieve the benefits of SOA. Read the fascinating story of how an online bookstore built a platform that it now rents out to others.]

For a while, I lost faith. Influenced by a few cynical colleagues, I too began to think SOA was marketing hype and nothing more. But then, as I continued to work in the shared services domain and had the opportunity to review more solution designs, I had an epiphany. Most of the designs I was seeing used SOA products (usually an ESB) or were implemented as Web Services, yet I could see they were still tightly-coupled at the level of the application design, i.e., the data. Also, it was very common for solution designers to use the wrong tool for the job, simply because it was the one they were most familiar with. They even seemed to lack a conceptual ability to tell which tool would be right for a given requirement.

With that epiphany, I revisited my understanding of SOA. I realised that solution architects needed to be educated to produce SOA-compliant application designs, otherwise all the investment made by their organisations in SOA tool suites would be a waste. Worse still, SOA itself would be unfairly blamed for the waste of resources, when it in fact remains the best hope to reduce waste, improve business agility and reduce operational risk.

I've been tossing this idea around in my head for at least a couple of years now, the idea that a lightweight method is required to get solution architects up to speed with the required concepts quickly. Unfortunately, most solution architects would bristle if it was suggested to them that they don't really understand SOA. "Can't you see we're using an ESB?" would be the defensive response. That the response is a non sequitur would be lost on them. One can use an ESB and still come up with a design that is not SOA-compliant. How can we educate solution architects when they don't know what they don't know?

Fortuitously, I had a chat about this in August this year with Sanjiva Weerawarana, the CEO of the innovative Open Source middleware company, WSO2. Our needs meshed perfectly. WSO2 has a full suite of middleware products based on SOA concepts and Web Service standards. They are compact and not bloated, fairly straightforward to install and use, fully Open Source under the Apache Software Licence, and for which WSO2 offers a very attractively priced support model as well as other professional services. For such an innovative range of value-for-money products, the level of awareness in the customer community has been surprisingly low. Sanjiva has been trying to raise the level of awareness in the industry about his company's products for some time now. For too long WSO2's pitch had been aimed at developers (techies talking to techies), but for a real breakthrough, they needed to target decision-makers and decision-influencers. They were looking for a way to reach the solution architect with a compelling message.

And here I was, trying to educate the same solution architect about SOA using a new, lightweight approach. So Sanjiva and I came to an understanding. I would do a paid consulting assignment with WSO2 for a few months and turn out a few white papers to present their offerings to an audience that was higher up the corporate food chain than the developers. In return (in addition to helping me pay the bills), they would provide me a vehicle to popularise some of my ideas on SOA, especially the lightweight methodology I came to call "Practical SOA". I guess this approach is the good cop to Jeff Bezos's bad cop :-).

The first of those white papers ("Practical SOA for the Solution Architect") has now been completed and is available on WSO2's website. It's my audacious hope that a Solution Architect can read it in half an hour and be immediately effective on their project thanks to a simpler and more powerful mental model of SOA. A short summary of the paper is available for you to skim through, but I would encourage you to download the full paper (a free registration is required), since it has much more detailed descriptions, extensive explanations for the final conclusions and a couple of industry examples to drive home the concepts. So please have a read and, if you like it, recommend it to all your architect- and designer-type friends. (At my own request, my name doesn't appear on this document because I don't want to dilute the appeal of the method by causing it to look like a mere individual's opinion. WSO2's cachet is better than mine!)

A second white paper that is currently in the works will describe the full suite of WSO2's products and how they map to the framework established in the first. Stay tuned for that too. The first white paper will equip you with concepts. The second will equip you with know-how about a comprehensive set of reasonably-priced tools. Together, they should provide a customer organisation with excellent value for money and the long sought-for return on their investment in SOA.

Steve Jobs and Dennis Ritchie

This hasn't been a great week to be a computer pioneer. Dennis Ritchie has now followed Steve Jobs off the stage and into that great big computer industry in the sky. Two giants in a single week.

Steve Jobs

Dennis Ritchie

I guess, unlike most people, I have been touched by Ritchie much more than by Jobs. I have never owned a Mac or an iPhone, and regretted buying an iPod as soon as I realised there was no way to bypass the iTunes straitjacket. iPod generation 4 worked with the gtkPod application on my Ubuntu desktop, but generation 5 corrected that shocking oversight, and the Apple empire, with a sigh of relief, regained its pristine purity, shutting out the great unwashed once more. That ended my dalliance with Jobs and his closed system. I don't fancy handcuffs even when they're haute mode.

I feel sorry about the death of Steve Jobs the human being. I have no sympathy for the worldview that he represented, of closed systems, slimy lawyers and patent lawsuits.

Dennis Ritchie, of course, was the polar opposite of Jobs. Those who have read Asimov's science fiction trilogy Foundation may remember that there were two Foundations, a well-known one at the periphery of the Galactic empire, and the other, a secret one, located "at the opposite end of the galaxy". Many characters in the novel tried searching for the Second Foundation along the opposite edge of the galaxy where they thought it would be, but its actual location was right at the centre of the empire! The term "opposite" was meant in a sociological sense, not a physical one.

And in true Foundation-esque fashion, Ritchie's contribution to mankind, while in a sense the opposite of Jobs's, was not a rival closed system but an open one. Along with Ken Thompson, he wrote the most open operating system of its time, -- Unix.

Ken Thompson

The popular web article "The Last Dinosaur and The Tarpits of Doom" has a matchless passage describing the world at the time of Unix's birth.

In 1970, primitive proprietary operating systems bestrode the landscape like mighty dinosaurs: Prime's PrimeOS, DEC's RSTS, RT-11, etc. (with VAX/VMS soon to come), IBM's innumerable offerings, CDC's Scope and of course dominating the scientific workstation market, Apollo's Domain.

Who would then have dared to predict the fall of such giants?

What force could topple such entrenched operating systems, backed by massive industry investment, hacker culture and customer loyalty?

Today, of course, we all know the answer:

In 1975 Bell Labs released Unix.

• Unix had no support from its creator, AT&T: Buy the magtape and don't call us. (AT&T was legally barred from entering the operating system market.)
  
• Unix had no support from any existing vendor: None had the slightest interest in backing, supporting or developing an alternative to its proprietary operating systems offerings.

• Unix had zero customer base: Nobody had ever heard of it, nobody was requesting it.
  
• Unix had zero marketing: Nobody had any reason to spend money building mindshare for it.

A one-sided competition?

Decidedly: Unix wiped all workstation competition off the map in less than fifteen years.

On April 12, 1989, HP bought up Apollo at a fire-sale price, putting out of its misery the last remaining proprietary operating system vendor in the workstation world, and the workstation proprietary OS era was over: Unix was left alone in the workstation market.

In fifteen years, a [magnetic] tape and an idea had effectively destroyed all opposition: Every workstation vendor was either supporting Unix or out of business.

Let me add one more point to that. At the heart of Apple's operating systems is a version of Unix (BSD Unix). Steve Jobs's business empire took a freely available operating system, layered a user-friendly graphical interface over it, and without a word of thanks, proceeded to build a proprietary edifice that was as closed as its enabling technology was open.

So thank you, Dennis Ritchie, for giving us today's Mac.

Ritchie was an inventor second to none. People today forget one of the main reasons Unix is considered "open". Before Unix, an operating system was written for a specific processor chip, in the assembly language corresponding to that chip. One of the key factors that made Unix open was the fact that it could be ported to any chip at all. More than 90% of an operating system's logic is in fact independent of the underlying hardware architecture. Less than 10% is specific to the chip. That's why only very low-level code in Unix is written in assembly language. That's the only part that needs to be re-written when porting Unix to a different processor architecture.

Once the operating system was liberated from its ties to hardware, any hardware manufacturer could port Unix to their computers. That's the openness that destroyed the proprietary dinosaurs and created the world we see today. We have Thompson's and Ritchie's genius to thank for that. In the next generation, Linux proceeded to wipe out proprietary Unix variants to take over the server room.

Today, Google's Android has Linux at its core. So now Ritchie's invention has taken over the server, a significant part of the desktop (through the Mac) and an increasingly dominant part of the smartphone and tablet markets (through Android and Apple's iOS). Not bad for a simple and open operating system!

Now we know that 90% of Unix is written in a higher-level language, and therein hangs another tale. At the time Thompson and Ritchie wrote Unix, there was no suitable high-level language to write an operating system in. It had to have the higher-level constructs of most modern, structured, procedural programming languages. Yet it also had to provide sufficient control over low-level constructs like memory addresses and file structures. This was a challenge that may have stumped other people and caused them to compromise in some way. Not Ritchie. Necessity for him was the mother of invention - the invention of the C programming language. Together with Brian Kernighan, Dennis Ritchie created the first C compiler, and it is astonishing that the language has hardly had to change since their version to the present day. The standardised version of their language, ANSI C, is largely the same as their original one, with just minor changes. Now that's vision for you.

Brian Kernighan

C inspired C++, Java, JavaScript, Perl, C# and a whole bunch of other languages. Any language with curly braces and semicolons owes an intellectual debt to Ritchie and Kernighan.

The laptop that I'm composing this on runs Ubuntu Linux, another variant of Unix. Most of Linux is written in C. I'm probably not fully aware of the extent to which I owe Ritchie a debt of gratitude, as the one common factor in the creation of both Unix and C.

By the way, if you think Unix has an ugly user interface because of its command line, there are two rebuttals to that argument. The trivial one is that modern Unix variants like Linux have very sophisticated and friendly user interfaces indeed. The deeper rebuttal is that there is beauty and power in the Unix command line that MacOS has eagerly embraced as an offering to the "power user".

User Interface experts Don Gentner and Jakob Nielsen write in their classic paper The Anti-Mac Interface:

The see-and-point principle states that users interact with the computer by pointing at the objects they can see on the screen. It's as if we have thrown away a million years of evolution, lost our facility with expressive language, and been reduced to pointing at objects in the immediate environment. Mouse buttons and modifier keys give us a vocabulary equivalent to a few different grunts. We have lost all the power of language, and can no longer talk about objects that are not immediately visible (all files more than one week old), objects that don't exist yet (future messages from my boss), or unknown objects (any guides to restaurants in Boston).

Like they said. As an advocate of the power of the Unix command line, I rest my case.

Unix is such a unique phenomenon in the world of computing that noted academic Prof Martin Vermeer believes it should be treated as a basic element of literacy, alongside the three Rs.

And so a tumultuous week has gone by, and the computer industry mourns its two luminaries. Among computer pioneers, Steve Jobs was the shiny user interface, slick and popular. Ritchie was the kernel, unseen and unknown to the masses, yet the workhorse that made everything else possible, including the user interface. He may be less widely mourned, but no less mourned.

And I like to think Ritchie rushed after Jobs to make sure the Pearly Gates stayed open to all!

Google's Aimless Darting

Search Engine vendors should be in the clarification business, not the muddying business. All the more frustrating to read the news about Dart, Google's new web programming language to replace JavaScript.

My only reaction is - WHY???

Yes, JavaScript is not exactly a perfect language. It has warts, huge ones. That's been known for a long time. Is that sufficient reason to throw the whole language overboard and try and popularise a new one? Especially after CoffeeScript has already done the job? JavaScript has good parts as well as bad, and I'm not talking about just its technical aspects. Its ubiquity is a major strength. Replacing it is an exercise with very doubtful prospects. I don't think even Google can pull it off.

CoffeeScript has taken the right approach, in my opinion. CoffeeScript is JavaScript with lead shielding over the reactor core. And that was all that was required. CoffeeScript on the server side uses Nodejs to run scripts anyway, so one can write server-side code in CoffeeScript and run it with "coffee" instead of with "node", and the ugliness and danger of JavaScript can be neatly sidestepped. Even on the client side, a single line such as

<script type="text/JavaScript" src="coffeescript.js"></script>

will allow you to write the rest of your client-side code in CoffeeScript, because coffeescript.js is a minimised library that will let your browser interpret CoffeeScript natively. Your application code will look like this:

<script type="text/CoffeeScript">
# CoffeeScript code
</script>


What is Dart going to do for us beyond that, functionality-wise? Technically, Dart won't even replace JavaScript, because it will compile to JavaScript. Does that sound familiar? Because that's just what CoffeeScript does as well! Was it so hard for Google to get behind CoffeeScript? Some NIH at work, I think.

Now the frustrating thing is that Dart will waste at least some developer mindshare and bandwidth when the world should have been just getting on with the job - using CoffeeScript plus jQuery on the client side, CoffeeScript plus Nodejs on the server side.

What a waste!

Life above the Service Tier (Change of Links)

Google Groups no longer allows file uploads. Worse, they went back on a commitment to keep the files that were already uploaded accessible. Now all the files I uploaded to the "wisdomofganesh" group have to find alternate homes (if I still have a local copy, that is), and I have to update all the links from my blog. Totally uncool of Google. I'm very annoyed.

Update 15/11/2012: I've changed again from mesfichiers.org to slideshare, since that seems to be the new place to upload shareable stuff.

Anyway, since the following docs seem to be the most frequently accessed, here are the new links:

1. Life above the Service Tier (white paper)

2. Life above the Service Tier (presentation)

3. SOFEA in a SOA Ecosystem (diagram)

Amazon Cloud Drive Tech Talk - Sydney, 10-10-2011

I attended the Amazon Cloud Drive Tech Talk today in Sydney. This was held on the 39th floor of the Citigroup building at 2 Park Street in the CBD. The views from the window are awesome, by the way.

That's Park Street stretching away towards King's Cross

Those red slugs are the new Metro double-buses

I reached the venue just after 1800, when registration was scheduled to begin. There were eats, bottles of beer and canned carbonated drinks in the room, but none suitable for a vegetarian teetotaller trying to lose a few calories, so I drank a glass of water instead and waited for the proceedings to begin.

The event kicked off promptly at 1830. John Scott of the Android Australia User Group made a short speech introducing Piragash Velummylum of Amazon, who had flown in from the US.


Then Piragash took the stage and spoke. I must say the Amazon Cloud Drive turned out to be more Amazon Recruitment Drive than anything else! Piragash and a colleague Brad (who spoke for half a minute) made no secret of the fact that they were in town to recruit developers for their Seattle office. The tech talk was like a campus recruitment talk - just enough data to pique the interest of developers wanting to work on cool technology.


The unofficial motto of the Amazon Cloud Drive service is "anything digital, securely stored, accessible anywhere". Piragash spoke a bit about Amazon CEO Jeff Bezos, who he projected to be a likeable nerd who listens to everybody and does the right thing. [Piragash must have a weird sense of humour. Just read this piece and the comments that follow. I wouldn't want to work for Jeff Bezos in a hundred years.]

Amazon Cloud Drive is meant to address three customer pain points:

• Multiple music downloads
• Moving files from one store to another
• Data loss

He said something about Amazon MP3, which seems to be similar to Apple's iTunes (Piragash sidestepped a question from the audience on a comparison with iTunes, saying they had a policy not to talk about competitors, but mentioned that the Amazon version allowed upload of customer content). I think the story might have begun with Amazon MP3 which catered to music files. Then Amazon Cloud Drive came along which was more general-purpose and catered to videos and other kinds of files as well. That's what I gathered.

He emphasised a few times that Amazon Cloud Drive is for customer content, not purchased content (also called studio content or catalog content). It's like DropBox, I guess. It's said to be free, but there was no mention about storage limits. Unlike DropBox, they don't do de-duplication of files (yet), and certainly not across users. They would need to sort out licensing before they did that sort of thing, and he made a statement to the effect that Amazon is DMCA-compliant.

They obviously leverage off their other technologies, i.e., AWS (Amazon Web Services). They currently use S3 (Scalable Storage Solution), not EC2 (Elastic Cloud Computing), but they may use that in future if required.

Amazon Cloud Drive has three layers. Storage is provided by S3. They've defined a hierarchical file system on top of that (which has long been a demand of S3 users). Finally, they have metadata on top that defines relationships and queries, making the file system much more useful.

These are some of the operations supported at each of those levels:

S3: Upload, download

Filesystem: Copy, move, delete, list, recycle, restore, create

Metadata: Select, get, put, delete (Aha, REST! But Select seems to have replaced Post. Puzzling.)

Piragash also talked about some of their technical challenges.

Scaling was the major one (and continues to be a major focus of research and innovation). The principles Amazon follows to ensure scalability are:

1. "Don't get in the way", i.e., let AWS and Akamai do the job they know best, don't interpose Amazon Cloud Drive between those systems and the user. Rather, allow Amazon Cloud Drive to be proxied by CDNs (Content Delivery Networks) like Akamai.

2. "Be flexible", i.e., be forward-thinking and don't prevent services like allowing a zipped set of files to be downloaded.

Security is another major concern. Content is meant to be private and personal. As part of their PaaS offering, Amazon provides an Identity and Access Management (IAM) system, and Amazon Cloud Drive uses IAM and AWS to control the generation of time-bound tokens. Delegated S3 access has an extra security token in the URL that expires after a certain period, so people can't pass content URLs around and have them accessible indefinitely to anyone who gets the URL. Can customers share their content with others, or is this purely private? Piragash's response: "Stay tuned".

Then Piragash briefly talked about the Kindle Fire, due for release in about a month. Incidentally, Kindle for iPad is said to use HTML5 throughout and to look like a native app.

There's also a new business called Amazon Fresh, currently only rolled out to the Seattle area. Order your groceries online at midnight, and have them delivered to your door by 6 am!

There was a little talk about version synchronisation strategies and rules for conflict resolution. They support multiple schemes for different domains. Sometimes they use automated algorithms, and at other times they let the customer resolve conflicts.

Asked about an API for Cloud Drive, Piragash would only say, "Stay tuned".

He mentioned that it was important to weed out "phantom requirements" and to concentrate on solving the customer's real problems.

A member of the audience remarked that Australia really needed an edge server located here to reduce latency. Piragash merely smiled acknowledgement. Someone else asked about the number of servers used by Amazon, but Piragash could not talk about that.

That was the end of the talk, and Piragash invited people to stay and talk to him and his colleagues. He called for CVs and said they'd be in Sydney the whole week, recruiting people to be based out of Seattle.

So if you like rain throughout the year and don't mind sharing a city with Microsoft, do apply to Amazon for a job.

[What was even more interesting than the technology talk was a nugget of cultural trivia that I've written about on my other blog.]

Aakash - The Sky's the Limit

Android tablet prices went down even further with the announcement of the Indian-made tablet "Aakash" (Sanskrit for "sky", pronounced "aakaash"). The price to students (admittedly with a government subsidy) is projected to be $35. Even without the subsidy, the retail price should still be a groundbreaking $60. The aim is to bridge the "digital divide" and allow less affluent sections of society to participate in the digital economy.

I like free markets, but given the tendency for cartels to form in most market segments (negating the freedom that true liquidity would deliver) and the distortions of patent law that entrench the power of large corporations, I also think governments need to step in from time to time to ensure equity. Untrammelled capitalism isn't going to bridge the digital divide. There has to be a deus ex machina that kicks in at critical junctures.

Stepping back a bit to take a historical view, the primary difference between the 19th and 20th centuries was not in scientific knowledge or even technological invention (because many 20th century inventions were known even in the 19th), but in the mass production and mass consumption of such technology. In analogous fashion, I can see the potential for India to become a technology "enabler" for the world's poorer half (or two-thirds), democratising technology usage across a geographical span rather than a historical one.

I remember the dire warnings a decade ago about an AIDS epidemic that was poised to devastate Africa, and the utterly shameful behaviour of the big Western pharma companies in refusing to lower the prices of AIDS medication to save millions of lives. They tried to use patent law to block any attempt by other parties to provide cheaper medication. If the intellectual wherewithal had been entirely lacking in the Third World, they may well have got away with it. But Indian pharma companies were able to produce AIDS medication, and more importantly, produce it at a much lower price point that African countries could afford, and the Third World as a whole managed to vote their way around the IP laws that the US (and other Western countries) have forced all others to sign. A humanitarian disaster was averted that would have dwarfed the Holocaust, and India had a large (though also largely unsung) role to play in averting it.

More recently, Tata Motors established a radically lower price point for cars ($3000). The Nano is a revolutionary breakthrough, and it isn't a toy either. If a car can survive on Indian roads, it will positively thrive anywhere else ;-). The Nano is going through a few teething problems right now, but I have a sense that in a decade or two, it will be one of the world's iconic car brands, perhaps the most ubiquitous. The Nano could change the image of car ownership as an indication of wealth.

And now, with Aakash, India is once again bringing technology within the reach of ordinary people, at Third World prices. The term "reasonably priced" means something very different in Western countries. Even middle class people in Third World countries cannot afford these "reasonably priced" products. In Marxist terminology, as long as the "means of production" were concentrated in Western hands, there wasn't much the rest of the world could do about it. They either paid those prices (exorbitant by their standards) or simply did without. Now they have a choice. Incongruous as it may seem, India is swooping in in shining armour to save the day.

I do have some reservations, though. Indian ingenuity has never been in doubt. What's in doubt is India's institutional ability to follow through, to execute, to deliver. India has always been a muddle-through country rather than a reliably-achieve country. Even the Nano is a case in point. The political shenanigans that preceded its launch very nearly canned the project. If I was someone big in the Indian government, I would not look at this as just a private sector enterprise that is none of the government's business. I would see it as a national enterprise where India has a chance to put its stamp on the world and change it for the better, and I would provide Tata Motors with all the support needed to manufacture and sell the Nano in volume, worldwide.

[I'm not going to argue about whether an invention that serves to consume more fossil fuels is going to change the world for the better. The environmentalists don't seem to have an easy answer to the developmental issue of stagnation-versus-pollution either.]

Again on the topic of the Indian character, I remember my student days at IIT Madras when a new Siemens mainframe computer was delivered to the institute. This was in the mid-eighties. The box was too big to go up the stairs of the computer centre. Many students and professors watched as the workmen, mostly uneducated, rigged up a makeshift pulley and with the help of ropes, winched the box up the side of the building, and others leaned over the parapet wall on the first floor and hauled it in. Mission accomplished with a minimum of technology (and no insurance!)

Watching this, one of my professors who had done his PhD in the US and worked there for a while, remarked wisely, "We Indians are great at improvisation. The danger is that we will be satisfied with our ability to improvise, and fail to develop real systems."

I see no evidence that India has improved significantly on the systems front. Flashes of Indian brilliance, like the Nano and the Aakash, will remain just flashes in the pan unless the country learns to be more disciplined about delivery. Virtually every developed country emphasises delivery discipline. Only when India masters that can the world look forward to a steady stream of dirt-cheap high technology that will change billions, not just millions, of lives.

Meeting of Android Australia User Group - Sydney 05/10/2011

I attended my first meeting of the Android Australia User Group - Sydney this evening.

The organiser, John Scott, started with some general announcements, which should be of interest to many.

- There is an Amazon Cloud Drive Developer Tech Talk in Sydney on Monday October 10, 2011.

- Google Developer Day will be on November 8, 2011 in Sydney.

Those interested should register for these events. They're both free, as far as I know.

John then kicked off the first talk of the evening, which was a brief overview of Motorola's IDE for Android application development - MOTODEV Studio. According to John, this IDE is better than the Google Android plugin for Eclipse, which seems to be the de facto IDE for Android, because it has a few extra features like the generation of boilerplate code to make the developer's life easier, and also graphical tools to manage SQLite databases. I found a couple of other discussions on MOTODEV Studio vs Eclipse here and here, but they seem somewhat dated.

The second talk was by Gianpaolo De Biase, a developer with AppCast. Gianpaolo talked about some real-life applications that his company has built (one of which is JustStartWalking, an app for an initiative of the same name from the Chiropractors' Association of Australia). He also discussed two important architectural decisions that his company made when developing these products.

1. When dealing with local databases, Gianpaolo recommends using an Object-Relational Mapping (ORM) tool rather than the raw SQLite API provided by Google. The tool he used is OrmLite, which is Open Source like Hibernate but more lightweight and adequate for the simpler data structures needed for local storage on mobile devices.

2. When making REST calls to remote servers, he recommends using the Spring Android module with its REST Template in preference to raw HttpClient. He did encounter some bugs with Spring Android in the areas of cookie management and SSL certificates, but believes the product is rapidly maturing.

As a side benefit of both OrmLite and Spring Android being annotation-based, Gianpaolo was able to have a single set of domain objects. Each object had two sets of annotations applied, one for persistence and one for interfacing with REST services. [I'm a bit suspicious of the latter annotation, since it looks like the design combines domain objects and message documents into a common entity, a form of tight coupling I've long warned against.]

After a short break, we had our third talk of the evening by James Zaki, a freelance developer with goCatch. This was a high-level description of the goCatch app, which brings together cab drivers and prospective passengers. There was general satisfaction expressed around the room in favour of this cartel-breaking app, since the taxi companies and CabCharge engage in significant rent-seeking behaviour at the expense of both drivers and passengers. goCatch allows the two to bypass the middlemen. I had some reservations about one of the aspects of the goCatch design as described by James, i.e., its statefulness, which led to problems of synchronisation of state held on devices with that on the server. Perhaps a suitable set of messages based on idempotence could solve the problem. I didn't have time to discuss this offline with James.

Our fourth and final talk of the evening was by Darren Younger, CTO of IPScape, in whose offices the meeting was held. IPScape is a provider of cloud-based contact centre solutions catering to both voice and web. One of their interesting applications allows mobile device users to make phone calls not through the device's native telephony capabilities, but through the IPScape app. The server then initiates regular (teleconference-style) phone contact with both the caller and the receiver. The advantage of this is that the server can record the call. Many financial service providers are required by law to record all customer conversations, and it is easier for them to use this app rather than approach the telcos for a voice recording service. A developer API may be coming in a few months.

I also met another attendee, Nanik Tolaram, an amateur Android enthusiast with his own Android-related website.

I picked up a few useful tidbits of information over the course of the evening.

Samsung sent a couple of people to the meeting. They seem keen to understand the size and strength of the Android developer community. Samsung wants to carve out a unique niche even within the Android ecosystem and they have their own app store separate from the generic Android one.

DroidDraw is a testing tool for Android User Interfaces.

Google has a cloud-to-device API.

developer.android.com is Google's portal for Android developers.

Balsamiq is a commercial tool to mock up UIs, including mobile UIs. Pencil seems to be a good Open Source equivalent, and Lumzy is a free one.

Two of the presenters talked about their negative experiences with outsourcing. Although the outsourcing countries were as varied as Israel, India and Singapore, there seem to be some common problems caused by distance as well as the seeming cultural inability of some developers to look beyond the literal specification and to understand the higher abstraction that an application is trying to implement. Errors in the documentation of some specs led to literal implementation of those features even though they patently made no sense. Outsourcing sites like Freelancer.com seem very cost-effective, but the elapsed time to obtain a working solution negates those benefits. Examples: $200 and 4.5 months to develop an app, $800 and 9 months to develop another. The moral of the story seems to be to hire good local developers so communication problems are reduced and results are achieved quickly.

The Android Australia User Group is a good place for developers to hang out. The organisations that some of the speakers represented are looking for developers, and this may be a good way to get introduced.

Daylight Saving Time - Ubuntu Passes the Test

60 seconds after 0159 on October 2nd is when New South Wales, Australia mysteriously loses an hour to Daylight Saving Time and the time becomes 0300.

I was awake to record what our computers did. My wife's Windows 7 machine showed 0300 all right. I would have been surprised if it hadn't. Microsoft needs to show some degree of competence for all that market share!

What was personally gratifying to me was that my Ubuntu Linux machines also neatly skipped the hour and showed 0300.

I'm going to bed now with a smile.