The Difference Between Knowledge And Wisdom

I realised today that I have become wise. I'm writing this out of a sense of amusement rather than vanity.

I had a discussion with a client about some changes she wanted made on an application. I realised even as she was speaking what changes I would need to make to the underlying database in order to display the results she wanted on the screen. After that conversation, I sat down to write out the SQL script that would make the required changes to the database.

But I didn't run it.

And at that moment, I realised the difference between knowledge and wisdom.

Knowledge refers to the capacity to understand how to implement something.

Wisdom is the meta-knowledge, born of years of experience, that it's rarely that straightforward.

It's the experience of receiving another email or phone call from the client within a few hours saying, "Hey Ganesh, I just discussed this with the team, and we realised that we need the system to do something more. So when you implement your change, can you please ensure that it also does X?"

It's the experience of looking at a script I wrote earlier and thinking, "Hey, why am I doing it like this? I can do it much more simply in this other way!"

And then I have to change the script I initially wrote, and undo all the changes I had made earlier.

That's really what's changed in me. I knew SQL even 25 years ago, and I could possibly have come up with the same script even back then.

That's knowledge, and I haven't significantly added to that knowledge in so many years.

The difference is that 25 years ago, I would have implemented that change immediately.

Today, I have a voice inside me that says, "Just wait. Save this script in a file and look at it again tomorrow."

That's wisdom.

Death of the Directory

The LDAP directory is a strange beast. It's a data store, but it's hierarchical in nature, rather like the old hierarchical databases most of us only read about in database textbooks. Hierarchical models, as we know, have inferior expressive power when compared to network and relational models. So shouldn't LDAP directories suffer from similar limitations?

It turns out that they do have similar limitations, and the industry as a whole has been going along with the convenient illusion that the emperor has clothes.

Talking to some old industry hands, I became gradually convinced of something that I've suspected for a while - that the LDAP directory was an invention that solved a temporary problem. The problem has gone away, but the awkward solution remains with us.

Twenty years ago, relational databases were relatively slow and cumbersome. Even reads were slow. Relational databases were therefore not a good fit for read-mostly use cases.

Enter the directory server. Directory servers were designed to be very fast at lookups. They were abysmally slow at updates, but that didn't matter. They were designed mainly for lookups, not for updates. Systems requiring on-line transaction processing needed relational databases.

Somewhere along the way, relational databases got faster. Much faster. Faster at reads as well. And somewhere there, LDAP directories lost their raison d'etre. Only the world never noticed.

Today, LDAP directories do some things well - a very small number of things. They perform authentication very well. Checking a user's credentials (an encrypted password, for example) is a directory server's bread-and-butter. Plus it can enforce password policies, lock accounts on a certain number of failed attempts, etc. These are all built-in features, whereas relational databases are too general-purpose to be able to do these out of the box.

But relational databases are excellent at modelling complex relationships. Directories suck at this. If we try and shoehorn complex structures into a hierarchical model, we will end up with highly unsatisfactory results. Not only will it be inelegant and difficult to understand, the resulting directory bloat will also slow down the system, negating the performance advantage that prompted the choice of the directory in the first place.

The bottomline is, anyone looking to implement a "directory" today would be better off with a hybrid model. They should use an LDAP directory for the limited tasks which it is good at (authentication), and should use a relational database to model and store all other information. They then gain the best of both worlds. The directory server will hold just a handful of attributes needed for authentication.

Hybrids are all very fine, but how do we bridge between a relational database and an LDAP directory? In other words, what do we use to link a record in a database with the corresponding entry in the directory? One candidate would suggest itself to anyone who has been following this blog lately - the Universally Unique ID (UUID).

Of course, most relational databases don't support a native UUID datatype. And many DBAs are used to having identity columns for tables, which are autogenerated by the DBMS. In such cases, the UUID could be another attribute of the same table with a unique constraint defined on it. In other words, the UUID is another candidate key, and this is the one that is used to link an entity within the database to corresponding attributes in other systems. It is truly universal and unique.

How do we provision databases and directories so that their records are consistent? There may be no theoretically satisfactory answer (such as a two-phase commit transaction across both), but there are a number of real-world solutions that are adequate. Organisations may find that the hybrid data store is a pragmatic approach after all.