Sunday, October 20, 2019

How To Understand Systems - My Secret Superpower

In Poul Anderson's SF story "The Problem of Pain", there is the following quote:

The primitive faiths see God or the gods, as power; the higher ones see Him as justice; the highest see Him as love.

There is also the quote attributed to Eleanor Roosevelt:

Great minds discuss ideas; average minds discuss events; small minds discuss people.

The common thread through both of these quotes is that there is a lower and a higher way to think, with some intermediate level too.

Over my 32-year IT career, I have come across three ways by which people tend to understand systems, and I would express my findings in an analogous manner:

The shallowest understanding of systems comes from a study of user interfaces; an intermediate level of understanding comes from a study of processes; the deepest level of understanding comes from a study of data.
Let me elaborate.

There are many people who work in the IT industry, and not all of them have undergone rigorous study or qualification in IT disciplines. Many have just drifted into it from other functions, and they often believe they can make a contribution based on their understanding of the field and of the software systems they are familiar with. Yet a lot of them exaggerate their competence, because one cannot understand IT systems by a superficial understanding of how they seem to work. There is a lot more to it than this.

The most common misunderstanding is seen in clerical-level operations staff who have only experienced IT systems as users. All they know of a system is its user interface. They see the system as a series of screens, with fields and buttons on each. If asked to describe the system, they will proceed to describe what a user will do on each screen - "Fill up this field, check this box, select a value from this dropdown, then click 'Next'. On the next screen, select this value, select from one of the radio buttons, then click 'Submit'."

Clearly, this level of understanding is superficial, and often misleading, when it comes to determining how to improve or extend a system to do something more.

Senior operations professionals and business people tend to understand systems a lot better than the clerical-level staff who have only experienced a system through its screens. They know what processes and process steps are being represented by those screens. This insight could result in suggestions for improvement of the system in various ways, since processes could be optimised over time. But processes too are limited in the insight they can provide into a system's design.

Many developers too are stuck at the level of understanding systems through process logic alone.

The ultimate level of insight into a system's design comes from an understanding of its data model. What entities are represented in the system? How are they related to each other? What constraints govern those relationships? In formal practice, this is known as Entity-Relationship Modelling.

The actual data stored in these data structures forms a secondary source of important information. The relative volumes of data in different data structures provide valuable information of their own, and usage patterns provide another dimension.

I identify most as a data analyst. When people describe systems to me, I mentally draw Entity-Relationship Diagrams. This has now become second nature to me. Unless I understand the entities in a system and their relationships, I'm not comfortable that I understand anything at all. As processes are explained to me, my understanding of them is formed in terms of the data entities being populated, updated, relationships established, etc. In other words, I see processes in terms of their effects on data. And when I see screens, I can see them as manifestations of process steps, just as business people can.

Three ways of understanding systems, from the lowest (most superficial and least effective) to the highest (most insightful and effective) - Click to expand

My focus on data is my secret superpower. It has helped me whenever I have been called in to consult on a problem. I have been able to diagnose problems within hours, sometimes within the first hour, where others have been stumped for days. I have been able to propose innovative and elegant approaches to enhance the capability of systems, without the elaborate and expensive projects proposed by others. When you can see data first and foremost, a process as something that changes data, and a screen as simply representing a step in a process, you can understand a system in its entirety. Your depth of vision is then so great, and your insights so profound, as to amaze coworkers who have not been similarly trained.

Yet it is not magic. It is very simple science. The focus on data - data modelling, data design - is fundamentally important to an IT professional. Alas, it is not more widely taught in the IT industry.