For a long time, I’ve wanted to write something about the state of our software practices. It’s always proven quite challenging as I find myself unerringly drawn towards philosophy, creativity, engineering and a myriad of other voluminous subjects. Producing something succinct has proven consistently elusive. They say you can’t force these things and so it would appear as it’s taken some writing from Leslie Lamport to help me distil out some specific points that I want to make.
The article that started this chain of events is Blueprints in which Dr Lamport discusses the practice of coding. I found it thought provoking yet judging by the comments many felt it was irrelevant, out of date or simply wrong. Reading through those comments and a tweet discussion with Nic Ferrier led me to a bunch of observations which appear below.
Foundations
It appears that the focus on “practical” aspects of systems building (e.g. knowing how to code in popular industry languages rather than the fundamentals that underpin them all) has significantly impacted the corpus of common knowledge. Specification as a practice is not well understood:
- It can be formal or informal – ultimately the end-goal determines what is appropriate. Formal specifications provide the opportunity for proof and verification which in critical systems is highly desirable. The relevance goes beyond critical systems though to any situation where high confidence in a piece of behaviour is required.
- Specification in its various forms isn’t a theoretical exercise – there are a number of examples of its application in real systems. Google mention it in a variety of circumstances including Chubby and Spanner.
- Proving correctness can be done via formal specification, it cannot be achieved by testing. Imagine standing in a dark room with a pencil beam torch trying to establish what’s in the room, dimensions etc. This will take a long time and things are easily missed unless you cover the entire room with that pencil beam (which will take forever). Formal specs allow you to simply turn the light on in the room.
- BDD, TDD and the like are testing processes and thus cannot be directly compared with specification (certainly not the formal variety) which is a tool.
Abstraction
The ability to deal in abstraction is important for the disciplines of architecture and design. However there is a more fundamental need to satisfy when coding, the limits of the individual mind to retain and reason about detail. Our only tool for coping with systems of detail larger than can be held in an individual mind is abstraction. Abstraction creates coarser constructs that hide some of the detail and allow us to scale our reasoning to broader levels. It also makes it possible to communicate and test our reasoning with others.
Reading the responses to Lamport’s article shows that some of us are too literal in our interpretation not pausing to consider the more abstract possibilities. We are unbalanced in our view, focused on detail and specifics in a world where grey and uncertainty (a natural consequence of dropping detail for sake of abstraction) play a critical role. Some examples:
- Software systems have nothing in common with buildings. Consider for a moment the challenge of changing an old or large system to cope with a radical new requirement say going from single machine to massively distributed. Is that so different from taking an old Victorian-age school and putting in the trunking and cabling required for modern systems development? In both cases, there will be a desire for an understanding of the current structure (dare I say blueprints?), then some consideration of options, perhaps some testing out of tools and practices before actually doing the work (which undoubtedly will be iterative, component by component or room by room).
- Skyscrapers are big systems, toolsheds are small ones. Lamport himself states otherwise in the article: “While the specs I write are almost all informal, occasionally a piece of code is sufficiently subtle, or sufficiently critical, that it should be specified formally — either for precision or for using tools to check it. I’ve only had to do that about a half dozen times during the past dozen years.” He’s clearly talking about pieces of code, could be one method or a couple of classes or indeed entire systems.
Being too literal, ignoring the grey and reducing abstractions to strict constructs (ironic considering the vehement resistance to formal specification because it’s too constraining) has ramifications beyond design quality for aspects such as human communication, essential in any good team, agile or otherwise. We stop ourselves from considering the greater context, the bigger possibilities which might explain why some techies cannot easily relate to customer needs.
Research
A couple of Google searches reveals that Lamport has a body of work (including tools) related to reasoning about concurrent systems using specifications. This is notable because it focuses on the non-functional, something not often seen in discussions pertaining to TDD or BDD. For example, have you ever run across a test specification like this?
Will sort n items, distributed in any of the following orders (already sorted, exponential etc), in (n log n) time subject to the availability of memory being sufficient to hold 4 * n.
Returning to Lamport’s work, how often would you see any explicit treatment of concurrency or parallelism in applications of BDD or TDD? Isn’t consideration of these non-functionals relevant?
Some have argued that Lamport as the author has the responsibility for including all of this in his article. Really? Are we saying that an audience has a right to a complete, finished work that they can just apply verbatim, without thought or further development? Do all the best films have a definitive ending? Of course not, because there’s value in allowing a viewer to invent and go further.
Lamport opened the door to an opportunity for personal development and an improvement in the quality of one’s work, maybe some innovation too. Those who sought no further reading (a mere google away) and pronounced what he was writing irrelevant or covered by BDD or TDD have missed out.
These failures to dig deeper and put into context lead to stagnation of our practice (ironic given the focus on “practical” aspects such as coding). Research is essential to learning and growth but seemingly is becoming a lost art to many.
Wrapping Up
Lamport wrote in his article: “Few programmers write even a rough sketch of what their programs will do before they start coding. Most programmers regard anything that doesn’t generate code to be a waste of time.”
Meanwhile, Wired headed the piece with this statement: “With widespread access to free, online coding courses and tools, “coding” has become the new writing – the everyman’s skill.”
Given many of the responses to the article it seems that the readership proved Lamport right and Wired wrong. Not everyone possesses the skill to code competently and those that think it’s just about code are missing the key factors to make themselves so.