Obsessing Over Testing

Testing in many software development organisations, is the focus of complaints whenever something goes wrong. The discussion that follows such an incident tends to concentrate on answering a single question:

“Why didn’t our testing catch this?”

The assumption seemingly being made is that all possible issues can be caught via the method of testing. Yet that requires:

1. 100% functional coverage¹ from application all the way through the operating system and hardware. This is typically too costly to be viable in both resource and time for many organisations. Even with infinite resource and time, the human mind cannot concoct all the necessary test cases.
2. A complete replication of load patterns that will be experienced in real-world deployment. Assuming one can muster the necessary hardware to do this, one might be able to account for current load patterns but not the new ones created as the result of releasing new functionality.
3. Dead and live locks to be entirely deterministic such that they can be reliably reproduced.

Of course, testing isn’t the only means of validating a system’s behaviour prior to real-world use. One could apply formal methods for example.

In spite of all this, many organisations persist in the pursuit of testing as the sole means of achieving quality. Why?

I wonder if a key contributor might be the risk averse or fearful mindset that is so prevalent? For the risk averse, failure is unpalatable, to be avoided at all costs. For these folks, risk management is about complete elimination rather than constrained mitigation ².

Or maybe the linear-think we all indulge in to one degree or another likes the neatness and certainty that testing seemingly brings?

My own feeling is that a more productive mindset would be one that accepts failures will occur and sets about limiting the consequences. That is managing the downside that comes with being unable to completely validate our systems beforehand. One might call it an anti-fragile³ approach.

A good demonstration of this mindset at work can be seen in the recent tribulations of Philae. The team could have pronounced the mission a disaster and proceeded to critique their approach to delivering the systems. Instead, they set about making the most of the situation. They worked out means for maximising the experimental opportunities⁴ and, in parallel devised options for addressing the fundamental problem of landing in a place that limited the available solar energy.

Of course, the Philae team applied the mindset on the fly⁵. If we were to apply it to the practice of deploying new features, how might that look?

1. We’d put mechanisms in place to limit the cost of a failed release on future releases. We’d avoid rollbacks wherever possible preferring the use of feature switches and traffic routing.
2. We’d introduce swim-lanes to confine runtime damage wherever possible.
3. We’d focus on making it easy to capture as much runtime behaviour and state as possible for purposes of diagnosis and repair.
4. We’d adopt a practice of dark-launching. In this way, we could ensure customers don’t see the consequences of a failure.

All of these practices would allow us to forego the more burdensome aspects of pre-production testing such as load prediction and find out by experimentation in the real-world.

How might one persuade a risk-averse mindset to adopt this kind of approach? I’m not sure:

1. Experimentation requires that we accept unpredictable outcomes.
2. Whilst some organisations are using these practices, they are not well-known, they are thus new to many.
3. Implementing the infrastructure for these practices takes effort without relatively immediate and obvious benefit.

None of these things would make for appealing dialogue to the fearful. Perhaps it comes down to establishing a bottom-up revolution. A process of identifying potential rebels, sharing the ideas with them and enlisting the positive responders to introduce the practices one small step at a time. As the advantages become too obvious to ignore, others would want to know how they’re being achieved and then adopt it themselves.

No, it’s not going to be that easy is it?

Updated to include a reference to Mars Rover upgrade mechanisms.