Agile swear words is a semi-serious series about words that have lost their meaning or actively sabotage product management. Or words that just give me the ick.
We all believe in it
Everyone believes in quality. Who wants to give their customer a low-quality experience? Who wants to build rubbish software?
Quality is often invoked and rarely defined.
Someone will say, we have to invest in quality, and everyone will nod along, holding their own definitions in their head. When you ask for those definitions, it’s amazing how much they can vary. Quality is used as a catch-all, covering everything from test coverage to latency, from architectural choices to code linting styles. All of these are quality, and none of them are full definitions.
It’s so rare that a definition of ‘quality’ is actually written down, that it makes sense that people have these views. None of them are wrong, which isn’t quite the same thing as being right.
But we don’t know what it means
I’ve spoken with a number of people recently that say quality is about meeting the acceptance criteria for a piece of work. This aligns with SCRUM’s perspective on the definition of done. ”The Definition of Done is a formal description of the state of the Increment when it meets the quality measures required for the product.” This is problematic. Complete doesn’t mean the same thing as quality, and the “quality measures required for the product” gestures at what this means rather than defining it.
The definition of quality that matters is how the customer or user experiences your software. Scrum can’t measure this because it doesn’t reach out into when the customer is using your product.
Too often, the lens of quality is turned inwards. For example, code coverage, while an important protector of the customer experience, is not a predictor of it. Increasing coverage from sixty to eighty per cent gets announced as a quality improvement even though it says nothing at all about what any customer will experience.
As with the Definition of Done, we have to look beyond the immediate environs of the delivery team to arrive at a complete perspective on quality. The way teams think about and measure quality falls into three separate categories, each of which offers different insights. They shouldn’t all have one label.
Delivery quality
”DORA’s software delivery performance metrics focus on a team’s ability to deliver software safely, quickly, and efficiently.”. DORA established that the perceived trade-off between delivery speed and system stability is illusory. The best teams deliver more stable software faster. This demonstrates a quality of delivered software. It doesn’t necessarily mean the customer is getting what they want or need.
Correctness
Another measure of quality focuses on the correctness of the build. Defect counts and escaped bugs feature here. Review and testing cycles exist to prevent defects impacting the customer experience. It seems strangely self-congratulatory (and possibly a sign of perverse incentives) that companies would seek to minimise this number in the name of “quality.” A release free of defects does not guarantee a positive customer experience.
System quality
Non-functional requirements are often referred to as quality measures. Measures such as availability, latency, error rates, and resilience all impact how the customer experiences your software. These are genuinely useful, and organisations invest in Service Level Objectives (SLOs) and Service Level Indicators (SLIs) to track these. Everyone experiences latency and availability, but these don’t tell you if the software can do what the customer needs. Monitoring and observability are key to ensuring that any issues with the system are known and acted upon before they become customer-impacting. However, these measures are taken at the service boundary. A SRE can look at a completely green dashboard while a customer struggles with a poorly-designed workflow.
A good example of this is the UK government’s Verify programme, which was found to have been ”an innovative technical success and is performing to specification [but] it is not producing the promised benefits.” The project, which was supposed to enable people to verify their identities online, reported a success rate of 48% in 2019. This had been projected to be 90% in 2015. The government’s Universal Credit scheme was a key customer, but only 38% of Universal Credit claimants could successfully use the system.
More than capable
Delivery quality, correctness and system quality all name capabilities. These are not outputs that people shipped, but aspects of those outputs. People care about these aspects, which is why it’s easy to mistake them for “quality.” You could have all of them and still have failed your customer.
Here are some questions that can guide teams toward a higher-quality customer experience:
Did the capability solve the identified customer problem?
Is the customer satisfied with the capability and its cost?
Does the customer trust the capability?
Is there any risk or vulnerability associated with the capability?
Does the product stand up in conditions that nobody predicted or tested?
Precedence
The reason people measure capabilities is that capabilities are prerequisites for outcomes. You can’t get a good outcome without solid capabilities. But those capabilities don’t guarantee a positive outcome. These things shouldn’t be conflated, and too often they are. Too often, the outcome is not measured at all. It may not be instrumented, it may take too long to arrive. Structurally, a team may not be set up or incentivised to measure outcomes. Also, part of an outcome is qualitative. Customer satisfaction and trust can only be measured through surveys or interviews, not through system instrumentation. That doesn’t mean it’s not important, but it can seem esoteric to a delivery team.
The mark of quality
Quality is the one thing we all believe in but none of us define. That’s because we use the word to describe different things. We need to break out delivery quality, system quality and correctness. Before nodding along next time someone says “we need to invest in quality,” let’s find out what they mean.
A reminder
This is part of a semi-serious series of words that should be removed from the agile lexicon. Follow the links below for earlier entries.
The only f-word I won’t say: Agile swear words part 1
Commitment-phobia: Agile swear words part 2
My vendetta against the v-word: Agile swear words part 3
Let’s leave grooming at the altar: Agile swear words part 4




