Off-centre
If we're all driving the same car, what's left to compete on?
An old word is taking on new meaning. As execution becomes commoditised through the use of LLMs, deciding what to do and how to do it becomes more important, not less. Taste is the word.
The revival of taste as a concept has been prompted by the success of LLMs in changing the way we approach work. As we outsource more of our planning and design to LLMs, so more of that work is going to tend toward the mean of the LLM’s training data.
The effect is measurable. A 2024 study found that while generative AI helped writers create better stories, the use of generative AI made outputs more similar to each other. The weakest writers saw the highest improvements in their writing as a result of using generative AI.
In other words, the beneficial effect that people feel of using LLMs is real, but so are the concerns that more LLM usage leads to more similarity, and a loss of originality in creative endeavour. Individual quality increases but collective variety falls. That’s the trade-off that LLM usage encourages, and it’s the one being made when organisations are focused on AI productivity gains, and people are being laid off so that more money can be spent on AI. The danger is that these companies are buying their way out of competitive advantage and investing instead in a new age of homogeneity.
The central reservation
For two decades, execution was a credible competitive advantage because it was expensive and unevenly distributed. Some teams shipped, most did not. Over time, “we can build it” became confused with “we will win.” But what happens when everyone can build it, and it’s almost the same thing?
Previous changes to software development focused on reducing execution costs. Offshoring, open source, no-code, all offered the promise of reduced execution variability while protecting your design. AI seems to offer more of the same.
However, if you’re using AI to compress your product development lifecycle, then you’ve moved beyond execution to design and planning. This may mean you speed up, but when everyone runs the same averaging engine, outputs converge. The thing that used to differentiate companies, the act of execution, has commoditised. What is exposed is the layer that used to be invisible: the choice of what to execute on.
If the engine tends toward the median, you can steer for the outlier. You have to be prepared to be the adversarial partner in the conversation. You need to continually challenge the LLM, enforcing your taste so the engine helps you produce a better result.
Stepping off the median
The choice of what to execute on is a bet, informed by data and by taste.
The best bets will be fuelled by the willingness to step off the central tendency, to be aware of the pull to the median, and to push back. A product manager might pull one of her tool’s suggestions and try the opposite out of curiosity. An engineer might refuse the first plausible answer because something in it feels wrong. A founder might insist on a thing nobody else thinks is necessary because she has lived with the problem long enough to know better. From outside, all three look like contrarianism. From inside, all three are simply paying attention. Paying attention isn’t the same thing as being right.
Another bet is to leave things un-built. When execution was expensive, the “do not build” decision was often made by capacity constraints as much as any customer data. Now that the price of execution is falling, the constraint is weakened, and the filtering has to be done by judgement. An agent will happily build anything you ask it to. You need to know what not to ask for.
Under the hood
Concerns about taste may lead an organisation to train a custom agent on internal data. The PRDs, the retrospectives, the wiki, the way teams document the way they work. The thinking is that the agent will produce outputs that look like the company instead of outputs that look like the internet.
This does not fix the problem. It creates a new one. Assuming the organisation has good documentation, then the LLM will produce a higher-fidelity average. It will reproduce with great confidence the way the company has done things; past-shaped answers to present problems. It will be on-brand and more competent at making the company’s previous output. What it won’t do is solve a novel problem in a novel way. Taste is forward-leaning. It bets on what has not existed yet.
If the organisation has poor documentation, then the internal agent is unlikely to be any better than a generic LLM anyway.
A useful version of the custom agent would do the opposite job. It would interrogate a draft. It would challenge the customer assumption and surface the unstated constraint. This is the agent worth building. The one that draws attention to the trade-offs being made. The agent encodes the rules of engagement. The operator gives the output its taste.
Speed limits
Whether driven by contrarianism or attention, any choice is a bet, and the number of bets that pay off is sobering. According to a review of published numbers, at Microsoft, A/B tests divide roughly into thirds: a third produce the expected result, a third move nothing, a third move metrics in the wrong direction. Google’s and Netflix’s published numbers are far worse (and probably more honest). Those odds don’t improve when the bet is the obvious one. The median choice fails as often as any other, and on the rare occasion it lands, it hands you an advantage everyone already has. It’s unlikely that the savings of faster execution will outweigh the costs of the wrong decision.
Many companies will spend the next few years optimising the cost of execution, congratulating themselves on the savings. The risk is that they will find their products becoming more interchangeable with their competitors’ as the moat of taste disappears.
Other organisations will look slower from the outside, and very different on the inside. They will be paying for taste, protecting it, hiring for it, building tools that help their operators refuse the obvious answer, and testing the bets that come out.
Now that execution is cheap, the price of not having taste could be existential.