The perfect shot
Part 2 of 3: what you get from the demo and what you don't
This is the second part of a three-part mini-series.
Read Part 1 here
In film, the shoot becomes the easy part when everything else is done. It’s an absolute breeze compared to post-production. The edit suite is where the film moves from content to product: the cut, the grade, the sound, the thousand invisible corrections that turn footage into something an audience will sit through to the end. The hardest parts of software development happen in the equivalent of the edit suite. The earlier parts, getting to the prototype, are like creating a storyboard.
A demo answers one question. Does this idea have potential? If we take SVPG’s four big risks of feasibility, viability, user value and usability, the demo helps to address usability, and potentially feasibility. The biggest risks of viability and user value are still out there. Can you turn this into a live, scalable product? Will users get sufficient value out of it for you to make returns on your investment?
The chemistry read
The ability to generate a high-quality, high-impact prototype accelerates the feeling that you’re on the right track. People leave the demo believing they are most of the way to delivering a product that survives contact with the world. The distance, though, hasn’t narrowed.
Scalability, security, reliability are all bills that are yet to fall due. The customer who does something you never anticipated with the product hasn’t appeared on the horizon. What has changed is that the demo used to be more obviously unfinished, so you could feel the weight of what was still coming. Now it feels done, and the remaining work stops feeling pending, or at least seems heavily discounted.
Then it arrives, unannounced, anyway.
A demo is a chemistry read. Two actors read a single scene, the room feels a spark, and the spark is real. But it cannot tell you whether the film gets financed, whether the performances hold across two hours and a difficult shoot, whether anyone will sit through the whole film. Everything that makes it a film is still ahead.
When the product is the model
And here the hard work may change, because we’re now building different types of products, and only one of them maps onto the way we have done things before.
If you are developing a traditional software product, you’re building something deterministic. You can build the way you always did, or use AI to accelerate. You test it the way you always did whether using AI or not. Your tests are there to ensure it does what you told it to, or it does not, and you can prove which, and the tests will return the same results until something in the code changes.
In traditional software, a green test stays green; one passing run is a promise about every identical run to come. A model makes no such promise. Its output is a sample from a distribution you have not measured, so a passing run tells you the distribution is not empty, and nothing else. A demo that works is evidence the way a single coin flip is evidence about a coin: it proves heads is possible, and tells you nothing about how often it lands tails, or which inputs send it there. The failures that cost you aren’t yet visible: the inputs you would never have thought to try, and the cases where the model is fluent and wrong at once.
In a hybrid, where AI is being embedded in a traditional app, the old rules still hold everywhere the code is deterministic; the new ones take over the moment the path runs through the model. A model in the middle of deterministic logic is not half the problem; it is a new one, because the code downstream cannot tell a right answer from a plausible one, and acts on both the same way.
So the hard part moves. Deterministic testing asks whether the code does what you intended, and ends in a green light. Evaluation asks whether the model is good enough, and how badly it fails when it is not, a question with no green light at the end. The work becomes the harness that scores it, the guardrails that fence it, the monitoring that catches it drifting; most of the engineering is now around the model, not in it. An AI product is not a reel you can print and screen unchanged. It is a live show that varies with each performance. It needs a stage manager standing in the wings because there is no take you can lock.
And the same system has to be able to say why it did what it did. A model returns an answer, not a reason, and “the model decided” is fast running out of road as an explanation. A lender in the United States has not been permitted to hide behind an algorithm too complex to interpret since 2022; from August 2026 the EU gives anyone subject to a high-risk AI decision a right to a clear and meaningful explanation of it. So alongside proving the product behaves well enough, often enough, you have to be able to reconstruct after the fact why it reached the conclusion it did on the one occasion that mattered, for a customer, or for a regulator who will not be impressed by a shrug.
That work is nowhere in the demo. It needs to be in the product.
From chemistry read to premiere
The demo makes people believe, and the belief is valuable. It can help guide the team through the path to ten thousand concurrent users, all interacting with the system in different ways. It becomes dangerous when it allows people to imagine that they’re at a premiere rather than a chemistry read.
Pinocchio didn’t become a real boy overnight. He had to put in the work to learn how to exist in the world.