When the buyer can run the eval, the demo theater ends

The software demo is a performance, and everyone in the room knows it. The salesperson drives. The data is clean because someone cleaned it last night. The path through the product is the one path that works, walked a hundred times before you saw it. The hard question you actually have, will this survive my messy data, my weird edge case, my Tuesday, is the one question the demo is structured to never reach.

We all agreed to pretend otherwise. We sat in the conference room, watched the curated path light up green, and called it evidence.

It was never evidence. A demo proves a curated happy path on the vendor’s data, on the vendor’s schedule. An eval proves the real one, on your data, with your edge cases, in front of you. When the buyer can run the eval, the theater ends.

That sentence is the whole post. The rest is why the demo was always theater, what replaces it, and why this matters more for AI software than it ever did for the old kind.

Why the demo was always a magic trick

Start with what a demo actually is, mechanically. It’s a sequence of inputs the vendor chose, run against data the vendor controls, narrated by the person whose job depends on you saying yes. Every degree of freedom in that sentence belongs to the seller.

The naive read is that a demo shows you the product. It doesn’t. It shows you the best possible run of the product, the one the vendor rehearsed, on the inputs the vendor knows it handles, skipping the inputs it doesn’t. A demo is the product’s highlight reel narrated as if it were the game.

And we knew. The honest buyer has always carried a silent discount: subtract thirty percent for the demo glow, assume the real thing is rougher. But that discount is a guess. You don’t know if it’s thirty percent or ninety. You don’t know which thirty percent, is the rough part the thing you’ll use daily or the thing you’ll touch twice a year? The demo gives you a number you can’t calibrate and a confidence you didn’t earn.

For ordinary software, you could survive this. A CRM either has a field or it doesn’t; you could click around in a trial and find out. The demo was a sales aid, the trial was the real test, and the gap between them was annoying but bounded.

Then the software stopped being deterministic. And the gap stopped being bounded.

AI broke the demo, because AI doesn’t behave the same twice

Here’s the thing that changes everything, and it’s worth stating plainly. Old software did the same thing every time. You clicked the button, the form saved. The demo and your Tuesday ran the same code on the same logic; only the data differed. If it saved in the demo, it saved for you.

An AI feature does not work this way. Ask an agent to triage an inbox, draft a reply, reconcile two records, decide whether a refund is warranted, and the answer depends on the wording, the context, the ten things around the one thing, and a model that is allowed to be a little different each time. The demo showed you one run. Your Tuesday is a different run. The fact that the demo run was good tells you almost nothing about the distribution of runs you’ll actually get.

This is the trap that kills AI pilots. The demo dazzles. The buyer signs. Then the thing meets real inputs, the email written in three languages, the customer who contradicts themselves, the record with a typo in the field everything keys on, and the behavior the demo never showed is exactly the behavior production is made of. The vendor showed the one run. You bought the distribution.

A demo shows you one run. You’re buying the distribution.

So the naive defense, “let me try it myself in a trial”, helps, but only a little, and only if you happen to try the inputs that break it. You won’t, because you don’t know which ones those are. You’re a buyer poking at a product for an afternoon. The vendor spent months learning where it’s thin and has every incentive not to point you there.

What you need isn’t a longer demo or a hands-on trial. You need a way to ask the product the same hard question, the same way, every time, and to ask it on your inputs, not theirs. You need the thing engineers building these systems already use to keep them honest. You need an eval.

What an eval is, and why it ends the theater

An eval is the unglamorous core of how anyone builds AI seriously. Strip away the jargon and it’s three things: a set of real cases, a way to run the product against all of them, and a fixed rule for scoring each result the same way every time. That’s it. Cases, runs, a scorer.

The naive version of “testing AI” is to chat with it and form an impression. You try a few things, it feels smart, you nod. That impression is a demo you gave yourself, same flaw, fewer rehearsals. A handful of casual prompts can’t tell you a pass rate, can’t tell you which cases fail, and can’t be re-run identically next week to see if a new version got better or quietly worse.

The eval version is the opposite of an impression. Suppose you’re an accounting practice evaluating whether an agent can reconcile your month-end. The eval isn’t “watch it reconcile one clean ledger.” It’s: take fifty of your real reconciliations, the gnarly ones, the ones with the duplicate entry and the misdated payment and the line item nobody can explain, run the agent against all fifty, and score each against the answer you already know is right. Out comes a number. Forty-one of fifty correct, and here are the nine it missed, and here is exactly why each one missed.

Now compare what you’re holding. The demo handed you a feeling and a guess-discount. The eval handed you a pass rate on the cases you care about and a named list of every failure. One is theater. The other is a measurement.

And the asymmetry flips. In a demo, the vendor controls every input and you control nothing. In an eval you control the inputs, they’re your cases, your edge cases, your Tuesday, and the vendor controls nothing except whether their product survives them. The salesperson can’t drive. The data can’t be cleaned the night before, because it’s your data. The one path that works can’t be the only path tried, because you bring the paths.

The vendor can rehearse a demo. The vendor cannot rehearse your data.

That single shift, who picks the inputs, is the entire difference between a performance and a proof.

The buyer becomes the one who writes the test

Here’s the part that sounds like it should be hard and turns out to be the point.

For decades, the test that mattered, the one that decided whether software actually did the job, lived inside the vendor. Their QA, their staging, their internal benchmarks. You, the buyer, got a demo and a trial and a contract, and you took the rest on faith. The proof was on the seller’s side of the table, and you were asked to trust the report of it.

The naive future just moves the demo online: a fancier sandbox, a self-serve tour, a chatbot that demos itself. Same theater, new venue. The vendor still picks the path; you still watch.

The real shift is that the test moves to your side of the table. You arrive at the evaluation not with a wish list of features but with a folder of cases, the real situations your business runs on, with the answers you already know. The product’s job is no longer to look good for forty minutes. Its job is to survive your folder. Pass rate, by case, re-runnable next quarter to prove it didn’t regress. The buyer stops being an audience and becomes the author of the test.

This is why AI buying will not look like software buying. The thing that made the demo necessary, that you couldn’t see how it’d behave on your reality until you’d already paid, is exactly the thing an eval removes. You see the behavior on your reality first. The signature comes after the number, not before it.

The vendor who fears this is telling you something

There’s a tell worth naming, because it’s the most useful signal a buyer will get this decade.

Ask a vendor to run their product against your cases, scored your way, in front of you, and watch what happens. The vendor who has built something real says yes, send the cases. They’ve already run a thousand evals harder than yours; one more is Tuesday. The vendor who’s been living on the demo glow flinches. Suddenly there are reasons it has to be their environment, their sample data, a tailored proof-of-concept we’ll scope over six weeks. The flinch is the answer. A product confident in its distribution doesn’t need to control the inputs.

This isn’t hostility. It’s the same logic that makes a good engineering team trust a review more than a reputation: the proof should live in the test, not in the claim. A vendor offering to be evaluated on your terms is offering you the thing the demo could never give, a number you didn’t have to discount.

The ones who built on theater will resist hardest, and they’ll resist with process: more meetings, more curated environments, more reasons the honest test is “premature.” The ones who built on something real will hand you the keys, because the eval is where they win.

The turn: stop being the audience

Here’s the part that isn’t about software at all.

For your whole career, buying tools has cast you as an audience. You sat in the room and watched someone else’s rehearsed run and were asked to feel convinced. The most important decision, will this actually work for us, was made on a feeling manufactured by the person selling. You were a spectator at your own purchase, and the discount you carried in your head was the only power you had, and it was a guess.

That role was never a good fit for you. You’re the one who knows where your business is messy. You know the customer who contradicts themselves, the ledger line nobody can explain, the edge case that shows up every quarter and breaks everything. That knowledge is the most valuable thing in any evaluation, and the demo was built to keep it out of the room, because the moment your hard cases enter, the highlight reel ends.

The eval invites them in. It takes the one thing you have that the vendor doesn’t, the truth of your own reality, and makes it the test. You stop clapping for the run someone else chose and start grading the runs you chose. The decision moves from a feeling you were handed to a number you produced. That’s not a smaller job. It’s finally the right one.

---

That’s part of what we’re building toward at Apollo, software you don’t have to take on faith, that proves itself against your reality before you commit to it, because an AI-native company runs on evidence, not on the glow of a good demo. If you’ve ever signed off the strength of a beautiful demo and watched the real thing arrive rougher, you already know why the next era of buying belongs to the person who writes the test.