Demos are not evidence

Every lab has one: the demo video where the agent glides through a task, opening the right files, filling the right fields, never once clicking the wrong thing. These videos are fun to make and useful for recruiting. As evidence, they are close to worthless, and it is worth being precise about why.

A demo is a single run, selected after many runs, presented without its failure rate. The viewer sees the outcome and infers a capability. But capability lives in the distribution, not in the best sample. An agent that finishes a form correctly one time in twenty looks identical to an agent that does it nineteen times out of twenty, if you only show the one.

What we want to see instead

Nothing exotic. A table of tasks, a number of attempts each, the share that finished, and the failures described plainly enough that someone could reproduce them. Add the prompt, the model version, and how the tasks were chosen. That is most of what a scientific reader needs.

• How many runs behind each number?
• Who picked the tasks, and before or after seeing results?
• What did failure look like when it happened?
• Could a stranger rerun this and land near the same numbers?

We are writing this partly at ourselves. It is tempting to cut the tape where the run went well. The norm we are trying to set while we are small: demos live on a separate page from evidence, and nothing crosses between them without a task list attached.