Building a small task suite for document workflows
Scope notes for our first evaluation artifact.
Our first evaluation artifact is deliberately narrow: routine document workflows. Filing an expense against a policy, reconciling an invoice line with a purchase order, drafting a reply that cites the right clause. Tasks of this shape are common, checkable, and boring in the way that makes measurement meaningful.
What is in scope
- Single-document comprehension with verifiable answers
- Two-to-three step workflows across a small fixed file set
- Checks that a script can run, not a human judge
What is out, and why
- Open-ended browsing: impossible to pin down deterministically at our size
- Long-horizon planning: worth studying, but not until shorter tasks are solid
- Anything scored by vibe, including ours
The suite will be small, maybe forty tasks at v0, and that is a feature. A suite small enough to read end to end can be argued about line by line, which is where evaluation actually improves. We will publish the task files, the checker scripts, and the choices we rejected, before publishing any model scores.