What we read this spring
Five papers that shaped how we think about agent evaluation.
Short annotations, kept honest. These shaped our thinking; none of them should be read as endorsements of every choice they make.
- ReAct: Synergizing Reasoning and Acting in Language Models (Yao et al., ICLR 2023). The framing everyone builds on. We keep returning to its examples of reasoning traces that look right and act wrong.
- AgentBench: Evaluating LLMs as Agents (Liu et al., 2023). Early proof that aggregate benchmark scores hide wildly uneven per-environment behavior. The per-task breakdowns matter more than the headline number.
- SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (Jimenez et al., 2023). A lesson in grounding evaluation in artifacts that already exist. Its test-based checking shaped how we think about automatic verification.
- GAIA: A Benchmark for General AI Assistants (Mialon et al., 2023). Questions humans answer easily and models do not. Useful reminder that difficulty is not monotonic across audiences.
- OSWorld: Benchmarking Multimodal Agents on Open-Ended Tasks in Real Computer Environments (Xie et al., 2024). Ambitious environment design. Reading their setup notes taught us as much as the results.
The pattern across all five: the interesting content sits in appendices and failure cases. Headline tables tell you who won a snapshot; the setup tells you whether the game was worth winning.