Penn AI Seminar featuring Mayur Naik (University of Pennsylvania)

MathDuels: A Self-Play Benchmark That Grows

As frontier language models attain near-ceiling performance on static mathematical benchmarks, existing evaluations are increasingly unable to differentiate model capabilities, largely because they cast models solely as solvers of fixed problem sets. I will present MathDuels, a self-play benchmark in which models occupy dual roles: each authors math problems under adversarial prompting and solves problems authored by every other participant. A generation pipeline produces and filters the problems, and a psychometric model jointly estimates solver abilities and problem difficulties, from which author quality is derived. Experiments across 19 frontier models reveal that authoring and solving capabilities are partially decoupled, and that dual-role evaluation exposes capability separations invisible in single-role benchmarks. As newer models enter the arena, they produce problems that defeat previously dominant solvers, so the benchmark’s difficulty co-evolves with participant strength rather than saturating at a fixed ceiling. A public leaderboard at mathduels.ai is updated as new models are released.

Zoom