BixBench3

Benchmarking AI agents on research-study-scale computational biology tasks

BixBench3 evaluates the ability of AI agents to carry out analyses on the scale of complete research studies. Each task mirrors how scientists often use agents today: the scientist sets the research objective and a general methodological plan, then delegates implementation to the agent. Each task is based on a published computational biology paper. The agent must build a working analysis pipeline that analyzes the paper's raw data and produce structured artifacts supporting its findings. Those artifacts – such as protein abundance matrices or tables of differentially expressed genes – are graded programmatically against reference artifacts from the published study, measuring how well the agent executed the required analyses. We evaluated 13 frontier language models across 20 BixBench3 tasks encompassing the generation of 138 unique artifacts. Their average scores ranged from 0.00 for Gemini 3.1 Flash Lite to 0.48 for GPT 5.6 Sol. These were long and computationally intensive runs: an average attempt took 6.8 hours, processed 102 million tokens, and cost $43 while the largest consumed 1.07 billion tokens, ran for 24 hours, and cost $525.