AI agents are increasingly being used in scientific research. To understand the strengths and weaknesses of these systems, it is critical to measure their abilities using rigorous benchmarks. However, most existing scientific benchmarks evaluate agents on relatively short tasks – consisting mostly of analyses that would take a human on the scale of an hour or so to complete. Here we introduce BixBench3, a benchmark that evaluates the ability of AI agents to carry out analyses on the scale of complete research studies.
We designed BixBench3 to mirror how scientists often work with AI agents today: the scientist chooses the research question and high-level methods, then delegates the implementation details to the agent. Accordingly, each BixBench3 task asks the agent to investigate a research objective by implementing a set of specified analyses following a general methodological plan.
BixBench3 tasks are based on published computational biology papers to ensure that they reflect real-world scientific research. The agent must build a working analysis pipeline starting from the paper’s raw data, which produces structured artifacts supporting its findings. Those artifacts – such as protein abundance matrices or tables of differentially expressed genes – are graded programmatically against reference artifacts from the published study, measuring how well the agent executed the required analyses.
We evaluated 13 frontier language models across 20 BixBench3 tasks. Their average scores ranged from 0.0 for Gemini 3.1 Flash Lite to 0.48 for GPT 5.6 Sol. These were long and computationally intensive runs: an average attempt took 6.8 hours, processed 102 million tokens, and cost $43 while the largest consumed 1.07 billion tokens, ran for 24 hours, and cost $525.
Our results suggest that AI agents are approaching the ability to complete computational biology work at the scale of entire research studies. Yet performance remained far from perfect and varied sharply between tasks. Agents could coherently carry out many analyses in sequence, with each step comparable in scope to the individual bioinformatics tasks that challenged models only a year ago, but still struggled to execute a specified analytical plan reliably from raw data to final outputs.
BixBench3 spans scientific domains and data types
The 20 BixBench3 tasks cover a range of biological data and scientific domains. Transcriptomic data appears in 16 tasks, epigenomic data in six, proteomic data in two, and genomic and microbiome data in one each. Six tasks combine multiple types of omics data within the same analysis. The scientific domains represented in BixBench3 broadly follow those found in recent bioRxiv papers.
The tasks are large in both their inputs and expected outputs. Each task contains 67 GB of raw data on average, with individual datasets ranging from 7 to 241 GB. Agents must produce 4–14 graded artifacts per task.
The artifacts capture successive stages of each analysis. Of the 138 artifacts, 56 are produced directly from the raw data, including outputs such as read-count matrices derived from raw .fastq files of RNA-seq data. Another 44 require a second stage of analysis, such as testing for differential gene expression, while 38 require three or more stages, such as identifying enriched biological pathways. Together, these artifacts form a dependency graph from the raw inputs to the most downstream results.
Each artifact is graded against the corresponding reference artifact from the published study, and overall task score is the proportion of artifacts that were faithfully reproduced.
(Left) Each benchmark task is constructed by decomposing a published study into a directed acyclic graph (DAG) of analyses producing data artifacts. Agent-produced versions of those artifacts are programmatically scored against corresponding published artifacts. (Right) Artifact-level pass outcomes for 13 models across 20 BixBench3 tasks, comprising 138 requested artifacts per model and 1,794 model–artifact evaluations. Filled squares denote passed artifacts, empty squares denote failed artifacts, and the bars on the right show each model’s overall BixBench3 score.
Performance varies sharply across tasks and analysis depth
Across the 20 BixBench3 tasks, GPT 5.6 Sol was the highest-scoring model with an average task score of 0.48 (i.e., 48% of possible points). Kimi K3 followed at 0.47, GLM 5.2 at 0.46, and Opus 4.8 at 0.46. However, these overall averages conceal large differences in where each model succeeded and failed.
Performance generally declined as agents moved further through the analysis pipeline. Across models, artifacts produced directly from the raw data scored 0.30 on average, while those requiring three or more stages scored 0.24.
Performance of different models on BixBench3.
Models with similar average scores often succeeded on different tasks, revealing distinct expertise and blind spots. GLM 5.2 and GPT 5.5 were separated by only 0.02 in their overall scores, but scored 0.5 and 0.25, respectively, on a task integrating ChIP-seq and gene expression measurements to investigate how Polycomb/RING1 affect gene activity in Drosophila.
Performance also varied across scientific domains and data sizes. Models scored highest on tasks with smaller datasets and on those focused on microbiology, evolution and ecology, and immunology; they scored lowest on tasks with larger datasets and on those focused on neuroscience or cell biology.
The most expensive models were not the most accurate
The average cost of a BixBench3 task varied 367-fold across models, from $0.35 to $129.14. Although cost and performance were generally related, several models offered different tradeoffs between accuracy and cost. GPT 5.6 Sol achieved the highest score, however Kimi K3 scored only 0.01 lower while costing 55% less.
Note: We accessed GLM 5.2 through OpenRouter, which routes requests across several companies serving the model. Each provider maintains its own cache, so a request sent to a different provider could not reuse context cached during an earlier request. As a result, GLM 5.2 achieved a cache hit rate of 41.4%, compared with 96.0% for the other models, and therefore an inflated cost. This lower cache rate reflects our OpenRouter configuration rather than an inherent limitation of GLM 5.2; keeping requests with the same provider could achieve a much higher rate. We therefore show both the amount we paid and an estimate of what the same runs would have cost at the average cache-hit rate of the other models.
Average cost per task for each model and the relationship between cost and task score.
Low-scoring agents often failed before completing the analysis
To understand why agents failed, we used a separate LLM judge to review each run’s methods report, execution trace, grades, and output files. The judge could assign up to ten failure-mode tags from a fixed list, including issues like incorrect methods, incomplete data, malformed outputs, premature termination, and repetitive retry loops.
The three highest-scoring models received 31–51 tags across all tasks, while the two lowest-scoring models received 102–117. Across models, the number of failure tags was strongly negatively correlated with mean task score (Spearman’s ρ = -0.92).
Premature termination and repetitive retry loops were most closely associated with low scores. These failures occurred approximately 2.0 and 2.2 times more often, respectively, in the 25% of runs with the lowest scores than across all runs. Environment setup failures, incomplete data, and synthetic or placeholder outputs were also more common among low-scoring runs. Among the 65 runs with the 25% highest scores, only one was tagged as ending prematurely and none as entering a repetitive retry loop.
Limitations
BixBench3 uses a strict grading scheme, comparing the artifacts produced by an agent with the corresponding reference artifacts from the published paper. This approach allows us to precisely compare the agent’s work with published results, but also engenders some limitations. This precision comes at the expense of flexibility; each task must specify the analyses and methods that should be used, otherwise a scientifically valid alternative could produce outputs that differ from the published reference artifacts and receive a low score. BixBench3 therefore evaluates how well agents execute a defined research objective and methodological plan, rather than whether they can decide which questions or analyses are worth pursuing. The benchmark is also subject to limitations inherited from its source papers, including any errors in the published artifacts or methods that may exist.
Implications
To our knowledge, BixBench3 is the longest-horizon biology benchmark reported to date, as measured by runtime and token usage. The strongest models scored almost 0.5 on average across tasks, meaning that they reproduced approximately half of the requested artifacts to such a degree so as to preserve the original scientific interpretation. This suggests that AI agents are approaching the ability to perform computational work at the scale of complete research studies, but important limitations remain.
Our results point to three axes that today’s frontier agents must improve on to reliably execute the study-scale analysis pipelines tested here: data size, analysis depth, and domain expertise. Agents scored lowest on tasks with the largest raw datasets, most sequential analyses, and in particular scientific domains. These results directly inform our work at Edison, where we are building Kosmos, which uses a world model to maintain coherence over long research horizons, and domain-specific models such as the chemistry-focused ether0.
Code to run the benchmark can be found in the BixBench3 repository, the dataset on Hugging Face, and further results on the BixBench3 benchmark page. The manuscript can be found here.