Inside scientific benchmarks
BixBench asks 205 bioinformatics questions. On 61 of them the answer is a number, checked against a stated interval rather than judged for meaning. Those are the questions a paper is built on — and they are the ones where ARC pulls clearly ahead.
At Heureka we aim for best-in-class performance and continuous improvement, which requires measuring results in objective, scientifically rigorous ways. BixBench is an independent benchmark for computational biology. ARC is built for novel discovery that reaches beyond bioinformatics, but it is still useful to test it across these research applications. Below, we explore what that exercise found.
We compare ARC against Claude Code throughout — the strongest comparator we could run under an identical protocol.
Why these 61 questions
Most agent benchmarks published this year report one number per agent. That number mixes together questions with a verifiable answer and questions where a language model was asked whether two paragraphs mean the same thing. While performance on both matters, averaging them hides the objective measure that counts most for data-driven discovery.
BixBench labels every question with how its answer is checked. We did not invent this split and we cannot influence it — it ships with the dataset:
range_verifier— 61 questions. The answer is a number, and the grader is asked one mechanical question: does it fall inside the stated interval? No judgement about meaning is involved.str_verifier— 61 questions. Exact string match first, then an LLM judge (subjective).llm_verifier— 83 questions. An LLM judge decides whether the answer is equivalent (subjective).
This post is about the first group, because it is the slice where grading leaves least room for interpretation: the grader is comparing a number against an interval, not deciding whether two paragraphs mean the same thing. To test how much the grader matters, we graded all 205 questions twice, using two judges from different vendors: Opus 5 and GPT-5.6 Sol. The two judges agreed on 98.4% of verdicts, and ARC stays ahead on these 61 questions under both — the size of the gap moves, the direction does not.
The arithmetically-graded questions
Mean per-question correctness across all 61, three repeats per agent.
The lead is not an artifact of one configuration
A single significant result is a hypothesis. We ran the same 61 questions at two model tiers and with ARC configured two different ways, and the direction never changed:
Every configuration we ran
Accuracy on the same 61 arithmetic questions. Within each tier both agents run the same model.
The bottom row matters as much as the top one. Drop both agents to a cheaper, faster model tier and the gap persists: 40.7% against 36.9%. This is not one lucky pairing of harness and model.
Where the lead comes from
Broken out by the benchmark's own subject labels. We print the number of distinct datasets behind each row alongside the question count, because BixBench questions cluster — several often draw on the same underlying data, so a question count on its own overstates the evidence. One label is left off the chart: machine learning and AI, where neither agent answered its two questions correctly.
Arithmetic questions, by field
Accuracy on each subject label.
How this compares with what else has been published
BixBench has been run before. Phylo Bio's evaluation put Biomni Lab at 52.2%, Edison Analysis at 42.4%, Claude Code on Opus 4.6 at 39.5% and the OpenAI Agents SDK on GPT-5.2 at 38.5%. LiteFold report their own agent, Rosalind, at 51.4% alongside the same comparators.
Those numbers are useful as building blocks for understanding the landscape. But each is a single figure per agent, without a repeat count, an interval, or a breakdown by how the answer was graded.
Worth noting who did report repeats: the benchmark's own authors. The BixBench paper says so plainly — "to account for stochastic trajectories, we run each analysis in parallel five times to calculate overall performance" — and its appendix reports accuracy per question and per capsule across those replicates. The care was there at the source. What tends to get lost is everything downstream of it — by the time a result reaches a comparison table, five runs have collapsed into one number.
- Whether the gap is larger than the agent's disagreement with itself — here, 7–9% of questions flip between identical runs.
- Whether it holds on a held-out set, rather than only on the questions where the effect was first found.
- Whether it comes from broad capability or from a few questions sharing one dataset.
- Where an agent is actually strong: a headline that averages every grading mode hides a 7.7-point lead on the questions graded arithmetically.
The chart below is every BixBench result we are aware of, on the full 205-question benchmark rather than the arithmetic slice.
BixBench, everything reported so far
Full 205-question benchmark. Gray bars are figures published by others.
How benchmark performance was measured
Everything outside the two agents was held identical. Same Docker image built from the same base, same non-root user, same freshly-extracted workspace, byte-identical task prompt injected through each agent's own documented hook, and the same wall-clock stopping rule. The only intended variable is the harness.
Three repeats per question, because a single run is not a stable measurement: each agent disagrees with itself on 7–9% of questions across identical repeats. The figures published elsewhere do not report a repeat count — and 38 of the 59 datasets ship a solved reference notebook into the working directory, which we removed for this exercise.
Prompts and grading come from BixBench itself, generated verbatim from upstream and pinned by a test. Where a judge is used at all it receives the question, the reference answer and the response — nothing that identifies which agent produced it.
The answer key ships with the data. 38 of the 59 datasets include the solved reference notebook — CapsuleNotebook-<uuid>_executed.ipynb — inside the agent's working directory, and its cell outputs carry the answer. That includes every dataset behind the 61 arithmetic questions. Upstream's extraction removes it, but the filter is easy to weaken by accident: match the directory by exact name rather than substring, or glob only the root, and the notebook survives. Check the workspace is clean immediately before the agent starts rather than trusting extraction to have worked.
The arithmetic slice is the one where the judge has least room to interpret, as described above. Reported alongside these results: on the full 205-question benchmark, ARC and Claude Code finish within a point of each other, and the complete breakdown — every grading mode, every subject label, both model tiers, and a second grading pass by a different vendor's model — is kept with the data.
ARC is the engine inside Heureka Bench, and most of what makes it useful only exists there: the project context it reads on the first turn, the lab notebook it writes findings back into, the provenance trail, and the dataset cards. In this benchmark, none of that is present — ARC runs in a bare directory with its Bench-only tools switched off. We benchmarked the engine with the car removed.
Why ARC is built for biology
Biology does not run on plausible answers. It runs on numbers that are right — a p-value, a count, a coefficient, the figure a reviewer will pull on and the claim your name must stand behind. That is a different standard from code that compiles, and it is the one we build to. On the questions BixBench scores against a stated interval — the questions that look like the work required in computational biology — ARC leads by 7.7 points, more than doubles the accuracy on genomic variant analysis, and holds that direction at every configuration we ran.
A general coding agent is optimized for something that executes. A research harness has to be optimized for something that is true, and that you can still defend six months from now — which is why ARC records each step while it is doing it, and why it lives in a lab notebook rather than a terminal.