Benchmarks
Qiskit HumanEval
Nala's score on IBM's public benchmark for writing Qiskit code, and exactly how we measured it.
85 of 151 tasks passed (56.3%)
75 of the 106 tasks that can pass in our sandbox (70.8%)
What this benchmark is
Qiskit HumanEval is a public benchmark of 151 short Qiskit programming tasks, each with its own correctness test. It measures whether a model can write quantum code that actually runs and gives the right answer, not how fast or how elegant that code is.
How Leona ran it
We ran Nala the same way a user does: the production model, one attempt per task, through Nala's full plan, generate, run and review loop. A task counts as passed only if Nala actually delivered code that passes the benchmark's own test. A candidate that never reached delivery is scored as failed, whatever it contains.
Why 106, not 151
The sandbox that runs generated code has no internet access, on purpose. 45 of the 151 tasks' own reference solutions need something the sandbox blocks: 44 need IBM's cloud runtime library, and 1 more needs to write a file to disk. Those 45 cannot pass no matter how good the generated code is, so 106 is the highest score physically possible in this sandbox.
Before we spent anything
We ran two free checks against the harness itself first: submitting each task's own correct answer, and submitting deliberately wrong code. The correct answers passed 106 of 151 (106 is the same 106-task ceiling above); the wrong code passed 0. Both matched what was already on record, so the harness had not drifted before we spent anything.
Two runs
The first run measured Nala before the review-step fix (PR 1010). We re-ran the full benchmark after the fix merged, so the table below shows what actually changed, not what we expected to change.
| Run | Date | Model | Passed | Of the 106 gradable | Spend | Wall time |
|---|---|---|---|---|---|---|
| First run, before the review-step fix | 2026-09-23 | deepseek-v4-pro | 68 / 151 | 64 / 106 | $6.47 | 2.46h |
| Re-run, after the review-step fix (PR 1010) | 2026-09-25 | deepseek-v4-pro | 85 / 151 | 75 / 106 | $4.81 | 1.84h |
What changed: PR 1010 fixed three checks in Nala's review step that were treating delivered-but-correct code as failed for reasons that did not apply to how the task was written; a real failure still blocks delivery after the fix. The score moved from 68 of 151 (45%) to 85 of 151 (56.3%), and from 64 to 75 of the 106 gradable tasks. Each run is measured once, so part of that change may be sampling rather than the fix itself.
Notes on each run
- First run, before the review-step fix: Nala's review step was rejecting some code that had already passed the benchmark's own test, before it ever reached a user. PR 1010 fixes that.
- Re-run, after the review-step fix (PR 1010): After the fix to Nala's review step (PR 1010); 10 of the 45 tasks the sandbox blocks by design passed through the model's own workarounds.
What this score does not show
- Each score above is from one run. Run-to-run variance for this benchmark is not measured, so repeating either run could land on a different number.
- Model output varies between runs of the same code. In one internal check, a task that had passed in an earlier run failed on a later run with nothing about the code changed.
- 106 is the highest score physically possible in this sandbox, not the highest score physically possible for the model. See "Why 106, not 151" above.
How we measured it
The code that runs this benchmark is part of the public repository.
View the harness on GitHub