Normalize 384 samples, combine them into 48 pools, and prepare two aliquots per pool on a simulated Hamilton STARlet.
Samples often need to be brought to a common concentration before they can be combined for downstream analysis. At plate scale, this requires coordinating many transfers while preserving sample identity, mixing thoroughly, and using the instrument efficiently.
Agents control the simulated liquid handler through VENUS, PyLabRobot, or direct firmware commands. They normalize and mix the samples, assemble and mix the pools, and dispense the final aliquots. The task tests accurate execution of a large liquid-handling workflow, with scoring based on completed milestones and instrument execution time, excluding time spent reasoning.
Scoring
Reach all six required milestones. Instrument speed does not affect Pass@1.
RESULTS
Normalize 384 samples, combine them into 48 pools, and prepare two aliquots per pool on a simulated Hamilton STARlet.
Model
Pass@1
Cost ($)Model/API inference cost only, averaged per task attempt. Lab and labor costs are not reported. Repeats are averaged using the same task and family weights as scores.
GPT–6 Astraxhigh
100.0%
$38.33
Claude Fable 5.1xhigh
0.0%
$23.93
Claude Opus 5xhigh
0.0%
$50.73
GPT–5.6 Solxhigh
0.0%
$37.63
Grok 4.6xhigh
0.0%
$91.51
Gemini 3.8 Flashhigh
0.0%
$18.13
0%50%100%
Model trajectories
One example per model.
Run grading
✓ Pass
Measurement
Recorded result
Outcome
Score
10.0 %
✓ Pass
Pass condition: Reach all six required milestones. Instrument speed does not affect Pass@1.
✓
Samples normalized
at 2740.844 s · 785 errors · machine time 129.274 s
✓
Samples mixed
at 2740.999 s · 0 errors · machine time 129.297 s
✓
Pools assembled
at 2741.173 s · 0 errors · machine time 129.339 s
✓
Pools mixed
at 2806.744 s · 0 errors · machine time 144.438 s
✓
Aliquots made
at 2807.308 s · 0 errors · machine time 144.548 s
✓
Finished
at 2807.485 s · 0 errors · machine time 144.634 s
✓
Final state
balances correct; plates seated and tools parked
✓
Record
complete
✓
benchmark environment
registered
✓
golden method
six sealed files match
✓
agent boundary
all STARlet work is in interval
✓
time budget
2640.74915846996 / 7200 s
✓
recorded evidence
complete
✓
orchestration
complete
Measurement
Recorded result
Outcome
Score
23.6 %
× Fail
Pass condition: Reach all six required milestones. Instrument speed does not affect Pass@1.
✓
Samples normalized
at 5707.783 s · 22 errors · machine time 22.680 s
✓
Samples mixed
at 5708.049 s · 0 errors · machine time 22.709 s
✓
Pools assembled
at 6580.255 s · 4 errors · machine time 50.689 s
✓
Pools mixed
at 6591.067 s · 0 errors · machine time 52.474 s
✓
Aliquots made
at 6591.972 s · 0 errors · machine time 52.581 s
×
Finished
not reached
×
Final state
final conditions not met
×
Record
VENUS method markers are out of order; VENUS method markers are out of order; VENUS method markers are out of order; VENUS method markers are out of order; VENUS method markers are out of order; VENUS method markers are out of order
✓
benchmark environment
registered
✓
golden method
six sealed files match
✓
agent boundary
all STARlet work is in interval
✓
time budget
5197.557809133999 / 7200 s
✓
recorded evidence
complete
✓
orchestration
complete
Measurement
Recorded result
Outcome
Score
17.8 %
× Fail
Pass condition: Reach all six required milestones. Instrument speed does not affect Pass@1.
✓
Samples normalized
at 5735.777 s · 112 errors · machine time 36.716 s
✓
Samples mixed
at 5735.777 s · 0 errors · machine time 36.716 s
×
Pools assembled
not reached
×
Pools mixed
not reached
×
Aliquots made
not reached
×
Finished
not reached
×
Final state
final conditions not met
×
Record
VENUS method execution is incomplete
✓
benchmark environment
registered
✓
golden method
six sealed files match
✓
agent boundary
all STARlet work is in interval
×
time budget
7200.0001968 / 7200 s
✓
recorded evidence
complete
×
orchestration
TimeoutError: agent exceeded the wall budget
Measurement
Recorded result
Outcome
Score
0.0 %
× Fail
Pass condition: Reach all six required milestones. Instrument speed does not affect Pass@1.
×
Samples normalized
not reached
×
Samples mixed
not reached
×
Pools assembled
not reached
×
Pools mixed
not reached
×
Aliquots made
not reached
×
Finished
not reached
×
Final state
final conditions not met
×
Record
VENUS method execution is incomplete; VENUS execution has no task firmware to correlate
✓
benchmark environment
registered
✓
golden method
six sealed files match
✓
agent boundary
all STARlet work is in interval
×
time budget
7200.000125848004 / 7200 s
✓
recorded evidence
complete
×
orchestration
TimeoutError: agent exceeded the wall budget
Measurement
Recorded result
Outcome
Score
0.0 %
× Fail
Pass condition: Reach all six required milestones. Instrument speed does not affect Pass@1.
×
Samples normalized
not reached
×
Samples mixed
not reached
×
Pools assembled
not reached
×
Pools mixed
not reached
×
Aliquots made
not reached
×
Finished
not reached
×
Final state
final conditions not met
✓
Record
complete
✓
benchmark environment
registered
✓
golden method
six sealed files match
✓
agent boundary
all STARlet work is in interval
✓
time budget
5641.659670982999 / 7200 s
✓
recorded evidence
complete
✓
orchestration
complete
Measurement
Recorded result
Outcome
Score
0.0 %
× Fail
Pass condition: Reach all six required milestones. Instrument speed does not affect Pass@1.
GPT–6 Astraxhigh
Claude Fable 5.1xhigh
Grok 4.6xhigh
Gemini 3.8 Flashhigh