Method
When comma publishes a new driving model on openpilot master or Hugging Face (commaai/openpilot_driving_models), our evaluation runner scores it on a fixed suite and publishes its results page. Model pages also give paired differences from lebowski (the July 2026 877M model that runs on the truck) and from the preceding release.
The benchmark has two halves. The truck half runs recorded drives from a 2011 Mercedes G55 AMG in open-loop replay and closed-loop simulation; raw G55 drives stay private and their scores are published. The public half runs comma's public two-camera segments, whose per-segment results are published as CSVs (below).
What each section measures
| Section | Input | What is scored |
|---|---|---|
| Public segments, open loop | comma's public segments that have both road cameras, one segment as the unit, each car's own camera intrinsics. Segments where the car never moves are dropped. | The model runs on the recorded frames. Its outputs are compared with what the human driver did next: lateral error, speed error a few seconds ahead, and how fast the action changes. |
| Public segments, closed loop | The same kind of segments, run through comma's public worldmodel-4B with comma's ideal car, a vehicle that follows each command exactly. |
The model's own steering moves the car and the world model renders what the camera would see next. Scored as distance from the path the human drove. |
| Truck, open loop | Windows from the truck's recorded drives, fed to the model the way the truck runs it: the narrow camera, mounted 1.45 m up. A replay of lebowski on these windows matches the truck's live records (correlation 0.99 to 1.000 on the lateral action). | Speed and lateral error against what the truck did, and lateral gain: the model's lateral action over the truck's actual lateral motion, where 1 is right. |
| Truck, closed loop | The truck windows, run through worldmodel-4B with a simulation of the G55's steering and powertrain in place of the ideal car, starting from the truck's state in its CAN recording of the 2 s before the window. |
Distance from the driven path at 2, 4 and 7.2 s, and the RMS lateral and longitudinal jerk of the requests. 98 windows from 13 drives and 5 camera setups. |
| Truck, recovery tests | Closed loop after a disturbance: pushed 0.5 m sideways, or turned 2° off heading. A test behind a braking lead car runs on the 2 windows, from 1 drive, where a car ahead brakes. | How far off the driven path the model is 2 and 4 s after the disturbance, and how hard it steers to recover. |
Each section on a model page lists its own metrics, units, sample size and condition. A section that was not run for a model says so; it is never filled in from another model.
Unit of evidence and intervals
Every value is a mean over units: a 60 s segment in the public sections, a window (one stretch of one recorded drive) in the truck sections. Units are grouped by the drive they come from: comma's 39 moving public segments come from 23 drives, and the 28 closed-loop ones from 19. The interval beside a value is a 95% bootstrap over drives: resample whole drives with replacement, recompute the mean over all their units, repeat 4000 times, and keep the 2.5th and 97.5th percentiles. Three adjacent minutes of one drive therefore count as one draw, not three.
With fewer than 10 drives there is no interval. In simulation, a nominal 95% interval of this kind covers the true mean about 90% of the time at 10 drives and 79% at 4, and less when the data are skewed. The G55 open loop comes from 12 drives, the closed loop from 13 and the recovery tests from 13, so they carry intervals. The braking-lead test comes from 1 drive, so it reports the mean, the median and win counts, including the number of drives in which the model did better, and no interval.
A delta compares a model with a baseline on the units both were scored on. It is model minus baseline, computed per unit and then bootstrapped the same way, so differences between cars, roads and light that affect both models cancel. The fraction beside it counts the units where the model did better; for lateral action rate and the lead of the lateral request, which have no better direction, it counts the units where the model's value was lower.
A blue dot or a red diamond means the delta's interval excludes zero (a grey triangle, for a measure with no better direction); a ring means the data do not separate the two models; a dash means there is no interval. Each page shows dozens of these comparisons with no correction for multiple tests, so a delta whose interval only just excludes zero is weak evidence. Where one segment dominates a mean, the median beside it shows it.
Two measures need care. The lead of the lateral request is the shift, searched from 0 to 1.5 s, at which the model's lateral
request best correlates with the car's later lateral motion: a larger value means the request matches motion further ahead,
so it comes earlier. Smoothing a request also moves that peak, so the lead is not a timing measure on its own and has no
better direction. A value at either end of the search is only a bound and is left out, so its
count can be below the section's. The plan speed error at 3 s reads the model's plan output, which does not separate
models built on one frozen base: it is identical for the four October models on base f22e06e5.
Every public interval can be recomputed from the per-segment CSVs.
The site reports values; it does not rank models or combine tests into one score. Each test has its own table, every model's mean with its 95% interval and its median, and every table lists models newest release first. The site build checks every page and refuses to publish one that lists models in any other order, prints a place or a medal, or prints a combined score.
Why open loop and closed loop can disagree
Open loop asks whether the model's output matches what a human did on a recorded drive. The output never changes the next frame, so a model can copy the human closely and still drift once its own steering moves the car. Closed loop feeds the steering back through a world model, and small errors build up.
In our 2026-10-27 ablation study, open-loop and closed-loop results disagreed sharply. comma's recent models pair a frozen base policy with a reinforcement-learned residual head. With the residual head disabled, the frozen base achieved the lowest open-loop lateral error of any model on comma's public segments (0.074 m/s² vs lebowski's 0.109 m/s²), yet drifted 1.8 m off the driven path at 4 s in closed loop after a small yaw disturbance (where lebowski stayed within 0.4 m). Enabling the RL residual head pulled all nine closed-loop rollouts 1.2 to 1.3 m closer to the path while lowering open-loop lateral action rate by 33 to 40% and shifting lateral requests 0.13 to 0.26 s earlier. Open-loop imitation scoring counts both the earlier lead and the smoother steering profile as error against the human driver.
This matches published work: on the nuPlan benchmark, open-loop and closed-loop planning scores were negatively correlated (Dauner et al., 2023). The site shows every test side by side and does not weigh one against another.
A closed-loop example
One public segment in closed loop. Each model drove comma's ideal car for 7.2 s from the same recorded start while worldmodel-4B generated the camera view it saw. Recorded shows the drive as logged. Dashed yellow is the driver's path, solid is the model's own, the ring is where each is at 2 s. Distance from the driver's path at 2 s: lebowski 1.77 m, Mountain Dew 1.08 m, MD v3 / Flip Flop 0.62 m.
worldmodel-4B generates one 256×128 frame per 0.2 s step, 5 frames per second. The frames are shown at that resolution, scaled to the screen by the browser, with no frames added between them. The page draws the paths, rings and distances from the logged positions, moving between the 0.2 s steps.
Segment chevy_silverado_s1 from comma's public CI data, at 12.5 m/s on a left curve, chosen because it is the segment where these three models differ most at 2 s. On every other segment all three are within 0.35 m of the driver's path at 2 s. Seed 0 of the two the scores average, re-run to keep the frames; its paths are within 0.15 m of the scored run.
What the truck sections can and cannot show
They show how each model behaves on one truck: its 1.45 m camera height, its cameras and its steering response, on its recorded drives and in one simulator version.
- They come from one truck. Nothing in them says how a model drives another vehicle.
- The windows come from 12 to 14 drives of one truck. Each section shows its count, and the conditions (day, night, speed, camera setup) are not balanced.
- Others cannot reproduce them: the drives are private, and only aggregate scores and a few checked stills are published.
- Closed loop is a simulation. The world model is weakest at night, rolls out 7.2 s at most, and pulls late frames toward what was actually recorded, so differences that appear only past about 4 s carry little weight. In a separate test it showed 0 to 20% of an added lateral offset 3 to 4 s after it, so the front page shows 2 s first.
- No newer model has steered the truck. It drives on lebowski, and every other model is scored by replay.
Stills are a few camera frames per condition with the model's path drawn on them, each checked so that nothing in it identifies the truck, its owner or the place.
Benchmark revisions
A result belongs to a benchmark revision: the protocol (the windows, tests and metrics; geval-1 since
2026-10-27) and, for the G55 closed loop and recovery tests, the G55 simulator. Every results file records both, and the
site refuses a table that mixes two simulators. A new simulator means every model is rescored, so a change between revisions
is a change of the benchmark, not of a model. The G55 simulators so far:
g55-2026-10-26: G55 steering and powertrain simulation as fitted on 2026-10-26; each window starts steady at its recorded speed.g55-2026-10-31: G55 steering and powertrain simulation with the steering centre and understeer as refitted by 2026-10-31; each window starts from the truck's state in its CAN recording of the 2 s before it.
The log lists each change with its date.
Harness check on comma's public segments
The same harness runs every release on comma's own public drives, from other cars. These numbers are not about the G55: they show the harness gives sensible results on the data comma trains on, and anyone can rerun them from the public repository.
comma's public segments, closed loop
comma's public two-camera segments in worldmodel-4B with comma's ideal car, a vehicle that follows each command exactly.
comma's public segments, open loop
comma's public two-camera segments replayed through each model, against the recorded car's lateral acceleration.
Downloads
One results file per model, in the format described in schema.txt, and one CSV per public section with a row per segment. G55 sections publish scores only.