Log

Every note, release, test run and change to the tests since 7 Nov 2023, newest first. Also as an RSS feed.

November 2026

6
Test change
The site is renamed My Self-Driving G-Wagon and moves to https://my-self-driving-g-wagon.drose.io; openpilot-gwagon-evals.drose.io redirects there.
No value changed.
6
Test change
The Signals page lists every message in the DBC files, the steering-column message the gateway sends for cruise control included; the build no longer refuses command frame IDs.
No value changed.
6
Test change
The site is dense: one front page with the truck's parts and what is on each, every release's G55 results with intervals in one table, the findings, the open questions, the log, the signals and a glossary.
New: a page per part (the engine ECU first), the truck's parts list and a W463 CAN signals table built from the DBC files. The build refuses drive, window and session ids, host names, file paths, private IP and Bluetooth addresses, the VIN prefix, money, the owner's name and the gateway's command frame IDs (`geval.boundary`). No value changed.
6
Test change
The site is a log of the G55 work.
The tests are the three on the G55's own drives; comma's public segments move to a harness check on the Method page. The N16 x-ray on comma's public segments is taken down until it runs on the G55's own drives. Notes record the work behind the tests. No value changed.
5
Release
f22e06e5 uploaded to Hugging Face
G55 closed loop
14.1 cm (10.9 to 17.1)
G55 recovery tests
48.1 cm (39.1 to 59.6)
G55 open loop
0.181 m/s² (0.153 to 0.187)
5
Tested
1 model scored, suite geval-1
f22e06e5
5
Test change
The site follows the releases over time instead of reporting one snapshot: a timeline of every release by model family, a feed of releases, test runs and test changes (also as RSS), one page per test with every release's 95% interval, newest first, and one page per release.
A new section, Inside the models, shows which of the 32 squares in the N16 nav model's view move its steering on 4 of comma's segments, and a check of that map. No value changed.
4
Note
N16's frozen vision tokens do not replace a box detector
A head on N16's frozen vision tokens found 2.7 % of vehicles at IoU 0.5, against 55.5 % for YOLO26s on the same frames. Added to the box, the tokens gave a speed error of 3.88 to 4.44 m/s against 2.31 m/s for the analytical tracker.
4
Note
Camera position moves where N16 holds the truck in its lane
In simulation, moving N16's virtual camera sideways moves its held lane position by -0.97 m per metre of shift. A road-plane warp for a lens 0.239 m right of centre took back about half of the +0.191 m median shift in the hold and restored 0.394 of 1.0 m pushes against 0.197 uncorrected (base 0.509).
4
Note
A trained steering correction on N16 recovers more often in simulation
On 44 held-out simulator windows, a TD3-trained correction on N16 recovered from 1.5 m pushes 0.76 of the time against 0.58 for its untrained starting point (+0.18 [+0.13, +0.33]). Against the driver's recorded line the gain was +0.07 at 1.5 m and +0.01 at 1.0 m.
3
Note
The camera loop ran below 20 Hz because OpenBLAS threads spun
On N16 drives the camera loop ran 17.7 to 18.8 Hz: a serial decoder skipped ticks while OpenBLAS worker threads used 1.75 cores. With one OpenBLAS thread per service, three drives held 19.91 to 19.92 Hz.

October 2026

31
Note
comma's models turn the wrong way first because of the base policy
In 258 of 258 pushes where a full comma model first steered with the push, its residual-off copy did too. The residual removes 61-88% of these turns but sits at its 1.0 m/s² bound on the rest.
31
Tested
9 models scored, suite geval-1
f22b8b34, 6b5b705f, N16 nav model (96244b1f), MD v3 / Flip Flop (1365126a), Mountain Dew (8d736683), 870a4823, Cinque v3 (f78ed37d), Cinque v2 (1a421175), lebowski
31
Test change
The site is a plain results report: one table per test, each model's mean with its 95% interval and its median, newest release first.
The G55 score, the tiers on the page, the findings and the trend charts are gone, and the build refuses any list of models in another order and any combined score. Each results file now records its benchmark revision: protocol geval-1 and, for the G55 closed loop and recovery tests, the simulator (g55-2026-10-31 since the rerun below). In that rerun the comparison with the preceding release was missing for Mountain Dew and made against MD v3 instead of N16 for 6b5b705f, in the G55 closed-loop and recovery tables; both are corrected, and no value changed.
commit 7f1f94f
31
Test change
G55 closed loop and the recovery tests rerun for all 9 models on the current fit of the G55 simulation (steering centre and understeer refitted).
Each window now starts from the truck's state in its CAN recording of the 2 s before it instead of a steady start; one window of 98 has no valid CAN there and starts steady. Windows and tests are unchanged. Medians at 2 s are now 8.3 to 12.0 cm (they were 12.0 to 15.3 cm), and 8 of 8 models are closer to the path than lebowski with the paired interval below zero (it was 5 of 8). On the G55 score the resamples no longer separate any model: one tier of 9 (it was 7 models, then Cinque v3 and lebowski).
commit aebff69
31
Test change
The leaderboard shows tiers instead of places 1 to 9 and medals.
Whole drives are resampled 2000 times, and a model sits in a higher tier only when it beat every model in the lower tiers in at least 95% of the resamples; inside a tier, models are listed newest first. On the G55 score the 9 closed-loop models form two tiers: 7 models, then Cinque v3 and lebowski. The site build now fails if a page prints a place, a medal or a tier the resamples do not support.
commit dc0d9be
30
Note
After a push, N16 still encodes the truck's offset and heading
A linear probe reads the truck's lateral offset from N16's vision tokens at R² 0.84-0.93 and from its base summary at 0.87-0.98 on held-out sessions. The wrong-way first turn comes from the base's readout of yaw rate, not from missing information.
30
Test change
Redesigned leaderboard and ranking system: models are ranked 1..N by a 4-pillar G55 Composite Score (40% closed-loop path keeping at 2 s, 25% disturbance recovery at 2 s, 20% open-loop tracking across all 18 models, 15% closed-loop smoothness) with interactive tabs to re-rank by any individual test or release date, replacing the marginal-median tie-rank that grouped 8 of 9 models at rank 1.
Added an interactive per-camera-setup breakdown matrix across all 5 G55 camera hardware setups (`by_group`).
29
Note
Teaching comma's world model to keep a sideways offset
By day comma's world model as shipped already puts the road at a commanded offset (road gain 1.05 at 3 s). Fine-tuning made it overshoot (1.24), reward training did not pass its gate, and no version holds the road at night.
29
Note
A truck model learned from the G55's CAN predicts motion better
On 12 held-out drives a recurrent network learned from the truck's CAN data has a sideways error of 0.29 m after 4 s of free running, against 7.34 m for the hand-built model and 0.66 m for the same equations refit. It fails a stopping check, so the simulator's default stays hand-built.
29
Release
f22b8b34 uploaded to Hugging Face
G55 closed loop
14.9 cm (11.4 to 17.8)
G55 recovery tests
48.3 cm (39.2 to 59.5)
G55 open loop
0.177 m/s² (0.147 to 0.187)
28
Tested
9 models scored, suite geval-1
WMI, Dark Souls 2, The Cool People's, North Nevada, Steam Powered, Filet o Fish, North Dakota, Certified Herbalist, Farmville
28
Test change
G55 closed loop and the heading and off-centre recovery tests now run on 98 windows from 13 drives and 5 camera setups (they were 18 windows from 4 drives; the braking lead keeps its 2 windows), with results per camera setup beside the pooled number.
With 10 or more drives the closed-loop ranking now uses intervals. Drives whose recorded camera calibration was carried over from another drive are left out. The 8 current models rescored.
28
Test change
G55 open loop now runs on 119 windows of 30 s from 12 drives and 5 camera setups (it was 4 windows from 3 drives), with results per camera setup beside the pooled number.
Both cameras feed the model; on the V13 setup the part of comma's narrow view the 8 mm lens misses comes from the wide camera. Drives whose recorded camera calibration was carried over from another drive are left out. All 17 models rescored.
28
Test change
The front page is a leaderboard: best first by the G55 closed-loop median, medals for the first three, a toggle for newest first, bars from zero with lebowski's value as a dashed line, recovery and open-loop cells shaded by rank within the column.
Colours now mean what a model is: shipped or on master (blue), Hugging Face upload only (orange), lebowski (green), earlier models (grey). Charts over time follow, with the region better than lebowski shaded.
28
Test change
The front page opens with four small G55 charts over time and one line per model, newest first: closed-loop median distance from the recorded path at 2 s as a dot, lebowski included, then the recovery medians and the open-loop lateral error.
It was a two-screen chart ordered by windows closer than lebowski. Ranks and intervals still wait for 10 drives. Findings, model summaries and the bars on each model's row use the G55 results; comma's segments stay in the harness check. The G55 clip at the top of the page is removed. Scoring runs in the log are grouped per day.
27
Test change
comma's earlier driving models added, openpilot 0.9.5 (November 2023) to 0.11.1: nine models from Farmville to WMI, each run with its own release's inputs, camera warp and command and checked against comma's CI model replay of that model.
Open loop on the G55 and on comma's segments; closed loop not yet. Their float16 feature normalisation runs in float32 here (in float16 it overflowed on some frames under ONNX Runtime). New over-time panel: G55 open-loop lateral error.
27
Test change
The G55 sections lead: the ranking is G55 closed loop at 2 s, ranked with intervals once it has 10 or more drives and until then ordered by windows closer than lebowski.
comma's public drives move to a harness check near the bottom; the world-model player moves to the method page.
27
Test change
The closed-loop video is replaced by a player: the world model's 256×128 frames as generated, the paths drawn by the page over them, one view at a time.
27
Test change
Over-time charts and this log replace the one-off results section.
27
Test change
Closed-loop video of one public segment, with each model's path drawn on the generated view.
commit fe1a5ed
27
Test change
"Lateral lag" corrected to the lead of the lateral request over the car's motion: a larger value means the request comes earlier.
No better direction is marked for it or for lateral action rate.
commit 8cebf52
27
Test change
Models ranked by the median distance from the recorded path at 2 s in closed loop; the 4 s "primary" horizon is dropped.
commit 8cebf52
27
Test change
Results recomputed with the new intervals from the stored per-segment values, no new runs.
commit 1b06fee
27
Test change
Intervals resample whole drives; no interval below 10 drives, so G55 sections report win counts.
Lead values at the 0 and 1.5 s search limits are left out.
commit 102710f
27
Test change
Release watcher started: Hugging Face commaai/openpilot_driving_models and openpilot's branches, every 20 minutes.
commit 7856299
27
Test change
G55 recovery after a 0.5 m offset added, 18 windows.
commit a98409f
27
Test change
Closed loop on comma's public segments added: 28 segments in worldmodel-4B, comma's ideal car, seeds 0 and 1.
commit f0ec83f
27
Test change
First edition: 8 models, scores backfilled from the 2026-10-27 deep dive.
commit 3a0056f
24
Release
MD v3 / Flip Flop (1365126a) merged to openpilot master again
23
Release
Mountain Dew (8d736683) shipped in openpilot v0.11.2
17
Release
6b5b705f uploaded to Hugging Face
G55 closed loop
13.9 cm (10.8 to 17.2)
G55 recovery tests
50.0 cm (40.7 to 61.5)
G55 open loop
0.161 m/s² (0.135 to 0.171)
15
Release
N16 nav model (96244b1f) uploaded to Hugging Face
G55 closed loop
13.6 cm (10.0 to 17.9)
G55 recovery tests
48.4 cm (38.5 to 61.4)
G55 open loop
0.171 m/s² (0.144 to 0.183)
14
Release
MD v3 / Flip Flop (1365126a) merged to openpilot master
G55 closed loop
13.7 cm (10.9 to 16.6)
G55 recovery tests
51.7 cm (42.5 to 62.9)
G55 open loop
0.156 m/s² (0.126 to 0.168)
10
Release
Mountain Dew (8d736683) merged to openpilot master
G55 closed loop
13.2 cm (10.2 to 15.8)
G55 recovery tests
49.2 cm (39.9 to 60.5)
G55 open loop
0.164 m/s² (0.134 to 0.175)
8
Release
870a4823 uploaded to Hugging Face
G55 closed loop
13.3 cm (10.7 to 15.4)
G55 recovery tests
51.8 cm (42.3 to 63.4)
G55 open loop
0.157 m/s² (0.129 to 0.169)

September 2026

18
Release
Cinque v2 (1a421175) uploaded to Hugging Face
18
Release
Cinque v3 (f78ed37d) merged to openpilot master
G55 closed loop
15.0 cm (12.4 to 17.2)
G55 recovery tests
51.0 cm (41.4 to 62.7)
G55 open loop
0.167 m/s² (0.143 to 0.178)
18
Release
Cinque v3 (f78ed37d) uploaded to Hugging Face
G55 closed loop
15.0 cm (12.4 to 17.2)
G55 recovery tests
51.0 cm (41.4 to 62.7)
G55 open loop
0.167 m/s² (0.143 to 0.178)
11
Release
Cinque v2 (1a421175) merged to openpilot master
G55 closed loop
13.3 cm (10.7 to 15.6)
G55 recovery tests
50.1 cm (40.4 to 62.2)
G55 open loop
0.161 m/s² (0.140 to 0.170)

July 2026

1
Release
lebowski merged to openpilot master
G55 closed loop
17.0 cm (13.7 to 20.3)
G55 recovery tests
45.3 cm (38.8 to 53.7)
G55 open loop
0.173 m/s² (0.138 to 0.184)

Log

November 2026

October 2026

September 2026

July 2026

January 2026

December 2025

October 2025

August 2025

March 2025

April 2024

February 2024

November 2023