My Self-Driving G-Wagon

comma's public segments, open loop

comma's public two-camera segments replayed through each model, against the recorded car's lateral acceleration.

39 segments from 23 drives. Run on 19 of 19 releases. When the test changes, every release is run again.

Lateral error

Lower is better. Bar: 95% interval from resampling whole drives; tick: mean. Rows are releases, newest first; a gap between two releases smaller than their intervals does not separate them.
0.100.15m/s²
  1. f22e06e55 Nov 2026, RL residual0.148 m/s² (0.119 to 0.179)
  2. f22b8b3429 Oct 2026, RL residual0.148 m/s² (0.116 to 0.183)
  3. 6b5b705f17 Oct 2026, RL residual0.137 m/s² (0.108 to 0.168)
  4. N16 nav model15 Oct 2026, RL residual0.152 m/s² (0.119 to 0.185)
  5. MD v3 / Flip Flop14 Oct 2026, RL residual0.129 m/s² (0.101 to 0.159)
  6. Mountain Dew10 Oct 2026, RL residual0.143 m/s² (0.113 to 0.176)
  7. 870a48238 Oct 2026, RL residual0.134 m/s² (0.106 to 0.163)
  8. Cinque v318 Sep 2026, Big RL0.137 m/s² (0.108 to 0.166)
  9. Cinque v211 Sep 2026, Big RL0.147 m/s² (0.113 to 0.183)
  10. lebowski1 Jul 2026, lebowski0.109 m/s² (0.085 to 0.134)
  11. WMI19 Jan 2026, Vision + policy0.143 m/s² (0.109 to 0.181)
  12. Dark Souls 212 Dec 2025, Vision + policy0.129 m/s² (0.099 to 0.163)
  13. The Cool People's20 Oct 2025, Vision + policy0.124 m/s² (0.096 to 0.154)
  14. North Nevada9 Oct 2025, Vision + policy0.120 m/s² (0.094 to 0.149)
  15. Steam Powered17 Aug 2025, Vision + policy0.113 m/s² (0.088 to 0.141)
  16. Filet o Fish7 Mar 2025, Vision + policy0.139 m/s² (0.110 to 0.172)
  17. North Dakota29 Apr 2024, One network0.135 m/s² (0.106 to 0.167)
  18. Certified Herbalist19 Feb 2024, One network0.139 m/s² (0.108 to 0.173)
  19. Farmville7 Nov 2023, One network0.139 m/s² (0.108 to 0.174)
The table: mean, interval and median

comma's public segments, open loop

comma's public two-camera segments replayed through each model, against the recorded car's lateral acceleration. 39 segments from 23 drives for every model.

Mean over segments with its 95% interval (resampling whole drives), median beneath. Newest release first.
Modelfirst public · statusLateral errorm/s²
f22e06e52026-11-05, Hugging Face checkpoint (unshipped)0.148 m/s²[0.119, 0.179]median 0.135
f22b8b342026-10-29, Hugging Face checkpoint (unshipped)0.148 m/s²[0.116, 0.183]median 0.136
6b5b705f2026-10-17, Hugging Face checkpoint (unshipped)0.137 m/s²[0.108, 0.168]median 0.118
N16 nav model 96244b1f2026-10-15, Hugging Face checkpoint (unshipped)0.152 m/s²[0.119, 0.185]median 0.134
MD v3 / Flip Flop 1365126a2026-10-14, merged to openpilot master (2026-10-14), updated 2026-10-240.129 m/s²[0.101, 0.159]median 0.110
Mountain Dew 8d7366832026-10-10, shipped in openpilot v0.11.2 (master 2026-10-10)0.143 m/s²[0.113, 0.176]median 0.127
870a48232026-10-08, Hugging Face checkpoint (unshipped)0.134 m/s²[0.106, 0.163]median 0.119
Cinque v3 f78ed37d2026-09-18, merged to openpilot master (2026-09-18)0.137 m/s²[0.108, 0.166]median 0.124
Cinque v2 1a4211752026-09-11, merged to openpilot master (2026-09-11)0.147 m/s²[0.113, 0.183]median 0.120
lebowski2026-07-01, merged to openpilot master (2026-07-01)0.109 m/s²[0.085, 0.134]median 0.087
WMI2026-01-19, merged to openpilot master (2026-01-19)0.143 m/s²[0.109, 0.181]median 0.134
Dark Souls 22025-12-12, merged to openpilot master (2025-12-12)0.129 m/s²[0.099, 0.163]median 0.095
The Cool People's2025-10-20, merged to openpilot master (2025-10-20)0.124 m/s²[0.096, 0.154]median 0.099
North Nevada2025-10-09, merged to openpilot master (2025-10-09)0.120 m/s²[0.094, 0.149]median 0.091
Steam Powered2025-08-17, merged to openpilot master (2025-08-17)0.113 m/s²[0.088, 0.141]median 0.106
Filet o Fish2025-03-07, merged to openpilot master (2025-03-07)0.139 m/s²[0.110, 0.172]median 0.130
North Dakota2024-04-29, merged to openpilot master (2024-04-29)0.135 m/s²[0.106, 0.167]median 0.131
Certified Herbalist2024-02-19, merged to openpilot master (2024-02-19)0.139 m/s²[0.108, 0.173]median 0.153
Farmville2023-11-07, merged to openpilot master (2023-11-07)0.139 m/s²[0.108, 0.174]median 0.115
Conditions and limits

Measured. comma's public two-camera segments (openpilot CI data), native intrinsics and the device's own calibration, both cameras; every model on the same decoded frames. Truth is what the recording car did next: carState speed and acceleration, yaw rate from the device's cameraOdometry. Segments where the car never moves have no data. 39 of the 45 listed segments have a score, from 23 drives and 20 cars.

Conditions and limits

Measured. comma's public two-camera segments (openpilot CI data), native intrinsics and the device's own calibration, both cameras; every model on the same decoded frames. Truth is what the recording car did next: carState speed and acceleration, yaw rate from the device's cameraOdometry. Segments where the car never moves have no data. 39 of the 45 listed segments have a score, from 23 drives and 20 cars.