Releases / RL residual
MD v3 / Flip Flop 1365126a
On openpilot master since 14 Oct 2026. First public 14 Oct 2026. 382.9M parameters.
Where it sits
RL residualbase + learned correction
OctNov
Tests
- G55 closed loopDistance from the recorded path at 2 s13.7 cm (10.9 to 16.6)101520cmdone 31 Oct 2026
- G55 recovery testsAfter a 2° heading error: distance at 2 s51.7 cm (42.5 to 62.9)405060cmdone 31 Oct 2026
- G55 open loopLateral error0.156 m/s² (0.126 to 0.168)0.150.200.25m/s²done 31 Oct 2026
Bar: this release's 95% interval, tick its mean. Grey dots: the means of the other releases in the RL residual family. Each axis spans every release the test has run on.
Tested 2026-10-31 23:59 UTC: protocol geval-1, harness d8eb40eeba; G55 simulator g55-2026-10-31.
Every metric, with paired differences
better, 95% interval excludes zero Better, 95% interval excludes zeroworse, 95% interval excludes zero Worse, interval excludes zerolower, 95% interval excludes zero; no better directionhigher, 95% interval excludes zero; no better direction Lower or higher, no better direction (action rate, lead)not separated, interval includes zero Not separatedno interval (too few drives) No interval: fewer than 10 drives
G55 closed loop
98 windows from 13 drives.
Measured. worldmodel-4B as in the public closed loop, but the model drives a simulation of the G55's measured steering and powertrain response instead of the ideal car. The simulation starts from the truck's state (speed, yaw rate, steering, gear) in its CAN recording of the 2 s before the window; one window, whose 2 s before are not all valid CAN, starts steady at its recorded speed. Lane-following windows from the G55's drives, day and night; distance is to the driver's recorded path; seeds 0 and 1, averaged per window. 98 windows (59 day, 39 night).
| Metric | Mean | vs lebowski | vs Mountain Dew |
|---|---|---|---|
| Distance from the recorded path at 2 sm, lower is better | 0.137[0.109, 0.166]median 0.085 | better, 95% interval excludes zero−0.033[−0.044, −0.022]better on 64/98 | not separated, interval includes zero+0.005[−0.004, +0.015]better on 42/98 |
| Distance from the recorded path at 4 sm, lower is better | 0.539[0.392, 0.692]median 0.318 | not separated, interval includes zero+0.019[−0.130, +0.177]better on 48/98 | not separated, interval includes zero+0.031[−0.004, +0.076]better on 44/98 |
| Distance from the recorded path at 7.2 sm, lower is better | 2.53[1.66, 3.59]median 0.88 | not separated, interval includes zero+0.55[−0.46, +1.71]better on 34/98 | worse, 95% interval excludes zero+0.48[+0.19, +0.94]better on 32/98 |
| Share of episodes ever more than 1 m off the pathlower is better | 0.454[0.384, 0.554]median 0.500 | not separated, interval includes zero+0.026[−0.049, +0.108]better on 11/98 | worse, 95% interval excludes zero+0.051[+0.024, +0.082]better on 2/98 |
| Requested lateral jerk, RMSm/s³, lower is better | 0.286[0.231, 0.339]median 0.178 | better, 95% interval excludes zero−0.256[−0.351, −0.171]better on 86/98 | worse, 95% interval excludes zero+0.022[+0.005, +0.039]better on 42/98 |
| Requested longitudinal jerk, RMSm/s³, lower is better | 0.365[0.299, 0.447]median 0.277 | not separated, interval includes zero−0.021[−0.100, +0.074]better on 65/98 | not separated, interval includes zero−0.006[−0.033, +0.019]better on 62/98 |
| Speed error against the recorded speed, RMSm/s, lower is better | 1.22[1.02, 1.42]median 0.97 | not separated, interval includes zero+0.14[−0.02, +0.30]better on 38/98 | not separated, interval includes zero0.00[−0.08, +0.09]better on 44/98 |
By G55 camera setup (5 setups)
| Metric | V14 (8 mm + IMX662)19 win, 2 drv | V13 (8 mm + IMX662)18 win, 3 drv | Stock aimed up14 win, 2 drv | Stock new case4 win, 1 drv | Stock dual43 win, 5 drv |
|---|---|---|---|---|---|
| Distance from the recorded path at 2 sm | 0.122med 0.070 | 0.186med 0.097 | 0.124med 0.085 | 0.050med 0.049 | 0.135med 0.097 |
| Distance from the recorded path at 4 sm | 0.759med 0.329 | 0.551med 0.276 | 0.700med 0.518 | 0.257med 0.249 | 0.411med 0.288 |
| Distance from the recorded path at 7.2 sm | 3.70med 0.78 | 1.40med 0.92 | 4.43med 3.98 | 1.55med 1.47 | 1.96med 0.67 |
| Share of episodes ever more than 1 m off the path | 0.421med 0.000 | 0.417med 0.500 | 0.750med 1.000 | 0.500med 0.500 | 0.384med 0.000 |
| Requested lateral jerk, RMSm/s³ | 0.193med 0.159 | 0.233med 0.154 | 0.461med 0.244 | 0.267med 0.210 | 0.294med 0.195 |
| Requested longitudinal jerk, RMSm/s³ | 0.302med 0.291 | 0.318med 0.227 | 0.306med 0.264 | 0.441med 0.416 | 0.425med 0.337 |
| Speed error against the recorded speed, RMSm/s | 1.30med 0.92 | 0.86med 0.74 | 1.56med 1.43 | 1.05med 0.84 | 1.24med 1.07 |
G55 recovery tests
100 windows from 14 drives.
Measured. The truck closed loop with a disturbance while the model drives; only the tests listed ran. 2° off heading: an extra 2°/s yaw for the first second. 98 windows (59 day, 39 night). 0.5 m off centre: an extra yaw-rate pulse, out and back over the first 2 s, that would move an uncorrected car 0.5 m sideways with no net heading change. 98 windows (59 day, 39 night). Braking lead: windows where the car ahead braked; the gap is derived from the lead distance seen on the recording minus how far the model ran ahead of the recorded car. 2 windows (2 day, 0 night).
| Metric | Mean | vs lebowski | vs Mountain Dew |
|---|---|---|---|
| 2° off heading: distance from the path at 2 sm, lower is better | 0.517[0.425, 0.629]median 0.435n = 98 | worse, 95% interval excludes zero+0.065[+0.028, +0.103]better on 34/98 | worse, 95% interval excludes zero+0.026[+0.022, +0.029]better on 8/98 |
| 2° off heading: distance from the path at 4 sm, lower is better | 0.788[0.575, 1.041]median 0.466n = 98 | not separated, interval includes zero+0.086[−0.036, +0.189]better on 38/98 | not separated, interval includes zero+0.041[−0.001, +0.099]better on 47/98 |
| 2° off heading: requested lateral jerk, RMSm/s³, lower is better | 0.371[0.335, 0.408]median 0.303n = 98 | better, 95% interval excludes zero−0.516[−0.586, −0.441]better on 95/98 | not separated, interval includes zero+0.009[−0.017, +0.036]better on 39/98 |
| 0.5 m off centre: distance from the path at 2 sm, lower is better | 0.285[0.260, 0.310]median 0.238n = 98 | not separated, interval includes zero+0.018[−0.004, +0.040]better on 34/98 | worse, 95% interval excludes zero+0.027[+0.021, +0.034]better on 14/98 |
| 0.5 m off centre: distance from the path at 4 sm, lower is better | 0.680[0.530, 0.830]median 0.369n = 98 | not separated, interval includes zero+0.018[−0.064, +0.118]better on 63/98 | not separated, interval includes zero+0.021[−0.036, +0.091]better on 53/98 |
| 0.5 m off centre: requested lateral jerk, RMSm/s³, lower is better | 0.514[0.469, 0.563]median 0.418n = 98 | better, 95% interval excludes zero−0.597[−0.668, −0.515]better on 95/98 | not separated, interval includes zero−0.038[−0.084, +0.002]better on 57/98 |
| Braking lead: distance ahead of the recorded car at 4 sm, lower is better | −0.565no intervalmedian −0.565n = 2 | no interval (too few drives)+0.773no intervalbetter on 1/20/1 drives | no interval (too few drives)−0.248no intervalbetter on 1/21/1 drives |
| Braking lead: smallest gap to the lead (derived)m, higher is better | 25.7no intervalmedian 25.7n = 2 | no interval (too few drives)−1.8no intervalbetter on 1/20/1 drives | no interval (too few drives)+0.8no intervalbetter on 1/21/1 drives |
| Braking lead: speed error against the recorded speed, RMSm/s, lower is better | 1.26no intervalmedian 1.26n = 2 | no interval (too few drives)+0.06no intervalbetter on 1/20/1 drives | no interval (too few drives)+0.01no intervalbetter on 0/20/1 drives |
By G55 camera setup (5 setups)
| Metric | V14 (8 mm + IMX662)19 win, 2 drv | V13 (8 mm + IMX662)18 win, 3 drv | Stock aimed up14 win, 2 drv | Stock new case4 win, 1 drv | Stock dual45 win, 6 drv |
|---|---|---|---|---|---|
| 2° off heading: distance from the path at 2 sm | 0.434med 0.432 | 0.885med 0.834 | 0.416med 0.389 | 0.379med 0.401 | 0.447med 0.390 |
| 2° off heading: distance from the path at 4 sm | 0.801med 0.337 | 1.311med 0.865 | 0.735med 0.515 | 0.397med 0.312 | 0.618med 0.432 |
| 2° off heading: requested lateral jerk, RMSm/s³ | 0.315med 0.268 | 0.447med 0.402 | 0.397med 0.320 | 0.329med 0.219 | 0.360med 0.294 |
| 0.5 m off centre: distance from the path at 2 sm | 0.256med 0.225 | 0.293med 0.217 | 0.293med 0.262 | 0.319med 0.235 | 0.288med 0.243 |
| 0.5 m off centre: distance from the path at 4 sm | 0.766med 0.384 | 0.590med 0.227 | 0.848med 0.603 | 0.759med 0.583 | 0.619med 0.356 |
| 0.5 m off centre: requested lateral jerk, RMSm/s³ | 0.424med 0.419 | 0.435med 0.382 | 0.642med 0.546 | 0.637med 0.407 | 0.533med 0.439 |
| Braking lead: distance ahead of the recorded car at 4 sm | — | — | — | — | −0.565med −0.565 |
| Braking lead: smallest gap to the lead (derived)m | — | — | — | — | 25.7med 25.7 |
| Braking lead: speed error against the recorded speed, RMSm/s | — | — | — | — | 1.26med 1.26 |
G55 open loop
119 windows from 12 drives.
Measured. The G55's recorded drives replayed through the model with both of the truck's cameras (each setup's own wide-camera calibration), road-plane warp to 1.45 m. Where the narrow lens does not cover comma's narrow view (the 8 mm lens in the V13 case), those pixels come from the wide camera, brightness matched. Truth is the truck's CAN speed and yaw rate. 30 s windows after a 5 s warm-up; window = unit, drive = cluster; results are also given per camera setup. Corpus windows of 30 s from the truck's recorded drives, scored per camera setup.
| Metric | Mean | vs lebowski | vs Mountain Dew |
|---|---|---|---|
| Lateral errorm/s², lower is better | 0.156[0.126, 0.168]median 0.144 | better, 95% interval excludes zero−0.017[−0.024, −0.008]better on 85/119 | better, 95% interval excludes zero−0.008[−0.011, −0.006]better on 102/119 |
| Lateral error at the model's own best leadm/s², lower is better | 0.143[0.116, 0.154]median 0.131 | better, 95% interval excludes zero−0.021[−0.027, −0.013]better on 95/119 | better, 95% interval excludes zero−0.003[−0.005, −0.001]better on 75/119 |
| Lateral correlation with the car's later motionhigher is better | 0.820[0.786, 0.850]median 0.916 | better, 95% interval excludes zero+0.053[+0.024, +0.065]better on 92/119 | better, 95% interval excludes zero+0.017[+0.012, +0.028]better on 110/119 |
| Lateral gain (1 = same size as the car's motion)closer to 1 is better; the delta is the change in distance from 1 | 0.789[0.770, 0.875]median 0.810 | not separated, interval includes zero0.000[−0.022, +0.030]better on 54/119 | better, 95% interval excludes zero−0.008[−0.036, −0.002]better on 65/119 |
| Lateral action rate, RMS (smoothness)m/s³, no better direction | 1.12[0.61, 1.35]median 0.82 | lower, 95% interval excludes zero; no better direction−1.17[−1.29, −0.55]lower on 118/119 | not separated, interval includes zero+0.03[−0.06, +0.05]lower on 44/119 |
| Lead of the lateral request over the car's motions, no better direction | 0.641[0.578, 0.681]median 0.600n = 118 | not separated, interval includes zero+0.025[−0.050, +0.115]lower on 40/118 | lower, 95% interval excludes zero; no better direction−0.075[−0.092, −0.061]lower on 88/118 |
| Longitudinal errorm/s², lower is better | 0.350[0.268, 0.397]median 0.360 | not separated, interval includes zero−0.019[−0.064, +0.004]better on 69/119 | not separated, interval includes zero−0.004[−0.032, +0.003]better on 66/119 |
| Longitudinal correlationhigher is better | 0.334[0.279, 0.584]median 0.283 | not separated, interval includes zero+0.007[−0.039, +0.037]better on 68/119 | not separated, interval includes zero−0.005[−0.011, +0.002]better on 51/119 |
| Longitudinal error while brakingm/s², lower is better | 0.479[0.443, 0.619]median 0.475n = 81 | not separated, interval includes zero−0.019[−0.041, +0.072]better on 52/81 | not separated, interval includes zero+0.004[−0.007, +0.049]better on 37/81 |
| Longitudinal action rate, RMS (smoothness)m/s³, no better direction | 4.41[2.03, 5.75]median 4.28 | lower, 95% interval excludes zero; no better direction−1.20[−1.79, −0.81]lower on 93/119 | lower, 95% interval excludes zero; no better direction−0.08[−0.10, −0.01]lower on 84/119 |
| Plan speed error at 3 sm/s, lower is better | 1.89[1.15, 2.30]median 1.84 | not separated, interval includes zero−0.11[−0.35, +0.31]better on 78/119 | not separated, interval includes zeroidenticalon all 119 |
By G55 camera setup (5 setups)
| Metric | V14 (8 mm + IMX662)8 win, 2 drv | V13 (8 mm + IMX662)92 win, 3 drv | Stock aimed up4 win, 2 drv | Stock new case2 win, 1 drv | Stock dual13 win, 4 drv |
|---|---|---|---|---|---|
| Lateral errorm/s² | 0.144med 0.144 | 0.162med 0.153 | 0.119med 0.119 | 0.141med 0.141 | 0.132med 0.120 |
| Lateral error at the model's own best leadm/s² | 0.137med 0.129 | 0.148med 0.136 | 0.108med 0.109 | 0.136med 0.136 | 0.121med 0.111 |
| Lateral correlation with the car's later motion | 0.871med 0.922 | 0.811med 0.919 | 0.832med 0.917 | 0.821med 0.821 | 0.844med 0.866 |
| Lateral gain (1 = same size as the car's motion) | 0.890med 0.895 | 0.762med 0.803 | 0.840med 0.903 | 0.778med 0.778 | 0.910med 0.903 |
| Lateral action rate, RMS (smoothness)m/s³ | 0.59med 0.55 | 1.26med 0.99 | 0.70med 0.68 | 0.76med 0.76 | 0.62med 0.60 |
| Lead of the lateral request over the car's motions | 0.537med 0.475 | 0.648med 0.600 | 0.688med 0.625 | 0.625med 0.625 | 0.646med 0.600 |
| Longitudinal errorm/s² | 0.346med 0.285 | 0.359med 0.372 | 0.404med 0.378 | 0.380med 0.380 | 0.267med 0.249 |
| Longitudinal correlation | 0.683med 0.721 | 0.254med 0.234 | 0.514med 0.481 | 0.528med 0.528 | 0.600med 0.727 |
| Longitudinal error while brakingm/s² | 0.634med 0.670 | 0.451med 0.441 | 0.513med 0.513 | 0.427med 0.427 | 0.639med 0.680 |
| Longitudinal action rate, RMS (smoothness)m/s³ | 4.57med 4.45 | 5.00med 5.04 | 1.93med 1.85 | 2.34med 2.34 | 1.18med 1.16 |
| Plan speed error at 3 sm/s | 1.09med 1.06 | 2.12med 1.98 | 1.44med 1.51 | 2.13med 2.13 | 0.81med 0.83 |
Limits
- Open-loop scores compare a model's action with what the recording driver or car did; a model that drives differently but well is scored as wrong. comma's RL residual makes the lateral action smoother, and earlier than its base's by 0.10 to 0.26 s (measured on the G55 and on 16 of comma's cars); both score as error here. Measured on the 2026-10-27 deep dive: open-loop accuracy did not predict closed-loop driving.
- Closed-loop numbers come from a generative world model, not a car. In a separate test on the G55's windows, worldmodel-4B followed an added yaw command through 2 s but showed 0 to 20% of an added lateral offset 3 to 4 s later; past about 4 s the recorded anchor frames also pull every model back toward the recorded path. The front page reports 2 s.
- Intervals are 95% percentile bootstraps that resample whole drives (the drive a public segment was cut from, the recorded drive a truck window comes from), so segments of one drive count once. Below 10 drives there is no interval: the braking-lead test comes from one drive and reports means and win counts only.
- Paired deltas are model minus baseline on the units both have (for lateral gain, the change in its distance from 1); wins count units where the model is better in the metric's direction.
- The lead of the lateral request is the shift, searched from 0 to 1.5 s, at which the request best correlates with the car's later lateral motion; a larger value means the request comes earlier. A value at either edge is censored and left out. Plan speed error at 3 s reads the plan head, which does not separate models built on one frozen base: it is identical on every segment for the four f22e06e5 models, and for Cinque v2 and v3.
- Truck open loop comes from few drives, and the camera setup changed between them (lens, housing, aim); the pooled number mixes setups, so read the per-setup numbers too. The 8 mm setups' wide camera recorded about 7-10 fps, so its frame can be up to ~60 ms from the narrow one. On the V14 setup up to 7 of a window's 700 steps are ~100 ms (one camera frame dropped). The camera's aim is the truck's live calibration learned on each drive; drives whose recorded calibration was carried over from another drive are left out.
- Truck closed loop: the steering and powertrain simulation is fitted to the G55, not validated on it in closed loop; the reference is the driver's own line, not the lane centre. The pooled number mixes camera setups, so read the per-setup numbers too. On the V14 setup the wide frame can be up to 90 ms from the simulated one (its camera recorded 7-9 fps).
- Recovery tests are disturbances rendered by the world model (yaw it renders; a one-frame jump it does not, so none is used). The braking-lead test has 2 windows from one drive, whose recorded calibration was carried over from an earlier drive.
The model file
- Model id
1365126a_12864- SHA-256
35976b11be3904eb78e32f7dc55a6abf8e9433ac5aa391a1998f1e9ce001b199- File tested
- openpilot-master,
9fba666fd4(source), 2026-10-24 - Parameters
- 382.9M
- GMAC per step
- 46.7
- Notes
- On openpilot master as MD v3 (2026-10-14, PR #39087) and again as Flip Flop (2026-10-24, PR #39156), same weights. Scored from the 2026-10-24 Hugging Face re-upload, weights identical to the 2026-10-16 upload. Vision base f22e06e5 with an RL residual head.
Results file (results/1365126a_12864.json)
History of this page
- Tests run (3 of 3), results published
- MD v3 / Flip Flop (1365126a) merged to openpilot master again
- MD v3 / Flip Flop (1365126a) merged to openpilot master