{
 "schema": 1,
 "model": {
  "id": "wmi",
  "name": "WMI",
  "sha256": "1dc66bc06f250b577653ccbeaa2c6521b3d46749f601d0a1a366419e929ca438+1edea5bb56f876db4cec97c150799513f6a59373f3ad152d55e4dcaab1b809e3",
  "source": {
   "kind": "openpilot-master",
   "ref": "0b41b42f7b23637e3bbb1f7495d318d954f971e7",
   "url": "https://github.com/commaai/openpilot/pull/36798",
   "date": "2026-01-19"
  },
  "first_public": "2026-01-19",
  "on_master": "2026-01-19",
  "params_m": null,
  "gmac": null,
  "family": "vision-policy",
  "notes": "Named in the openpilot 0.11.0 release notes. The 0.11.0 and 0.11.1 tags carry the CD210 model (PR #37050, master 2026-02-03): a re-export of these weights, every weight tensor byte for byte the same (policy checkpoint a27b3122/100 in both), so it is scored once, here. Same vision model as Dark Souls 2 (6a7d09ad/200). Plan only, commands from modeld's get_action_from_model."
 },
 "scored": {
  "date": "2026-10-28T04:04:40Z",
  "suite": "geval-1",
  "code": "d8eb40eeba87"
 },
 "baselines": [
  "lebowski_877m",
  "dark_souls_2"
 ],
 "sections": {
  "public_open_loop": {
   "status": "ok",
   "unit_of_evidence": "segment",
   "n": 39,
   "clusters": 23,
   "condition": "Measured. comma's public two-camera segments (openpilot CI data), native intrinsics and the device's own calibration, both cameras; every model on the same decoded frames. Truth is what the recording car did next: carState speed and acceleration, yaw rate from the device's cameraOdometry. Segments where the car never moves have no data. 39 of the 45 listed segments have a score, from 23 drives and 20 cars.",
   "baselines": [
    "lebowski_877m",
    "dark_souls_2"
   ],
   "metrics": [
    {
     "key": "lat_mae",
     "label": "Lateral error",
     "unit": "m/s²",
     "better": "lower",
     "value": 0.1433,
     "median": 0.1338,
     "ci": [
      0.1094,
      0.1806
     ],
     "n": 39,
     "clusters": 23,
     "paired": {
      "lebowski_877m": {
       "delta": 0.03462,
       "median": 0.02248,
       "ci": [
        0.02222,
        0.04909
       ],
       "n": 39,
       "wins": 4,
       "ties": 0,
       "clusters": 23,
       "cluster_wins": 3
      },
      "dark_souls_2": {
       "delta": 0.01389,
       "median": 0.008091,
       "ci": [
        0.008002,
        0.02054
       ],
       "n": 39,
       "wins": 10,
       "ties": 0,
       "clusters": 23,
       "cluster_wins": 4
      }
     }
    },
    {
     "key": "lat_mae_bl",
     "label": "Lateral error at the model's own best lead",
     "unit": "m/s²",
     "better": "lower",
     "value": 0.14,
     "median": 0.1322,
     "ci": [
      0.1068,
      0.1764
     ],
     "n": 39,
     "clusters": 23,
     "paired": {
      "lebowski_877m": {
       "delta": 0.03855,
       "median": 0.02968,
       "ci": [
        0.02697,
        0.05239
       ],
       "n": 39,
       "wins": 3,
       "ties": 0,
       "clusters": 23,
       "cluster_wins": 1
      },
      "dark_souls_2": {
       "delta": 0.01446,
       "median": 0.008327,
       "ci": [
        0.008465,
        0.02151
       ],
       "n": 39,
       "wins": 9,
       "ties": 0,
       "clusters": 23,
       "cluster_wins": 4
      }
     }
    },
    {
     "key": "lat_corr",
     "label": "Lateral correlation with the car's later motion",
     "unit": "",
     "better": "higher",
     "value": 0.8064,
     "median": 0.9202,
     "ci": [
      0.7236,
      0.8812
     ],
     "n": 39,
     "clusters": 23,
     "paired": {
      "lebowski_877m": {
       "delta": -0.05118,
       "median": -0.01704,
       "ci": [
        -0.1193,
        -0.009631
       ],
       "n": 39,
       "wins": 8,
       "ties": 0,
       "clusters": 23,
       "cluster_wins": 4
      },
      "dark_souls_2": {
       "delta": -0.005382,
       "median": -0.0006701,
       "ci": [
        -0.01624,
        0.005425
       ],
       "n": 39,
       "wins": 17,
       "ties": 0,
       "clusters": 23,
       "cluster_wins": 9
      }
     }
    },
    {
     "key": "lat_gain",
     "label": "Lateral gain (1 = same size as the car's motion)",
     "unit": "",
     "better": "near1",
     "value": 0.8381,
     "median": 0.8964,
     "ci": [
      0.741,
      0.933
     ],
     "n": 39,
     "clusters": 23,
     "paired": {
      "lebowski_877m": {
       "delta": 0.01473,
       "median": 0.0002165,
       "ci": [
        -0.01598,
        0.05477
       ],
       "n": 39,
       "wins": 19,
       "ties": 0,
       "clusters": 23,
       "cluster_wins": 9
      },
      "dark_souls_2": {
       "delta": -0.01807,
       "median": -0.0005144,
       "ci": [
        -0.03811,
        0.0006691
       ],
       "n": 39,
       "wins": 20,
       "ties": 0,
       "clusters": 23,
       "cluster_wins": 13
      }
     }
    },
    {
     "key": "lat_rate_rms",
     "label": "Lateral action rate, RMS (smoothness)",
     "unit": "m/s³",
     "better": "lower",
     "value": 0.4931,
     "median": 0.4519,
     "ci": [
      0.3893,
      0.6084
     ],
     "n": 39,
     "clusters": 23,
     "paired": {
      "lebowski_877m": {
       "delta": -0.1707,
       "median": -0.1462,
       "ci": [
        -0.1991,
        -0.139
       ],
       "n": 39,
       "wins": 38,
       "ties": 0,
       "clusters": 23,
       "cluster_wins": 23
      },
      "dark_souls_2": {
       "delta": 0.03851,
       "median": 0.02257,
       "ci": [
        0.0233,
        0.05368
       ],
       "n": 39,
       "wins": 8,
       "ties": 0,
       "clusters": 23,
       "cluster_wins": 5
      }
     }
    },
    {
     "key": "lat_best_lag",
     "label": "Lead of the lateral request over the car's motion",
     "unit": "s",
     "better": "lower",
     "value": 0.5414,
     "median": 0.45,
     "ci": [
      0.4437,
      0.6557
     ],
     "n": 35,
     "clusters": 21,
     "paired": {
      "lebowski_877m": {
       "delta": 0.1561,
       "median": 0.1,
       "ci": [
        0.09516,
        0.2307
       ],
       "n": 33,
       "wins": 2,
       "ties": 6,
       "clusters": 21,
       "cluster_wins": 0
      },
      "dark_souls_2": {
       "delta": 0.002941,
       "median": 0.0,
       "ci": [
        -0.02581,
        0.03
       ],
       "n": 34,
       "wins": 10,
       "ties": 14,
       "clusters": 21,
       "cluster_wins": 7
      }
     }
    },
    {
     "key": "lon_mae",
     "label": "Longitudinal error",
     "unit": "m/s²",
     "better": "lower",
     "value": 0.343,
     "median": 0.3113,
     "ci": [
      0.296,
      0.3964
     ],
     "n": 39,
     "clusters": 23,
     "paired": {
      "lebowski_877m": {
       "delta": 0.03484,
       "median": 0.04056,
       "ci": [
        0.01582,
        0.05542
       ],
       "n": 39,
       "wins": 7,
       "ties": 0,
       "clusters": 23,
       "cluster_wins": 4
      },
      "dark_souls_2": {
       "delta": 0.008149,
       "median": 0.003755,
       "ci": [
        -0.004282,
        0.02097
       ],
       "n": 39,
       "wins": 17,
       "ties": 0,
       "clusters": 23,
       "cluster_wins": 8
      }
     }
    },
    {
     "key": "lon_corr",
     "label": "Longitudinal correlation",
     "unit": "",
     "better": "higher",
     "value": 0.6653,
     "median": 0.7913,
     "ci": [
      0.5511,
      0.7608
     ],
     "n": 39,
     "clusters": 23,
     "paired": {
      "lebowski_877m": {
       "delta": -0.05241,
       "median": -0.02221,
       "ci": [
        -0.102,
        -0.01091
       ],
       "n": 39,
       "wins": 10,
       "ties": 0,
       "clusters": 23,
       "cluster_wins": 5
      },
      "dark_souls_2": {
       "delta": -0.01831,
       "median": -0.01152,
       "ci": [
        -0.04603,
        0.005418
       ],
       "n": 39,
       "wins": 14,
       "ties": 0,
       "clusters": 23,
       "cluster_wins": 7
      }
     }
    },
    {
     "key": "lon_brake_mae",
     "label": "Longitudinal error while braking",
     "unit": "m/s²",
     "better": "lower",
     "value": 0.4898,
     "median": 0.4604,
     "ci": [
      0.3981,
      0.5882
     ],
     "n": 30,
     "clusters": 21,
     "paired": {
      "lebowski_877m": {
       "delta": 0.05552,
       "median": 0.03184,
       "ci": [
        0.008448,
        0.1073
       ],
       "n": 30,
       "wins": 11,
       "ties": 0,
       "clusters": 21,
       "cluster_wins": 6
      },
      "dark_souls_2": {
       "delta": 0.0559,
       "median": 0.05894,
       "ci": [
        0.01694,
        0.1014
       ],
       "n": 30,
       "wins": 8,
       "ties": 0,
       "clusters": 21,
       "cluster_wins": 4
      }
     }
    },
    {
     "key": "lon_rate_rms",
     "label": "Longitudinal action rate, RMS (smoothness)",
     "unit": "m/s³",
     "better": "lower",
     "value": 0.4204,
     "median": 0.3686,
     "ci": [
      0.3687,
      0.4781
     ],
     "n": 39,
     "clusters": 23,
     "paired": {
      "lebowski_877m": {
       "delta": -0.9126,
       "median": -0.8803,
       "ci": [
        -1.073,
        -0.7682
       ],
       "n": 39,
       "wins": 39,
       "ties": 0,
       "clusters": 23,
       "cluster_wins": 23
      },
      "dark_souls_2": {
       "delta": 0.00713,
       "median": 0.002712,
       "ci": [
        -0.02339,
        0.03843
       ],
       "n": 39,
       "wins": 18,
       "ties": 0,
       "clusters": 23,
       "cluster_wins": 10
      }
     }
    },
    {
     "key": "v3_mae",
     "label": "Plan speed error at 3 s",
     "unit": "m/s",
     "better": "lower",
     "value": 1.0,
     "median": 0.9655,
     "ci": [
      0.8558,
      1.163
     ],
     "n": 39,
     "clusters": 23,
     "paired": {
      "lebowski_877m": {
       "delta": 0.2845,
       "median": 0.2897,
       "ci": [
        0.1966,
        0.363
       ],
       "n": 39,
       "wins": 6,
       "ties": 0,
       "clusters": 23,
       "cluster_wins": 3
      },
      "dark_souls_2": {
       "delta": 0.03707,
       "median": 0.0628,
       "ci": [
        -0.004393,
        0.08144
       ],
       "n": 39,
       "wins": 14,
       "ties": 0,
       "clusters": 23,
       "cluster_wins": 8
      }
     }
    }
   ],
   "per_unit_file": "per_unit/wmi/public_open_loop.csv"
  },
  "public_closed_loop": {
   "status": "not_run",
   "unit_of_evidence": "segment",
   "n": 0,
   "condition": "Measured. comma's worldmodel-4B generates each next frame at 5 Hz from 9 history frames, 5 recorded anchor frames and the car's pose; the model drives comma's ideal car (the requested curvature and acceleration happen at once) for 7.2 s from the recorded start. Distance is to the recording car's own path. One window per segment, nearest the middle, where the car holds at least 5 m/s for the window and the 4.8 s before it; segments with no such stretch are left out. Seeds 0 and 1, averaged per segment.",
   "baselines": [],
   "metrics": [],
   "per_unit_file": null
  },
  "truck_open_loop": {
   "status": "ok",
   "unit_of_evidence": "window",
   "n": 119,
   "clusters": 12,
   "condition": "Measured. The G55's recorded drives replayed through the model with both of the truck's cameras (each setup's own wide-camera calibration), road-plane warp to 1.45 m. Where the narrow lens does not cover comma's narrow view (the 8 mm lens in the V13 case), those pixels come from the wide camera, brightness matched. Truth is the truck's CAN speed and yaw rate. 30 s windows after a 5 s warm-up; window = unit, drive = cluster; results are also given per camera setup. Corpus windows of 30 s from the truck's recorded drives, scored per camera setup.",
   "baselines": [
    "lebowski_877m",
    "dark_souls_2"
   ],
   "metrics": [
    {
     "key": "lat_mae",
     "label": "Lateral error",
     "unit": "m/s²",
     "better": "lower",
     "value": 0.2307,
     "median": 0.2187,
     "ci": [
      0.1983,
      0.2591
     ],
     "n": 119,
     "clusters": 12,
     "paired": {
      "lebowski_877m": {
       "delta": 0.0579,
       "median": 0.04539,
       "ci": [
        0.03869,
        0.09505
       ],
       "n": 119,
       "wins": 13,
       "ties": 0,
       "clusters": 12,
       "cluster_wins": 0
      },
      "dark_souls_2": {
       "delta": 0.05863,
       "median": 0.05203,
       "ci": [
        0.04789,
        0.07484
       ],
       "n": 119,
       "wins": 3,
       "ties": 0,
       "clusters": 12,
       "cluster_wins": 0
      }
     }
    },
    {
     "key": "lat_mae_bl",
     "label": "Lateral error at the model's own best lead",
     "unit": "m/s²",
     "better": "lower",
     "value": 0.2263,
     "median": 0.2149,
     "ci": [
      0.1943,
      0.2551
     ],
     "n": 119,
     "clusters": 12,
     "paired": {
      "lebowski_877m": {
       "delta": 0.06179,
       "median": 0.04932,
       "ci": [
        0.04259,
        0.0987
       ],
       "n": 119,
       "wins": 12,
       "ties": 0,
       "clusters": 12,
       "cluster_wins": 0
      },
      "dark_souls_2": {
       "delta": 0.0581,
       "median": 0.05178,
       "ci": [
        0.04736,
        0.07416
       ],
       "n": 119,
       "wins": 4,
       "ties": 0,
       "clusters": 12,
       "cluster_wins": 0
      }
     }
    },
    {
     "key": "lat_corr",
     "label": "Lateral correlation with the car's later motion",
     "unit": "",
     "better": "higher",
     "value": 0.7857,
     "median": 0.8586,
     "ci": [
      0.7341,
      0.8149
     ],
     "n": 119,
     "clusters": 12,
     "paired": {
      "lebowski_877m": {
       "delta": 0.0189,
       "median": 0.003058,
       "ci": [
        -0.0292,
        0.03853
       ],
       "n": 119,
       "wins": 64,
       "ties": 0,
       "clusters": 12,
       "cluster_wins": 5
      },
      "dark_souls_2": {
       "delta": -0.04195,
       "median": -0.03255,
       "ci": [
        -0.06111,
        -0.03294
       ],
       "n": 119,
       "wins": 16,
       "ties": 0,
       "clusters": 12,
       "cluster_wins": 0
      }
     }
    },
    {
     "key": "lat_gain",
     "label": "Lateral gain (1 = same size as the car's motion)",
     "unit": "",
     "better": "near1",
     "value": 1.003,
     "median": 0.8949,
     "ci": [
      0.9373,
      1.171
     ],
     "n": 119,
     "clusters": 12,
     "paired": {
      "lebowski_877m": {
       "delta": 0.03594,
       "median": 0.02201,
       "ci": [
        -0.0238,
        0.161
       ],
       "n": 119,
       "wins": 53,
       "ties": 0,
       "clusters": 12,
       "cluster_wins": 4
      },
      "dark_souls_2": {
       "delta": 0.08011,
       "median": 0.03622,
       "ci": [
        0.03248,
        0.1505
       ],
       "n": 119,
       "wins": 37,
       "ties": 0,
       "clusters": 12,
       "cluster_wins": 5
      }
     }
    },
    {
     "key": "lat_rate_rms",
     "label": "Lateral action rate, RMS (smoothness)",
     "unit": "m/s³",
     "better": "lower",
     "value": 0.8184,
     "median": 0.7514,
     "ci": [
      0.5474,
      0.9014
     ],
     "n": 119,
     "clusters": 12,
     "paired": {
      "lebowski_877m": {
       "delta": -1.472,
       "median": -1.264,
       "ci": [
        -1.77,
        -0.6495
       ],
       "n": 119,
       "wins": 119,
       "ties": 0,
       "clusters": 12,
       "cluster_wins": 12
      },
      "dark_souls_2": {
       "delta": 0.1006,
       "median": 0.09805,
       "ci": [
        0.07411,
        0.159
       ],
       "n": 119,
       "wins": 26,
       "ties": 0,
       "clusters": 12,
       "cluster_wins": 0
      }
     }
    },
    {
     "key": "lat_best_lag",
     "label": "Lead of the lateral request over the car's motion",
     "unit": "s",
     "better": "lower",
     "value": 0.7597,
     "median": 0.75,
     "ci": [
      0.7302,
      0.8233
     ],
     "n": 119,
     "clusters": 12,
     "paired": {
      "lebowski_877m": {
       "delta": 0.1374,
       "median": 0.15,
       "ci": [
        0.1041,
        0.2672
       ],
       "n": 119,
       "wins": 22,
       "ties": 9,
       "clusters": 12,
       "cluster_wins": 0
      },
      "dark_souls_2": {
       "delta": 0.05924,
       "median": 0.05,
       "ci": [
        0.03602,
        0.1034
       ],
       "n": 119,
       "wins": 21,
       "ties": 28,
       "clusters": 12,
       "cluster_wins": 0
      }
     }
    },
    {
     "key": "lon_mae",
     "label": "Longitudinal error",
     "unit": "m/s²",
     "better": "lower",
     "value": 0.4393,
     "median": 0.4093,
     "ci": [
      0.2755,
      0.5379
     ],
     "n": 119,
     "clusters": 12,
     "paired": {
      "lebowski_877m": {
       "delta": 0.07064,
       "median": 0.01743,
       "ci": [
        -0.0329,
        0.1292
       ],
       "n": 119,
       "wins": 41,
       "ties": 0,
       "clusters": 12,
       "cluster_wins": 5
      },
      "dark_souls_2": {
       "delta": 0.1137,
       "median": 0.03327,
       "ci": [
        -0.006542,
        0.1883
       ],
       "n": 119,
       "wins": 33,
       "ties": 0,
       "clusters": 12,
       "cluster_wins": 5
      }
     }
    },
    {
     "key": "lon_corr",
     "label": "Longitudinal correlation",
     "unit": "",
     "better": "higher",
     "value": 0.3481,
     "median": 0.3333,
     "ci": [
      0.2923,
      0.5831
     ],
     "n": 119,
     "clusters": 12,
     "paired": {
      "lebowski_877m": {
       "delta": 0.02119,
       "median": 0.03683,
       "ci": [
        -0.03396,
        0.03968
       ],
       "n": 119,
       "wins": 65,
       "ties": 0,
       "clusters": 12,
       "cluster_wins": 6
      },
      "dark_souls_2": {
       "delta": -0.01355,
       "median": -0.007271,
       "ci": [
        -0.04115,
        0.07699
       ],
       "n": 119,
       "wins": 58,
       "ties": 0,
       "clusters": 12,
       "cluster_wins": 6
      }
     }
    },
    {
     "key": "lon_brake_mae",
     "label": "Longitudinal error while braking",
     "unit": "m/s²",
     "better": "lower",
     "value": 0.4092,
     "median": 0.3882,
     "ci": [
      0.3479,
      0.6431
     ],
     "n": 81,
     "clusters": 12,
     "paired": {
      "lebowski_877m": {
       "delta": -0.08963,
       "median": -0.07537,
       "ci": [
        -0.1313,
        0.08148
       ],
       "n": 81,
       "wins": 58,
       "ties": 0,
       "clusters": 12,
       "cluster_wins": 5
      },
      "dark_souls_2": {
       "delta": -0.2609,
       "median": -0.2531,
       "ci": [
        -0.3817,
        0.08774
       ],
       "n": 81,
       "wins": 65,
       "ties": 0,
       "clusters": 12,
       "cluster_wins": 4
      }
     }
    },
    {
     "key": "lon_rate_rms",
     "label": "Longitudinal action rate, RMS (smoothness)",
     "unit": "m/s³",
     "better": "lower",
     "value": 0.5588,
     "median": 0.5106,
     "ci": [
      0.4014,
      0.6307
     ],
     "n": 119,
     "clusters": 12,
     "paired": {
      "lebowski_877m": {
       "delta": -5.051,
       "median": -4.395,
       "ci": [
        -6.085,
        -2.823
       ],
       "n": 119,
       "wins": 119,
       "ties": 0,
       "clusters": 12,
       "cluster_wins": 12
      },
      "dark_souls_2": {
       "delta": -0.03307,
       "median": -0.07143,
       "ci": [
        -0.144,
        0.03326
       ],
       "n": 119,
       "wins": 75,
       "ties": 0,
       "clusters": 12,
       "cluster_wins": 10
      }
     }
    },
    {
     "key": "v3_mae",
     "label": "Plan speed error at 3 s",
     "unit": "m/s",
     "better": "lower",
     "value": 2.816,
     "median": 2.794,
     "ci": [
      1.309,
      3.19
     ],
     "n": 119,
     "clusters": 12,
     "paired": {
      "lebowski_877m": {
       "delta": 0.8192,
       "median": 0.7149,
       "ci": [
        0.1173,
        1.122
       ],
       "n": 119,
       "wins": 27,
       "ties": 0,
       "clusters": 12,
       "cluster_wins": 5
      },
      "dark_souls_2": {
       "delta": 0.3713,
       "median": 0.1275,
       "ci": [
        -0.2545,
        0.7328
       ],
       "n": 119,
       "wins": 43,
       "ties": 0,
       "clusters": 12,
       "cluster_wins": 8
      }
     }
    }
   ],
   "per_unit_file": null,
   "by_group": {
    "stock-aimedup": {
     "n": 4,
     "clusters": 2,
     "metrics": [
      {
       "key": "lat_mae",
       "value": 0.2673,
       "median": 0.2363,
       "ci": null,
       "n": 4,
       "clusters": 2,
       "paired": {
        "lebowski_877m": {
         "delta": 0.1262,
         "median": 0.07297,
         "ci": null,
         "n": 4,
         "wins": 0,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 0
        },
        "dark_souls_2": {
         "delta": 0.1516,
         "median": 0.1195,
         "ci": null,
         "n": 4,
         "wins": 0,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 0
        }
       }
      },
      {
       "key": "lat_mae_bl",
       "value": 0.2571,
       "median": 0.2185,
       "ci": null,
       "n": 4,
       "clusters": 2,
       "paired": {
        "lebowski_877m": {
         "delta": 0.1175,
         "median": 0.05637,
         "ci": null,
         "n": 4,
         "wins": 0,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 0
        },
        "dark_souls_2": {
         "delta": 0.1492,
         "median": 0.1106,
         "ci": null,
         "n": 4,
         "wins": 0,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 0
        }
       }
      },
      {
       "key": "lat_corr",
       "value": 0.7743,
       "median": 0.888,
       "ci": null,
       "n": 4,
       "clusters": 2,
       "paired": {
        "lebowski_877m": {
         "delta": -0.02084,
         "median": -0.009227,
         "ci": null,
         "n": 4,
         "wins": 2,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 1
        },
        "dark_souls_2": {
         "delta": -0.04752,
         "median": -0.0354,
         "ci": null,
         "n": 4,
         "wins": 0,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 0
        }
       }
      },
      {
       "key": "lat_gain",
       "value": 1.319,
       "median": 1.488,
       "ci": null,
       "n": 4,
       "clusters": 2,
       "paired": {
        "lebowski_877m": {
         "delta": 0.205,
         "median": 0.132,
         "ci": null,
         "n": 4,
         "wins": 0,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 0
        },
        "dark_souls_2": {
         "delta": 0.2769,
         "median": 0.3589,
         "ci": null,
         "n": 4,
         "wins": 1,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 0
        }
       }
      },
      {
       "key": "lat_rate_rms",
       "value": 0.7449,
       "median": 0.7647,
       "ci": null,
       "n": 4,
       "clusters": 2,
       "paired": {
        "lebowski_877m": {
         "delta": -0.408,
         "median": -0.4473,
         "ci": null,
         "n": 4,
         "wins": 4,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 2
        },
        "dark_souls_2": {
         "delta": 0.3729,
         "median": 0.3606,
         "ci": null,
         "n": 4,
         "wins": 0,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 0
        }
       }
      },
      {
       "key": "lat_best_lag",
       "value": 0.9125,
       "median": 0.9,
       "ci": null,
       "n": 4,
       "clusters": 2,
       "paired": {
        "lebowski_877m": {
         "delta": 0.4,
         "median": 0.4,
         "ci": null,
         "n": 4,
         "wins": 0,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 0
        },
        "dark_souls_2": {
         "delta": 0.0375,
         "median": 0.025,
         "ci": null,
         "n": 4,
         "wins": 2,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 0
        }
       }
      },
      {
       "key": "lon_mae",
       "value": 0.6163,
       "median": 0.5752,
       "ci": null,
       "n": 4,
       "clusters": 2,
       "paired": {
        "lebowski_877m": {
         "delta": -0.1614,
         "median": -0.1514,
         "ci": null,
         "n": 4,
         "wins": 4,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 2
        },
        "dark_souls_2": {
         "delta": 0.1192,
         "median": 0.1189,
         "ci": null,
         "n": 4,
         "wins": 0,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 0
        }
       }
      },
      {
       "key": "lon_corr",
       "value": 0.5072,
       "median": 0.5111,
       "ci": null,
       "n": 4,
       "clusters": 2,
       "paired": {
        "lebowski_877m": {
         "delta": -0.02822,
         "median": -0.07609,
         "ci": null,
         "n": 4,
         "wins": 2,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 1
        },
        "dark_souls_2": {
         "delta": 0.3307,
         "median": 0.317,
         "ci": null,
         "n": 4,
         "wins": 4,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 2
        }
       }
      },
      {
       "key": "lon_brake_mae",
       "value": 0.2909,
       "median": 0.2909,
       "ci": null,
       "n": 2,
       "clusters": 2,
       "paired": {
        "lebowski_877m": {
         "delta": 0.1029,
         "median": 0.1029,
         "ci": null,
         "n": 2,
         "wins": 0,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 0
        },
        "dark_souls_2": {
         "delta": -0.7228,
         "median": -0.7228,
         "ci": null,
         "n": 2,
         "wins": 2,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 2
        }
       }
      },
      {
       "key": "lon_rate_rms",
       "value": 0.3436,
       "median": 0.3046,
       "ci": null,
       "n": 4,
       "clusters": 2,
       "paired": {
        "lebowski_877m": {
         "delta": -2.951,
         "median": -3.078,
         "ci": null,
         "n": 4,
         "wins": 4,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 2
        },
        "dark_souls_2": {
         "delta": -0.09146,
         "median": -0.09173,
         "ci": null,
         "n": 4,
         "wins": 4,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 2
        }
       }
      },
      {
       "key": "v3_mae",
       "value": 1.657,
       "median": 1.8,
       "ci": null,
       "n": 4,
       "clusters": 2,
       "paired": {
        "lebowski_877m": {
         "delta": -0.7315,
         "median": -0.6648,
         "ci": null,
         "n": 4,
         "wins": 4,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 2
        },
        "dark_souls_2": {
         "delta": -0.6717,
         "median": -0.9345,
         "ci": null,
         "n": 4,
         "wins": 3,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 2
        }
       }
      }
     ]
    },
    "stock-dual": {
     "n": 13,
     "clusters": 4,
     "metrics": [
      {
       "key": "lat_mae",
       "value": 0.192,
       "median": 0.1945,
       "ci": null,
       "n": 13,
       "clusters": 4,
       "paired": {
        "lebowski_877m": {
         "delta": 0.04678,
         "median": 0.0382,
         "ci": null,
         "n": 13,
         "wins": 1,
         "ties": 0,
         "clusters": 4,
         "cluster_wins": 0
        },
        "dark_souls_2": {
         "delta": 0.04956,
         "median": 0.04638,
         "ci": null,
         "n": 13,
         "wins": 2,
         "ties": 0,
         "clusters": 4,
         "cluster_wins": 0
        }
       }
      },
      {
       "key": "lat_mae_bl",
       "value": 0.1869,
       "median": 0.1945,
       "ci": null,
       "n": 13,
       "clusters": 4,
       "paired": {
        "lebowski_877m": {
         "delta": 0.04759,
         "median": 0.05008,
         "ci": null,
         "n": 13,
         "wins": 1,
         "ties": 0,
         "clusters": 4,
         "cluster_wins": 0
        },
        "dark_souls_2": {
         "delta": 0.04855,
         "median": 0.04717,
         "ci": null,
         "n": 13,
         "wins": 2,
         "ties": 0,
         "clusters": 4,
         "cluster_wins": 0
        }
       }
      },
      {
       "key": "lat_corr",
       "value": 0.7982,
       "median": 0.7974,
       "ci": null,
       "n": 13,
       "clusters": 4,
       "paired": {
        "lebowski_877m": {
         "delta": -0.01512,
         "median": -0.02601,
         "ci": null,
         "n": 13,
         "wins": 4,
         "ties": 0,
         "clusters": 4,
         "cluster_wins": 2
        },
        "dark_souls_2": {
         "delta": -0.04719,
         "median": -0.03416,
         "ci": null,
         "n": 13,
         "wins": 2,
         "ties": 0,
         "clusters": 4,
         "cluster_wins": 0
        }
       }
      },
      {
       "key": "lat_gain",
       "value": 1.0,
       "median": 0.9566,
       "ci": null,
       "n": 13,
       "clusters": 4,
       "paired": {
        "lebowski_877m": {
         "delta": 0.0141,
         "median": 0.0226,
         "ci": null,
         "n": 13,
         "wins": 6,
         "ties": 0,
         "clusters": 4,
         "cluster_wins": 2
        },
        "dark_souls_2": {
         "delta": -0.0132,
         "median": -0.007102,
         "ci": null,
         "n": 13,
         "wins": 8,
         "ties": 0,
         "clusters": 4,
         "cluster_wins": 3
        }
       }
      },
      {
       "key": "lat_rate_rms",
       "value": 0.4753,
       "median": 0.4584,
       "ci": null,
       "n": 13,
       "clusters": 4,
       "paired": {
        "lebowski_877m": {
         "delta": -0.7006,
         "median": -0.6394,
         "ci": null,
         "n": 13,
         "wins": 13,
         "ties": 0,
         "clusters": 4,
         "cluster_wins": 4
        },
        "dark_souls_2": {
         "delta": 0.07845,
         "median": 0.09515,
         "ci": null,
         "n": 13,
         "wins": 4,
         "ties": 0,
         "clusters": 4,
         "cluster_wins": 0
        }
       }
      },
      {
       "key": "lat_best_lag",
       "value": 0.7654,
       "median": 0.75,
       "ci": null,
       "n": 13,
       "clusters": 4,
       "paired": {
        "lebowski_877m": {
         "delta": 0.2538,
         "median": 0.25,
         "ci": null,
         "n": 13,
         "wins": 1,
         "ties": 0,
         "clusters": 4,
         "cluster_wins": 0
        },
        "dark_souls_2": {
         "delta": 0.1038,
         "median": 0.1,
         "ci": null,
         "n": 13,
         "wins": 1,
         "ties": 3,
         "clusters": 4,
         "cluster_wins": 0
        }
       }
      },
      {
       "key": "lon_mae",
       "value": 0.2722,
       "median": 0.2672,
       "ci": null,
       "n": 13,
       "clusters": 4,
       "paired": {
        "lebowski_877m": {
         "delta": 0.04393,
         "median": 0.03168,
         "ci": null,
         "n": 13,
         "wins": 0,
         "ties": 0,
         "clusters": 4,
         "cluster_wins": 0
        },
        "dark_souls_2": {
         "delta": -0.009204,
         "median": 0.002898,
         "ci": null,
         "n": 13,
         "wins": 5,
         "ties": 0,
         "clusters": 4,
         "cluster_wins": 2
        }
       }
      },
      {
       "key": "lon_corr",
       "value": 0.5729,
       "median": 0.6983,
       "ci": null,
       "n": 13,
       "clusters": 4,
       "paired": {
        "lebowski_877m": {
         "delta": -0.06125,
         "median": -0.05313,
         "ci": null,
         "n": 13,
         "wins": 3,
         "ties": 0,
         "clusters": 4,
         "cluster_wins": 0
        },
        "dark_souls_2": {
         "delta": -0.002717,
         "median": -0.04751,
         "ci": null,
         "n": 13,
         "wins": 4,
         "ties": 0,
         "clusters": 4,
         "cluster_wins": 2
        }
       }
      },
      {
       "key": "lon_brake_mae",
       "value": 0.7592,
       "median": 0.7751,
       "ci": null,
       "n": 6,
       "clusters": 4,
       "paired": {
        "lebowski_877m": {
         "delta": 0.1119,
         "median": 0.101,
         "ci": null,
         "n": 6,
         "wins": 0,
         "ties": 0,
         "clusters": 4,
         "cluster_wins": 0
        },
        "dark_souls_2": {
         "delta": 0.1773,
         "median": 0.1339,
         "ci": null,
         "n": 6,
         "wins": 0,
         "ties": 0,
         "clusters": 4,
         "cluster_wins": 0
        }
       }
      },
      {
       "key": "lon_rate_rms",
       "value": 0.3557,
       "median": 0.3349,
       "ci": null,
       "n": 13,
       "clusters": 4,
       "paired": {
        "lebowski_877m": {
         "delta": -1.858,
         "median": -1.88,
         "ci": null,
         "n": 13,
         "wins": 13,
         "ties": 0,
         "clusters": 4,
         "cluster_wins": 4
        },
        "dark_souls_2": {
         "delta": -0.06027,
         "median": -0.07143,
         "ci": null,
         "n": 13,
         "wins": 11,
         "ties": 0,
         "clusters": 4,
         "cluster_wins": 3
        }
       }
      },
      {
       "key": "v3_mae",
       "value": 1.143,
       "median": 1.089,
       "ci": null,
       "n": 13,
       "clusters": 4,
       "paired": {
        "lebowski_877m": {
         "delta": 0.3023,
         "median": 0.292,
         "ci": null,
         "n": 13,
         "wins": 2,
         "ties": 0,
         "clusters": 4,
         "cluster_wins": 1
        },
        "dark_souls_2": {
         "delta": 0.03323,
         "median": 0.07194,
         "ci": null,
         "n": 13,
         "wins": 5,
         "ties": 0,
         "clusters": 4,
         "cluster_wins": 2
        }
       }
      }
     ]
    },
    "stock-newcase": {
     "n": 2,
     "clusters": 1,
     "metrics": [
      {
       "key": "lat_mae",
       "value": 0.176,
       "median": 0.176,
       "ci": null,
       "n": 2,
       "clusters": 1,
       "paired": {
        "lebowski_877m": {
         "delta": 0.04172,
         "median": 0.04172,
         "ci": null,
         "n": 2,
         "wins": 0,
         "ties": 0,
         "clusters": 1,
         "cluster_wins": 0
        },
        "dark_souls_2": {
         "delta": 0.00246,
         "median": 0.00246,
         "ci": null,
         "n": 2,
         "wins": 0,
         "ties": 0,
         "clusters": 1,
         "cluster_wins": 0
        }
       }
      },
      {
       "key": "lat_mae_bl",
       "value": 0.1749,
       "median": 0.1749,
       "ci": null,
       "n": 2,
       "clusters": 1,
       "paired": {
        "lebowski_877m": {
         "delta": 0.03976,
         "median": 0.03976,
         "ci": null,
         "n": 2,
         "wins": 0,
         "ties": 0,
         "clusters": 1,
         "cluster_wins": 0
        },
        "dark_souls_2": {
         "delta": 0.004967,
         "median": 0.004967,
         "ci": null,
         "n": 2,
         "wins": 0,
         "ties": 0,
         "clusters": 1,
         "cluster_wins": 0
        }
       }
      },
      {
       "key": "lat_corr",
       "value": 0.8314,
       "median": 0.8314,
       "ci": null,
       "n": 2,
       "clusters": 1,
       "paired": {
        "lebowski_877m": {
         "delta": -0.01616,
         "median": -0.01616,
         "ci": null,
         "n": 2,
         "wins": 1,
         "ties": 0,
         "clusters": 1,
         "cluster_wins": 0
        },
        "dark_souls_2": {
         "delta": -0.01607,
         "median": -0.01607,
         "ci": null,
         "n": 2,
         "wins": 0,
         "ties": 0,
         "clusters": 1,
         "cluster_wins": 0
        }
       }
      },
      {
       "key": "lat_gain",
       "value": 0.8515,
       "median": 0.8515,
       "ci": null,
       "n": 2,
       "clusters": 1,
       "paired": {
        "lebowski_877m": {
         "delta": -0.06158,
         "median": -0.06158,
         "ci": null,
         "n": 2,
         "wins": 1,
         "ties": 0,
         "clusters": 1,
         "cluster_wins": 1
        },
        "dark_souls_2": {
         "delta": -0.04053,
         "median": -0.04053,
         "ci": null,
         "n": 2,
         "wins": 1,
         "ties": 0,
         "clusters": 1,
         "cluster_wins": 1
        }
       }
      },
      {
       "key": "lat_rate_rms",
       "value": 0.678,
       "median": 0.678,
       "ci": null,
       "n": 2,
       "clusters": 1,
       "paired": {
        "lebowski_877m": {
         "delta": -0.5976,
         "median": -0.5976,
         "ci": null,
         "n": 2,
         "wins": 2,
         "ties": 0,
         "clusters": 1,
         "cluster_wins": 1
        },
        "dark_souls_2": {
         "delta": 0.05168,
         "median": 0.05168,
         "ci": null,
         "n": 2,
         "wins": 0,
         "ties": 0,
         "clusters": 1,
         "cluster_wins": 0
        }
       }
      },
      {
       "key": "lat_best_lag",
       "value": 0.65,
       "median": 0.65,
       "ci": null,
       "n": 2,
       "clusters": 1,
       "paired": {
        "lebowski_877m": {
         "delta": 0.15,
         "median": 0.15,
         "ci": null,
         "n": 2,
         "wins": 0,
         "ties": 0,
         "clusters": 1,
         "cluster_wins": 0
        },
        "dark_souls_2": {
         "delta": 0.1,
         "median": 0.1,
         "ci": null,
         "n": 2,
         "wins": 0,
         "ties": 0,
         "clusters": 1,
         "cluster_wins": 0
        }
       }
      },
      {
       "key": "lon_mae",
       "value": 0.3639,
       "median": 0.3639,
       "ci": null,
       "n": 2,
       "clusters": 1,
       "paired": {
        "lebowski_877m": {
         "delta": 0.001431,
         "median": 0.001431,
         "ci": null,
         "n": 2,
         "wins": 0,
         "ties": 0,
         "clusters": 1,
         "cluster_wins": 0
        },
        "dark_souls_2": {
         "delta": -0.05202,
         "median": -0.05202,
         "ci": null,
         "n": 2,
         "wins": 2,
         "ties": 0,
         "clusters": 1,
         "cluster_wins": 1
        }
       }
      },
      {
       "key": "lon_corr",
       "value": 0.5602,
       "median": 0.5602,
       "ci": null,
       "n": 2,
       "clusters": 1,
       "paired": {
        "lebowski_877m": {
         "delta": 0.07357,
         "median": 0.07357,
         "ci": null,
         "n": 2,
         "wins": 2,
         "ties": 0,
         "clusters": 1,
         "cluster_wins": 1
        },
        "dark_souls_2": {
         "delta": 0.09202,
         "median": 0.09202,
         "ci": null,
         "n": 2,
         "wins": 2,
         "ties": 0,
         "clusters": 1,
         "cluster_wins": 1
        }
       }
      },
      {
       "key": "lon_brake_mae",
       "value": 0.4521,
       "median": 0.4521,
       "ci": null,
       "n": 2,
       "clusters": 1,
       "paired": {
        "lebowski_877m": {
         "delta": -0.05124,
         "median": -0.05124,
         "ci": null,
         "n": 2,
         "wins": 2,
         "ties": 0,
         "clusters": 1,
         "cluster_wins": 1
        },
        "dark_souls_2": {
         "delta": 0.1293,
         "median": 0.1293,
         "ci": null,
         "n": 2,
         "wins": 0,
         "ties": 0,
         "clusters": 1,
         "cluster_wins": 0
        }
       }
      },
      {
       "key": "lon_rate_rms",
       "value": 0.522,
       "median": 0.522,
       "ci": null,
       "n": 2,
       "clusters": 1,
       "paired": {
        "lebowski_877m": {
         "delta": -2.834,
         "median": -2.834,
         "ci": null,
         "n": 2,
         "wins": 2,
         "ties": 0,
         "clusters": 1,
         "cluster_wins": 1
        },
        "dark_souls_2": {
         "delta": -0.07786,
         "median": -0.07786,
         "ci": null,
         "n": 2,
         "wins": 2,
         "ties": 0,
         "clusters": 1,
         "cluster_wins": 1
        }
       }
      },
      {
       "key": "v3_mae",
       "value": 1.793,
       "median": 1.793,
       "ci": null,
       "n": 2,
       "clusters": 1,
       "paired": {
        "lebowski_877m": {
         "delta": -0.3011,
         "median": -0.3011,
         "ci": null,
         "n": 2,
         "wins": 2,
         "ties": 0,
         "clusters": 1,
         "cluster_wins": 1
        },
        "dark_souls_2": {
         "delta": -0.3544,
         "median": -0.3544,
         "ci": null,
         "n": 2,
         "wins": 2,
         "ties": 0,
         "clusters": 1,
         "cluster_wins": 1
        }
       }
      }
     ]
    },
    "v13-8mm": {
     "n": 92,
     "clusters": 3,
     "metrics": [
      {
       "key": "lat_mae",
       "value": 0.2394,
       "median": 0.224,
       "ci": null,
       "n": 92,
       "clusters": 3,
       "paired": {
        "lebowski_877m": {
         "delta": 0.05785,
         "median": 0.04393,
         "ci": null,
         "n": 92,
         "wins": 12,
         "ties": 0,
         "clusters": 3,
         "cluster_wins": 0
        },
        "dark_souls_2": {
         "delta": 0.05921,
         "median": 0.05566,
         "ci": null,
         "n": 92,
         "wins": 1,
         "ties": 0,
         "clusters": 3,
         "cluster_wins": 0
        }
       }
      },
      {
       "key": "lat_mae_bl",
       "value": 0.2349,
       "median": 0.2225,
       "ci": null,
       "n": 92,
       "clusters": 3,
       "paired": {
        "lebowski_877m": {
         "delta": 0.06275,
         "median": 0.04911,
         "ci": null,
         "n": 92,
         "wins": 11,
         "ties": 0,
         "clusters": 3,
         "cluster_wins": 0
        },
        "dark_souls_2": {
         "delta": 0.05866,
         "median": 0.05452,
         "ci": null,
         "n": 92,
         "wins": 2,
         "ties": 0,
         "clusters": 3,
         "cluster_wins": 0
        }
       }
      },
      {
       "key": "lat_corr",
       "value": 0.7808,
       "median": 0.8605,
       "ci": null,
       "n": 92,
       "clusters": 3,
       "paired": {
        "lebowski_877m": {
         "delta": 0.03233,
         "median": 0.007223,
         "ci": null,
         "n": 92,
         "wins": 55,
         "ties": 0,
         "clusters": 3,
         "cluster_wins": 2
        },
        "dark_souls_2": {
         "delta": -0.04269,
         "median": -0.03323,
         "ci": null,
         "n": 92,
         "wins": 14,
         "ties": 0,
         "clusters": 3,
         "cluster_wins": 0
        }
       }
      },
      {
       "key": "lat_gain",
       "value": 0.9964,
       "median": 0.8725,
       "ci": null,
       "n": 92,
       "clusters": 3,
       "paired": {
        "lebowski_877m": {
         "delta": 0.033,
         "median": 0.005379,
         "ci": null,
         "n": 92,
         "wins": 45,
         "ties": 0,
         "clusters": 3,
         "cluster_wins": 1
        },
        "dark_souls_2": {
         "delta": 0.09309,
         "median": 0.05253,
         "ci": null,
         "n": 92,
         "wins": 25,
         "ties": 0,
         "clusters": 3,
         "cluster_wins": 0
        }
       }
      },
      {
       "key": "lat_rate_rms",
       "value": 0.8998,
       "median": 0.8075,
       "ci": null,
       "n": 92,
       "clusters": 3,
       "paired": {
        "lebowski_877m": {
         "delta": -1.718,
         "median": -1.538,
         "ci": null,
         "n": 92,
         "wins": 92,
         "ties": 0,
         "clusters": 3,
         "cluster_wins": 3
        },
        "dark_souls_2": {
         "delta": 0.09809,
         "median": 0.109,
         "ci": null,
         "n": 92,
         "wins": 21,
         "ties": 0,
         "clusters": 3,
         "cluster_wins": 0
        }
       }
      },
      {
       "key": "lat_best_lag",
       "value": 0.756,
       "median": 0.7,
       "ci": null,
       "n": 92,
       "clusters": 3,
       "paired": {
        "lebowski_877m": {
         "delta": 0.09565,
         "median": 0.15,
         "ci": null,
         "n": 92,
         "wins": 21,
         "ties": 9,
         "clusters": 3,
         "cluster_wins": 0
        },
        "dark_souls_2": {
         "delta": 0.05109,
         "median": 0.05,
         "ci": null,
         "n": 92,
         "wins": 18,
         "ties": 21,
         "clusters": 3,
         "cluster_wins": 0
        }
       }
      },
      {
       "key": "lon_mae",
       "value": 0.4649,
       "median": 0.4369,
       "ci": null,
       "n": 92,
       "clusters": 3,
       "paired": {
        "lebowski_877m": {
         "delta": 0.08888,
         "median": 0.01959,
         "ci": null,
         "n": 92,
         "wins": 35,
         "ties": 0,
         "clusters": 3,
         "cluster_wins": 2
        },
        "dark_souls_2": {
         "delta": 0.1451,
         "median": 0.05936,
         "ci": null,
         "n": 92,
         "wins": 22,
         "ties": 0,
         "clusters": 3,
         "cluster_wins": 1
        }
       }
      },
      {
       "key": "lon_corr",
       "value": 0.2724,
       "median": 0.2954,
       "ci": null,
       "n": 92,
       "clusters": 3,
       "paired": {
        "lebowski_877m": {
         "delta": 0.03873,
         "median": 0.05942,
         "ci": null,
         "n": 92,
         "wins": 56,
         "ties": 0,
         "clusters": 3,
         "cluster_wins": 3
        },
        "dark_souls_2": {
         "delta": -0.02946,
         "median": -0.0002073,
         "ci": null,
         "n": 92,
         "wins": 46,
         "ties": 0,
         "clusters": 3,
         "cluster_wins": 1
        }
       }
      },
      {
       "key": "lon_brake_mae",
       "value": 0.3568,
       "median": 0.3267,
       "ci": null,
       "n": 65,
       "clusters": 3,
       "paired": {
        "lebowski_877m": {
         "delta": -0.1323,
         "median": -0.1244,
         "ci": null,
         "n": 65,
         "wins": 55,
         "ties": 0,
         "clusters": 3,
         "cluster_wins": 3
        },
        "dark_souls_2": {
         "delta": -0.3359,
         "median": -0.3352,
         "ci": null,
         "n": 65,
         "wins": 62,
         "ties": 0,
         "clusters": 3,
         "cluster_wins": 2
        }
       }
      },
      {
       "key": "lon_rate_rms",
       "value": 0.6018,
       "median": 0.5287,
       "ci": null,
       "n": 92,
       "clusters": 3,
       "paired": {
        "lebowski_877m": {
         "delta": -5.731,
         "median": -5.435,
         "ci": null,
         "n": 92,
         "wins": 92,
         "ties": 0,
         "clusters": 3,
         "cluster_wins": 3
        },
        "dark_souls_2": {
         "delta": -0.02209,
         "median": -0.04638,
         "ci": null,
         "n": 92,
         "wins": 51,
         "ties": 0,
         "clusters": 3,
         "cluster_wins": 2
        }
       }
      },
      {
       "key": "v3_mae",
       "value": 3.278,
       "median": 3.164,
       "ci": null,
       "n": 92,
       "clusters": 3,
       "paired": {
        "lebowski_877m": {
         "delta": 1.044,
         "median": 0.96,
         "ci": null,
         "n": 92,
         "wins": 16,
         "ties": 0,
         "clusters": 3,
         "cluster_wins": 0
        },
        "dark_souls_2": {
         "delta": 0.5093,
         "median": 0.2253,
         "ci": null,
         "n": 92,
         "wins": 30,
         "ties": 0,
         "clusters": 3,
         "cluster_wins": 2
        }
       }
      }
     ]
    },
    "v14-8mm": {
     "n": 8,
     "clusters": 2,
     "metrics": [
      {
       "key": "lat_mae",
       "value": 0.1878,
       "median": 0.1852,
       "ci": null,
       "n": 8,
       "clusters": 2,
       "paired": {
        "lebowski_877m": {
         "delta": 0.04643,
         "median": 0.04551,
         "ci": null,
         "n": 8,
         "wins": 0,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 0
        },
        "dark_souls_2": {
         "delta": 0.03429,
         "median": 0.03592,
         "ci": null,
         "n": 8,
         "wins": 0,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 0
        }
       }
      },
      {
       "key": "lat_mae_bl",
       "value": 0.1886,
       "median": 0.1883,
       "ci": null,
       "n": 8,
       "clusters": 2,
       "paired": {
        "lebowski_877m": {
         "delta": 0.05146,
         "median": 0.05228,
         "ci": null,
         "n": 8,
         "wins": 0,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 0
        },
        "dark_souls_2": {
         "delta": 0.03496,
         "median": 0.03959,
         "ci": null,
         "n": 8,
         "wins": 0,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 0
        }
       }
      },
      {
       "key": "lat_corr",
       "value": 0.8164,
       "median": 0.9035,
       "ci": null,
       "n": 8,
       "clusters": 2,
       "paired": {
        "lebowski_877m": {
         "delta": -0.05162,
         "median": -0.02725,
         "ci": null,
         "n": 8,
         "wins": 2,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 0
        },
        "dark_souls_2": {
         "delta": -0.0286,
         "median": -0.02696,
         "ci": null,
         "n": 8,
         "wins": 0,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 0
        }
       }
      },
      {
       "key": "lat_gain",
       "value": 0.9637,
       "median": 0.9783,
       "ci": null,
       "n": 8,
       "clusters": 2,
       "paired": {
        "lebowski_877m": {
         "delta": 0.04516,
         "median": 0.04855,
         "ci": null,
         "n": 8,
         "wins": 1,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 0
        },
        "dark_souls_2": {
         "delta": 0.01416,
         "median": 0.008526,
         "ci": null,
         "n": 8,
         "wins": 2,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 1
        }
       }
      },
      {
       "key": "lat_rate_rms",
       "value": 0.5115,
       "median": 0.4405,
       "ci": null,
       "n": 8,
       "clusters": 2,
       "paired": {
        "lebowski_877m": {
         "delta": -0.6441,
         "median": -0.4901,
         "ci": null,
         "n": 8,
         "wins": 8,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 2
        },
        "dark_souls_2": {
         "delta": 0.0409,
         "median": 0.03622,
         "ci": null,
         "n": 8,
         "wins": 1,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 0
        }
       }
      },
      {
       "key": "lat_best_lag",
       "value": 0.7438,
       "median": 0.625,
       "ci": null,
       "n": 8,
       "clusters": 2,
       "paired": {
        "lebowski_877m": {
         "delta": 0.2938,
         "median": 0.2,
         "ci": null,
         "n": 8,
         "wins": 0,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 0
        },
        "dark_souls_2": {
         "delta": 0.08125,
         "median": 0.025,
         "ci": null,
         "n": 8,
         "wins": 0,
         "ties": 4,
         "clusters": 2,
         "cluster_wins": 0
        }
       }
      },
      {
       "key": "lon_mae",
       "value": 0.347,
       "median": 0.3117,
       "ci": null,
       "n": 8,
       "clusters": 2,
       "paired": {
        "lebowski_877m": {
         "delta": 0.03762,
         "median": 0.0379,
         "ci": null,
         "n": 8,
         "wins": 2,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 1
        },
        "dark_souls_2": {
         "delta": -0.01019,
         "median": 0.004769,
         "ci": null,
         "n": 8,
         "wins": 4,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 1
        }
       }
      },
      {
       "key": "lon_corr",
       "value": 0.7197,
       "median": 0.7707,
       "ci": null,
       "n": 8,
       "clusters": 2,
       "paired": {
        "lebowski_877m": {
         "delta": -0.03497,
         "median": -0.0243,
         "ci": null,
         "n": 8,
         "wins": 2,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 1
        },
        "dark_souls_2": {
         "delta": -0.0467,
         "median": -0.0306,
         "ci": null,
         "n": 8,
         "wins": 2,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 0
        }
       }
      },
      {
       "key": "lon_brake_mae",
       "value": 0.6515,
       "median": 0.6918,
       "ci": null,
       "n": 6,
       "clusters": 2,
       "paired": {
        "lebowski_877m": {
         "delta": 0.09393,
         "median": 0.1026,
         "ci": null,
         "n": 6,
         "wins": 1,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 1
        },
        "dark_souls_2": {
         "delta": 0.1368,
         "median": 0.1471,
         "ci": null,
         "n": 6,
         "wins": 1,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 0
        }
       }
      },
      {
       "key": "lon_rate_rms",
       "value": 0.5115,
       "median": 0.4581,
       "ci": null,
       "n": 8,
       "clusters": 2,
       "paired": {
        "lebowski_877m": {
         "delta": -4.022,
         "median": -3.763,
         "ci": null,
         "n": 8,
         "wins": 8,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 2
        },
        "dark_souls_2": {
         "delta": -0.07468,
         "median": -0.07766,
         "ci": null,
         "n": 8,
         "wins": 7,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 2
        }
       }
      },
      {
       "key": "v3_mae",
       "value": 1.061,
       "median": 1.031,
       "ci": null,
       "n": 8,
       "clusters": 2,
       "paired": {
        "lebowski_877m": {
         "delta": 0.1284,
         "median": 0.1101,
         "ci": null,
         "n": 8,
         "wins": 3,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 1
        },
        "dark_souls_2": {
         "delta": 0.03633,
         "median": 0.0478,
         "ci": null,
         "n": 8,
         "wins": 3,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 1
        }
       }
      }
     ]
    }
   }
  },
  "truck_closed_loop": {
   "status": "not_run",
   "unit_of_evidence": "window",
   "n": 0,
   "condition": "Measured. worldmodel-4B as in the public closed loop, but the model drives a simulation of the G55's measured steering and powertrain response instead of the ideal car. The simulation starts from the truck's state (speed, yaw rate, steering, gear) in its CAN recording of the 2 s before the window; one window, whose 2 s before are not all valid CAN, starts steady at its recorded speed. Lane-following windows from the G55's drives, day and night; distance is to the driver's recorded path; seeds 0 and 1, averaged per window.",
   "baselines": [],
   "metrics": [],
   "per_unit_file": null
  },
  "truck_recovery": {
   "status": "not_run",
   "unit_of_evidence": "window",
   "n": 0,
   "condition": "Measured. The truck closed loop with a disturbance while the model drives; only the tests listed ran.",
   "baselines": [],
   "metrics": [],
   "per_unit_file": null
  }
 },
 "stills": [],
 "limits": [
  "Open-loop scores compare a model's action with what the recording driver or car did; a model that drives differently but well is scored as wrong. comma's RL residual makes the lateral action smoother, and earlier than its base's by 0.10 to 0.26 s (measured on the G55 and on 16 of comma's cars); both score as error here. Measured on the 2026-10-27 deep dive: open-loop accuracy did not predict closed-loop driving.",
  "Closed-loop numbers come from a generative world model, not a car. In a separate test on the G55's windows, worldmodel-4B followed an added yaw command through 2 s but showed 0 to 20% of an added lateral offset 3 to 4 s later; past about 4 s the recorded anchor frames also pull every model back toward the recorded path. The front page reports 2 s.",
  "Intervals are 95% percentile bootstraps that resample whole drives (the drive a public segment was cut from, the recorded drive a truck window comes from), so segments of one drive count once. Below 10 drives there is no interval: the braking-lead test comes from one drive and reports means and win counts only.",
  "Paired deltas are model minus baseline on the units both have (for lateral gain, the change in its distance from 1); wins count units where the model is better in the metric's direction.",
  "The lead of the lateral request is the shift, searched from 0 to 1.5 s, at which the request best correlates with the car's later lateral motion; a larger value means the request comes earlier. A value at either edge is censored and left out. Plan speed error at 3 s reads the plan head, which does not separate models built on one frozen base: it is identical on every segment for the four f22e06e5 models, and for Cinque v2 and v3.",
  "Truck open loop comes from few drives, and the camera setup changed between them (lens, housing, aim); the pooled number mixes setups, so read the per-setup numbers too. The 8 mm setups' wide camera recorded about 7-10 fps, so its frame can be up to ~60 ms from the narrow one. On the V14 setup up to 7 of a window's 700 steps are ~100 ms (one camera frame dropped). The camera's aim is the truck's live calibration learned on each drive; drives whose recorded calibration was carried over from another drive are left out.",
  "A historical model, run outside comma's software with its own release's inputs, camera warp and command (engines/legacy.py). Checked against comma's CI model replay of this model (made on a comma device, 60 frames of one route): the plan's lateral position is 0.025 m RMS from comma's output; every other model here is 1.23 m or more from it on the same check.",
  "Trained for the comma devices and cameras of its time, not for the G55's camera or for every camera among comma's public segments: of those 45 segments 26 record the OX03C10 sensor, 6 the AR0231, 4 the OS04C10 and 9 none; 30 were recorded 2021 to 2024 and 15 come from devices whose route names carry no date.",
  "Not run in closed loop: the G55 closed-loop and recovery tests need this release's controller inside the world-model loop, which is not built yet.",
  "The commands are 0.10's modeld (get_action_from_model on the plan, lateral delay = steer actuator delay + 0.2 s, smoothed over 0.1 s lateral and 0.3 s longitudinal); the newest models are scored on their own unsmoothed action head, so lateral action rates are not comparable across these eras."
 ]
}