{
 "schema": 1,
 "model": {
  "id": "dark_souls_2",
  "name": "Dark Souls 2",
  "sha256": "1dc66bc06f250b577653ccbeaa2c6521b3d46749f601d0a1a366419e929ca438+f8fe9a71b0fd428a045a82ed50790179f77aa664391198f078e11e7b2cb2c2d7",
  "source": {
   "kind": "openpilot-master",
   "ref": "9421e1cbfebdb2970afd777cf4642bc5af5795cf",
   "url": "https://github.com/commaai/openpilot/pull/36849",
   "date": "2025-12-12"
  },
  "first_public": "2025-12-12",
  "on_master": "2025-12-12",
  "params_m": null,
  "gmac": null,
  "family": "vision-policy",
  "notes": "In the openpilot 0.10.3 tag. Plan only, commands from modeld's get_action_from_model. Vision 6a7d09ad/200, policy 4b78e2e6/400."
 },
 "scored": {
  "date": "2026-10-28T04:04:40Z",
  "suite": "geval-1",
  "code": "d8eb40eeba87"
 },
 "baselines": [
  "lebowski_877m",
  "the_cool_peoples"
 ],
 "sections": {
  "public_open_loop": {
   "status": "ok",
   "unit_of_evidence": "segment",
   "n": 39,
   "clusters": 23,
   "condition": "Measured. comma's public two-camera segments (openpilot CI data), native intrinsics and the device's own calibration, both cameras; every model on the same decoded frames. Truth is what the recording car did next: carState speed and acceleration, yaw rate from the device's cameraOdometry. Segments where the car never moves have no data. 39 of the 45 listed segments have a score, from 23 drives and 20 cars.",
   "baselines": [
    "lebowski_877m",
    "the_cool_peoples"
   ],
   "metrics": [
    {
     "key": "lat_mae",
     "label": "Lateral error",
     "unit": "m/s²",
     "better": "lower",
     "value": 0.1294,
     "median": 0.09522,
     "ci": [
      0.09924,
      0.1633
     ],
     "n": 39,
     "clusters": 23,
     "paired": {
      "lebowski_877m": {
       "delta": 0.02073,
       "median": 0.01616,
       "ci": [
        0.01035,
        0.03274
       ],
       "n": 39,
       "wins": 7,
       "ties": 0,
       "clusters": 23,
       "cluster_wins": 4
      },
      "the_cool_peoples": {
       "delta": 0.00575,
       "median": 0.00275,
       "ci": [
        0.001619,
        0.01101
       ],
       "n": 39,
       "wins": 15,
       "ties": 0,
       "clusters": 23,
       "cluster_wins": 6
      }
     }
    },
    {
     "key": "lat_mae_bl",
     "label": "Lateral error at the model's own best lead",
     "unit": "m/s²",
     "better": "lower",
     "value": 0.1255,
     "median": 0.09467,
     "ci": [
      0.09526,
      0.1586
     ],
     "n": 39,
     "clusters": 23,
     "paired": {
      "lebowski_877m": {
       "delta": 0.0241,
       "median": 0.01854,
       "ci": [
        0.01492,
        0.035
       ],
       "n": 39,
       "wins": 7,
       "ties": 0,
       "clusters": 23,
       "cluster_wins": 2
      },
      "the_cool_peoples": {
       "delta": 0.006903,
       "median": 0.004826,
       "ci": [
        0.001574,
        0.01364
       ],
       "n": 39,
       "wins": 15,
       "ties": 0,
       "clusters": 23,
       "cluster_wins": 6
      }
     }
    },
    {
     "key": "lat_corr",
     "label": "Lateral correlation with the car's later motion",
     "unit": "",
     "better": "higher",
     "value": 0.8118,
     "median": 0.9321,
     "ci": [
      0.7217,
      0.8903
     ],
     "n": 39,
     "clusters": 23,
     "paired": {
      "lebowski_877m": {
       "delta": -0.0458,
       "median": -0.01946,
       "ci": [
        -0.1195,
        -0.00557
       ],
       "n": 39,
       "wins": 11,
       "ties": 0,
       "clusters": 23,
       "cluster_wins": 7
      },
      "the_cool_peoples": {
       "delta": -0.01768,
       "median": -0.003134,
       "ci": [
        -0.03854,
        -0.001526
       ],
       "n": 39,
       "wins": 12,
       "ties": 0,
       "clusters": 23,
       "cluster_wins": 5
      }
     }
    },
    {
     "key": "lat_gain",
     "label": "Lateral gain (1 = same size as the car's motion)",
     "unit": "",
     "better": "near1",
     "value": 0.8103,
     "median": 0.8923,
     "ci": [
      0.7031,
      0.9146
     ],
     "n": 39,
     "clusters": 23,
     "paired": {
      "lebowski_877m": {
       "delta": 0.03279,
       "median": 0.005498,
       "ci": [
        -0.005259,
        0.07819
       ],
       "n": 39,
       "wins": 17,
       "ties": 0,
       "clusters": 23,
       "cluster_wins": 8
      },
      "the_cool_peoples": {
       "delta": 0.03897,
       "median": -0.00118,
       "ci": [
        -0.003555,
        0.09978
       ],
       "n": 39,
       "wins": 20,
       "ties": 0,
       "clusters": 23,
       "cluster_wins": 8
      }
     }
    },
    {
     "key": "lat_rate_rms",
     "label": "Lateral action rate, RMS (smoothness)",
     "unit": "m/s³",
     "better": "lower",
     "value": 0.4546,
     "median": 0.4159,
     "ci": [
      0.3608,
      0.5598
     ],
     "n": 39,
     "clusters": 23,
     "paired": {
      "lebowski_877m": {
       "delta": -0.2093,
       "median": -0.1888,
       "ci": [
        -0.2464,
        -0.1675
       ],
       "n": 39,
       "wins": 38,
       "ties": 0,
       "clusters": 23,
       "cluster_wins": 22
      },
      "the_cool_peoples": {
       "delta": 0.01188,
       "median": 0.005346,
       "ci": [
        0.000976,
        0.02438
       ],
       "n": 39,
       "wins": 15,
       "ties": 0,
       "clusters": 23,
       "cluster_wins": 6
      }
     }
    },
    {
     "key": "lat_best_lag",
     "label": "Lead of the lateral request over the car's motion",
     "unit": "s",
     "better": "lower",
     "value": 0.5338,
     "median": 0.45,
     "ci": [
      0.436,
      0.6431
     ],
     "n": 34,
     "clusters": 21,
     "paired": {
      "lebowski_877m": {
       "delta": 0.1437,
       "median": 0.1,
       "ci": [
        0.09473,
        0.2037
       ],
       "n": 32,
       "wins": 2,
       "ties": 5,
       "clusters": 20,
       "cluster_wins": 1
      },
      "the_cool_peoples": {
       "delta": 0.03824,
       "median": 0.05,
       "ci": [
        0.01405,
        0.06429
       ],
       "n": 34,
       "wins": 6,
       "ties": 9,
       "clusters": 21,
       "cluster_wins": 5
      }
     }
    },
    {
     "key": "lon_mae",
     "label": "Longitudinal error",
     "unit": "m/s²",
     "better": "lower",
     "value": 0.3348,
     "median": 0.3114,
     "ci": [
      0.2904,
      0.3853
     ],
     "n": 39,
     "clusters": 23,
     "paired": {
      "lebowski_877m": {
       "delta": 0.0267,
       "median": 0.02979,
       "ci": [
        0.01177,
        0.04157
       ],
       "n": 39,
       "wins": 9,
       "ties": 0,
       "clusters": 23,
       "cluster_wins": 3
      },
      "the_cool_peoples": {
       "delta": 0.001951,
       "median": 0.001905,
       "ci": [
        -0.007839,
        0.01099
       ],
       "n": 39,
       "wins": 19,
       "ties": 0,
       "clusters": 23,
       "cluster_wins": 11
      }
     }
    },
    {
     "key": "lon_corr",
     "label": "Longitudinal correlation",
     "unit": "",
     "better": "higher",
     "value": 0.6836,
     "median": 0.8016,
     "ci": [
      0.5812,
      0.7732
     ],
     "n": 39,
     "clusters": 23,
     "paired": {
      "lebowski_877m": {
       "delta": -0.0341,
       "median": -0.01869,
       "ci": [
        -0.06038,
        -0.01192
       ],
       "n": 39,
       "wins": 9,
       "ties": 0,
       "clusters": 23,
       "cluster_wins": 7
      },
      "the_cool_peoples": {
       "delta": -0.01852,
       "median": -0.00727,
       "ci": [
        -0.04018,
        -0.001521
       ],
       "n": 39,
       "wins": 15,
       "ties": 0,
       "clusters": 23,
       "cluster_wins": 8
      }
     }
    },
    {
     "key": "lon_brake_mae",
     "label": "Longitudinal error while braking",
     "unit": "m/s²",
     "better": "lower",
     "value": 0.4339,
     "median": 0.3924,
     "ci": [
      0.3614,
      0.5119
     ],
     "n": 30,
     "clusters": 21,
     "paired": {
      "lebowski_877m": {
       "delta": -0.0003794,
       "median": -0.009371,
       "ci": [
        -0.03538,
        0.03525
       ],
       "n": 30,
       "wins": 17,
       "ties": 0,
       "clusters": 21,
       "cluster_wins": 13
      },
      "the_cool_peoples": {
       "delta": 0.006703,
       "median": 0.009738,
       "ci": [
        -0.01957,
        0.02983
       ],
       "n": 30,
       "wins": 13,
       "ties": 0,
       "clusters": 21,
       "cluster_wins": 9
      }
     }
    },
    {
     "key": "lon_rate_rms",
     "label": "Longitudinal action rate, RMS (smoothness)",
     "unit": "m/s³",
     "better": "lower",
     "value": 0.4133,
     "median": 0.379,
     "ci": [
      0.3825,
      0.4492
     ],
     "n": 39,
     "clusters": 23,
     "paired": {
      "lebowski_877m": {
       "delta": -0.9197,
       "median": -0.9422,
       "ci": [
        -1.078,
        -0.7771
       ],
       "n": 39,
       "wins": 39,
       "ties": 0,
       "clusters": 23,
       "cluster_wins": 23
      },
      "the_cool_peoples": {
       "delta": 0.02178,
       "median": 0.01923,
       "ci": [
        0.0008416,
        0.04012
       ],
       "n": 39,
       "wins": 9,
       "ties": 0,
       "clusters": 23,
       "cluster_wins": 8
      }
     }
    },
    {
     "key": "v3_mae",
     "label": "Plan speed error at 3 s",
     "unit": "m/s",
     "better": "lower",
     "value": 0.9632,
     "median": 0.9125,
     "ci": [
      0.832,
      1.117
     ],
     "n": 39,
     "clusters": 23,
     "paired": {
      "lebowski_877m": {
       "delta": 0.2475,
       "median": 0.211,
       "ci": [
        0.178,
        0.3117
       ],
       "n": 39,
       "wins": 4,
       "ties": 0,
       "clusters": 23,
       "cluster_wins": 1
      },
      "the_cool_peoples": {
       "delta": 0.04729,
       "median": 0.02479,
       "ci": [
        0.002126,
        0.09236
       ],
       "n": 39,
       "wins": 15,
       "ties": 0,
       "clusters": 23,
       "cluster_wins": 8
      }
     }
    }
   ],
   "per_unit_file": "per_unit/dark_souls_2/public_open_loop.csv"
  },
  "public_closed_loop": {
   "status": "not_run",
   "unit_of_evidence": "segment",
   "n": 0,
   "condition": "Measured. comma's worldmodel-4B generates each next frame at 5 Hz from 9 history frames, 5 recorded anchor frames and the car's pose; the model drives comma's ideal car (the requested curvature and acceleration happen at once) for 7.2 s from the recorded start. Distance is to the recording car's own path. One window per segment, nearest the middle, where the car holds at least 5 m/s for the window and the 4.8 s before it; segments with no such stretch are left out. Seeds 0 and 1, averaged per segment.",
   "baselines": [],
   "metrics": [],
   "per_unit_file": null
  },
  "truck_open_loop": {
   "status": "ok",
   "unit_of_evidence": "window",
   "n": 119,
   "clusters": 12,
   "condition": "Measured. The G55's recorded drives replayed through the model with both of the truck's cameras (each setup's own wide-camera calibration), road-plane warp to 1.45 m. Where the narrow lens does not cover comma's narrow view (the 8 mm lens in the V13 case), those pixels come from the wide camera, brightness matched. Truth is the truck's CAN speed and yaw rate. 30 s windows after a 5 s warm-up; window = unit, drive = cluster; results are also given per camera setup. Corpus windows of 30 s from the truck's recorded drives, scored per camera setup.",
   "baselines": [
    "lebowski_877m",
    "the_cool_peoples"
   ],
   "metrics": [
    {
     "key": "lat_mae",
     "label": "Lateral error",
     "unit": "m/s²",
     "better": "lower",
     "value": 0.172,
     "median": 0.1626,
     "ci": [
      0.145,
      0.1894
     ],
     "n": 119,
     "clusters": 12,
     "paired": {
      "lebowski_877m": {
       "delta": -0.0007356,
       "median": -0.004906,
       "ci": [
        -0.01607,
        0.02545
       ],
       "n": 119,
       "wins": 66,
       "ties": 0,
       "clusters": 12,
       "cluster_wins": 5
      },
      "the_cool_peoples": {
       "delta": 0.004043,
       "median": 0.00531,
       "ci": [
        -0.0005742,
        0.008421
       ],
       "n": 119,
       "wins": 49,
       "ties": 0,
       "clusters": 12,
       "cluster_wins": 4
      }
     }
    },
    {
     "key": "lat_mae_bl",
     "label": "Lateral error at the model's own best lead",
     "unit": "m/s²",
     "better": "lower",
     "value": 0.1682,
     "median": 0.1632,
     "ci": [
      0.1414,
      0.1866
     ],
     "n": 119,
     "clusters": 12,
     "paired": {
      "lebowski_877m": {
       "delta": 0.003684,
       "median": 0.00137,
       "ci": [
        -0.01197,
        0.03046
       ],
       "n": 119,
       "wins": 59,
       "ties": 0,
       "clusters": 12,
       "cluster_wins": 4
      },
      "the_cool_peoples": {
       "delta": 0.004484,
       "median": 0.00634,
       "ci": [
        -0.001188,
        0.008031
       ],
       "n": 119,
       "wins": 53,
       "ties": 0,
       "clusters": 12,
       "cluster_wins": 4
      }
     }
    },
    {
     "key": "lat_corr",
     "label": "Lateral correlation with the car's later motion",
     "unit": "",
     "better": "higher",
     "value": 0.8277,
     "median": 0.9069,
     "ci": [
      0.7861,
      0.8489
     ],
     "n": 119,
     "clusters": 12,
     "paired": {
      "lebowski_877m": {
       "delta": 0.06085,
       "median": 0.02707,
       "ci": [
        0.0166,
        0.07405
       ],
       "n": 119,
       "wins": 94,
       "ties": 0,
       "clusters": 12,
       "cluster_wins": 7
      },
      "the_cool_peoples": {
       "delta": -0.004203,
       "median": -0.0007657,
       "ci": [
        -0.01516,
        0.0002377
       ],
       "n": 119,
       "wins": 58,
       "ties": 0,
       "clusters": 12,
       "cluster_wins": 2
      }
     }
    },
    {
     "key": "lat_gain",
     "label": "Lateral gain (1 = same size as the car's motion)",
     "unit": "",
     "better": "near1",
     "value": 0.9304,
     "median": 0.9092,
     "ci": [
      0.893,
      1.024
     ],
     "n": 119,
     "clusters": 12,
     "paired": {
      "lebowski_877m": {
       "delta": -0.04416,
       "median": -0.03268,
       "ci": [
        -0.0751,
        0.03544
       ],
       "n": 119,
       "wins": 75,
       "ties": 0,
       "clusters": 12,
       "cluster_wins": 6
      },
      "the_cool_peoples": {
       "delta": 0.003363,
       "median": 0.008937,
       "ci": [
        -0.01089,
        0.02789
       ],
       "n": 119,
       "wins": 54,
       "ties": 0,
       "clusters": 12,
       "cluster_wins": 5
      }
     }
    },
    {
     "key": "lat_rate_rms",
     "label": "Lateral action rate, RMS (smoothness)",
     "unit": "m/s³",
     "better": "lower",
     "value": 0.7178,
     "median": 0.635,
     "ci": [
      0.4409,
      0.7817
     ],
     "n": 119,
     "clusters": 12,
     "paired": {
      "lebowski_877m": {
       "delta": -1.572,
       "median": -1.362,
       "ci": [
        -1.851,
        -0.7614
       ],
       "n": 119,
       "wins": 119,
       "ties": 0,
       "clusters": 12,
       "cluster_wins": 12
      },
      "the_cool_peoples": {
       "delta": -0.06308,
       "median": -0.04217,
       "ci": [
        -0.08866,
        0.01222
       ],
       "n": 119,
       "wins": 84,
       "ties": 0,
       "clusters": 12,
       "cluster_wins": 6
      }
     }
    },
    {
     "key": "lat_best_lag",
     "label": "Lead of the lateral request over the car's motion",
     "unit": "s",
     "better": "lower",
     "value": 0.7004,
     "median": 0.65,
     "ci": [
      0.6712,
      0.7291
     ],
     "n": 119,
     "clusters": 12,
     "paired": {
      "lebowski_877m": {
       "delta": 0.07815,
       "median": 0.15,
       "ci": [
        0.015,
        0.1803
       ],
       "n": 119,
       "wins": 29,
       "ties": 8,
       "clusters": 12,
       "cluster_wins": 2
      },
      "the_cool_peoples": {
       "delta": 0.007563,
       "median": 0.0,
       "ci": [
        -0.01367,
        0.05758
       ],
       "n": 119,
       "wins": 37,
       "ties": 31,
       "clusters": 12,
       "cluster_wins": 5
      }
     }
    },
    {
     "key": "lon_mae",
     "label": "Longitudinal error",
     "unit": "m/s²",
     "better": "lower",
     "value": 0.3257,
     "median": 0.3081,
     "ci": [
      0.276,
      0.3618
     ],
     "n": 119,
     "clusters": 12,
     "paired": {
      "lebowski_877m": {
       "delta": -0.04302,
       "median": -0.03394,
       "ci": [
        -0.0664,
        0.01446
       ],
       "n": 119,
       "wins": 82,
       "ties": 0,
       "clusters": 12,
       "cluster_wins": 5
      },
      "the_cool_peoples": {
       "delta": 0.006396,
       "median": 0.004434,
       "ci": [
        0.001331,
        0.01806
       ],
       "n": 119,
       "wins": 50,
       "ties": 0,
       "clusters": 12,
       "cluster_wins": 3
      }
     }
    },
    {
     "key": "lon_corr",
     "label": "Longitudinal correlation",
     "unit": "",
     "better": "higher",
     "value": 0.3616,
     "median": 0.3446,
     "ci": [
      0.2861,
      0.5455
     ],
     "n": 119,
     "clusters": 12,
     "paired": {
      "lebowski_877m": {
       "delta": 0.03474,
       "median": 0.03701,
       "ci": [
        -0.09271,
        0.06815
       ],
       "n": 119,
       "wins": 76,
       "ties": 0,
       "clusters": 12,
       "cluster_wins": 6
      },
      "the_cool_peoples": {
       "delta": 0.000429,
       "median": 0.006433,
       "ci": [
        -0.04602,
        0.01189
       ],
       "n": 119,
       "wins": 65,
       "ties": 0,
       "clusters": 12,
       "cluster_wins": 6
      }
     }
    },
    {
     "key": "lon_brake_mae",
     "label": "Longitudinal error while braking",
     "unit": "m/s²",
     "better": "lower",
     "value": 0.6701,
     "median": 0.6575,
     "ci": [
      0.5139,
      0.7361
     ],
     "n": 81,
     "clusters": 12,
     "paired": {
      "lebowski_877m": {
       "delta": 0.1713,
       "median": 0.1663,
       "ci": [
        -0.03933,
        0.2563
       ],
       "n": 81,
       "wins": 21,
       "ties": 0,
       "clusters": 12,
       "cluster_wins": 7
      },
      "the_cool_peoples": {
       "delta": 0.03043,
       "median": 0.04372,
       "ci": [
        -0.07356,
        0.06899
       ],
       "n": 81,
       "wins": 33,
       "ties": 0,
       "clusters": 12,
       "cluster_wins": 10
      }
     }
    },
    {
     "key": "lon_rate_rms",
     "label": "Longitudinal action rate, RMS (smoothness)",
     "unit": "m/s³",
     "better": "lower",
     "value": 0.5919,
     "median": 0.5879,
     "ci": [
      0.496,
      0.6355
     ],
     "n": 119,
     "clusters": 12,
     "paired": {
      "lebowski_877m": {
       "delta": -5.018,
       "median": -4.311,
       "ci": [
        -6.119,
        -2.727
       ],
       "n": 119,
       "wins": 119,
       "ties": 0,
       "clusters": 12,
       "cluster_wins": 12
      },
      "the_cool_peoples": {
       "delta": -0.04883,
       "median": -0.014,
       "ci": [
        -0.09539,
        0.02724
       ],
       "n": 119,
       "wins": 66,
       "ties": 0,
       "clusters": 12,
       "cluster_wins": 4
      }
     }
    },
    {
     "key": "v3_mae",
     "label": "Plan speed error at 3 s",
     "unit": "m/s",
     "better": "lower",
     "value": 2.445,
     "median": 2.271,
     "ci": [
      1.453,
      3.269
     ],
     "n": 119,
     "clusters": 12,
     "paired": {
      "lebowski_877m": {
       "delta": 0.4478,
       "median": 0.3189,
       "ci": [
        0.002916,
        1.262
       ],
       "n": 119,
       "wins": 38,
       "ties": 0,
       "clusters": 12,
       "cluster_wins": 4
      },
      "the_cool_peoples": {
       "delta": 0.07088,
       "median": 0.06922,
       "ci": [
        -0.02648,
        0.2631
       ],
       "n": 119,
       "wins": 49,
       "ties": 0,
       "clusters": 12,
       "cluster_wins": 3
      }
     }
    }
   ],
   "per_unit_file": null,
   "by_group": {
    "stock-aimedup": {
     "n": 4,
     "clusters": 2,
     "metrics": [
      {
       "key": "lat_mae",
       "value": 0.1157,
       "median": 0.1168,
       "ci": null,
       "n": 4,
       "clusters": 2,
       "paired": {
        "lebowski_877m": {
         "delta": -0.02545,
         "median": -0.03302,
         "ci": null,
         "n": 4,
         "wins": 3,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 2
        },
        "the_cool_peoples": {
         "delta": 0.01318,
         "median": 0.01189,
         "ci": null,
         "n": 4,
         "wins": 1,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 0
        }
       }
      },
      {
       "key": "lat_mae_bl",
       "value": 0.1079,
       "median": 0.1079,
       "ci": null,
       "n": 4,
       "clusters": 2,
       "paired": {
        "lebowski_877m": {
         "delta": -0.03164,
         "median": -0.04275,
         "ci": null,
         "n": 4,
         "wins": 3,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 2
        },
        "the_cool_peoples": {
         "delta": 0.007653,
         "median": 0.00701,
         "ci": null,
         "n": 4,
         "wins": 2,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 0
        }
       }
      },
      {
       "key": "lat_corr",
       "value": 0.8218,
       "median": 0.9234,
       "ci": null,
       "n": 4,
       "clusters": 2,
       "paired": {
        "lebowski_877m": {
         "delta": 0.02668,
         "median": 0.01843,
         "ci": null,
         "n": 4,
         "wins": 2,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 1
        },
        "the_cool_peoples": {
         "delta": -0.04441,
         "median": -0.02468,
         "ci": null,
         "n": 4,
         "wins": 0,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 0
        }
       }
      },
      {
       "key": "lat_gain",
       "value": 0.8196,
       "median": 0.8951,
       "ci": null,
       "n": 4,
       "clusters": 2,
       "paired": {
        "lebowski_877m": {
         "delta": -0.07191,
         "median": -0.1319,
         "ci": null,
         "n": 4,
         "wins": 3,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 1
        },
        "the_cool_peoples": {
         "delta": 0.04417,
         "median": 0.04694,
         "ci": null,
         "n": 4,
         "wins": 1,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 1
        }
       }
      },
      {
       "key": "lat_rate_rms",
       "value": 0.3721,
       "median": 0.3818,
       "ci": null,
       "n": 4,
       "clusters": 2,
       "paired": {
        "lebowski_877m": {
         "delta": -0.7809,
         "median": -0.8079,
         "ci": null,
         "n": 4,
         "wins": 4,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 2
        },
        "the_cool_peoples": {
         "delta": 0.01504,
         "median": 0.0004982,
         "ci": null,
         "n": 4,
         "wins": 2,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 1
        }
       }
      },
      {
       "key": "lat_best_lag",
       "value": 0.875,
       "median": 0.925,
       "ci": null,
       "n": 4,
       "clusters": 2,
       "paired": {
        "lebowski_877m": {
         "delta": 0.3625,
         "median": 0.325,
         "ci": null,
         "n": 4,
         "wins": 0,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 0
        },
        "the_cool_peoples": {
         "delta": 0.1625,
         "median": 0.175,
         "ci": null,
         "n": 4,
         "wins": 0,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 0
        }
       }
      },
      {
       "key": "lon_mae",
       "value": 0.4971,
       "median": 0.4623,
       "ci": null,
       "n": 4,
       "clusters": 2,
       "paired": {
        "lebowski_877m": {
         "delta": -0.2806,
         "median": -0.2682,
         "ci": null,
         "n": 4,
         "wins": 4,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 2
        },
        "the_cool_peoples": {
         "delta": -0.007554,
         "median": -0.03756,
         "ci": null,
         "n": 4,
         "wins": 3,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 1
        }
       }
      },
      {
       "key": "lon_corr",
       "value": 0.1765,
       "median": -0.001676,
       "ci": null,
       "n": 4,
       "clusters": 2,
       "paired": {
        "lebowski_877m": {
         "delta": -0.3589,
         "median": -0.357,
         "ci": null,
         "n": 4,
         "wins": 2,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 1
        },
        "the_cool_peoples": {
         "delta": -0.1508,
         "median": -0.1258,
         "ci": null,
         "n": 4,
         "wins": 1,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 0
        }
       }
      },
      {
       "key": "lon_brake_mae",
       "value": 1.014,
       "median": 1.014,
       "ci": null,
       "n": 2,
       "clusters": 2,
       "paired": {
        "lebowski_877m": {
         "delta": 0.8257,
         "median": 0.8257,
         "ci": null,
         "n": 2,
         "wins": 0,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 0
        },
        "the_cool_peoples": {
         "delta": -0.04676,
         "median": -0.04676,
         "ci": null,
         "n": 2,
         "wins": 2,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 2
        }
       }
      },
      {
       "key": "lon_rate_rms",
       "value": 0.435,
       "median": 0.413,
       "ci": null,
       "n": 4,
       "clusters": 2,
       "paired": {
        "lebowski_877m": {
         "delta": -2.859,
         "median": -2.95,
         "ci": null,
         "n": 4,
         "wins": 4,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 2
        },
        "the_cool_peoples": {
         "delta": -0.0299,
         "median": 0.02292,
         "ci": null,
         "n": 4,
         "wins": 1,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 1
        }
       }
      },
      {
       "key": "v3_mae",
       "value": 2.329,
       "median": 2.327,
       "ci": null,
       "n": 4,
       "clusters": 2,
       "paired": {
        "lebowski_877m": {
         "delta": -0.05988,
         "median": 0.3679,
         "ci": null,
         "n": 4,
         "wins": 1,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 1
        },
        "the_cool_peoples": {
         "delta": 0.624,
         "median": 0.7611,
         "ci": null,
         "n": 4,
         "wins": 1,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 0
        }
       }
      }
     ]
    },
    "stock-dual": {
     "n": 13,
     "clusters": 4,
     "metrics": [
      {
       "key": "lat_mae",
       "value": 0.1424,
       "median": 0.1273,
       "ci": null,
       "n": 13,
       "clusters": 4,
       "paired": {
        "lebowski_877m": {
         "delta": -0.002785,
         "median": -0.001805,
         "ci": null,
         "n": 13,
         "wins": 7,
         "ties": 0,
         "clusters": 4,
         "cluster_wins": 2
        },
        "the_cool_peoples": {
         "delta": -0.004744,
         "median": -0.009879,
         "ci": null,
         "n": 13,
         "wins": 9,
         "ties": 0,
         "clusters": 4,
         "cluster_wins": 2
        }
       }
      },
      {
       "key": "lat_mae_bl",
       "value": 0.1383,
       "median": 0.1246,
       "ci": null,
       "n": 13,
       "clusters": 4,
       "paired": {
        "lebowski_877m": {
         "delta": -0.0009633,
         "median": 0.002698,
         "ci": null,
         "n": 13,
         "wins": 5,
         "ties": 0,
         "clusters": 4,
         "cluster_wins": 1
        },
        "the_cool_peoples": {
         "delta": -0.004941,
         "median": -0.01028,
         "ci": null,
         "n": 13,
         "wins": 9,
         "ties": 0,
         "clusters": 4,
         "cluster_wins": 2
        }
       }
      },
      {
       "key": "lat_corr",
       "value": 0.8454,
       "median": 0.8399,
       "ci": null,
       "n": 13,
       "clusters": 4,
       "paired": {
        "lebowski_877m": {
         "delta": 0.03207,
         "median": 0.002477,
         "ci": null,
         "n": 13,
         "wins": 8,
         "ties": 0,
         "clusters": 4,
         "cluster_wins": 2
        },
        "the_cool_peoples": {
         "delta": -0.008145,
         "median": -0.003681,
         "ci": null,
         "n": 13,
         "wins": 4,
         "ties": 0,
         "clusters": 4,
         "cluster_wins": 0
        }
       }
      },
      {
       "key": "lat_gain",
       "value": 0.9715,
       "median": 0.9255,
       "ci": null,
       "n": 13,
       "clusters": 4,
       "paired": {
        "lebowski_877m": {
         "delta": 0.0273,
         "median": 0.01298,
         "ci": null,
         "n": 13,
         "wins": 6,
         "ties": 0,
         "clusters": 4,
         "cluster_wins": 2
        },
        "the_cool_peoples": {
         "delta": 0.01581,
         "median": 0.0382,
         "ci": null,
         "n": 13,
         "wins": 5,
         "ties": 0,
         "clusters": 4,
         "cluster_wins": 1
        }
       }
      },
      {
       "key": "lat_rate_rms",
       "value": 0.3968,
       "median": 0.3451,
       "ci": null,
       "n": 13,
       "clusters": 4,
       "paired": {
        "lebowski_877m": {
         "delta": -0.7791,
         "median": -0.76,
         "ci": null,
         "n": 13,
         "wins": 13,
         "ties": 0,
         "clusters": 4,
         "cluster_wins": 4
        },
        "the_cool_peoples": {
         "delta": 0.007914,
         "median": 0.002478,
         "ci": null,
         "n": 13,
         "wins": 6,
         "ties": 0,
         "clusters": 4,
         "cluster_wins": 1
        }
       }
      },
      {
       "key": "lat_best_lag",
       "value": 0.6615,
       "median": 0.6,
       "ci": null,
       "n": 13,
       "clusters": 4,
       "paired": {
        "lebowski_877m": {
         "delta": 0.15,
         "median": 0.15,
         "ci": null,
         "n": 13,
         "wins": 1,
         "ties": 1,
         "clusters": 4,
         "cluster_wins": 0
        },
        "the_cool_peoples": {
         "delta": 0.01154,
         "median": 0.0,
         "ci": null,
         "n": 13,
         "wins": 5,
         "ties": 2,
         "clusters": 4,
         "cluster_wins": 3
        }
       }
      },
      {
       "key": "lon_mae",
       "value": 0.2814,
       "median": 0.2668,
       "ci": null,
       "n": 13,
       "clusters": 4,
       "paired": {
        "lebowski_877m": {
         "delta": 0.05313,
         "median": 0.04763,
         "ci": null,
         "n": 13,
         "wins": 1,
         "ties": 0,
         "clusters": 4,
         "cluster_wins": 0
        },
        "the_cool_peoples": {
         "delta": 0.01926,
         "median": 0.01282,
         "ci": null,
         "n": 13,
         "wins": 3,
         "ties": 0,
         "clusters": 4,
         "cluster_wins": 0
        }
       }
      },
      {
       "key": "lon_corr",
       "value": 0.5756,
       "median": 0.6725,
       "ci": null,
       "n": 13,
       "clusters": 4,
       "paired": {
        "lebowski_877m": {
         "delta": -0.05854,
         "median": -0.0229,
         "ci": null,
         "n": 13,
         "wins": 3,
         "ties": 0,
         "clusters": 4,
         "cluster_wins": 1
        },
        "the_cool_peoples": {
         "delta": -0.04228,
         "median": -0.000489,
         "ci": null,
         "n": 13,
         "wins": 6,
         "ties": 0,
         "clusters": 4,
         "cluster_wins": 1
        }
       }
      },
      {
       "key": "lon_brake_mae",
       "value": 0.5818,
       "median": 0.5351,
       "ci": null,
       "n": 6,
       "clusters": 4,
       "paired": {
        "lebowski_877m": {
         "delta": -0.06547,
         "median": -0.07472,
         "ci": null,
         "n": 6,
         "wins": 5,
         "ties": 0,
         "clusters": 4,
         "cluster_wins": 3
        },
        "the_cool_peoples": {
         "delta": -0.05363,
         "median": -0.05703,
         "ci": null,
         "n": 6,
         "wins": 5,
         "ties": 0,
         "clusters": 4,
         "cluster_wins": 3
        }
       }
      },
      {
       "key": "lon_rate_rms",
       "value": 0.416,
       "median": 0.4084,
       "ci": null,
       "n": 13,
       "clusters": 4,
       "paired": {
        "lebowski_877m": {
         "delta": -1.798,
         "median": -1.815,
         "ci": null,
         "n": 13,
         "wins": 13,
         "ties": 0,
         "clusters": 4,
         "cluster_wins": 4
        },
        "the_cool_peoples": {
         "delta": 0.02132,
         "median": 0.01832,
         "ci": null,
         "n": 13,
         "wins": 3,
         "ties": 0,
         "clusters": 4,
         "cluster_wins": 1
        }
       }
      },
      {
       "key": "v3_mae",
       "value": 1.11,
       "median": 1.055,
       "ci": null,
       "n": 13,
       "clusters": 4,
       "paired": {
        "lebowski_877m": {
         "delta": 0.2691,
         "median": 0.2675,
         "ci": null,
         "n": 13,
         "wins": 2,
         "ties": 0,
         "clusters": 4,
         "cluster_wins": 1
        },
        "the_cool_peoples": {
         "delta": 0.1376,
         "median": 0.07868,
         "ci": null,
         "n": 13,
         "wins": 2,
         "ties": 0,
         "clusters": 4,
         "cluster_wins": 0
        }
       }
      }
     ]
    },
    "stock-newcase": {
     "n": 2,
     "clusters": 1,
     "metrics": [
      {
       "key": "lat_mae",
       "value": 0.1736,
       "median": 0.1736,
       "ci": null,
       "n": 2,
       "clusters": 1,
       "paired": {
        "lebowski_877m": {
         "delta": 0.03926,
         "median": 0.03926,
         "ci": null,
         "n": 2,
         "wins": 0,
         "ties": 0,
         "clusters": 1,
         "cluster_wins": 0
        },
        "the_cool_peoples": {
         "delta": 0.01037,
         "median": 0.01037,
         "ci": null,
         "n": 2,
         "wins": 0,
         "ties": 0,
         "clusters": 1,
         "cluster_wins": 0
        }
       }
      },
      {
       "key": "lat_mae_bl",
       "value": 0.17,
       "median": 0.17,
       "ci": null,
       "n": 2,
       "clusters": 1,
       "paired": {
        "lebowski_877m": {
         "delta": 0.0348,
         "median": 0.0348,
         "ci": null,
         "n": 2,
         "wins": 0,
         "ties": 0,
         "clusters": 1,
         "cluster_wins": 0
        },
        "the_cool_peoples": {
         "delta": 0.01016,
         "median": 0.01016,
         "ci": null,
         "n": 2,
         "wins": 0,
         "ties": 0,
         "clusters": 1,
         "cluster_wins": 0
        }
       }
      },
      {
       "key": "lat_corr",
       "value": 0.8475,
       "median": 0.8475,
       "ci": null,
       "n": 2,
       "clusters": 1,
       "paired": {
        "lebowski_877m": {
         "delta": -8.999e-05,
         "median": -8.999e-05,
         "ci": null,
         "n": 2,
         "wins": 1,
         "ties": 0,
         "clusters": 1,
         "cluster_wins": 0
        },
        "the_cool_peoples": {
         "delta": 0.01433,
         "median": 0.01433,
         "ci": null,
         "n": 2,
         "wins": 1,
         "ties": 0,
         "clusters": 1,
         "cluster_wins": 1
        }
       }
      },
      {
       "key": "lat_gain",
       "value": 0.8109,
       "median": 0.8109,
       "ci": null,
       "n": 2,
       "clusters": 1,
       "paired": {
        "lebowski_877m": {
         "delta": -0.02106,
         "median": -0.02106,
         "ci": null,
         "n": 2,
         "wins": 2,
         "ties": 0,
         "clusters": 1,
         "cluster_wins": 1
        },
        "the_cool_peoples": {
         "delta": -0.04482,
         "median": -0.04482,
         "ci": null,
         "n": 2,
         "wins": 2,
         "ties": 0,
         "clusters": 1,
         "cluster_wins": 1
        }
       }
      },
      {
       "key": "lat_rate_rms",
       "value": 0.6263,
       "median": 0.6263,
       "ci": null,
       "n": 2,
       "clusters": 1,
       "paired": {
        "lebowski_877m": {
         "delta": -0.6493,
         "median": -0.6493,
         "ci": null,
         "n": 2,
         "wins": 2,
         "ties": 0,
         "clusters": 1,
         "cluster_wins": 1
        },
        "the_cool_peoples": {
         "delta": 0.1643,
         "median": 0.1643,
         "ci": null,
         "n": 2,
         "wins": 0,
         "ties": 0,
         "clusters": 1,
         "cluster_wins": 0
        }
       }
      },
      {
       "key": "lat_best_lag",
       "value": 0.55,
       "median": 0.55,
       "ci": null,
       "n": 2,
       "clusters": 1,
       "paired": {
        "lebowski_877m": {
         "delta": 0.05,
         "median": 0.05,
         "ci": null,
         "n": 2,
         "wins": 0,
         "ties": 1,
         "clusters": 1,
         "cluster_wins": 0
        },
        "the_cool_peoples": {
         "delta": -0.05,
         "median": -0.05,
         "ci": null,
         "n": 2,
         "wins": 1,
         "ties": 1,
         "clusters": 1,
         "cluster_wins": 1
        }
       }
      },
      {
       "key": "lon_mae",
       "value": 0.416,
       "median": 0.416,
       "ci": null,
       "n": 2,
       "clusters": 1,
       "paired": {
        "lebowski_877m": {
         "delta": 0.05345,
         "median": 0.05345,
         "ci": null,
         "n": 2,
         "wins": 0,
         "ties": 0,
         "clusters": 1,
         "cluster_wins": 0
        },
        "the_cool_peoples": {
         "delta": 0.02639,
         "median": 0.02639,
         "ci": null,
         "n": 2,
         "wins": 0,
         "ties": 0,
         "clusters": 1,
         "cluster_wins": 0
        }
       }
      },
      {
       "key": "lon_corr",
       "value": 0.4682,
       "median": 0.4682,
       "ci": null,
       "n": 2,
       "clusters": 1,
       "paired": {
        "lebowski_877m": {
         "delta": -0.01845,
         "median": -0.01845,
         "ci": null,
         "n": 2,
         "wins": 0,
         "ties": 0,
         "clusters": 1,
         "cluster_wins": 0
        },
        "the_cool_peoples": {
         "delta": 0.01106,
         "median": 0.01106,
         "ci": null,
         "n": 2,
         "wins": 1,
         "ties": 0,
         "clusters": 1,
         "cluster_wins": 1
        }
       }
      },
      {
       "key": "lon_brake_mae",
       "value": 0.3227,
       "median": 0.3227,
       "ci": null,
       "n": 2,
       "clusters": 1,
       "paired": {
        "lebowski_877m": {
         "delta": -0.1806,
         "median": -0.1806,
         "ci": null,
         "n": 2,
         "wins": 2,
         "ties": 0,
         "clusters": 1,
         "cluster_wins": 1
        },
        "the_cool_peoples": {
         "delta": -0.03148,
         "median": -0.03148,
         "ci": null,
         "n": 2,
         "wins": 1,
         "ties": 0,
         "clusters": 1,
         "cluster_wins": 1
        }
       }
      },
      {
       "key": "lon_rate_rms",
       "value": 0.5999,
       "median": 0.5999,
       "ci": null,
       "n": 2,
       "clusters": 1,
       "paired": {
        "lebowski_877m": {
         "delta": -2.756,
         "median": -2.756,
         "ci": null,
         "n": 2,
         "wins": 2,
         "ties": 0,
         "clusters": 1,
         "cluster_wins": 1
        },
        "the_cool_peoples": {
         "delta": 0.02492,
         "median": 0.02492,
         "ci": null,
         "n": 2,
         "wins": 1,
         "ties": 0,
         "clusters": 1,
         "cluster_wins": 0
        }
       }
      },
      {
       "key": "v3_mae",
       "value": 2.147,
       "median": 2.147,
       "ci": null,
       "n": 2,
       "clusters": 1,
       "paired": {
        "lebowski_877m": {
         "delta": 0.05331,
         "median": 0.05331,
         "ci": null,
         "n": 2,
         "wins": 1,
         "ties": 0,
         "clusters": 1,
         "cluster_wins": 0
        },
        "the_cool_peoples": {
         "delta": 0.07152,
         "median": 0.07152,
         "ci": null,
         "n": 2,
         "wins": 1,
         "ties": 0,
         "clusters": 1,
         "cluster_wins": 0
        }
       }
      }
     ]
    },
    "v13-8mm": {
     "n": 92,
     "clusters": 3,
     "metrics": [
      {
       "key": "lat_mae",
       "value": 0.1802,
       "median": 0.1706,
       "ci": null,
       "n": 92,
       "clusters": 3,
       "paired": {
        "lebowski_877m": {
         "delta": -0.001361,
         "median": -0.006174,
         "ci": null,
         "n": 92,
         "wins": 53,
         "ties": 0,
         "clusters": 3,
         "cluster_wins": 1
        },
        "the_cool_peoples": {
         "delta": 0.00546,
         "median": 0.006005,
         "ci": null,
         "n": 92,
         "wins": 33,
         "ties": 0,
         "clusters": 3,
         "cluster_wins": 0
        }
       }
      },
      {
       "key": "lat_mae_bl",
       "value": 0.1763,
       "median": 0.1669,
       "ci": null,
       "n": 92,
       "clusters": 3,
       "paired": {
        "lebowski_877m": {
         "delta": 0.004086,
         "median": -0.002466,
         "ci": null,
         "n": 92,
         "wins": 49,
         "ties": 0,
         "clusters": 3,
         "cluster_wins": 1
        },
        "the_cool_peoples": {
         "delta": 0.006226,
         "median": 0.00756,
         "ci": null,
         "n": 92,
         "wins": 36,
         "ties": 0,
         "clusters": 3,
         "cluster_wins": 0
        }
       }
      },
      {
       "key": "lat_corr",
       "value": 0.8235,
       "median": 0.9077,
       "ci": null,
       "n": 92,
       "clusters": 3,
       "paired": {
        "lebowski_877m": {
         "delta": 0.07502,
         "median": 0.0283,
         "ci": null,
         "n": 92,
         "wins": 78,
         "ties": 0,
         "clusters": 3,
         "cluster_wins": 3
        },
        "the_cool_peoples": {
         "delta": -0.001839,
         "median": 0.001432,
         "ci": null,
         "n": 92,
         "wins": 49,
         "ties": 0,
         "clusters": 3,
         "cluster_wins": 1
        }
       }
      },
      {
       "key": "lat_gain",
       "value": 0.9288,
       "median": 0.9049,
       "ci": null,
       "n": 92,
       "clusters": 3,
       "paired": {
        "lebowski_877m": {
         "delta": -0.06009,
         "median": -0.05223,
         "ci": null,
         "n": 92,
         "wins": 61,
         "ties": 0,
         "clusters": 3,
         "cluster_wins": 2
        },
        "the_cool_peoples": {
         "delta": 0.003704,
         "median": 0.008966,
         "ci": null,
         "n": 92,
         "wins": 40,
         "ties": 0,
         "clusters": 3,
         "cluster_wins": 1
        }
       }
      },
      {
       "key": "lat_rate_rms",
       "value": 0.8017,
       "median": 0.7418,
       "ci": null,
       "n": 92,
       "clusters": 3,
       "paired": {
        "lebowski_877m": {
         "delta": -1.816,
         "median": -1.639,
         "ci": null,
         "n": 92,
         "wins": 92,
         "ties": 0,
         "clusters": 3,
         "cluster_wins": 3
        },
        "the_cool_peoples": {
         "delta": -0.08587,
         "median": -0.07203,
         "ci": null,
         "n": 92,
         "wins": 72,
         "ties": 0,
         "clusters": 3,
         "cluster_wins": 3
        }
       }
      },
      {
       "key": "lat_best_lag",
       "value": 0.7049,
       "median": 0.65,
       "ci": null,
       "n": 92,
       "clusters": 3,
       "paired": {
        "lebowski_877m": {
         "delta": 0.04457,
         "median": 0.1,
         "ci": null,
         "n": 92,
         "wins": 28,
         "ties": 6,
         "clusters": 3,
         "cluster_wins": 2
        },
        "the_cool_peoples": {
         "delta": -0.00163,
         "median": 0.0,
         "ci": null,
         "n": 92,
         "wins": 31,
         "ties": 23,
         "clusters": 3,
         "cluster_wins": 1
        }
       }
      },
      {
       "key": "lon_mae",
       "value": 0.3198,
       "median": 0.3077,
       "ci": null,
       "n": 92,
       "clusters": 3,
       "paired": {
        "lebowski_877m": {
         "delta": -0.05627,
         "median": -0.05298,
         "ci": null,
         "n": 92,
         "wins": 76,
         "ties": 0,
         "clusters": 3,
         "cluster_wins": 3
        },
        "the_cool_peoples": {
         "delta": 0.00334,
         "median": 0.001659,
         "ci": null,
         "n": 92,
         "wins": 42,
         "ties": 0,
         "clusters": 3,
         "cluster_wins": 2
        }
       }
      },
      {
       "key": "lon_corr",
       "value": 0.3019,
       "median": 0.3011,
       "ci": null,
       "n": 92,
       "clusters": 3,
       "paired": {
        "lebowski_877m": {
         "delta": 0.06819,
         "median": 0.07625,
         "ci": null,
         "n": 92,
         "wins": 66,
         "ties": 0,
         "clusters": 3,
         "cluster_wins": 3
        },
        "the_cool_peoples": {
         "delta": 0.01249,
         "median": 0.01554,
         "ci": null,
         "n": 92,
         "wins": 52,
         "ties": 0,
         "clusters": 3,
         "cluster_wins": 2
        }
       }
      },
      {
       "key": "lon_brake_mae",
       "value": 0.6927,
       "median": 0.671,
       "ci": null,
       "n": 65,
       "clusters": 3,
       "paired": {
        "lebowski_877m": {
         "delta": 0.2036,
         "median": 0.2172,
         "ci": null,
         "n": 65,
         "wins": 10,
         "ties": 0,
         "clusters": 3,
         "cluster_wins": 1
        },
        "the_cool_peoples": {
         "delta": 0.05423,
         "median": 0.06194,
         "ci": null,
         "n": 65,
         "wins": 19,
         "ties": 0,
         "clusters": 3,
         "cluster_wins": 2
        }
       }
      },
      {
       "key": "lon_rate_rms",
       "value": 0.6239,
       "median": 0.608,
       "ci": null,
       "n": 92,
       "clusters": 3,
       "paired": {
        "lebowski_877m": {
         "delta": -5.709,
         "median": -5.407,
         "ci": null,
         "n": 92,
         "wins": 92,
         "ties": 0,
         "clusters": 3,
         "cluster_wins": 3
        },
        "the_cool_peoples": {
         "delta": -0.06436,
         "median": -0.03137,
         "ci": null,
         "n": 92,
         "wins": 57,
         "ties": 0,
         "clusters": 3,
         "cluster_wins": 1
        }
       }
      },
      {
       "key": "v3_mae",
       "value": 2.769,
       "median": 2.674,
       "ci": null,
       "n": 92,
       "clusters": 3,
       "paired": {
        "lebowski_877m": {
         "delta": 0.5347,
         "median": 0.3931,
         "ci": null,
         "n": 92,
         "wins": 31,
         "ties": 0,
         "clusters": 3,
         "cluster_wins": 1
        },
        "the_cool_peoples": {
         "delta": 0.05479,
         "median": 0.09695,
         "ci": null,
         "n": 92,
         "wins": 37,
         "ties": 0,
         "clusters": 3,
         "cluster_wins": 1
        }
       }
      }
     ]
    },
    "v14-8mm": {
     "n": 8,
     "clusters": 2,
     "metrics": [
      {
       "key": "lat_mae",
       "value": 0.1535,
       "median": 0.1606,
       "ci": null,
       "n": 8,
       "clusters": 2,
       "paired": {
        "lebowski_877m": {
         "delta": 0.01214,
         "median": 0.01559,
         "ci": null,
         "n": 8,
         "wins": 3,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 0
        },
        "the_cool_peoples": {
         "delta": -0.004121,
         "median": -0.007493,
         "ci": null,
         "n": 8,
         "wins": 6,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 2
        }
       }
      },
      {
       "key": "lat_mae_bl",
       "value": 0.1536,
       "median": 0.1583,
       "ci": null,
       "n": 8,
       "clusters": 2,
       "paired": {
        "lebowski_877m": {
         "delta": 0.0165,
         "median": 0.01856,
         "ci": null,
         "n": 8,
         "wins": 2,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 0
        },
        "the_cool_peoples": {
         "delta": -0.003248,
         "median": -0.007754,
         "ci": null,
         "n": 8,
         "wins": 6,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 2
        }
       }
      },
      {
       "key": "lat_corr",
       "value": 0.845,
       "median": 0.9264,
       "ci": null,
       "n": 8,
       "clusters": 2,
       "paired": {
        "lebowski_877m": {
         "delta": -0.02301,
         "median": 0.009447,
         "ci": null,
         "n": 8,
         "wins": 5,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 1
        },
        "the_cool_peoples": {
         "delta": -0.00951,
         "median": 0.0002507,
         "ci": null,
         "n": 8,
         "wins": 4,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 0
        }
       }
      },
      {
       "key": "lat_gain",
       "value": 0.9668,
       "median": 0.9558,
       "ci": null,
       "n": 8,
       "clusters": 2,
       "paired": {
        "lebowski_877m": {
         "delta": 0.031,
         "median": 0.02528,
         "ci": null,
         "n": 8,
         "wins": 3,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 0
        },
        "the_cool_peoples": {
         "delta": -0.02915,
         "median": -0.007439,
         "ci": null,
         "n": 8,
         "wins": 6,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 1
        }
       }
      },
      {
       "key": "lat_rate_rms",
       "value": 0.4706,
       "median": 0.4254,
       "ci": null,
       "n": 8,
       "clusters": 2,
       "paired": {
        "lebowski_877m": {
         "delta": -0.685,
         "median": -0.5796,
         "ci": null,
         "n": 8,
         "wins": 8,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 2
        },
        "the_cool_peoples": {
         "delta": -0.01218,
         "median": -0.002814,
         "ci": null,
         "n": 8,
         "wins": 4,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 1
        }
       }
      },
      {
       "key": "lat_best_lag",
       "value": 0.6625,
       "median": 0.6,
       "ci": null,
       "n": 8,
       "clusters": 2,
       "paired": {
        "lebowski_877m": {
         "delta": 0.2125,
         "median": 0.2,
         "ci": null,
         "n": 8,
         "wins": 0,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 0
        },
        "the_cool_peoples": {
         "delta": 0.04375,
         "median": 0.0,
         "ci": null,
         "n": 8,
         "wins": 0,
         "ties": 5,
         "clusters": 2,
         "cluster_wins": 0
        }
       }
      },
      {
       "key": "lon_mae",
       "value": 0.3572,
       "median": 0.2959,
       "ci": null,
       "n": 8,
       "clusters": 2,
       "paired": {
        "lebowski_877m": {
         "delta": 0.04781,
         "median": 0.02571,
         "ci": null,
         "n": 8,
         "wins": 1,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 0
        },
        "the_cool_peoples": {
         "delta": 0.02261,
         "median": 0.01901,
         "ci": null,
         "n": 8,
         "wins": 2,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 0
        }
       }
      },
      {
       "key": "lon_corr",
       "value": 0.7664,
       "median": 0.7819,
       "ci": null,
       "n": 8,
       "clusters": 2,
       "paired": {
        "lebowski_877m": {
         "delta": 0.01172,
         "median": 0.02804,
         "ci": null,
         "n": 8,
         "wins": 5,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 1
        },
        "the_cool_peoples": {
         "delta": 0.004125,
         "median": 0.01088,
         "ci": null,
         "n": 8,
         "wins": 5,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 2
        }
       }
      },
      {
       "key": "lon_brake_mae",
       "value": 0.5148,
       "median": 0.5947,
       "ci": null,
       "n": 6,
       "clusters": 2,
       "paired": {
        "lebowski_877m": {
         "delta": -0.04283,
         "median": -0.02311,
         "ci": null,
         "n": 6,
         "wins": 4,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 2
        },
        "the_cool_peoples": {
         "delta": -0.09701,
         "median": -0.07967,
         "ci": null,
         "n": 6,
         "wins": 6,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 2
        }
       }
      },
      {
       "key": "lon_rate_rms",
       "value": 0.5862,
       "median": 0.5465,
       "ci": null,
       "n": 8,
       "clusters": 2,
       "paired": {
        "lebowski_877m": {
         "delta": -3.947,
         "median": -3.624,
         "ci": null,
         "n": 8,
         "wins": 8,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 2
        },
        "the_cool_peoples": {
         "delta": -0.01212,
         "median": -0.01543,
         "ci": null,
         "n": 8,
         "wins": 4,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 1
        }
       }
      },
      {
       "key": "v3_mae",
       "value": 1.025,
       "median": 0.9662,
       "ci": null,
       "n": 8,
       "clusters": 2,
       "paired": {
        "lebowski_877m": {
         "delta": 0.09212,
         "median": 0.0624,
         "ci": null,
         "n": 8,
         "wins": 3,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 1
        },
        "the_cool_peoples": {
         "delta": -0.1292,
         "median": -0.1387,
         "ci": null,
         "n": 8,
         "wins": 8,
         "ties": 0,
         "clusters": 2,
         "cluster_wins": 2
        }
       }
      }
     ]
    }
   }
  },
  "truck_closed_loop": {
   "status": "not_run",
   "unit_of_evidence": "window",
   "n": 0,
   "condition": "Measured. worldmodel-4B as in the public closed loop, but the model drives a simulation of the G55's measured steering and powertrain response instead of the ideal car. The simulation starts from the truck's state (speed, yaw rate, steering, gear) in its CAN recording of the 2 s before the window; one window, whose 2 s before are not all valid CAN, starts steady at its recorded speed. Lane-following windows from the G55's drives, day and night; distance is to the driver's recorded path; seeds 0 and 1, averaged per window.",
   "baselines": [],
   "metrics": [],
   "per_unit_file": null
  },
  "truck_recovery": {
   "status": "not_run",
   "unit_of_evidence": "window",
   "n": 0,
   "condition": "Measured. The truck closed loop with a disturbance while the model drives; only the tests listed ran.",
   "baselines": [],
   "metrics": [],
   "per_unit_file": null
  }
 },
 "stills": [],
 "limits": [
  "Open-loop scores compare a model's action with what the recording driver or car did; a model that drives differently but well is scored as wrong. comma's RL residual makes the lateral action smoother, and earlier than its base's by 0.10 to 0.26 s (measured on the G55 and on 16 of comma's cars); both score as error here. Measured on the 2026-10-27 deep dive: open-loop accuracy did not predict closed-loop driving.",
  "Closed-loop numbers come from a generative world model, not a car. In a separate test on the G55's windows, worldmodel-4B followed an added yaw command through 2 s but showed 0 to 20% of an added lateral offset 3 to 4 s later; past about 4 s the recorded anchor frames also pull every model back toward the recorded path. The front page reports 2 s.",
  "Intervals are 95% percentile bootstraps that resample whole drives (the drive a public segment was cut from, the recorded drive a truck window comes from), so segments of one drive count once. Below 10 drives there is no interval: the braking-lead test comes from one drive and reports means and win counts only.",
  "Paired deltas are model minus baseline on the units both have (for lateral gain, the change in its distance from 1); wins count units where the model is better in the metric's direction.",
  "The lead of the lateral request is the shift, searched from 0 to 1.5 s, at which the request best correlates with the car's later lateral motion; a larger value means the request comes earlier. A value at either edge is censored and left out. Plan speed error at 3 s reads the plan head, which does not separate models built on one frozen base: it is identical on every segment for the four f22e06e5 models, and for Cinque v2 and v3.",
  "Truck open loop comes from few drives, and the camera setup changed between them (lens, housing, aim); the pooled number mixes setups, so read the per-setup numbers too. The 8 mm setups' wide camera recorded about 7-10 fps, so its frame can be up to ~60 ms from the narrow one. On the V14 setup up to 7 of a window's 700 steps are ~100 ms (one camera frame dropped). The camera's aim is the truck's live calibration learned on each drive; drives whose recorded calibration was carried over from another drive are left out.",
  "A historical model, run outside comma's software with its own release's inputs, camera warp and command (engines/legacy.py). Checked against comma's CI model replay of this model (made on a comma device, 60 frames of one route): the plan's lateral position is 0.067 m RMS from comma's output; every other model here is 0.98 m or more from it on the same check.",
  "Trained for the comma devices and cameras of its time, not for the G55's camera or for every camera among comma's public segments: of those 45 segments 26 record the OX03C10 sensor, 6 the AR0231, 4 the OS04C10 and 9 none; 30 were recorded 2021 to 2024 and 15 come from devices whose route names carry no date.",
  "Not run in closed loop: the G55 closed-loop and recovery tests need this release's controller inside the world-model loop, which is not built yet.",
  "The commands are 0.10's modeld (get_action_from_model on the plan, lateral delay = steer actuator delay + 0.2 s, smoothed over 0.1 s lateral and 0.3 s longitudinal); the newest models are scored on their own unsmoothed action head, so lateral action rates are not comparable across these eras."
 ]
}