{ "schema": 1, "model": { "id": "f22b8b34_16384", "name": null, "sha256": "6274eb4a3f3eb7ad1ab341d222bbba2ba709b06645dbbd15ac4971d6cb22ab21", "source": { "kind": "huggingface", "ref": "84c72ed5381a64b8630890b861cddb5adf5ae8aa", "url": "https://huggingface.co/commaai/openpilot_driving_models/blob/84c72ed5381a64b8630890b861cddb5adf5ae8aa/f22b8b34-0983-4185-a453-a87b73798d61/16384/big_driving_supercombo.onnx", "date": "2026-10-29" }, "first_public": "2026-10-29", "on_master": null, "params_m": 382.9, "gmac": 50.1, "family": "residual", "notes": "Resolved from hf:f22b8b34/16384; checkpoint f22e06e5-3b3f-48d2-a6b9-2f32dc20e707/56320/f22b8b34-0983-4185-a453-a87b73798d61/16384." }, "scored": { "date": "2026-10-31T23:59:50Z", "suite": "geval-1", "code": "d8eb40eeba87" }, "baselines": [ "lebowski_877m", "6b5b705f_12800" ], "sections": { "public_open_loop": { "status": "ok", "unit_of_evidence": "segment", "n": 39, "clusters": 23, "condition": "Measured. comma's public two-camera segments (openpilot CI data), native intrinsics and the device's own calibration, both cameras; every model on the same decoded frames. Truth is what the recording car did next: carState speed and acceleration, yaw rate from the device's cameraOdometry. Segments where the car never moves have no data. 39 of the 45 listed segments have a score, from 23 drives and 20 cars.", "baselines": [ "lebowski_877m", "6b5b705f_12800" ], "metrics": [ { "key": "lat_mae", "label": "Lateral error", "unit": "m/s²", "better": "lower", "value": 0.1484, "median": 0.1362, "ci": [ 0.1161, 0.183 ], "n": 39, "clusters": 23, "paired": { "lebowski_877m": { "delta": 0.03974, "median": 0.01881, "ci": [ 0.02547, 0.0558 ], "n": 39, "wins": 3, "ties": 0, "clusters": 23, "cluster_wins": 1 }, "6b5b705f_12800": { "delta": 0.01159, "median": 0.003549, "ci": [ 0.00379, 0.02146 ], "n": 39, "wins": 14, "ties": 0, "clusters": 23, "cluster_wins": 6 } } }, { "key": "lat_mae_bl", "label": "Lateral error at the model's own best lead", "unit": "m/s²", "better": "lower", "value": 0.1391, "median": 0.1285, "ci": [ 0.1079, 0.1732 ], "n": 39, "clusters": 23, "paired": { "lebowski_877m": { "delta": 0.03766, "median": 0.01913, "ci": [ 0.0224, 0.05431 ], "n": 39, "wins": 6, "ties": 0, "clusters": 23, "cluster_wins": 2 }, "6b5b705f_12800": { "delta": 0.01354, "median": 0.0061, "ci": [ 0.004992, 0.02396 ], "n": 39, "wins": 15, "ties": 0, "clusters": 23, "cluster_wins": 7 } } }, { "key": "lat_corr", "label": "Lateral correlation with the car's later motion", "unit": "", "better": "higher", "value": 0.8195, "median": 0.9305, "ci": [ 0.7346, 0.898 ], "n": 39, "clusters": 23, "paired": { "lebowski_877m": { "delta": -0.03813, "median": -0.01214, "ci": [ -0.07546, -0.007561 ], "n": 39, "wins": 8, "ties": 0, "clusters": 23, "cluster_wins": 5 }, "6b5b705f_12800": { "delta": -0.01724, "median": -0.002878, "ci": [ -0.04437, 0.005692 ], "n": 39, "wins": 13, "ties": 0, "clusters": 23, "cluster_wins": 10 } } }, { "key": "lat_gain", "label": "Lateral gain (1 = same size as the car's motion)", "unit": "", "better": "near1", "value": 0.7176, "median": 0.7613, "ci": [ 0.6148, 0.8262 ], "n": 39, "clusters": 23, "paired": { "lebowski_877m": { "delta": 0.1092, "median": 0.08304, "ci": [ 0.05596, 0.1728 ], "n": 39, "wins": 10, "ties": 0, "clusters": 23, "cluster_wins": 6 }, "6b5b705f_12800": { "delta": 0.08101, "median": 0.03865, "ci": [ 0.0279, 0.151 ], "n": 39, "wins": 10, "ties": 0, "clusters": 23, "cluster_wins": 6 } } }, { "key": "lat_rate_rms", "label": "Lateral action rate, RMS (smoothness)", "unit": "m/s³", "better": "lower", "value": 0.47, "median": 0.4, "ci": [ 0.3698, 0.5909 ], "n": 39, "clusters": 23, "paired": { "lebowski_877m": { "delta": -0.1939, "median": -0.1959, "ci": [ -0.2421, -0.141 ], "n": 39, "wins": 37, "ties": 0, "clusters": 23, "cluster_wins": 22 }, "6b5b705f_12800": { "delta": 0.006277, "median": 0.00124, "ci": [ -0.009239, 0.02885 ], "n": 39, "wins": 19, "ties": 0, "clusters": 23, "cluster_wins": 11 } } }, { "key": "lat_best_lag", "label": "Lead of the lateral request over the car's motion", "unit": "s", "better": "lower", "value": 0.525, "median": 0.475, "ci": [ 0.4333, 0.6258 ], "n": 34, "clusters": 22, "paired": { "lebowski_877m": { "delta": 0.1394, "median": 0.1, "ci": [ 0.07317, 0.2203 ], "n": 33, "wins": 4, "ties": 7, "clusters": 22, "cluster_wins": 2 }, "6b5b705f_12800": { "delta": -0.02353, "median": 0.0, "ci": [ -0.08056, 0.02116 ], "n": 34, "wins": 6, "ties": 16, "clusters": 22, "cluster_wins": 5 } } }, { "key": "lon_mae", "label": "Longitudinal error", "unit": "m/s²", "better": "lower", "value": 0.3317, "median": 0.3239, "ci": [ 0.2877, 0.3789 ], "n": 39, "clusters": 23, "paired": { "lebowski_877m": { "delta": 0.0236, "median": 0.02202, "ci": [ 0.004488, 0.04667 ], "n": 39, "wins": 11, "ties": 0, "clusters": 23, "cluster_wins": 6 }, "6b5b705f_12800": { "delta": -0.03616, "median": -0.02979, "ci": [ -0.05246, -0.01948 ], "n": 39, "wins": 33, "ties": 0, "clusters": 23, "cluster_wins": 21 } } }, { "key": "lon_corr", "label": "Longitudinal correlation", "unit": "", "better": "higher", "value": 0.7324, "median": 0.8539, "ci": [ 0.6357, 0.8177 ], "n": 39, "clusters": 23, "paired": { "lebowski_877m": { "delta": 0.01467, "median": -0.002793, "ci": [ -0.01283, 0.04004 ], "n": 39, "wins": 19, "ties": 0, "clusters": 23, "cluster_wins": 12 }, "6b5b705f_12800": { "delta": 0.05806, "median": 0.02712, "ci": [ 0.02954, 0.09375 ], "n": 39, "wins": 33, "ties": 0, "clusters": 23, "cluster_wins": 21 } } }, { "key": "lon_brake_mae", "label": "Longitudinal error while braking", "unit": "m/s²", "better": "lower", "value": 0.3967, "median": 0.3812, "ci": [ 0.3426, 0.4609 ], "n": 30, "clusters": 21, "paired": { "lebowski_877m": { "delta": -0.03763, "median": -0.02438, "ci": [ -0.05564, -0.01678 ], "n": 30, "wins": 22, "ties": 0, "clusters": 21, "cluster_wins": 17 }, "6b5b705f_12800": { "delta": -0.06382, "median": -0.06155, "ci": [ -0.09222, -0.03882 ], "n": 30, "wins": 22, "ties": 0, "clusters": 21, "cluster_wins": 18 } } }, { "key": "lon_rate_rms", "label": "Longitudinal action rate, RMS (smoothness)", "unit": "m/s³", "better": "lower", "value": 0.8696, "median": 0.7311, "ci": [ 0.7473, 1.013 ], "n": 39, "clusters": 23, "paired": { "lebowski_877m": { "delta": -0.4634, "median": -0.388, "ci": [ -0.6014, -0.3457 ], "n": 39, "wins": 37, "ties": 0, "clusters": 23, "cluster_wins": 23 }, "6b5b705f_12800": { "delta": -0.1437, "median": -0.1259, "ci": [ -0.18, -0.1083 ], "n": 39, "wins": 36, "ties": 0, "clusters": 23, "cluster_wins": 21 } } }, { "key": "v3_mae", "label": "Plan speed error at 3 s", "unit": "m/s", "better": "lower", "value": 0.7602, "median": 0.7121, "ci": [ 0.6567, 0.888 ], "n": 39, "clusters": 23, "paired": { "lebowski_877m": { "delta": 0.04442, "median": 0.04624, "ci": [ 0.009032, 0.07746 ], "n": 39, "wins": 12, "ties": 0, "clusters": 23, "cluster_wins": 8 }, "6b5b705f_12800": { "delta": 0.0, "median": 0.0, "ci": [ 0.0, 0.0 ], "n": 39, "wins": 0, "ties": 39, "clusters": 23, "cluster_wins": 0 } } } ], "per_unit_file": "per_unit/f22b8b34_16384/public_open_loop.csv" }, "public_closed_loop": { "status": "not_run", "unit_of_evidence": "segment", "n": 0, "condition": "Measured. comma's worldmodel-4B generates each next frame at 5 Hz from 9 history frames, 5 recorded anchor frames and the car's pose; the model drives comma's ideal car (the requested curvature and acceleration happen at once) for 7.2 s from the recorded start. Distance is to the recording car's own path. One window per segment, nearest the middle, where the car holds at least 5 m/s for the window and the 4.8 s before it; segments with no such stretch are left out. Seeds 0 and 1, averaged per segment.", "baselines": [], "metrics": [], "per_unit_file": null }, "truck_open_loop": { "status": "ok", "unit_of_evidence": "window", "n": 119, "clusters": 12, "condition": "Measured. The G55's recorded drives replayed through the model with both of the truck's cameras (each setup's own wide-camera calibration), road-plane warp to 1.45 m. Where the narrow lens does not cover comma's narrow view (the 8 mm lens in the V13 case), those pixels come from the wide camera, brightness matched. Truth is the truck's CAN speed and yaw rate. 30 s windows after a 5 s warm-up; window = unit, drive = cluster; results are also given per camera setup. Corpus windows of 30 s from the truck's recorded drives, scored per camera setup.", "baselines": [ "lebowski_877m", "6b5b705f_12800" ], "metrics": [ { "key": "lat_mae", "label": "Lateral error", "unit": "m/s²", "better": "lower", "value": 0.1767, "median": 0.1661, "ci": [ 0.1469, 0.1874 ], "n": 119, "clusters": 12, "paired": { "lebowski_877m": { "delta": 0.00398, "median": 0.003223, "ci": [ -0.0004753, 0.0156 ], "n": 119, "wins": 52, "ties": 0, "clusters": 12, "cluster_wins": 3 }, "6b5b705f_12800": { "delta": 0.01532, "median": 0.01554, "ci": [ 0.009269, 0.01843 ], "n": 119, "wins": 26, "ties": 0, "clusters": 12, "cluster_wins": 1 } } }, { "key": "lat_mae_bl", "label": "Lateral error at the model's own best lead", "unit": "m/s²", "better": "lower", "value": 0.1622, "median": 0.1499, "ci": [ 0.1368, 0.1705 ], "n": 119, "clusters": 12, "paired": { "lebowski_877m": { "delta": -0.002283, "median": -0.002383, "ci": [ -0.006025, 0.009656 ], "n": 119, "wins": 63, "ties": 0, "clusters": 12, "cluster_wins": 6 }, "6b5b705f_12800": { "delta": 0.01261, "median": 0.01234, "ci": [ 0.009247, 0.01718 ], "n": 119, "wins": 29, "ties": 0, "clusters": 12, "cluster_wins": 2 } } }, { "key": "lat_corr", "label": "Lateral correlation with the car's later motion", "unit": "", "better": "higher", "value": 0.8046, "median": 0.893, "ci": [ 0.7661, 0.8297 ], "n": 119, "clusters": 12, "paired": { "lebowski_877m": { "delta": 0.03775, "median": 0.01333, "ci": [ 0.00594, 0.04708 ], "n": 119, "wins": 81, "ties": 0, "clusters": 12, "cluster_wins": 6 }, "6b5b705f_12800": { "delta": -0.01171, "median": -0.00923, "ci": [ -0.01715, -0.005224 ], "n": 119, "wins": 26, "ties": 0, "clusters": 12, "cluster_wins": 2 } } }, { "key": "lat_gain", "label": "Lateral gain (1 = same size as the car's motion)", "unit": "", "better": "near1", "value": 0.7985, "median": 0.8063, "ci": [ 0.7756, 0.8916 ], "n": 119, "clusters": 12, "paired": { "lebowski_877m": { "delta": 0.008, "median": 0.02313, "ci": [ -0.01427, 0.04453 ], "n": 119, "wins": 47, "ties": 0, "clusters": 12, "cluster_wins": 5 }, "6b5b705f_12800": { "delta": 0.03583, "median": 0.04649, "ci": [ 0.0243, 0.04947 ], "n": 119, "wins": 31, "ties": 0, "clusters": 12, "cluster_wins": 2 } } }, { "key": "lat_rate_rms", "label": "Lateral action rate, RMS (smoothness)", "unit": "m/s³", "better": "lower", "value": 1.181, "median": 0.8939, "ci": [ 0.6684, 1.423 ], "n": 119, "clusters": 12, "paired": { "lebowski_877m": { "delta": -1.109, "median": -0.9853, "ci": [ -1.229, -0.4879 ], "n": 119, "wins": 118, "ties": 0, "clusters": 12, "cluster_wins": 12 }, "6b5b705f_12800": { "delta": 0.02585, "median": 0.02934, "ci": [ -0.0009281, 0.03989 ], "n": 119, "wins": 37, "ties": 0, "clusters": 12, "cluster_wins": 5 } } }, { "key": "lat_best_lag", "label": "Lead of the lateral request over the car's motion", "unit": "s", "better": "lower", "value": 0.6727, "median": 0.65, "ci": [ 0.6138, 0.7033 ], "n": 119, "clusters": 12, "paired": { "lebowski_877m": { "delta": 0.05042, "median": 0.1, "ci": [ -0.01549, 0.1296 ], "n": 119, "wins": 33, "ties": 12, "clusters": 12, "cluster_wins": 4 }, "6b5b705f_12800": { "delta": 0.04873, "median": 0.05, "ci": [ -0.01111, 0.06596 ], "n": 118, "wins": 19, "ties": 34, "clusters": 12, "cluster_wins": 5 } } }, { "key": "lon_mae", "label": "Longitudinal error", "unit": "m/s²", "better": "lower", "value": 0.3569, "median": 0.3635, "ci": [ 0.2785, 0.4083 ], "n": 119, "clusters": 12, "paired": { "lebowski_877m": { "delta": -0.01173, "median": -0.01075, "ci": [ -0.0243, 0.02351 ], "n": 119, "wins": 67, "ties": 0, "clusters": 12, "cluster_wins": 5 }, "6b5b705f_12800": { "delta": -0.01096, "median": -0.01883, "ci": [ -0.02565, 0.03562 ], "n": 119, "wins": 87, "ties": 0, "clusters": 12, "cluster_wins": 8 } } }, { "key": "lon_corr", "label": "Longitudinal correlation", "unit": "", "better": "higher", "value": 0.34, "median": 0.2869, "ci": [ 0.2868, 0.5878 ], "n": 119, "clusters": 12, "paired": { "lebowski_877m": { "delta": 0.01315, "median": 0.01159, "ci": [ -0.02886, 0.05619 ], "n": 119, "wins": 66, "ties": 0, "clusters": 12, "cluster_wins": 4 }, "6b5b705f_12800": { "delta": 0.03202, "median": 0.02046, "ci": [ 0.02196, 0.0707 ], "n": 119, "wins": 86, "ties": 0, "clusters": 12, "cluster_wins": 12 } } }, { "key": "lon_brake_mae", "label": "Longitudinal error while braking", "unit": "m/s²", "better": "lower", "value": 0.477, "median": 0.4573, "ci": [ 0.4452, 0.5839 ], "n": 81, "clusters": 12, "paired": { "lebowski_877m": { "delta": -0.02178, "median": -0.02488, "ci": [ -0.05804, 0.02761 ], "n": 81, "wins": 50, "ties": 0, "clusters": 12, "cluster_wins": 7 }, "6b5b705f_12800": { "delta": -0.01174, "median": 0.002543, "ci": [ -0.14, 0.01856 ], "n": 81, "wins": 39, "ties": 0, "clusters": 12, "cluster_wins": 11 } } }, { "key": "lon_rate_rms", "label": "Longitudinal action rate, RMS (smoothness)", "unit": "m/s³", "better": "lower", "value": 4.246, "median": 4.074, "ci": [ 1.938, 5.553 ], "n": 119, "clusters": 12, "paired": { "lebowski_877m": { "delta": -1.363, "median": -1.393, "ci": [ -1.915, -0.9337 ], "n": 119, "wins": 93, "ties": 0, "clusters": 12, "cluster_wins": 12 }, "6b5b705f_12800": { "delta": -0.2281, "median": -0.1825, "ci": [ -0.305, -0.1588 ], "n": 119, "wins": 94, "ties": 0, "clusters": 12, "cluster_wins": 12 } } }, { "key": "v3_mae", "label": "Plan speed error at 3 s", "unit": "m/s", "better": "lower", "value": 1.889, "median": 1.838, "ci": [ 1.152, 2.3 ], "n": 119, "clusters": 12, "paired": { "lebowski_877m": { "delta": -0.1087, "median": -0.125, "ci": [ -0.3518, 0.3059 ], "n": 119, "wins": 78, "ties": 0, "clusters": 12, "cluster_wins": 5 }, "6b5b705f_12800": { "delta": -2.287e-05, "median": -3.358e-05, "ci": [ -0.0001093, 5.602e-05 ], "n": 119, "wins": 66, "ties": 1, "clusters": 12, "cluster_wins": 4 } } } ], "per_unit_file": null, "by_group": { "stock-aimedup": { "n": 4, "clusters": 2, "metrics": [ { "key": "lat_mae", "value": 0.1164, "median": 0.1147, "ci": null, "n": 4, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.0247, "median": -0.02946, "ci": null, "n": 4, "wins": 3, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "6b5b705f_12800": { "delta": 0.002181, "median": 0.003878, "ci": null, "n": 4, "wins": 1, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "lat_mae_bl", "value": 0.1064, "median": 0.105, "ci": null, "n": 4, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.03316, "median": -0.04033, "ci": null, "n": 4, "wins": 3, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "6b5b705f_12800": { "delta": -0.0004011, "median": 0.002824, "ci": null, "n": 4, "wins": 2, "ties": 0, "clusters": 2, "cluster_wins": 1 } } }, { "key": "lat_corr", "value": 0.8201, "median": 0.8954, "ci": null, "n": 4, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.02496, "median": 0.01669, "ci": null, "n": 4, "wins": 3, "ties": 0, "clusters": 2, "cluster_wins": 1 }, "6b5b705f_12800": { "delta": -0.01962, "median": -0.02256, "ci": null, "n": 4, "wins": 0, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "lat_gain", "value": 0.7811, "median": 0.8413, "ci": null, "n": 4, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.1172, "median": -0.07812, "ci": null, "n": 4, "wins": 3, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "6b5b705f_12800": { "delta": 0.03195, "median": 0.0555, "ci": null, "n": 4, "wins": 1, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "lat_rate_rms", "value": 0.846, "median": 0.7988, "ci": null, "n": 4, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.3069, "median": -0.3035, "ci": null, "n": 4, "wins": 4, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "6b5b705f_12800": { "delta": 0.004681, "median": 0.04043, "ci": null, "n": 4, "wins": 1, "ties": 0, "clusters": 2, "cluster_wins": 1 } } }, { "key": "lat_best_lag", "value": 0.6, "median": 0.625, "ci": null, "n": 4, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.0875, "median": 0.075, "ci": null, "n": 4, "wins": 1, "ties": 0, "clusters": 2, "cluster_wins": 1 }, "6b5b705f_12800": { "delta": -0.025, "median": 0.025, "ci": null, "n": 4, "wins": 1, "ties": 1, "clusters": 2, "cluster_wins": 1 } } }, { "key": "lon_mae", "value": 0.7039, "median": 0.6611, "ci": null, "n": 4, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.07378, "median": -0.0569, "ci": null, "n": 4, "wins": 4, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "6b5b705f_12800": { "delta": 0.286, "median": 0.2733, "ci": null, "n": 4, "wins": 0, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "lon_corr", "value": 0.5887, "median": 0.6701, "ci": null, "n": 4, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.05329, "median": 0.03216, "ci": null, "n": 4, "wins": 3, "ties": 0, "clusters": 2, "cluster_wins": 1 }, "6b5b705f_12800": { "delta": 0.1773, "median": 0.1451, "ci": null, "n": 4, "wins": 3, "ties": 0, "clusters": 2, "cluster_wins": 2 } } }, { "key": "lon_brake_mae", "value": 0.2598, "median": 0.2598, "ci": null, "n": 2, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.07182, "median": 0.07182, "ci": null, "n": 2, "wins": 0, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "6b5b705f_12800": { "delta": -0.416, "median": -0.416, "ci": null, "n": 2, "wins": 2, "ties": 0, "clusters": 2, "cluster_wins": 2 } } }, { "key": "lon_rate_rms", "value": 1.808, "median": 1.671, "ci": null, "n": 4, "clusters": 2, "paired": { "lebowski_877m": { "delta": -1.486, "median": -1.402, "ci": null, "n": 4, "wins": 4, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "6b5b705f_12800": { "delta": -0.4442, "median": -0.3842, "ci": null, "n": 4, "wins": 4, "ties": 0, "clusters": 2, "cluster_wins": 2 } } }, { "key": "v3_mae", "value": 1.443, "median": 1.507, "ci": null, "n": 4, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.9456, "median": -0.9687, "ci": null, "n": 4, "wins": 4, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "6b5b705f_12800": { "delta": 8.445e-05, "median": 4.85e-05, "ci": null, "n": 4, "wins": 2, "ties": 0, "clusters": 2, "cluster_wins": 0 } } } ] }, "stock-dual": { "n": 13, "clusters": 4, "metrics": [ { "key": "lat_mae", "value": 0.1626, "median": 0.1509, "ci": null, "n": 13, "clusters": 4, "paired": { "lebowski_877m": { "delta": 0.01739, "median": 0.008061, "ci": null, "n": 13, "wins": 3, "ties": 0, "clusters": 4, "cluster_wins": 0 }, "6b5b705f_12800": { "delta": 0.01979, "median": 0.0189, "ci": null, "n": 13, "wins": 2, "ties": 0, "clusters": 4, "cluster_wins": 0 } } }, { "key": "lat_mae_bl", "value": 0.1503, "median": 0.149, "ci": null, "n": 13, "clusters": 4, "paired": { "lebowski_877m": { "delta": 0.01106, "median": 0.00905, "ci": null, "n": 13, "wins": 5, "ties": 0, "clusters": 4, "cluster_wins": 1 }, "6b5b705f_12800": { "delta": 0.02002, "median": 0.01921, "ci": null, "n": 13, "wins": 1, "ties": 0, "clusters": 4, "cluster_wins": 0 } } }, { "key": "lat_corr", "value": 0.8209, "median": 0.8354, "ci": null, "n": 13, "clusters": 4, "paired": { "lebowski_877m": { "delta": 0.00755, "median": -0.01643, "ci": null, "n": 13, "wins": 5, "ties": 0, "clusters": 4, "cluster_wins": 2 }, "6b5b705f_12800": { "delta": -0.007856, "median": -0.006979, "ci": null, "n": 13, "wins": 3, "ties": 0, "clusters": 4, "cluster_wins": 0 } } }, { "key": "lat_gain", "value": 0.9379, "median": 0.9254, "ci": null, "n": 13, "clusters": 4, "paired": { "lebowski_877m": { "delta": 0.04301, "median": 0.03298, "ci": null, "n": 13, "wins": 3, "ties": 0, "clusters": 4, "cluster_wins": 1 }, "6b5b705f_12800": { "delta": 0.01739, "median": 0.001688, "ci": null, "n": 13, "wins": 6, "ties": 0, "clusters": 4, "cluster_wins": 1 } } }, { "key": "lat_rate_rms", "value": 0.6709, "median": 0.68, "ci": null, "n": 13, "clusters": 4, "paired": { "lebowski_877m": { "delta": -0.505, "median": -0.4422, "ci": null, "n": 13, "wins": 13, "ties": 0, "clusters": 4, "cluster_wins": 4 }, "6b5b705f_12800": { "delta": 0.02524, "median": 0.03855, "ci": null, "n": 13, "wins": 4, "ties": 0, "clusters": 4, "cluster_wins": 1 } } }, { "key": "lat_best_lag", "value": 0.6731, "median": 0.7, "ci": null, "n": 13, "clusters": 4, "paired": { "lebowski_877m": { "delta": 0.1615, "median": 0.15, "ci": null, "n": 13, "wins": 2, "ties": 0, "clusters": 4, "cluster_wins": 1 }, "6b5b705f_12800": { "delta": 0.003846, "median": 0.0, "ci": null, "n": 13, "wins": 4, "ties": 4, "clusters": 4, "cluster_wins": 2 } } }, { "key": "lon_mae", "value": 0.2739, "median": 0.2651, "ci": null, "n": 13, "clusters": 4, "paired": { "lebowski_877m": { "delta": 0.04565, "median": 0.02741, "ci": null, "n": 13, "wins": 2, "ties": 0, "clusters": 4, "cluster_wins": 0 }, "6b5b705f_12800": { "delta": -0.03471, "median": -0.03321, "ci": null, "n": 13, "wins": 11, "ties": 0, "clusters": 4, "cluster_wins": 3 } } }, { "key": "lon_corr", "value": 0.5833, "median": 0.7318, "ci": null, "n": 13, "clusters": 4, "paired": { "lebowski_877m": { "delta": -0.05089, "median": 0.01159, "ci": null, "n": 13, "wins": 7, "ties": 0, "clusters": 4, "cluster_wins": 0 }, "6b5b705f_12800": { "delta": 0.06023, "median": 0.04611, "ci": null, "n": 13, "wins": 11, "ties": 0, "clusters": 4, "cluster_wins": 4 } } }, { "key": "lon_brake_mae", "value": 0.623, "median": 0.665, "ci": null, "n": 6, "clusters": 4, "paired": { "lebowski_877m": { "delta": -0.02426, "median": -0.006943, "ci": null, "n": 6, "wins": 3, "ties": 0, "clusters": 4, "cluster_wins": 2 }, "6b5b705f_12800": { "delta": -0.122, "median": -0.1309, "ci": null, "n": 6, "wins": 6, "ties": 0, "clusters": 4, "cluster_wins": 4 } } }, { "key": "lon_rate_rms", "value": 1.143, "median": 1.12, "ci": null, "n": 13, "clusters": 4, "paired": { "lebowski_877m": { "delta": -1.071, "median": -1.101, "ci": null, "n": 13, "wins": 13, "ties": 0, "clusters": 4, "cluster_wins": 4 }, "6b5b705f_12800": { "delta": -0.1868, "median": -0.2395, "ci": null, "n": 13, "wins": 10, "ties": 0, "clusters": 4, "cluster_wins": 4 } } }, { "key": "v3_mae", "value": 0.8066, "median": 0.8321, "ci": null, "n": 13, "clusters": 4, "paired": { "lebowski_877m": { "delta": -0.03457, "median": -0.06049, "ci": null, "n": 13, "wins": 8, "ties": 0, "clusters": 4, "cluster_wins": 2 }, "6b5b705f_12800": { "delta": 3.25e-06, "median": -4.745e-05, "ci": null, "n": 13, "wins": 7, "ties": 0, "clusters": 4, "cluster_wins": 2 } } } ] }, "stock-newcase": { "n": 2, "clusters": 1, "metrics": [ { "key": "lat_mae", "value": 0.1371, "median": 0.1371, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": 0.002767, "median": 0.002767, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "6b5b705f_12800": { "delta": -0.01307, "median": -0.01307, "ci": null, "n": 2, "wins": 2, "ties": 0, "clusters": 1, "cluster_wins": 1 } } }, { "key": "lat_mae_bl", "value": 0.1322, "median": 0.1322, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.002979, "median": -0.002979, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "6b5b705f_12800": { "delta": -0.01283, "median": -0.01283, "ci": null, "n": 2, "wins": 2, "ties": 0, "clusters": 1, "cluster_wins": 1 } } }, { "key": "lat_corr", "value": 0.8249, "median": 0.8249, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.02267, "median": -0.02267, "ci": null, "n": 2, "wins": 0, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "6b5b705f_12800": { "delta": 0.009384, "median": 0.009384, "ci": null, "n": 2, "wins": 2, "ties": 0, "clusters": 1, "cluster_wins": 1 } } }, { "key": "lat_gain", "value": 0.8114, "median": 0.8114, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.0215, "median": -0.0215, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "6b5b705f_12800": { "delta": -0.005521, "median": -0.005521, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 1 } } }, { "key": "lat_rate_rms", "value": 0.7991, "median": 0.7991, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.4765, "median": -0.4765, "ci": null, "n": 2, "wins": 2, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "6b5b705f_12800": { "delta": -0.01788, "median": -0.01788, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 1 } } }, { "key": "lat_best_lag", "value": 0.6, "median": 0.6, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": 0.1, "median": 0.1, "ci": null, "n": 2, "wins": 0, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "6b5b705f_12800": { "delta": 0.0, "median": 0.0, "ci": null, "n": 2, "wins": 0, "ties": 2, "clusters": 1, "cluster_wins": 0 } } }, { "key": "lon_mae", "value": 0.3717, "median": 0.3717, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": 0.009216, "median": 0.009216, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "6b5b705f_12800": { "delta": -0.007695, "median": -0.007695, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 1 } } }, { "key": "lon_corr", "value": 0.5235, "median": 0.5235, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": 0.03689, "median": 0.03689, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "6b5b705f_12800": { "delta": 0.02129, "median": 0.02129, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 1 } } }, { "key": "lon_brake_mae", "value": 0.3906, "median": 0.3906, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.1127, "median": -0.1127, "ci": null, "n": 2, "wins": 2, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "6b5b705f_12800": { "delta": -0.05765, "median": -0.05765, "ci": null, "n": 2, "wins": 2, "ties": 0, "clusters": 1, "cluster_wins": 1 } } }, { "key": "lon_rate_rms", "value": 2.26, "median": 2.26, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": -1.096, "median": -1.096, "ci": null, "n": 2, "wins": 2, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "6b5b705f_12800": { "delta": -0.6257, "median": -0.6257, "ci": null, "n": 2, "wins": 2, "ties": 0, "clusters": 1, "cluster_wins": 1 } } }, { "key": "v3_mae", "value": 2.129, "median": 2.129, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": 0.03494, "median": 0.03494, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "6b5b705f_12800": { "delta": 6.709e-05, "median": 6.709e-05, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 } } } ] }, "v13-8mm": { "n": 92, "clusters": 3, "metrics": [ { "key": "lat_mae", "value": 0.1824, "median": 0.172, "ci": null, "n": 92, "clusters": 3, "paired": { "lebowski_877m": { "delta": 0.0008183, "median": 0.001515, "ci": null, "n": 92, "wins": 45, "ties": 0, "clusters": 3, "cluster_wins": 1 }, "6b5b705f_12800": { "delta": 0.01521, "median": 0.01506, "ci": null, "n": 92, "wins": 21, "ties": 0, "clusters": 3, "cluster_wins": 0 } } }, { "key": "lat_mae_bl", "value": 0.1669, "median": 0.1615, "ci": null, "n": 92, "clusters": 3, "paired": { "lebowski_877m": { "delta": -0.005292, "median": -0.00606, "ci": null, "n": 92, "wins": 54, "ties": 0, "clusters": 3, "cluster_wins": 2 }, "6b5b705f_12800": { "delta": 0.01178, "median": 0.01026, "ci": null, "n": 92, "wins": 24, "ties": 0, "clusters": 3, "cluster_wins": 0 } } }, { "key": "lat_corr", "value": 0.7972, "median": 0.8935, "ci": null, "n": 92, "clusters": 3, "paired": { "lebowski_877m": { "delta": 0.04873, "median": 0.01884, "ci": null, "n": 92, "wins": 71, "ties": 0, "clusters": 3, "cluster_wins": 3 }, "6b5b705f_12800": { "delta": -0.01266, "median": -0.009102, "ci": null, "n": 92, "wins": 19, "ties": 0, "clusters": 3, "cluster_wins": 0 } } }, { "key": "lat_gain", "value": 0.7703, "median": 0.7914, "ci": null, "n": 92, "clusters": 3, "paired": { "lebowski_877m": { "delta": 0.005694, "median": 0.02109, "ci": null, "n": 92, "wins": 38, "ties": 0, "clusters": 3, "cluster_wins": 1 }, "6b5b705f_12800": { "delta": 0.03935, "median": 0.0506, "ci": null, "n": 92, "wins": 21, "ties": 0, "clusters": 3, "cluster_wins": 0 } } }, { "key": "lat_rate_rms", "value": 1.323, "median": 1.073, "ci": null, "n": 92, "clusters": 3, "paired": { "lebowski_877m": { "delta": -1.295, "median": -1.251, "ci": null, "n": 92, "wins": 91, "ties": 0, "clusters": 3, "cluster_wins": 3 }, "6b5b705f_12800": { "delta": 0.02684, "median": 0.03058, "ci": null, "n": 92, "wins": 31, "ties": 0, "clusters": 3, "cluster_wins": 2 } } }, { "key": "lat_best_lag", "value": 0.6837, "median": 0.7, "ci": null, "n": 92, "clusters": 3, "paired": { "lebowski_877m": { "delta": 0.02337, "median": 0.05, "ci": null, "n": 92, "wins": 30, "ties": 11, "clusters": 3, "cluster_wins": 2 }, "6b5b705f_12800": { "delta": 0.06209, "median": 0.05, "ci": null, "n": 91, "wins": 13, "ties": 24, "clusters": 3, "cluster_wins": 1 } } }, { "key": "lon_mae", "value": 0.3531, "median": 0.3685, "ci": null, "n": 92, "clusters": 3, "paired": { "lebowski_877m": { "delta": -0.02289, "median": -0.01782, "ci": null, "n": 92, "wins": 59, "ties": 0, "clusters": 3, "cluster_wins": 3 }, "6b5b705f_12800": { "delta": -0.01922, "median": -0.01671, "ci": null, "n": 92, "wins": 67, "ties": 0, "clusters": 3, "cluster_wins": 2 } } }, { "key": "lon_corr", "value": 0.261, "median": 0.2469, "ci": null, "n": 92, "clusters": 3, "paired": { "lebowski_877m": { "delta": 0.0273, "median": 0.03386, "ci": null, "n": 92, "wins": 53, "ties": 0, "clusters": 3, "cluster_wins": 2 }, "6b5b705f_12800": { "delta": 0.02071, "median": 0.01751, "ci": null, "n": 92, "wins": 63, "ties": 0, "clusters": 3, "cluster_wins": 3 } } }, { "key": "lon_brake_mae", "value": 0.4581, "median": 0.4568, "ci": null, "n": 65, "clusters": 3, "paired": { "lebowski_877m": { "delta": -0.03101, "median": -0.02592, "ci": null, "n": 65, "wins": 42, "ties": 0, "clusters": 3, "cluster_wins": 3 }, "6b5b705f_12800": { "delta": 0.0153, "median": 0.0214, "ci": null, "n": 65, "wins": 23, "ties": 0, "clusters": 3, "cluster_wins": 2 } } }, { "key": "lon_rate_rms", "value": 4.818, "median": 4.881, "ci": null, "n": 92, "clusters": 3, "paired": { "lebowski_877m": { "delta": -1.514, "median": -1.745, "ci": null, "n": 92, "wins": 71, "ties": 0, "clusters": 3, "cluster_wins": 3 }, "6b5b705f_12800": { "delta": -0.219, "median": -0.1647, "ci": null, "n": 92, "wins": 73, "ties": 0, "clusters": 3, "cluster_wins": 3 } } }, { "key": "v3_mae", "value": 2.125, "median": 1.978, "ci": null, "n": 92, "clusters": 3, "paired": { "lebowski_877m": { "delta": -0.1094, "median": -0.2266, "ci": null, "n": 92, "wins": 63, "ties": 0, "clusters": 3, "cluster_wins": 1 }, "6b5b705f_12800": { "delta": -4.2e-05, "median": -5.763e-05, "ci": null, "n": 92, "wins": 53, "ties": 1, "clusters": 3, "cluster_wins": 2 } } } ] }, "v14-8mm": { "n": 8, "clusters": 2, "metrics": [ { "key": "lat_mae", "value": 0.1746, "median": 0.1788, "ci": null, "n": 8, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.03319, "median": 0.03305, "ci": null, "n": 8, "wins": 0, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "6b5b705f_12800": { "delta": 0.02301, "median": 0.02561, "ci": null, "n": 8, "wins": 0, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "lat_mae_bl", "value": 0.1633, "median": 0.1616, "ci": null, "n": 8, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.02625, "median": 0.02851, "ci": null, "n": 8, "wins": 0, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "6b5b705f_12800": { "delta": 0.02298, "median": 0.02555, "ci": null, "n": 8, "wins": 0, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "lat_corr", "value": 0.8501, "median": 0.9089, "ci": null, "n": 8, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.01792, "median": -0.01273, "ci": null, "n": 8, "wins": 2, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "6b5b705f_12800": { "delta": -0.008315, "median": -0.01162, "ci": null, "n": 8, "wins": 2, "ties": 0, "clusters": 2, "cluster_wins": 1 } } }, { "key": "lat_gain", "value": 0.9021, "median": 0.9196, "ci": null, "n": 8, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.0476, "median": 0.02898, "ci": null, "n": 8, "wins": 2, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "6b5b705f_12800": { "delta": 0.03755, "median": 0.05077, "ci": null, "n": 8, "wins": 2, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "lat_rate_rms", "value": 0.6477, "median": 0.5911, "ci": null, "n": 8, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.5079, "median": -0.4259, "ci": null, "n": 8, "wins": 8, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "6b5b705f_12800": { "delta": 0.03684, "median": 0.02831, "ci": null, "n": 8, "wins": 0, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "lat_best_lag", "value": 0.6, "median": 0.6, "ci": null, "n": 8, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.15, "median": 0.125, "ci": null, "n": 8, "wins": 0, "ties": 1, "clusters": 2, "cluster_wins": 0 }, "6b5b705f_12800": { "delta": 0.01875, "median": 0.025, "ci": null, "n": 8, "wins": 1, "ties": 3, "clusters": 2, "cluster_wins": 1 } } }, { "key": "lon_mae", "value": 0.3585, "median": 0.291, "ci": null, "n": 8, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.04914, "median": 0.03828, "ci": null, "n": 8, "wins": 1, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "6b5b705f_12800": { "delta": -0.02665, "median": -0.02319, "ci": null, "n": 8, "wins": 8, "ties": 0, "clusters": 2, "cluster_wins": 2 } } }, { "key": "lon_corr", "value": 0.6832, "median": 0.6954, "ci": null, "n": 8, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.07153, "median": -0.04595, "ci": null, "n": 8, "wins": 2, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "6b5b705f_12800": { "delta": 0.04639, "median": 0.02298, "ci": null, "n": 8, "wins": 8, "ties": 0, "clusters": 2, "cluster_wins": 2 } } }, { "key": "lon_brake_mae", "value": 0.6375, "median": 0.7169, "ci": null, "n": 6, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.07988, "median": 0.03361, "ci": null, "n": 6, "wins": 3, "ties": 0, "clusters": 2, "cluster_wins": 1 }, "6b5b705f_12800": { "delta": -0.04427, "median": -0.05238, "ci": null, "n": 6, "wins": 6, "ties": 0, "clusters": 2, "cluster_wins": 2 } } }, { "key": "lon_rate_rms", "value": 4.426, "median": 4.237, "ci": null, "n": 8, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.1068, "median": 0.1067, "ci": null, "n": 8, "wins": 3, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "6b5b705f_12800": { "delta": -0.1922, "median": -0.1956, "ci": null, "n": 8, "wins": 5, "ties": 0, "clusters": 2, "cluster_wins": 2 } } }, { "key": "v3_mae", "value": 1.095, "median": 1.066, "ci": null, "n": 8, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.1623, "median": 0.1656, "ci": null, "n": 8, "wins": 2, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "6b5b705f_12800": { "delta": 7.847e-05, "median": 8.341e-05, "ci": null, "n": 8, "wins": 3, "ties": 0, "clusters": 2, "cluster_wins": 0 } } } ] } } }, "truck_closed_loop": { "status": "ok", "unit_of_evidence": "window", "n": 98, "clusters": 13, "condition": "Measured. worldmodel-4B as in the public closed loop, but the model drives a simulation of the G55's measured steering and powertrain response instead of the ideal car. The simulation starts from the truck's state (speed, yaw rate, steering, gear) in its CAN recording of the 2 s before the window; one window, whose 2 s before are not all valid CAN, starts steady at its recorded speed. Lane-following windows from the G55's drives, day and night; distance is to the driver's recorded path; seeds 0 and 1, averaged per window. 98 windows (59 day, 39 night).", "baselines": [ "lebowski_877m", "6b5b705f_12800" ], "metrics": [ { "key": "xtrack_2s", "label": "Distance from the recorded path at 2 s", "unit": "m", "better": "lower", "value": 0.149, "median": 0.09884, "ci": [ 0.1139, 0.1779 ], "n": 98, "clusters": 13, "paired": { "lebowski_877m": { "delta": -0.02052, "median": -0.01634, "ci": [ -0.036, -0.006792 ], "n": 98, "wins": 61, "ties": 0, "clusters": 13, "cluster_wins": 10 }, "6b5b705f_12800": { "delta": 0.01013, "median": 0.005328, "ci": [ -0.002087, 0.01952 ], "n": 98, "wins": 40, "ties": 0, "clusters": 13, "cluster_wins": 3 } } }, { "key": "xtrack_4s", "label": "Distance from the recorded path at 4 s", "unit": "m", "better": "lower", "value": 0.5153, "median": 0.3099, "ci": [ 0.3719, 0.6506 ], "n": 98, "clusters": 13, "paired": { "lebowski_877m": { "delta": -0.004768, "median": -0.02458, "ci": [ -0.1687, 0.1378 ], "n": 98, "wins": 58, "ties": 0, "clusters": 13, "cluster_wins": 6 }, "6b5b705f_12800": { "delta": -0.06145, "median": -0.0005961, "ci": [ -0.1971, 0.02362 ], "n": 98, "wins": 50, "ties": 0, "clusters": 13, "cluster_wins": 7 } } }, { "key": "xtrack_7s", "label": "Distance from the recorded path at 7.2 s", "unit": "m", "better": "lower", "value": 2.028, "median": 0.8116, "ci": [ 1.362, 2.936 ], "n": 98, "clusters": 13, "paired": { "lebowski_877m": { "delta": 0.04907, "median": -0.03221, "ci": [ -0.7938, 1.022 ], "n": 98, "wins": 55, "ties": 0, "clusters": 13, "cluster_wins": 6 }, "6b5b705f_12800": { "delta": -0.9795, "median": -0.05951, "ci": [ -1.704, -0.3842 ], "n": 98, "wins": 61, "ties": 0, "clusters": 13, "cluster_wins": 10 } } }, { "key": "fail_1m", "label": "Share of episodes ever more than 1 m off the path", "unit": "", "better": "lower", "value": 0.4235, "median": 0.0, "ci": [ 0.3206, 0.5357 ], "n": 98, "clusters": 13, "paired": { "lebowski_877m": { "delta": -0.005102, "median": 0.0, "ci": [ -0.05556, 0.06081 ], "n": 98, "wins": 13, "ties": 75, "clusters": 13, "cluster_wins": 6 }, "6b5b705f_12800": { "delta": 0.005102, "median": 0.0, "ci": [ -0.04001, 0.05189 ], "n": 98, "wins": 7, "ties": 85, "clusters": 13, "cluster_wins": 5 } } }, { "key": "lat_jerk", "label": "Requested lateral jerk, RMS", "unit": "m/s³", "better": "lower", "value": 0.2715, "median": 0.1859, "ci": [ 0.2314, 0.3125 ], "n": 98, "clusters": 13, "paired": { "lebowski_877m": { "delta": -0.2704, "median": -0.1884, "ci": [ -0.3514, -0.1956 ], "n": 98, "wins": 91, "ties": 0, "clusters": 13, "cluster_wins": 13 }, "6b5b705f_12800": { "delta": -0.01459, "median": 0.00419, "ci": [ -0.05043, 0.01755 ], "n": 98, "wins": 45, "ties": 0, "clusters": 13, "cluster_wins": 8 } } }, { "key": "long_jerk", "label": "Requested longitudinal jerk, RMS", "unit": "m/s³", "better": "lower", "value": 0.3337, "median": 0.2652, "ci": [ 0.2877, 0.3805 ], "n": 98, "clusters": 13, "paired": { "lebowski_877m": { "delta": -0.05258, "median": -0.08311, "ci": [ -0.1085, 0.007609 ], "n": 98, "wins": 71, "ties": 0, "clusters": 13, "cluster_wins": 9 }, "6b5b705f_12800": { "delta": -0.03955, "median": -0.03197, "ci": [ -0.0629, -0.01965 ], "n": 98, "wins": 67, "ties": 0, "clusters": 13, "cluster_wins": 11 } } }, { "key": "speed_err", "label": "Speed error against the recorded speed, RMS", "unit": "m/s", "better": "lower", "value": 1.201, "median": 0.9812, "ci": [ 0.9754, 1.442 ], "n": 98, "clusters": 13, "paired": { "lebowski_877m": { "delta": 0.1183, "median": 0.1071, "ci": [ -0.02692, 0.2622 ], "n": 98, "wins": 43, "ties": 0, "clusters": 13, "cluster_wins": 4 }, "6b5b705f_12800": { "delta": -0.1098, "median": -0.0864, "ci": [ -0.198, -0.01127 ], "n": 98, "wins": 62, "ties": 0, "clusters": 13, "cluster_wins": 9 } } } ], "per_unit_file": null, "by_group": { "stock-aimedup": { "n": 14, "clusters": 2, "metrics": [ { "key": "xtrack_2s", "value": 0.1359, "median": 0.1038, "ci": null, "n": 14, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.01438, "median": 0.006313, "ci": null, "n": 14, "wins": 7, "ties": 0, "clusters": 2, "cluster_wins": 1 }, "6b5b705f_12800": { "delta": 0.01431, "median": 0.006176, "ci": null, "n": 14, "wins": 5, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "xtrack_4s", "value": 0.5652, "median": 0.4137, "ci": null, "n": 14, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.1141, "median": 0.06052, "ci": null, "n": 14, "wins": 6, "ties": 0, "clusters": 2, "cluster_wins": 1 }, "6b5b705f_12800": { "delta": -0.007025, "median": -0.03032, "ci": null, "n": 14, "wins": 8, "ties": 0, "clusters": 2, "cluster_wins": 1 } } }, { "key": "xtrack_7s", "value": 1.798, "median": 1.236, "ci": null, "n": 14, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.1065, "median": -0.2491, "ci": null, "n": 14, "wins": 10, "ties": 0, "clusters": 2, "cluster_wins": 1 }, "6b5b705f_12800": { "delta": -3.049, "median": -1.455, "ci": null, "n": 14, "wins": 10, "ties": 0, "clusters": 2, "cluster_wins": 2 } } }, { "key": "fail_1m", "value": 0.75, "median": 1.0, "ci": null, "n": 14, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.07143, "median": 0.0, "ci": null, "n": 14, "wins": 2, "ties": 8, "clusters": 2, "cluster_wins": 0 }, "6b5b705f_12800": { "delta": 0.03571, "median": 0.0, "ci": null, "n": 14, "wins": 2, "ties": 10, "clusters": 2, "cluster_wins": 1 } } }, { "key": "lat_jerk", "value": 0.4068, "median": 0.2408, "ci": null, "n": 14, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.164, "median": -0.2057, "ci": null, "n": 14, "wins": 11, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "6b5b705f_12800": { "delta": 0.04371, "median": 0.02341, "ci": null, "n": 14, "wins": 6, "ties": 0, "clusters": 2, "cluster_wins": 1 } } }, { "key": "long_jerk", "value": 0.2986, "median": 0.2551, "ci": null, "n": 14, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.1961, "median": -0.1744, "ci": null, "n": 14, "wins": 13, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "6b5b705f_12800": { "delta": -0.05034, "median": -0.05827, "ci": null, "n": 14, "wins": 10, "ties": 0, "clusters": 2, "cluster_wins": 2 } } }, { "key": "speed_err", "value": 1.838, "median": 1.633, "ci": null, "n": 14, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.07386, "median": 0.05554, "ci": null, "n": 14, "wins": 6, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "6b5b705f_12800": { "delta": 0.1356, "median": 0.7779, "ci": null, "n": 14, "wins": 5, "ties": 0, "clusters": 2, "cluster_wins": 1 } } } ] }, "stock-dual": { "n": 43, "clusters": 5, "metrics": [ { "key": "xtrack_2s", "value": 0.1525, "median": 0.1082, "ci": null, "n": 43, "clusters": 5, "paired": { "lebowski_877m": { "delta": -0.01559, "median": -0.04663, "ci": null, "n": 43, "wins": 29, "ties": 0, "clusters": 5, "cluster_wins": 4 }, "6b5b705f_12800": { "delta": 0.01819, "median": 0.01737, "ci": null, "n": 43, "wins": 16, "ties": 0, "clusters": 5, "cluster_wins": 1 } } }, { "key": "xtrack_4s", "value": 0.4272, "median": 0.2738, "ci": null, "n": 43, "clusters": 5, "paired": { "lebowski_877m": { "delta": -0.02507, "median": -0.04813, "ci": null, "n": 43, "wins": 27, "ties": 0, "clusters": 5, "cluster_wins": 3 }, "6b5b705f_12800": { "delta": 0.01178, "median": 0.01614, "ci": null, "n": 43, "wins": 20, "ties": 0, "clusters": 5, "cluster_wins": 2 } } }, { "key": "xtrack_7s", "value": 1.844, "median": 0.6215, "ci": null, "n": 43, "clusters": 5, "paired": { "lebowski_877m": { "delta": -0.175, "median": -0.03181, "ci": null, "n": 43, "wins": 23, "ties": 0, "clusters": 5, "cluster_wins": 3 }, "6b5b705f_12800": { "delta": -0.4046, "median": -0.04893, "ci": null, "n": 43, "wins": 24, "ties": 0, "clusters": 5, "cluster_wins": 4 } } }, { "key": "fail_1m", "value": 0.3372, "median": 0.0, "ci": null, "n": 43, "clusters": 5, "paired": { "lebowski_877m": { "delta": 0.0, "median": 0.0, "ci": null, "n": 43, "wins": 6, "ties": 33, "clusters": 5, "cluster_wins": 2 }, "6b5b705f_12800": { "delta": -0.01163, "median": 0.0, "ci": null, "n": 43, "wins": 2, "ties": 40, "clusters": 5, "cluster_wins": 1 } } }, { "key": "lat_jerk", "value": 0.2766, "median": 0.1893, "ci": null, "n": 43, "clusters": 5, "paired": { "lebowski_877m": { "delta": -0.2762, "median": -0.1864, "ci": null, "n": 43, "wins": 39, "ties": 0, "clusters": 5, "cluster_wins": 5 }, "6b5b705f_12800": { "delta": -0.02531, "median": 0.005819, "ci": null, "n": 43, "wins": 20, "ties": 0, "clusters": 5, "cluster_wins": 4 } } }, { "key": "long_jerk", "value": 0.3557, "median": 0.2742, "ci": null, "n": 43, "clusters": 5, "paired": { "lebowski_877m": { "delta": -0.01849, "median": -0.07361, "ci": null, "n": 43, "wins": 31, "ties": 0, "clusters": 5, "cluster_wins": 3 }, "6b5b705f_12800": { "delta": -0.06241, "median": -0.06265, "ci": null, "n": 43, "wins": 35, "ties": 0, "clusters": 5, "cluster_wins": 5 } } }, { "key": "speed_err", "value": 1.171, "median": 0.977, "ci": null, "n": 43, "clusters": 5, "paired": { "lebowski_877m": { "delta": 0.08789, "median": 0.09667, "ci": null, "n": 43, "wins": 21, "ties": 0, "clusters": 5, "cluster_wins": 2 }, "6b5b705f_12800": { "delta": -0.1227, "median": -0.1201, "ci": null, "n": 43, "wins": 31, "ties": 0, "clusters": 5, "cluster_wins": 4 } } } ] }, "stock-newcase": { "n": 4, "clusters": 1, "metrics": [ { "key": "xtrack_2s", "value": 0.03341, "median": 0.02876, "ci": null, "n": 4, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.09146, "median": -0.08888, "ci": null, "n": 4, "wins": 3, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "6b5b705f_12800": { "delta": -0.01198, "median": -0.008265, "ci": null, "n": 4, "wins": 2, "ties": 0, "clusters": 1, "cluster_wins": 1 } } }, { "key": "xtrack_4s", "value": 0.1858, "median": 0.1736, "ci": null, "n": 4, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.05484, "median": 0.007601, "ci": null, "n": 4, "wins": 2, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "6b5b705f_12800": { "delta": -0.03354, "median": -0.0358, "ci": null, "n": 4, "wins": 2, "ties": 0, "clusters": 1, "cluster_wins": 1 } } }, { "key": "xtrack_7s", "value": 0.9489, "median": 0.8592, "ci": null, "n": 4, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.6868, "median": -0.7317, "ci": null, "n": 4, "wins": 4, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "6b5b705f_12800": { "delta": -0.8031, "median": -0.6561, "ci": null, "n": 4, "wins": 4, "ties": 0, "clusters": 1, "cluster_wins": 1 } } }, { "key": "fail_1m", "value": 0.5, "median": 0.5, "ci": null, "n": 4, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.125, "median": 0.0, "ci": null, "n": 4, "wins": 1, "ties": 3, "clusters": 1, "cluster_wins": 1 }, "6b5b705f_12800": { "delta": -0.125, "median": 0.0, "ci": null, "n": 4, "wins": 1, "ties": 3, "clusters": 1, "cluster_wins": 1 } } }, { "key": "lat_jerk", "value": 0.329, "median": 0.2453, "ci": null, "n": 4, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.4108, "median": -0.2684, "ci": null, "n": 4, "wins": 4, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "6b5b705f_12800": { "delta": 0.007628, "median": 0.01667, "ci": null, "n": 4, "wins": 2, "ties": 0, "clusters": 1, "cluster_wins": 0 } } }, { "key": "long_jerk", "value": 0.3393, "median": 0.3355, "ci": null, "n": 4, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.04989, "median": -0.06464, "ci": null, "n": 4, "wins": 3, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "6b5b705f_12800": { "delta": -0.0615, "median": -0.05757, "ci": null, "n": 4, "wins": 2, "ties": 0, "clusters": 1, "cluster_wins": 1 } } }, { "key": "speed_err", "value": 0.7499, "median": 0.6306, "ci": null, "n": 4, "clusters": 1, "paired": { "lebowski_877m": { "delta": 0.0711, "median": 0.199, "ci": null, "n": 4, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "6b5b705f_12800": { "delta": 0.04417, "median": 0.1112, "ci": null, "n": 4, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 } } } ] }, "v13-8mm": { "n": 18, "clusters": 3, "metrics": [ { "key": "xtrack_2s", "value": 0.1858, "median": 0.09661, "ci": null, "n": 18, "clusters": 3, "paired": { "lebowski_877m": { "delta": -0.0315, "median": -0.01788, "ci": null, "n": 18, "wins": 11, "ties": 0, "clusters": 3, "cluster_wins": 3 }, "6b5b705f_12800": { "delta": -0.009858, "median": -0.003576, "ci": null, "n": 18, "wins": 10, "ties": 0, "clusters": 3, "cluster_wins": 1 } } }, { "key": "xtrack_4s", "value": 0.4854, "median": 0.2591, "ci": null, "n": 18, "clusters": 3, "paired": { "lebowski_877m": { "delta": -0.3167, "median": -0.03649, "ci": null, "n": 18, "wins": 13, "ties": 0, "clusters": 3, "cluster_wins": 1 }, "6b5b705f_12800": { "delta": -0.3325, "median": 0.00059, "ci": null, "n": 18, "wins": 9, "ties": 0, "clusters": 3, "cluster_wins": 2 } } }, { "key": "xtrack_7s", "value": 1.301, "median": 0.8462, "ci": null, "n": 18, "clusters": 3, "paired": { "lebowski_877m": { "delta": -1.324, "median": 0.0005568, "ci": null, "n": 18, "wins": 9, "ties": 0, "clusters": 3, "cluster_wins": 1 }, "6b5b705f_12800": { "delta": -1.435, "median": 0.009284, "ci": null, "n": 18, "wins": 8, "ties": 0, "clusters": 3, "cluster_wins": 2 } } }, { "key": "fail_1m", "value": 0.3333, "median": 0.0, "ci": null, "n": 18, "clusters": 3, "paired": { "lebowski_877m": { "delta": -0.05556, "median": 0.0, "ci": null, "n": 18, "wins": 2, "ties": 16, "clusters": 3, "cluster_wins": 2 }, "6b5b705f_12800": { "delta": 0.02778, "median": 0.0, "ci": null, "n": 18, "wins": 1, "ties": 15, "clusters": 3, "cluster_wins": 1 } } }, { "key": "lat_jerk", "value": 0.2029, "median": 0.1707, "ci": null, "n": 18, "clusters": 3, "paired": { "lebowski_877m": { "delta": -0.1765, "median": -0.1195, "ci": null, "n": 18, "wins": 18, "ties": 0, "clusters": 3, "cluster_wins": 3 }, "6b5b705f_12800": { "delta": -0.08966, "median": -0.009685, "ci": null, "n": 18, "wins": 13, "ties": 0, "clusters": 3, "cluster_wins": 3 } } }, { "key": "long_jerk", "value": 0.3074, "median": 0.2211, "ci": null, "n": 18, "clusters": 3, "paired": { "lebowski_877m": { "delta": 0.002766, "median": -0.01379, "ci": null, "n": 18, "wins": 11, "ties": 0, "clusters": 3, "cluster_wins": 1 }, "6b5b705f_12800": { "delta": -0.02527, "median": -0.02149, "ci": null, "n": 18, "wins": 12, "ties": 0, "clusters": 3, "cluster_wins": 3 } } }, { "key": "speed_err", "value": 0.8182, "median": 0.6641, "ci": null, "n": 18, "clusters": 3, "paired": { "lebowski_877m": { "delta": -0.1265, "median": -0.02198, "ci": null, "n": 18, "wins": 10, "ties": 0, "clusters": 3, "cluster_wins": 2 }, "6b5b705f_12800": { "delta": -0.2362, "median": -0.1467, "ci": null, "n": 18, "wins": 13, "ties": 0, "clusters": 3, "cluster_wins": 2 } } } ] }, "v14-8mm": { "n": 19, "clusters": 2, "metrics": [ { "key": "xtrack_2s", "value": 0.1401, "median": 0.08717, "ci": null, "n": 19, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.01085, "median": -0.006152, "ci": null, "n": 19, "wins": 11, "ties": 0, "clusters": 2, "cluster_wins": 1 }, "6b5b705f_12800": { "delta": 0.0124, "median": 0.01411, "ci": null, "n": 19, "wins": 7, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "xtrack_4s", "value": 0.7754, "median": 0.4557, "ci": null, "n": 19, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.2596, "median": -0.001524, "ci": null, "n": 19, "wins": 10, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "6b5b705f_12800": { "delta": -0.01638, "median": -0.03642, "ci": null, "n": 19, "wins": 11, "ties": 0, "clusters": 2, "cluster_wins": 1 } } }, { "key": "xtrack_7s", "value": 3.531, "median": 0.7297, "ci": null, "n": 19, "clusters": 2, "paired": { "lebowski_877m": { "delta": 2.127, "median": 0.113, "ci": null, "n": 19, "wins": 9, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "6b5b705f_12800": { "delta": -0.361, "median": -0.0482, "ci": null, "n": 19, "wins": 15, "ties": 0, "clusters": 2, "cluster_wins": 1 } } }, { "key": "fail_1m", "value": 0.4474, "median": 0.0, "ci": null, "n": 19, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.0, "median": 0.0, "ci": null, "n": 19, "wins": 2, "ties": 15, "clusters": 2, "cluster_wins": 1 }, "6b5b705f_12800": { "delta": 0.02632, "median": 0.0, "ci": null, "n": 19, "wins": 1, "ties": 17, "clusters": 2, "cluster_wins": 1 } } }, { "key": "lat_jerk", "value": 0.2133, "median": 0.1843, "ci": null, "n": 19, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.3952, "median": -0.2749, "ci": null, "n": 19, "wins": 19, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "6b5b705f_12800": { "delta": 0.03317, "median": 0.0218, "ci": null, "n": 19, "wins": 4, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "long_jerk", "value": 0.3334, "median": 0.2919, "ci": null, "n": 19, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.07703, "median": -0.07843, "ci": null, "n": 19, "wins": 13, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "6b5b705f_12800": { "delta": 0.01122, "median": 0.001023, "ci": null, "n": 19, "wins": 8, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "speed_err", "value": 1.257, "median": 0.9786, "ci": null, "n": 19, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.4618, "median": 0.4016, "ci": null, "n": 19, "wins": 5, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "6b5b705f_12800": { "delta": -0.1741, "median": -0.03731, "ci": null, "n": 19, "wins": 12, "ties": 0, "clusters": 2, "cluster_wins": 2 } } } ] } }, "simulator": "g55-2026-10-31" }, "truck_recovery": { "status": "ok", "unit_of_evidence": "window", "n": 100, "clusters": 14, "condition": "Measured. The truck closed loop with a disturbance while the model drives; only the tests listed ran. 2° off heading: an extra 2°/s yaw for the first second. 98 windows (59 day, 39 night). 0.5 m off centre: an extra yaw-rate pulse, out and back over the first 2 s, that would move an uncorrected car 0.5 m sideways with no net heading change. 98 windows (59 day, 39 night). Braking lead: windows where the car ahead braked; the gap is derived from the lead distance seen on the recording minus how far the model ran ahead of the recorded car. 2 windows (2 day, 0 night).", "baselines": [ "lebowski_877m", "6b5b705f_12800" ], "metrics": [ { "key": "heading_xtrack_2s", "label": "2° off heading: distance from the path at 2 s", "unit": "m", "better": "lower", "value": 0.4833, "median": 0.3996, "ci": [ 0.3916, 0.5954 ], "n": 98, "clusters": 13, "paired": { "lebowski_877m": { "delta": 0.03041, "median": 0.02461, "ci": [ -0.005783, 0.06971 ], "n": 98, "wins": 43, "ties": 0, "clusters": 13, "cluster_wins": 5 }, "6b5b705f_12800": { "delta": -0.01689, "median": -0.01641, "ci": [ -0.02512, -0.008343 ], "n": 98, "wins": 69, "ties": 0, "clusters": 13, "cluster_wins": 10 } } }, { "key": "heading_xtrack_4s", "label": "2° off heading: distance from the path at 4 s", "unit": "m", "better": "lower", "value": 0.7032, "median": 0.4362, "ci": [ 0.5169, 0.9172 ], "n": 98, "clusters": 13, "paired": { "lebowski_877m": { "delta": 0.0006493, "median": -0.01695, "ci": [ -0.1392, 0.1235 ], "n": 98, "wins": 50, "ties": 0, "clusters": 13, "cluster_wins": 4 }, "6b5b705f_12800": { "delta": -0.07494, "median": -0.04163, "ci": [ -0.1285, -0.026 ], "n": 98, "wins": 66, "ties": 0, "clusters": 13, "cluster_wins": 12 } } }, { "key": "heading_lat_jerk", "label": "2° off heading: requested lateral jerk, RMS", "unit": "m/s³", "better": "lower", "value": 0.4026, "median": 0.3137, "ci": [ 0.3415, 0.4724 ], "n": 98, "clusters": 13, "paired": { "lebowski_877m": { "delta": -0.4844, "median": -0.4605, "ci": [ -0.5633, -0.4033 ], "n": 98, "wins": 92, "ties": 0, "clusters": 13, "cluster_wins": 13 }, "6b5b705f_12800": { "delta": 0.02504, "median": 0.004706, "ci": [ -0.0116, 0.063 ], "n": 98, "wins": 44, "ties": 0, "clusters": 13, "cluster_wins": 6 } } }, { "key": "offcentre_xtrack_2s", "label": "0.5 m off centre: distance from the path at 2 s", "unit": "m", "better": "lower", "value": 0.2626, "median": 0.2202, "ci": [ 0.2377, 0.2863 ], "n": 98, "clusters": 13, "paired": { "lebowski_877m": { "delta": -0.004426, "median": 0.006933, "ci": [ -0.02741, 0.01926 ], "n": 98, "wins": 45, "ties": 0, "clusters": 13, "cluster_wins": 7 }, "6b5b705f_12800": { "delta": -0.00441, "median": -0.005375, "ci": [ -0.01091, 0.002825 ], "n": 98, "wins": 56, "ties": 0, "clusters": 13, "cluster_wins": 8 } } }, { "key": "offcentre_xtrack_4s", "label": "0.5 m off centre: distance from the path at 4 s", "unit": "m", "better": "lower", "value": 0.6586, "median": 0.3982, "ci": [ 0.5297, 0.7813 ], "n": 98, "clusters": 13, "paired": { "lebowski_877m": { "delta": -0.004248, "median": -0.03426, "ci": [ -0.07712, 0.06988 ], "n": 98, "wins": 58, "ties": 0, "clusters": 13, "cluster_wins": 8 }, "6b5b705f_12800": { "delta": -0.05171, "median": -0.003829, "ci": [ -0.08858, -0.01848 ], "n": 98, "wins": 53, "ties": 0, "clusters": 13, "cluster_wins": 10 } } }, { "key": "offcentre_lat_jerk", "label": "0.5 m off centre: requested lateral jerk, RMS", "unit": "m/s³", "better": "lower", "value": 0.5338, "median": 0.4277, "ci": [ 0.4765, 0.5865 ], "n": 98, "clusters": 13, "paired": { "lebowski_877m": { "delta": -0.5766, "median": -0.5839, "ci": [ -0.6503, -0.504 ], "n": 98, "wins": 95, "ties": 0, "clusters": 13, "cluster_wins": 13 }, "6b5b705f_12800": { "delta": -0.03285, "median": 0.001431, "ci": [ -0.08797, 0.01113 ], "n": 98, "wins": 46, "ties": 0, "clusters": 13, "cluster_wins": 6 } } }, { "key": "brake_ahead_4s", "label": "Braking lead: distance ahead of the recorded car at 4 s", "unit": "m", "better": "lower", "value": -0.5526, "median": -0.5526, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": 0.7849, "median": 0.7849, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "6b5b705f_12800": { "delta": -0.1834, "median": -0.1834, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 1 } } }, { "key": "brake_gap_min", "label": "Braking lead: smallest gap to the lead (derived)", "unit": "m", "better": "higher", "value": 26.02, "median": 26.02, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": -1.437, "median": -1.437, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "6b5b705f_12800": { "delta": 0.2957, "median": 0.2957, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 1 } } }, { "key": "brake_speed_err", "label": "Braking lead: speed error against the recorded speed, RMS", "unit": "m/s", "better": "lower", "value": 1.133, "median": 1.133, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.07059, "median": -0.07059, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "6b5b705f_12800": { "delta": 0.01654, "median": 0.01654, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 } } } ], "per_unit_file": null, "by_group": { "stock-aimedup": { "n": 14, "clusters": 2, "metrics": [ { "key": "heading_xtrack_2s", "value": 0.3886, "median": 0.3719, "ci": null, "n": 14, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.003195, "median": -0.0004837, "ci": null, "n": 14, "wins": 8, "ties": 0, "clusters": 2, "cluster_wins": 1 }, "6b5b705f_12800": { "delta": -0.04138, "median": -0.04021, "ci": null, "n": 14, "wins": 14, "ties": 0, "clusters": 2, "cluster_wins": 2 } } }, { "key": "heading_xtrack_4s", "value": 0.6922, "median": 0.5184, "ci": null, "n": 14, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.1392, "median": -0.001257, "ci": null, "n": 14, "wins": 7, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "6b5b705f_12800": { "delta": -0.06076, "median": -0.08802, "ci": null, "n": 14, "wins": 8, "ties": 0, "clusters": 2, "cluster_wins": 2 } } }, { "key": "heading_lat_jerk", "value": 0.4581, "median": 0.2845, "ci": null, "n": 14, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.3098, "median": -0.2768, "ci": null, "n": 14, "wins": 12, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "6b5b705f_12800": { "delta": 0.06537, "median": 0.0277, "ci": null, "n": 14, "wins": 6, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "offcentre_xtrack_2s", "value": 0.2719, "median": 0.2516, "ci": null, "n": 14, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.009002, "median": 0.01379, "ci": null, "n": 14, "wins": 6, "ties": 0, "clusters": 2, "cluster_wins": 1 }, "6b5b705f_12800": { "delta": -0.02174, "median": -0.02397, "ci": null, "n": 14, "wins": 11, "ties": 0, "clusters": 2, "cluster_wins": 2 } } }, { "key": "offcentre_xtrack_4s", "value": 0.6242, "median": 0.5274, "ci": null, "n": 14, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.03444, "median": 0.06658, "ci": null, "n": 14, "wins": 7, "ties": 0, "clusters": 2, "cluster_wins": 1 }, "6b5b705f_12800": { "delta": -0.01993, "median": -0.01379, "ci": null, "n": 14, "wins": 9, "ties": 0, "clusters": 2, "cluster_wins": 1 } } }, { "key": "offcentre_lat_jerk", "value": 0.6376, "median": 0.5222, "ci": null, "n": 14, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.4241, "median": -0.5938, "ci": null, "n": 14, "wins": 12, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "6b5b705f_12800": { "delta": -0.1282, "median": 0.00843, "ci": null, "n": 14, "wins": 7, "ties": 0, "clusters": 2, "cluster_wins": 1 } } } ] }, "stock-dual": { "n": 45, "clusters": 6, "metrics": [ { "key": "heading_xtrack_2s", "value": 0.41, "median": 0.3531, "ci": null, "n": 43, "clusters": 5, "paired": { "lebowski_877m": { "delta": -0.005596, "median": -0.01143, "ci": null, "n": 43, "wins": 23, "ties": 0, "clusters": 5, "cluster_wins": 3 }, "6b5b705f_12800": { "delta": -0.01166, "median": -0.01254, "ci": null, "n": 43, "wins": 30, "ties": 0, "clusters": 5, "cluster_wins": 4 } } }, { "key": "heading_xtrack_4s", "value": 0.5413, "median": 0.341, "ci": null, "n": 43, "clusters": 5, "paired": { "lebowski_877m": { "delta": -0.09494, "median": -0.04548, "ci": null, "n": 43, "wins": 26, "ties": 0, "clusters": 5, "cluster_wins": 3 }, "6b5b705f_12800": { "delta": -0.06003, "median": -0.04554, "ci": null, "n": 43, "wins": 28, "ties": 0, "clusters": 5, "cluster_wins": 5 } } }, { "key": "heading_lat_jerk", "value": 0.4379, "median": 0.3185, "ci": null, "n": 43, "clusters": 5, "paired": { "lebowski_877m": { "delta": -0.4585, "median": -0.4414, "ci": null, "n": 43, "wins": 39, "ties": 0, "clusters": 5, "cluster_wins": 5 }, "6b5b705f_12800": { "delta": 0.05624, "median": 0.01106, "ci": null, "n": 43, "wins": 14, "ties": 0, "clusters": 5, "cluster_wins": 1 } } }, { "key": "offcentre_xtrack_2s", "value": 0.2609, "median": 0.2214, "ci": null, "n": 43, "clusters": 5, "paired": { "lebowski_877m": { "delta": -0.03846, "median": -0.02679, "ci": null, "n": 43, "wins": 27, "ties": 0, "clusters": 5, "cluster_wins": 5 }, "6b5b705f_12800": { "delta": -0.005231, "median": -0.005256, "ci": null, "n": 43, "wins": 26, "ties": 0, "clusters": 5, "cluster_wins": 3 } } }, { "key": "offcentre_xtrack_4s", "value": 0.6445, "median": 0.3989, "ci": null, "n": 43, "clusters": 5, "paired": { "lebowski_877m": { "delta": -0.02593, "median": -0.04013, "ci": null, "n": 43, "wins": 25, "ties": 0, "clusters": 5, "cluster_wins": 4 }, "6b5b705f_12800": { "delta": -0.05694, "median": 0.005329, "ci": null, "n": 43, "wins": 21, "ties": 0, "clusters": 5, "cluster_wins": 4 } } }, { "key": "offcentre_lat_jerk", "value": 0.5798, "median": 0.4371, "ci": null, "n": 43, "clusters": 5, "paired": { "lebowski_877m": { "delta": -0.5968, "median": -0.5801, "ci": null, "n": 43, "wins": 42, "ties": 0, "clusters": 5, "cluster_wins": 5 }, "6b5b705f_12800": { "delta": -0.01854, "median": 0.004821, "ci": null, "n": 43, "wins": 20, "ties": 0, "clusters": 5, "cluster_wins": 2 } } }, { "key": "brake_ahead_4s", "value": -0.5526, "median": -0.5526, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": 0.7849, "median": 0.7849, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "6b5b705f_12800": { "delta": -0.1834, "median": -0.1834, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 1 } } }, { "key": "brake_gap_min", "value": 26.02, "median": 26.02, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": -1.437, "median": -1.437, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "6b5b705f_12800": { "delta": 0.2957, "median": 0.2957, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 1 } } }, { "key": "brake_speed_err", "value": 1.133, "median": 1.133, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.07059, "median": -0.07059, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "6b5b705f_12800": { "delta": 0.01654, "median": 0.01654, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 } } } ] }, "stock-newcase": { "n": 4, "clusters": 1, "metrics": [ { "key": "heading_xtrack_2s", "value": 0.3523, "median": 0.3703, "ci": null, "n": 4, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.09827, "median": -0.1133, "ci": null, "n": 4, "wins": 3, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "6b5b705f_12800": { "delta": 0.003201, "median": 0.006632, "ci": null, "n": 4, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 } } }, { "key": "heading_xtrack_4s", "value": 0.2627, "median": 0.1672, "ci": null, "n": 4, "clusters": 1, "paired": { "lebowski_877m": { "delta": 0.04298, "median": 0.05728, "ci": null, "n": 4, "wins": 2, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "6b5b705f_12800": { "delta": -0.03732, "median": -0.05788, "ci": null, "n": 4, "wins": 3, "ties": 0, "clusters": 1, "cluster_wins": 1 } } }, { "key": "heading_lat_jerk", "value": 0.3147, "median": 0.2361, "ci": null, "n": 4, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.6575, "median": -0.6379, "ci": null, "n": 4, "wins": 4, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "6b5b705f_12800": { "delta": -0.03563, "median": -0.04191, "ci": null, "n": 4, "wins": 3, "ties": 0, "clusters": 1, "cluster_wins": 1 } } }, { "key": "offcentre_xtrack_2s", "value": 0.2966, "median": 0.2219, "ci": null, "n": 4, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.003192, "median": 0.02719, "ci": null, "n": 4, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "6b5b705f_12800": { "delta": 0.005707, "median": 0.004676, "ci": null, "n": 4, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 } } }, { "key": "offcentre_xtrack_4s", "value": 0.5236, "median": 0.4675, "ci": null, "n": 4, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.2041, "median": -0.08799, "ci": null, "n": 4, "wins": 4, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "6b5b705f_12800": { "delta": -0.136, "median": -0.06794, "ci": null, "n": 4, "wins": 4, "ties": 0, "clusters": 1, "cluster_wins": 1 } } }, { "key": "offcentre_lat_jerk", "value": 0.6872, "median": 0.4447, "ci": null, "n": 4, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.7102, "median": -0.6603, "ci": null, "n": 4, "wins": 4, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "6b5b705f_12800": { "delta": 0.03145, "median": 0.02332, "ci": null, "n": 4, "wins": 2, "ties": 0, "clusters": 1, "cluster_wins": 0 } } } ] }, "v13-8mm": { "n": 18, "clusters": 3, "metrics": [ { "key": "heading_xtrack_2s", "value": 0.8528, "median": 0.801, "ci": null, "n": 18, "clusters": 3, "paired": { "lebowski_877m": { "delta": 0.1598, "median": 0.1847, "ci": null, "n": 18, "wins": 3, "ties": 0, "clusters": 3, "cluster_wins": 0 }, "6b5b705f_12800": { "delta": -0.01632, "median": 0.002702, "ci": null, "n": 18, "wins": 8, "ties": 0, "clusters": 3, "cluster_wins": 2 } } }, { "key": "heading_xtrack_4s", "value": 1.134, "median": 0.7096, "ci": null, "n": 18, "clusters": 3, "paired": { "lebowski_877m": { "delta": -0.09721, "median": 0.01346, "ci": null, "n": 18, "wins": 9, "ties": 0, "clusters": 3, "cluster_wins": 1 }, "6b5b705f_12800": { "delta": -0.1618, "median": -0.02552, "ci": null, "n": 18, "wins": 14, "ties": 0, "clusters": 3, "cluster_wins": 2 } } }, { "key": "heading_lat_jerk", "value": 0.4027, "median": 0.4008, "ci": null, "n": 18, "clusters": 3, "paired": { "lebowski_877m": { "delta": -0.4703, "median": -0.4617, "ci": null, "n": 18, "wins": 18, "ties": 0, "clusters": 3, "cluster_wins": 3 }, "6b5b705f_12800": { "delta": -0.03977, "median": -0.01487, "ci": null, "n": 18, "wins": 14, "ties": 0, "clusters": 3, "cluster_wins": 3 } } }, { "key": "offcentre_xtrack_2s", "value": 0.2835, "median": 0.217, "ci": null, "n": 18, "clusters": 3, "paired": { "lebowski_877m": { "delta": 0.02544, "median": 0.04518, "ci": null, "n": 18, "wins": 6, "ties": 0, "clusters": 3, "cluster_wins": 0 }, "6b5b705f_12800": { "delta": 0.007654, "median": 0.01188, "ci": null, "n": 18, "wins": 6, "ties": 0, "clusters": 3, "cluster_wins": 1 } } }, { "key": "offcentre_xtrack_4s", "value": 0.6192, "median": 0.3263, "ci": null, "n": 18, "clusters": 3, "paired": { "lebowski_877m": { "delta": -0.07389, "median": -0.01232, "ci": null, "n": 18, "wins": 10, "ties": 0, "clusters": 3, "cluster_wins": 2 }, "6b5b705f_12800": { "delta": -0.07157, "median": 0.01938, "ci": null, "n": 18, "wins": 8, "ties": 0, "clusters": 3, "cluster_wins": 2 } } }, { "key": "offcentre_lat_jerk", "value": 0.4198, "median": 0.3815, "ci": null, "n": 18, "clusters": 3, "paired": { "lebowski_877m": { "delta": -0.462, "median": -0.4878, "ci": null, "n": 18, "wins": 18, "ties": 0, "clusters": 3, "cluster_wins": 3 }, "6b5b705f_12800": { "delta": -0.06048, "median": -0.01079, "ci": null, "n": 18, "wins": 12, "ties": 0, "clusters": 3, "cluster_wins": 3 } } } ] }, "v14-8mm": { "n": 19, "clusters": 2, "metrics": [ { "key": "heading_xtrack_2s", "value": 0.3967, "median": 0.3825, "ci": null, "n": 19, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.04118, "median": 0.0281, "ci": null, "n": 19, "wins": 6, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "6b5b705f_12800": { "delta": -0.01545, "median": -0.01984, "ci": null, "n": 19, "wins": 16, "ties": 0, "clusters": 2, "cluster_wins": 2 } } }, { "key": "heading_xtrack_4s", "value": 0.7622, "median": 0.3901, "ci": null, "n": 19, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.1987, "median": 0.07295, "ci": null, "n": 19, "wins": 6, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "6b5b705f_12800": { "delta": -0.04474, "median": -0.04425, "ci": null, "n": 19, "wins": 13, "ties": 0, "clusters": 2, "cluster_wins": 2 } } }, { "key": "heading_lat_jerk", "value": 0.3001, "median": 0.2827, "ci": null, "n": 19, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.6483, "median": -0.5372, "ci": null, "n": 19, "wins": 19, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "6b5b705f_12800": { "delta": -0.001128, "median": 0.009065, "ci": null, "n": 19, "wins": 7, "ties": 0, "clusters": 2, "cluster_wins": 1 } } }, { "key": "offcentre_xtrack_2s", "value": 0.2326, "median": 0.2069, "ci": null, "n": 19, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.03414, "median": 0.05002, "ci": null, "n": 19, "wins": 5, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "6b5b705f_12800": { "delta": -0.003337, "median": -0.0116, "ci": null, "n": 19, "wins": 12, "ties": 0, "clusters": 2, "cluster_wins": 2 } } }, { "key": "offcentre_xtrack_4s", "value": 0.7818, "median": 0.3941, "ci": null, "n": 19, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.1244, "median": -0.07211, "ci": null, "n": 19, "wins": 12, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "6b5b705f_12800": { "delta": -0.02675, "median": -0.03093, "ci": null, "n": 19, "wins": 11, "ties": 0, "clusters": 2, "cluster_wins": 2 } } }, { "key": "offcentre_lat_jerk", "value": 0.4287, "median": 0.4226, "ci": null, "n": 19, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.7238, "median": -0.6656, "ci": null, "n": 19, "wins": 19, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "6b5b705f_12800": { "delta": 0.01769, "median": 0.02441, "ci": null, "n": 19, "wins": 5, "ties": 0, "clusters": 2, "cluster_wins": 0 } } } ] } }, "simulator": "g55-2026-10-31" } }, "stills": [], "limits": [ "Open-loop scores compare a model's action with what the recording driver or car did; a model that drives differently but well is scored as wrong. comma's RL residual makes the lateral action smoother, and earlier than its base's by 0.10 to 0.26 s (measured on the G55 and on 16 of comma's cars); both score as error here. Measured on the 2026-10-27 deep dive: open-loop accuracy did not predict closed-loop driving.", "Closed-loop numbers come from a generative world model, not a car. In a separate test on the G55's windows, worldmodel-4B followed an added yaw command through 2 s but showed 0 to 20% of an added lateral offset 3 to 4 s later; past about 4 s the recorded anchor frames also pull every model back toward the recorded path. The front page reports 2 s.", "Intervals are 95% percentile bootstraps that resample whole drives (the drive a public segment was cut from, the recorded drive a truck window comes from), so segments of one drive count once. Below 10 drives there is no interval: the braking-lead test comes from one drive and reports means and win counts only.", "Paired deltas are model minus baseline on the units both have (for lateral gain, the change in its distance from 1); wins count units where the model is better in the metric's direction.", "The lead of the lateral request is the shift, searched from 0 to 1.5 s, at which the request best correlates with the car's later lateral motion; a larger value means the request comes earlier. A value at either edge is censored and left out. Plan speed error at 3 s reads the plan head, which does not separate models built on one frozen base: it is identical on every segment for the four f22e06e5 models, and for Cinque v2 and v3.", "Truck open loop comes from few drives, and the camera setup changed between them (lens, housing, aim); the pooled number mixes setups, so read the per-setup numbers too. The 8 mm setups' wide camera recorded about 7-10 fps, so its frame can be up to ~60 ms from the narrow one. On the V14 setup up to 7 of a window's 700 steps are ~100 ms (one camera frame dropped). The camera's aim is the truck's live calibration learned on each drive; drives whose recorded calibration was carried over from another drive are left out.", "Truck closed loop: the steering and powertrain simulation is fitted to the G55, not validated on it in closed loop; the reference is the driver's own line, not the lane centre. The pooled number mixes camera setups, so read the per-setup numbers too. On the V14 setup the wide frame can be up to 90 ms from the simulated one (its camera recorded 7-9 fps).", "Recovery tests are disturbances rendered by the world model (yaw it renders; a one-frame jump it does not, so none is used). The braking-lead test has 2 windows from one drive, whose recorded calibration was carried over from an earlier drive." ] }