{ "schema": 1, "model": { "id": "f22e06e5_56320", "name": null, "sha256": "909a140019050a9d7f00db22dd4f4085928d2af9474a82244672ddab964f95a8", "source": { "kind": "huggingface", "ref": "ea306f534159ac9a3caab0ac3bd4a5f82f9cd805", "url": "https://huggingface.co/commaai/openpilot_driving_models/blob/ea306f534159ac9a3caab0ac3bd4a5f82f9cd805/ea2f516f-f758-42be-a245-913bb818a64f/32768/big_driving_supercombo.onnx", "date": "2026-11-05" }, "first_public": "2026-11-05", "on_master": null, "params_m": 382.9, "gmac": 46.7, "family": "residual", "notes": "Resolved from hf:ea2f516f/32768; checkpoint f22e06e5-3b3f-48d2-a6b9-2f32dc20e707/56320/ea2f516f-f758-42be-a245-913bb818a64f." }, "scored": { "date": "2026-11-05T06:41:33Z", "suite": "geval-1", "code": "0194c656104b" }, "baselines": [ "lebowski_877m", "f22b8b34_16384" ], "sections": { "public_open_loop": { "status": "ok", "unit_of_evidence": "segment", "n": 39, "clusters": 23, "condition": "Measured. comma's public two-camera segments (openpilot CI data), native intrinsics and the device's own calibration, both cameras; every model on the same decoded frames. Truth is what the recording car did next: carState speed and acceleration, yaw rate from the device's cameraOdometry. Segments where the car never moves have no data. 39 of the 45 listed segments have a score, from 23 drives and 20 cars.", "baselines": [ "lebowski_877m", "f22b8b34_16384" ], "metrics": [ { "key": "lat_mae", "label": "Lateral error", "unit": "m/s²", "better": "lower", "value": 0.1482, "median": 0.1347, "ci": [ 0.1188, 0.1792 ], "n": 39, "clusters": 23, "paired": { "lebowski_877m": { "delta": 0.03952, "median": 0.02478, "ci": [ 0.02841, 0.05089 ], "n": 39, "wins": 4, "ties": 0, "clusters": 23, "cluster_wins": 1 }, "f22b8b34_16384": { "delta": -0.000226, "median": 0.0002584, "ci": [ -0.007318, 0.006307 ], "n": 39, "wins": 19, "ties": 0, "clusters": 23, "cluster_wins": 10 } } }, { "key": "lat_mae_bl", "label": "Lateral error at the model's own best lead", "unit": "m/s²", "better": "lower", "value": 0.1392, "median": 0.1263, "ci": [ 0.1102, 0.1692 ], "n": 39, "clusters": 23, "paired": { "lebowski_877m": { "delta": 0.03775, "median": 0.0236, "ci": [ 0.02573, 0.0498 ], "n": 39, "wins": 6, "ties": 0, "clusters": 23, "cluster_wins": 3 }, "f22b8b34_16384": { "delta": 8.466e-05, "median": 0.0004167, "ci": [ -0.00768, 0.006923 ], "n": 39, "wins": 19, "ties": 0, "clusters": 23, "cluster_wins": 11 } } }, { "key": "lat_corr", "label": "Lateral correlation with the car's later motion", "unit": "", "better": "higher", "value": 0.8236, "median": 0.9336, "ci": [ 0.7371, 0.9024 ], "n": 39, "clusters": 23, "paired": { "lebowski_877m": { "delta": -0.03398, "median": -0.01073, "ci": [ -0.08168, 0.0003673 ], "n": 39, "wins": 9, "ties": 0, "clusters": 23, "cluster_wins": 7 }, "f22b8b34_16384": { "delta": 0.004147, "median": 0.002009, "ci": [ -0.0144, 0.02134 ], "n": 39, "wins": 25, "ties": 0, "clusters": 23, "cluster_wins": 12 } } }, { "key": "lat_gain", "label": "Lateral gain (1 = same size as the car's motion)", "unit": "", "better": "near1", "value": 0.6981, "median": 0.7544, "ci": [ 0.5709, 0.82 ], "n": 39, "clusters": 23, "paired": { "lebowski_877m": { "delta": 0.1223, "median": 0.05619, "ci": [ 0.06091, 0.1933 ], "n": 39, "wins": 10, "ties": 0, "clusters": 23, "cluster_wins": 4 }, "f22b8b34_16384": { "delta": 0.01308, "median": -0.008342, "ci": [ -0.02346, 0.06479 ], "n": 39, "wins": 22, "ties": 0, "clusters": 23, "cluster_wins": 11 } } }, { "key": "lat_rate_rms", "label": "Lateral action rate, RMS (smoothness)", "unit": "m/s³", "better": "lower", "value": 0.4637, "median": 0.3951, "ci": [ 0.362, 0.5822 ], "n": 39, "clusters": 23, "paired": { "lebowski_877m": { "delta": -0.2002, "median": -0.1837, "ci": [ -0.241, -0.1569 ], "n": 39, "wins": 37, "ties": 0, "clusters": 23, "cluster_wins": 22 }, "f22b8b34_16384": { "delta": -0.006277, "median": -0.004882, "ci": [ -0.02056, 0.004974 ], "n": 39, "wins": 22, "ties": 0, "clusters": 23, "cluster_wins": 10 } } }, { "key": "lat_best_lag", "label": "Lead of the lateral request over the car's motion", "unit": "s", "better": "lower", "value": 0.4971, "median": 0.45, "ci": [ 0.4029, 0.6048 ], "n": 35, "clusters": 21, "paired": { "lebowski_877m": { "delta": 0.09853, "median": 0.05, "ci": [ 0.03371, 0.181 ], "n": 34, "wins": 7, "ties": 2, "clusters": 21, "cluster_wins": 3 }, "f22b8b34_16384": { "delta": 0.01364, "median": 0.0, "ci": [ -0.01731, 0.05715 ], "n": 33, "wins": 9, "ties": 16, "clusters": 21, "cluster_wins": 8 } } }, { "key": "lon_mae", "label": "Longitudinal error", "unit": "m/s²", "better": "lower", "value": 0.3349, "median": 0.321, "ci": [ 0.2901, 0.3844 ], "n": 39, "clusters": 23, "paired": { "lebowski_877m": { "delta": 0.0268, "median": 0.02377, "ci": [ 0.00917, 0.04806 ], "n": 39, "wins": 14, "ties": 0, "clusters": 23, "cluster_wins": 5 }, "f22b8b34_16384": { "delta": 0.003191, "median": -0.002338, "ci": [ -0.01043, 0.01836 ], "n": 39, "wins": 21, "ties": 0, "clusters": 23, "cluster_wins": 12 } } }, { "key": "lon_corr", "label": "Longitudinal correlation", "unit": "", "better": "higher", "value": 0.6947, "median": 0.8346, "ci": [ 0.5704, 0.7988 ], "n": 39, "clusters": 23, "paired": { "lebowski_877m": { "delta": -0.02305, "median": -0.007389, "ci": [ -0.07229, 0.02255 ], "n": 39, "wins": 15, "ties": 0, "clusters": 23, "cluster_wins": 9 }, "f22b8b34_16384": { "delta": -0.03772, "median": -0.005913, "ci": [ -0.0852, -0.002659 ], "n": 39, "wins": 17, "ties": 0, "clusters": 23, "cluster_wins": 10 } } }, { "key": "lon_brake_mae", "label": "Longitudinal error while braking", "unit": "m/s²", "better": "lower", "value": 0.4375, "median": 0.4105, "ci": [ 0.3695, 0.52 ], "n": 30, "clusters": 21, "paired": { "lebowski_877m": { "delta": 0.003241, "median": 0.01572, "ci": [ -0.02881, 0.04411 ], "n": 30, "wins": 12, "ties": 0, "clusters": 21, "cluster_wins": 9 }, "f22b8b34_16384": { "delta": 0.04088, "median": 0.02614, "ci": [ 0.01511, 0.07458 ], "n": 30, "wins": 8, "ties": 0, "clusters": 21, "cluster_wins": 5 } } }, { "key": "lon_rate_rms", "label": "Longitudinal action rate, RMS (smoothness)", "unit": "m/s³", "better": "lower", "value": 0.895, "median": 0.7385, "ci": [ 0.7642, 1.051 ], "n": 39, "clusters": 23, "paired": { "lebowski_877m": { "delta": -0.438, "median": -0.3805, "ci": [ -0.5821, -0.3144 ], "n": 39, "wins": 36, "ties": 0, "clusters": 23, "cluster_wins": 23 }, "f22b8b34_16384": { "delta": 0.02536, "median": 0.02662, "ci": [ 0.006149, 0.04262 ], "n": 39, "wins": 11, "ties": 0, "clusters": 23, "cluster_wins": 6 } } }, { "key": "v3_mae", "label": "Plan speed error at 3 s", "unit": "m/s", "better": "lower", "value": 0.7602, "median": 0.7121, "ci": [ 0.6567, 0.888 ], "n": 39, "clusters": 23, "paired": { "lebowski_877m": { "delta": 0.04442, "median": 0.04624, "ci": [ 0.009032, 0.07746 ], "n": 39, "wins": 12, "ties": 0, "clusters": 23, "cluster_wins": 8 }, "f22b8b34_16384": { "delta": 0.0, "median": 0.0, "ci": [ 0.0, 0.0 ], "n": 39, "wins": 0, "ties": 39, "clusters": 23, "cluster_wins": 0 } } } ], "per_unit_file": "per_unit/f22e06e5_56320/public_open_loop.csv" }, "public_closed_loop": { "status": "not_run", "unit_of_evidence": "segment", "n": 0, "condition": "Measured. comma's worldmodel-4B generates each next frame at 5 Hz from 9 history frames, 5 recorded anchor frames and the car's pose; the model drives comma's ideal car (the requested curvature and acceleration happen at once) for 7.2 s from the recorded start. Distance is to the recording car's own path. One window per segment, nearest the middle, where the car holds at least 5 m/s for the window and the 4.8 s before it; segments with no such stretch are left out. Seeds 0 and 1, averaged per segment.", "baselines": [], "metrics": [], "per_unit_file": null }, "truck_open_loop": { "status": "ok", "unit_of_evidence": "window", "n": 119, "clusters": 12, "condition": "Measured. The G55's recorded drives replayed through the model with both of the truck's cameras (each setup's own wide-camera calibration), road-plane warp to 1.45 m. Where the narrow lens does not cover comma's narrow view (the 8 mm lens in the V13 case), those pixels come from the wide camera, brightness matched. Truth is the truck's CAN speed and yaw rate. 30 s windows after a 5 s warm-up; window = unit, drive = cluster; results are also given per camera setup. Corpus windows of 30 s from the truck's recorded drives, scored per camera setup.", "baselines": [ "lebowski_877m", "f22b8b34_16384" ], "metrics": [ { "key": "lat_mae", "label": "Lateral error", "unit": "m/s²", "better": "lower", "value": 0.181, "median": 0.1764, "ci": [ 0.1526, 0.1869 ], "n": 119, "clusters": 12, "paired": { "lebowski_877m": { "delta": 0.008219, "median": 0.008335, "ci": [ 0.0001299, 0.02408 ], "n": 119, "wins": 45, "ties": 0, "clusters": 12, "cluster_wins": 5 }, "f22b8b34_16384": { "delta": 0.004239, "median": 0.002138, "ci": [ -0.004057, 0.01797 ], "n": 119, "wins": 50, "ties": 0, "clusters": 12, "cluster_wins": 7 } } }, { "key": "lat_mae_bl", "label": "Lateral error at the model's own best lead", "unit": "m/s²", "better": "lower", "value": 0.166, "median": 0.1613, "ci": [ 0.1398, 0.1719 ], "n": 119, "clusters": 12, "paired": { "lebowski_877m": { "delta": 0.001518, "median": 0.0004359, "ci": [ -0.007744, 0.01899 ], "n": 119, "wins": 59, "ties": 0, "clusters": 12, "cluster_wins": 5 }, "f22b8b34_16384": { "delta": 0.003802, "median": 0.003289, "ci": [ -0.005366, 0.01694 ], "n": 119, "wins": 51, "ties": 0, "clusters": 12, "cluster_wins": 8 } } }, { "key": "lat_corr", "label": "Lateral correlation with the car's later motion", "unit": "", "better": "higher", "value": 0.7984, "median": 0.891, "ci": [ 0.7481, 0.8256 ], "n": 119, "clusters": 12, "paired": { "lebowski_877m": { "delta": 0.03156, "median": 0.01091, "ci": [ -0.005378, 0.04149 ], "n": 119, "wins": 75, "ties": 0, "clusters": 12, "cluster_wins": 6 }, "f22b8b34_16384": { "delta": -0.006197, "median": -0.001762, "ci": [ -0.019, -0.0009121 ], "n": 119, "wins": 49, "ties": 0, "clusters": 12, "cluster_wins": 5 } } }, { "key": "lat_gain", "label": "Lateral gain (1 = same size as the car's motion)", "unit": "", "better": "near1", "value": 0.8212, "median": 0.8376, "ci": [ 0.798, 0.8888 ], "n": 119, "clusters": 12, "paired": { "lebowski_877m": { "delta": -0.003591, "median": 0.006133, "ci": [ -0.02666, 0.04971 ], "n": 119, "wins": 55, "ties": 0, "clusters": 12, "cluster_wins": 6 }, "f22b8b34_16384": { "delta": -0.01159, "median": -0.0172, "ci": [ -0.03309, 0.01288 ], "n": 119, "wins": 77, "ties": 0, "clusters": 12, "cluster_wins": 7 } } }, { "key": "lat_rate_rms", "label": "Lateral action rate, RMS (smoothness)", "unit": "m/s³", "better": "lower", "value": 1.184, "median": 0.8977, "ci": [ 0.6738, 1.425 ], "n": 119, "clusters": 12, "paired": { "lebowski_877m": { "delta": -1.106, "median": -1.029, "ci": [ -1.226, -0.4838 ], "n": 119, "wins": 118, "ties": 0, "clusters": 12, "cluster_wins": 12 }, "f22b8b34_16384": { "delta": 0.003012, "median": -0.006421, "ci": [ -0.0009295, 0.009966 ], "n": 119, "wins": 66, "ties": 0, "clusters": 12, "cluster_wins": 4 } } }, { "key": "lat_best_lag", "label": "Lead of the lateral request over the car's motion", "unit": "s", "better": "lower", "value": 0.6668, "median": 0.65, "ci": [ 0.6204, 0.7242 ], "n": 119, "clusters": 12, "paired": { "lebowski_877m": { "delta": 0.04454, "median": 0.05, "ci": [ -0.02685, 0.1539 ], "n": 119, "wins": 31, "ties": 13, "clusters": 12, "cluster_wins": 2 }, "f22b8b34_16384": { "delta": -0.005882, "median": 0.0, "ci": [ -0.02259, 0.03982 ], "n": 119, "wins": 36, "ties": 54, "clusters": 12, "cluster_wins": 4 } } }, { "key": "lon_mae", "label": "Longitudinal error", "unit": "m/s²", "better": "lower", "value": 0.3678, "median": 0.3657, "ci": [ 0.2887, 0.4142 ], "n": 119, "clusters": 12, "paired": { "lebowski_877m": { "delta": -0.0008678, "median": 0.002121, "ci": [ -0.03511, 0.02486 ], "n": 119, "wins": 59, "ties": 0, "clusters": 12, "cluster_wins": 4 }, "f22b8b34_16384": { "delta": 0.01086, "median": 0.01183, "ci": [ -0.02601, 0.02176 ], "n": 119, "wins": 37, "ties": 0, "clusters": 12, "cluster_wins": 6 } } }, { "key": "lon_corr", "label": "Longitudinal correlation", "unit": "", "better": "higher", "value": 0.3345, "median": 0.297, "ci": [ 0.283, 0.5739 ], "n": 119, "clusters": 12, "paired": { "lebowski_877m": { "delta": 0.007679, "median": 0.01035, "ci": [ -0.05029, 0.04783 ], "n": 119, "wins": 65, "ties": 0, "clusters": 12, "cluster_wins": 6 }, "f22b8b34_16384": { "delta": -0.005468, "median": -0.006947, "ci": [ -0.02719, 0.00474 ], "n": 119, "wins": 55, "ties": 0, "clusters": 12, "cluster_wins": 5 } } }, { "key": "lon_brake_mae", "label": "Longitudinal error while braking", "unit": "m/s²", "better": "lower", "value": 0.4627, "median": 0.4369, "ci": [ 0.4184, 0.6624 ], "n": 81, "clusters": 12, "paired": { "lebowski_877m": { "delta": -0.03614, "median": -0.06301, "ci": [ -0.06342, 0.1036 ], "n": 81, "wins": 55, "ties": 0, "clusters": 12, "cluster_wins": 4 }, "f22b8b34_16384": { "delta": -0.01437, "median": -0.02095, "ci": [ -0.03747, 0.08739 ], "n": 81, "wins": 50, "ties": 0, "clusters": 12, "cluster_wins": 3 } } }, { "key": "lon_rate_rms", "label": "Longitudinal action rate, RMS (smoothness)", "unit": "m/s³", "better": "lower", "value": 4.507, "median": 4.223, "ci": [ 2.049, 5.934 ], "n": 119, "clusters": 12, "paired": { "lebowski_877m": { "delta": -1.103, "median": -1.188, "ci": [ -1.832, -0.6452 ], "n": 119, "wins": 90, "ties": 0, "clusters": 12, "cluster_wins": 10 }, "f22b8b34_16384": { "delta": 0.2603, "median": 0.1455, "ci": [ 0.05685, 0.3815 ], "n": 119, "wins": 32, "ties": 0, "clusters": 12, "cluster_wins": 1 } } }, { "key": "v3_mae", "label": "Plan speed error at 3 s", "unit": "m/s", "better": "lower", "value": 1.889, "median": 1.838, "ci": [ 1.152, 2.3 ], "n": 119, "clusters": 12, "paired": { "lebowski_877m": { "delta": -0.1087, "median": -0.125, "ci": [ -0.3518, 0.3059 ], "n": 119, "wins": 78, "ties": 0, "clusters": 12, "cluster_wins": 5 }, "f22b8b34_16384": { "delta": 0.0, "median": 0.0, "ci": [ 0.0, 0.0 ], "n": 119, "wins": 0, "ties": 119, "clusters": 12, "cluster_wins": 0 } } } ], "per_unit_file": null, "by_group": { "stock-aimedup": { "n": 4, "clusters": 2, "metrics": [ { "key": "lat_mae", "value": 0.1158, "median": 0.1163, "ci": null, "n": 4, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.02534, "median": -0.0279, "ci": null, "n": 4, "wins": 3, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "f22b8b34_16384": { "delta": -0.0006366, "median": -0.0009851, "ci": null, "n": 4, "wins": 3, "ties": 0, "clusters": 2, "cluster_wins": 1 } } }, { "key": "lat_mae_bl", "value": 0.1086, "median": 0.1099, "ci": null, "n": 4, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.03098, "median": -0.03358, "ci": null, "n": 4, "wins": 4, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "f22b8b34_16384": { "delta": 0.002178, "median": 0.003541, "ci": null, "n": 4, "wins": 1, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "lat_corr", "value": 0.8376, "median": 0.8972, "ci": null, "n": 4, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.04241, "median": 0.04491, "ci": null, "n": 4, "wins": 3, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "f22b8b34_16384": { "delta": 0.01746, "median": 0.0067, "ci": null, "n": 4, "wins": 3, "ties": 0, "clusters": 2, "cluster_wins": 2 } } }, { "key": "lat_gain", "value": 0.8276, "median": 0.9138, "ci": null, "n": 4, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.1637, "median": -0.1576, "ci": null, "n": 4, "wins": 3, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "f22b8b34_16384": { "delta": -0.04653, "median": -0.05461, "ci": null, "n": 4, "wins": 3, "ties": 0, "clusters": 2, "cluster_wins": 2 } } }, { "key": "lat_rate_rms", "value": 0.8599, "median": 0.8016, "ci": null, "n": 4, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.293, "median": -0.3076, "ci": null, "n": 4, "wins": 4, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "f22b8b34_16384": { "delta": 0.01392, "median": 0.002853, "ci": null, "n": 4, "wins": 2, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "lat_best_lag", "value": 0.5125, "median": 0.55, "ci": null, "n": 4, "clusters": 2, "paired": { "lebowski_877m": { "delta": 2.776e-17, "median": 0.075, "ci": null, "n": 4, "wins": 1, "ties": 0, "clusters": 2, "cluster_wins": 1 }, "f22b8b34_16384": { "delta": -0.0875, "median": -0.075, "ci": null, "n": 4, "wins": 2, "ties": 2, "clusters": 2, "cluster_wins": 2 } } }, { "key": "lon_mae", "value": 0.5233, "median": 0.471, "ci": null, "n": 4, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.2544, "median": -0.247, "ci": null, "n": 4, "wins": 4, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "f22b8b34_16384": { "delta": -0.1806, "median": -0.1829, "ci": null, "n": 4, "wins": 4, "ties": 0, "clusters": 2, "cluster_wins": 2 } } }, { "key": "lon_corr", "value": 0.5614, "median": 0.6257, "ci": null, "n": 4, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.02605, "median": 0.05419, "ci": null, "n": 4, "wins": 3, "ties": 0, "clusters": 2, "cluster_wins": 1 }, "f22b8b34_16384": { "delta": -0.02724, "median": -0.04435, "ci": null, "n": 4, "wins": 1, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "lon_brake_mae", "value": 0.419, "median": 0.419, "ci": null, "n": 2, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.2311, "median": 0.2311, "ci": null, "n": 2, "wins": 0, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "f22b8b34_16384": { "delta": 0.1592, "median": 0.1592, "ci": null, "n": 2, "wins": 0, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "lon_rate_rms", "value": 2.097, "median": 1.826, "ci": null, "n": 4, "clusters": 2, "paired": { "lebowski_877m": { "delta": -1.197, "median": -1.205, "ci": null, "n": 4, "wins": 4, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "f22b8b34_16384": { "delta": 0.2888, "median": 0.3409, "ci": null, "n": 4, "wins": 1, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "v3_mae", "value": 1.443, "median": 1.507, "ci": null, "n": 4, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.9456, "median": -0.9687, "ci": null, "n": 4, "wins": 4, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "f22b8b34_16384": { "delta": 0.0, "median": 0.0, "ci": null, "n": 4, "wins": 0, "ties": 4, "clusters": 2, "cluster_wins": 0 } } } ] }, "stock-dual": { "n": 13, "clusters": 4, "metrics": [ { "key": "lat_mae", "value": 0.1563, "median": 0.1467, "ci": null, "n": 13, "clusters": 4, "paired": { "lebowski_877m": { "delta": 0.01111, "median": 0.006381, "ci": null, "n": 13, "wins": 5, "ties": 0, "clusters": 4, "cluster_wins": 1 }, "f22b8b34_16384": { "delta": -0.006278, "median": -0.00168, "ci": null, "n": 13, "wins": 8, "ties": 0, "clusters": 4, "cluster_wins": 2 } } }, { "key": "lat_mae_bl", "value": 0.1405, "median": 0.1394, "ci": null, "n": 13, "clusters": 4, "paired": { "lebowski_877m": { "delta": 0.001273, "median": 0.00397, "ci": null, "n": 13, "wins": 6, "ties": 0, "clusters": 4, "cluster_wins": 1 }, "f22b8b34_16384": { "delta": -0.009787, "median": -0.007912, "ci": null, "n": 13, "wins": 9, "ties": 0, "clusters": 4, "cluster_wins": 4 } } }, { "key": "lat_corr", "value": 0.8084, "median": 0.8362, "ci": null, "n": 13, "clusters": 4, "paired": { "lebowski_877m": { "delta": -0.004925, "median": -0.01919, "ci": null, "n": 13, "wins": 4, "ties": 0, "clusters": 4, "cluster_wins": 1 }, "f22b8b34_16384": { "delta": -0.01247, "median": -0.00566, "ci": null, "n": 13, "wins": 5, "ties": 0, "clusters": 4, "cluster_wins": 2 } } }, { "key": "lat_gain", "value": 0.892, "median": 0.8482, "ci": null, "n": 13, "clusters": 4, "paired": { "lebowski_877m": { "delta": 0.03873, "median": 0.04781, "ci": null, "n": 13, "wins": 5, "ties": 0, "clusters": 4, "cluster_wins": 1 }, "f22b8b34_16384": { "delta": -0.004277, "median": 0.0002683, "ci": null, "n": 13, "wins": 5, "ties": 0, "clusters": 4, "cluster_wins": 2 } } }, { "key": "lat_rate_rms", "value": 0.6795, "median": 0.6333, "ci": null, "n": 13, "clusters": 4, "paired": { "lebowski_877m": { "delta": -0.4964, "median": -0.4889, "ci": null, "n": 13, "wins": 13, "ties": 0, "clusters": 4, "cluster_wins": 4 }, "f22b8b34_16384": { "delta": 0.008585, "median": -7.212e-06, "ci": null, "n": 13, "wins": 7, "ties": 0, "clusters": 4, "cluster_wins": 1 } } }, { "key": "lat_best_lag", "value": 0.7154, "median": 0.65, "ci": null, "n": 13, "clusters": 4, "paired": { "lebowski_877m": { "delta": 0.2038, "median": 0.15, "ci": null, "n": 13, "wins": 0, "ties": 0, "clusters": 4, "cluster_wins": 0 }, "f22b8b34_16384": { "delta": 0.04231, "median": 0.0, "ci": null, "n": 13, "wins": 1, "ties": 6, "clusters": 4, "cluster_wins": 0 } } }, { "key": "lon_mae", "value": 0.301, "median": 0.2701, "ci": null, "n": 13, "clusters": 4, "paired": { "lebowski_877m": { "delta": 0.07279, "median": 0.09096, "ci": null, "n": 13, "wins": 2, "ties": 0, "clusters": 4, "cluster_wins": 0 }, "f22b8b34_16384": { "delta": 0.02713, "median": 0.02047, "ci": null, "n": 13, "wins": 4, "ties": 0, "clusters": 4, "cluster_wins": 2 } } }, { "key": "lon_corr", "value": 0.5448, "median": 0.6973, "ci": null, "n": 13, "clusters": 4, "paired": { "lebowski_877m": { "delta": -0.08938, "median": -0.05288, "ci": null, "n": 13, "wins": 6, "ties": 0, "clusters": 4, "cluster_wins": 1 }, "f22b8b34_16384": { "delta": -0.03849, "median": -0.02227, "ci": null, "n": 13, "wins": 6, "ties": 0, "clusters": 4, "cluster_wins": 1 } } }, { "key": "lon_brake_mae", "value": 0.7236, "median": 0.8106, "ci": null, "n": 6, "clusters": 4, "paired": { "lebowski_877m": { "delta": 0.07636, "median": 0.04105, "ci": null, "n": 6, "wins": 3, "ties": 0, "clusters": 4, "cluster_wins": 1 }, "f22b8b34_16384": { "delta": 0.1006, "median": 0.08507, "ci": null, "n": 6, "wins": 0, "ties": 0, "clusters": 4, "cluster_wins": 0 } } }, { "key": "lon_rate_rms", "value": 1.24, "median": 1.186, "ci": null, "n": 13, "clusters": 4, "paired": { "lebowski_877m": { "delta": -0.9741, "median": -0.9287, "ci": null, "n": 13, "wins": 13, "ties": 0, "clusters": 4, "cluster_wins": 4 }, "f22b8b34_16384": { "delta": 0.09651, "median": 0.05123, "ci": null, "n": 13, "wins": 3, "ties": 0, "clusters": 4, "cluster_wins": 0 } } }, { "key": "v3_mae", "value": 0.8066, "median": 0.8321, "ci": null, "n": 13, "clusters": 4, "paired": { "lebowski_877m": { "delta": -0.03457, "median": -0.06049, "ci": null, "n": 13, "wins": 8, "ties": 0, "clusters": 4, "cluster_wins": 2 }, "f22b8b34_16384": { "delta": 0.0, "median": 0.0, "ci": null, "n": 13, "wins": 0, "ties": 13, "clusters": 4, "cluster_wins": 0 } } } ] }, "stock-newcase": { "n": 2, "clusters": 1, "metrics": [ { "key": "lat_mae", "value": 0.1334, "median": 0.1334, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.0009076, "median": -0.0009076, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "f22b8b34_16384": { "delta": -0.003674, "median": -0.003674, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 1 } } }, { "key": "lat_mae_bl", "value": 0.1259, "median": 0.1259, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.009236, "median": -0.009236, "ci": null, "n": 2, "wins": 2, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "f22b8b34_16384": { "delta": -0.006258, "median": -0.006258, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 1 } } }, { "key": "lat_corr", "value": 0.8262, "median": 0.8262, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.0214, "median": -0.0214, "ci": null, "n": 2, "wins": 0, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "f22b8b34_16384": { "delta": 0.001265, "median": 0.001265, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 1 } } }, { "key": "lat_gain", "value": 0.7445, "median": 0.7445, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": 0.04543, "median": 0.04543, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "f22b8b34_16384": { "delta": 0.06693, "median": 0.06693, "ci": null, "n": 2, "wins": 0, "ties": 0, "clusters": 1, "cluster_wins": 0 } } }, { "key": "lat_rate_rms", "value": 0.7773, "median": 0.7773, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.4983, "median": -0.4983, "ci": null, "n": 2, "wins": 2, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "f22b8b34_16384": { "delta": -0.02176, "median": -0.02176, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 1 } } }, { "key": "lat_best_lag", "value": 0.625, "median": 0.625, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": 0.125, "median": 0.125, "ci": null, "n": 2, "wins": 0, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "f22b8b34_16384": { "delta": 0.025, "median": 0.025, "ci": null, "n": 2, "wins": 0, "ties": 1, "clusters": 1, "cluster_wins": 0 } } }, { "key": "lon_mae", "value": 0.3803, "median": 0.3803, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": 0.01778, "median": 0.01778, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "f22b8b34_16384": { "delta": 0.008568, "median": 0.008568, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 } } }, { "key": "lon_corr", "value": 0.5435, "median": 0.5435, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": 0.05691, "median": 0.05691, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "f22b8b34_16384": { "delta": 0.02003, "median": 0.02003, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 1 } } }, { "key": "lon_brake_mae", "value": 0.3771, "median": 0.3771, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.1262, "median": -0.1262, "ci": null, "n": 2, "wins": 2, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "f22b8b34_16384": { "delta": -0.01347, "median": -0.01347, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 1 } } }, { "key": "lon_rate_rms", "value": 2.505, "median": 2.505, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.8511, "median": -0.8511, "ci": null, "n": 2, "wins": 2, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "f22b8b34_16384": { "delta": 0.2448, "median": 0.2448, "ci": null, "n": 2, "wins": 0, "ties": 0, "clusters": 1, "cluster_wins": 0 } } }, { "key": "v3_mae", "value": 2.129, "median": 2.129, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": 0.03494, "median": 0.03494, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "f22b8b34_16384": { "delta": 0.0, "median": 0.0, "ci": null, "n": 2, "wins": 0, "ties": 2, "clusters": 1, "cluster_wins": 0 } } } ] }, "v13-8mm": { "n": 92, "clusters": 3, "metrics": [ { "key": "lat_mae", "value": 0.1889, "median": 0.1818, "ci": null, "n": 92, "clusters": 3, "paired": { "lebowski_877m": { "delta": 0.007353, "median": 0.008021, "ci": null, "n": 92, "wins": 36, "ties": 0, "clusters": 3, "cluster_wins": 1 }, "f22b8b34_16384": { "delta": 0.006534, "median": 0.004173, "ci": null, "n": 92, "wins": 34, "ties": 0, "clusters": 3, "cluster_wins": 1 } } }, { "key": "lat_mae_bl", "value": 0.1737, "median": 0.1655, "ci": null, "n": 92, "clusters": 3, "paired": { "lebowski_877m": { "delta": 0.001552, "median": -0.0007217, "ci": null, "n": 92, "wins": 47, "ties": 0, "clusters": 3, "cluster_wins": 1 }, "f22b8b34_16384": { "delta": 0.006845, "median": 0.004976, "ci": null, "n": 92, "wins": 34, "ties": 0, "clusters": 3, "cluster_wins": 1 } } }, { "key": "lat_corr", "value": 0.7913, "median": 0.8981, "ci": null, "n": 92, "clusters": 3, "paired": { "lebowski_877m": { "delta": 0.04282, "median": 0.01698, "ci": null, "n": 92, "wins": 67, "ties": 0, "clusters": 3, "cluster_wins": 3 }, "f22b8b34_16384": { "delta": -0.005908, "median": -0.001814, "ci": null, "n": 92, "wins": 38, "ties": 0, "clusters": 3, "cluster_wins": 0 } } }, { "key": "lat_gain", "value": 0.8063, "median": 0.8233, "ci": null, "n": 92, "clusters": 3, "paired": { "lebowski_877m": { "delta": -0.007656, "median": 0.003084, "ci": null, "n": 92, "wins": 43, "ties": 0, "clusters": 3, "cluster_wins": 2 }, "f22b8b34_16384": { "delta": -0.01335, "median": -0.01981, "ci": null, "n": 92, "wins": 64, "ties": 0, "clusters": 3, "cluster_wins": 2 } } }, { "key": "lat_rate_rms", "value": 1.325, "median": 1.046, "ci": null, "n": 92, "clusters": 3, "paired": { "lebowski_877m": { "delta": -1.293, "median": -1.246, "ci": null, "n": 92, "wins": 91, "ties": 0, "clusters": 3, "cluster_wins": 3 }, "f22b8b34_16384": { "delta": 0.002457, "median": -0.008143, "ci": null, "n": 92, "wins": 51, "ties": 0, "clusters": 3, "cluster_wins": 1 } } }, { "key": "lat_best_lag", "value": 0.6707, "median": 0.65, "ci": null, "n": 92, "clusters": 3, "paired": { "lebowski_877m": { "delta": 0.01033, "median": 0.05, "ci": null, "n": 92, "wins": 30, "ties": 13, "clusters": 3, "cluster_wins": 1 }, "f22b8b34_16384": { "delta": -0.01304, "median": 0.0, "ci": null, "n": 92, "wins": 31, "ties": 42, "clusters": 3, "cluster_wins": 2 } } }, { "key": "lon_mae", "value": 0.3717, "median": 0.3815, "ci": null, "n": 92, "clusters": 3, "paired": { "lebowski_877m": { "delta": -0.00433, "median": -0.004855, "ci": null, "n": 92, "wins": 52, "ties": 0, "clusters": 3, "cluster_wins": 2 }, "f22b8b34_16384": { "delta": 0.01856, "median": 0.01322, "ci": null, "n": 92, "wins": 24, "ties": 0, "clusters": 3, "cluster_wins": 0 } } }, { "key": "lon_corr", "value": 0.2601, "median": 0.2471, "ci": null, "n": 92, "clusters": 3, "paired": { "lebowski_877m": { "delta": 0.02641, "median": 0.02465, "ci": null, "n": 92, "wins": 54, "ties": 0, "clusters": 3, "cluster_wins": 3 }, "f22b8b34_16384": { "delta": -0.0008869, "median": -0.001565, "ci": null, "n": 92, "wins": 44, "ties": 0, "clusters": 3, "cluster_wins": 2 } } }, { "key": "lon_brake_mae", "value": 0.4208, "median": 0.43, "ci": null, "n": 65, "clusters": 3, "paired": { "lebowski_877m": { "delta": -0.06827, "median": -0.07715, "ci": null, "n": 65, "wins": 49, "ties": 0, "clusters": 3, "cluster_wins": 2 }, "f22b8b34_16384": { "delta": -0.03726, "median": -0.03428, "ci": null, "n": 65, "wins": 49, "ties": 0, "clusters": 3, "cluster_wins": 2 } } }, { "key": "lon_rate_rms", "value": 5.1, "median": 5.291, "ci": null, "n": 92, "clusters": 3, "paired": { "lebowski_877m": { "delta": -1.233, "median": -1.615, "ci": null, "n": 92, "wins": 68, "ties": 0, "clusters": 3, "cluster_wins": 3 }, "f22b8b34_16384": { "delta": 0.2813, "median": 0.1571, "ci": null, "n": 92, "wins": 27, "ties": 0, "clusters": 3, "cluster_wins": 1 } } }, { "key": "v3_mae", "value": 2.125, "median": 1.978, "ci": null, "n": 92, "clusters": 3, "paired": { "lebowski_877m": { "delta": -0.1094, "median": -0.2266, "ci": null, "n": 92, "wins": 63, "ties": 0, "clusters": 3, "cluster_wins": 1 }, "f22b8b34_16384": { "delta": 0.0, "median": 0.0, "ci": null, "n": 92, "wins": 0, "ties": 92, "clusters": 3, "cluster_wins": 0 } } } ] }, "v14-8mm": { "n": 8, "clusters": 2, "metrics": [ { "key": "lat_mae", "value": 0.1739, "median": 0.18, "ci": null, "n": 8, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.03255, "median": 0.02681, "ci": null, "n": 8, "wins": 0, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "f22b8b34_16384": { "delta": -0.0006472, "median": -0.002798, "ci": null, "n": 8, "wins": 4, "ties": 0, "clusters": 2, "cluster_wins": 2 } } }, { "key": "lat_mae_bl", "value": 0.1576, "median": 0.1627, "ci": null, "n": 8, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.02046, "median": 0.02046, "ci": null, "n": 8, "wins": 0, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "f22b8b34_16384": { "delta": -0.005785, "median": -0.006143, "ci": null, "n": 8, "wins": 6, "ties": 0, "clusters": 2, "cluster_wins": 2 } } }, { "key": "lat_corr", "value": 0.8371, "median": 0.9001, "ci": null, "n": 8, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.03092, "median": -0.026, "ci": null, "n": 8, "wins": 1, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "f22b8b34_16384": { "delta": -0.013, "median": -0.008192, "ci": null, "n": 8, "wins": 2, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "lat_gain", "value": 0.8925, "median": 0.9301, "ci": null, "n": 8, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.04219, "median": 0.01929, "ci": null, "n": 8, "wins": 3, "ties": 0, "clusters": 2, "cluster_wins": 1 }, "f22b8b34_16384": { "delta": -0.005412, "median": -0.001666, "ci": null, "n": 8, "wins": 5, "ties": 0, "clusters": 2, "cluster_wins": 1 } } }, { "key": "lat_rate_rms", "value": 0.6488, "median": 0.5504, "ci": null, "n": 8, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.5068, "median": -0.4652, "ci": null, "n": 8, "wins": 8, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "f22b8b34_16384": { "delta": 0.001087, "median": -0.02253, "ci": null, "n": 8, "wins": 5, "ties": 0, "clusters": 2, "cluster_wins": 1 } } }, { "key": "lat_best_lag", "value": 0.6312, "median": 0.6, "ci": null, "n": 8, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.1812, "median": 0.15, "ci": null, "n": 8, "wins": 0, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "f22b8b34_16384": { "delta": 0.03125, "median": 0.0, "ci": null, "n": 8, "wins": 2, "ties": 3, "clusters": 2, "cluster_wins": 0 } } }, { "key": "lon_mae", "value": 0.3507, "median": 0.2995, "ci": null, "n": 8, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.04134, "median": 0.02966, "ci": null, "n": 8, "wins": 0, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "f22b8b34_16384": { "delta": -0.007797, "median": 0.002762, "ci": null, "n": 8, "wins": 4, "ties": 0, "clusters": 2, "cluster_wins": 2 } } }, { "key": "lon_corr", "value": 0.6832, "median": 0.7012, "ci": null, "n": 8, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.07152, "median": -0.05756, "ci": null, "n": 8, "wins": 1, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "f22b8b34_16384": { "delta": 1.471e-05, "median": -0.007839, "ci": null, "n": 8, "wins": 3, "ties": 0, "clusters": 2, "cluster_wins": 1 } } }, { "key": "lon_brake_mae", "value": 0.6979, "median": 0.7685, "ci": null, "n": 6, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.1403, "median": 0.08257, "ci": null, "n": 6, "wins": 1, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "f22b8b34_16384": { "delta": 0.06044, "median": 0.05522, "ci": null, "n": 6, "wins": 0, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "lon_rate_rms", "value": 4.702, "median": 4.421, "ci": null, "n": 8, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.1685, "median": 0.1365, "ci": null, "n": 8, "wins": 3, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "f22b8b34_16384": { "delta": 0.2753, "median": 0.198, "ci": null, "n": 8, "wins": 1, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "v3_mae", "value": 1.095, "median": 1.066, "ci": null, "n": 8, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.1623, "median": 0.1656, "ci": null, "n": 8, "wins": 2, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "f22b8b34_16384": { "delta": 0.0, "median": 0.0, "ci": null, "n": 8, "wins": 0, "ties": 8, "clusters": 2, "cluster_wins": 0 } } } ] } } }, "truck_closed_loop": { "status": "ok", "unit_of_evidence": "window", "n": 98, "clusters": 13, "condition": "Measured. worldmodel-4B as in the public closed loop, but the model drives a simulation of the G55's measured steering and powertrain response instead of the ideal car. The simulation starts from the truck's state (speed, yaw rate, steering, gear) in its CAN recording of the 2 s before the window; one window, whose 2 s before are not all valid CAN, starts steady at its recorded speed. Lane-following windows from the G55's drives, day and night; distance is to the driver's recorded path; seeds 0 and 1, averaged per window. 98 windows (59 day, 39 night).", "baselines": [ "lebowski_877m", "f22b8b34_16384" ], "metrics": [ { "key": "xtrack_2s", "label": "Distance from the recorded path at 2 s", "unit": "m", "better": "lower", "value": 0.1415, "median": 0.1034, "ci": [ 0.109, 0.1712 ], "n": 98, "clusters": 13, "paired": { "lebowski_877m": { "delta": -0.02799, "median": -0.02192, "ci": [ -0.04393, -0.01282 ], "n": 98, "wins": 59, "ties": 0, "clusters": 13, "cluster_wins": 10 }, "f22b8b34_16384": { "delta": -0.007474, "median": -0.001994, "ci": [ -0.01563, 0.0007721 ], "n": 98, "wins": 51, "ties": 0, "clusters": 13, "cluster_wins": 8 } } }, { "key": "xtrack_4s", "label": "Distance from the recorded path at 4 s", "unit": "m", "better": "lower", "value": 0.5295, "median": 0.3627, "ci": [ 0.4036, 0.6667 ], "n": 98, "clusters": 13, "paired": { "lebowski_877m": { "delta": 0.009504, "median": 0.02689, "ci": [ -0.1566, 0.1558 ], "n": 98, "wins": 47, "ties": 0, "clusters": 13, "cluster_wins": 5 }, "f22b8b34_16384": { "delta": 0.01427, "median": 0.01963, "ci": [ -0.03065, 0.06209 ], "n": 98, "wins": 42, "ties": 0, "clusters": 13, "cluster_wins": 5 } } }, { "key": "xtrack_7s", "label": "Distance from the recorded path at 7.2 s", "unit": "m", "better": "lower", "value": 2.455, "median": 0.89, "ci": [ 1.663, 3.455 ], "n": 98, "clusters": 13, "paired": { "lebowski_877m": { "delta": 0.476, "median": 0.03042, "ci": [ -0.4486, 1.558 ], "n": 98, "wins": 45, "ties": 0, "clusters": 13, "cluster_wins": 4 }, "f22b8b34_16384": { "delta": 0.4269, "median": 0.06819, "ci": [ 0.1342, 0.791 ], "n": 98, "wins": 37, "ties": 0, "clusters": 13, "cluster_wins": 3 } } }, { "key": "fail_1m", "label": "Share of episodes ever more than 1 m off the path", "unit": "", "better": "lower", "value": 0.4592, "median": 0.5, "ci": [ 0.3926, 0.5357 ], "n": 98, "clusters": 13, "paired": { "lebowski_877m": { "delta": 0.03061, "median": 0.0, "ci": [ -0.01415, 0.09091 ], "n": 98, "wins": 9, "ties": 77, "clusters": 13, "cluster_wins": 3 }, "f22b8b34_16384": { "delta": 0.03571, "median": 0.0, "ci": [ -0.02273, 0.08763 ], "n": 98, "wins": 5, "ties": 80, "clusters": 13, "cluster_wins": 3 } } }, { "key": "lat_jerk", "label": "Requested lateral jerk, RMS", "unit": "m/s³", "better": "lower", "value": 0.3018, "median": 0.2146, "ci": [ 0.2534, 0.3538 ], "n": 98, "clusters": 13, "paired": { "lebowski_877m": { "delta": -0.2401, "median": -0.1693, "ci": [ -0.3091, -0.1707 ], "n": 98, "wins": 85, "ties": 0, "clusters": 13, "cluster_wins": 13 }, "f22b8b34_16384": { "delta": 0.03029, "median": 0.01157, "ci": [ 0.001298, 0.06826 ], "n": 98, "wins": 35, "ties": 0, "clusters": 13, "cluster_wins": 3 } } }, { "key": "long_jerk", "label": "Requested longitudinal jerk, RMS", "unit": "m/s³", "better": "lower", "value": 0.3841, "median": 0.3189, "ci": [ 0.3281, 0.4425 ], "n": 98, "clusters": 13, "paired": { "lebowski_877m": { "delta": -0.002172, "median": -0.04113, "ci": [ -0.0642, 0.06783 ], "n": 98, "wins": 59, "ties": 0, "clusters": 13, "cluster_wins": 7 }, "f22b8b34_16384": { "delta": 0.05041, "median": 0.03338, "ci": [ 0.0263, 0.07366 ], "n": 98, "wins": 24, "ties": 0, "clusters": 13, "cluster_wins": 1 } } }, { "key": "speed_err", "label": "Speed error against the recorded speed, RMS", "unit": "m/s", "better": "lower", "value": 1.255, "median": 1.104, "ci": [ 1.065, 1.447 ], "n": 98, "clusters": 13, "paired": { "lebowski_877m": { "delta": 0.1719, "median": 0.1367, "ci": [ 0.02686, 0.3057 ], "n": 98, "wins": 39, "ties": 0, "clusters": 13, "cluster_wins": 5 }, "f22b8b34_16384": { "delta": 0.05362, "median": 0.07613, "ci": [ -0.05097, 0.1495 ], "n": 98, "wins": 42, "ties": 0, "clusters": 13, "cluster_wins": 5 } } } ], "per_unit_file": null, "by_group": { "stock-aimedup": { "n": 14, "clusters": 2, "metrics": [ { "key": "xtrack_2s", "value": 0.1122, "median": 0.07435, "ci": null, "n": 14, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.03812, "median": -0.01335, "ci": null, "n": 14, "wins": 9, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "f22b8b34_16384": { "delta": -0.02373, "median": -0.003978, "ci": null, "n": 14, "wins": 9, "ties": 0, "clusters": 2, "cluster_wins": 2 } } }, { "key": "xtrack_4s", "value": 0.5411, "median": 0.5904, "ci": null, "n": 14, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.09001, "median": 0.134, "ci": null, "n": 14, "wins": 6, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "f22b8b34_16384": { "delta": -0.02406, "median": 0.007712, "ci": null, "n": 14, "wins": 7, "ties": 0, "clusters": 2, "cluster_wins": 1 } } }, { "key": "xtrack_7s", "value": 3.087, "median": 2.794, "ci": null, "n": 14, "clusters": 2, "paired": { "lebowski_877m": { "delta": 1.182, "median": 0.03786, "ci": null, "n": 14, "wins": 7, "ties": 0, "clusters": 2, "cluster_wins": 1 }, "f22b8b34_16384": { "delta": 1.288, "median": 0.3074, "ci": null, "n": 14, "wins": 5, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "fail_1m", "value": 0.6429, "median": 1.0, "ci": null, "n": 14, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.03571, "median": 0.0, "ci": null, "n": 14, "wins": 3, "ties": 8, "clusters": 2, "cluster_wins": 1 }, "f22b8b34_16384": { "delta": -0.1071, "median": 0.0, "ci": null, "n": 14, "wins": 3, "ties": 9, "clusters": 2, "cluster_wins": 2 } } }, { "key": "lat_jerk", "value": 0.4215, "median": 0.2539, "ci": null, "n": 14, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.1494, "median": -0.1808, "ci": null, "n": 14, "wins": 10, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "f22b8b34_16384": { "delta": 0.01464, "median": 0.006356, "ci": null, "n": 14, "wins": 6, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "long_jerk", "value": 0.3404, "median": 0.2998, "ci": null, "n": 14, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.1543, "median": -0.1321, "ci": null, "n": 14, "wins": 12, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "f22b8b34_16384": { "delta": 0.0418, "median": 0.03277, "ci": null, "n": 14, "wins": 4, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "speed_err", "value": 1.633, "median": 1.711, "ci": null, "n": 14, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.1307, "median": -0.3379, "ci": null, "n": 14, "wins": 9, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "f22b8b34_16384": { "delta": -0.2045, "median": -0.5406, "ci": null, "n": 14, "wins": 9, "ties": 0, "clusters": 2, "cluster_wins": 2 } } } ] }, "stock-dual": { "n": 43, "clusters": 5, "metrics": [ { "key": "xtrack_2s", "value": 0.1449, "median": 0.1128, "ci": null, "n": 43, "clusters": 5, "paired": { "lebowski_877m": { "delta": -0.02323, "median": -0.02941, "ci": null, "n": 43, "wins": 27, "ties": 0, "clusters": 5, "cluster_wins": 3 }, "f22b8b34_16384": { "delta": -0.007647, "median": -0.00694, "ci": null, "n": 43, "wins": 23, "ties": 0, "clusters": 5, "cluster_wins": 4 } } }, { "key": "xtrack_4s", "value": 0.4502, "median": 0.3594, "ci": null, "n": 43, "clusters": 5, "paired": { "lebowski_877m": { "delta": -0.002059, "median": -0.01868, "ci": null, "n": 43, "wins": 22, "ties": 0, "clusters": 5, "cluster_wins": 3 }, "f22b8b34_16384": { "delta": 0.02301, "median": 0.01411, "ci": null, "n": 43, "wins": 19, "ties": 0, "clusters": 5, "cluster_wins": 2 } } }, { "key": "xtrack_7s", "value": 2.253, "median": 0.6495, "ci": null, "n": 43, "clusters": 5, "paired": { "lebowski_877m": { "delta": 0.234, "median": 0.09582, "ci": null, "n": 43, "wins": 18, "ties": 0, "clusters": 5, "cluster_wins": 1 }, "f22b8b34_16384": { "delta": 0.409, "median": 0.06495, "ci": null, "n": 43, "wins": 15, "ties": 0, "clusters": 5, "cluster_wins": 1 } } }, { "key": "fail_1m", "value": 0.4186, "median": 0.0, "ci": null, "n": 43, "clusters": 5, "paired": { "lebowski_877m": { "delta": 0.0814, "median": 0.0, "ci": null, "n": 43, "wins": 3, "ties": 34, "clusters": 5, "cluster_wins": 1 }, "f22b8b34_16384": { "delta": 0.0814, "median": 0.0, "ci": null, "n": 43, "wins": 1, "ties": 35, "clusters": 5, "cluster_wins": 0 } } }, { "key": "lat_jerk", "value": 0.3313, "median": 0.2176, "ci": null, "n": 43, "clusters": 5, "paired": { "lebowski_877m": { "delta": -0.2214, "median": -0.1616, "ci": null, "n": 43, "wins": 37, "ties": 0, "clusters": 5, "cluster_wins": 5 }, "f22b8b34_16384": { "delta": 0.05475, "median": 0.01664, "ci": null, "n": 43, "wins": 15, "ties": 0, "clusters": 5, "cluster_wins": 1 } } }, { "key": "long_jerk", "value": 0.4254, "median": 0.3257, "ci": null, "n": 43, "clusters": 5, "paired": { "lebowski_877m": { "delta": 0.05116, "median": -0.04815, "ci": null, "n": 43, "wins": 26, "ties": 0, "clusters": 5, "cluster_wins": 3 }, "f22b8b34_16384": { "delta": 0.06964, "median": 0.05006, "ci": null, "n": 43, "wins": 6, "ties": 0, "clusters": 5, "cluster_wins": 0 } } }, { "key": "speed_err", "value": 1.311, "median": 1.224, "ci": null, "n": 43, "clusters": 5, "paired": { "lebowski_877m": { "delta": 0.2279, "median": 0.1633, "ci": null, "n": 43, "wins": 18, "ties": 0, "clusters": 5, "cluster_wins": 1 }, "f22b8b34_16384": { "delta": 0.14, "median": 0.1984, "ci": null, "n": 43, "wins": 15, "ties": 0, "clusters": 5, "cluster_wins": 1 } } } ] }, "stock-newcase": { "n": 4, "clusters": 1, "metrics": [ { "key": "xtrack_2s", "value": 0.03573, "median": 0.02148, "ci": null, "n": 4, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.08914, "median": -0.0849, "ci": null, "n": 4, "wins": 3, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "f22b8b34_16384": { "delta": 0.002326, "median": 0.0006764, "ci": null, "n": 4, "wins": 2, "ties": 0, "clusters": 1, "cluster_wins": 0 } } }, { "key": "xtrack_4s", "value": 0.2022, "median": 0.1447, "ci": null, "n": 4, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.03842, "median": 0.01183, "ci": null, "n": 4, "wins": 2, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "f22b8b34_16384": { "delta": 0.01642, "median": -0.001113, "ci": null, "n": 4, "wins": 2, "ties": 0, "clusters": 1, "cluster_wins": 0 } } }, { "key": "xtrack_7s", "value": 1.429, "median": 1.383, "ci": null, "n": 4, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.2065, "median": -0.2015, "ci": null, "n": 4, "wins": 3, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "f22b8b34_16384": { "delta": 0.4803, "median": 0.1148, "ci": null, "n": 4, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 } } }, { "key": "fail_1m", "value": 0.625, "median": 0.75, "ci": null, "n": 4, "clusters": 1, "paired": { "lebowski_877m": { "delta": 0.0, "median": 0.0, "ci": null, "n": 4, "wins": 0, "ties": 4, "clusters": 1, "cluster_wins": 0 }, "f22b8b34_16384": { "delta": 0.125, "median": 0.0, "ci": null, "n": 4, "wins": 0, "ties": 3, "clusters": 1, "cluster_wins": 0 } } }, { "key": "lat_jerk", "value": 0.2582, "median": 0.221, "ci": null, "n": 4, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.4815, "median": -0.3642, "ci": null, "n": 4, "wins": 4, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "f22b8b34_16384": { "delta": -0.07079, "median": -0.04581, "ci": null, "n": 4, "wins": 3, "ties": 0, "clusters": 1, "cluster_wins": 1 } } }, { "key": "long_jerk", "value": 0.4758, "median": 0.501, "ci": null, "n": 4, "clusters": 1, "paired": { "lebowski_877m": { "delta": 0.0866, "median": 0.008515, "ci": null, "n": 4, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "f22b8b34_16384": { "delta": 0.1365, "median": 0.07947, "ci": null, "n": 4, "wins": 0, "ties": 0, "clusters": 1, "cluster_wins": 0 } } }, { "key": "speed_err", "value": 0.9113, "median": 0.9046, "ci": null, "n": 4, "clusters": 1, "paired": { "lebowski_877m": { "delta": 0.2325, "median": 0.3915, "ci": null, "n": 4, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "f22b8b34_16384": { "delta": 0.1614, "median": 0.2336, "ci": null, "n": 4, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 } } } ] }, "v13-8mm": { "n": 18, "clusters": 3, "metrics": [ { "key": "xtrack_2s", "value": 0.1931, "median": 0.1017, "ci": null, "n": 18, "clusters": 3, "paired": { "lebowski_877m": { "delta": -0.02424, "median": -0.03027, "ci": null, "n": 18, "wins": 11, "ties": 0, "clusters": 3, "cluster_wins": 2 }, "f22b8b34_16384": { "delta": 0.007266, "median": 0.004843, "ci": null, "n": 18, "wins": 7, "ties": 0, "clusters": 3, "cluster_wins": 1 } } }, { "key": "xtrack_4s", "value": 0.5046, "median": 0.2968, "ci": null, "n": 18, "clusters": 3, "paired": { "lebowski_877m": { "delta": -0.2975, "median": -0.009622, "ci": null, "n": 18, "wins": 10, "ties": 0, "clusters": 3, "cluster_wins": 1 }, "f22b8b34_16384": { "delta": 0.01919, "median": 0.0155, "ci": null, "n": 18, "wins": 7, "ties": 0, "clusters": 3, "cluster_wins": 1 } } }, { "key": "xtrack_7s", "value": 1.368, "median": 0.89, "ci": null, "n": 18, "clusters": 3, "paired": { "lebowski_877m": { "delta": -1.256, "median": 0.02413, "ci": null, "n": 18, "wins": 9, "ties": 0, "clusters": 3, "cluster_wins": 1 }, "f22b8b34_16384": { "delta": 0.0676, "median": 0.05684, "ci": null, "n": 18, "wins": 6, "ties": 0, "clusters": 3, "cluster_wins": 1 } } }, { "key": "fail_1m", "value": 0.4167, "median": 0.25, "ci": null, "n": 18, "clusters": 3, "paired": { "lebowski_877m": { "delta": 0.02778, "median": 0.0, "ci": null, "n": 18, "wins": 1, "ties": 15, "clusters": 3, "cluster_wins": 0 }, "f22b8b34_16384": { "delta": 0.08333, "median": 0.0, "ci": null, "n": 18, "wins": 0, "ties": 16, "clusters": 3, "cluster_wins": 0 } } }, { "key": "lat_jerk", "value": 0.2352, "median": 0.2081, "ci": null, "n": 18, "clusters": 3, "paired": { "lebowski_877m": { "delta": -0.1442, "median": -0.0991, "ci": null, "n": 18, "wins": 16, "ties": 0, "clusters": 3, "cluster_wins": 3 }, "f22b8b34_16384": { "delta": 0.03227, "median": 0.03299, "ci": null, "n": 18, "wins": 3, "ties": 0, "clusters": 3, "cluster_wins": 0 } } }, { "key": "long_jerk", "value": 0.3073, "median": 0.2181, "ci": null, "n": 18, "clusters": 3, "paired": { "lebowski_877m": { "delta": 0.00275, "median": -0.01502, "ci": null, "n": 18, "wins": 10, "ties": 0, "clusters": 3, "cluster_wins": 1 }, "f22b8b34_16384": { "delta": -1.618e-05, "median": 0.01253, "ci": null, "n": 18, "wins": 8, "ties": 0, "clusters": 3, "cluster_wins": 1 } } }, { "key": "speed_err", "value": 0.8876, "median": 0.7122, "ci": null, "n": 18, "clusters": 3, "paired": { "lebowski_877m": { "delta": -0.05699, "median": 0.04131, "ci": null, "n": 18, "wins": 7, "ties": 0, "clusters": 3, "cluster_wins": 2 }, "f22b8b34_16384": { "delta": 0.06948, "median": 0.09135, "ci": null, "n": 18, "wins": 6, "ties": 0, "clusters": 3, "cluster_wins": 1 } } } ] }, "v14-8mm": { "n": 19, "clusters": 2, "metrics": [ { "key": "xtrack_2s", "value": 0.1289, "median": 0.1181, "ci": null, "n": 19, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.02198, "median": 0.01487, "ci": null, "n": 19, "wins": 9, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "f22b8b34_16384": { "delta": -0.01113, "median": -0.002793, "ci": null, "n": 19, "wins": 10, "ties": 0, "clusters": 2, "cluster_wins": 1 } } }, { "key": "xtrack_4s", "value": 0.793, "median": 0.3912, "ci": null, "n": 19, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.2772, "median": 0.07953, "ci": null, "n": 19, "wins": 7, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "f22b8b34_16384": { "delta": 0.01762, "median": 0.05297, "ci": null, "n": 19, "wins": 7, "ties": 0, "clusters": 2, "cluster_wins": 1 } } }, { "key": "xtrack_7s", "value": 3.694, "median": 0.8012, "ci": null, "n": 19, "clusters": 2, "paired": { "lebowski_877m": { "delta": 2.289, "median": 0.01317, "ci": null, "n": 19, "wins": 8, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "f22b8b34_16384": { "delta": 0.1622, "median": -0.01625, "ci": null, "n": 19, "wins": 10, "ties": 0, "clusters": 2, "cluster_wins": 1 } } }, { "key": "fail_1m", "value": 0.4211, "median": 0.0, "ci": null, "n": 19, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.02632, "median": 0.0, "ci": null, "n": 19, "wins": 2, "ties": 16, "clusters": 2, "cluster_wins": 1 }, "f22b8b34_16384": { "delta": -0.02632, "median": 0.0, "ci": null, "n": 19, "wins": 1, "ties": 17, "clusters": 2, "cluster_wins": 1 } } }, { "key": "lat_jerk", "value": 0.2192, "median": 0.1859, "ci": null, "n": 19, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.3894, "median": -0.2497, "ci": null, "n": 19, "wins": 18, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "f22b8b34_16384": { "delta": 0.005864, "median": 0.003769, "ci": null, "n": 19, "wins": 8, "ties": 0, "clusters": 2, "cluster_wins": 1 } } }, { "key": "long_jerk", "value": 0.3763, "median": 0.3958, "ci": null, "n": 19, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.03414, "median": -0.03127, "ci": null, "n": 19, "wins": 10, "ties": 0, "clusters": 2, "cluster_wins": 1 }, "f22b8b34_16384": { "delta": 0.04289, "median": 0.01474, "ci": null, "n": 19, "wins": 6, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "speed_err", "value": 1.267, "median": 0.897, "ci": null, "n": 19, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.4725, "median": 0.1598, "ci": null, "n": 19, "wins": 4, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "f22b8b34_16384": { "delta": 0.01071, "median": -0.03135, "ci": null, "n": 19, "wins": 11, "ties": 0, "clusters": 2, "cluster_wins": 1 } } } ] } }, "simulator": "g55-2026-10-31" }, "truck_recovery": { "status": "ok", "unit_of_evidence": "window", "n": 100, "clusters": 14, "condition": "Measured. The truck closed loop with a disturbance while the model drives; only the tests listed ran. 2° off heading: an extra 2°/s yaw for the first second. 98 windows (59 day, 39 night). 0.5 m off centre: an extra yaw-rate pulse, out and back over the first 2 s, that would move an uncorrected car 0.5 m sideways with no net heading change. 98 windows (59 day, 39 night). Braking lead: windows where the car ahead braked; the gap is derived from the lead distance seen on the recording minus how far the model ran ahead of the recorded car. 2 windows (2 day, 0 night).", "baselines": [ "lebowski_877m", "f22b8b34_16384" ], "metrics": [ { "key": "heading_xtrack_2s", "label": "2° off heading: distance from the path at 2 s", "unit": "m", "better": "lower", "value": 0.4813, "median": 0.4101, "ci": [ 0.3906, 0.5959 ], "n": 98, "clusters": 13, "paired": { "lebowski_877m": { "delta": 0.02834, "median": 0.01003, "ci": [ -0.005244, 0.06753 ], "n": 98, "wins": 44, "ties": 0, "clusters": 13, "cluster_wins": 5 }, "f22b8b34_16384": { "delta": -0.002067, "median": -0.003774, "ci": [ -0.008549, 0.005764 ], "n": 98, "wins": 56, "ties": 0, "clusters": 13, "cluster_wins": 7 } } }, { "key": "heading_xtrack_4s", "label": "2° off heading: distance from the path at 4 s", "unit": "m", "better": "lower", "value": 0.7261, "median": 0.4541, "ci": [ 0.5327, 0.9425 ], "n": 98, "clusters": 13, "paired": { "lebowski_877m": { "delta": 0.02355, "median": 0.04345, "ci": [ -0.1169, 0.1445 ], "n": 98, "wins": 40, "ties": 0, "clusters": 13, "cluster_wins": 4 }, "f22b8b34_16384": { "delta": 0.0229, "median": 0.02173, "ci": [ -0.02287, 0.05325 ], "n": 98, "wins": 40, "ties": 0, "clusters": 13, "cluster_wins": 2 } } }, { "key": "heading_lat_jerk", "label": "2° off heading: requested lateral jerk, RMS", "unit": "m/s³", "better": "lower", "value": 0.4004, "median": 0.3125, "ci": [ 0.3569, 0.4463 ], "n": 98, "clusters": 13, "paired": { "lebowski_877m": { "delta": -0.4865, "median": -0.4504, "ci": [ -0.5581, -0.4088 ], "n": 98, "wins": 94, "ties": 0, "clusters": 13, "cluster_wins": 13 }, "f22b8b34_16384": { "delta": -0.002142, "median": 0.007645, "ci": [ -0.03818, 0.03704 ], "n": 98, "wins": 40, "ties": 0, "clusters": 13, "cluster_wins": 5 } } }, { "key": "offcentre_xtrack_2s", "label": "0.5 m off centre: distance from the path at 2 s", "unit": "m", "better": "lower", "value": 0.2536, "median": 0.2131, "ci": [ 0.2277, 0.281 ], "n": 98, "clusters": 13, "paired": { "lebowski_877m": { "delta": -0.01344, "median": -0.004788, "ci": [ -0.03983, 0.01272 ], "n": 98, "wins": 51, "ties": 0, "clusters": 13, "cluster_wins": 6 }, "f22b8b34_16384": { "delta": -0.009018, "median": -0.008413, "ci": [ -0.01754, 0.0009304 ], "n": 98, "wins": 64, "ties": 0, "clusters": 13, "cluster_wins": 8 } } }, { "key": "offcentre_xtrack_4s", "label": "0.5 m off centre: distance from the path at 4 s", "unit": "m", "better": "lower", "value": 0.6567, "median": 0.3895, "ci": [ 0.5346, 0.7833 ], "n": 98, "clusters": 13, "paired": { "lebowski_877m": { "delta": -0.006178, "median": -0.02896, "ci": [ -0.08674, 0.08509 ], "n": 98, "wins": 54, "ties": 0, "clusters": 13, "cluster_wins": 6 }, "f22b8b34_16384": { "delta": -0.00193, "median": -0.01438, "ci": [ -0.03934, 0.044 ], "n": 98, "wins": 53, "ties": 0, "clusters": 13, "cluster_wins": 5 } } }, { "key": "offcentre_lat_jerk", "label": "0.5 m off centre: requested lateral jerk, RMS", "unit": "m/s³", "better": "lower", "value": 0.5441, "median": 0.4465, "ci": [ 0.4865, 0.5982 ], "n": 98, "clusters": 13, "paired": { "lebowski_877m": { "delta": -0.5663, "median": -0.5516, "ci": [ -0.6421, -0.4973 ], "n": 98, "wins": 96, "ties": 0, "clusters": 13, "cluster_wins": 13 }, "f22b8b34_16384": { "delta": 0.01034, "median": 0.01845, "ci": [ -0.0285, 0.04048 ], "n": 98, "wins": 32, "ties": 0, "clusters": 13, "cluster_wins": 5 } } }, { "key": "brake_ahead_4s", "label": "Braking lead: distance ahead of the recorded car at 4 s", "unit": "m", "better": "lower", "value": -0.02247, "median": -0.02247, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": 1.315, "median": 1.315, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "f22b8b34_16384": { "delta": 0.5301, "median": 0.5301, "ci": null, "n": 2, "wins": 0, "ties": 0, "clusters": 1, "cluster_wins": 0 } } }, { "key": "brake_gap_min", "label": "Braking lead: smallest gap to the lead (derived)", "unit": "m", "better": "higher", "value": 23.63, "median": 23.63, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": -3.833, "median": -3.833, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "f22b8b34_16384": { "delta": -2.396, "median": -2.396, "ci": null, "n": 2, "wins": 0, "ties": 0, "clusters": 1, "cluster_wins": 0 } } }, { "key": "brake_speed_err", "label": "Braking lead: speed error against the recorded speed, RMS", "unit": "m/s", "better": "lower", "value": 1.382, "median": 1.382, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": 0.1789, "median": 0.1789, "ci": null, "n": 2, "wins": 0, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "f22b8b34_16384": { "delta": 0.2495, "median": 0.2495, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 } } } ], "per_unit_file": null, "by_group": { "stock-aimedup": { "n": 14, "clusters": 2, "metrics": [ { "key": "heading_xtrack_2s", "value": 0.3932, "median": 0.3403, "ci": null, "n": 14, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.001453, "median": -0.007706, "ci": null, "n": 14, "wins": 8, "ties": 0, "clusters": 2, "cluster_wins": 1 }, "f22b8b34_16384": { "delta": 0.004648, "median": 0.004485, "ci": null, "n": 14, "wins": 6, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "heading_xtrack_4s", "value": 0.5957, "median": 0.5276, "ci": null, "n": 14, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.04269, "median": 0.0812, "ci": null, "n": 14, "wins": 6, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "f22b8b34_16384": { "delta": -0.09652, "median": -0.01645, "ci": null, "n": 14, "wins": 7, "ties": 0, "clusters": 2, "cluster_wins": 1 } } }, { "key": "heading_lat_jerk", "value": 0.4286, "median": 0.3103, "ci": null, "n": 14, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.3394, "median": -0.2922, "ci": null, "n": 14, "wins": 13, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "f22b8b34_16384": { "delta": -0.02953, "median": -0.002394, "ci": null, "n": 14, "wins": 7, "ties": 0, "clusters": 2, "cluster_wins": 1 } } }, { "key": "offcentre_xtrack_2s", "value": 0.2487, "median": 0.2387, "ci": null, "n": 14, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.0142, "median": -0.01101, "ci": null, "n": 14, "wins": 9, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "f22b8b34_16384": { "delta": -0.02321, "median": -0.01559, "ci": null, "n": 14, "wins": 11, "ties": 0, "clusters": 2, "cluster_wins": 2 } } }, { "key": "offcentre_xtrack_4s", "value": 0.653, "median": 0.6779, "ci": null, "n": 14, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.06317, "median": 0.04123, "ci": null, "n": 14, "wins": 7, "ties": 0, "clusters": 2, "cluster_wins": 1 }, "f22b8b34_16384": { "delta": 0.02874, "median": 0.04645, "ci": null, "n": 14, "wins": 6, "ties": 0, "clusters": 2, "cluster_wins": 1 } } }, { "key": "offcentre_lat_jerk", "value": 0.6106, "median": 0.543, "ci": null, "n": 14, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.4512, "median": -0.5088, "ci": null, "n": 14, "wins": 13, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "f22b8b34_16384": { "delta": -0.02702, "median": 0.03034, "ci": null, "n": 14, "wins": 4, "ties": 0, "clusters": 2, "cluster_wins": 1 } } } ] }, "stock-dual": { "n": 45, "clusters": 6, "metrics": [ { "key": "heading_xtrack_2s", "value": 0.4052, "median": 0.3516, "ci": null, "n": 43, "clusters": 5, "paired": { "lebowski_877m": { "delta": -0.01037, "median": -0.03202, "ci": null, "n": 43, "wins": 25, "ties": 0, "clusters": 5, "cluster_wins": 3 }, "f22b8b34_16384": { "delta": -0.004777, "median": -0.004404, "ci": null, "n": 43, "wins": 26, "ties": 0, "clusters": 5, "cluster_wins": 3 } } }, { "key": "heading_xtrack_4s", "value": 0.5946, "median": 0.3992, "ci": null, "n": 43, "clusters": 5, "paired": { "lebowski_877m": { "delta": -0.04156, "median": 0.0172, "ci": null, "n": 43, "wins": 19, "ties": 0, "clusters": 5, "cluster_wins": 3 }, "f22b8b34_16384": { "delta": 0.05338, "median": 0.02673, "ci": null, "n": 43, "wins": 19, "ties": 0, "clusters": 5, "cluster_wins": 0 } } }, { "key": "heading_lat_jerk", "value": 0.4253, "median": 0.3138, "ci": null, "n": 43, "clusters": 5, "paired": { "lebowski_877m": { "delta": -0.4711, "median": -0.4467, "ci": null, "n": 43, "wins": 40, "ties": 0, "clusters": 5, "cluster_wins": 5 }, "f22b8b34_16384": { "delta": -0.01262, "median": 0.009394, "ci": null, "n": 43, "wins": 16, "ties": 0, "clusters": 5, "cluster_wins": 3 } } }, { "key": "offcentre_xtrack_2s", "value": 0.2496, "median": 0.2144, "ci": null, "n": 43, "clusters": 5, "paired": { "lebowski_877m": { "delta": -0.04972, "median": -0.03722, "ci": null, "n": 43, "wins": 29, "ties": 0, "clusters": 5, "cluster_wins": 4 }, "f22b8b34_16384": { "delta": -0.01126, "median": -0.0119, "ci": null, "n": 43, "wins": 28, "ties": 0, "clusters": 5, "cluster_wins": 3 } } }, { "key": "offcentre_xtrack_4s", "value": 0.5984, "median": 0.3619, "ci": null, "n": 43, "clusters": 5, "paired": { "lebowski_877m": { "delta": -0.07198, "median": -0.08086, "ci": null, "n": 43, "wins": 28, "ties": 0, "clusters": 5, "cluster_wins": 3 }, "f22b8b34_16384": { "delta": -0.04604, "median": -0.03847, "ci": null, "n": 43, "wins": 26, "ties": 0, "clusters": 5, "cluster_wins": 3 } } }, { "key": "offcentre_lat_jerk", "value": 0.5986, "median": 0.4582, "ci": null, "n": 43, "clusters": 5, "paired": { "lebowski_877m": { "delta": -0.578, "median": -0.5599, "ci": null, "n": 43, "wins": 42, "ties": 0, "clusters": 5, "cluster_wins": 5 }, "f22b8b34_16384": { "delta": 0.01875, "median": 0.02315, "ci": null, "n": 43, "wins": 14, "ties": 0, "clusters": 5, "cluster_wins": 2 } } }, { "key": "brake_ahead_4s", "value": -0.02247, "median": -0.02247, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": 1.315, "median": 1.315, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "f22b8b34_16384": { "delta": 0.5301, "median": 0.5301, "ci": null, "n": 2, "wins": 0, "ties": 0, "clusters": 1, "cluster_wins": 0 } } }, { "key": "brake_gap_min", "value": 23.63, "median": 23.63, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": -3.833, "median": -3.833, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "f22b8b34_16384": { "delta": -2.396, "median": -2.396, "ci": null, "n": 2, "wins": 0, "ties": 0, "clusters": 1, "cluster_wins": 0 } } }, { "key": "brake_speed_err", "value": 1.382, "median": 1.382, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": 0.1789, "median": 0.1789, "ci": null, "n": 2, "wins": 0, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "f22b8b34_16384": { "delta": 0.2495, "median": 0.2495, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 } } } ] }, "stock-newcase": { "n": 4, "clusters": 1, "metrics": [ { "key": "heading_xtrack_2s", "value": 0.3811, "median": 0.4084, "ci": null, "n": 4, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.06947, "median": -0.08333, "ci": null, "n": 4, "wins": 3, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "f22b8b34_16384": { "delta": 0.0288, "median": 0.02766, "ci": null, "n": 4, "wins": 0, "ties": 0, "clusters": 1, "cluster_wins": 0 } } }, { "key": "heading_xtrack_4s", "value": 0.3044, "median": 0.2114, "ci": null, "n": 4, "clusters": 1, "paired": { "lebowski_877m": { "delta": 0.08474, "median": 0.1091, "ci": null, "n": 4, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "f22b8b34_16384": { "delta": 0.04176, "median": 0.04056, "ci": null, "n": 4, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 } } }, { "key": "heading_lat_jerk", "value": 0.3372, "median": 0.2181, "ci": null, "n": 4, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.635, "median": -0.5625, "ci": null, "n": 4, "wins": 4, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "f22b8b34_16384": { "delta": 0.02251, "median": 0.01689, "ci": null, "n": 4, "wins": 2, "ties": 0, "clusters": 1, "cluster_wins": 0 } } }, { "key": "offcentre_xtrack_2s", "value": 0.3431, "median": 0.2594, "ci": null, "n": 4, "clusters": 1, "paired": { "lebowski_877m": { "delta": 0.04325, "median": 0.07142, "ci": null, "n": 4, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "f22b8b34_16384": { "delta": 0.04644, "median": 0.04423, "ci": null, "n": 4, "wins": 0, "ties": 0, "clusters": 1, "cluster_wins": 0 } } }, { "key": "offcentre_xtrack_4s", "value": 0.7792, "median": 0.5153, "ci": null, "n": 4, "clusters": 1, "paired": { "lebowski_877m": { "delta": 0.05152, "median": -0.0402, "ci": null, "n": 4, "wins": 2, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "f22b8b34_16384": { "delta": 0.2556, "median": 0.04778, "ci": null, "n": 4, "wins": 2, "ties": 0, "clusters": 1, "cluster_wins": 0 } } }, { "key": "offcentre_lat_jerk", "value": 0.6168, "median": 0.3877, "ci": null, "n": 4, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.7806, "median": -0.6931, "ci": null, "n": 4, "wins": 4, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "f22b8b34_16384": { "delta": -0.07042, "median": -0.08167, "ci": null, "n": 4, "wins": 4, "ties": 0, "clusters": 1, "cluster_wins": 1 } } } ] }, "v13-8mm": { "n": 18, "clusters": 3, "metrics": [ { "key": "heading_xtrack_2s", "value": 0.8523, "median": 0.7926, "ci": null, "n": 18, "clusters": 3, "paired": { "lebowski_877m": { "delta": 0.1592, "median": 0.1744, "ci": null, "n": 18, "wins": 2, "ties": 0, "clusters": 3, "cluster_wins": 0 }, "f22b8b34_16384": { "delta": -0.0005418, "median": 0.005227, "ci": null, "n": 18, "wins": 8, "ties": 0, "clusters": 3, "cluster_wins": 2 } } }, { "key": "heading_xtrack_4s", "value": 1.15, "median": 0.7175, "ci": null, "n": 18, "clusters": 3, "paired": { "lebowski_877m": { "delta": -0.08168, "median": 0.04448, "ci": null, "n": 18, "wins": 7, "ties": 0, "clusters": 3, "cluster_wins": 1 }, "f22b8b34_16384": { "delta": 0.01553, "median": -0.001378, "ci": null, "n": 18, "wins": 9, "ties": 0, "clusters": 3, "cluster_wins": 1 } } }, { "key": "heading_lat_jerk", "value": 0.4065, "median": 0.4071, "ci": null, "n": 18, "clusters": 3, "paired": { "lebowski_877m": { "delta": -0.4665, "median": -0.4493, "ci": null, "n": 18, "wins": 18, "ties": 0, "clusters": 3, "cluster_wins": 3 }, "f22b8b34_16384": { "delta": 0.003856, "median": 0.009926, "ci": null, "n": 18, "wins": 7, "ties": 0, "clusters": 3, "cluster_wins": 1 } } }, { "key": "offcentre_xtrack_2s", "value": 0.2813, "median": 0.2024, "ci": null, "n": 18, "clusters": 3, "paired": { "lebowski_877m": { "delta": 0.02322, "median": 0.04503, "ci": null, "n": 18, "wins": 7, "ties": 0, "clusters": 3, "cluster_wins": 0 }, "f22b8b34_16384": { "delta": -0.002223, "median": -0.005967, "ci": null, "n": 18, "wins": 12, "ties": 0, "clusters": 3, "cluster_wins": 1 } } }, { "key": "offcentre_xtrack_4s", "value": 0.6081, "median": 0.3156, "ci": null, "n": 18, "clusters": 3, "paired": { "lebowski_877m": { "delta": -0.08493, "median": 0.01581, "ci": null, "n": 18, "wins": 8, "ties": 0, "clusters": 3, "cluster_wins": 2 }, "f22b8b34_16384": { "delta": -0.01104, "median": -0.01928, "ci": null, "n": 18, "wins": 10, "ties": 0, "clusters": 3, "cluster_wins": 1 } } }, { "key": "offcentre_lat_jerk", "value": 0.4699, "median": 0.4254, "ci": null, "n": 18, "clusters": 3, "paired": { "lebowski_877m": { "delta": -0.4119, "median": -0.4126, "ci": null, "n": 18, "wins": 18, "ties": 0, "clusters": 3, "cluster_wins": 3 }, "f22b8b34_16384": { "delta": 0.05005, "median": 0.0324, "ci": null, "n": 18, "wins": 1, "ties": 0, "clusters": 3, "cluster_wins": 0 } } } ] }, "v14-8mm": { "n": 19, "clusters": 2, "metrics": [ { "key": "heading_xtrack_2s", "value": 0.3879, "median": 0.3762, "ci": null, "n": 19, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.03236, "median": 0.04316, "ci": null, "n": 19, "wins": 6, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "f22b8b34_16384": { "delta": -0.008828, "median": -0.01077, "ci": null, "n": 19, "wins": 16, "ties": 0, "clusters": 2, "cluster_wins": 2 } } }, { "key": "heading_xtrack_4s", "value": 0.8071, "median": 0.3833, "ci": null, "n": 19, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.2436, "median": 0.09532, "ci": null, "n": 19, "wins": 7, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "f22b8b34_16384": { "delta": 0.04493, "median": 0.04869, "ci": null, "n": 19, "wins": 4, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "heading_lat_jerk", "value": 0.331, "median": 0.2956, "ci": null, "n": 19, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.6174, "median": -0.5143, "ci": null, "n": 19, "wins": 19, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "f22b8b34_16384": { "delta": 0.03087, "median": 0.003933, "ci": null, "n": 19, "wins": 8, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "offcentre_xtrack_2s", "value": 0.221, "median": 0.2029, "ci": null, "n": 19, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.02254, "median": 0.04417, "ci": null, "n": 19, "wins": 5, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "f22b8b34_16384": { "delta": -0.0116, "median": -0.007802, "ci": null, "n": 19, "wins": 13, "ties": 0, "clusters": 2, "cluster_wins": 2 } } }, { "key": "offcentre_xtrack_4s", "value": 0.8115, "median": 0.4389, "ci": null, "n": 19, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.1541, "median": 0.01352, "ci": null, "n": 19, "wins": 9, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "f22b8b34_16384": { "delta": 0.02972, "median": 0.02195, "ci": null, "n": 19, "wins": 9, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "offcentre_lat_jerk", "value": 0.4269, "median": 0.3993, "ci": null, "n": 19, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.7256, "median": -0.6709, "ci": null, "n": 19, "wins": 19, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "f22b8b34_16384": { "delta": -0.001792, "median": 0.0008395, "ci": null, "n": 19, "wins": 9, "ties": 0, "clusters": 2, "cluster_wins": 1 } } } ] } }, "simulator": "g55-2026-10-31" } }, "stills": [], "limits": [ "Open-loop scores compare a model's action with what the recording driver or car did; a model that drives differently but well is scored as wrong. comma's RL residual makes the lateral action smoother, and earlier than its base's by 0.10 to 0.26 s (measured on the G55 and on 16 of comma's cars); both score as error here. Measured on the 2026-10-27 deep dive: open-loop accuracy did not predict closed-loop driving.", "Closed-loop numbers come from a generative world model, not a car. In a separate test on the G55's windows, worldmodel-4B followed an added yaw command through 2 s but showed 0 to 20% of an added lateral offset 3 to 4 s later; past about 4 s the recorded anchor frames also pull every model back toward the recorded path. The front page reports 2 s.", "Intervals are 95% percentile bootstraps that resample whole drives (the drive a public segment was cut from, the recorded drive a truck window comes from), so segments of one drive count once. Below 10 drives there is no interval: the braking-lead test comes from one drive and reports means and win counts only.", "Paired deltas are model minus baseline on the units both have (for lateral gain, the change in its distance from 1); wins count units where the model is better in the metric's direction.", "The lead of the lateral request is the shift, searched from 0 to 1.5 s, at which the request best correlates with the car's later lateral motion; a larger value means the request comes earlier. A value at either edge is censored and left out. Plan speed error at 3 s reads the plan head, which does not separate models built on one frozen base: it is identical on every segment for the four f22e06e5 models, and for Cinque v2 and v3.", "Truck open loop comes from few drives, and the camera setup changed between them (lens, housing, aim); the pooled number mixes setups, so read the per-setup numbers too. The 8 mm setups' wide camera recorded about 7-10 fps, so its frame can be up to ~60 ms from the narrow one. On the V14 setup up to 7 of a window's 700 steps are ~100 ms (one camera frame dropped). The camera's aim is the truck's live calibration learned on each drive; drives whose recorded calibration was carried over from another drive are left out.", "Truck closed loop: the steering and powertrain simulation is fitted to the G55, not validated on it in closed loop; the reference is the driver's own line, not the lane centre. The pooled number mixes camera setups, so read the per-setup numbers too. On the V14 setup the wide frame can be up to 90 ms from the simulated one (its camera recorded 7-9 fps).", "Recovery tests are disturbances rendered by the world model (yaw it renders; a one-frame jump it does not, so none is used). The braking-lead test has 2 windows from one drive, whose recorded calibration was carried over from an earlier drive." ] }