{ "schema": 1, "model": { "id": "1365126a_12864", "name": "MD v3 / Flip Flop", "sha256": "35976b11be3904eb78e32f7dc55a6abf8e9433ac5aa391a1998f1e9ce001b199", "source": { "kind": "openpilot-master", "ref": "9fba666fd4", "url": "https://github.com/commaai/openpilot/pull/39156", "date": "2026-10-24" }, "first_public": "2026-10-14", "on_master": "2026-10-14", "params_m": 382.9, "gmac": 46.7, "family": "residual", "notes": "On openpilot master as MD v3 (2026-10-14, PR #39087) and again as Flip Flop (2026-10-24, PR #39156), same weights. Scored from the 2026-10-24 Hugging Face re-upload, weights identical to the 2026-10-16 upload. Vision base f22e06e5 with an RL residual head." }, "scored": { "date": "2026-10-31T23:59:17Z", "suite": "geval-1", "code": "d8eb40eeba87" }, "baselines": [ "lebowski_877m", "8d736683_12864" ], "sections": { "public_open_loop": { "status": "ok", "unit_of_evidence": "segment", "n": 39, "clusters": 23, "condition": "Measured. comma's public two-camera segments (openpilot CI data), native intrinsics and the device's own calibration, both cameras; every model on the same decoded frames. Truth is what the recording car did next: carState speed and acceleration, yaw rate from the device's cameraOdometry. Segments where the car never moves have no data. 39 of the 45 listed segments have a score, from 23 drives and 20 cars.", "baselines": [ "lebowski_877m", "8d736683_12864" ], "metrics": [ { "key": "lat_mae", "label": "Lateral error", "unit": "m/s²", "better": "lower", "value": 0.1288, "median": 0.1097, "ci": [ 0.1012, 0.1586 ], "n": 39, "clusters": 23, "paired": { "lebowski_877m": { "delta": 0.02008, "median": 0.01146, "ci": [ 0.009954, 0.03181 ], "n": 39, "wins": 10, "ties": 0, "clusters": 23, "cluster_wins": 5 }, "8d736683_12864": { "delta": -0.01469, "median": -0.01419, "ci": [ -0.02012, -0.009449 ], "n": 39, "wins": 36, "ties": 0, "clusters": 23, "cluster_wins": 22 } } }, { "key": "lat_mae_bl", "label": "Lateral error at the model's own best lead", "unit": "m/s²", "better": "lower", "value": 0.1194, "median": 0.09844, "ci": [ 0.09337, 0.1483 ], "n": 39, "clusters": 23, "paired": { "lebowski_877m": { "delta": 0.01798, "median": 0.009241, "ci": [ 0.006761, 0.02947 ], "n": 39, "wins": 10, "ties": 0, "clusters": 23, "cluster_wins": 7 }, "8d736683_12864": { "delta": -0.01276, "median": -0.008698, "ci": [ -0.01842, -0.007039 ], "n": 39, "wins": 36, "ties": 0, "clusters": 23, "cluster_wins": 21 } } }, { "key": "lat_corr", "label": "Lateral correlation with the car's later motion", "unit": "", "better": "higher", "value": 0.8467, "median": 0.9394, "ci": [ 0.7586, 0.9207 ], "n": 39, "clusters": 23, "paired": { "lebowski_877m": { "delta": -0.01093, "median": -0.004578, "ci": [ -0.05281, 0.02237 ], "n": 39, "wins": 12, "ties": 0, "clusters": 23, "cluster_wins": 7 }, "8d736683_12864": { "delta": 0.02603, "median": 0.01448, "ci": [ 0.01299, 0.04159 ], "n": 39, "wins": 35, "ties": 0, "clusters": 23, "cluster_wins": 22 } } }, { "key": "lat_gain", "label": "Lateral gain (1 = same size as the car's motion)", "unit": "", "better": "near1", "value": 0.7565, "median": 0.7947, "ci": [ 0.6661, 0.851 ], "n": 39, "clusters": 23, "paired": { "lebowski_877m": { "delta": 0.06431, "median": 0.0333, "ci": [ 0.01161, 0.1173 ], "n": 39, "wins": 13, "ties": 0, "clusters": 23, "cluster_wins": 7 }, "8d736683_12864": { "delta": -0.02435, "median": -0.01904, "ci": [ -0.04598, -0.002486 ], "n": 39, "wins": 32, "ties": 0, "clusters": 23, "cluster_wins": 19 } } }, { "key": "lat_rate_rms", "label": "Lateral action rate, RMS (smoothness)", "unit": "m/s³", "better": "lower", "value": 0.4514, "median": 0.3967, "ci": [ 0.3533, 0.5663 ], "n": 39, "clusters": 23, "paired": { "lebowski_877m": { "delta": -0.2125, "median": -0.2186, "ci": [ -0.2556, -0.1659 ], "n": 39, "wins": 38, "ties": 0, "clusters": 23, "cluster_wins": 22 }, "8d736683_12864": { "delta": -0.02257, "median": -0.01781, "ci": [ -0.04, -0.01083 ], "n": 39, "wins": 32, "ties": 0, "clusters": 23, "cluster_wins": 20 } } }, { "key": "lat_best_lag", "label": "Lead of the lateral request over the car's motion", "unit": "s", "better": "lower", "value": 0.47, "median": 0.45, "ci": [ 0.3767, 0.5735 ], "n": 35, "clusters": 22, "paired": { "lebowski_877m": { "delta": 0.08824, "median": 0.05, "ci": [ 0.03485, 0.1518 ], "n": 34, "wins": 7, "ties": 6, "clusters": 22, "cluster_wins": 3 }, "8d736683_12864": { "delta": -0.1271, "median": -0.05, "ci": [ -0.1914, -0.07727 ], "n": 35, "wins": 30, "ties": 4, "clusters": 22, "cluster_wins": 20 } } }, { "key": "lon_mae", "label": "Longitudinal error", "unit": "m/s²", "better": "lower", "value": 0.3007, "median": 0.2675, "ci": [ 0.2606, 0.3466 ], "n": 39, "clusters": 23, "paired": { "lebowski_877m": { "delta": -0.007425, "median": 0.002673, "ci": [ -0.02295, 0.009765 ], "n": 39, "wins": 19, "ties": 0, "clusters": 23, "cluster_wins": 10 }, "8d736683_12864": { "delta": -0.0124, "median": -0.01082, "ci": [ -0.01898, -0.005299 ], "n": 39, "wins": 30, "ties": 0, "clusters": 23, "cluster_wins": 18 } } }, { "key": "lon_corr", "label": "Longitudinal correlation", "unit": "", "better": "higher", "value": 0.7387, "median": 0.8496, "ci": [ 0.6402, 0.8236 ], "n": 39, "clusters": 23, "paired": { "lebowski_877m": { "delta": 0.02094, "median": 0.007609, "ci": [ -0.00439, 0.04466 ], "n": 39, "wins": 24, "ties": 0, "clusters": 23, "cluster_wins": 13 }, "8d736683_12864": { "delta": 0.01473, "median": 0.009904, "ci": [ 0.0003902, 0.02666 ], "n": 39, "wins": 31, "ties": 0, "clusters": 23, "cluster_wins": 18 } } }, { "key": "lon_brake_mae", "label": "Longitudinal error while braking", "unit": "m/s²", "better": "lower", "value": 0.4005, "median": 0.3657, "ci": [ 0.3402, 0.4711 ], "n": 30, "clusters": 21, "paired": { "lebowski_877m": { "delta": -0.03376, "median": -0.02843, "ci": [ -0.05555, -0.01068 ], "n": 30, "wins": 20, "ties": 0, "clusters": 21, "cluster_wins": 15 }, "8d736683_12864": { "delta": -0.02694, "median": -0.01994, "ci": [ -0.04015, -0.01507 ], "n": 30, "wins": 24, "ties": 0, "clusters": 21, "cluster_wins": 17 } } }, { "key": "lon_rate_rms", "label": "Longitudinal action rate, RMS (smoothness)", "unit": "m/s³", "better": "lower", "value": 0.8898, "median": 0.7731, "ci": [ 0.7801, 1.02 ], "n": 39, "clusters": 23, "paired": { "lebowski_877m": { "delta": -0.4432, "median": -0.3429, "ci": [ -0.5735, -0.3293 ], "n": 39, "wins": 37, "ties": 0, "clusters": 23, "cluster_wins": 23 }, "8d736683_12864": { "delta": -0.007157, "median": -0.002395, "ci": [ -0.03248, 0.0153 ], "n": 39, "wins": 21, "ties": 0, "clusters": 23, "cluster_wins": 13 } } }, { "key": "v3_mae", "label": "Plan speed error at 3 s", "unit": "m/s", "better": "lower", "value": 0.7602, "median": 0.7121, "ci": [ 0.6567, 0.888 ], "n": 39, "clusters": 23, "paired": { "lebowski_877m": { "delta": 0.04442, "median": 0.04624, "ci": [ 0.009032, 0.07746 ], "n": 39, "wins": 12, "ties": 0, "clusters": 23, "cluster_wins": 8 }, "8d736683_12864": { "delta": 0.0, "median": 0.0, "ci": [ 0.0, 0.0 ], "n": 39, "wins": 0, "ties": 39, "clusters": 23, "cluster_wins": 0 } } } ], "per_unit_file": "per_unit/1365126a_12864/public_open_loop.csv" }, "public_closed_loop": { "status": "ok", "unit_of_evidence": "segment", "n": 28, "clusters": 19, "condition": "Measured. comma's worldmodel-4B generates each next frame at 5 Hz from 9 history frames, 5 recorded anchor frames and the car's pose; the model drives comma's ideal car (the requested curvature and acceleration happen at once) for 7.2 s from the recorded start. Distance is to the recording car's own path. One window per segment, nearest the middle, where the car holds at least 5 m/s for the window and the 4.8 s before it; segments with no such stretch are left out. Seeds 0 and 1, averaged per segment. 28 of the 45 listed segments have a score, from 19 drives and 17 cars.", "baselines": [ "lebowski_877m", "8d736683_12864" ], "metrics": [ { "key": "xtrack_2s", "label": "Distance from the recorded path at 2 s", "unit": "m", "better": "lower", "value": 0.08729, "median": 0.07398, "ci": [ 0.05635, 0.1345 ], "n": 28, "clusters": 19, "paired": { "lebowski_877m": { "delta": -0.0604, "median": -0.008358, "ci": [ -0.1468, -0.005501 ], "n": 28, "wins": 17, "ties": 0, "clusters": 19, "cluster_wins": 12 }, "8d736683_12864": { "delta": -0.009095, "median": 0.009605, "ci": [ -0.04794, 0.01409 ], "n": 28, "wins": 11, "ties": 0, "clusters": 19, "cluster_wins": 7 } } }, { "key": "xtrack_4s", "label": "Distance from the recorded path at 4 s", "unit": "m", "better": "lower", "value": 0.1701, "median": 0.1268, "ci": [ 0.114, 0.2327 ], "n": 28, "clusters": 19, "paired": { "lebowski_877m": { "delta": -0.1081, "median": 0.01134, "ci": [ -0.3106, 0.01546 ], "n": 28, "wins": 12, "ties": 0, "clusters": 19, "cluster_wins": 8 }, "8d736683_12864": { "delta": -0.05086, "median": 0.007072, "ci": [ -0.2045, 0.03854 ], "n": 28, "wins": 12, "ties": 0, "clusters": 19, "cluster_wins": 7 } } }, { "key": "xtrack_7s", "label": "Distance from the recorded path at 7.2 s", "unit": "m", "better": "lower", "value": 0.9987, "median": 0.303, "ci": [ 0.3331, 1.9 ], "n": 28, "clusters": 19, "paired": { "lebowski_877m": { "delta": -0.04528, "median": 0.024, "ci": [ -0.621, 0.5269 ], "n": 28, "wins": 10, "ties": 0, "clusters": 19, "cluster_wins": 6 }, "8d736683_12864": { "delta": -0.02599, "median": 0.03136, "ci": [ -0.174, 0.08682 ], "n": 28, "wins": 11, "ties": 0, "clusters": 19, "cluster_wins": 5 } } }, { "key": "fail_1m", "label": "Share of episodes ever more than 1 m off the path", "unit": "", "better": "lower", "value": 0.1429, "median": 0.0, "ci": [ 0.0, 0.3 ], "n": 28, "clusters": 19, "paired": { "lebowski_877m": { "delta": -0.08929, "median": 0.0, "ci": [ -0.2046, 0.0 ], "n": 28, "wins": 3, "ties": 25, "clusters": 19, "cluster_wins": 3 }, "8d736683_12864": { "delta": -0.03571, "median": 0.0, "ci": [ -0.1111, 0.0 ], "n": 28, "wins": 1, "ties": 27, "clusters": 19, "cluster_wins": 1 } } }, { "key": "lat_jerk", "label": "Requested lateral jerk, RMS", "unit": "m/s³", "better": "lower", "value": 0.2008, "median": 0.09599, "ci": [ 0.1261, 0.2847 ], "n": 28, "clusters": 19, "paired": { "lebowski_877m": { "delta": -0.1248, "median": -0.08186, "ci": [ -0.1828, -0.06414 ], "n": 28, "wins": 25, "ties": 0, "clusters": 19, "cluster_wins": 16 }, "8d736683_12864": { "delta": -0.001252, "median": -0.005006, "ci": [ -0.02517, 0.02934 ], "n": 28, "wins": 19, "ties": 0, "clusters": 19, "cluster_wins": 14 } } }, { "key": "long_jerk", "label": "Requested longitudinal jerk, RMS", "unit": "m/s³", "better": "lower", "value": 0.3563, "median": 0.3015, "ci": [ 0.2746, 0.4546 ], "n": 28, "clusters": 19, "paired": { "lebowski_877m": { "delta": 0.009958, "median": -0.01544, "ci": [ -0.03029, 0.05998 ], "n": 28, "wins": 15, "ties": 0, "clusters": 19, "cluster_wins": 11 }, "8d736683_12864": { "delta": 0.01534, "median": 0.01154, "ci": [ -0.001606, 0.03174 ], "n": 28, "wins": 9, "ties": 0, "clusters": 19, "cluster_wins": 5 } } }, { "key": "speed_err", "label": "Speed error against the recorded speed, RMS", "unit": "m/s", "better": "lower", "value": 1.176, "median": 0.6985, "ci": [ 0.8381, 1.557 ], "n": 28, "clusters": 19, "paired": { "lebowski_877m": { "delta": 0.07429, "median": 0.1045, "ci": [ -0.1291, 0.269 ], "n": 28, "wins": 8, "ties": 0, "clusters": 19, "cluster_wins": 7 }, "8d736683_12864": { "delta": -0.06572, "median": -0.04657, "ci": [ -0.127, 0.001138 ], "n": 28, "wins": 17, "ties": 0, "clusters": 19, "cluster_wins": 11 } } } ], "per_unit_file": "per_unit/1365126a_12864/public_closed_loop.csv" }, "truck_open_loop": { "status": "ok", "unit_of_evidence": "window", "n": 119, "clusters": 12, "condition": "Measured. The G55's recorded drives replayed through the model with both of the truck's cameras (each setup's own wide-camera calibration), road-plane warp to 1.45 m. Where the narrow lens does not cover comma's narrow view (the 8 mm lens in the V13 case), those pixels come from the wide camera, brightness matched. Truth is the truck's CAN speed and yaw rate. 30 s windows after a 5 s warm-up; window = unit, drive = cluster; results are also given per camera setup. Corpus windows of 30 s from the truck's recorded drives, scored per camera setup.", "baselines": [ "lebowski_877m", "8d736683_12864" ], "metrics": [ { "key": "lat_mae", "label": "Lateral error", "unit": "m/s²", "better": "lower", "value": 0.1559, "median": 0.1438, "ci": [ 0.126, 0.1683 ], "n": 119, "clusters": 12, "paired": { "lebowski_877m": { "delta": -0.01682, "median": -0.0146, "ci": [ -0.02432, -0.007891 ], "n": 119, "wins": 85, "ties": 0, "clusters": 12, "cluster_wins": 9 }, "8d736683_12864": { "delta": -0.008089, "median": -0.00762, "ci": [ -0.01116, -0.006196 ], "n": 119, "wins": 102, "ties": 0, "clusters": 12, "cluster_wins": 10 } } }, { "key": "lat_mae_bl", "label": "Lateral error at the model's own best lead", "unit": "m/s²", "better": "lower", "value": 0.143, "median": 0.131, "ci": [ 0.1163, 0.1536 ], "n": 119, "clusters": 12, "paired": { "lebowski_877m": { "delta": -0.02147, "median": -0.01995, "ci": [ -0.02653, -0.01263 ], "n": 119, "wins": 95, "ties": 0, "clusters": 12, "cluster_wins": 10 }, "8d736683_12864": { "delta": -0.002569, "median": -0.002202, "ci": [ -0.005338, -0.000908 ], "n": 119, "wins": 75, "ties": 0, "clusters": 12, "cluster_wins": 11 } } }, { "key": "lat_corr", "label": "Lateral correlation with the car's later motion", "unit": "", "better": "higher", "value": 0.8196, "median": 0.9161, "ci": [ 0.7858, 0.8497 ], "n": 119, "clusters": 12, "paired": { "lebowski_877m": { "delta": 0.05279, "median": 0.01981, "ci": [ 0.02435, 0.06549 ], "n": 119, "wins": 92, "ties": 0, "clusters": 12, "cluster_wins": 9 }, "8d736683_12864": { "delta": 0.01663, "median": 0.01144, "ci": [ 0.01158, 0.02753 ], "n": 119, "wins": 110, "ties": 0, "clusters": 12, "cluster_wins": 12 } } }, { "key": "lat_gain", "label": "Lateral gain (1 = same size as the car's motion)", "unit": "", "better": "near1", "value": 0.7895, "median": 0.8102, "ci": [ 0.77, 0.8755 ], "n": 119, "clusters": 12, "paired": { "lebowski_877m": { "delta": 0.0004234, "median": 0.004503, "ci": [ -0.02215, 0.02995 ], "n": 119, "wins": 54, "ties": 0, "clusters": 12, "cluster_wins": 4 }, "8d736683_12864": { "delta": -0.008323, "median": -0.003263, "ci": [ -0.03648, -0.002037 ], "n": 119, "wins": 65, "ties": 0, "clusters": 12, "cluster_wins": 11 } } }, { "key": "lat_rate_rms", "label": "Lateral action rate, RMS (smoothness)", "unit": "m/s³", "better": "lower", "value": 1.12, "median": 0.8189, "ci": [ 0.6143, 1.355 ], "n": 119, "clusters": 12, "paired": { "lebowski_877m": { "delta": -1.17, "median": -1.082, "ci": [ -1.289, -0.5487 ], "n": 119, "wins": 118, "ties": 0, "clusters": 12, "cluster_wins": 12 }, "8d736683_12864": { "delta": 0.02694, "median": 0.01602, "ci": [ -0.0578, 0.04622 ], "n": 119, "wins": 44, "ties": 0, "clusters": 12, "cluster_wins": 8 } } }, { "key": "lat_best_lag", "label": "Lead of the lateral request over the car's motion", "unit": "s", "better": "lower", "value": 0.6411, "median": 0.6, "ci": [ 0.5778, 0.6809 ], "n": 118, "clusters": 12, "paired": { "lebowski_877m": { "delta": 0.02458, "median": 0.05, "ci": [ -0.05, 0.1146 ], "n": 118, "wins": 40, "ties": 18, "clusters": 12, "cluster_wins": 3 }, "8d736683_12864": { "delta": -0.07542, "median": -0.1, "ci": [ -0.09167, -0.06141 ], "n": 118, "wins": 88, "ties": 28, "clusters": 12, "cluster_wins": 10 } } }, { "key": "lon_mae", "label": "Longitudinal error", "unit": "m/s²", "better": "lower", "value": 0.3499, "median": 0.3603, "ci": [ 0.2677, 0.397 ], "n": 119, "clusters": 12, "paired": { "lebowski_877m": { "delta": -0.01877, "median": -0.01217, "ci": [ -0.06377, 0.004259 ], "n": 119, "wins": 69, "ties": 0, "clusters": 12, "cluster_wins": 5 }, "8d736683_12864": { "delta": -0.004287, "median": -0.0006367, "ci": [ -0.03248, 0.002781 ], "n": 119, "wins": 66, "ties": 0, "clusters": 12, "cluster_wins": 8 } } }, { "key": "lon_corr", "label": "Longitudinal correlation", "unit": "", "better": "higher", "value": 0.3342, "median": 0.2833, "ci": [ 0.2793, 0.5842 ], "n": 119, "clusters": 12, "paired": { "lebowski_877m": { "delta": 0.007321, "median": 0.01468, "ci": [ -0.0389, 0.03704 ], "n": 119, "wins": 68, "ties": 0, "clusters": 12, "cluster_wins": 6 }, "8d736683_12864": { "delta": -0.004906, "median": -0.004094, "ci": [ -0.01053, 0.002126 ], "n": 119, "wins": 51, "ties": 0, "clusters": 12, "cluster_wins": 4 } } }, { "key": "lon_brake_mae", "label": "Longitudinal error while braking", "unit": "m/s²", "better": "lower", "value": 0.4795, "median": 0.4752, "ci": [ 0.4427, 0.6186 ], "n": 81, "clusters": 12, "paired": { "lebowski_877m": { "delta": -0.01933, "median": -0.03879, "ci": [ -0.04079, 0.07156 ], "n": 81, "wins": 52, "ties": 0, "clusters": 12, "cluster_wins": 6 }, "8d736683_12864": { "delta": 0.003548, "median": 0.003686, "ci": [ -0.006772, 0.04861 ], "n": 81, "wins": 37, "ties": 0, "clusters": 12, "cluster_wins": 2 } } }, { "key": "lon_rate_rms", "label": "Longitudinal action rate, RMS (smoothness)", "unit": "m/s³", "better": "lower", "value": 4.406, "median": 4.28, "ci": [ 2.027, 5.746 ], "n": 119, "clusters": 12, "paired": { "lebowski_877m": { "delta": -1.204, "median": -1.263, "ci": [ -1.787, -0.8054 ], "n": 119, "wins": 93, "ties": 0, "clusters": 12, "cluster_wins": 11 }, "8d736683_12864": { "delta": -0.07842, "median": -0.07187, "ci": [ -0.1037, -0.01001 ], "n": 119, "wins": 84, "ties": 0, "clusters": 12, "cluster_wins": 9 } } }, { "key": "v3_mae", "label": "Plan speed error at 3 s", "unit": "m/s", "better": "lower", "value": 1.889, "median": 1.838, "ci": [ 1.152, 2.301 ], "n": 119, "clusters": 12, "paired": { "lebowski_877m": { "delta": -0.1086, "median": -0.1249, "ci": [ -0.3518, 0.306 ], "n": 119, "wins": 78, "ties": 0, "clusters": 12, "cluster_wins": 5 }, "8d736683_12864": { "delta": 0.0, "median": 0.0, "ci": [ 0.0, 0.0 ], "n": 119, "wins": 0, "ties": 119, "clusters": 12, "cluster_wins": 0 } } } ], "per_unit_file": null, "by_group": { "stock-aimedup": { "n": 4, "clusters": 2, "metrics": [ { "key": "lat_mae", "value": 0.1188, "median": 0.1194, "ci": null, "n": 4, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.02235, "median": -0.02475, "ci": null, "n": 4, "wins": 3, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "8d736683_12864": { "delta": 0.004355, "median": 0.00317, "ci": null, "n": 4, "wins": 2, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "lat_mae_bl", "value": 0.1084, "median": 0.1087, "ci": null, "n": 4, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.03113, "median": -0.03482, "ci": null, "n": 4, "wins": 4, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "8d736683_12864": { "delta": 0.00366, "median": -0.001688, "ci": null, "n": 4, "wins": 3, "ties": 0, "clusters": 2, "cluster_wins": 1 } } }, { "key": "lat_corr", "value": 0.8322, "median": 0.9167, "ci": null, "n": 4, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.0371, "median": 0.01673, "ci": null, "n": 4, "wins": 2, "ties": 0, "clusters": 2, "cluster_wins": 1 }, "8d736683_12864": { "delta": 0.02067, "median": 0.0194, "ci": null, "n": 4, "wins": 4, "ties": 0, "clusters": 2, "cluster_wins": 2 } } }, { "key": "lat_gain", "value": 0.8402, "median": 0.9026, "ci": null, "n": 4, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.1715, "median": -0.1394, "ci": null, "n": 4, "wins": 3, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "8d736683_12864": { "delta": -0.1229, "median": -0.119, "ci": null, "n": 4, "wins": 4, "ties": 0, "clusters": 2, "cluster_wins": 2 } } }, { "key": "lat_rate_rms", "value": 0.7037, "median": 0.6838, "ci": null, "n": 4, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.4493, "median": -0.4258, "ci": null, "n": 4, "wins": 4, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "8d736683_12864": { "delta": -0.1126, "median": -0.09715, "ci": null, "n": 4, "wins": 4, "ties": 0, "clusters": 2, "cluster_wins": 2 } } }, { "key": "lat_best_lag", "value": 0.6875, "median": 0.625, "ci": null, "n": 4, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.175, "median": 0.175, "ci": null, "n": 4, "wins": 0, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "8d736683_12864": { "delta": 0.0125, "median": -0.025, "ci": null, "n": 4, "wins": 2, "ties": 1, "clusters": 2, "cluster_wins": 1 } } }, { "key": "lon_mae", "value": 0.4043, "median": 0.3778, "ci": null, "n": 4, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.3734, "median": -0.3769, "ci": null, "n": 4, "wins": 4, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "8d736683_12864": { "delta": -0.1724, "median": -0.1628, "ci": null, "n": 4, "wins": 4, "ties": 0, "clusters": 2, "cluster_wins": 2 } } }, { "key": "lon_corr", "value": 0.5138, "median": 0.4807, "ci": null, "n": 4, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.02157, "median": 0.04109, "ci": null, "n": 4, "wins": 3, "ties": 0, "clusters": 2, "cluster_wins": 1 }, "8d736683_12864": { "delta": -0.0221, "median": 0.02689, "ci": null, "n": 4, "wins": 3, "ties": 0, "clusters": 2, "cluster_wins": 1 } } }, { "key": "lon_brake_mae", "value": 0.5127, "median": 0.5127, "ci": null, "n": 2, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.3248, "median": 0.3248, "ci": null, "n": 2, "wins": 0, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "8d736683_12864": { "delta": 0.1735, "median": 0.1735, "ci": null, "n": 2, "wins": 0, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "lon_rate_rms", "value": 1.929, "median": 1.853, "ci": null, "n": 4, "clusters": 2, "paired": { "lebowski_877m": { "delta": -1.365, "median": -1.333, "ci": null, "n": 4, "wins": 4, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "8d736683_12864": { "delta": 0.03409, "median": 0.05075, "ci": null, "n": 4, "wins": 1, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "v3_mae", "value": 1.443, "median": 1.507, "ci": null, "n": 4, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.9457, "median": -0.9688, "ci": null, "n": 4, "wins": 4, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "8d736683_12864": { "delta": 0.0, "median": 0.0, "ci": null, "n": 4, "wins": 0, "ties": 4, "clusters": 2, "cluster_wins": 0 } } } ] }, "stock-dual": { "n": 13, "clusters": 4, "metrics": [ { "key": "lat_mae", "value": 0.1321, "median": 0.1196, "ci": null, "n": 13, "clusters": 4, "paired": { "lebowski_877m": { "delta": -0.01314, "median": -0.01491, "ci": null, "n": 13, "wins": 9, "ties": 0, "clusters": 4, "cluster_wins": 3 }, "8d736683_12864": { "delta": -0.01102, "median": -0.009812, "ci": null, "n": 13, "wins": 12, "ties": 0, "clusters": 4, "cluster_wins": 4 } } }, { "key": "lat_mae_bl", "value": 0.1206, "median": 0.1113, "ci": null, "n": 13, "clusters": 4, "paired": { "lebowski_877m": { "delta": -0.01869, "median": -0.01995, "ci": null, "n": 13, "wins": 9, "ties": 0, "clusters": 4, "cluster_wins": 4 }, "8d736683_12864": { "delta": -0.005401, "median": -0.004092, "ci": null, "n": 13, "wins": 10, "ties": 0, "clusters": 4, "cluster_wins": 4 } } }, { "key": "lat_corr", "value": 0.8441, "median": 0.8658, "ci": null, "n": 13, "clusters": 4, "paired": { "lebowski_877m": { "delta": 0.03082, "median": 0.01646, "ci": null, "n": 13, "wins": 7, "ties": 0, "clusters": 4, "cluster_wins": 4 }, "8d736683_12864": { "delta": 0.0267, "median": 0.02045, "ci": null, "n": 13, "wins": 12, "ties": 0, "clusters": 4, "cluster_wins": 4 } } }, { "key": "lat_gain", "value": 0.9098, "median": 0.903, "ci": null, "n": 13, "clusters": 4, "paired": { "lebowski_877m": { "delta": 0.02587, "median": 0.004386, "ci": null, "n": 13, "wins": 6, "ties": 0, "clusters": 4, "cluster_wins": 1 }, "8d736683_12864": { "delta": -0.02311, "median": -0.01929, "ci": null, "n": 13, "wins": 8, "ties": 0, "clusters": 4, "cluster_wins": 4 } } }, { "key": "lat_rate_rms", "value": 0.6154, "median": 0.605, "ci": null, "n": 13, "clusters": 4, "paired": { "lebowski_877m": { "delta": -0.5605, "median": -0.5172, "ci": null, "n": 13, "wins": 13, "ties": 0, "clusters": 4, "cluster_wins": 4 }, "8d736683_12864": { "delta": -0.08662, "median": -0.1062, "ci": null, "n": 13, "wins": 12, "ties": 0, "clusters": 4, "cluster_wins": 3 } } }, { "key": "lat_best_lag", "value": 0.6462, "median": 0.6, "ci": null, "n": 13, "clusters": 4, "paired": { "lebowski_877m": { "delta": 0.1346, "median": 0.1, "ci": null, "n": 13, "wins": 3, "ties": 0, "clusters": 4, "cluster_wins": 1 }, "8d736683_12864": { "delta": -0.09231, "median": -0.1, "ci": null, "n": 13, "wins": 13, "ties": 0, "clusters": 4, "cluster_wins": 4 } } }, { "key": "lon_mae", "value": 0.2666, "median": 0.249, "ci": null, "n": 13, "clusters": 4, "paired": { "lebowski_877m": { "delta": 0.03839, "median": 0.03047, "ci": null, "n": 13, "wins": 2, "ties": 0, "clusters": 4, "cluster_wins": 0 }, "8d736683_12864": { "delta": -0.004784, "median": -0.002275, "ci": null, "n": 13, "wins": 8, "ties": 0, "clusters": 4, "cluster_wins": 3 } } }, { "key": "lon_corr", "value": 0.5997, "median": 0.7271, "ci": null, "n": 13, "clusters": 4, "paired": { "lebowski_877m": { "delta": -0.03445, "median": -0.0227, "ci": null, "n": 13, "wins": 6, "ties": 0, "clusters": 4, "cluster_wins": 1 }, "8d736683_12864": { "delta": 0.009624, "median": 0.0103, "ci": null, "n": 13, "wins": 10, "ties": 0, "clusters": 4, "cluster_wins": 3 } } }, { "key": "lon_brake_mae", "value": 0.6389, "median": 0.6796, "ci": null, "n": 6, "clusters": 4, "paired": { "lebowski_877m": { "delta": -0.008381, "median": -0.008678, "ci": null, "n": 6, "wins": 3, "ties": 0, "clusters": 4, "cluster_wins": 1 }, "8d736683_12864": { "delta": 0.04109, "median": 0.0439, "ci": null, "n": 6, "wins": 1, "ties": 0, "clusters": 4, "cluster_wins": 0 } } }, { "key": "lon_rate_rms", "value": 1.177, "median": 1.16, "ci": null, "n": 13, "clusters": 4, "paired": { "lebowski_877m": { "delta": -1.037, "median": -1.107, "ci": null, "n": 13, "wins": 13, "ties": 0, "clusters": 4, "cluster_wins": 4 }, "8d736683_12864": { "delta": -0.03446, "median": -0.02309, "ci": null, "n": 13, "wins": 9, "ties": 0, "clusters": 4, "cluster_wins": 4 } } }, { "key": "v3_mae", "value": 0.8066, "median": 0.8323, "ci": null, "n": 13, "clusters": 4, "paired": { "lebowski_877m": { "delta": -0.03457, "median": -0.06044, "ci": null, "n": 13, "wins": 8, "ties": 0, "clusters": 4, "cluster_wins": 2 }, "8d736683_12864": { "delta": 0.0, "median": 0.0, "ci": null, "n": 13, "wins": 0, "ties": 13, "clusters": 4, "cluster_wins": 0 } } } ] }, "stock-newcase": { "n": 2, "clusters": 1, "metrics": [ { "key": "lat_mae", "value": 0.1412, "median": 0.1412, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": 0.006868, "median": 0.006868, "ci": null, "n": 2, "wins": 0, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "8d736683_12864": { "delta": -0.002995, "median": -0.002995, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 1 } } }, { "key": "lat_mae_bl", "value": 0.1361, "median": 0.1361, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": 0.0009226, "median": 0.0009226, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "8d736683_12864": { "delta": -0.0009953, "median": -0.0009953, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 1 } } }, { "key": "lat_corr", "value": 0.8206, "median": 0.8206, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.02694, "median": -0.02694, "ci": null, "n": 2, "wins": 0, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "8d736683_12864": { "delta": 0.006546, "median": 0.006546, "ci": null, "n": 2, "wins": 2, "ties": 0, "clusters": 1, "cluster_wins": 1 } } }, { "key": "lat_gain", "value": 0.7783, "median": 0.7783, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": 0.01158, "median": 0.01158, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "8d736683_12864": { "delta": -0.002936, "median": -0.002936, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 1 } } }, { "key": "lat_rate_rms", "value": 0.7628, "median": 0.7628, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.5128, "median": -0.5128, "ci": null, "n": 2, "wins": 2, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "8d736683_12864": { "delta": -0.0425, "median": -0.0425, "ci": null, "n": 2, "wins": 2, "ties": 0, "clusters": 1, "cluster_wins": 1 } } }, { "key": "lat_best_lag", "value": 0.625, "median": 0.625, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": 0.125, "median": 0.125, "ci": null, "n": 2, "wins": 0, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "8d736683_12864": { "delta": 0.025, "median": 0.025, "ci": null, "n": 2, "wins": 0, "ties": 1, "clusters": 1, "cluster_wins": 0 } } }, { "key": "lon_mae", "value": 0.3796, "median": 0.3796, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": 0.01705, "median": 0.01705, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "8d736683_12864": { "delta": 0.003412, "median": 0.003412, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 } } }, { "key": "lon_corr", "value": 0.5276, "median": 0.5276, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": 0.04093, "median": 0.04093, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "8d736683_12864": { "delta": -0.01228, "median": -0.01228, "ci": null, "n": 2, "wins": 0, "ties": 0, "clusters": 1, "cluster_wins": 0 } } }, { "key": "lon_brake_mae", "value": 0.4267, "median": 0.4267, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.07658, "median": -0.07658, "ci": null, "n": 2, "wins": 2, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "8d736683_12864": { "delta": 0.03775, "median": 0.03775, "ci": null, "n": 2, "wins": 0, "ties": 0, "clusters": 1, "cluster_wins": 0 } } }, { "key": "lon_rate_rms", "value": 2.338, "median": 2.338, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": -1.018, "median": -1.018, "ci": null, "n": 2, "wins": 2, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "8d736683_12864": { "delta": -0.01187, "median": -0.01187, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 1 } } }, { "key": "v3_mae", "value": 2.129, "median": 2.129, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": 0.03487, "median": 0.03487, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "8d736683_12864": { "delta": 0.0, "median": 0.0, "ci": null, "n": 2, "wins": 0, "ties": 2, "clusters": 1, "cluster_wins": 0 } } } ] }, "v13-8mm": { "n": 92, "clusters": 3, "metrics": [ { "key": "lat_mae", "value": 0.1623, "median": 0.1531, "ci": null, "n": 92, "clusters": 3, "paired": { "lebowski_877m": { "delta": -0.01934, "median": -0.01561, "ci": null, "n": 92, "wins": 71, "ties": 0, "clusters": 3, "cluster_wins": 3 }, "8d736683_12864": { "delta": -0.007942, "median": -0.007574, "ci": null, "n": 92, "wins": 80, "ties": 0, "clusters": 3, "cluster_wins": 3 } } }, { "key": "lat_mae_bl", "value": 0.1484, "median": 0.136, "ci": null, "n": 92, "clusters": 3, "paired": { "lebowski_877m": { "delta": -0.02378, "median": -0.02233, "ci": null, "n": 92, "wins": 78, "ties": 0, "clusters": 3, "cluster_wins": 3 }, "8d736683_12864": { "delta": -0.002414, "median": -0.001764, "ci": null, "n": 92, "wins": 56, "ties": 0, "clusters": 3, "cluster_wins": 3 } } }, { "key": "lat_corr", "value": 0.8111, "median": 0.9195, "ci": null, "n": 92, "clusters": 3, "paired": { "lebowski_877m": { "delta": 0.06262, "median": 0.02505, "ci": null, "n": 92, "wins": 77, "ties": 0, "clusters": 3, "cluster_wins": 3 }, "8d736683_12864": { "delta": 0.0143, "median": 0.008964, "ci": null, "n": 92, "wins": 84, "ties": 0, "clusters": 3, "cluster_wins": 3 } } }, { "key": "lat_gain", "value": 0.7617, "median": 0.8033, "ci": null, "n": 92, "clusters": 3, "paired": { "lebowski_877m": { "delta": 0.0008938, "median": 0.007778, "ci": null, "n": 92, "wins": 41, "ties": 0, "clusters": 3, "cluster_wins": 1 }, "8d736683_12864": { "delta": -0.0005583, "median": -0.001231, "ci": null, "n": 92, "wins": 47, "ties": 0, "clusters": 3, "cluster_wins": 2 } } }, { "key": "lat_rate_rms", "value": 1.263, "median": 0.988, "ci": null, "n": 92, "clusters": 3, "paired": { "lebowski_877m": { "delta": -1.355, "median": -1.311, "ci": null, "n": 92, "wins": 91, "ties": 0, "clusters": 3, "cluster_wins": 3 }, "8d736683_12864": { "delta": 0.0546, "median": 0.03445, "ci": null, "n": 92, "wins": 22, "ties": 0, "clusters": 3, "cluster_wins": 0 } } }, { "key": "lat_best_lag", "value": 0.6478, "median": 0.6, "ci": null, "n": 91, "clusters": 3, "paired": { "lebowski_877m": { "delta": -0.005495, "median": 0.0, "ci": null, "n": 91, "wins": 37, "ties": 15, "clusters": 3, "cluster_wins": 2 }, "8d736683_12864": { "delta": -0.07692, "median": -0.05, "ci": null, "n": 91, "wins": 66, "ties": 25, "clusters": 3, "cluster_wins": 3 } } }, { "key": "lon_mae", "value": 0.359, "median": 0.3722, "ci": null, "n": 92, "clusters": 3, "paired": { "lebowski_877m": { "delta": -0.01705, "median": -0.01934, "ci": null, "n": 92, "wins": 60, "ties": 0, "clusters": 3, "cluster_wins": 3 }, "8d736683_12864": { "delta": 0.002487, "median": -0.0003212, "ci": null, "n": 92, "wins": 48, "ties": 0, "clusters": 3, "cluster_wins": 2 } } }, { "key": "lon_corr", "value": 0.2543, "median": 0.2339, "ci": null, "n": 92, "clusters": 3, "paired": { "lebowski_877m": { "delta": 0.02064, "median": 0.02573, "ci": null, "n": 92, "wins": 55, "ties": 0, "clusters": 3, "cluster_wins": 3 }, "8d736683_12864": { "delta": -0.006139, "median": -0.006438, "ci": null, "n": 92, "wins": 35, "ties": 0, "clusters": 3, "cluster_wins": 0 } } }, { "key": "lon_brake_mae", "value": 0.4511, "median": 0.4406, "ci": null, "n": 65, "clusters": 3, "paired": { "lebowski_877m": { "delta": -0.03799, "median": -0.05023, "ci": null, "n": 65, "wins": 45, "ties": 0, "clusters": 3, "cluster_wins": 3 }, "8d736683_12864": { "delta": -0.005988, "median": -0.001598, "ci": null, "n": 65, "wins": 34, "ties": 0, "clusters": 3, "cluster_wins": 2 } } }, { "key": "lon_rate_rms", "value": 5.001, "median": 5.04, "ci": null, "n": 92, "clusters": 3, "paired": { "lebowski_877m": { "delta": -1.332, "median": -1.54, "ci": null, "n": 92, "wins": 71, "ties": 0, "clusters": 3, "cluster_wins": 3 }, "8d736683_12864": { "delta": -0.09797, "median": -0.09456, "ci": null, "n": 92, "wins": 69, "ties": 0, "clusters": 3, "cluster_wins": 3 } } }, { "key": "v3_mae", "value": 2.125, "median": 1.978, "ci": null, "n": 92, "clusters": 3, "paired": { "lebowski_877m": { "delta": -0.1094, "median": -0.2267, "ci": null, "n": 92, "wins": 63, "ties": 0, "clusters": 3, "cluster_wins": 1 }, "8d736683_12864": { "delta": 0.0, "median": 0.0, "ci": null, "n": 92, "wins": 0, "ties": 92, "clusters": 3, "cluster_wins": 0 } } } ] }, "v14-8mm": { "n": 8, "clusters": 2, "metrics": [ { "key": "lat_mae", "value": 0.1444, "median": 0.144, "ci": null, "n": 8, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.002969, "median": 0.006028, "ci": null, "n": 8, "wins": 2, "ties": 0, "clusters": 2, "cluster_wins": 1 }, "8d736683_12864": { "delta": -0.01251, "median": -0.01243, "ci": null, "n": 8, "wins": 7, "ties": 0, "clusters": 2, "cluster_wins": 2 } } }, { "key": "lat_mae_bl", "value": 0.1369, "median": 0.1291, "ci": null, "n": 8, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.000229, "median": 0.001394, "ci": null, "n": 8, "wins": 3, "ties": 0, "clusters": 2, "cluster_wins": 1 }, "8d736683_12864": { "delta": -0.003268, "median": -0.003463, "ci": null, "n": 8, "wins": 5, "ties": 0, "clusters": 2, "cluster_wins": 2 } } }, { "key": "lat_corr", "value": 0.8712, "median": 0.9218, "ci": null, "n": 8, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.003257, "median": 0.005731, "ci": null, "n": 8, "wins": 6, "ties": 0, "clusters": 2, "cluster_wins": 1 }, "8d736683_12864": { "delta": 0.02747, "median": 0.01506, "ci": null, "n": 8, "wins": 8, "ties": 0, "clusters": 2, "cluster_wins": 2 } } }, { "key": "lat_gain", "value": 0.8901, "median": 0.8953, "ci": null, "n": 8, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.03682, "median": 0.02892, "ci": null, "n": 8, "wins": 3, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "8d736683_12864": { "delta": -0.01764, "median": -0.02544, "ci": null, "n": 8, "wins": 5, "ties": 0, "clusters": 2, "cluster_wins": 2 } } }, { "key": "lat_rate_rms", "value": 0.5917, "median": 0.5507, "ci": null, "n": 8, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.5639, "median": -0.4661, "ci": null, "n": 8, "wins": 8, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "8d736683_12864": { "delta": -0.0195, "median": -0.01657, "ci": null, "n": 8, "wins": 4, "ties": 0, "clusters": 2, "cluster_wins": 2 } } }, { "key": "lat_best_lag", "value": 0.5375, "median": 0.475, "ci": null, "n": 8, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.0875, "median": 0.075, "ci": null, "n": 8, "wins": 0, "ties": 3, "clusters": 2, "cluster_wins": 0 }, "8d736683_12864": { "delta": -0.1, "median": -0.1, "ci": null, "n": 8, "wins": 7, "ties": 1, "clusters": 2, "cluster_wins": 2 } } }, { "key": "lon_mae", "value": 0.3462, "median": 0.2852, "ci": null, "n": 8, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.03686, "median": 0.03374, "ci": null, "n": 8, "wins": 2, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "8d736683_12864": { "delta": 0.0007623, "median": -0.001227, "ci": null, "n": 8, "wins": 5, "ties": 0, "clusters": 2, "cluster_wins": 1 } } }, { "key": "lon_corr", "value": 0.6827, "median": 0.7207, "ci": null, "n": 8, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.07196, "median": -0.05247, "ci": null, "n": 8, "wins": 3, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "8d736683_12864": { "delta": -0.003899, "median": -0.003059, "ci": null, "n": 8, "wins": 3, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "lon_brake_mae", "value": 0.6339, "median": 0.67, "ci": null, "n": 6, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.07635, "median": 0.06655, "ci": null, "n": 6, "wins": 2, "ties": 0, "clusters": 2, "cluster_wins": 1 }, "8d736683_12864": { "delta": 0.001255, "median": 0.004618, "ci": null, "n": 6, "wins": 2, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "lon_rate_rms", "value": 4.565, "median": 4.445, "ci": null, "n": 8, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.03163, "median": 0.2397, "ci": null, "n": 8, "wins": 3, "ties": 0, "clusters": 2, "cluster_wins": 1 }, "8d736683_12864": { "delta": 0.002036, "median": 0.02192, "ci": null, "n": 8, "wins": 4, "ties": 0, "clusters": 2, "cluster_wins": 1 } } }, { "key": "v3_mae", "value": 1.095, "median": 1.065, "ci": null, "n": 8, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.1622, "median": 0.1656, "ci": null, "n": 8, "wins": 2, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "8d736683_12864": { "delta": 0.0, "median": 0.0, "ci": null, "n": 8, "wins": 0, "ties": 8, "clusters": 2, "cluster_wins": 0 } } } ] } } }, "truck_closed_loop": { "status": "ok", "unit_of_evidence": "window", "n": 98, "clusters": 13, "condition": "Measured. worldmodel-4B as in the public closed loop, but the model drives a simulation of the G55's measured steering and powertrain response instead of the ideal car. The simulation starts from the truck's state (speed, yaw rate, steering, gear) in its CAN recording of the 2 s before the window; one window, whose 2 s before are not all valid CAN, starts steady at its recorded speed. Lane-following windows from the G55's drives, day and night; distance is to the driver's recorded path; seeds 0 and 1, averaged per window. 98 windows (59 day, 39 night).", "baselines": [ "lebowski_877m", "8d736683_12864" ], "metrics": [ { "key": "xtrack_2s", "label": "Distance from the recorded path at 2 s", "unit": "m", "better": "lower", "value": 0.1366, "median": 0.08513, "ci": [ 0.1094, 0.1663 ], "n": 98, "clusters": 13, "paired": { "lebowski_877m": { "delta": -0.03286, "median": -0.03125, "ci": [ -0.04376, -0.02211 ], "n": 98, "wins": 64, "ties": 0, "clusters": 13, "cluster_wins": 12 }, "8d736683_12864": { "delta": 0.004932, "median": 0.007312, "ci": [ -0.004076, 0.01482 ], "n": 98, "wins": 42, "ties": 0, "clusters": 13, "cluster_wins": 5 } } }, { "key": "xtrack_4s", "label": "Distance from the recorded path at 4 s", "unit": "m", "better": "lower", "value": 0.5395, "median": 0.3184, "ci": [ 0.3923, 0.6918 ], "n": 98, "clusters": 13, "paired": { "lebowski_877m": { "delta": 0.01944, "median": 0.006087, "ci": [ -0.1295, 0.1766 ], "n": 98, "wins": 48, "ties": 0, "clusters": 13, "cluster_wins": 5 }, "8d736683_12864": { "delta": 0.03138, "median": 0.006141, "ci": [ -0.003798, 0.07633 ], "n": 98, "wins": 44, "ties": 0, "clusters": 13, "cluster_wins": 4 } } }, { "key": "xtrack_7s", "label": "Distance from the recorded path at 7.2 s", "unit": "m", "better": "lower", "value": 2.531, "median": 0.88, "ci": [ 1.661, 3.592 ], "n": 98, "clusters": 13, "paired": { "lebowski_877m": { "delta": 0.5517, "median": 0.08127, "ci": [ -0.4561, 1.706 ], "n": 98, "wins": 34, "ties": 0, "clusters": 13, "cluster_wins": 4 }, "8d736683_12864": { "delta": 0.4849, "median": 0.09913, "ci": [ 0.1895, 0.9448 ], "n": 98, "wins": 32, "ties": 0, "clusters": 13, "cluster_wins": 0 } } }, { "key": "fail_1m", "label": "Share of episodes ever more than 1 m off the path", "unit": "", "better": "lower", "value": 0.4541, "median": 0.5, "ci": [ 0.3836, 0.5542 ], "n": 98, "clusters": 13, "paired": { "lebowski_877m": { "delta": 0.02551, "median": 0.0, "ci": [ -0.04903, 0.1081 ], "n": 98, "wins": 11, "ties": 73, "clusters": 13, "cluster_wins": 5 }, "8d736683_12864": { "delta": 0.05102, "median": 0.0, "ci": [ 0.02358, 0.0822 ], "n": 98, "wins": 2, "ties": 85, "clusters": 13, "cluster_wins": 0 } } }, { "key": "lat_jerk", "label": "Requested lateral jerk, RMS", "unit": "m/s³", "better": "lower", "value": 0.2859, "median": 0.1778, "ci": [ 0.2307, 0.3391 ], "n": 98, "clusters": 13, "paired": { "lebowski_877m": { "delta": -0.256, "median": -0.1909, "ci": [ -0.3511, -0.1708 ], "n": 98, "wins": 86, "ties": 0, "clusters": 13, "cluster_wins": 13 }, "8d736683_12864": { "delta": 0.0219, "median": 0.006657, "ci": [ 0.004696, 0.03921 ], "n": 98, "wins": 42, "ties": 0, "clusters": 13, "cluster_wins": 5 } } }, { "key": "long_jerk", "label": "Requested longitudinal jerk, RMS", "unit": "m/s³", "better": "lower", "value": 0.365, "median": 0.2768, "ci": [ 0.2988, 0.4474 ], "n": 98, "clusters": 13, "paired": { "lebowski_877m": { "delta": -0.02126, "median": -0.05103, "ci": [ -0.1002, 0.07444 ], "n": 98, "wins": 65, "ties": 0, "clusters": 13, "cluster_wins": 8 }, "8d736683_12864": { "delta": -0.0058, "median": -0.008786, "ci": [ -0.0333, 0.01927 ], "n": 98, "wins": 62, "ties": 0, "clusters": 13, "cluster_wins": 6 } } }, { "key": "speed_err", "label": "Speed error against the recorded speed, RMS", "unit": "m/s", "better": "lower", "value": 1.219, "median": 0.9687, "ci": [ 1.017, 1.419 ], "n": 98, "clusters": 13, "paired": { "lebowski_877m": { "delta": 0.1364, "median": 0.1346, "ci": [ -0.02489, 0.2999 ], "n": 98, "wins": 38, "ties": 0, "clusters": 13, "cluster_wins": 7 }, "8d736683_12864": { "delta": 0.002198, "median": 0.02242, "ci": [ -0.08161, 0.08619 ], "n": 98, "wins": 44, "ties": 0, "clusters": 13, "cluster_wins": 7 } } } ], "per_unit_file": null, "by_group": { "stock-aimedup": { "n": 14, "clusters": 2, "metrics": [ { "key": "xtrack_2s", "value": 0.1241, "median": 0.08513, "ci": null, "n": 14, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.02619, "median": -0.008255, "ci": null, "n": 14, "wins": 7, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "8d736683_12864": { "delta": 0.003018, "median": 0.01401, "ci": null, "n": 14, "wins": 5, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "xtrack_4s", "value": 0.7, "median": 0.5178, "ci": null, "n": 14, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.2489, "median": 0.07267, "ci": null, "n": 14, "wins": 5, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "8d736683_12864": { "delta": 0.1567, "median": 0.03412, "ci": null, "n": 14, "wins": 5, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "xtrack_7s", "value": 4.433, "median": 3.978, "ci": null, "n": 14, "clusters": 2, "paired": { "lebowski_877m": { "delta": 2.528, "median": 0.8649, "ci": null, "n": 14, "wins": 3, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "8d736683_12864": { "delta": 1.897, "median": 1.46, "ci": null, "n": 14, "wins": 2, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "fail_1m", "value": 0.75, "median": 1.0, "ci": null, "n": 14, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.07143, "median": 0.0, "ci": null, "n": 14, "wins": 3, "ties": 7, "clusters": 2, "cluster_wins": 1 }, "8d736683_12864": { "delta": 0.07143, "median": 0.0, "ci": null, "n": 14, "wins": 1, "ties": 10, "clusters": 2, "cluster_wins": 0 } } }, { "key": "lat_jerk", "value": 0.4613, "median": 0.2436, "ci": null, "n": 14, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.1095, "median": -0.1426, "ci": null, "n": 14, "wins": 9, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "8d736683_12864": { "delta": 0.07701, "median": 0.03865, "ci": null, "n": 14, "wins": 5, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "long_jerk", "value": 0.3061, "median": 0.2639, "ci": null, "n": 14, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.1886, "median": -0.1422, "ci": null, "n": 14, "wins": 13, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "8d736683_12864": { "delta": -0.02386, "median": -0.03063, "ci": null, "n": 14, "wins": 12, "ties": 0, "clusters": 2, "cluster_wins": 2 } } }, { "key": "speed_err", "value": 1.558, "median": 1.434, "ci": null, "n": 14, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.2059, "median": -0.677, "ci": null, "n": 14, "wins": 9, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "8d736683_12864": { "delta": -0.1798, "median": -0.3206, "ci": null, "n": 14, "wins": 9, "ties": 0, "clusters": 2, "cluster_wins": 2 } } } ] }, "stock-dual": { "n": 43, "clusters": 5, "metrics": [ { "key": "xtrack_2s", "value": 0.1349, "median": 0.09727, "ci": null, "n": 43, "clusters": 5, "paired": { "lebowski_877m": { "delta": -0.0332, "median": -0.0394, "ci": null, "n": 43, "wins": 29, "ties": 0, "clusters": 5, "cluster_wins": 4 }, "8d736683_12864": { "delta": 0.007055, "median": 0.007877, "ci": null, "n": 43, "wins": 17, "ties": 0, "clusters": 5, "cluster_wins": 1 } } }, { "key": "xtrack_4s", "value": 0.4114, "median": 0.2878, "ci": null, "n": 43, "clusters": 5, "paired": { "lebowski_877m": { "delta": -0.04087, "median": -0.04213, "ci": null, "n": 43, "wins": 24, "ties": 0, "clusters": 5, "cluster_wins": 3 }, "8d736683_12864": { "delta": -0.009353, "median": -0.008311, "ci": null, "n": 43, "wins": 25, "ties": 0, "clusters": 5, "cluster_wins": 3 } } }, { "key": "xtrack_7s", "value": 1.959, "median": 0.6737, "ci": null, "n": 43, "clusters": 5, "paired": { "lebowski_877m": { "delta": -0.06066, "median": 0.08085, "ci": null, "n": 43, "wins": 15, "ties": 0, "clusters": 5, "cluster_wins": 2 }, "8d736683_12864": { "delta": 0.3327, "median": 0.06014, "ci": null, "n": 43, "wins": 15, "ties": 0, "clusters": 5, "cluster_wins": 0 } } }, { "key": "fail_1m", "value": 0.3837, "median": 0.0, "ci": null, "n": 43, "clusters": 5, "paired": { "lebowski_877m": { "delta": 0.04651, "median": 0.0, "ci": null, "n": 43, "wins": 4, "ties": 32, "clusters": 5, "cluster_wins": 2 }, "8d736683_12864": { "delta": 0.04651, "median": 0.0, "ci": null, "n": 43, "wins": 1, "ties": 37, "clusters": 5, "cluster_wins": 0 } } }, { "key": "lat_jerk", "value": 0.2938, "median": 0.1953, "ci": null, "n": 43, "clusters": 5, "paired": { "lebowski_877m": { "delta": -0.259, "median": -0.2073, "ci": null, "n": 43, "wins": 39, "ties": 0, "clusters": 5, "cluster_wins": 5 }, "8d736683_12864": { "delta": 0.01618, "median": -0.001883, "ci": null, "n": 43, "wins": 22, "ties": 0, "clusters": 5, "cluster_wins": 2 } } }, { "key": "long_jerk", "value": 0.4248, "median": 0.3369, "ci": null, "n": 43, "clusters": 5, "paired": { "lebowski_877m": { "delta": 0.05054, "median": -0.02916, "ci": null, "n": 43, "wins": 25, "ties": 0, "clusters": 5, "cluster_wins": 3 }, "8d736683_12864": { "delta": 0.003221, "median": -0.008722, "ci": null, "n": 43, "wins": 27, "ties": 0, "clusters": 5, "cluster_wins": 2 } } }, { "key": "speed_err", "value": 1.238, "median": 1.069, "ci": null, "n": 43, "clusters": 5, "paired": { "lebowski_877m": { "delta": 0.1549, "median": 0.1129, "ci": null, "n": 43, "wins": 19, "ties": 0, "clusters": 5, "cluster_wins": 2 }, "8d736683_12864": { "delta": 0.01649, "median": 0.03638, "ci": null, "n": 43, "wins": 18, "ties": 0, "clusters": 5, "cluster_wins": 2 } } } ] }, "stock-newcase": { "n": 4, "clusters": 1, "metrics": [ { "key": "xtrack_2s", "value": 0.04954, "median": 0.04916, "ci": null, "n": 4, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.07532, "median": -0.07235, "ci": null, "n": 4, "wins": 3, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "8d736683_12864": { "delta": -0.001556, "median": -0.003973, "ci": null, "n": 4, "wins": 2, "ties": 0, "clusters": 1, "cluster_wins": 1 } } }, { "key": "xtrack_4s", "value": 0.2574, "median": 0.2489, "ci": null, "n": 4, "clusters": 1, "paired": { "lebowski_877m": { "delta": 0.01682, "median": 0.04023, "ci": null, "n": 4, "wins": 2, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "8d736683_12864": { "delta": 0.05059, "median": 0.1161, "ci": null, "n": 4, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 } } }, { "key": "xtrack_7s", "value": 1.553, "median": 1.467, "ci": null, "n": 4, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.08268, "median": 0.1123, "ci": null, "n": 4, "wins": 2, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "8d736683_12864": { "delta": 0.4766, "median": 0.367, "ci": null, "n": 4, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 } } }, { "key": "fail_1m", "value": 0.5, "median": 0.5, "ci": null, "n": 4, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.125, "median": 0.0, "ci": null, "n": 4, "wins": 1, "ties": 3, "clusters": 1, "cluster_wins": 1 }, "8d736683_12864": { "delta": 0.0, "median": 0.0, "ci": null, "n": 4, "wins": 0, "ties": 4, "clusters": 1, "cluster_wins": 0 } } }, { "key": "lat_jerk", "value": 0.2665, "median": 0.2096, "ci": null, "n": 4, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.4732, "median": -0.3577, "ci": null, "n": 4, "wins": 4, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "8d736683_12864": { "delta": 0.02214, "median": 0.01687, "ci": null, "n": 4, "wins": 2, "ties": 0, "clusters": 1, "cluster_wins": 0 } } }, { "key": "long_jerk", "value": 0.441, "median": 0.4162, "ci": null, "n": 4, "clusters": 1, "paired": { "lebowski_877m": { "delta": 0.05176, "median": 0.01439, "ci": null, "n": 4, "wins": 2, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "8d736683_12864": { "delta": 0.06524, "median": 0.06389, "ci": null, "n": 4, "wins": 2, "ties": 0, "clusters": 1, "cluster_wins": 0 } } }, { "key": "speed_err", "value": 1.047, "median": 0.8382, "ci": null, "n": 4, "clusters": 1, "paired": { "lebowski_877m": { "delta": 0.3681, "median": 0.4731, "ci": null, "n": 4, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "8d736683_12864": { "delta": 0.3644, "median": 0.1773, "ci": null, "n": 4, "wins": 0, "ties": 0, "clusters": 1, "cluster_wins": 0 } } } ] }, "v13-8mm": { "n": 18, "clusters": 3, "metrics": [ { "key": "xtrack_2s", "value": 0.1858, "median": 0.0966, "ci": null, "n": 18, "clusters": 3, "paired": { "lebowski_877m": { "delta": -0.03155, "median": -0.02585, "ci": null, "n": 18, "wins": 14, "ties": 0, "clusters": 3, "cluster_wins": 3 }, "8d736683_12864": { "delta": 0.01363, "median": 0.00754, "ci": null, "n": 18, "wins": 6, "ties": 0, "clusters": 3, "cluster_wins": 1 } } }, { "key": "xtrack_4s", "value": 0.5514, "median": 0.2758, "ci": null, "n": 18, "clusters": 3, "paired": { "lebowski_877m": { "delta": -0.2507, "median": 0.003688, "ci": null, "n": 18, "wins": 9, "ties": 0, "clusters": 3, "cluster_wins": 1 }, "8d736683_12864": { "delta": 0.06956, "median": 0.01243, "ci": null, "n": 18, "wins": 4, "ties": 0, "clusters": 3, "cluster_wins": 0 } } }, { "key": "xtrack_7s", "value": 1.401, "median": 0.9197, "ci": null, "n": 18, "clusters": 3, "paired": { "lebowski_877m": { "delta": -1.224, "median": 0.05456, "ci": null, "n": 18, "wins": 6, "ties": 0, "clusters": 3, "cluster_wins": 1 }, "8d736683_12864": { "delta": 0.0986, "median": 0.03897, "ci": null, "n": 18, "wins": 5, "ties": 0, "clusters": 3, "cluster_wins": 0 } } }, { "key": "fail_1m", "value": 0.4167, "median": 0.5, "ci": null, "n": 18, "clusters": 3, "paired": { "lebowski_877m": { "delta": 0.02778, "median": 0.0, "ci": null, "n": 18, "wins": 1, "ties": 15, "clusters": 3, "cluster_wins": 0 }, "8d736683_12864": { "delta": 0.08333, "median": 0.0, "ci": null, "n": 18, "wins": 0, "ties": 16, "clusters": 3, "cluster_wins": 0 } } }, { "key": "lat_jerk", "value": 0.2329, "median": 0.1539, "ci": null, "n": 18, "clusters": 3, "paired": { "lebowski_877m": { "delta": -0.1465, "median": -0.1286, "ci": null, "n": 18, "wins": 15, "ties": 0, "clusters": 3, "cluster_wins": 3 }, "8d736683_12864": { "delta": 0.01272, "median": 0.005605, "ci": null, "n": 18, "wins": 7, "ties": 0, "clusters": 3, "cluster_wins": 2 } } }, { "key": "long_jerk", "value": 0.3177, "median": 0.2266, "ci": null, "n": 18, "clusters": 3, "paired": { "lebowski_877m": { "delta": 0.01311, "median": -0.005136, "ci": null, "n": 18, "wins": 11, "ties": 0, "clusters": 3, "cluster_wins": 1 }, "8d736683_12864": { "delta": 0.007682, "median": 0.007344, "ci": null, "n": 18, "wins": 7, "ties": 0, "clusters": 3, "cluster_wins": 1 } } }, { "key": "speed_err", "value": 0.8584, "median": 0.7409, "ci": null, "n": 18, "clusters": 3, "paired": { "lebowski_877m": { "delta": -0.08621, "median": 0.04906, "ci": null, "n": 18, "wins": 6, "ties": 0, "clusters": 3, "cluster_wins": 3 }, "8d736683_12864": { "delta": 0.1034, "median": 0.05405, "ci": null, "n": 18, "wins": 6, "ties": 0, "clusters": 3, "cluster_wins": 1 } } } ] }, "v14-8mm": { "n": 19, "clusters": 2, "metrics": [ { "key": "xtrack_2s", "value": 0.1216, "median": 0.07043, "ci": null, "n": 19, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.02931, "median": -0.02463, "ci": null, "n": 19, "wins": 11, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "8d736683_12864": { "delta": -0.005332, "median": -0.005139, "ci": null, "n": 19, "wins": 12, "ties": 0, "clusters": 2, "cluster_wins": 2 } } }, { "key": "xtrack_4s", "value": 0.7591, "median": 0.3289, "ci": null, "n": 19, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.2433, "median": 0.05484, "ci": null, "n": 19, "wins": 8, "ties": 0, "clusters": 2, "cluster_wins": 1 }, "8d736683_12864": { "delta": -0.008984, "median": 0.001741, "ci": null, "n": 19, "wins": 9, "ties": 0, "clusters": 2, "cluster_wins": 1 } } }, { "key": "xtrack_7s", "value": 3.701, "median": 0.7762, "ci": null, "n": 19, "clusters": 2, "paired": { "lebowski_877m": { "delta": 2.297, "median": 0.01118, "ci": null, "n": 19, "wins": 8, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "8d736683_12864": { "delta": 0.1565, "median": 0.05425, "ci": null, "n": 19, "wins": 9, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "fail_1m", "value": 0.4211, "median": 0.0, "ci": null, "n": 19, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.02632, "median": 0.0, "ci": null, "n": 19, "wins": 2, "ties": 16, "clusters": 2, "cluster_wins": 1 }, "8d736683_12864": { "delta": 0.02632, "median": 0.0, "ci": null, "n": 19, "wins": 0, "ties": 18, "clusters": 2, "cluster_wins": 0 } } }, { "key": "lat_jerk", "value": 0.1931, "median": 0.1593, "ci": null, "n": 19, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.4154, "median": -0.2692, "ci": null, "n": 19, "wins": 19, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "8d736683_12864": { "delta": 0.002911, "median": 0.01589, "ci": null, "n": 19, "wins": 6, "ties": 0, "clusters": 2, "cluster_wins": 1 } } }, { "key": "long_jerk", "value": 0.302, "median": 0.2909, "ci": null, "n": 19, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.1084, "median": -0.1016, "ci": null, "n": 19, "wins": 14, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "8d736683_12864": { "delta": -0.04064, "median": -0.006269, "ci": null, "n": 19, "wins": 14, "ties": 0, "clusters": 2, "cluster_wins": 1 } } }, { "key": "speed_err", "value": 1.304, "median": 0.9182, "ci": null, "n": 19, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.5089, "median": 0.2626, "ci": null, "n": 19, "wins": 3, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "8d736683_12864": { "delta": -0.0682, "median": -0.1018, "ci": null, "n": 19, "wins": 11, "ties": 0, "clusters": 2, "cluster_wins": 2 } } } ] } }, "simulator": "g55-2026-10-31" }, "truck_recovery": { "status": "ok", "unit_of_evidence": "window", "n": 100, "clusters": 14, "condition": "Measured. The truck closed loop with a disturbance while the model drives; only the tests listed ran. 2° off heading: an extra 2°/s yaw for the first second. 98 windows (59 day, 39 night). 0.5 m off centre: an extra yaw-rate pulse, out and back over the first 2 s, that would move an uncorrected car 0.5 m sideways with no net heading change. 98 windows (59 day, 39 night). Braking lead: windows where the car ahead braked; the gap is derived from the lead distance seen on the recording minus how far the model ran ahead of the recorded car. 2 windows (2 day, 0 night).", "baselines": [ "lebowski_877m", "8d736683_12864" ], "metrics": [ { "key": "heading_xtrack_2s", "label": "2° off heading: distance from the path at 2 s", "unit": "m", "better": "lower", "value": 0.5174, "median": 0.4347, "ci": [ 0.4255, 0.6293 ], "n": 98, "clusters": 13, "paired": { "lebowski_877m": { "delta": 0.06453, "median": 0.05581, "ci": [ 0.028, 0.1035 ], "n": 98, "wins": 34, "ties": 0, "clusters": 13, "cluster_wins": 4 }, "8d736683_12864": { "delta": 0.02586, "median": 0.02288, "ci": [ 0.02213, 0.02881 ], "n": 98, "wins": 8, "ties": 0, "clusters": 13, "cluster_wins": 0 } } }, { "key": "heading_xtrack_4s", "label": "2° off heading: distance from the path at 4 s", "unit": "m", "better": "lower", "value": 0.7885, "median": 0.4656, "ci": [ 0.575, 1.041 ], "n": 98, "clusters": 13, "paired": { "lebowski_877m": { "delta": 0.08597, "median": 0.04848, "ci": [ -0.03636, 0.1893 ], "n": 98, "wins": 38, "ties": 0, "clusters": 13, "cluster_wins": 4 }, "8d736683_12864": { "delta": 0.04059, "median": 0.008763, "ci": [ -0.001116, 0.09882 ], "n": 98, "wins": 47, "ties": 0, "clusters": 13, "cluster_wins": 7 } } }, { "key": "heading_lat_jerk", "label": "2° off heading: requested lateral jerk, RMS", "unit": "m/s³", "better": "lower", "value": 0.3712, "median": 0.3035, "ci": [ 0.3348, 0.4084 ], "n": 98, "clusters": 13, "paired": { "lebowski_877m": { "delta": -0.5157, "median": -0.4674, "ci": [ -0.5862, -0.4414 ], "n": 98, "wins": 95, "ties": 0, "clusters": 13, "cluster_wins": 13 }, "8d736683_12864": { "delta": 0.00892, "median": 0.008712, "ci": [ -0.01708, 0.0363 ], "n": 98, "wins": 39, "ties": 0, "clusters": 13, "cluster_wins": 4 } } }, { "key": "offcentre_xtrack_2s", "label": "0.5 m off centre: distance from the path at 2 s", "unit": "m", "better": "lower", "value": 0.285, "median": 0.2379, "ci": [ 0.2595, 0.3096 ], "n": 98, "clusters": 13, "paired": { "lebowski_877m": { "delta": 0.01796, "median": 0.02847, "ci": [ -0.004059, 0.04037 ], "n": 98, "wins": 34, "ties": 0, "clusters": 13, "cluster_wins": 4 }, "8d736683_12864": { "delta": 0.02727, "median": 0.02348, "ci": [ 0.02057, 0.034 ], "n": 98, "wins": 14, "ties": 0, "clusters": 13, "cluster_wins": 0 } } }, { "key": "offcentre_xtrack_4s", "label": "0.5 m off centre: distance from the path at 4 s", "unit": "m", "better": "lower", "value": 0.6805, "median": 0.3689, "ci": [ 0.5297, 0.8303 ], "n": 98, "clusters": 13, "paired": { "lebowski_877m": { "delta": 0.01763, "median": -0.0308, "ci": [ -0.06412, 0.1185 ], "n": 98, "wins": 63, "ties": 0, "clusters": 13, "cluster_wins": 7 }, "8d736683_12864": { "delta": 0.02117, "median": -0.01108, "ci": [ -0.03609, 0.09066 ], "n": 98, "wins": 53, "ties": 0, "clusters": 13, "cluster_wins": 6 } } }, { "key": "offcentre_lat_jerk", "label": "0.5 m off centre: requested lateral jerk, RMS", "unit": "m/s³", "better": "lower", "value": 0.5138, "median": 0.4178, "ci": [ 0.4685, 0.5631 ], "n": 98, "clusters": 13, "paired": { "lebowski_877m": { "delta": -0.5966, "median": -0.5779, "ci": [ -0.6685, -0.5152 ], "n": 98, "wins": 95, "ties": 0, "clusters": 13, "cluster_wins": 13 }, "8d736683_12864": { "delta": -0.03775, "median": -0.006996, "ci": [ -0.08369, 0.001866 ], "n": 98, "wins": 57, "ties": 0, "clusters": 13, "cluster_wins": 7 } } }, { "key": "brake_ahead_4s", "label": "Braking lead: distance ahead of the recorded car at 4 s", "unit": "m", "better": "lower", "value": -0.5645, "median": -0.5645, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": 0.773, "median": 0.773, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "8d736683_12864": { "delta": -0.2477, "median": -0.2477, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 1 } } }, { "key": "brake_gap_min", "label": "Braking lead: smallest gap to the lead (derived)", "unit": "m", "better": "higher", "value": 25.7, "median": 25.7, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": -1.756, "median": -1.756, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "8d736683_12864": { "delta": 0.8148, "median": 0.8148, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 1 } } }, { "key": "brake_speed_err", "label": "Braking lead: speed error against the recorded speed, RMS", "unit": "m/s", "better": "lower", "value": 1.26, "median": 1.26, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": 0.05657, "median": 0.05657, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "8d736683_12864": { "delta": 0.012, "median": 0.012, "ci": null, "n": 2, "wins": 0, "ties": 0, "clusters": 1, "cluster_wins": 0 } } } ], "per_unit_file": null, "by_group": { "stock-aimedup": { "n": 14, "clusters": 2, "metrics": [ { "key": "heading_xtrack_2s", "value": 0.416, "median": 0.3894, "ci": null, "n": 14, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.02417, "median": 0.0205, "ci": null, "n": 14, "wins": 7, "ties": 0, "clusters": 2, "cluster_wins": 1 }, "8d736683_12864": { "delta": 0.02015, "median": 0.01647, "ci": null, "n": 14, "wins": 1, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "heading_xtrack_4s", "value": 0.7352, "median": 0.5146, "ci": null, "n": 14, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.1822, "median": 0.1009, "ci": null, "n": 14, "wins": 5, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "8d736683_12864": { "delta": 0.04122, "median": 0.02444, "ci": null, "n": 14, "wins": 6, "ties": 0, "clusters": 2, "cluster_wins": 1 } } }, { "key": "heading_lat_jerk", "value": 0.3974, "median": 0.3202, "ci": null, "n": 14, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.3706, "median": -0.346, "ci": null, "n": 14, "wins": 13, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "8d736683_12864": { "delta": 0.001517, "median": 0.006028, "ci": null, "n": 14, "wins": 6, "ties": 0, "clusters": 2, "cluster_wins": 1 } } }, { "key": "offcentre_xtrack_2s", "value": 0.2928, "median": 0.2621, "ci": null, "n": 14, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.02992, "median": 0.01469, "ci": null, "n": 14, "wins": 5, "ties": 0, "clusters": 2, "cluster_wins": 1 }, "8d736683_12864": { "delta": 0.02652, "median": 0.01829, "ci": null, "n": 14, "wins": 2, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "offcentre_xtrack_4s", "value": 0.8479, "median": 0.6033, "ci": null, "n": 14, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.2581, "median": 0.1157, "ci": null, "n": 14, "wins": 6, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "8d736683_12864": { "delta": 0.1789, "median": 0.08183, "ci": null, "n": 14, "wins": 4, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "offcentre_lat_jerk", "value": 0.642, "median": 0.5456, "ci": null, "n": 14, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.4197, "median": -0.5562, "ci": null, "n": 14, "wins": 12, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "8d736683_12864": { "delta": -0.04949, "median": -0.01209, "ci": null, "n": 14, "wins": 8, "ties": 0, "clusters": 2, "cluster_wins": 1 } } } ] }, "stock-dual": { "n": 45, "clusters": 6, "metrics": [ { "key": "heading_xtrack_2s", "value": 0.4465, "median": 0.3897, "ci": null, "n": 43, "clusters": 5, "paired": { "lebowski_877m": { "delta": 0.03089, "median": 0.007853, "ci": null, "n": 43, "wins": 19, "ties": 0, "clusters": 5, "cluster_wins": 2 }, "8d736683_12864": { "delta": 0.03029, "median": 0.02596, "ci": null, "n": 43, "wins": 1, "ties": 0, "clusters": 5, "cluster_wins": 0 } } }, { "key": "heading_xtrack_4s", "value": 0.6181, "median": 0.432, "ci": null, "n": 43, "clusters": 5, "paired": { "lebowski_877m": { "delta": -0.01812, "median": -0.008539, "ci": null, "n": 43, "wins": 22, "ties": 0, "clusters": 5, "cluster_wins": 3 }, "8d736683_12864": { "delta": 0.01981, "median": 0.01429, "ci": null, "n": 43, "wins": 20, "ties": 0, "clusters": 5, "cluster_wins": 2 } } }, { "key": "heading_lat_jerk", "value": 0.3599, "median": 0.2938, "ci": null, "n": 43, "clusters": 5, "paired": { "lebowski_877m": { "delta": -0.5365, "median": -0.497, "ci": null, "n": 43, "wins": 42, "ties": 0, "clusters": 5, "cluster_wins": 5 }, "8d736683_12864": { "delta": -0.01665, "median": 0.0009469, "ci": null, "n": 43, "wins": 21, "ties": 0, "clusters": 5, "cluster_wins": 3 } } }, { "key": "offcentre_xtrack_2s", "value": 0.2885, "median": 0.2431, "ci": null, "n": 43, "clusters": 5, "paired": { "lebowski_877m": { "delta": -0.01085, "median": 0.003479, "ci": null, "n": 43, "wins": 21, "ties": 0, "clusters": 5, "cluster_wins": 3 }, "8d736683_12864": { "delta": 0.03603, "median": 0.02825, "ci": null, "n": 43, "wins": 3, "ties": 0, "clusters": 5, "cluster_wins": 0 } } }, { "key": "offcentre_xtrack_4s", "value": 0.6189, "median": 0.3561, "ci": null, "n": 43, "clusters": 5, "paired": { "lebowski_877m": { "delta": -0.05155, "median": -0.09528, "ci": null, "n": 43, "wins": 29, "ties": 0, "clusters": 5, "cluster_wins": 4 }, "8d736683_12864": { "delta": 0.002745, "median": -0.02594, "ci": null, "n": 43, "wins": 26, "ties": 0, "clusters": 5, "cluster_wins": 3 } } }, { "key": "offcentre_lat_jerk", "value": 0.5331, "median": 0.4388, "ci": null, "n": 43, "clusters": 5, "paired": { "lebowski_877m": { "delta": -0.6435, "median": -0.6209, "ci": null, "n": 43, "wins": 42, "ties": 0, "clusters": 5, "cluster_wins": 5 }, "8d736683_12864": { "delta": -0.064, "median": -0.02604, "ci": null, "n": 43, "wins": 32, "ties": 0, "clusters": 5, "cluster_wins": 4 } } }, { "key": "brake_ahead_4s", "value": -0.5645, "median": -0.5645, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": 0.773, "median": 0.773, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "8d736683_12864": { "delta": -0.2477, "median": -0.2477, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 1 } } }, { "key": "brake_gap_min", "value": 25.7, "median": 25.7, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": -1.756, "median": -1.756, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "8d736683_12864": { "delta": 0.8148, "median": 0.8148, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 1 } } }, { "key": "brake_speed_err", "value": 1.26, "median": 1.26, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": 0.05657, "median": 0.05657, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "8d736683_12864": { "delta": 0.012, "median": 0.012, "ci": null, "n": 2, "wins": 0, "ties": 0, "clusters": 1, "cluster_wins": 0 } } } ] }, "stock-newcase": { "n": 4, "clusters": 1, "metrics": [ { "key": "heading_xtrack_2s", "value": 0.3793, "median": 0.4012, "ci": null, "n": 4, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.07123, "median": -0.09327, "ci": null, "n": 4, "wins": 3, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "8d736683_12864": { "delta": 0.02169, "median": 0.01798, "ci": null, "n": 4, "wins": 0, "ties": 0, "clusters": 1, "cluster_wins": 0 } } }, { "key": "heading_xtrack_4s", "value": 0.3973, "median": 0.3116, "ci": null, "n": 4, "clusters": 1, "paired": { "lebowski_877m": { "delta": 0.1776, "median": 0.1199, "ci": null, "n": 4, "wins": 0, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "8d736683_12864": { "delta": 0.126, "median": 0.1226, "ci": null, "n": 4, "wins": 0, "ties": 0, "clusters": 1, "cluster_wins": 0 } } }, { "key": "heading_lat_jerk", "value": 0.3287, "median": 0.2193, "ci": null, "n": 4, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.6435, "median": -0.5808, "ci": null, "n": 4, "wins": 4, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "8d736683_12864": { "delta": 0.04153, "median": 0.01379, "ci": null, "n": 4, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 } } }, { "key": "offcentre_xtrack_2s", "value": 0.3192, "median": 0.2353, "ci": null, "n": 4, "clusters": 1, "paired": { "lebowski_877m": { "delta": 0.01937, "median": 0.05442, "ci": null, "n": 4, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "8d736683_12864": { "delta": 0.004527, "median": 0.008633, "ci": null, "n": 4, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 } } }, { "key": "offcentre_xtrack_4s", "value": 0.7586, "median": 0.583, "ci": null, "n": 4, "clusters": 1, "paired": { "lebowski_877m": { "delta": 0.0309, "median": 0.02758, "ci": null, "n": 4, "wins": 2, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "8d736683_12864": { "delta": 0.06026, "median": 0.07037, "ci": null, "n": 4, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 } } }, { "key": "offcentre_lat_jerk", "value": 0.6375, "median": 0.4072, "ci": null, "n": 4, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.7598, "median": -0.674, "ci": null, "n": 4, "wins": 4, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "8d736683_12864": { "delta": -0.03029, "median": -0.0242, "ci": null, "n": 4, "wins": 3, "ties": 0, "clusters": 1, "cluster_wins": 1 } } } ] }, "v13-8mm": { "n": 18, "clusters": 3, "metrics": [ { "key": "heading_xtrack_2s", "value": 0.8848, "median": 0.8342, "ci": null, "n": 18, "clusters": 3, "paired": { "lebowski_877m": { "delta": 0.1918, "median": 0.2195, "ci": null, "n": 18, "wins": 2, "ties": 0, "clusters": 3, "cluster_wins": 0 }, "8d736683_12864": { "delta": 0.02178, "median": 0.01941, "ci": null, "n": 18, "wins": 4, "ties": 0, "clusters": 3, "cluster_wins": 0 } } }, { "key": "heading_xtrack_4s", "value": 1.311, "median": 0.8645, "ci": null, "n": 18, "clusters": 3, "paired": { "lebowski_877m": { "delta": 0.07941, "median": 0.09476, "ci": null, "n": 18, "wins": 7, "ties": 0, "clusters": 3, "cluster_wins": 1 }, "8d736683_12864": { "delta": 0.1361, "median": 0.005608, "ci": null, "n": 18, "wins": 8, "ties": 0, "clusters": 3, "cluster_wins": 2 } } }, { "key": "heading_lat_jerk", "value": 0.4465, "median": 0.4019, "ci": null, "n": 18, "clusters": 3, "paired": { "lebowski_877m": { "delta": -0.4265, "median": -0.4254, "ci": null, "n": 18, "wins": 17, "ties": 0, "clusters": 3, "cluster_wins": 3 }, "8d736683_12864": { "delta": 0.06697, "median": 0.03407, "ci": null, "n": 18, "wins": 4, "ties": 0, "clusters": 3, "cluster_wins": 0 } } }, { "key": "offcentre_xtrack_2s", "value": 0.2933, "median": 0.2166, "ci": null, "n": 18, "clusters": 3, "paired": { "lebowski_877m": { "delta": 0.03525, "median": 0.05228, "ci": null, "n": 18, "wins": 5, "ties": 0, "clusters": 3, "cluster_wins": 0 }, "8d736683_12864": { "delta": 0.01411, "median": 0.01551, "ci": null, "n": 18, "wins": 6, "ties": 0, "clusters": 3, "cluster_wins": 0 } } }, { "key": "offcentre_xtrack_4s", "value": 0.5903, "median": 0.2272, "ci": null, "n": 18, "clusters": 3, "paired": { "lebowski_877m": { "delta": -0.1027, "median": -0.03924, "ci": null, "n": 18, "wins": 12, "ties": 0, "clusters": 3, "cluster_wins": 2 }, "8d736683_12864": { "delta": 0.006634, "median": 0.0006841, "ci": null, "n": 18, "wins": 9, "ties": 0, "clusters": 3, "cluster_wins": 1 } } }, { "key": "offcentre_lat_jerk", "value": 0.4353, "median": 0.3818, "ci": null, "n": 18, "clusters": 3, "paired": { "lebowski_877m": { "delta": -0.4465, "median": -0.4413, "ci": null, "n": 18, "wins": 18, "ties": 0, "clusters": 3, "cluster_wins": 3 }, "8d736683_12864": { "delta": 0.01787, "median": 0.009646, "ci": null, "n": 18, "wins": 6, "ties": 0, "clusters": 3, "cluster_wins": 0 } } } ] }, "v14-8mm": { "n": 19, "clusters": 2, "metrics": [ { "key": "heading_xtrack_2s", "value": 0.4339, "median": 0.4324, "ci": null, "n": 19, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.07843, "median": 0.08264, "ci": null, "n": 19, "wins": 3, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "8d736683_12864": { "delta": 0.02479, "median": 0.02184, "ci": null, "n": 19, "wins": 2, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "heading_xtrack_4s", "value": 0.801, "median": 0.3372, "ci": null, "n": 19, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.2375, "median": 0.09678, "ci": null, "n": 19, "wins": 4, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "8d736683_12864": { "delta": -0.02135, "median": -0.02843, "ci": null, "n": 19, "wins": 13, "ties": 0, "clusters": 2, "cluster_wins": 2 } } }, { "key": "heading_lat_jerk", "value": 0.3152, "median": 0.2683, "ci": null, "n": 19, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.6333, "median": -0.533, "ci": null, "n": 19, "wins": 19, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "8d736683_12864": { "delta": 0.01039, "median": 0.009745, "ci": null, "n": 19, "wins": 7, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "offcentre_xtrack_2s", "value": 0.2561, "median": 0.2247, "ci": null, "n": 19, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.05766, "median": 0.08654, "ci": null, "n": 19, "wins": 2, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "8d736683_12864": { "delta": 0.02525, "median": 0.02311, "ci": null, "n": 19, "wins": 2, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "offcentre_xtrack_4s", "value": 0.7656, "median": 0.3839, "ci": null, "n": 19, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.1082, "median": -0.02583, "ci": null, "n": 19, "wins": 14, "ties": 0, "clusters": 2, "cluster_wins": 1 }, "8d736683_12864": { "delta": -0.04778, "median": -0.05867, "ci": null, "n": 19, "wins": 13, "ties": 0, "clusters": 2, "cluster_wins": 2 } } }, { "key": "offcentre_lat_jerk", "value": 0.424, "median": 0.419, "ci": null, "n": 19, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.7285, "median": -0.6974, "ci": null, "n": 19, "wins": 19, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "8d736683_12864": { "delta": -0.02397, "median": 0.008684, "ci": null, "n": 19, "wins": 8, "ties": 0, "clusters": 2, "cluster_wins": 1 } } } ] } }, "simulator": "g55-2026-10-31" } }, "stills": [], "limits": [ "Open-loop scores compare a model's action with what the recording driver or car did; a model that drives differently but well is scored as wrong. comma's RL residual makes the lateral action smoother, and earlier than its base's by 0.10 to 0.26 s (measured on the G55 and on 16 of comma's cars); both score as error here. Measured on the 2026-10-27 deep dive: open-loop accuracy did not predict closed-loop driving.", "Closed-loop numbers come from a generative world model, not a car. In a separate test on the G55's windows, worldmodel-4B followed an added yaw command through 2 s but showed 0 to 20% of an added lateral offset 3 to 4 s later; past about 4 s the recorded anchor frames also pull every model back toward the recorded path. The front page reports 2 s.", "Intervals are 95% percentile bootstraps that resample whole drives (the drive a public segment was cut from, the recorded drive a truck window comes from), so segments of one drive count once. Below 10 drives there is no interval: the braking-lead test comes from one drive and reports means and win counts only.", "Paired deltas are model minus baseline on the units both have (for lateral gain, the change in its distance from 1); wins count units where the model is better in the metric's direction.", "The lead of the lateral request is the shift, searched from 0 to 1.5 s, at which the request best correlates with the car's later lateral motion; a larger value means the request comes earlier. A value at either edge is censored and left out. Plan speed error at 3 s reads the plan head, which does not separate models built on one frozen base: it is identical on every segment for the four f22e06e5 models, and for Cinque v2 and v3.", "Truck open loop comes from few drives, and the camera setup changed between them (lens, housing, aim); the pooled number mixes setups, so read the per-setup numbers too. The 8 mm setups' wide camera recorded about 7-10 fps, so its frame can be up to ~60 ms from the narrow one. On the V14 setup up to 7 of a window's 700 steps are ~100 ms (one camera frame dropped). The camera's aim is the truck's live calibration learned on each drive; drives whose recorded calibration was carried over from another drive are left out.", "Truck closed loop: the steering and powertrain simulation is fitted to the G55, not validated on it in closed loop; the reference is the driver's own line, not the lane centre. The pooled number mixes camera setups, so read the per-setup numbers too. On the V14 setup the wide frame can be up to 90 ms from the simulated one (its camera recorded 7-9 fps).", "Recovery tests are disturbances rendered by the world model (yaw it renders; a one-frame jump it does not, so none is used). The braking-lead test has 2 windows from one drive, whose recorded calibration was carried over from an earlier drive." ] }