{ "schema": 1, "model": { "id": "96244b1f_12800", "name": "N16 nav model", "sha256": "01effc679eb46d6036ce0302673a87fe2dcfcc95a679daeeb929679e53c3647a", "source": { "kind": "huggingface", "ref": "3fe6f3dea4fa9613dd3582b9a3828e97d307ad9e", "url": "https://huggingface.co/commaai/openpilot_driving_models/blob/3fe6f3dea4fa9613dd3582b9a3828e97d307ad9e/96244b1f-6c4e-4adc-848d-eccd0b44b182/12800/big_driving_supercombo.onnx", "date": "2026-10-15" }, "first_public": "2026-10-15", "on_master": null, "params_m": 393.67, "gmac": 51.9, "family": "residual", "notes": "Uploaded as \"add N16 nav model\"; never shipped. Takes a navigation map and route inputs, fed zeros here (no route). Vision base 2931d3d9 with an RL residual head." }, "scored": { "date": "2026-10-31T23:59:43Z", "suite": "geval-1", "code": "d8eb40eeba87" }, "baselines": [ "lebowski_877m", "1365126a_12864" ], "sections": { "public_open_loop": { "status": "ok", "unit_of_evidence": "segment", "n": 39, "clusters": 23, "condition": "Measured. comma's public two-camera segments (openpilot CI data), native intrinsics and the device's own calibration, both cameras; every model on the same decoded frames. Truth is what the recording car did next: carState speed and acceleration, yaw rate from the device's cameraOdometry. Segments where the car never moves have no data. 39 of the 45 listed segments have a score, from 23 drives and 20 cars.", "baselines": [ "lebowski_877m", "1365126a_12864" ], "metrics": [ { "key": "lat_mae", "label": "Lateral error", "unit": "m/s²", "better": "lower", "value": 0.1519, "median": 0.1341, "ci": [ 0.1195, 0.1853 ], "n": 39, "clusters": 23, "paired": { "lebowski_877m": { "delta": 0.04319, "median": 0.03024, "ci": [ 0.02943, 0.05641 ], "n": 39, "wins": 3, "ties": 0, "clusters": 23, "cluster_wins": 0 }, "1365126a_12864": { "delta": 0.02311, "median": 0.02055, "ci": [ 0.01317, 0.03306 ], "n": 39, "wins": 5, "ties": 0, "clusters": 23, "cluster_wins": 2 } } }, { "key": "lat_mae_bl", "label": "Lateral error at the model's own best lead", "unit": "m/s²", "better": "lower", "value": 0.1419, "median": 0.1298, "ci": [ 0.1099, 0.1754 ], "n": 39, "clusters": 23, "paired": { "lebowski_877m": { "delta": 0.04045, "median": 0.02249, "ci": [ 0.02482, 0.05574 ], "n": 39, "wins": 8, "ties": 0, "clusters": 23, "cluster_wins": 4 }, "1365126a_12864": { "delta": 0.02248, "median": 0.01325, "ci": [ 0.01213, 0.03308 ], "n": 39, "wins": 6, "ties": 0, "clusters": 23, "cluster_wins": 3 } } }, { "key": "lat_corr", "label": "Lateral correlation with the car's later motion", "unit": "", "better": "higher", "value": 0.8277, "median": 0.9202, "ci": [ 0.7457, 0.8971 ], "n": 39, "clusters": 23, "paired": { "lebowski_877m": { "delta": -0.02992, "median": -0.02039, "ci": [ -0.07169, 0.005037 ], "n": 39, "wins": 9, "ties": 0, "clusters": 23, "cluster_wins": 7 }, "1365126a_12864": { "delta": -0.01899, "median": -0.01371, "ci": [ -0.03721, -0.004165 ], "n": 39, "wins": 6, "ties": 0, "clusters": 23, "cluster_wins": 5 } } }, { "key": "lat_gain", "label": "Lateral gain (1 = same size as the car's motion)", "unit": "", "better": "near1", "value": 0.7337, "median": 0.7214, "ci": [ 0.6437, 0.8274 ], "n": 39, "clusters": 23, "paired": { "lebowski_877m": { "delta": 0.09198, "median": 0.05853, "ci": [ 0.01007, 0.1668 ], "n": 39, "wins": 11, "ties": 0, "clusters": 23, "cluster_wins": 8 }, "1365126a_12864": { "delta": 0.02767, "median": 0.03633, "ci": [ -0.02422, 0.07631 ], "n": 39, "wins": 9, "ties": 0, "clusters": 23, "cluster_wins": 6 } } }, { "key": "lat_rate_rms", "label": "Lateral action rate, RMS (smoothness)", "unit": "m/s³", "better": "lower", "value": 0.4711, "median": 0.4217, "ci": [ 0.3799, 0.5762 ], "n": 39, "clusters": 23, "paired": { "lebowski_877m": { "delta": -0.1928, "median": -0.1776, "ci": [ -0.2352, -0.1455 ], "n": 39, "wins": 37, "ties": 0, "clusters": 23, "cluster_wins": 23 }, "1365126a_12864": { "delta": 0.01968, "median": 0.02603, "ci": [ -0.007571, 0.03907 ], "n": 39, "wins": 6, "ties": 0, "clusters": 23, "cluster_wins": 4 } } }, { "key": "lat_best_lag", "label": "Lead of the lateral request over the car's motion", "unit": "s", "better": "lower", "value": 0.5386, "median": 0.5, "ci": [ 0.4289, 0.6596 ], "n": 35, "clusters": 22, "paired": { "lebowski_877m": { "delta": 0.1544, "median": 0.125, "ci": [ 0.08485, 0.2396 ], "n": 34, "wins": 4, "ties": 4, "clusters": 22, "cluster_wins": 1 }, "1365126a_12864": { "delta": 0.06857, "median": 0.05, "ci": [ 0.03444, 0.113 ], "n": 35, "wins": 3, "ties": 10, "clusters": 22, "cluster_wins": 1 } } }, { "key": "lon_mae", "label": "Longitudinal error", "unit": "m/s²", "better": "lower", "value": 0.3167, "median": 0.2983, "ci": [ 0.2742, 0.366 ], "n": 39, "clusters": 23, "paired": { "lebowski_877m": { "delta": 0.008513, "median": 0.005901, "ci": [ -0.008462, 0.02783 ], "n": 39, "wins": 14, "ties": 0, "clusters": 23, "cluster_wins": 8 }, "1365126a_12864": { "delta": 0.01594, "median": 0.01219, "ci": [ 0.004487, 0.02746 ], "n": 39, "wins": 14, "ties": 0, "clusters": 23, "cluster_wins": 9 } } }, { "key": "lon_corr", "label": "Longitudinal correlation", "unit": "", "better": "higher", "value": 0.7287, "median": 0.8474, "ci": [ 0.6257, 0.8173 ], "n": 39, "clusters": 23, "paired": { "lebowski_877m": { "delta": 0.01099, "median": 0.008126, "ci": [ -0.01493, 0.03581 ], "n": 39, "wins": 23, "ties": 0, "clusters": 23, "cluster_wins": 14 }, "1365126a_12864": { "delta": -0.009947, "median": -0.009404, "ci": [ -0.02125, 0.003718 ], "n": 39, "wins": 13, "ties": 0, "clusters": 23, "cluster_wins": 7 } } }, { "key": "lon_brake_mae", "label": "Longitudinal error while braking", "unit": "m/s²", "better": "lower", "value": 0.4363, "median": 0.4287, "ci": [ 0.3729, 0.5087 ], "n": 30, "clusters": 21, "paired": { "lebowski_877m": { "delta": 0.001968, "median": 0.02602, "ci": [ -0.02514, 0.03511 ], "n": 30, "wins": 12, "ties": 0, "clusters": 21, "cluster_wins": 9 }, "1365126a_12864": { "delta": 0.03573, "median": 0.04055, "ci": [ 0.01322, 0.06181 ], "n": 30, "wins": 7, "ties": 0, "clusters": 21, "cluster_wins": 4 } } }, { "key": "lon_rate_rms", "label": "Longitudinal action rate, RMS (smoothness)", "unit": "m/s³", "better": "lower", "value": 0.9005, "median": 0.8208, "ci": [ 0.7969, 1.014 ], "n": 39, "clusters": 23, "paired": { "lebowski_877m": { "delta": -0.4325, "median": -0.3706, "ci": [ -0.5612, -0.3217 ], "n": 39, "wins": 36, "ties": 0, "clusters": 23, "cluster_wins": 22 }, "1365126a_12864": { "delta": 0.01069, "median": 0.01384, "ci": [ -0.02498, 0.0431 ], "n": 39, "wins": 13, "ties": 0, "clusters": 23, "cluster_wins": 7 } } }, { "key": "v3_mae", "label": "Plan speed error at 3 s", "unit": "m/s", "better": "lower", "value": 0.7667, "median": 0.7474, "ci": [ 0.6565, 0.9018 ], "n": 39, "clusters": 23, "paired": { "lebowski_877m": { "delta": 0.05091, "median": 0.04574, "ci": [ 0.01517, 0.08501 ], "n": 39, "wins": 13, "ties": 0, "clusters": 23, "cluster_wins": 7 }, "1365126a_12864": { "delta": 0.006492, "median": 0.005619, "ci": [ -0.01238, 0.02415 ], "n": 39, "wins": 18, "ties": 0, "clusters": 23, "cluster_wins": 10 } } } ], "per_unit_file": "per_unit/96244b1f_12800/public_open_loop.csv" }, "public_closed_loop": { "status": "ok", "unit_of_evidence": "segment", "n": 28, "clusters": 19, "condition": "Measured. comma's worldmodel-4B generates each next frame at 5 Hz from 9 history frames, 5 recorded anchor frames and the car's pose; the model drives comma's ideal car (the requested curvature and acceleration happen at once) for 7.2 s from the recorded start. Distance is to the recording car's own path. One window per segment, nearest the middle, where the car holds at least 5 m/s for the window and the 4.8 s before it; segments with no such stretch are left out. Seeds 0 and 1, averaged per segment. 28 of the 45 listed segments have a score, from 19 drives and 17 cars.", "baselines": [ "lebowski_877m", "1365126a_12864" ], "metrics": [ { "key": "xtrack_2s", "label": "Distance from the recorded path at 2 s", "unit": "m", "better": "lower", "value": 0.08837, "median": 0.05381, "ci": [ 0.04714, 0.1595 ], "n": 28, "clusters": 19, "paired": { "lebowski_877m": { "delta": -0.05932, "median": -0.01924, "ci": [ -0.1236, -0.01545 ], "n": 28, "wins": 22, "ties": 0, "clusters": 19, "cluster_wins": 18 }, "1365126a_12864": { "delta": 0.001077, "median": -0.01035, "ci": [ -0.01874, 0.02818 ], "n": 28, "wins": 18, "ties": 0, "clusters": 19, "cluster_wins": 12 } } }, { "key": "xtrack_4s", "label": "Distance from the recorded path at 4 s", "unit": "m", "better": "lower", "value": 0.1702, "median": 0.09467, "ci": [ 0.1029, 0.2549 ], "n": 28, "clusters": 19, "paired": { "lebowski_877m": { "delta": -0.108, "median": -0.003776, "ci": [ -0.271, -0.004106 ], "n": 28, "wins": 14, "ties": 0, "clusters": 19, "cluster_wins": 9 }, "1365126a_12864": { "delta": 9.161e-05, "median": -0.01572, "ci": [ -0.03576, 0.04611 ], "n": 28, "wins": 16, "ties": 0, "clusters": 19, "cluster_wins": 11 } } }, { "key": "xtrack_7s", "label": "Distance from the recorded path at 7.2 s", "unit": "m", "better": "lower", "value": 1.129, "median": 0.2739, "ci": [ 0.3285, 2.275 ], "n": 28, "clusters": 19, "paired": { "lebowski_877m": { "delta": 0.08475, "median": -0.02438, "ci": [ -0.5086, 0.891 ], "n": 28, "wins": 15, "ties": 0, "clusters": 19, "cluster_wins": 12 }, "1365126a_12864": { "delta": 0.13, "median": -0.02175, "ci": [ -0.0804, 0.4371 ], "n": 28, "wins": 16, "ties": 0, "clusters": 19, "cluster_wins": 12 } } }, { "key": "fail_1m", "label": "Share of episodes ever more than 1 m off the path", "unit": "", "better": "lower", "value": 0.2321, "median": 0.0, "ci": [ 0.08065, 0.4062 ], "n": 28, "clusters": 19, "paired": { "lebowski_877m": { "delta": 0.0, "median": 0.0, "ci": [ -0.05362, 0.05357 ], "n": 28, "wins": 1, "ties": 26, "clusters": 19, "cluster_wins": 1 }, "1365126a_12864": { "delta": 0.08929, "median": 0.0, "ci": [ 0.0, 0.2083 ], "n": 28, "wins": 0, "ties": 25, "clusters": 19, "cluster_wins": 0 } } }, { "key": "lat_jerk", "label": "Requested lateral jerk, RMS", "unit": "m/s³", "better": "lower", "value": 0.205, "median": 0.1286, "ci": [ 0.1422, 0.2754 ], "n": 28, "clusters": 19, "paired": { "lebowski_877m": { "delta": -0.1206, "median": -0.08962, "ci": [ -0.1684, -0.07085 ], "n": 28, "wins": 25, "ties": 0, "clusters": 19, "cluster_wins": 17 }, "1365126a_12864": { "delta": 0.004135, "median": 0.006992, "ci": [ -0.02477, 0.03123 ], "n": 28, "wins": 10, "ties": 0, "clusters": 19, "cluster_wins": 7 } } }, { "key": "long_jerk", "label": "Requested longitudinal jerk, RMS", "unit": "m/s³", "better": "lower", "value": 0.3904, "median": 0.3332, "ci": [ 0.292, 0.504 ], "n": 28, "clusters": 19, "paired": { "lebowski_877m": { "delta": 0.04405, "median": -0.004609, "ci": [ -0.006412, 0.102 ], "n": 28, "wins": 14, "ties": 0, "clusters": 19, "cluster_wins": 9 }, "1365126a_12864": { "delta": 0.03409, "median": 0.004636, "ci": [ -0.002911, 0.07827 ], "n": 28, "wins": 12, "ties": 0, "clusters": 19, "cluster_wins": 9 } } }, { "key": "speed_err", "label": "Speed error against the recorded speed, RMS", "unit": "m/s", "better": "lower", "value": 1.402, "median": 1.101, "ci": [ 0.9945, 1.855 ], "n": 28, "clusters": 19, "paired": { "lebowski_877m": { "delta": 0.3003, "median": 0.2428, "ci": [ 0.0483, 0.5662 ], "n": 28, "wins": 7, "ties": 0, "clusters": 19, "cluster_wins": 4 }, "1365126a_12864": { "delta": 0.226, "median": 0.2687, "ci": [ 0.07779, 0.367 ], "n": 28, "wins": 6, "ties": 0, "clusters": 19, "cluster_wins": 4 } } } ], "per_unit_file": "per_unit/96244b1f_12800/public_closed_loop.csv" }, "truck_open_loop": { "status": "ok", "unit_of_evidence": "window", "n": 119, "clusters": 12, "condition": "Measured. The G55's recorded drives replayed through the model with both of the truck's cameras (each setup's own wide-camera calibration), road-plane warp to 1.45 m. Where the narrow lens does not cover comma's narrow view (the 8 mm lens in the V13 case), those pixels come from the wide camera, brightness matched. Truth is the truck's CAN speed and yaw rate. 30 s windows after a 5 s warm-up; window = unit, drive = cluster; results are also given per camera setup. Corpus windows of 30 s from the truck's recorded drives, scored per camera setup.", "baselines": [ "lebowski_877m", "1365126a_12864" ], "metrics": [ { "key": "lat_mae", "label": "Lateral error", "unit": "m/s²", "better": "lower", "value": 0.1707, "median": 0.1579, "ci": [ 0.1445, 0.1831 ], "n": 119, "clusters": 12, "paired": { "lebowski_877m": { "delta": -0.002087, "median": -0.002345, "ci": [ -0.0116, 0.01585 ], "n": 119, "wins": 65, "ties": 0, "clusters": 12, "cluster_wins": 4 }, "1365126a_12864": { "delta": 0.01474, "median": 0.01225, "ci": [ 0.01149, 0.02508 ], "n": 119, "wins": 24, "ties": 0, "clusters": 12, "cluster_wins": 0 } } }, { "key": "lat_mae_bl", "label": "Lateral error at the model's own best lead", "unit": "m/s²", "better": "lower", "value": 0.1499, "median": 0.1341, "ci": [ 0.1281, 0.1598 ], "n": 119, "clusters": 12, "paired": { "lebowski_877m": { "delta": -0.01462, "median": -0.01615, "ci": [ -0.02151, 0.004281 ], "n": 119, "wins": 88, "ties": 0, "clusters": 12, "cluster_wins": 6 }, "1365126a_12864": { "delta": 0.006848, "median": 0.005181, "ci": [ 0.003612, 0.01843 ], "n": 119, "wins": 39, "ties": 0, "clusters": 12, "cluster_wins": 1 } } }, { "key": "lat_corr", "label": "Lateral correlation with the car's later motion", "unit": "", "better": "higher", "value": 0.7968, "median": 0.8921, "ci": [ 0.7543, 0.8178 ], "n": 119, "clusters": 12, "paired": { "lebowski_877m": { "delta": 0.02997, "median": 0.01074, "ci": [ -0.01583, 0.04166 ], "n": 119, "wins": 77, "ties": 0, "clusters": 12, "cluster_wins": 5 }, "1365126a_12864": { "delta": -0.02283, "median": -0.01527, "ci": [ -0.03946, -0.01827 ], "n": 119, "wins": 15, "ties": 0, "clusters": 12, "cluster_wins": 0 } } }, { "key": "lat_gain", "label": "Lateral gain (1 = same size as the car's motion)", "unit": "", "better": "near1", "value": 0.7752, "median": 0.7876, "ci": [ 0.7502, 0.849 ], "n": 119, "clusters": 12, "paired": { "lebowski_877m": { "delta": 0.02267, "median": 0.02996, "ci": [ 0.01228, 0.0537 ], "n": 119, "wins": 48, "ties": 0, "clusters": 12, "cluster_wins": 2 }, "1365126a_12864": { "delta": 0.02224, "median": 0.02154, "ci": [ -0.0008981, 0.05183 ], "n": 119, "wins": 44, "ties": 0, "clusters": 12, "cluster_wins": 2 } } }, { "key": "lat_rate_rms", "label": "Lateral action rate, RMS (smoothness)", "unit": "m/s³", "better": "lower", "value": 1.084, "median": 0.8208, "ci": [ 0.643, 1.307 ], "n": 119, "clusters": 12, "paired": { "lebowski_877m": { "delta": -1.207, "median": -1.172, "ci": [ -1.351, -0.4559 ], "n": 119, "wins": 118, "ties": 0, "clusters": 12, "cluster_wins": 12 }, "1365126a_12864": { "delta": -0.03632, "median": -0.002605, "ci": [ -0.06707, 0.0816 ], "n": 119, "wins": 63, "ties": 0, "clusters": 12, "cluster_wins": 3 } } }, { "key": "lat_best_lag", "label": "Lead of the lateral request over the car's motion", "unit": "s", "better": "lower", "value": 0.7513, "median": 0.7, "ci": [ 0.7029, 0.7883 ], "n": 118, "clusters": 12, "paired": { "lebowski_877m": { "delta": 0.1347, "median": 0.15, "ci": [ 0.06977, 0.2375 ], "n": 118, "wins": 20, "ties": 10, "clusters": 12, "cluster_wins": 0 }, "1365126a_12864": { "delta": 0.1102, "median": 0.1, "ci": [ 0.09753, 0.1373 ], "n": 118, "wins": 3, "ties": 17, "clusters": 12, "cluster_wins": 0 } } }, { "key": "lon_mae", "label": "Longitudinal error", "unit": "m/s²", "better": "lower", "value": 0.3764, "median": 0.3814, "ci": [ 0.2743, 0.4302 ], "n": 119, "clusters": 12, "paired": { "lebowski_877m": { "delta": 0.007694, "median": 0.0221, "ci": [ -0.06076, 0.02707 ], "n": 119, "wins": 43, "ties": 0, "clusters": 12, "cluster_wins": 4 }, "1365126a_12864": { "delta": 0.02647, "median": 0.02967, "ci": [ -0.0089, 0.03619 ], "n": 119, "wins": 29, "ties": 0, "clusters": 12, "cluster_wins": 7 } } }, { "key": "lon_corr", "label": "Longitudinal correlation", "unit": "", "better": "higher", "value": 0.301, "median": 0.2324, "ci": [ 0.2359, 0.6172 ], "n": 119, "clusters": 12, "paired": { "lebowski_877m": { "delta": -0.02588, "median": -0.01291, "ci": [ -0.04006, 0.01408 ], "n": 119, "wins": 54, "ties": 0, "clusters": 12, "cluster_wins": 5 }, "1365126a_12864": { "delta": -0.0332, "median": -0.02835, "ci": [ -0.04663, 0.0302 ], "n": 119, "wins": 41, "ties": 0, "clusters": 12, "cluster_wins": 6 } } }, { "key": "lon_brake_mae", "label": "Longitudinal error while braking", "unit": "m/s²", "better": "lower", "value": 0.4643, "median": 0.4411, "ci": [ 0.433, 0.6138 ], "n": 81, "clusters": 12, "paired": { "lebowski_877m": { "delta": -0.03453, "median": -0.02718, "ci": [ -0.06908, 0.06995 ], "n": 81, "wins": 50, "ties": 0, "clusters": 12, "cluster_wins": 6 }, "1365126a_12864": { "delta": -0.0152, "median": -0.02028, "ci": [ -0.05614, 0.007892 ], "n": 81, "wins": 51, "ties": 0, "clusters": 12, "cluster_wins": 8 } } }, { "key": "lon_rate_rms", "label": "Longitudinal action rate, RMS (smoothness)", "unit": "m/s³", "better": "lower", "value": 5.571, "median": 5.563, "ci": [ 2.49, 7.012 ], "n": 119, "clusters": 12, "paired": { "lebowski_877m": { "delta": -0.03909, "median": -0.598, "ci": [ -0.8042, 0.2982 ], "n": 119, "wins": 73, "ties": 0, "clusters": 12, "cluster_wins": 9 }, "1365126a_12864": { "delta": 1.165, "median": 1.16, "ci": [ 0.1513, 1.42 ], "n": 119, "wins": 27, "ties": 0, "clusters": 12, "cluster_wins": 4 } } }, { "key": "v3_mae", "label": "Plan speed error at 3 s", "unit": "m/s", "better": "lower", "value": 1.91, "median": 2.0, "ci": [ 1.059, 2.095 ], "n": 119, "clusters": 12, "paired": { "lebowski_877m": { "delta": -0.08764, "median": -0.05556, "ci": [ -0.2032, 0.002216 ], "n": 119, "wins": 72, "ties": 0, "clusters": 12, "cluster_wins": 9 }, "1365126a_12864": { "delta": 0.021, "median": 0.04143, "ci": [ -0.4093, 0.2468 ], "n": 119, "wins": 55, "ties": 0, "clusters": 12, "cluster_wins": 9 } } } ], "per_unit_file": null, "by_group": { "stock-aimedup": { "n": 4, "clusters": 2, "metrics": [ { "key": "lat_mae", "value": 0.1362, "median": 0.1354, "ci": null, "n": 4, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.004892, "median": -0.003977, "ci": null, "n": 4, "wins": 2, "ties": 0, "clusters": 2, "cluster_wins": 1 }, "1365126a_12864": { "delta": 0.01745, "median": 0.01691, "ci": null, "n": 4, "wins": 0, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "lat_mae_bl", "value": 0.1137, "median": 0.1106, "ci": null, "n": 4, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.02592, "median": -0.03134, "ci": null, "n": 4, "wins": 4, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "1365126a_12864": { "delta": 0.005207, "median": 0.004496, "ci": null, "n": 4, "wins": 1, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "lat_corr", "value": 0.7718, "median": 0.8711, "ci": null, "n": 4, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.02336, "median": -0.03337, "ci": null, "n": 4, "wins": 1, "ties": 0, "clusters": 2, "cluster_wins": 1 }, "1365126a_12864": { "delta": -0.06046, "median": -0.05438, "ci": null, "n": 4, "wins": 0, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "lat_gain", "value": 0.6789, "median": 0.7259, "ci": null, "n": 4, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.01501, "median": 0.03732, "ci": null, "n": 4, "wins": 1, "ties": 0, "clusters": 2, "cluster_wins": 1 }, "1365126a_12864": { "delta": 0.1565, "median": 0.1767, "ci": null, "n": 4, "wins": 0, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "lat_rate_rms", "value": 0.7934, "median": 0.7955, "ci": null, "n": 4, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.3596, "median": -0.3999, "ci": null, "n": 4, "wins": 4, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "1365126a_12864": { "delta": 0.08968, "median": 0.07476, "ci": null, "n": 4, "wins": 1, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "lat_best_lag", "value": 0.8625, "median": 0.85, "ci": null, "n": 4, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.35, "median": 0.325, "ci": null, "n": 4, "wins": 0, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "1365126a_12864": { "delta": 0.175, "median": 0.2, "ci": null, "n": 4, "wins": 0, "ties": 1, "clusters": 2, "cluster_wins": 0 } } }, { "key": "lon_mae", "value": 0.3858, "median": 0.3735, "ci": null, "n": 4, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.3918, "median": -0.4105, "ci": null, "n": 4, "wins": 4, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "1365126a_12864": { "delta": -0.01847, "median": -0.03187, "ci": null, "n": 4, "wins": 2, "ties": 0, "clusters": 2, "cluster_wins": 1 } } }, { "key": "lon_corr", "value": 0.6746, "median": 0.7939, "ci": null, "n": 4, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.1392, "median": 0.07567, "ci": null, "n": 4, "wins": 4, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "1365126a_12864": { "delta": 0.1608, "median": 0.07069, "ci": null, "n": 4, "wins": 3, "ties": 0, "clusters": 2, "cluster_wins": 1 } } }, { "key": "lon_brake_mae", "value": 0.5249, "median": 0.5249, "ci": null, "n": 2, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.3369, "median": 0.3369, "ci": null, "n": 2, "wins": 0, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "1365126a_12864": { "delta": 0.01215, "median": 0.01215, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 2, "cluster_wins": 1 } } }, { "key": "lon_rate_rms", "value": 1.705, "median": 1.629, "ci": null, "n": 4, "clusters": 2, "paired": { "lebowski_877m": { "delta": -1.589, "median": -1.631, "ci": null, "n": 4, "wins": 4, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "1365126a_12864": { "delta": -0.2237, "median": -0.2235, "ci": null, "n": 4, "wins": 4, "ties": 0, "clusters": 2, "cluster_wins": 2 } } }, { "key": "v3_mae", "value": 1.475, "median": 1.299, "ci": null, "n": 4, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.9138, "median": -0.8763, "ci": null, "n": 4, "wins": 4, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "1365126a_12864": { "delta": 0.03193, "median": -0.03439, "ci": null, "n": 4, "wins": 2, "ties": 0, "clusters": 2, "cluster_wins": 1 } } } ] }, "stock-dual": { "n": 13, "clusters": 4, "metrics": [ { "key": "lat_mae", "value": 0.1627, "median": 0.1473, "ci": null, "n": 13, "clusters": 4, "paired": { "lebowski_877m": { "delta": 0.01748, "median": 0.01485, "ci": null, "n": 13, "wins": 3, "ties": 0, "clusters": 4, "cluster_wins": 0 }, "1365126a_12864": { "delta": 0.03062, "median": 0.02795, "ci": null, "n": 13, "wins": 1, "ties": 0, "clusters": 4, "cluster_wins": 0 } } }, { "key": "lat_mae_bl", "value": 0.1461, "median": 0.1485, "ci": null, "n": 13, "clusters": 4, "paired": { "lebowski_877m": { "delta": 0.0068, "median": 0.004647, "ci": null, "n": 13, "wins": 4, "ties": 0, "clusters": 4, "cluster_wins": 1 }, "1365126a_12864": { "delta": 0.02549, "median": 0.02359, "ci": null, "n": 13, "wins": 0, "ties": 0, "clusters": 4, "cluster_wins": 0 } } }, { "key": "lat_corr", "value": 0.8028, "median": 0.8205, "ci": null, "n": 13, "clusters": 4, "paired": { "lebowski_877m": { "delta": -0.01053, "median": -0.02186, "ci": null, "n": 13, "wins": 5, "ties": 0, "clusters": 4, "cluster_wins": 1 }, "1365126a_12864": { "delta": -0.04135, "median": -0.03562, "ci": null, "n": 13, "wins": 0, "ties": 0, "clusters": 4, "cluster_wins": 0 } } }, { "key": "lat_gain", "value": 0.8797, "median": 0.846, "ci": null, "n": 13, "clusters": 4, "paired": { "lebowski_877m": { "delta": 0.06125, "median": 0.04983, "ci": null, "n": 13, "wins": 2, "ties": 0, "clusters": 4, "cluster_wins": 1 }, "1365126a_12864": { "delta": 0.03538, "median": 0.05375, "ci": null, "n": 13, "wins": 4, "ties": 0, "clusters": 4, "cluster_wins": 0 } } }, { "key": "lat_rate_rms", "value": 0.7482, "median": 0.6656, "ci": null, "n": 13, "clusters": 4, "paired": { "lebowski_877m": { "delta": -0.4277, "median": -0.4543, "ci": null, "n": 13, "wins": 13, "ties": 0, "clusters": 4, "cluster_wins": 4 }, "1365126a_12864": { "delta": 0.1328, "median": 0.1425, "ci": null, "n": 13, "wins": 1, "ties": 0, "clusters": 4, "cluster_wins": 0 } } }, { "key": "lat_best_lag", "value": 0.7654, "median": 0.75, "ci": null, "n": 13, "clusters": 4, "paired": { "lebowski_877m": { "delta": 0.2538, "median": 0.3, "ci": null, "n": 13, "wins": 0, "ties": 1, "clusters": 4, "cluster_wins": 0 }, "1365126a_12864": { "delta": 0.1192, "median": 0.15, "ci": null, "n": 13, "wins": 1, "ties": 2, "clusters": 4, "cluster_wins": 0 } } }, { "key": "lon_mae", "value": 0.2611, "median": 0.2519, "ci": null, "n": 13, "clusters": 4, "paired": { "lebowski_877m": { "delta": 0.03288, "median": 0.01405, "ci": null, "n": 13, "wins": 4, "ties": 0, "clusters": 4, "cluster_wins": 0 }, "1365126a_12864": { "delta": -0.005507, "median": -0.01206, "ci": null, "n": 13, "wins": 8, "ties": 0, "clusters": 4, "cluster_wins": 3 } } }, { "key": "lon_corr", "value": 0.6199, "median": 0.7343, "ci": null, "n": 13, "clusters": 4, "paired": { "lebowski_877m": { "delta": -0.01424, "median": -0.003834, "ci": null, "n": 13, "wins": 6, "ties": 0, "clusters": 4, "cluster_wins": 1 }, "1365126a_12864": { "delta": 0.02021, "median": 0.008418, "ci": null, "n": 13, "wins": 7, "ties": 0, "clusters": 4, "cluster_wins": 4 } } }, { "key": "lon_brake_mae", "value": 0.6227, "median": 0.669, "ci": null, "n": 6, "clusters": 4, "paired": { "lebowski_877m": { "delta": -0.02456, "median": 0.03007, "ci": null, "n": 6, "wins": 3, "ties": 0, "clusters": 4, "cluster_wins": 2 }, "1365126a_12864": { "delta": -0.01618, "median": -0.04829, "ci": null, "n": 6, "wins": 4, "ties": 0, "clusters": 4, "cluster_wins": 3 } } }, { "key": "lon_rate_rms", "value": 1.244, "median": 1.285, "ci": null, "n": 13, "clusters": 4, "paired": { "lebowski_877m": { "delta": -0.9701, "median": -0.9213, "ci": null, "n": 13, "wins": 13, "ties": 0, "clusters": 4, "cluster_wins": 4 }, "1365126a_12864": { "delta": 0.06709, "median": 0.09686, "ci": null, "n": 13, "wins": 5, "ties": 0, "clusters": 4, "cluster_wins": 1 } } }, { "key": "v3_mae", "value": 0.7878, "median": 0.7611, "ci": null, "n": 13, "clusters": 4, "paired": { "lebowski_877m": { "delta": -0.05341, "median": -0.007929, "ci": null, "n": 13, "wins": 8, "ties": 0, "clusters": 4, "cluster_wins": 3 }, "1365126a_12864": { "delta": -0.01884, "median": -0.0316, "ci": null, "n": 13, "wins": 8, "ties": 0, "clusters": 4, "cluster_wins": 3 } } } ] }, "stock-newcase": { "n": 2, "clusters": 1, "metrics": [ { "key": "lat_mae", "value": 0.1493, "median": 0.1493, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": 0.01497, "median": 0.01497, "ci": null, "n": 2, "wins": 0, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "1365126a_12864": { "delta": 0.008105, "median": 0.008105, "ci": null, "n": 2, "wins": 0, "ties": 0, "clusters": 1, "cluster_wins": 0 } } }, { "key": "lat_mae_bl", "value": 0.1445, "median": 0.1445, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": 0.009287, "median": 0.009287, "ci": null, "n": 2, "wins": 0, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "1365126a_12864": { "delta": 0.008364, "median": 0.008364, "ci": null, "n": 2, "wins": 0, "ties": 0, "clusters": 1, "cluster_wins": 0 } } }, { "key": "lat_corr", "value": 0.8136, "median": 0.8136, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.03401, "median": -0.03401, "ci": null, "n": 2, "wins": 0, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "1365126a_12864": { "delta": -0.007071, "median": -0.007071, "ci": null, "n": 2, "wins": 0, "ties": 0, "clusters": 1, "cluster_wins": 0 } } }, { "key": "lat_gain", "value": 0.7884, "median": 0.7884, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": 0.001455, "median": 0.001455, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "1365126a_12864": { "delta": -0.01012, "median": -0.01012, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 1 } } }, { "key": "lat_rate_rms", "value": 0.7964, "median": 0.7964, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.4792, "median": -0.4792, "ci": null, "n": 2, "wins": 2, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "1365126a_12864": { "delta": 0.03361, "median": 0.03361, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 } } }, { "key": "lat_best_lag", "value": 0.65, "median": 0.65, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": 0.15, "median": 0.15, "ci": null, "n": 2, "wins": 0, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "1365126a_12864": { "delta": 0.025, "median": 0.025, "ci": null, "n": 2, "wins": 0, "ties": 1, "clusters": 1, "cluster_wins": 0 } } }, { "key": "lon_mae", "value": 0.3638, "median": 0.3638, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": 0.001265, "median": 0.001265, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "1365126a_12864": { "delta": -0.01578, "median": -0.01578, "ci": null, "n": 2, "wins": 2, "ties": 0, "clusters": 1, "cluster_wins": 1 } } }, { "key": "lon_corr", "value": 0.5003, "median": 0.5003, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": 0.01365, "median": 0.01365, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "1365126a_12864": { "delta": -0.02728, "median": -0.02728, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 } } }, { "key": "lon_brake_mae", "value": 0.5084, "median": 0.5084, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": 0.005061, "median": 0.005061, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "1365126a_12864": { "delta": 0.08164, "median": 0.08164, "ci": null, "n": 2, "wins": 0, "ties": 0, "clusters": 1, "cluster_wins": 0 } } }, { "key": "lon_rate_rms", "value": 1.626, "median": 1.626, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": -1.73, "median": -1.73, "ci": null, "n": 2, "wins": 2, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "1365126a_12864": { "delta": -0.7118, "median": -0.7118, "ci": null, "n": 2, "wins": 2, "ties": 0, "clusters": 1, "cluster_wins": 1 } } }, { "key": "v3_mae", "value": 1.902, "median": 1.902, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.1915, "median": -0.1915, "ci": null, "n": 2, "wins": 2, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "1365126a_12864": { "delta": -0.2264, "median": -0.2264, "ci": null, "n": 2, "wins": 2, "ties": 0, "clusters": 1, "cluster_wins": 1 } } } ] }, "v13-8mm": { "n": 92, "clusters": 3, "metrics": [ { "key": "lat_mae", "value": 0.1738, "median": 0.1646, "ci": null, "n": 92, "clusters": 3, "paired": { "lebowski_877m": { "delta": -0.007759, "median": -0.008295, "ci": null, "n": 92, "wins": 60, "ties": 0, "clusters": 3, "cluster_wins": 3 }, "1365126a_12864": { "delta": 0.01158, "median": 0.009977, "ci": null, "n": 92, "wins": 23, "ties": 0, "clusters": 3, "cluster_wins": 0 } } }, { "key": "lat_mae_bl", "value": 0.1517, "median": 0.1341, "ci": null, "n": 92, "clusters": 3, "paired": { "lebowski_877m": { "delta": -0.02042, "median": -0.01947, "ci": null, "n": 92, "wins": 79, "ties": 0, "clusters": 3, "cluster_wins": 3 }, "1365126a_12864": { "delta": 0.00336, "median": 0.003277, "ci": null, "n": 92, "wins": 38, "ties": 0, "clusters": 3, "cluster_wins": 1 } } }, { "key": "lat_corr", "value": 0.7932, "median": 0.9003, "ci": null, "n": 92, "clusters": 3, "paired": { "lebowski_877m": { "delta": 0.04476, "median": 0.01891, "ci": null, "n": 92, "wins": 68, "ties": 0, "clusters": 3, "cluster_wins": 3 }, "1365126a_12864": { "delta": -0.01786, "median": -0.01319, "ci": null, "n": 92, "wins": 15, "ties": 0, "clusters": 3, "cluster_wins": 0 } } }, { "key": "lat_gain", "value": 0.7553, "median": 0.7855, "ci": null, "n": 92, "clusters": 3, "paired": { "lebowski_877m": { "delta": 0.01421, "median": 0.01976, "ci": null, "n": 92, "wins": 41, "ties": 0, "clusters": 3, "cluster_wins": 0 }, "1365126a_12864": { "delta": 0.01332, "median": 0.01763, "ci": null, "n": 92, "wins": 38, "ties": 0, "clusters": 3, "cluster_wins": 1 } } }, { "key": "lat_rate_rms", "value": 1.187, "median": 0.9182, "ci": null, "n": 92, "clusters": 3, "paired": { "lebowski_877m": { "delta": -1.431, "median": -1.375, "ci": null, "n": 92, "wins": 91, "ties": 0, "clusters": 3, "cluster_wins": 3 }, "1365126a_12864": { "delta": -0.07608, "median": -0.06669, "ci": null, "n": 92, "wins": 56, "ties": 0, "clusters": 3, "cluster_wins": 3 } } }, { "key": "lat_best_lag", "value": 0.7555, "median": 0.7, "ci": null, "n": 91, "clusters": 3, "paired": { "lebowski_877m": { "delta": 0.1022, "median": 0.1, "ci": null, "n": 91, "wins": 20, "ties": 9, "clusters": 3, "cluster_wins": 0 }, "1365126a_12864": { "delta": 0.1077, "median": 0.1, "ci": null, "n": 91, "wins": 2, "ties": 13, "clusters": 3, "cluster_wins": 0 } } }, { "key": "lon_mae", "value": 0.3961, "median": 0.4024, "ci": null, "n": 92, "clusters": 3, "paired": { "lebowski_877m": { "delta": 0.02007, "median": 0.02214, "ci": null, "n": 92, "wins": 32, "ties": 0, "clusters": 3, "cluster_wins": 1 }, "1365126a_12864": { "delta": 0.03711, "median": 0.03611, "ci": null, "n": 92, "wins": 12, "ties": 0, "clusters": 3, "cluster_wins": 0 } } }, { "key": "lon_corr", "value": 0.202, "median": 0.1677, "ci": null, "n": 92, "clusters": 3, "paired": { "lebowski_877m": { "delta": -0.03165, "median": -0.02784, "ci": null, "n": 92, "wins": 40, "ties": 0, "clusters": 3, "cluster_wins": 1 }, "1365126a_12864": { "delta": -0.05229, "median": -0.04272, "ci": null, "n": 92, "wins": 26, "ties": 0, "clusters": 3, "cluster_wins": 0 } } }, { "key": "lon_brake_mae", "value": 0.4304, "median": 0.4317, "ci": null, "n": 65, "clusters": 3, "paired": { "lebowski_877m": { "delta": -0.05874, "median": -0.04341, "ci": null, "n": 65, "wins": 44, "ties": 0, "clusters": 3, "cluster_wins": 3 }, "1365126a_12864": { "delta": -0.02074, "median": -0.02438, "ci": null, "n": 65, "wins": 43, "ties": 0, "clusters": 3, "cluster_wins": 3 } } }, { "key": "lon_rate_rms", "value": 6.482, "median": 6.51, "ci": null, "n": 92, "clusters": 3, "paired": { "lebowski_877m": { "delta": 0.1487, "median": -0.2819, "ci": null, "n": 92, "wins": 50, "ties": 0, "clusters": 3, "cluster_wins": 2 }, "1365126a_12864": { "delta": 1.48, "median": 1.444, "ci": null, "n": 92, "wins": 14, "ties": 0, "clusters": 3, "cluster_wins": 0 } } }, { "key": "v3_mae", "value": 2.172, "median": 2.147, "ci": null, "n": 92, "clusters": 3, "paired": { "lebowski_877m": { "delta": -0.06194, "median": -0.05169, "ci": null, "n": 92, "wins": 53, "ties": 0, "clusters": 3, "cluster_wins": 2 }, "1365126a_12864": { "delta": 0.04744, "median": 0.1543, "ci": null, "n": 92, "wins": 36, "ties": 0, "clusters": 3, "cluster_wins": 2 } } } ] }, "v14-8mm": { "n": 8, "clusters": 2, "metrics": [ { "key": "lat_mae", "value": 0.1699, "median": 0.1673, "ci": null, "n": 8, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.02849, "median": 0.02514, "ci": null, "n": 8, "wins": 0, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "1365126a_12864": { "delta": 0.02552, "median": 0.02124, "ci": null, "n": 8, "wins": 0, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "lat_mae_bl", "value": 0.154, "median": 0.1485, "ci": null, "n": 8, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.01688, "median": 0.01252, "ci": null, "n": 8, "wins": 1, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "1365126a_12864": { "delta": 0.01711, "median": 0.01161, "ci": null, "n": 8, "wins": 0, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "lat_corr", "value": 0.8362, "median": 0.9129, "ci": null, "n": 8, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.03173, "median": -0.01493, "ci": null, "n": 8, "wins": 3, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "1365126a_12864": { "delta": -0.03499, "median": -0.0189, "ci": null, "n": 8, "wins": 0, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "lat_gain", "value": 0.8787, "median": 0.8528, "ci": null, "n": 8, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.0813, "median": 0.07457, "ci": null, "n": 8, "wins": 3, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "1365126a_12864": { "delta": 0.04448, "median": 0.04669, "ci": null, "n": 8, "wins": 1, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "lat_rate_rms", "value": 0.6574, "median": 0.5384, "ci": null, "n": 8, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.4982, "median": -0.4925, "ci": null, "n": 8, "wins": 8, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "1365126a_12864": { "delta": 0.06564, "median": 0.01258, "ci": null, "n": 8, "wins": 4, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "lat_best_lag", "value": 0.65, "median": 0.625, "ci": null, "n": 8, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.2, "median": 0.15, "ci": null, "n": 8, "wins": 0, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "1365126a_12864": { "delta": 0.1125, "median": 0.1, "ci": null, "n": 8, "wins": 0, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "lon_mae", "value": 0.3352, "median": 0.3197, "ci": null, "n": 8, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.02585, "median": 0.03006, "ci": null, "n": 8, "wins": 2, "ties": 0, "clusters": 2, "cluster_wins": 1 }, "1365126a_12864": { "delta": -0.011, "median": -0.005292, "ci": null, "n": 8, "wins": 5, "ties": 0, "clusters": 2, "cluster_wins": 2 } } }, { "key": "lon_corr", "value": 0.6838, "median": 0.7426, "ci": null, "n": 8, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.0709, "median": -0.05464, "ci": null, "n": 8, "wins": 3, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "1365126a_12864": { "delta": 0.001062, "median": -0.002505, "ci": null, "n": 8, "wins": 4, "ties": 0, "clusters": 2, "cluster_wins": 1 } } }, { "key": "lon_brake_mae", "value": 0.6384, "median": 0.6803, "ci": null, "n": 6, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.08076, "median": 0.08004, "ci": null, "n": 6, "wins": 2, "ties": 0, "clusters": 2, "cluster_wins": 1 }, "1365126a_12864": { "delta": 0.004418, "median": -0.001792, "ci": null, "n": 6, "wins": 3, "ties": 0, "clusters": 2, "cluster_wins": 1 } } }, { "key": "lon_rate_rms", "value": 5.045, "median": 4.82, "ci": null, "n": 8, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.5119, "median": 0.06986, "ci": null, "n": 8, "wins": 4, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "1365126a_12864": { "delta": 0.4803, "median": 0.4273, "ci": null, "n": 8, "wins": 2, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "v3_mae", "value": 0.9327, "median": 0.9285, "ci": null, "n": 8, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.0002404, "median": -0.001614, "ci": null, "n": 8, "wins": 5, "ties": 0, "clusters": 2, "cluster_wins": 1 }, "1365126a_12864": { "delta": -0.162, "median": -0.1857, "ci": null, "n": 8, "wins": 7, "ties": 0, "clusters": 2, "cluster_wins": 2 } } } ] } } }, "truck_closed_loop": { "status": "ok", "unit_of_evidence": "window", "n": 98, "clusters": 13, "condition": "Measured. worldmodel-4B as in the public closed loop, but the model drives a simulation of the G55's measured steering and powertrain response instead of the ideal car. The simulation starts from the truck's state (speed, yaw rate, steering, gear) in its CAN recording of the 2 s before the window; one window, whose 2 s before are not all valid CAN, starts steady at its recorded speed. Lane-following windows from the G55's drives, day and night; distance is to the driver's recorded path; seeds 0 and 1, averaged per window. 98 windows (59 day, 39 night).", "baselines": [ "lebowski_877m", "1365126a_12864" ], "metrics": [ { "key": "xtrack_2s", "label": "Distance from the recorded path at 2 s", "unit": "m", "better": "lower", "value": 0.1361, "median": 0.08325, "ci": [ 0.09959, 0.179 ], "n": 98, "clusters": 13, "paired": { "lebowski_877m": { "delta": -0.0334, "median": -0.02585, "ci": [ -0.04801, -0.0197 ], "n": 98, "wins": 64, "ties": 0, "clusters": 13, "cluster_wins": 12 }, "1365126a_12864": { "delta": -0.0005412, "median": -0.002301, "ci": [ -0.01387, 0.01459 ], "n": 98, "wins": 53, "ties": 0, "clusters": 13, "cluster_wins": 7 } } }, { "key": "xtrack_4s", "label": "Distance from the recorded path at 4 s", "unit": "m", "better": "lower", "value": 0.5554, "median": 0.2841, "ci": [ 0.3572, 0.7924 ], "n": 98, "clusters": 13, "paired": { "lebowski_877m": { "delta": 0.03534, "median": -0.01536, "ci": [ -0.04831, 0.1348 ], "n": 98, "wins": 52, "ties": 0, "clusters": 13, "cluster_wins": 8 }, "1365126a_12864": { "delta": 0.0159, "median": -0.01726, "ci": [ -0.1103, 0.1892 ], "n": 98, "wins": 63, "ties": 0, "clusters": 13, "cluster_wins": 10 } } }, { "key": "xtrack_7s", "label": "Distance from the recorded path at 7.2 s", "unit": "m", "better": "lower", "value": 2.399, "median": 0.7292, "ci": [ 1.502, 3.423 ], "n": 98, "clusters": 13, "paired": { "lebowski_877m": { "delta": 0.4197, "median": 0.02513, "ci": [ -0.3519, 1.44 ], "n": 98, "wins": 46, "ties": 0, "clusters": 13, "cluster_wins": 5 }, "1365126a_12864": { "delta": -0.132, "median": -0.07199, "ci": [ -0.5589, 0.4898 ], "n": 98, "wins": 63, "ties": 0, "clusters": 13, "cluster_wins": 11 } } }, { "key": "fail_1m", "label": "Share of episodes ever more than 1 m off the path", "unit": "", "better": "lower", "value": 0.4031, "median": 0.0, "ci": [ 0.3069, 0.5124 ], "n": 98, "clusters": 13, "paired": { "lebowski_877m": { "delta": -0.02551, "median": 0.0, "ci": [ -0.06915, 0.02501 ], "n": 98, "wins": 12, "ties": 77, "clusters": 13, "cluster_wins": 5 }, "1365126a_12864": { "delta": -0.05102, "median": 0.0, "ci": [ -0.1072, 0.006421 ], "n": 98, "wins": 13, "ties": 79, "clusters": 13, "cluster_wins": 7 } } }, { "key": "lat_jerk", "label": "Requested lateral jerk, RMS", "unit": "m/s³", "better": "lower", "value": 0.2627, "median": 0.1933, "ci": [ 0.2258, 0.2985 ], "n": 98, "clusters": 13, "paired": { "lebowski_877m": { "delta": -0.2792, "median": -0.1845, "ci": [ -0.3603, -0.203 ], "n": 98, "wins": 91, "ties": 0, "clusters": 13, "cluster_wins": 13 }, "1365126a_12864": { "delta": -0.02321, "median": 0.002514, "ci": [ -0.04714, 0.002417 ], "n": 98, "wins": 48, "ties": 0, "clusters": 13, "cluster_wins": 6 } } }, { "key": "long_jerk", "label": "Requested longitudinal jerk, RMS", "unit": "m/s³", "better": "lower", "value": 0.3782, "median": 0.3051, "ci": [ 0.3356, 0.4157 ], "n": 98, "clusters": 13, "paired": { "lebowski_877m": { "delta": -0.008104, "median": -0.03339, "ci": [ -0.06796, 0.04686 ], "n": 98, "wins": 63, "ties": 0, "clusters": 13, "cluster_wins": 7 }, "1365126a_12864": { "delta": 0.01316, "median": 0.01608, "ci": [ -0.03925, 0.05692 ], "n": 98, "wins": 38, "ties": 0, "clusters": 13, "cluster_wins": 4 } } }, { "key": "speed_err", "label": "Speed error against the recorded speed, RMS", "unit": "m/s", "better": "lower", "value": 1.211, "median": 1.065, "ci": [ 1.02, 1.398 ], "n": 98, "clusters": 13, "paired": { "lebowski_877m": { "delta": 0.1284, "median": 0.07584, "ci": [ -0.02723, 0.2697 ], "n": 98, "wins": 40, "ties": 0, "clusters": 13, "cluster_wins": 5 }, "1365126a_12864": { "delta": -0.007942, "median": 0.02076, "ci": [ -0.1073, 0.1181 ], "n": 98, "wins": 48, "ties": 0, "clusters": 13, "cluster_wins": 9 } } } ], "per_unit_file": null, "by_group": { "stock-aimedup": { "n": 14, "clusters": 2, "metrics": [ { "key": "xtrack_2s", "value": 0.09137, "median": 0.06579, "ci": null, "n": 14, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.05895, "median": -0.04078, "ci": null, "n": 14, "wins": 8, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "1365126a_12864": { "delta": -0.03276, "median": -0.02638, "ci": null, "n": 14, "wins": 11, "ties": 0, "clusters": 2, "cluster_wins": 2 } } }, { "key": "xtrack_4s", "value": 0.4942, "median": 0.3466, "ci": null, "n": 14, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.04303, "median": -0.04108, "ci": null, "n": 14, "wins": 8, "ties": 0, "clusters": 2, "cluster_wins": 1 }, "1365126a_12864": { "delta": -0.2058, "median": -0.1203, "ci": null, "n": 14, "wins": 11, "ties": 0, "clusters": 2, "cluster_wins": 2 } } }, { "key": "xtrack_7s", "value": 3.497, "median": 1.909, "ci": null, "n": 14, "clusters": 2, "paired": { "lebowski_877m": { "delta": 1.592, "median": 0.423, "ci": null, "n": 14, "wins": 6, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "1365126a_12864": { "delta": -0.9363, "median": -0.4103, "ci": null, "n": 14, "wins": 8, "ties": 0, "clusters": 2, "cluster_wins": 2 } } }, { "key": "fail_1m", "value": 0.75, "median": 1.0, "ci": null, "n": 14, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.07143, "median": 0.0, "ci": null, "n": 14, "wins": 2, "ties": 8, "clusters": 2, "cluster_wins": 0 }, "1365126a_12864": { "delta": 0.0, "median": 0.0, "ci": null, "n": 14, "wins": 3, "ties": 8, "clusters": 2, "cluster_wins": 1 } } }, { "key": "lat_jerk", "value": 0.3812, "median": 0.2845, "ci": null, "n": 14, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.1896, "median": -0.1451, "ci": null, "n": 14, "wins": 11, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "1365126a_12864": { "delta": -0.08008, "median": 0.0007169, "ci": null, "n": 14, "wins": 7, "ties": 0, "clusters": 2, "cluster_wins": 2 } } }, { "key": "long_jerk", "value": 0.3312, "median": 0.3181, "ci": null, "n": 14, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.1635, "median": -0.1294, "ci": null, "n": 14, "wins": 14, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "1365126a_12864": { "delta": 0.02511, "median": 0.01408, "ci": null, "n": 14, "wins": 5, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "speed_err", "value": 1.429, "median": 1.379, "ci": null, "n": 14, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.3345, "median": -0.7257, "ci": null, "n": 14, "wins": 10, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "1365126a_12864": { "delta": -0.1286, "median": -0.03607, "ci": null, "n": 14, "wins": 7, "ties": 0, "clusters": 2, "cluster_wins": 2 } } } ] }, "stock-dual": { "n": 43, "clusters": 5, "metrics": [ { "key": "xtrack_2s", "value": 0.1322, "median": 0.09819, "ci": null, "n": 43, "clusters": 5, "paired": { "lebowski_877m": { "delta": -0.03589, "median": -0.03865, "ci": null, "n": 43, "wins": 30, "ties": 0, "clusters": 5, "cluster_wins": 5 }, "1365126a_12864": { "delta": -0.002685, "median": 3.728e-05, "ci": null, "n": 43, "wins": 21, "ties": 0, "clusters": 5, "cluster_wins": 3 } } }, { "key": "xtrack_4s", "value": 0.3884, "median": 0.2818, "ci": null, "n": 43, "clusters": 5, "paired": { "lebowski_877m": { "delta": -0.06382, "median": -0.0149, "ci": null, "n": 43, "wins": 23, "ties": 0, "clusters": 5, "cluster_wins": 4 }, "1365126a_12864": { "delta": -0.02295, "median": -0.009659, "ci": null, "n": 43, "wins": 22, "ties": 0, "clusters": 5, "cluster_wins": 4 } } }, { "key": "xtrack_7s", "value": 1.515, "median": 0.5646, "ci": null, "n": 43, "clusters": 5, "paired": { "lebowski_877m": { "delta": -0.5045, "median": 0.02424, "ci": null, "n": 43, "wins": 20, "ties": 0, "clusters": 5, "cluster_wins": 3 }, "1365126a_12864": { "delta": -0.4439, "median": -0.07804, "ci": null, "n": 43, "wins": 29, "ties": 0, "clusters": 5, "cluster_wins": 5 } } }, { "key": "fail_1m", "value": 0.3023, "median": 0.0, "ci": null, "n": 43, "clusters": 5, "paired": { "lebowski_877m": { "delta": -0.03488, "median": 0.0, "ci": null, "n": 43, "wins": 5, "ties": 35, "clusters": 5, "cluster_wins": 3 }, "1365126a_12864": { "delta": -0.0814, "median": 0.0, "ci": null, "n": 43, "wins": 6, "ties": 36, "clusters": 5, "cluster_wins": 3 } } }, { "key": "lat_jerk", "value": 0.2584, "median": 0.1936, "ci": null, "n": 43, "clusters": 5, "paired": { "lebowski_877m": { "delta": -0.2943, "median": -0.1957, "ci": null, "n": 43, "wins": 40, "ties": 0, "clusters": 5, "cluster_wins": 5 }, "1365126a_12864": { "delta": -0.03534, "median": 0.006389, "ci": null, "n": 43, "wins": 19, "ties": 0, "clusters": 5, "cluster_wins": 3 } } }, { "key": "long_jerk", "value": 0.4019, "median": 0.3048, "ci": null, "n": 43, "clusters": 5, "paired": { "lebowski_877m": { "delta": 0.02762, "median": -0.0266, "ci": null, "n": 43, "wins": 27, "ties": 0, "clusters": 5, "cluster_wins": 3 }, "1365126a_12864": { "delta": -0.02292, "median": 0.0149, "ci": null, "n": 43, "wins": 19, "ties": 0, "clusters": 5, "cluster_wins": 2 } } }, { "key": "speed_err", "value": 1.162, "median": 1.059, "ci": null, "n": 43, "clusters": 5, "paired": { "lebowski_877m": { "delta": 0.07853, "median": 0.06911, "ci": null, "n": 43, "wins": 18, "ties": 0, "clusters": 5, "cluster_wins": 2 }, "1365126a_12864": { "delta": -0.07633, "median": -0.0299, "ci": null, "n": 43, "wins": 25, "ties": 0, "clusters": 5, "cluster_wins": 4 } } } ] }, "stock-newcase": { "n": 4, "clusters": 1, "metrics": [ { "key": "xtrack_2s", "value": 0.05186, "median": 0.05138, "ci": null, "n": 4, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.073, "median": -0.06655, "ci": null, "n": 4, "wins": 3, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "1365126a_12864": { "delta": 0.002319, "median": 0.00507, "ci": null, "n": 4, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 } } }, { "key": "xtrack_4s", "value": 0.2516, "median": 0.2126, "ci": null, "n": 4, "clusters": 1, "paired": { "lebowski_877m": { "delta": 0.01098, "median": 0.09771, "ci": null, "n": 4, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "1365126a_12864": { "delta": -0.005847, "median": -0.03737, "ci": null, "n": 4, "wins": 3, "ties": 0, "clusters": 1, "cluster_wins": 1 } } }, { "key": "xtrack_7s", "value": 1.423, "median": 1.343, "ci": null, "n": 4, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.213, "median": -0.04024, "ci": null, "n": 4, "wins": 2, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "1365126a_12864": { "delta": -0.1303, "median": -0.1526, "ci": null, "n": 4, "wins": 3, "ties": 0, "clusters": 1, "cluster_wins": 1 } } }, { "key": "fail_1m", "value": 0.625, "median": 0.75, "ci": null, "n": 4, "clusters": 1, "paired": { "lebowski_877m": { "delta": 0.0, "median": 0.0, "ci": null, "n": 4, "wins": 0, "ties": 4, "clusters": 1, "cluster_wins": 0 }, "1365126a_12864": { "delta": 0.125, "median": 0.0, "ci": null, "n": 4, "wins": 0, "ties": 3, "clusters": 1, "cluster_wins": 0 } } }, { "key": "lat_jerk", "value": 0.2684, "median": 0.2276, "ci": null, "n": 4, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.4713, "median": -0.3473, "ci": null, "n": 4, "wins": 4, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "1365126a_12864": { "delta": 0.001902, "median": -0.006545, "ci": null, "n": 4, "wins": 3, "ties": 0, "clusters": 1, "cluster_wins": 0 } } }, { "key": "long_jerk", "value": 0.265, "median": 0.2689, "ci": null, "n": 4, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.1243, "median": -0.1479, "ci": null, "n": 4, "wins": 3, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "1365126a_12864": { "delta": -0.176, "median": -0.1515, "ci": null, "n": 4, "wins": 4, "ties": 0, "clusters": 1, "cluster_wins": 1 } } }, { "key": "speed_err", "value": 0.9416, "median": 0.9593, "ci": null, "n": 4, "clusters": 1, "paired": { "lebowski_877m": { "delta": 0.2628, "median": 0.2488, "ci": null, "n": 4, "wins": 2, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "1365126a_12864": { "delta": -0.1053, "median": 0.1908, "ci": null, "n": 4, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 1 } } } ] }, "v13-8mm": { "n": 18, "clusters": 3, "metrics": [ { "key": "xtrack_2s", "value": 0.215, "median": 0.08879, "ci": null, "n": 18, "clusters": 3, "paired": { "lebowski_877m": { "delta": -0.002281, "median": -0.01537, "ci": null, "n": 18, "wins": 12, "ties": 0, "clusters": 3, "cluster_wins": 2 }, "1365126a_12864": { "delta": 0.02927, "median": -0.001793, "ci": null, "n": 18, "wins": 10, "ties": 0, "clusters": 3, "cluster_wins": 1 } } }, { "key": "xtrack_4s", "value": 0.919, "median": 0.2175, "ci": null, "n": 18, "clusters": 3, "paired": { "lebowski_877m": { "delta": 0.1169, "median": -0.02021, "ci": null, "n": 18, "wins": 10, "ties": 0, "clusters": 3, "cluster_wins": 2 }, "1365126a_12864": { "delta": 0.3676, "median": -0.01366, "ci": null, "n": 18, "wins": 14, "ties": 0, "clusters": 3, "cluster_wins": 2 } } }, { "key": "xtrack_7s", "value": 2.662, "median": 0.9521, "ci": null, "n": 18, "clusters": 3, "paired": { "lebowski_877m": { "delta": 0.03696, "median": 0.02816, "ci": null, "n": 18, "wins": 7, "ties": 0, "clusters": 3, "cluster_wins": 1 }, "1365126a_12864": { "delta": 1.26, "median": -0.005106, "ci": null, "n": 18, "wins": 11, "ties": 0, "clusters": 3, "cluster_wins": 1 } } }, { "key": "fail_1m", "value": 0.3889, "median": 0.5, "ci": null, "n": 18, "clusters": 3, "paired": { "lebowski_877m": { "delta": 0.0, "median": 0.0, "ci": null, "n": 18, "wins": 1, "ties": 16, "clusters": 3, "cluster_wins": 0 }, "1365126a_12864": { "delta": -0.02778, "median": 0.0, "ci": null, "n": 18, "wins": 2, "ties": 15, "clusters": 3, "cluster_wins": 1 } } }, { "key": "lat_jerk", "value": 0.2299, "median": 0.1602, "ci": null, "n": 18, "clusters": 3, "paired": { "lebowski_877m": { "delta": -0.1495, "median": -0.1603, "ci": null, "n": 18, "wins": 17, "ties": 0, "clusters": 3, "cluster_wins": 3 }, "1365126a_12864": { "delta": -0.003059, "median": -0.008533, "ci": null, "n": 18, "wins": 11, "ties": 0, "clusters": 3, "cluster_wins": 1 } } }, { "key": "long_jerk", "value": 0.3795, "median": 0.2792, "ci": null, "n": 18, "clusters": 3, "paired": { "lebowski_877m": { "delta": 0.07487, "median": 0.0495, "ci": null, "n": 18, "wins": 7, "ties": 0, "clusters": 3, "cluster_wins": 0 }, "1365126a_12864": { "delta": 0.06176, "median": 0.04168, "ci": null, "n": 18, "wins": 5, "ties": 0, "clusters": 3, "cluster_wins": 1 } } }, { "key": "speed_err", "value": 1.163, "median": 1.054, "ci": null, "n": 18, "clusters": 3, "paired": { "lebowski_877m": { "delta": 0.2183, "median": 0.2301, "ci": null, "n": 18, "wins": 5, "ties": 0, "clusters": 3, "cluster_wins": 1 }, "1365126a_12864": { "delta": 0.3046, "median": 0.2969, "ci": null, "n": 18, "wins": 3, "ties": 0, "clusters": 3, "cluster_wins": 1 } } } ] }, "v14-8mm": { "n": 19, "clusters": 2, "metrics": [ { "key": "xtrack_2s", "value": 0.1208, "median": 0.0998, "ci": null, "n": 19, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.0301, "median": -0.009671, "ci": null, "n": 19, "wins": 11, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "1365126a_12864": { "delta": -0.0007972, "median": -0.0009314, "ci": null, "n": 19, "wins": 10, "ties": 0, "clusters": 2, "cluster_wins": 1 } } }, { "key": "xtrack_4s", "value": 0.6977, "median": 0.402, "ci": null, "n": 19, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.1819, "median": -0.008674, "ci": null, "n": 19, "wins": 10, "ties": 0, "clusters": 2, "cluster_wins": 1 }, "1365126a_12864": { "delta": -0.06137, "median": -0.01579, "ci": null, "n": 19, "wins": 13, "ties": 0, "clusters": 2, "cluster_wins": 1 } } }, { "key": "xtrack_7s", "value": 3.549, "median": 0.6326, "ci": null, "n": 19, "clusters": 2, "paired": { "lebowski_877m": { "delta": 2.144, "median": -0.04705, "ci": null, "n": 19, "wins": 11, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "1365126a_12864": { "delta": -0.1528, "median": -0.1007, "ci": null, "n": 19, "wins": 12, "ties": 0, "clusters": 2, "cluster_wins": 2 } } }, { "key": "fail_1m", "value": 0.3421, "median": 0.0, "ci": null, "n": 19, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.1053, "median": 0.0, "ci": null, "n": 19, "wins": 4, "ties": 14, "clusters": 2, "cluster_wins": 2 }, "1365126a_12864": { "delta": -0.07895, "median": 0.0, "ci": null, "n": 19, "wins": 2, "ties": 17, "clusters": 2, "cluster_wins": 2 } } }, { "key": "lat_jerk", "value": 0.2148, "median": 0.1743, "ci": null, "n": 19, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.3937, "median": -0.2585, "ci": null, "n": 19, "wins": 19, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "1365126a_12864": { "delta": 0.02175, "median": 0.01431, "ci": null, "n": 19, "wins": 8, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "long_jerk", "value": 0.3818, "median": 0.3753, "ci": null, "n": 19, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.02862, "median": -0.02546, "ci": null, "n": 19, "wins": 12, "ties": 0, "clusters": 2, "cluster_wins": 1 }, "1365126a_12864": { "delta": 0.07977, "median": 0.02799, "ci": null, "n": 19, "wins": 5, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "speed_err", "value": 1.264, "median": 0.9521, "ci": null, "n": 19, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.4691, "median": 0.2126, "ci": null, "n": 19, "wins": 5, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "1365126a_12864": { "delta": -0.03978, "median": -0.05899, "ci": null, "n": 19, "wins": 12, "ties": 0, "clusters": 2, "cluster_wins": 1 } } } ] } }, "simulator": "g55-2026-10-31" }, "truck_recovery": { "status": "ok", "unit_of_evidence": "window", "n": 100, "clusters": 14, "condition": "Measured. The truck closed loop with a disturbance while the model drives; only the tests listed ran. 2° off heading: an extra 2°/s yaw for the first second. 98 windows (59 day, 39 night). 0.5 m off centre: an extra yaw-rate pulse, out and back over the first 2 s, that would move an uncorrected car 0.5 m sideways with no net heading change. 98 windows (59 day, 39 night). Braking lead: windows where the car ahead braked; the gap is derived from the lead distance seen on the recording minus how far the model ran ahead of the recorded car. 2 windows (2 day, 0 night).", "baselines": [ "lebowski_877m", "1365126a_12864" ], "metrics": [ { "key": "heading_xtrack_2s", "label": "2° off heading: distance from the path at 2 s", "unit": "m", "better": "lower", "value": 0.4841, "median": 0.3881, "ci": [ 0.3848, 0.614 ], "n": 98, "clusters": 13, "paired": { "lebowski_877m": { "delta": 0.03117, "median": 0.01421, "ci": [ -0.009573, 0.08257 ], "n": 98, "wins": 44, "ties": 0, "clusters": 13, "cluster_wins": 6 }, "1365126a_12864": { "delta": -0.03335, "median": -0.03748, "ci": [ -0.04863, -0.01183 ], "n": 98, "wins": 87, "ties": 0, "clusters": 13, "cluster_wins": 12 } } }, { "key": "heading_xtrack_4s", "label": "2° off heading: distance from the path at 4 s", "unit": "m", "better": "lower", "value": 0.7579, "median": 0.4288, "ci": [ 0.5203, 1.111 ], "n": 98, "clusters": 13, "paired": { "lebowski_877m": { "delta": 0.05531, "median": 0.01666, "ci": [ -0.0906, 0.1885 ], "n": 98, "wins": 44, "ties": 0, "clusters": 13, "cluster_wins": 4 }, "1365126a_12864": { "delta": -0.03066, "median": -0.03572, "ci": [ -0.1243, 0.09185 ], "n": 98, "wins": 61, "ties": 0, "clusters": 13, "cluster_wins": 10 } } }, { "key": "heading_lat_jerk", "label": "2° off heading: requested lateral jerk, RMS", "unit": "m/s³", "better": "lower", "value": 0.3824, "median": 0.3152, "ci": [ 0.3395, 0.4251 ], "n": 98, "clusters": 13, "paired": { "lebowski_877m": { "delta": -0.5045, "median": -0.4783, "ci": [ -0.5749, -0.4299 ], "n": 98, "wins": 95, "ties": 0, "clusters": 13, "cluster_wins": 13 }, "1365126a_12864": { "delta": 0.01123, "median": 0.009977, "ci": [ -0.01643, 0.03789 ], "n": 98, "wins": 40, "ties": 0, "clusters": 13, "cluster_wins": 6 } } }, { "key": "offcentre_xtrack_2s", "label": "0.5 m off centre: distance from the path at 2 s", "unit": "m", "better": "lower", "value": 0.2583, "median": 0.2085, "ci": [ 0.2284, 0.2897 ], "n": 98, "clusters": 13, "paired": { "lebowski_877m": { "delta": -0.008756, "median": 6.953e-05, "ci": [ -0.03322, 0.01604 ], "n": 98, "wins": 49, "ties": 0, "clusters": 13, "cluster_wins": 5 }, "1365126a_12864": { "delta": -0.02671, "median": -0.0347, "ci": [ -0.04201, -0.009568 ], "n": 98, "wins": 78, "ties": 0, "clusters": 13, "cluster_wins": 11 } } }, { "key": "offcentre_xtrack_4s", "label": "0.5 m off centre: distance from the path at 4 s", "unit": "m", "better": "lower", "value": 0.6621, "median": 0.3713, "ci": [ 0.4908, 0.8554 ], "n": 98, "clusters": 13, "paired": { "lebowski_877m": { "delta": -0.0007278, "median": -0.03582, "ci": [ -0.1464, 0.1318 ], "n": 98, "wins": 62, "ties": 0, "clusters": 13, "cluster_wins": 9 }, "1365126a_12864": { "delta": -0.01835, "median": -0.0008111, "ci": [ -0.1742, 0.1528 ], "n": 98, "wins": 49, "ties": 0, "clusters": 13, "cluster_wins": 7 } } }, { "key": "offcentre_lat_jerk", "label": "0.5 m off centre: requested lateral jerk, RMS", "unit": "m/s³", "better": "lower", "value": 0.5488, "median": 0.4443, "ci": [ 0.5015, 0.5902 ], "n": 98, "clusters": 13, "paired": { "lebowski_877m": { "delta": -0.5616, "median": -0.5487, "ci": [ -0.6295, -0.4918 ], "n": 98, "wins": 94, "ties": 0, "clusters": 13, "cluster_wins": 13 }, "1365126a_12864": { "delta": 0.03498, "median": 0.04521, "ci": [ 0.0008119, 0.06588 ], "n": 98, "wins": 23, "ties": 0, "clusters": 13, "cluster_wins": 4 } } }, { "key": "brake_ahead_4s", "label": "Braking lead: distance ahead of the recorded car at 4 s", "unit": "m", "better": "lower", "value": -0.02443, "median": -0.02443, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": 1.313, "median": 1.313, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "1365126a_12864": { "delta": 0.5401, "median": 0.5401, "ci": null, "n": 2, "wins": 0, "ties": 0, "clusters": 1, "cluster_wins": 0 } } }, { "key": "brake_gap_min", "label": "Braking lead: smallest gap to the lead (derived)", "unit": "m", "better": "higher", "value": 25.78, "median": 25.78, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": -1.68, "median": -1.68, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "1365126a_12864": { "delta": 0.07616, "median": 0.07616, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 1 } } }, { "key": "brake_speed_err", "label": "Braking lead: speed error against the recorded speed, RMS", "unit": "m/s", "better": "lower", "value": 1.132, "median": 1.132, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.07142, "median": -0.07142, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "1365126a_12864": { "delta": -0.128, "median": -0.128, "ci": null, "n": 2, "wins": 2, "ties": 0, "clusters": 1, "cluster_wins": 1 } } } ], "per_unit_file": null, "by_group": { "stock-aimedup": { "n": 14, "clusters": 2, "metrics": [ { "key": "heading_xtrack_2s", "value": 0.3638, "median": 0.3401, "ci": null, "n": 14, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.028, "median": -0.04143, "ci": null, "n": 14, "wins": 10, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "1365126a_12864": { "delta": -0.05217, "median": -0.04542, "ci": null, "n": 14, "wins": 14, "ties": 0, "clusters": 2, "cluster_wins": 2 } } }, { "key": "heading_xtrack_4s", "value": 0.6209, "median": 0.4277, "ci": null, "n": 14, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.06792, "median": 0.01816, "ci": null, "n": 14, "wins": 6, "ties": 0, "clusters": 2, "cluster_wins": 1 }, "1365126a_12864": { "delta": -0.1143, "median": -0.08731, "ci": null, "n": 14, "wins": 9, "ties": 0, "clusters": 2, "cluster_wins": 2 } } }, { "key": "heading_lat_jerk", "value": 0.4373, "median": 0.3115, "ci": null, "n": 14, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.3307, "median": -0.3193, "ci": null, "n": 14, "wins": 14, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "1365126a_12864": { "delta": 0.03993, "median": 0.05009, "ci": null, "n": 14, "wins": 5, "ties": 0, "clusters": 2, "cluster_wins": 1 } } }, { "key": "offcentre_xtrack_2s", "value": 0.23, "median": 0.222, "ci": null, "n": 14, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.03292, "median": -0.04056, "ci": null, "n": 14, "wins": 9, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "1365126a_12864": { "delta": -0.06284, "median": -0.04907, "ci": null, "n": 14, "wins": 13, "ties": 0, "clusters": 2, "cluster_wins": 2 } } }, { "key": "offcentre_xtrack_4s", "value": 0.6123, "median": 0.3147, "ci": null, "n": 14, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.02249, "median": -0.0755, "ci": null, "n": 14, "wins": 9, "ties": 0, "clusters": 2, "cluster_wins": 1 }, "1365126a_12864": { "delta": -0.2356, "median": -0.1045, "ci": null, "n": 14, "wins": 11, "ties": 0, "clusters": 2, "cluster_wins": 2 } } }, { "key": "offcentre_lat_jerk", "value": 0.5797, "median": 0.5713, "ci": null, "n": 14, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.482, "median": -0.5515, "ci": null, "n": 14, "wins": 13, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "1365126a_12864": { "delta": -0.06233, "median": 0.06662, "ci": null, "n": 14, "wins": 4, "ties": 0, "clusters": 2, "cluster_wins": 2 } } } ] }, "stock-dual": { "n": 45, "clusters": 6, "metrics": [ { "key": "heading_xtrack_2s", "value": 0.403, "median": 0.3446, "ci": null, "n": 43, "clusters": 5, "paired": { "lebowski_877m": { "delta": -0.01259, "median": -0.02003, "ci": null, "n": 43, "wins": 25, "ties": 0, "clusters": 5, "cluster_wins": 3 }, "1365126a_12864": { "delta": -0.04349, "median": -0.03688, "ci": null, "n": 43, "wins": 39, "ties": 0, "clusters": 5, "cluster_wins": 5 } } }, { "key": "heading_xtrack_4s", "value": 0.5128, "median": 0.4203, "ci": null, "n": 43, "clusters": 5, "paired": { "lebowski_877m": { "delta": -0.1234, "median": -0.02178, "ci": null, "n": 43, "wins": 25, "ties": 0, "clusters": 5, "cluster_wins": 3 }, "1365126a_12864": { "delta": -0.1053, "median": -0.03471, "ci": null, "n": 43, "wins": 25, "ties": 0, "clusters": 5, "cluster_wins": 4 } } }, { "key": "heading_lat_jerk", "value": 0.3931, "median": 0.3144, "ci": null, "n": 43, "clusters": 5, "paired": { "lebowski_877m": { "delta": -0.5033, "median": -0.49, "ci": null, "n": 43, "wins": 41, "ties": 0, "clusters": 5, "cluster_wins": 5 }, "1365126a_12864": { "delta": 0.03316, "median": 0.0297, "ci": null, "n": 43, "wins": 11, "ties": 0, "clusters": 5, "cluster_wins": 0 } } }, { "key": "offcentre_xtrack_2s", "value": 0.2638, "median": 0.2134, "ci": null, "n": 43, "clusters": 5, "paired": { "lebowski_877m": { "delta": -0.03551, "median": -0.02453, "ci": null, "n": 43, "wins": 28, "ties": 0, "clusters": 5, "cluster_wins": 3 }, "1365126a_12864": { "delta": -0.02466, "median": -0.03641, "ci": null, "n": 43, "wins": 33, "ties": 0, "clusters": 5, "cluster_wins": 4 } } }, { "key": "offcentre_xtrack_4s", "value": 0.5623, "median": 0.3906, "ci": null, "n": 43, "clusters": 5, "paired": { "lebowski_877m": { "delta": -0.1081, "median": -0.04027, "ci": null, "n": 43, "wins": 29, "ties": 0, "clusters": 5, "cluster_wins": 4 }, "1365126a_12864": { "delta": -0.05658, "median": 0.03937, "ci": null, "n": 43, "wins": 13, "ties": 0, "clusters": 5, "cluster_wins": 2 } } }, { "key": "offcentre_lat_jerk", "value": 0.6049, "median": 0.4523, "ci": null, "n": 43, "clusters": 5, "paired": { "lebowski_877m": { "delta": -0.5718, "median": -0.5822, "ci": null, "n": 43, "wins": 41, "ties": 0, "clusters": 5, "cluster_wins": 5 }, "1365126a_12864": { "delta": 0.07177, "median": 0.05603, "ci": null, "n": 43, "wins": 10, "ties": 0, "clusters": 5, "cluster_wins": 1 } } }, { "key": "brake_ahead_4s", "value": -0.02443, "median": -0.02443, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": 1.313, "median": 1.313, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "1365126a_12864": { "delta": 0.5401, "median": 0.5401, "ci": null, "n": 2, "wins": 0, "ties": 0, "clusters": 1, "cluster_wins": 0 } } }, { "key": "brake_gap_min", "value": 25.78, "median": 25.78, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": -1.68, "median": -1.68, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "1365126a_12864": { "delta": 0.07616, "median": 0.07616, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 1 } } }, { "key": "brake_speed_err", "value": 1.132, "median": 1.132, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.07142, "median": -0.07142, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "1365126a_12864": { "delta": -0.128, "median": -0.128, "ci": null, "n": 2, "wins": 2, "ties": 0, "clusters": 1, "cluster_wins": 1 } } } ] }, "stock-newcase": { "n": 4, "clusters": 1, "metrics": [ { "key": "heading_xtrack_2s", "value": 0.3656, "median": 0.3728, "ci": null, "n": 4, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.08501, "median": -0.1002, "ci": null, "n": 4, "wins": 3, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "1365126a_12864": { "delta": -0.01378, "median": -0.02061, "ci": null, "n": 4, "wins": 3, "ties": 0, "clusters": 1, "cluster_wins": 1 } } }, { "key": "heading_xtrack_4s", "value": 0.3868, "median": 0.3448, "ci": null, "n": 4, "clusters": 1, "paired": { "lebowski_877m": { "delta": 0.1671, "median": 0.09434, "ci": null, "n": 4, "wins": 0, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "1365126a_12864": { "delta": -0.01046, "median": -0.0008375, "ci": null, "n": 4, "wins": 2, "ties": 0, "clusters": 1, "cluster_wins": 1 } } }, { "key": "heading_lat_jerk", "value": 0.3967, "median": 0.3747, "ci": null, "n": 4, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.5755, "median": -0.5201, "ci": null, "n": 4, "wins": 4, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "1365126a_12864": { "delta": 0.06795, "median": 0.05326, "ci": null, "n": 4, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 } } }, { "key": "offcentre_xtrack_2s", "value": 0.3041, "median": 0.2058, "ci": null, "n": 4, "clusters": 1, "paired": { "lebowski_877m": { "delta": 0.004278, "median": 0.02864, "ci": null, "n": 4, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "1365126a_12864": { "delta": -0.01509, "median": -0.02578, "ci": null, "n": 4, "wins": 3, "ties": 0, "clusters": 1, "cluster_wins": 1 } } }, { "key": "offcentre_xtrack_4s", "value": 0.7106, "median": 0.6562, "ci": null, "n": 4, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.01708, "median": 0.1008, "ci": null, "n": 4, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "1365126a_12864": { "delta": -0.04798, "median": 0.07318, "ci": null, "n": 4, "wins": 2, "ties": 0, "clusters": 1, "cluster_wins": 1 } } }, { "key": "offcentre_lat_jerk", "value": 0.6309, "median": 0.5273, "ci": null, "n": 4, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.7665, "median": -0.5563, "ci": null, "n": 4, "wins": 4, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "1365126a_12864": { "delta": -0.00668, "median": 0.05893, "ci": null, "n": 4, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 1 } } } ] }, "v13-8mm": { "n": 18, "clusters": 3, "metrics": [ { "key": "heading_xtrack_2s", "value": 0.8977, "median": 0.8421, "ci": null, "n": 18, "clusters": 3, "paired": { "lebowski_877m": { "delta": 0.2047, "median": 0.1919, "ci": null, "n": 18, "wins": 1, "ties": 0, "clusters": 3, "cluster_wins": 0 }, "1365126a_12864": { "delta": 0.0129, "median": -0.01884, "ci": null, "n": 18, "wins": 13, "ties": 0, "clusters": 3, "cluster_wins": 2 } } }, { "key": "heading_xtrack_4s", "value": 1.52, "median": 0.7541, "ci": null, "n": 18, "clusters": 3, "paired": { "lebowski_877m": { "delta": 0.2891, "median": 0.08123, "ci": null, "n": 18, "wins": 7, "ties": 0, "clusters": 3, "cluster_wins": 0 }, "1365126a_12864": { "delta": 0.2097, "median": -0.04054, "ci": null, "n": 18, "wins": 12, "ties": 0, "clusters": 3, "cluster_wins": 2 } } }, { "key": "heading_lat_jerk", "value": 0.402, "median": 0.3739, "ci": null, "n": 18, "clusters": 3, "paired": { "lebowski_877m": { "delta": -0.471, "median": -0.4812, "ci": null, "n": 18, "wins": 17, "ties": 0, "clusters": 3, "cluster_wins": 3 }, "1365126a_12864": { "delta": -0.04444, "median": -0.01569, "ci": null, "n": 18, "wins": 15, "ties": 0, "clusters": 3, "cluster_wins": 3 } } }, { "key": "offcentre_xtrack_2s", "value": 0.299, "median": 0.2151, "ci": null, "n": 18, "clusters": 3, "paired": { "lebowski_877m": { "delta": 0.04096, "median": 0.0526, "ci": null, "n": 18, "wins": 6, "ties": 0, "clusters": 3, "cluster_wins": 0 }, "1365126a_12864": { "delta": 0.00571, "median": -0.01728, "ci": null, "n": 18, "wins": 12, "ties": 0, "clusters": 3, "cluster_wins": 2 } } }, { "key": "offcentre_xtrack_4s", "value": 0.8377, "median": 0.2049, "ci": null, "n": 18, "clusters": 3, "paired": { "lebowski_877m": { "delta": 0.1446, "median": -0.03435, "ci": null, "n": 18, "wins": 11, "ties": 0, "clusters": 3, "cluster_wins": 2 }, "1365126a_12864": { "delta": 0.2473, "median": -0.01342, "ci": null, "n": 18, "wins": 11, "ties": 0, "clusters": 3, "cluster_wins": 1 } } }, { "key": "offcentre_lat_jerk", "value": 0.4733, "median": 0.4115, "ci": null, "n": 18, "clusters": 3, "paired": { "lebowski_877m": { "delta": -0.4085, "median": -0.4364, "ci": null, "n": 18, "wins": 17, "ties": 0, "clusters": 3, "cluster_wins": 3 }, "1365126a_12864": { "delta": 0.03802, "median": 0.01152, "ci": null, "n": 18, "wins": 6, "ties": 0, "clusters": 3, "cluster_wins": 0 } } } ] }, "v14-8mm": { "n": 19, "clusters": 2, "metrics": [ { "key": "heading_xtrack_2s", "value": 0.3894, "median": 0.382, "ci": null, "n": 19, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.03393, "median": 0.01786, "ci": null, "n": 19, "wins": 5, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "1365126a_12864": { "delta": -0.0445, "median": -0.04619, "ci": null, "n": 19, "wins": 18, "ties": 0, "clusters": 2, "cluster_wins": 2 } } }, { "key": "heading_xtrack_4s", "value": 0.7691, "median": 0.3939, "ci": null, "n": 19, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.2056, "median": 0.08804, "ci": null, "n": 19, "wins": 6, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "1365126a_12864": { "delta": -0.03199, "median": -0.04137, "ci": null, "n": 19, "wins": 13, "ties": 0, "clusters": 2, "cluster_wins": 1 } } }, { "key": "heading_lat_jerk", "value": 0.2964, "median": 0.2758, "ci": null, "n": 19, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.652, "median": -0.5428, "ci": null, "n": 19, "wins": 19, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "1365126a_12864": { "delta": -0.01876, "median": 0.003068, "ci": null, "n": 19, "wins": 8, "ties": 0, "clusters": 2, "cluster_wins": 2 } } }, { "key": "offcentre_xtrack_2s", "value": 0.2182, "median": 0.1938, "ci": null, "n": 19, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.01975, "median": 0.04353, "ci": null, "n": 19, "wins": 5, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "1365126a_12864": { "delta": -0.03791, "median": -0.04795, "ci": null, "n": 19, "wins": 17, "ties": 0, "clusters": 2, "cluster_wins": 2 } } }, { "key": "offcentre_xtrack_4s", "value": 0.7484, "median": 0.3665, "ci": null, "n": 19, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.09096, "median": -0.04153, "ci": null, "n": 19, "wins": 12, "ties": 0, "clusters": 2, "cluster_wins": 1 }, "1365126a_12864": { "delta": -0.01721, "median": -0.02399, "ci": null, "n": 19, "wins": 12, "ties": 0, "clusters": 2, "cluster_wins": 1 } } }, { "key": "offcentre_lat_jerk", "value": 0.4533, "median": 0.4297, "ci": null, "n": 19, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.6992, "median": -0.6514, "ci": null, "n": 19, "wins": 19, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "1365126a_12864": { "delta": 0.02929, "median": 0.0246, "ci": null, "n": 19, "wins": 2, "ties": 0, "clusters": 2, "cluster_wins": 0 } } } ] } }, "simulator": "g55-2026-10-31" } }, "stills": [], "limits": [ "Open-loop scores compare a model's action with what the recording driver or car did; a model that drives differently but well is scored as wrong. comma's RL residual makes the lateral action smoother, and earlier than its base's by 0.10 to 0.26 s (measured on the G55 and on 16 of comma's cars); both score as error here. Measured on the 2026-10-27 deep dive: open-loop accuracy did not predict closed-loop driving.", "Closed-loop numbers come from a generative world model, not a car. In a separate test on the G55's windows, worldmodel-4B followed an added yaw command through 2 s but showed 0 to 20% of an added lateral offset 3 to 4 s later; past about 4 s the recorded anchor frames also pull every model back toward the recorded path. The front page reports 2 s.", "Intervals are 95% percentile bootstraps that resample whole drives (the drive a public segment was cut from, the recorded drive a truck window comes from), so segments of one drive count once. Below 10 drives there is no interval: the braking-lead test comes from one drive and reports means and win counts only.", "Paired deltas are model minus baseline on the units both have (for lateral gain, the change in its distance from 1); wins count units where the model is better in the metric's direction.", "The lead of the lateral request is the shift, searched from 0 to 1.5 s, at which the request best correlates with the car's later lateral motion; a larger value means the request comes earlier. A value at either edge is censored and left out. Plan speed error at 3 s reads the plan head, which does not separate models built on one frozen base: it is identical on every segment for the four f22e06e5 models, and for Cinque v2 and v3.", "Truck open loop comes from few drives, and the camera setup changed between them (lens, housing, aim); the pooled number mixes setups, so read the per-setup numbers too. The 8 mm setups' wide camera recorded about 7-10 fps, so its frame can be up to ~60 ms from the narrow one. On the V14 setup up to 7 of a window's 700 steps are ~100 ms (one camera frame dropped). The camera's aim is the truck's live calibration learned on each drive; drives whose recorded calibration was carried over from another drive are left out.", "Truck closed loop: the steering and powertrain simulation is fitted to the G55, not validated on it in closed loop; the reference is the driver's own line, not the lane centre. The pooled number mixes camera setups, so read the per-setup numbers too. On the V14 setup the wide frame can be up to 90 ms from the simulated one (its camera recorded 7-9 fps).", "Recovery tests are disturbances rendered by the world model (yaw it renders; a one-frame jump it does not, so none is used). The braking-lead test has 2 windows from one drive, whose recorded calibration was carried over from an earlier drive.", "This model takes a navigation map and route; every run here feeds zeros (no route), which is not how it would be used with navigation." ] }