{ "schema": 1, "model": { "id": "6b5b705f_12800", "name": null, "sha256": "ceb38c593005b3338aee4ca057d5ad9ee64c58f8fdba0b41641a9d4cc6c36e36", "source": { "kind": "huggingface", "ref": "eed9e515dccd5aa9be969a2d6c57eb469174a97d", "url": "https://huggingface.co/commaai/openpilot_driving_models/blob/eed9e515dccd5aa9be969a2d6c57eb469174a97d/6b5b705f-4a05-4f30-9c37-9c51b8dc5ffc/12800/big_driving_supercombo.onnx", "date": "2026-10-17" }, "first_public": "2026-10-17", "on_master": null, "params_m": 382.9, "gmac": 50.1, "family": "residual", "notes": "Never shipped. Same frozen base as MD v3 (f22e06e5/56320, bit-identical), different RL residual head." }, "scored": { "date": "2026-10-31T23:59:36Z", "suite": "geval-1", "code": "d8eb40eeba87" }, "baselines": [ "lebowski_877m", "96244b1f_12800" ], "sections": { "public_open_loop": { "status": "ok", "unit_of_evidence": "segment", "n": 39, "clusters": 23, "condition": "Measured. comma's public two-camera segments (openpilot CI data), native intrinsics and the device's own calibration, both cameras; every model on the same decoded frames. Truth is what the recording car did next: carState speed and acceleration, yaw rate from the device's cameraOdometry. Segments where the car never moves have no data. 39 of the 45 listed segments have a score, from 23 drives and 20 cars.", "baselines": [ "lebowski_877m", "96244b1f_12800" ], "metrics": [ { "key": "lat_mae", "label": "Lateral error", "unit": "m/s²", "better": "lower", "value": 0.1369, "median": 0.1184, "ci": [ 0.1083, 0.168 ], "n": 39, "clusters": 23, "paired": { "lebowski_877m": { "delta": 0.02815, "median": 0.01788, "ci": [ 0.02008, 0.03738 ], "n": 39, "wins": 4, "ties": 0, "clusters": 23, "cluster_wins": 2 }, "96244b1f_12800": { "delta": -0.01504, "median": -0.008707, "ci": [ -0.02301, -0.006733 ], "n": 39, "wins": 25, "ties": 0, "clusters": 23, "cluster_wins": 16 } } }, { "key": "lat_mae_bl", "label": "Lateral error at the model's own best lead", "unit": "m/s²", "better": "lower", "value": 0.1255, "median": 0.1084, "ci": [ 0.09784, 0.1546 ], "n": 39, "clusters": 23, "paired": { "lebowski_877m": { "delta": 0.02412, "median": 0.0193, "ci": [ 0.0154, 0.03265 ], "n": 39, "wins": 4, "ties": 0, "clusters": 23, "cluster_wins": 2 }, "96244b1f_12800": { "delta": -0.01633, "median": -0.005026, "ci": [ -0.02588, -0.006817 ], "n": 39, "wins": 25, "ties": 0, "clusters": 23, "cluster_wins": 14 } } }, { "key": "lat_corr", "label": "Lateral correlation with the car's later motion", "unit": "", "better": "higher", "value": 0.8367, "median": 0.9426, "ci": [ 0.7483, 0.9117 ], "n": 39, "clusters": 23, "paired": { "lebowski_877m": { "delta": -0.02089, "median": -0.006692, "ci": [ -0.06623, 0.01376 ], "n": 39, "wins": 10, "ties": 0, "clusters": 23, "cluster_wins": 9 }, "96244b1f_12800": { "delta": 0.009036, "median": 0.007575, "ci": [ -0.001488, 0.02041 ], "n": 39, "wins": 28, "ties": 0, "clusters": 23, "cluster_wins": 15 } } }, { "key": "lat_gain", "label": "Lateral gain (1 = same size as the car's motion)", "unit": "", "better": "near1", "value": 0.823, "median": 0.8789, "ci": [ 0.7243, 0.9204 ], "n": 39, "clusters": 23, "paired": { "lebowski_877m": { "delta": 0.02818, "median": 0.03317, "ci": [ -0.03558, 0.08089 ], "n": 39, "wins": 13, "ties": 0, "clusters": 23, "cluster_wins": 8 }, "96244b1f_12800": { "delta": -0.0638, "median": -0.05086, "ci": [ -0.09652, -0.03529 ], "n": 39, "wins": 31, "ties": 0, "clusters": 23, "cluster_wins": 21 } } }, { "key": "lat_rate_rms", "label": "Lateral action rate, RMS (smoothness)", "unit": "m/s³", "better": "lower", "value": 0.4637, "median": 0.4102, "ci": [ 0.3706, 0.5709 ], "n": 39, "clusters": 23, "paired": { "lebowski_877m": { "delta": -0.2002, "median": -0.1941, "ci": [ -0.242, -0.1554 ], "n": 39, "wins": 38, "ties": 0, "clusters": 23, "cluster_wins": 23 }, "96244b1f_12800": { "delta": -0.007393, "median": -0.005065, "ci": [ -0.01968, 0.007545 ], "n": 39, "wins": 25, "ties": 0, "clusters": 23, "cluster_wins": 18 } } }, { "key": "lat_best_lag", "label": "Lead of the lateral request over the car's motion", "unit": "s", "better": "lower", "value": 0.5583, "median": 0.475, "ci": [ 0.4456, 0.6803 ], "n": 36, "clusters": 22, "paired": { "lebowski_877m": { "delta": 0.1686, "median": 0.05, "ci": [ 0.075, 0.2783 ], "n": 35, "wins": 4, "ties": 9, "clusters": 22, "cluster_wins": 2 }, "96244b1f_12800": { "delta": 1.269e-17, "median": -0.05, "ci": [ -0.04878, 0.06071 ], "n": 35, "wins": 18, "ties": 10, "clusters": 22, "cluster_wins": 14 } } }, { "key": "lon_mae", "label": "Longitudinal error", "unit": "m/s²", "better": "lower", "value": 0.3679, "median": 0.3559, "ci": [ 0.3214, 0.4163 ], "n": 39, "clusters": 23, "paired": { "lebowski_877m": { "delta": 0.05977, "median": 0.04645, "ci": [ 0.03506, 0.08638 ], "n": 39, "wins": 5, "ties": 0, "clusters": 23, "cluster_wins": 3 }, "96244b1f_12800": { "delta": 0.05125, "median": 0.0404, "ci": [ 0.03635, 0.06711 ], "n": 39, "wins": 3, "ties": 0, "clusters": 23, "cluster_wins": 0 } } }, { "key": "lon_corr", "label": "Longitudinal correlation", "unit": "", "better": "higher", "value": 0.6744, "median": 0.8139, "ci": [ 0.5505, 0.7805 ], "n": 39, "clusters": 23, "paired": { "lebowski_877m": { "delta": -0.04338, "median": -0.02692, "ci": [ -0.08778, -0.003138 ], "n": 39, "wins": 15, "ties": 0, "clusters": 23, "cluster_wins": 8 }, "96244b1f_12800": { "delta": -0.05438, "median": -0.03141, "ci": [ -0.08911, -0.026 ], "n": 39, "wins": 6, "ties": 0, "clusters": 23, "cluster_wins": 2 } } }, { "key": "lon_brake_mae", "label": "Longitudinal error while braking", "unit": "m/s²", "better": "lower", "value": 0.4605, "median": 0.4437, "ci": [ 0.3923, 0.5385 ], "n": 30, "clusters": 21, "paired": { "lebowski_877m": { "delta": 0.02619, "median": 0.03725, "ci": [ -0.005091, 0.06369 ], "n": 30, "wins": 10, "ties": 0, "clusters": 21, "cluster_wins": 7 }, "96244b1f_12800": { "delta": 0.02422, "median": 0.04011, "ci": [ -3.034e-05, 0.04829 ], "n": 30, "wins": 8, "ties": 0, "clusters": 21, "cluster_wins": 6 } } }, { "key": "lon_rate_rms", "label": "Longitudinal action rate, RMS (smoothness)", "unit": "m/s³", "better": "lower", "value": 1.013, "median": 0.9455, "ci": [ 0.8776, 1.16 ], "n": 39, "clusters": 23, "paired": { "lebowski_877m": { "delta": -0.3197, "median": -0.2801, "ci": [ -0.4533, -0.2006 ], "n": 39, "wins": 33, "ties": 0, "clusters": 23, "cluster_wins": 21 }, "96244b1f_12800": { "delta": 0.1128, "median": 0.08819, "ci": [ 0.0607, 0.166 ], "n": 39, "wins": 9, "ties": 0, "clusters": 23, "cluster_wins": 5 } } }, { "key": "v3_mae", "label": "Plan speed error at 3 s", "unit": "m/s", "better": "lower", "value": 0.7602, "median": 0.7121, "ci": [ 0.6567, 0.888 ], "n": 39, "clusters": 23, "paired": { "lebowski_877m": { "delta": 0.04442, "median": 0.04624, "ci": [ 0.009032, 0.07746 ], "n": 39, "wins": 12, "ties": 0, "clusters": 23, "cluster_wins": 8 }, "96244b1f_12800": { "delta": -0.006492, "median": -0.005619, "ci": [ -0.02415, 0.01238 ], "n": 39, "wins": 21, "ties": 0, "clusters": 23, "cluster_wins": 13 } } } ], "per_unit_file": "per_unit/6b5b705f_12800/public_open_loop.csv" }, "public_closed_loop": { "status": "ok", "unit_of_evidence": "segment", "n": 28, "clusters": 19, "condition": "Measured. comma's worldmodel-4B generates each next frame at 5 Hz from 9 history frames, 5 recorded anchor frames and the car's pose; the model drives comma's ideal car (the requested curvature and acceleration happen at once) for 7.2 s from the recorded start. Distance is to the recording car's own path. One window per segment, nearest the middle, where the car holds at least 5 m/s for the window and the 4.8 s before it; segments with no such stretch are left out. Seeds 0 and 1, averaged per segment. 28 of the 45 listed segments have a score, from 19 drives and 17 cars.", "baselines": [ "lebowski_877m", "96244b1f_12800" ], "metrics": [ { "key": "xtrack_2s", "label": "Distance from the recorded path at 2 s", "unit": "m", "better": "lower", "value": 0.1146, "median": 0.05992, "ci": [ 0.05477, 0.2194 ], "n": 28, "clusters": 19, "paired": { "lebowski_877m": { "delta": -0.03313, "median": -0.01006, "ci": [ -0.06378, -0.008695 ], "n": 28, "wins": 17, "ties": 0, "clusters": 19, "cluster_wins": 12 }, "96244b1f_12800": { "delta": 0.02619, "median": 0.007084, "ci": [ 0.001437, 0.06237 ], "n": 28, "wins": 11, "ties": 0, "clusters": 19, "cluster_wins": 6 } } }, { "key": "xtrack_4s", "label": "Distance from the recorded path at 4 s", "unit": "m", "better": "lower", "value": 0.2413, "median": 0.1199, "ci": [ 0.1071, 0.4602 ], "n": 28, "clusters": 19, "paired": { "lebowski_877m": { "delta": -0.03702, "median": -0.00498, "ci": [ -0.07501, -0.002356 ], "n": 28, "wins": 14, "ties": 0, "clusters": 19, "cluster_wins": 10 }, "96244b1f_12800": { "delta": 0.07103, "median": -0.00634, "ci": [ -0.00684, 0.2152 ], "n": 28, "wins": 15, "ties": 0, "clusters": 19, "cluster_wins": 9 } } }, { "key": "xtrack_7s", "label": "Distance from the recorded path at 7.2 s", "unit": "m", "better": "lower", "value": 1.125, "median": 0.3277, "ci": [ 0.4077, 2.048 ], "n": 28, "clusters": 19, "paired": { "lebowski_877m": { "delta": 0.08104, "median": 0.04283, "ci": [ -0.5261, 0.6981 ], "n": 28, "wins": 12, "ties": 0, "clusters": 19, "cluster_wins": 6 }, "96244b1f_12800": { "delta": -0.003708, "median": 0.01813, "ci": [ -0.2938, 0.2499 ], "n": 28, "wins": 12, "ties": 0, "clusters": 19, "cluster_wins": 7 } } }, { "key": "fail_1m", "label": "Share of episodes ever more than 1 m off the path", "unit": "", "better": "lower", "value": 0.2143, "median": 0.0, "ci": [ 0.07143, 0.375 ], "n": 28, "clusters": 19, "paired": { "lebowski_877m": { "delta": -0.01786, "median": 0.0, "ci": [ -0.1111, 0.05172 ], "n": 28, "wins": 1, "ties": 26, "clusters": 19, "cluster_wins": 1 }, "96244b1f_12800": { "delta": -0.01786, "median": 0.0, "ci": [ -0.125, 0.07143 ], "n": 28, "wins": 2, "ties": 24, "clusters": 19, "cluster_wins": 2 } } }, { "key": "lat_jerk", "label": "Requested lateral jerk, RMS", "unit": "m/s³", "better": "lower", "value": 0.1923, "median": 0.09458, "ci": [ 0.1228, 0.266 ], "n": 28, "clusters": 19, "paired": { "lebowski_877m": { "delta": -0.1333, "median": -0.09429, "ci": [ -0.1924, -0.07745 ], "n": 28, "wins": 25, "ties": 0, "clusters": 19, "cluster_wins": 16 }, "96244b1f_12800": { "delta": -0.01263, "median": -0.008068, "ci": [ -0.03973, 0.01226 ], "n": 28, "wins": 18, "ties": 0, "clusters": 19, "cluster_wins": 15 } } }, { "key": "long_jerk", "label": "Requested longitudinal jerk, RMS", "unit": "m/s³", "better": "lower", "value": 0.3439, "median": 0.3176, "ci": [ 0.2698, 0.434 ], "n": 28, "clusters": 19, "paired": { "lebowski_877m": { "delta": -0.002534, "median": -0.04448, "ci": [ -0.04653, 0.05358 ], "n": 28, "wins": 18, "ties": 0, "clusters": 19, "cluster_wins": 12 }, "96244b1f_12800": { "delta": -0.04659, "median": -0.01749, "ci": [ -0.1036, 0.004842 ], "n": 28, "wins": 17, "ties": 0, "clusters": 19, "cluster_wins": 12 } } }, { "key": "speed_err", "label": "Speed error against the recorded speed, RMS", "unit": "m/s", "better": "lower", "value": 1.357, "median": 0.7189, "ci": [ 0.9319, 1.783 ], "n": 28, "clusters": 19, "paired": { "lebowski_877m": { "delta": 0.2562, "median": 0.2651, "ci": [ -0.01344, 0.5088 ], "n": 28, "wins": 8, "ties": 0, "clusters": 19, "cluster_wins": 6 }, "96244b1f_12800": { "delta": -0.0441, "median": -0.08008, "ci": [ -0.2081, 0.0991 ], "n": 28, "wins": 18, "ties": 0, "clusters": 19, "cluster_wins": 12 } } } ], "per_unit_file": "per_unit/6b5b705f_12800/public_closed_loop.csv" }, "truck_open_loop": { "status": "ok", "unit_of_evidence": "window", "n": 119, "clusters": 12, "condition": "Measured. The G55's recorded drives replayed through the model with both of the truck's cameras (each setup's own wide-camera calibration), road-plane warp to 1.45 m. Where the narrow lens does not cover comma's narrow view (the 8 mm lens in the V13 case), those pixels come from the wide camera, brightness matched. Truth is the truck's CAN speed and yaw rate. 30 s windows after a 5 s warm-up; window = unit, drive = cluster; results are also given per camera setup. Corpus windows of 30 s from the truck's recorded drives, scored per camera setup.", "baselines": [ "lebowski_877m", "96244b1f_12800" ], "metrics": [ { "key": "lat_mae", "label": "Lateral error", "unit": "m/s²", "better": "lower", "value": 0.1614, "median": 0.1502, "ci": [ 0.1353, 0.1707 ], "n": 119, "clusters": 12, "paired": { "lebowski_877m": { "delta": -0.01134, "median": -0.008984, "ci": [ -0.01592, 0.001323 ], "n": 119, "wins": 73, "ties": 0, "clusters": 12, "cluster_wins": 7 }, "96244b1f_12800": { "delta": -0.009251, "median": -0.004847, "ci": [ -0.0168, -0.001451 ], "n": 119, "wins": 69, "ties": 0, "clusters": 12, "cluster_wins": 9 } } }, { "key": "lat_mae_bl", "label": "Lateral error at the model's own best lead", "unit": "m/s²", "better": "lower", "value": 0.1496, "median": 0.14, "ci": [ 0.1256, 0.1577 ], "n": 119, "clusters": 12, "paired": { "lebowski_877m": { "delta": -0.01489, "median": -0.01318, "ci": [ -0.01874, -0.004234 ], "n": 119, "wins": 86, "ties": 0, "clusters": 12, "cluster_wins": 8 }, "96244b1f_12800": { "delta": -0.0002657, "median": 0.00349, "ci": [ -0.01106, 0.005904 ], "n": 119, "wins": 50, "ties": 0, "clusters": 12, "cluster_wins": 8 } } }, { "key": "lat_corr", "label": "Lateral correlation with the car's later motion", "unit": "", "better": "higher", "value": 0.8163, "median": 0.9042, "ci": [ 0.7746, 0.8415 ], "n": 119, "clusters": 12, "paired": { "lebowski_877m": { "delta": 0.04946, "median": 0.0199, "ci": [ 0.01328, 0.05945 ], "n": 119, "wins": 91, "ties": 0, "clusters": 12, "cluster_wins": 7 }, "96244b1f_12800": { "delta": 0.01949, "median": 0.01035, "ci": [ 0.01563, 0.02947 ], "n": 119, "wins": 94, "ties": 0, "clusters": 12, "cluster_wins": 12 } } }, { "key": "lat_gain", "label": "Lateral gain (1 = same size as the car's motion)", "unit": "", "better": "near1", "value": 0.8338, "median": 0.8515, "ci": [ 0.8153, 0.9043 ], "n": 119, "clusters": 12, "paired": { "lebowski_877m": { "delta": -0.02783, "median": -0.01498, "ci": [ -0.04602, 0.01012 ], "n": 119, "wins": 68, "ties": 0, "clusters": 12, "cluster_wins": 7 }, "96244b1f_12800": { "delta": -0.0505, "median": -0.04659, "ci": [ -0.06485, -0.03052 ], "n": 119, "wins": 93, "ties": 0, "clusters": 12, "cluster_wins": 12 } } }, { "key": "lat_rate_rms", "label": "Lateral action rate, RMS (smoothness)", "unit": "m/s³", "better": "lower", "value": 1.155, "median": 0.8631, "ci": [ 0.6521, 1.384 ], "n": 119, "clusters": 12, "paired": { "lebowski_877m": { "delta": -1.135, "median": -1.021, "ci": [ -1.255, -0.5102 ], "n": 119, "wins": 117, "ties": 0, "clusters": 12, "cluster_wins": 12 }, "96244b1f_12800": { "delta": 0.07174, "median": 0.03648, "ci": [ -0.0406, 0.1113 ], "n": 119, "wins": 42, "ties": 0, "clusters": 12, "cluster_wins": 6 } } }, { "key": "lat_best_lag", "label": "Lead of the lateral request over the car's motion", "unit": "s", "better": "lower", "value": 0.6195, "median": 0.6, "ci": [ 0.5805, 0.6833 ], "n": 118, "clusters": 12, "paired": { "lebowski_877m": { "delta": 0.002966, "median": 0.0, "ci": [ -0.06627, 0.1281 ], "n": 118, "wins": 41, "ties": 21, "clusters": 12, "cluster_wins": 3 }, "96244b1f_12800": { "delta": -0.1318, "median": -0.1, "ci": [ -0.15, -0.08855 ], "n": 118, "wins": 101, "ties": 13, "clusters": 12, "cluster_wins": 12 } } }, { "key": "lon_mae", "label": "Longitudinal error", "unit": "m/s²", "better": "lower", "value": 0.3679, "median": 0.3688, "ci": [ 0.2826, 0.4183 ], "n": 119, "clusters": 12, "paired": { "lebowski_877m": { "delta": -0.0007675, "median": 0.006922, "ci": [ -0.04831, 0.03055 ], "n": 119, "wins": 56, "ties": 0, "clusters": 12, "cluster_wins": 4 }, "96244b1f_12800": { "delta": -0.008462, "median": -0.01812, "ci": [ -0.01967, 0.04213 ], "n": 119, "wins": 80, "ties": 0, "clusters": 12, "cluster_wins": 4 } } }, { "key": "lon_corr", "label": "Longitudinal correlation", "unit": "", "better": "higher", "value": 0.308, "median": 0.2773, "ci": [ 0.2607, 0.5221 ], "n": 119, "clusters": 12, "paired": { "lebowski_877m": { "delta": -0.01888, "median": 0.003423, "ci": [ -0.09795, 0.02149 ], "n": 119, "wins": 61, "ties": 0, "clusters": 12, "cluster_wins": 4 }, "96244b1f_12800": { "delta": 0.007006, "median": 0.005748, "ci": [ -0.09195, 0.0319 ], "n": 119, "wins": 63, "ties": 0, "clusters": 12, "cluster_wins": 4 } } }, { "key": "lon_brake_mae", "label": "Longitudinal error while braking", "unit": "m/s²", "better": "lower", "value": 0.4888, "median": 0.4509, "ci": [ 0.4419, 0.6888 ], "n": 81, "clusters": 12, "paired": { "lebowski_877m": { "delta": -0.01004, "median": -0.03496, "ci": [ -0.04034, 0.1452 ], "n": 81, "wins": 49, "ties": 0, "clusters": 12, "cluster_wins": 4 }, "96244b1f_12800": { "delta": 0.02448, "median": 0.01316, "ci": [ 0.00172, 0.1058 ], "n": 81, "wins": 36, "ties": 0, "clusters": 12, "cluster_wins": 3 } } }, { "key": "lon_rate_rms", "label": "Longitudinal action rate, RMS (smoothness)", "unit": "m/s³", "better": "lower", "value": 4.475, "median": 4.233, "ci": [ 2.147, 5.808 ], "n": 119, "clusters": 12, "paired": { "lebowski_877m": { "delta": -1.135, "median": -1.195, "ci": [ -1.71, -0.6833 ], "n": 119, "wins": 91, "ties": 0, "clusters": 12, "cluster_wins": 11 }, "96244b1f_12800": { "delta": -1.096, "median": -1.167, "ci": [ -1.349, 0.01694 ], "n": 119, "wins": 87, "ties": 0, "clusters": 12, "cluster_wins": 6 } } }, { "key": "v3_mae", "label": "Plan speed error at 3 s", "unit": "m/s", "better": "lower", "value": 1.889, "median": 1.838, "ci": [ 1.152, 2.301 ], "n": 119, "clusters": 12, "paired": { "lebowski_877m": { "delta": -0.1086, "median": -0.1249, "ci": [ -0.3518, 0.306 ], "n": 119, "wins": 78, "ties": 0, "clusters": 12, "cluster_wins": 5 }, "96244b1f_12800": { "delta": -0.021, "median": -0.04143, "ci": [ -0.2468, 0.4093 ], "n": 119, "wins": 64, "ties": 0, "clusters": 12, "cluster_wins": 3 } } } ], "per_unit_file": null, "by_group": { "stock-aimedup": { "n": 4, "clusters": 2, "metrics": [ { "key": "lat_mae", "value": 0.1142, "median": 0.1131, "ci": null, "n": 4, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.02689, "median": -0.03775, "ci": null, "n": 4, "wins": 3, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "96244b1f_12800": { "delta": -0.02199, "median": -0.02577, "ci": null, "n": 4, "wins": 3, "ties": 0, "clusters": 2, "cluster_wins": 2 } } }, { "key": "lat_mae_bl", "value": 0.1068, "median": 0.1086, "ci": null, "n": 4, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.03276, "median": -0.04683, "ci": null, "n": 4, "wins": 3, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "96244b1f_12800": { "delta": -0.006842, "median": -0.01195, "ci": null, "n": 4, "wins": 3, "ties": 0, "clusters": 2, "cluster_wins": 2 } } }, { "key": "lat_corr", "value": 0.8397, "median": 0.9121, "ci": null, "n": 4, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.04457, "median": 0.0328, "ci": null, "n": 4, "wins": 3, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "96244b1f_12800": { "delta": 0.06793, "median": 0.05326, "ci": null, "n": 4, "wins": 4, "ties": 0, "clusters": 2, "cluster_wins": 2 } } }, { "key": "lat_gain", "value": 0.8156, "median": 0.9058, "ci": null, "n": 4, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.1491, "median": -0.1426, "ci": null, "n": 4, "wins": 3, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "96244b1f_12800": { "delta": -0.1341, "median": -0.1775, "ci": null, "n": 4, "wins": 3, "ties": 0, "clusters": 2, "cluster_wins": 2 } } }, { "key": "lat_rate_rms", "value": 0.8413, "median": 0.7583, "ci": null, "n": 4, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.3116, "median": -0.3536, "ci": null, "n": 4, "wins": 4, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "96244b1f_12800": { "delta": 0.04799, "median": -0.03009, "ci": null, "n": 4, "wins": 2, "ties": 0, "clusters": 2, "cluster_wins": 1 } } }, { "key": "lat_best_lag", "value": 0.625, "median": 0.625, "ci": null, "n": 4, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.1125, "median": 0.05, "ci": null, "n": 4, "wins": 0, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "96244b1f_12800": { "delta": -0.2375, "median": -0.2, "ci": null, "n": 4, "wins": 4, "ties": 0, "clusters": 2, "cluster_wins": 2 } } }, { "key": "lon_mae", "value": 0.4179, "median": 0.4099, "ci": null, "n": 4, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.3598, "median": -0.3556, "ci": null, "n": 4, "wins": 4, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "96244b1f_12800": { "delta": 0.03206, "median": 0.02776, "ci": null, "n": 4, "wins": 1, "ties": 0, "clusters": 2, "cluster_wins": 1 } } }, { "key": "lon_corr", "value": 0.4114, "median": 0.357, "ci": null, "n": 4, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.124, "median": -0.04266, "ci": null, "n": 4, "wins": 2, "ties": 0, "clusters": 2, "cluster_wins": 1 }, "96244b1f_12800": { "delta": -0.2632, "median": -0.2089, "ci": null, "n": 4, "wins": 1, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "lon_brake_mae", "value": 0.6758, "median": 0.6758, "ci": null, "n": 2, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.4878, "median": 0.4878, "ci": null, "n": 2, "wins": 0, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "96244b1f_12800": { "delta": 0.1509, "median": 0.1509, "ci": null, "n": 2, "wins": 0, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "lon_rate_rms", "value": 2.253, "median": 2.175, "ci": null, "n": 4, "clusters": 2, "paired": { "lebowski_877m": { "delta": -1.042, "median": -1.023, "ci": null, "n": 4, "wins": 4, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "96244b1f_12800": { "delta": 0.5473, "median": 0.5465, "ci": null, "n": 4, "wins": 0, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "v3_mae", "value": 1.443, "median": 1.507, "ci": null, "n": 4, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.9457, "median": -0.9688, "ci": null, "n": 4, "wins": 4, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "96244b1f_12800": { "delta": -0.03193, "median": 0.03439, "ci": null, "n": 4, "wins": 2, "ties": 0, "clusters": 2, "cluster_wins": 1 } } } ] }, "stock-dual": { "n": 13, "clusters": 4, "metrics": [ { "key": "lat_mae", "value": 0.1428, "median": 0.1265, "ci": null, "n": 13, "clusters": 4, "paired": { "lebowski_877m": { "delta": -0.0024, "median": -0.0008259, "ci": null, "n": 13, "wins": 8, "ties": 0, "clusters": 4, "cluster_wins": 2 }, "96244b1f_12800": { "delta": -0.01988, "median": -0.0168, "ci": null, "n": 13, "wins": 11, "ties": 0, "clusters": 4, "cluster_wins": 4 } } }, { "key": "lat_mae_bl", "value": 0.1303, "median": 0.1181, "ci": null, "n": 13, "clusters": 4, "paired": { "lebowski_877m": { "delta": -0.00896, "median": -0.007424, "ci": null, "n": 13, "wins": 8, "ties": 0, "clusters": 4, "cluster_wins": 3 }, "96244b1f_12800": { "delta": -0.01576, "median": -0.01115, "ci": null, "n": 13, "wins": 10, "ties": 0, "clusters": 4, "cluster_wins": 3 } } }, { "key": "lat_corr", "value": 0.8287, "median": 0.8487, "ci": null, "n": 13, "clusters": 4, "paired": { "lebowski_877m": { "delta": 0.01541, "median": -0.01087, "ci": null, "n": 13, "wins": 6, "ties": 0, "clusters": 4, "cluster_wins": 2 }, "96244b1f_12800": { "delta": 0.02594, "median": 0.02434, "ci": null, "n": 13, "wins": 12, "ties": 0, "clusters": 4, "cluster_wins": 4 } } }, { "key": "lat_gain", "value": 0.9343, "median": 0.9127, "ci": null, "n": 13, "clusters": 4, "paired": { "lebowski_877m": { "delta": 0.02561, "median": 0.03945, "ci": null, "n": 13, "wins": 4, "ties": 0, "clusters": 4, "cluster_wins": 1 }, "96244b1f_12800": { "delta": -0.03564, "median": -0.02388, "ci": null, "n": 13, "wins": 11, "ties": 0, "clusters": 4, "cluster_wins": 4 } } }, { "key": "lat_rate_rms", "value": 0.6457, "median": 0.6143, "ci": null, "n": 13, "clusters": 4, "paired": { "lebowski_877m": { "delta": -0.5302, "median": -0.5078, "ci": null, "n": 13, "wins": 13, "ties": 0, "clusters": 4, "cluster_wins": 4 }, "96244b1f_12800": { "delta": -0.1025, "median": -0.1183, "ci": null, "n": 13, "wins": 10, "ties": 0, "clusters": 4, "cluster_wins": 3 } } }, { "key": "lat_best_lag", "value": 0.6692, "median": 0.6, "ci": null, "n": 13, "clusters": 4, "paired": { "lebowski_877m": { "delta": 0.1577, "median": 0.1, "ci": null, "n": 13, "wins": 1, "ties": 3, "clusters": 4, "cluster_wins": 0 }, "96244b1f_12800": { "delta": -0.09615, "median": -0.05, "ci": null, "n": 13, "wins": 9, "ties": 3, "clusters": 4, "cluster_wins": 4 } } }, { "key": "lon_mae", "value": 0.3086, "median": 0.2899, "ci": null, "n": 13, "clusters": 4, "paired": { "lebowski_877m": { "delta": 0.08036, "median": 0.07651, "ci": null, "n": 13, "wins": 1, "ties": 0, "clusters": 4, "cluster_wins": 0 }, "96244b1f_12800": { "delta": 0.04748, "median": 0.04867, "ci": null, "n": 13, "wins": 2, "ties": 0, "clusters": 4, "cluster_wins": 0 } } }, { "key": "lon_corr", "value": 0.5231, "median": 0.6524, "ci": null, "n": 13, "clusters": 4, "paired": { "lebowski_877m": { "delta": -0.1111, "median": -0.05026, "ci": null, "n": 13, "wins": 3, "ties": 0, "clusters": 4, "cluster_wins": 0 }, "96244b1f_12800": { "delta": -0.09688, "median": -0.0715, "ci": null, "n": 13, "wins": 2, "ties": 0, "clusters": 4, "cluster_wins": 0 } } }, { "key": "lon_brake_mae", "value": 0.745, "median": 0.8075, "ci": null, "n": 6, "clusters": 4, "paired": { "lebowski_877m": { "delta": 0.09776, "median": 0.1083, "ci": null, "n": 6, "wins": 2, "ties": 0, "clusters": 4, "cluster_wins": 1 }, "96244b1f_12800": { "delta": 0.1223, "median": 0.1627, "ci": null, "n": 6, "wins": 1, "ties": 0, "clusters": 4, "cluster_wins": 1 } } }, { "key": "lon_rate_rms", "value": 1.33, "median": 1.295, "ci": null, "n": 13, "clusters": 4, "paired": { "lebowski_877m": { "delta": -0.8838, "median": -0.8482, "ci": null, "n": 13, "wins": 13, "ties": 0, "clusters": 4, "cluster_wins": 4 }, "96244b1f_12800": { "delta": 0.08627, "median": 0.077, "ci": null, "n": 13, "wins": 4, "ties": 0, "clusters": 4, "cluster_wins": 1 } } }, { "key": "v3_mae", "value": 0.8066, "median": 0.8323, "ci": null, "n": 13, "clusters": 4, "paired": { "lebowski_877m": { "delta": -0.03457, "median": -0.06044, "ci": null, "n": 13, "wins": 8, "ties": 0, "clusters": 4, "cluster_wins": 2 }, "96244b1f_12800": { "delta": 0.01884, "median": 0.0316, "ci": null, "n": 13, "wins": 5, "ties": 0, "clusters": 4, "cluster_wins": 1 } } } ] }, "stock-newcase": { "n": 2, "clusters": 1, "metrics": [ { "key": "lat_mae", "value": 0.1501, "median": 0.1501, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": 0.01583, "median": 0.01583, "ci": null, "n": 2, "wins": 0, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "96244b1f_12800": { "delta": 0.0008618, "median": 0.0008618, "ci": null, "n": 2, "wins": 0, "ties": 0, "clusters": 1, "cluster_wins": 0 } } }, { "key": "lat_mae_bl", "value": 0.145, "median": 0.145, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": 0.009851, "median": 0.009851, "ci": null, "n": 2, "wins": 0, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "96244b1f_12800": { "delta": 0.0005646, "median": 0.0005646, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 } } }, { "key": "lat_corr", "value": 0.8155, "median": 0.8155, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.03205, "median": -0.03205, "ci": null, "n": 2, "wins": 0, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "96244b1f_12800": { "delta": 0.001961, "median": 0.001961, "ci": null, "n": 2, "wins": 2, "ties": 0, "clusters": 1, "cluster_wins": 1 } } }, { "key": "lat_gain", "value": 0.8059, "median": 0.8059, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.01598, "median": -0.01598, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "96244b1f_12800": { "delta": -0.01744, "median": -0.01744, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 1 } } }, { "key": "lat_rate_rms", "value": 0.8169, "median": 0.8169, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.4586, "median": -0.4586, "ci": null, "n": 2, "wins": 2, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "96244b1f_12800": { "delta": 0.02055, "median": 0.02055, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 } } }, { "key": "lat_best_lag", "value": 0.6, "median": 0.6, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": 0.1, "median": 0.1, "ci": null, "n": 2, "wins": 0, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "96244b1f_12800": { "delta": -0.05, "median": -0.05, "ci": null, "n": 2, "wins": 2, "ties": 0, "clusters": 1, "cluster_wins": 1 } } }, { "key": "lon_mae", "value": 0.3794, "median": 0.3794, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": 0.01691, "median": 0.01691, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "96244b1f_12800": { "delta": 0.01564, "median": 0.01564, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 } } }, { "key": "lon_corr", "value": 0.5022, "median": 0.5022, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": 0.01559, "median": 0.01559, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "96244b1f_12800": { "delta": 0.001944, "median": 0.001944, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 1 } } }, { "key": "lon_brake_mae", "value": 0.4482, "median": 0.4482, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.05506, "median": -0.05506, "ci": null, "n": 2, "wins": 2, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "96244b1f_12800": { "delta": -0.06012, "median": -0.06012, "ci": null, "n": 2, "wins": 2, "ties": 0, "clusters": 1, "cluster_wins": 1 } } }, { "key": "lon_rate_rms", "value": 2.886, "median": 2.886, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.4702, "median": -0.4702, "ci": null, "n": 2, "wins": 2, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "96244b1f_12800": { "delta": 1.26, "median": 1.26, "ci": null, "n": 2, "wins": 0, "ties": 0, "clusters": 1, "cluster_wins": 0 } } }, { "key": "v3_mae", "value": 2.129, "median": 2.129, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": 0.03487, "median": 0.03487, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "96244b1f_12800": { "delta": 0.2264, "median": 0.2264, "ci": null, "n": 2, "wins": 0, "ties": 0, "clusters": 1, "cluster_wins": 0 } } } ] }, "v13-8mm": { "n": 92, "clusters": 3, "metrics": [ { "key": "lat_mae", "value": 0.1672, "median": 0.1557, "ci": null, "n": 92, "clusters": 3, "paired": { "lebowski_877m": { "delta": -0.01439, "median": -0.01077, "ci": null, "n": 92, "wins": 60, "ties": 0, "clusters": 3, "cluster_wins": 3 }, "96244b1f_12800": { "delta": -0.006628, "median": -0.0004611, "ci": null, "n": 92, "wins": 48, "ties": 0, "clusters": 3, "cluster_wins": 1 } } }, { "key": "lat_mae_bl", "value": 0.1551, "median": 0.1434, "ci": null, "n": 92, "clusters": 3, "paired": { "lebowski_877m": { "delta": -0.01707, "median": -0.01459, "ci": null, "n": 92, "wins": 71, "ties": 0, "clusters": 3, "cluster_wins": 3 }, "96244b1f_12800": { "delta": 0.003352, "median": 0.00588, "ci": null, "n": 92, "wins": 29, "ties": 0, "clusters": 3, "cluster_wins": 1 } } }, { "key": "lat_corr", "value": 0.8099, "median": 0.9068, "ci": null, "n": 92, "clusters": 3, "paired": { "lebowski_877m": { "delta": 0.06139, "median": 0.02468, "ci": null, "n": 92, "wins": 78, "ties": 0, "clusters": 3, "cluster_wins": 3 }, "96244b1f_12800": { "delta": 0.01663, "median": 0.008338, "ci": null, "n": 92, "wins": 68, "ties": 0, "clusters": 3, "cluster_wins": 3 } } }, { "key": "lat_gain", "value": 0.8141, "median": 0.8318, "ci": null, "n": 92, "clusters": 3, "paired": { "lebowski_877m": { "delta": -0.03366, "median": -0.02027, "ci": null, "n": 92, "wins": 55, "ties": 0, "clusters": 3, "cluster_wins": 2 }, "96244b1f_12800": { "delta": -0.04787, "median": -0.04951, "ci": null, "n": 92, "wins": 70, "ties": 0, "clusters": 3, "cluster_wins": 3 } } }, { "key": "lat_rate_rms", "value": 1.296, "median": 1.002, "ci": null, "n": 92, "clusters": 3, "paired": { "lebowski_877m": { "delta": -1.322, "median": -1.263, "ci": null, "n": 92, "wins": 90, "ties": 0, "clusters": 3, "cluster_wins": 3 }, "96244b1f_12800": { "delta": 0.1088, "median": 0.05482, "ci": null, "n": 92, "wins": 25, "ties": 0, "clusters": 3, "cluster_wins": 0 } } }, { "key": "lat_best_lag", "value": 0.6159, "median": 0.6, "ci": null, "n": 91, "clusters": 3, "paired": { "lebowski_877m": { "delta": -0.03736, "median": 0.0, "ci": null, "n": 91, "wins": 40, "ties": 17, "clusters": 3, "cluster_wins": 3 }, "96244b1f_12800": { "delta": -0.1396, "median": -0.15, "ci": null, "n": 91, "wins": 80, "ties": 9, "clusters": 3, "cluster_wins": 3 } } }, { "key": "lon_mae", "value": 0.3724, "median": 0.3783, "ci": null, "n": 92, "clusters": 3, "paired": { "lebowski_877m": { "delta": -0.003662, "median": -0.01499, "ci": null, "n": 92, "wins": 50, "ties": 0, "clusters": 3, "cluster_wins": 2 }, "96244b1f_12800": { "delta": -0.02373, "median": -0.0278, "ci": null, "n": 92, "wins": 74, "ties": 0, "clusters": 3, "cluster_wins": 3 } } }, { "key": "lon_corr", "value": 0.2403, "median": 0.2372, "ci": null, "n": 92, "clusters": 3, "paired": { "lebowski_877m": { "delta": 0.006591, "median": 0.01713, "ci": null, "n": 92, "wins": 55, "ties": 0, "clusters": 3, "cluster_wins": 2 }, "96244b1f_12800": { "delta": 0.03824, "median": 0.0303, "ci": null, "n": 92, "wins": 57, "ties": 0, "clusters": 3, "cluster_wins": 3 } } }, { "key": "lon_brake_mae", "value": 0.4428, "median": 0.4217, "ci": null, "n": 65, "clusters": 3, "paired": { "lebowski_877m": { "delta": -0.04631, "median": -0.05765, "ci": null, "n": 65, "wins": 44, "ties": 0, "clusters": 3, "cluster_wins": 2 }, "96244b1f_12800": { "delta": 0.01242, "median": 0.006539, "ci": null, "n": 65, "wins": 30, "ties": 0, "clusters": 3, "cluster_wins": 1 } } }, { "key": "lon_rate_rms", "value": 5.037, "median": 5.133, "ci": null, "n": 92, "clusters": 3, "paired": { "lebowski_877m": { "delta": -1.295, "median": -1.55, "ci": null, "n": 92, "wins": 69, "ties": 0, "clusters": 3, "cluster_wins": 3 }, "96244b1f_12800": { "delta": -1.444, "median": -1.398, "ci": null, "n": 92, "wins": 77, "ties": 0, "clusters": 3, "cluster_wins": 3 } } }, { "key": "v3_mae", "value": 2.125, "median": 1.978, "ci": null, "n": 92, "clusters": 3, "paired": { "lebowski_877m": { "delta": -0.1094, "median": -0.2267, "ci": null, "n": 92, "wins": 63, "ties": 0, "clusters": 3, "cluster_wins": 1 }, "96244b1f_12800": { "delta": -0.04744, "median": -0.1543, "ci": null, "n": 92, "wins": 56, "ties": 0, "clusters": 3, "cluster_wins": 1 } } } ] }, "v14-8mm": { "n": 8, "clusters": 2, "metrics": [ { "key": "lat_mae", "value": 0.1516, "median": 0.1631, "ci": null, "n": 8, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.01019, "median": 0.009338, "ci": null, "n": 8, "wins": 2, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "96244b1f_12800": { "delta": -0.0183, "median": -0.02025, "ci": null, "n": 8, "wins": 7, "ties": 0, "clusters": 2, "cluster_wins": 2 } } }, { "key": "lat_mae_bl", "value": 0.1404, "median": 0.1458, "ci": null, "n": 8, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.00327, "median": 0.001223, "ci": null, "n": 8, "wins": 4, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "96244b1f_12800": { "delta": -0.01361, "median": -0.01023, "ci": null, "n": 8, "wins": 7, "ties": 0, "clusters": 2, "cluster_wins": 2 } } }, { "key": "lat_corr", "value": 0.8584, "median": 0.923, "ci": null, "n": 8, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.009606, "median": -0.003774, "ci": null, "n": 8, "wins": 4, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "96244b1f_12800": { "delta": 0.02212, "median": 0.01281, "ci": null, "n": 8, "wins": 8, "ties": 0, "clusters": 2, "cluster_wins": 2 } } }, { "key": "lat_gain", "value": 0.9142, "median": 0.9421, "ci": null, "n": 8, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.01005, "median": -0.01642, "ci": null, "n": 8, "wins": 5, "ties": 0, "clusters": 2, "cluster_wins": 1 }, "96244b1f_12800": { "delta": -0.07125, "median": -0.05887, "ci": null, "n": 8, "wins": 8, "ties": 0, "clusters": 2, "cluster_wins": 2 } } }, { "key": "lat_rate_rms", "value": 0.6108, "median": 0.5449, "ci": null, "n": 8, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.5447, "median": -0.456, "ci": null, "n": 8, "wins": 8, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "96244b1f_12800": { "delta": -0.04653, "median": 0.005393, "ci": null, "n": 8, "wins": 4, "ties": 0, "clusters": 2, "cluster_wins": 2 } } }, { "key": "lat_best_lag", "value": 0.5813, "median": 0.55, "ci": null, "n": 8, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.1312, "median": 0.125, "ci": null, "n": 8, "wins": 0, "ties": 1, "clusters": 2, "cluster_wins": 0 }, "96244b1f_12800": { "delta": -0.06875, "median": -0.075, "ci": null, "n": 8, "wins": 6, "ties": 1, "clusters": 2, "cluster_wins": 2 } } }, { "key": "lon_mae", "value": 0.3852, "median": 0.3063, "ci": null, "n": 8, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.07578, "median": 0.06657, "ci": null, "n": 8, "wins": 0, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "96244b1f_12800": { "delta": 0.04993, "median": 0.04881, "ci": null, "n": 8, "wins": 2, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "lon_corr", "value": 0.6368, "median": 0.6822, "ci": null, "n": 8, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.1179, "median": -0.08156, "ci": null, "n": 8, "wins": 0, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "96244b1f_12800": { "delta": -0.04702, "median": -0.0572, "ci": null, "n": 8, "wins": 2, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "lon_brake_mae", "value": 0.6817, "median": 0.7492, "ci": null, "n": 6, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.1241, "median": 0.07001, "ci": null, "n": 6, "wins": 1, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "96244b1f_12800": { "delta": 0.04338, "median": 0.01682, "ci": null, "n": 6, "wins": 3, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "lon_rate_rms", "value": 4.618, "median": 4.335, "ci": null, "n": 8, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.08539, "median": 0.1577, "ci": null, "n": 8, "wins": 3, "ties": 0, "clusters": 2, "cluster_wins": 1 }, "96244b1f_12800": { "delta": -0.4265, "median": -0.4994, "ci": null, "n": 8, "wins": 6, "ties": 0, "clusters": 2, "cluster_wins": 2 } } }, { "key": "v3_mae", "value": 1.095, "median": 1.065, "ci": null, "n": 8, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.1622, "median": 0.1656, "ci": null, "n": 8, "wins": 2, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "96244b1f_12800": { "delta": 0.162, "median": 0.1857, "ci": null, "n": 8, "wins": 1, "ties": 0, "clusters": 2, "cluster_wins": 0 } } } ] } } }, "truck_closed_loop": { "status": "ok", "unit_of_evidence": "window", "n": 98, "clusters": 13, "condition": "Measured. worldmodel-4B as in the public closed loop, but the model drives a simulation of the G55's measured steering and powertrain response instead of the ideal car. The simulation starts from the truck's state (speed, yaw rate, steering, gear) in its CAN recording of the 2 s before the window; one window, whose 2 s before are not all valid CAN, starts steady at its recorded speed. Lane-following windows from the G55's drives, day and night; distance is to the driver's recorded path; seeds 0 and 1, averaged per window. 98 windows (59 day, 39 night).", "baselines": [ "lebowski_877m", "96244b1f_12800" ], "metrics": [ { "key": "xtrack_2s", "label": "Distance from the recorded path at 2 s", "unit": "m", "better": "lower", "value": 0.1389, "median": 0.09476, "ci": [ 0.1076, 0.1719 ], "n": 98, "clusters": 13, "paired": { "lebowski_877m": { "delta": -0.03065, "median": -0.02147, "ci": [ -0.04317, -0.01952 ], "n": 98, "wins": 65, "ties": 0, "clusters": 13, "cluster_wins": 12 }, "96244b1f_12800": { "delta": 0.002753, "median": 0.004396, "ci": [ -0.008336, 0.01285 ], "n": 98, "wins": 42, "ties": 0, "clusters": 13, "cluster_wins": 7 } } }, { "key": "xtrack_4s", "label": "Distance from the recorded path at 4 s", "unit": "m", "better": "lower", "value": 0.5767, "median": 0.3, "ci": [ 0.3843, 0.7766 ], "n": 98, "clusters": 13, "paired": { "lebowski_877m": { "delta": 0.05668, "median": -0.01625, "ci": [ -0.02198, 0.1675 ], "n": 98, "wins": 54, "ties": 0, "clusters": 13, "cluster_wins": 6 }, "96244b1f_12800": { "delta": 0.02134, "median": 0.005347, "ci": [ -0.04749, 0.07448 ], "n": 98, "wins": 46, "ties": 0, "clusters": 13, "cluster_wins": 5 } } }, { "key": "xtrack_7s", "label": "Distance from the recorded path at 7.2 s", "unit": "m", "better": "lower", "value": 3.008, "median": 0.7577, "ci": [ 2.034, 4.077 ], "n": 98, "clusters": 13, "paired": { "lebowski_877m": { "delta": 1.029, "median": 0.07547, "ci": [ 0.1983, 2.101 ], "n": 98, "wins": 36, "ties": 0, "clusters": 13, "cluster_wins": 1 }, "96244b1f_12800": { "delta": 0.6088, "median": 0.09856, "ci": [ 0.3294, 0.9216 ], "n": 98, "wins": 36, "ties": 0, "clusters": 13, "cluster_wins": 1 } } }, { "key": "fail_1m", "label": "Share of episodes ever more than 1 m off the path", "unit": "", "better": "lower", "value": 0.4184, "median": 0.0, "ci": [ 0.3211, 0.526 ], "n": 98, "clusters": 13, "paired": { "lebowski_877m": { "delta": -0.0102, "median": 0.0, "ci": [ -0.06396, 0.05635 ], "n": 98, "wins": 11, "ties": 78, "clusters": 13, "cluster_wins": 5 }, "96244b1f_12800": { "delta": 0.01531, "median": 0.0, "ci": [ -0.03604, 0.06317 ], "n": 98, "wins": 7, "ties": 84, "clusters": 13, "cluster_wins": 3 } } }, { "key": "lat_jerk", "label": "Requested lateral jerk, RMS", "unit": "m/s³", "better": "lower", "value": 0.2861, "median": 0.1875, "ci": [ 0.2385, 0.3324 ], "n": 98, "clusters": 13, "paired": { "lebowski_877m": { "delta": -0.2558, "median": -0.1824, "ci": [ -0.3533, -0.1591 ], "n": 98, "wins": 88, "ties": 0, "clusters": 13, "cluster_wins": 13 }, "96244b1f_12800": { "delta": 0.02343, "median": 0.001961, "ci": [ -0.004518, 0.05377 ], "n": 98, "wins": 48, "ties": 0, "clusters": 13, "cluster_wins": 4 } } }, { "key": "long_jerk", "label": "Requested longitudinal jerk, RMS", "unit": "m/s³", "better": "lower", "value": 0.3732, "median": 0.3093, "ci": [ 0.3294, 0.4168 ], "n": 98, "clusters": 13, "paired": { "lebowski_877m": { "delta": -0.01303, "median": -0.0328, "ci": [ -0.06889, 0.04651 ], "n": 98, "wins": 62, "ties": 0, "clusters": 13, "cluster_wins": 8 }, "96244b1f_12800": { "delta": -0.004926, "median": 0.00979, "ci": [ -0.03134, 0.02265 ], "n": 98, "wins": 46, "ties": 0, "clusters": 13, "cluster_wins": 5 } } }, { "key": "speed_err", "label": "Speed error against the recorded speed, RMS", "unit": "m/s", "better": "lower", "value": 1.311, "median": 1.061, "ci": [ 1.066, 1.555 ], "n": 98, "clusters": 13, "paired": { "lebowski_877m": { "delta": 0.2281, "median": 0.1845, "ci": [ 0.05564, 0.4004 ], "n": 98, "wins": 37, "ties": 0, "clusters": 13, "cluster_wins": 4 }, "96244b1f_12800": { "delta": 0.09969, "median": 0.06311, "ci": [ -0.006685, 0.2056 ], "n": 98, "wins": 39, "ties": 0, "clusters": 13, "cluster_wins": 5 } } } ], "per_unit_file": null, "by_group": { "stock-aimedup": { "n": 14, "clusters": 2, "metrics": [ { "key": "xtrack_2s", "value": 0.1216, "median": 0.09476, "ci": null, "n": 14, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.02869, "median": -0.03028, "ci": null, "n": 14, "wins": 8, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "96244b1f_12800": { "delta": 0.03025, "median": 0.01355, "ci": null, "n": 14, "wins": 5, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "xtrack_4s", "value": 0.5722, "median": 0.4552, "ci": null, "n": 14, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.1211, "median": 0.05566, "ci": null, "n": 14, "wins": 7, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "96244b1f_12800": { "delta": 0.07808, "median": 0.1118, "ci": null, "n": 14, "wins": 5, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "xtrack_7s", "value": 4.848, "median": 5.292, "ci": null, "n": 14, "clusters": 2, "paired": { "lebowski_877m": { "delta": 2.943, "median": 1.226, "ci": null, "n": 14, "wins": 4, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "96244b1f_12800": { "delta": 1.351, "median": 0.2017, "ci": null, "n": 14, "wins": 5, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "fail_1m", "value": 0.7143, "median": 1.0, "ci": null, "n": 14, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.03571, "median": 0.0, "ci": null, "n": 14, "wins": 2, "ties": 9, "clusters": 2, "cluster_wins": 1 }, "96244b1f_12800": { "delta": -0.03571, "median": 0.0, "ci": null, "n": 14, "wins": 3, "ties": 9, "clusters": 2, "cluster_wins": 1 } } }, { "key": "lat_jerk", "value": 0.3631, "median": 0.2358, "ci": null, "n": 14, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.2077, "median": -0.2619, "ci": null, "n": 14, "wins": 11, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "96244b1f_12800": { "delta": -0.0181, "median": -0.04074, "ci": null, "n": 14, "wins": 9, "ties": 0, "clusters": 2, "cluster_wins": 1 } } }, { "key": "long_jerk", "value": 0.3489, "median": 0.3154, "ci": null, "n": 14, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.1457, "median": -0.101, "ci": null, "n": 14, "wins": 12, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "96244b1f_12800": { "delta": 0.01773, "median": 0.04146, "ci": null, "n": 14, "wins": 5, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "speed_err", "value": 1.702, "median": 1.43, "ci": null, "n": 14, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.06178, "median": -0.6326, "ci": null, "n": 14, "wins": 9, "ties": 0, "clusters": 2, "cluster_wins": 1 }, "96244b1f_12800": { "delta": 0.2727, "median": 0.04853, "ci": null, "n": 14, "wins": 5, "ties": 0, "clusters": 2, "cluster_wins": 0 } } } ] }, "stock-dual": { "n": 43, "clusters": 5, "metrics": [ { "key": "xtrack_2s", "value": 0.1343, "median": 0.1015, "ci": null, "n": 43, "clusters": 5, "paired": { "lebowski_877m": { "delta": -0.03378, "median": -0.04203, "ci": null, "n": 43, "wins": 33, "ties": 0, "clusters": 5, "cluster_wins": 4 }, "96244b1f_12800": { "delta": 0.002106, "median": 0.00177, "ci": null, "n": 43, "wins": 20, "ties": 0, "clusters": 5, "cluster_wins": 2 } } }, { "key": "xtrack_4s", "value": 0.4154, "median": 0.2789, "ci": null, "n": 43, "clusters": 5, "paired": { "lebowski_877m": { "delta": -0.03685, "median": -0.04101, "ci": null, "n": 43, "wins": 26, "ties": 0, "clusters": 5, "cluster_wins": 4 }, "96244b1f_12800": { "delta": 0.02697, "median": -0.00643, "ci": null, "n": 43, "wins": 24, "ties": 0, "clusters": 5, "cluster_wins": 3 } } }, { "key": "xtrack_7s", "value": 2.249, "median": 0.532, "ci": null, "n": 43, "clusters": 5, "paired": { "lebowski_877m": { "delta": 0.2295, "median": 0.09003, "ci": null, "n": 43, "wins": 15, "ties": 0, "clusters": 5, "cluster_wins": 1 }, "96244b1f_12800": { "delta": 0.7341, "median": 0.1125, "ci": null, "n": 43, "wins": 13, "ties": 0, "clusters": 5, "cluster_wins": 0 } } }, { "key": "fail_1m", "value": 0.3488, "median": 0.0, "ci": null, "n": 43, "clusters": 5, "paired": { "lebowski_877m": { "delta": 0.01163, "median": 0.0, "ci": null, "n": 43, "wins": 5, "ties": 33, "clusters": 5, "cluster_wins": 2 }, "96244b1f_12800": { "delta": 0.04651, "median": 0.0, "ci": null, "n": 43, "wins": 1, "ties": 39, "clusters": 5, "cluster_wins": 1 } } }, { "key": "lat_jerk", "value": 0.3019, "median": 0.1825, "ci": null, "n": 43, "clusters": 5, "paired": { "lebowski_877m": { "delta": -0.2508, "median": -0.1743, "ci": null, "n": 43, "wins": 39, "ties": 0, "clusters": 5, "cluster_wins": 5 }, "96244b1f_12800": { "delta": 0.04346, "median": 0.01726, "ci": null, "n": 43, "wins": 19, "ties": 0, "clusters": 5, "cluster_wins": 0 } } }, { "key": "long_jerk", "value": 0.4182, "median": 0.3367, "ci": null, "n": 43, "clusters": 5, "paired": { "lebowski_877m": { "delta": 0.04392, "median": -0.01867, "ci": null, "n": 43, "wins": 26, "ties": 0, "clusters": 5, "cluster_wins": 3 }, "96244b1f_12800": { "delta": 0.01631, "median": 0.02833, "ci": null, "n": 43, "wins": 18, "ties": 0, "clusters": 5, "cluster_wins": 1 } } }, { "key": "speed_err", "value": 1.294, "median": 1.008, "ci": null, "n": 43, "clusters": 5, "paired": { "lebowski_877m": { "delta": 0.2106, "median": 0.1974, "ci": null, "n": 43, "wins": 17, "ties": 0, "clusters": 5, "cluster_wins": 2 }, "96244b1f_12800": { "delta": 0.1321, "median": 0.08394, "ci": null, "n": 43, "wins": 15, "ties": 0, "clusters": 5, "cluster_wins": 2 } } } ] }, "stock-newcase": { "n": 4, "clusters": 1, "metrics": [ { "key": "xtrack_2s", "value": 0.04539, "median": 0.04095, "ci": null, "n": 4, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.07948, "median": -0.08061, "ci": null, "n": 4, "wins": 3, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "96244b1f_12800": { "delta": -0.006478, "median": 0.001826, "ci": null, "n": 4, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 1 } } }, { "key": "xtrack_4s", "value": 0.2193, "median": 0.1676, "ci": null, "n": 4, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.0213, "median": 0.04875, "ci": null, "n": 4, "wins": 2, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "96244b1f_12800": { "delta": -0.03228, "median": -0.04504, "ci": null, "n": 4, "wins": 2, "ties": 0, "clusters": 1, "cluster_wins": 1 } } }, { "key": "xtrack_7s", "value": 1.752, "median": 2.135, "ci": null, "n": 4, "clusters": 1, "paired": { "lebowski_877m": { "delta": 0.1163, "median": 0.106, "ci": null, "n": 4, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "96244b1f_12800": { "delta": 0.3292, "median": 0.005763, "ci": null, "n": 4, "wins": 2, "ties": 0, "clusters": 1, "cluster_wins": 0 } } }, { "key": "fail_1m", "value": 0.625, "median": 0.75, "ci": null, "n": 4, "clusters": 1, "paired": { "lebowski_877m": { "delta": 0.0, "median": 0.0, "ci": null, "n": 4, "wins": 0, "ties": 4, "clusters": 1, "cluster_wins": 0 }, "96244b1f_12800": { "delta": 0.0, "median": 0.0, "ci": null, "n": 4, "wins": 0, "ties": 4, "clusters": 1, "cluster_wins": 0 } } }, { "key": "lat_jerk", "value": 0.3214, "median": 0.2917, "ci": null, "n": 4, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.4184, "median": -0.3133, "ci": null, "n": 4, "wins": 4, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "96244b1f_12800": { "delta": 0.05293, "median": 0.04175, "ci": null, "n": 4, "wins": 2, "ties": 0, "clusters": 1, "cluster_wins": 0 } } }, { "key": "long_jerk", "value": 0.4008, "median": 0.4076, "ci": null, "n": 4, "clusters": 1, "paired": { "lebowski_877m": { "delta": 0.01161, "median": -0.007072, "ci": null, "n": 4, "wins": 2, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "96244b1f_12800": { "delta": 0.1359, "median": 0.1387, "ci": null, "n": 4, "wins": 0, "ties": 0, "clusters": 1, "cluster_wins": 0 } } }, { "key": "speed_err", "value": 0.7057, "median": 0.6036, "ci": null, "n": 4, "clusters": 1, "paired": { "lebowski_877m": { "delta": 0.02694, "median": 0.2527, "ci": null, "n": 4, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "96244b1f_12800": { "delta": -0.2358, "median": -0.4765, "ci": null, "n": 4, "wins": 3, "ties": 0, "clusters": 1, "cluster_wins": 1 } } } ] }, "v13-8mm": { "n": 18, "clusters": 3, "metrics": [ { "key": "xtrack_2s", "value": 0.1957, "median": 0.09915, "ci": null, "n": 18, "clusters": 3, "paired": { "lebowski_877m": { "delta": -0.02164, "median": -0.02137, "ci": null, "n": 18, "wins": 11, "ties": 0, "clusters": 3, "cluster_wins": 3 }, "96244b1f_12800": { "delta": -0.01936, "median": 0.00529, "ci": null, "n": 18, "wins": 8, "ties": 0, "clusters": 3, "cluster_wins": 3 } } }, { "key": "xtrack_4s", "value": 0.8179, "median": 0.2925, "ci": null, "n": 18, "clusters": 3, "paired": { "lebowski_877m": { "delta": 0.01583, "median": -0.03334, "ci": null, "n": 18, "wins": 12, "ties": 0, "clusters": 3, "cluster_wins": 1 }, "96244b1f_12800": { "delta": -0.1011, "median": -0.006297, "ci": null, "n": 18, "wins": 10, "ties": 0, "clusters": 3, "cluster_wins": 1 } } }, { "key": "xtrack_7s", "value": 2.736, "median": 0.8269, "ci": null, "n": 18, "clusters": 3, "paired": { "lebowski_877m": { "delta": 0.1111, "median": 0.03563, "ci": null, "n": 18, "wins": 8, "ties": 0, "clusters": 3, "cluster_wins": 0 }, "96244b1f_12800": { "delta": 0.07413, "median": -0.02651, "ci": null, "n": 18, "wins": 11, "ties": 0, "clusters": 3, "cluster_wins": 1 } } }, { "key": "fail_1m", "value": 0.3056, "median": 0.0, "ci": null, "n": 18, "clusters": 3, "paired": { "lebowski_877m": { "delta": -0.08333, "median": 0.0, "ci": null, "n": 18, "wins": 2, "ties": 16, "clusters": 3, "cluster_wins": 1 }, "96244b1f_12800": { "delta": -0.08333, "median": 0.0, "ci": null, "n": 18, "wins": 3, "ties": 15, "clusters": 3, "cluster_wins": 1 } } }, { "key": "lat_jerk", "value": 0.2926, "median": 0.1599, "ci": null, "n": 18, "clusters": 3, "paired": { "lebowski_877m": { "delta": -0.08685, "median": -0.1076, "ci": null, "n": 18, "wins": 15, "ties": 0, "clusters": 3, "cluster_wins": 3 }, "96244b1f_12800": { "delta": 0.06269, "median": 0.02781, "ci": null, "n": 18, "wins": 5, "ties": 0, "clusters": 3, "cluster_wins": 1 } } }, { "key": "long_jerk", "value": 0.3326, "median": 0.206, "ci": null, "n": 18, "clusters": 3, "paired": { "lebowski_877m": { "delta": 0.02804, "median": 0.0001073, "ci": null, "n": 18, "wins": 9, "ties": 0, "clusters": 3, "cluster_wins": 1 }, "96244b1f_12800": { "delta": -0.04683, "median": -0.01776, "ci": null, "n": 18, "wins": 12, "ties": 0, "clusters": 3, "cluster_wins": 2 } } }, { "key": "speed_err", "value": 1.054, "median": 0.7361, "ci": null, "n": 18, "clusters": 3, "paired": { "lebowski_877m": { "delta": 0.1097, "median": 0.1686, "ci": null, "n": 18, "wins": 6, "ties": 0, "clusters": 3, "cluster_wins": 1 }, "96244b1f_12800": { "delta": -0.1086, "median": -0.1292, "ci": null, "n": 18, "wins": 11, "ties": 0, "clusters": 3, "cluster_wins": 2 } } } ] }, "v14-8mm": { "n": 19, "clusters": 2, "metrics": [ { "key": "xtrack_2s", "value": 0.1277, "median": 0.1015, "ci": null, "n": 19, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.02325, "median": -0.00716, "ci": null, "n": 19, "wins": 10, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "96244b1f_12800": { "delta": 0.00685, "median": 0.006359, "ci": null, "n": 19, "wins": 8, "ties": 0, "clusters": 2, "cluster_wins": 1 } } }, { "key": "xtrack_4s", "value": 0.7918, "median": 0.3801, "ci": null, "n": 19, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.276, "median": 0.05522, "ci": null, "n": 19, "wins": 7, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "96244b1f_12800": { "delta": 0.09407, "median": 0.0598, "ci": null, "n": 19, "wins": 5, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "xtrack_7s", "value": 3.892, "median": 0.7779, "ci": null, "n": 19, "clusters": 2, "paired": { "lebowski_877m": { "delta": 2.488, "median": 0.002568, "ci": null, "n": 19, "wins": 8, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "96244b1f_12800": { "delta": 0.3437, "median": 0.1347, "ci": null, "n": 19, "wins": 5, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "fail_1m", "value": 0.4211, "median": 0.0, "ci": null, "n": 19, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.02632, "median": 0.0, "ci": null, "n": 19, "wins": 2, "ties": 16, "clusters": 2, "cluster_wins": 1 }, "96244b1f_12800": { "delta": 0.07895, "median": 0.0, "ci": null, "n": 19, "wins": 0, "ties": 17, "clusters": 2, "cluster_wins": 0 } } }, { "key": "lat_jerk", "value": 0.1801, "median": 0.1656, "ci": null, "n": 19, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.4284, "median": -0.2626, "ci": null, "n": 19, "wins": 19, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "96244b1f_12800": { "delta": -0.03473, "median": -0.01395, "ci": null, "n": 19, "wins": 13, "ties": 0, "clusters": 2, "cluster_wins": 2 } } }, { "key": "long_jerk", "value": 0.3222, "median": 0.3, "ci": null, "n": 19, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.08824, "median": -0.0989, "ci": null, "n": 19, "wins": 13, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "96244b1f_12800": { "delta": -0.05962, "median": -0.005558, "ci": null, "n": 19, "wins": 11, "ties": 0, "clusters": 2, "cluster_wins": 2 } } }, { "key": "speed_err", "value": 1.431, "median": 1.055, "ci": null, "n": 19, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.6359, "median": 0.4273, "ci": null, "n": 19, "wins": 4, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "96244b1f_12800": { "delta": 0.1668, "median": 0.1564, "ci": null, "n": 19, "wins": 5, "ties": 0, "clusters": 2, "cluster_wins": 0 } } } ] } }, "simulator": "g55-2026-10-31" }, "truck_recovery": { "status": "ok", "unit_of_evidence": "window", "n": 100, "clusters": 14, "condition": "Measured. The truck closed loop with a disturbance while the model drives; only the tests listed ran. 2° off heading: an extra 2°/s yaw for the first second. 98 windows (59 day, 39 night). 0.5 m off centre: an extra yaw-rate pulse, out and back over the first 2 s, that would move an uncorrected car 0.5 m sideways with no net heading change. 98 windows (59 day, 39 night). Braking lead: windows where the car ahead braked; the gap is derived from the lead distance seen on the recording minus how far the model ran ahead of the recorded car. 2 windows (2 day, 0 night).", "baselines": [ "lebowski_877m", "96244b1f_12800" ], "metrics": [ { "key": "heading_xtrack_2s", "label": "2° off heading: distance from the path at 2 s", "unit": "m", "better": "lower", "value": 0.5002, "median": 0.4209, "ci": [ 0.4066, 0.6147 ], "n": 98, "clusters": 13, "paired": { "lebowski_877m": { "delta": 0.0473, "median": 0.05833, "ci": [ 0.008299, 0.08823 ], "n": 98, "wins": 40, "ties": 0, "clusters": 13, "cluster_wins": 5 }, "96244b1f_12800": { "delta": 0.01612, "median": 0.02124, "ci": [ -0.005449, 0.03508 ], "n": 98, "wins": 27, "ties": 0, "clusters": 13, "cluster_wins": 4 } } }, { "key": "heading_xtrack_4s", "label": "2° off heading: distance from the path at 4 s", "unit": "m", "better": "lower", "value": 0.7781, "median": 0.4767, "ci": [ 0.5513, 1.038 ], "n": 98, "clusters": 13, "paired": { "lebowski_877m": { "delta": 0.07559, "median": 0.03009, "ci": [ -0.06813, 0.1907 ], "n": 98, "wins": 41, "ties": 0, "clusters": 13, "cluster_wins": 3 }, "96244b1f_12800": { "delta": 0.02028, "median": 0.008313, "ci": [ -0.1003, 0.1151 ], "n": 98, "wins": 46, "ties": 0, "clusters": 13, "cluster_wins": 7 } } }, { "key": "heading_lat_jerk", "label": "2° off heading: requested lateral jerk, RMS", "unit": "m/s³", "better": "lower", "value": 0.3775, "median": 0.3077, "ci": [ 0.3358, 0.4194 ], "n": 98, "clusters": 13, "paired": { "lebowski_877m": { "delta": -0.5094, "median": -0.4585, "ci": [ -0.581, -0.4322 ], "n": 98, "wins": 94, "ties": 0, "clusters": 13, "cluster_wins": 13 }, "96244b1f_12800": { "delta": -0.004901, "median": -0.008272, "ci": [ -0.03131, 0.02377 ], "n": 98, "wins": 57, "ties": 0, "clusters": 13, "cluster_wins": 7 } } }, { "key": "offcentre_xtrack_2s", "label": "0.5 m off centre: distance from the path at 2 s", "unit": "m", "better": "lower", "value": 0.267, "median": 0.2203, "ci": [ 0.2395, 0.2929 ], "n": 98, "clusters": 13, "paired": { "lebowski_877m": { "delta": -1.618e-05, "median": 0.005776, "ci": [ -0.02072, 0.02233 ], "n": 98, "wins": 44, "ties": 0, "clusters": 13, "cluster_wins": 5 }, "96244b1f_12800": { "delta": 0.00874, "median": 0.01678, "ci": [ -0.009991, 0.02657 ], "n": 98, "wins": 34, "ties": 0, "clusters": 13, "cluster_wins": 5 } } }, { "key": "offcentre_xtrack_4s", "label": "0.5 m off centre: distance from the path at 4 s", "unit": "m", "better": "lower", "value": 0.7103, "median": 0.3962, "ci": [ 0.5606, 0.8563 ], "n": 98, "clusters": 13, "paired": { "lebowski_877m": { "delta": 0.04746, "median": -0.03851, "ci": [ -0.008247, 0.1077 ], "n": 98, "wins": 55, "ties": 0, "clusters": 13, "cluster_wins": 6 }, "96244b1f_12800": { "delta": 0.04819, "median": 0.01464, "ci": [ -0.07693, 0.1713 ], "n": 98, "wins": 44, "ties": 0, "clusters": 13, "cluster_wins": 6 } } }, { "key": "offcentre_lat_jerk", "label": "0.5 m off centre: requested lateral jerk, RMS", "unit": "m/s³", "better": "lower", "value": 0.5666, "median": 0.4139, "ci": [ 0.4934, 0.6408 ], "n": 98, "clusters": 13, "paired": { "lebowski_877m": { "delta": -0.5438, "median": -0.516, "ci": [ -0.6436, -0.4399 ], "n": 98, "wins": 92, "ties": 0, "clusters": 13, "cluster_wins": 13 }, "96244b1f_12800": { "delta": 0.01785, "median": -0.02008, "ci": [ -0.03018, 0.07578 ], "n": 98, "wins": 63, "ties": 0, "clusters": 13, "cluster_wins": 7 } } }, { "key": "brake_ahead_4s", "label": "Braking lead: distance ahead of the recorded car at 4 s", "unit": "m", "better": "lower", "value": -0.3692, "median": -0.3692, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": 0.9683, "median": 0.9683, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "96244b1f_12800": { "delta": -0.3448, "median": -0.3448, "ci": null, "n": 2, "wins": 2, "ties": 0, "clusters": 1, "cluster_wins": 1 } } }, { "key": "brake_gap_min", "label": "Braking lead: smallest gap to the lead (derived)", "unit": "m", "better": "higher", "value": 25.73, "median": 25.73, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": -1.733, "median": -1.733, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "96244b1f_12800": { "delta": -0.05307, "median": -0.05307, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 } } }, { "key": "brake_speed_err", "label": "Braking lead: speed error against the recorded speed, RMS", "unit": "m/s", "better": "lower", "value": 1.116, "median": 1.116, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.08713, "median": -0.08713, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "96244b1f_12800": { "delta": -0.0157, "median": -0.0157, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 1 } } } ], "per_unit_file": null, "by_group": { "stock-aimedup": { "n": 14, "clusters": 2, "metrics": [ { "key": "heading_xtrack_2s", "value": 0.43, "median": 0.3994, "ci": null, "n": 14, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.03819, "median": 0.03875, "ci": null, "n": 14, "wins": 5, "ties": 0, "clusters": 2, "cluster_wins": 1 }, "96244b1f_12800": { "delta": 0.06619, "median": 0.06228, "ci": null, "n": 14, "wins": 0, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "heading_xtrack_4s", "value": 0.753, "median": 0.6034, "ci": null, "n": 14, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.2, "median": 0.06036, "ci": null, "n": 14, "wins": 5, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "96244b1f_12800": { "delta": 0.1321, "median": 0.07918, "ci": null, "n": 14, "wins": 3, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "heading_lat_jerk", "value": 0.3928, "median": 0.2841, "ci": null, "n": 14, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.3752, "median": -0.3332, "ci": null, "n": 14, "wins": 13, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "96244b1f_12800": { "delta": -0.04451, "median": -0.03535, "ci": null, "n": 14, "wins": 8, "ties": 0, "clusters": 2, "cluster_wins": 1 } } }, { "key": "offcentre_xtrack_2s", "value": 0.2937, "median": 0.2769, "ci": null, "n": 14, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.03075, "median": 0.01551, "ci": null, "n": 14, "wins": 4, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "96244b1f_12800": { "delta": 0.06366, "median": 0.04641, "ci": null, "n": 14, "wins": 0, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "offcentre_xtrack_4s", "value": 0.6442, "median": 0.5652, "ci": null, "n": 14, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.05436, "median": -0.02759, "ci": null, "n": 14, "wins": 8, "ties": 0, "clusters": 2, "cluster_wins": 1 }, "96244b1f_12800": { "delta": 0.03187, "median": 0.1069, "ci": null, "n": 14, "wins": 4, "ties": 0, "clusters": 2, "cluster_wins": 1 } } }, { "key": "offcentre_lat_jerk", "value": 0.7658, "median": 0.6453, "ci": null, "n": 14, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.2959, "median": -0.2996, "ci": null, "n": 14, "wins": 11, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "96244b1f_12800": { "delta": 0.1861, "median": 0.05728, "ci": null, "n": 14, "wins": 6, "ties": 0, "clusters": 2, "cluster_wins": 0 } } } ] }, "stock-dual": { "n": 45, "clusters": 6, "metrics": [ { "key": "heading_xtrack_2s", "value": 0.4216, "median": 0.3683, "ci": null, "n": 43, "clusters": 5, "paired": { "lebowski_877m": { "delta": 0.006064, "median": -0.01926, "ci": null, "n": 43, "wins": 25, "ties": 0, "clusters": 5, "cluster_wins": 3 }, "96244b1f_12800": { "delta": 0.01866, "median": 0.01788, "ci": null, "n": 43, "wins": 13, "ties": 0, "clusters": 5, "cluster_wins": 1 } } }, { "key": "heading_xtrack_4s", "value": 0.6013, "median": 0.3705, "ci": null, "n": 43, "clusters": 5, "paired": { "lebowski_877m": { "delta": -0.03491, "median": -0.01784, "ci": null, "n": 43, "wins": 24, "ties": 0, "clusters": 5, "cluster_wins": 3 }, "96244b1f_12800": { "delta": 0.08853, "median": 0.002009, "ci": null, "n": 43, "wins": 21, "ties": 0, "clusters": 5, "cluster_wins": 2 } } }, { "key": "heading_lat_jerk", "value": 0.3817, "median": 0.3065, "ci": null, "n": 43, "clusters": 5, "paired": { "lebowski_877m": { "delta": -0.5147, "median": -0.4704, "ci": null, "n": 43, "wins": 40, "ties": 0, "clusters": 5, "cluster_wins": 5 }, "96244b1f_12800": { "delta": -0.01143, "median": -0.01902, "ci": null, "n": 43, "wins": 30, "ties": 0, "clusters": 5, "cluster_wins": 4 } } }, { "key": "offcentre_xtrack_2s", "value": 0.2661, "median": 0.2178, "ci": null, "n": 43, "clusters": 5, "paired": { "lebowski_877m": { "delta": -0.03323, "median": -0.0371, "ci": null, "n": 43, "wins": 28, "ties": 0, "clusters": 5, "cluster_wins": 4 }, "96244b1f_12800": { "delta": 0.002284, "median": 0.01241, "ci": null, "n": 43, "wins": 17, "ties": 0, "clusters": 5, "cluster_wins": 2 } } }, { "key": "offcentre_xtrack_4s", "value": 0.7014, "median": 0.3956, "ci": null, "n": 43, "clusters": 5, "paired": { "lebowski_877m": { "delta": 0.031, "median": -0.0577, "ci": null, "n": 43, "wins": 27, "ties": 0, "clusters": 5, "cluster_wins": 3 }, "96244b1f_12800": { "delta": 0.1391, "median": -0.02912, "ci": null, "n": 43, "wins": 25, "ties": 0, "clusters": 5, "cluster_wins": 2 } } }, { "key": "offcentre_lat_jerk", "value": 0.5984, "median": 0.4385, "ci": null, "n": 43, "clusters": 5, "paired": { "lebowski_877m": { "delta": -0.5783, "median": -0.5147, "ci": null, "n": 43, "wins": 41, "ties": 0, "clusters": 5, "cluster_wins": 5 }, "96244b1f_12800": { "delta": -0.006489, "median": -0.02787, "ci": null, "n": 43, "wins": 27, "ties": 0, "clusters": 5, "cluster_wins": 3 } } }, { "key": "brake_ahead_4s", "value": -0.3692, "median": -0.3692, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": 0.9683, "median": 0.9683, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "96244b1f_12800": { "delta": -0.3448, "median": -0.3448, "ci": null, "n": 2, "wins": 2, "ties": 0, "clusters": 1, "cluster_wins": 1 } } }, { "key": "brake_gap_min", "value": 25.73, "median": 25.73, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": -1.733, "median": -1.733, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "96244b1f_12800": { "delta": -0.05307, "median": -0.05307, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 0 } } }, { "key": "brake_speed_err", "value": 1.116, "median": 1.116, "ci": null, "n": 2, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.08713, "median": -0.08713, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "96244b1f_12800": { "delta": -0.0157, "median": -0.0157, "ci": null, "n": 2, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 1 } } } ] }, "stock-newcase": { "n": 4, "clusters": 1, "metrics": [ { "key": "heading_xtrack_2s", "value": 0.3491, "median": 0.3601, "ci": null, "n": 4, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.1015, "median": -0.1199, "ci": null, "n": 4, "wins": 3, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "96244b1f_12800": { "delta": -0.01646, "median": -0.01986, "ci": null, "n": 4, "wins": 3, "ties": 0, "clusters": 1, "cluster_wins": 1 } } }, { "key": "heading_xtrack_4s", "value": 0.3, "median": 0.1811, "ci": null, "n": 4, "clusters": 1, "paired": { "lebowski_877m": { "delta": 0.0803, "median": 0.03406, "ci": null, "n": 4, "wins": 2, "ties": 0, "clusters": 1, "cluster_wins": 0 }, "96244b1f_12800": { "delta": -0.08681, "median": -0.08996, "ci": null, "n": 4, "wins": 3, "ties": 0, "clusters": 1, "cluster_wins": 1 } } }, { "key": "heading_lat_jerk", "value": 0.3503, "median": 0.2596, "ci": null, "n": 4, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.6218, "median": -0.5721, "ci": null, "n": 4, "wins": 4, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "96244b1f_12800": { "delta": -0.04631, "median": -0.01866, "ci": null, "n": 4, "wins": 3, "ties": 0, "clusters": 1, "cluster_wins": 1 } } }, { "key": "offcentre_xtrack_2s", "value": 0.2909, "median": 0.2173, "ci": null, "n": 4, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.008899, "median": 0.01753, "ci": null, "n": 4, "wins": 1, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "96244b1f_12800": { "delta": -0.01318, "median": -0.01111, "ci": null, "n": 4, "wins": 3, "ties": 0, "clusters": 1, "cluster_wins": 1 } } }, { "key": "offcentre_xtrack_4s", "value": 0.6596, "median": 0.5351, "ci": null, "n": 4, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.06811, "median": -0.0623, "ci": null, "n": 4, "wins": 3, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "96244b1f_12800": { "delta": -0.05103, "median": -0.1211, "ci": null, "n": 4, "wins": 3, "ties": 0, "clusters": 1, "cluster_wins": 1 } } }, { "key": "offcentre_lat_jerk", "value": 0.6557, "median": 0.4519, "ci": null, "n": 4, "clusters": 1, "paired": { "lebowski_877m": { "delta": -0.7416, "median": -0.6289, "ci": null, "n": 4, "wins": 4, "ties": 0, "clusters": 1, "cluster_wins": 1 }, "96244b1f_12800": { "delta": 0.02487, "median": -0.01112, "ci": null, "n": 4, "wins": 2, "ties": 0, "clusters": 1, "cluster_wins": 0 } } } ] }, "v13-8mm": { "n": 18, "clusters": 3, "metrics": [ { "key": "heading_xtrack_2s", "value": 0.8691, "median": 0.8218, "ci": null, "n": 18, "clusters": 3, "paired": { "lebowski_877m": { "delta": 0.1761, "median": 0.1887, "ci": null, "n": 18, "wins": 2, "ties": 0, "clusters": 3, "cluster_wins": 0 }, "96244b1f_12800": { "delta": -0.02857, "median": -0.01135, "ci": null, "n": 18, "wins": 10, "ties": 0, "clusters": 3, "cluster_wins": 2 } } }, { "key": "heading_xtrack_4s", "value": 1.296, "median": 0.7879, "ci": null, "n": 18, "clusters": 3, "paired": { "lebowski_877m": { "delta": 0.06463, "median": 0.07944, "ci": null, "n": 18, "wins": 5, "ties": 0, "clusters": 3, "cluster_wins": 0 }, "96244b1f_12800": { "delta": -0.2245, "median": -0.01902, "ci": null, "n": 18, "wins": 11, "ties": 0, "clusters": 3, "cluster_wins": 3 } } }, { "key": "heading_lat_jerk", "value": 0.4424, "median": 0.4059, "ci": null, "n": 18, "clusters": 3, "paired": { "lebowski_877m": { "delta": -0.4306, "median": -0.418, "ci": null, "n": 18, "wins": 18, "ties": 0, "clusters": 3, "cluster_wins": 3 }, "96244b1f_12800": { "delta": 0.04041, "median": 0.03264, "ci": null, "n": 18, "wins": 3, "ties": 0, "clusters": 3, "cluster_wins": 0 } } }, { "key": "offcentre_xtrack_2s", "value": 0.2758, "median": 0.195, "ci": null, "n": 18, "clusters": 3, "paired": { "lebowski_877m": { "delta": 0.01779, "median": 0.02932, "ci": null, "n": 18, "wins": 8, "ties": 0, "clusters": 3, "cluster_wins": 0 }, "96244b1f_12800": { "delta": -0.02317, "median": 0.001304, "ci": null, "n": 18, "wins": 9, "ties": 0, "clusters": 3, "cluster_wins": 2 } } }, { "key": "offcentre_xtrack_4s", "value": 0.6907, "median": 0.2427, "ci": null, "n": 18, "clusters": 3, "paired": { "lebowski_877m": { "delta": -0.002326, "median": -0.03324, "ci": null, "n": 18, "wins": 10, "ties": 0, "clusters": 3, "cluster_wins": 1 }, "96244b1f_12800": { "delta": -0.147, "median": 0.05671, "ci": null, "n": 18, "wins": 6, "ties": 0, "clusters": 3, "cluster_wins": 2 } } }, { "key": "offcentre_lat_jerk", "value": 0.4803, "median": 0.3863, "ci": null, "n": 18, "clusters": 3, "paired": { "lebowski_877m": { "delta": -0.4015, "median": -0.4277, "ci": null, "n": 18, "wins": 17, "ties": 0, "clusters": 3, "cluster_wins": 3 }, "96244b1f_12800": { "delta": 0.007003, "median": -0.004222, "ci": null, "n": 18, "wins": 10, "ties": 0, "clusters": 3, "cluster_wins": 2 } } } ] }, "v14-8mm": { "n": 19, "clusters": 2, "metrics": [ { "key": "heading_xtrack_2s", "value": 0.4121, "median": 0.3934, "ci": null, "n": 19, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.05664, "median": 0.06119, "ci": null, "n": 19, "wins": 5, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "96244b1f_12800": { "delta": 0.0227, "median": 0.01986, "ci": null, "n": 19, "wins": 1, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "heading_xtrack_4s", "value": 0.8069, "median": 0.4059, "ci": null, "n": 19, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.2434, "median": 0.1122, "ci": null, "n": 19, "wins": 5, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "96244b1f_12800": { "delta": 0.03788, "median": 0.01313, "ci": null, "n": 19, "wins": 8, "ties": 0, "clusters": 2, "cluster_wins": 1 } } }, { "key": "heading_lat_jerk", "value": 0.3012, "median": 0.2596, "ci": null, "n": 19, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.6472, "median": -0.5365, "ci": null, "n": 19, "wins": 19, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "96244b1f_12800": { "delta": 0.004842, "median": -0.008328, "ci": null, "n": 19, "wins": 13, "ties": 0, "clusters": 2, "cluster_wins": 1 } } }, { "key": "offcentre_xtrack_2s", "value": 0.2359, "median": 0.2025, "ci": null, "n": 19, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.03748, "median": 0.06771, "ci": null, "n": 19, "wins": 3, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "96244b1f_12800": { "delta": 0.01773, "median": 0.02485, "ci": null, "n": 19, "wins": 5, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "offcentre_xtrack_4s", "value": 0.8086, "median": 0.4141, "ci": null, "n": 19, "clusters": 2, "paired": { "lebowski_877m": { "delta": 0.1511, "median": 0.01212, "ci": null, "n": 19, "wins": 7, "ties": 0, "clusters": 2, "cluster_wins": 0 }, "96244b1f_12800": { "delta": 0.06016, "median": 0.06257, "ci": null, "n": 19, "wins": 6, "ties": 0, "clusters": 2, "cluster_wins": 0 } } }, { "key": "offcentre_lat_jerk", "value": 0.411, "median": 0.378, "ci": null, "n": 19, "clusters": 2, "paired": { "lebowski_877m": { "delta": -0.7415, "median": -0.7054, "ci": null, "n": 19, "wins": 19, "ties": 0, "clusters": 2, "cluster_wins": 2 }, "96244b1f_12800": { "delta": -0.0423, "median": -0.02916, "ci": null, "n": 19, "wins": 18, "ties": 0, "clusters": 2, "cluster_wins": 2 } } } ] } }, "simulator": "g55-2026-10-31" } }, "stills": [], "limits": [ "Open-loop scores compare a model's action with what the recording driver or car did; a model that drives differently but well is scored as wrong. comma's RL residual makes the lateral action smoother, and earlier than its base's by 0.10 to 0.26 s (measured on the G55 and on 16 of comma's cars); both score as error here. Measured on the 2026-10-27 deep dive: open-loop accuracy did not predict closed-loop driving.", "Closed-loop numbers come from a generative world model, not a car. In a separate test on the G55's windows, worldmodel-4B followed an added yaw command through 2 s but showed 0 to 20% of an added lateral offset 3 to 4 s later; past about 4 s the recorded anchor frames also pull every model back toward the recorded path. The front page reports 2 s.", "Intervals are 95% percentile bootstraps that resample whole drives (the drive a public segment was cut from, the recorded drive a truck window comes from), so segments of one drive count once. Below 10 drives there is no interval: the braking-lead test comes from one drive and reports means and win counts only.", "Paired deltas are model minus baseline on the units both have (for lateral gain, the change in its distance from 1); wins count units where the model is better in the metric's direction.", "The lead of the lateral request is the shift, searched from 0 to 1.5 s, at which the request best correlates with the car's later lateral motion; a larger value means the request comes earlier. A value at either edge is censored and left out. Plan speed error at 3 s reads the plan head, which does not separate models built on one frozen base: it is identical on every segment for the four f22e06e5 models, and for Cinque v2 and v3.", "Truck open loop comes from few drives, and the camera setup changed between them (lens, housing, aim); the pooled number mixes setups, so read the per-setup numbers too. The 8 mm setups' wide camera recorded about 7-10 fps, so its frame can be up to ~60 ms from the narrow one. On the V14 setup up to 7 of a window's 700 steps are ~100 ms (one camera frame dropped). The camera's aim is the truck's live calibration learned on each drive; drives whose recorded calibration was carried over from another drive are left out.", "Truck closed loop: the steering and powertrain simulation is fitted to the G55, not validated on it in closed loop; the reference is the driver's own line, not the lane centre. The pooled number mixes camera setups, so read the per-setup numbers too. On the V14 setup the wide frame can be up to 90 ms from the simulated one (its camera recorded 7-9 fps).", "Recovery tests are disturbances rendered by the world model (yaw it renders; a one-frame jump it does not, so none is used). The braking-lead test has 2 windows from one drive, whose recorded calibration was carried over from an earlier drive." ] }