Methodology & validation

The two indices

Every carrier receives two first-class indices. They are separate objects because one linear blend cannot rank both future out-of-service rate and future crashes: the two outcomes are pulled in opposite directions by fleet size, so a composite that is good at one is worse at the other. Each index is validated only against its own outcome.

Higher is riskier on both. pctl(·) is the carrier's empirical-CDF percentile over every scoreable carrier in the same daily run. The weights are frozen: they were chosen once, by a documented grid search on the 2024-08-16 origin only, and have not been re-fit on any later data. Every score carries score_version, a methodology note, every component's value, percentile and weight, and the index's historical-validation line, so each point can be traced.

Deliberately not a component: inspection volume (more inspections mean more exposure, not more risk per unit of exposure), carrier age (anti-predictive for crashes and near-zero for OOS in testing), and any current-snapshot fleet-size field such as power units.

Legacy composite

For backward compatibility the API also returns legacy_composite (carrier_score): 0.70 × the higher index + 0.30 × the lower index, plus all surcharges. It is not validated, not gated and not the headline. Read the two indices.

Percentiles and activity bands

Each component is ranked over the whole scored population ("global" percentiles). Every carrier is also assigned a 24-month inspection-activity band — 0, 1–2, 3–5, 6–10, 11–25, 26–60, 61–150, 151+ roadside inspections — which is reported as context with every score.

Ranking within the band instead of population-wide was tested as an ablation and chosen on the fit origin only, by AUC on each index's own label. Population-wide percentiles won for both indices (inspection 0.766 vs 0.760 — a wash; crash 0.690 vs 0.617 — within-band ranking removes exposure by construction), so the band is context, not the ranking basis. Within every band both indices retain their discrimination (see activity-band table).

Carriers with too little inspection history for a component receive a neutral (50th-percentile) weighting for that component, and every score reports a data_sufficiency figure (0–1) so a "50" from missing data is never confused with a "50" from average behaviour.

Flags and surcharges

Hard facts are shown as explicit flags and added as fixed surcharge points rather than hidden inside weights. Surcharge points are policy constants, not fitted parameters.

Only the active-OOS-order surcharge is live in the historical validation below; the insurance and reincarnation surcharges cannot yet be reconstructed point-in-time and are disabled there (see caveats).

Validation design

The question the validation answers is narrow and honest: if you had ranked carriers by this index on a past date, using only what was knowable on that date, how well would that ranking have separated the carriers that went on to have the outcome from those that did not?

1. Point-in-time features. The population is scored at a past origin date T using only inspections, crashes and OOS orders dated on or before T. Outcomes are read from the 12 months after T. Carriers first registered after T are excluded.
2. Cohort. Carriers with at least 3 roadside inspections in the 24 months before T (so a rate is measurable). Cohorts are defined by inspection activity only — never by current operating status, which would be survivorship bias.
3. Temporal holdouts. Weights were chosen on T1 = 2024-08-16 and nowhere else. T2 = 2025-02-16 and T3 = 2025-08-16 are the pre-registered gate holdouts. T4 = 2025-05-16 is a post-selection holdout added after the weights and gate were frozen; it played no part in any choice and is reported, not gated.
4. Provenance guard. The harness asserts that every evaluation origin is strictly after the fit origin, re-runs the weight search on T1 alone and checks it reproduces the shipped weights exactly (it does), and re-runs it with scrambled holdouts to confirm holdout data cannot influence the choice.
5. Pre-registered ship gate. On every gate holdout: inspection_risk AUC ≥ 0.75 on future OOS-rate top decile; crash_risk AUC ≥ 0.66 on any crash in 12 months; and each index's tie-aware decile calibration (Spearman) ≥ 0.90 on its own label. The composite is deliberately not gated.
6. Robustness. ±20 % multiplicative jitter on every weight (10 draws) to check the result is not an artefact of the exact weights.

"OOS-label n" is the subset for which a future 12-month OOS rate is defined (a rate needs inspections in the outcome window). Crash base rates fall at later origins partly because FMCSA crash uploads lag by weeks to months — the T3 and T4 crash labels are lower bounds.

Results (validation run 2026-08-17)

Ship gate — passed

"At the bar": on 2025-02-16 the crash index has 5 merged bins with one adjacent swap between the top two, which is exactly 0.90; T3 and T4 are fully monotone. AUC-ROC 0.5 = no signal, 1.0 = perfect ranking.

Discrimination per index, own label

Secondary crash label (fatal or injury crash in 12 m): crash_risk AUC 0.708 / 0.705 / 0.702 / 0.705 across the four origins, top-decile lift 2.1–2.7×. AUC-PR is read against the base rate. "Top-decile lift" = outcome rate in the riskiest 10 % ÷ population rate.

Calibration: observed outcome rate by index decile (holdouts)

Inspection risk index — share of carriers whose 12-month OOS rate landed in the population's top decile:

Crash risk index — share of carriers with any reportable crash in the next 12 months, tie-aware bins (consecutive deciles at one tied value merged):

The tied bin is the zero-crash point mass: carriers with no reportable crash in the prior 24 months all receive the same crash-index base value. Their forward crash rate is roughly a third of the population's top bins.

Discrimination within activity bands (own label)

Both indices improve with more inspection history, as expected of a rate; the small-band crash figures are the weakest numbers in this validation and should be read as such.

Robustness to the exact weights (±20 % jitter, 10 draws)

Free parameters vs cohort size

The transparent score has 5 free parameters: three inspection weights and three crash weights, each set summing to one (4 free), plus one legacy-composite blend weight. Surcharge points are policy constants. Percentile ranks add no parameters. Against a smallest cohort of 158,125 carriers that is more than 31,000 carriers per free parameter — overfitting of the transparent index is structurally negligible; the residual risk is the choice of objective and thresholds, which is why the post-selection holdout exists.

For contrast, a gradient-boosted comparison model (LightGBM, 400 trees, 22 point-in-time features) reaches AUC 0.77 on any-crash and 0.82 on OOS-rate on the same holdouts with tens of thousands of split parameters. It is research-only and is not served: an evidence report has to be traceable point by point.

Leakage and caveats

Read these before quoting any number above. Every figure is an upper bound on live performance until a run on true archived daily snapshots reproduces it.

• Census is a current snapshot. Fields such as fleet size and safety rating reflect today, not the origin date. The v0.5 indices use no census exposure field (power units appear nowhere in either index); registration date is used only to exclude carriers registered after T. The FMCSA safety-rating benchmark in the full report is therefore a leaky upper bound and is not quoted here.
• The inspection file is a rolling ~3-year window. For the 2024-08-16 fit origin the 24-month window holds only 12 months of data; for 2025-02-16, 18 months. Chronic features are truncated at early origins and the T1 cohort is smallest for that reason.
• Insurance and authority state are current, so the no-insurance surcharge is disabled in the backtest; the reincarnation surcharge is disabled because entity resolution runs on the current census and is not yet point-in-time.
• Crash upload lag. FMCSA crash records arrive weeks to months after the event. The 2025-08-16 (and to a lesser degree 2025-05-16) label windows are incomplete at the tail; those crash base rates are lower bounds and the crash-side AUCs are computed against a slightly under-labelled positive class.
• Raw crash counts are dominated by exposure. A 500-truck fleet almost surely has a crash next year; a one-truck fleet almost surely does not. A rate-based index is not built to rank that, which is why the crash index is judged on its own inspection-normalized construction and why within-band results are shown.
• Operating status is not used to define cohorts (survivorship bias); OOS orders are point-in-time by order and rescind date, but orders later purged from the current file would be missed (rare).
• Population percentiles are computed over every census carrier registered by T — the same construction as production — and evaluation is restricted to the cohort, so the metrics describe the population the product actually scores.

What the score is not

• It does not predict crashes, and we do not describe it that way. It is a documented ranking of carriers by their own inspection and crash history relative to peers, which in historical validation was associated with future outcomes at the levels shown above — for populations, not for any individual carrier.
• It is not a recommendation to use or avoid any carrier. It does not tell you what to do; it documents what was knowable.
• It is not FMCSA's Safety Measurement System, does not reproduce CSA BASIC percentiles, and does not use FMCSA's safety rating as an input.
• It is not a learned model. Weights are fixed and published; there is no per-carrier machine-learning inference in any served score.
• It is not legal advice. Whether a given selection record is defensible in a given case is a question for counsel.

Data: FMCSA/US DOT open-data census, inspection, crash, out-of-service, operating-authority and insurance datasets, archived daily. The full validation report (about 1,200 lines including every decile table, the v0/v0.3/v0.4 history and the learned-model comparison) is generated by the backtest harness; the tables above are rendered from it verbatim. Questions about the methodology: [email protected].