Model performance

This project's whole discipline rests on keeping tuned-against data separate from a true holdout — so these are shown as separate reports, never blended into one number. 2026 Season is the live, ongoing season — real predictions graded in real time as games finish. Validation (2023-2024) is the data every modeling decision in this project was tuned against. Test (2025) is sealed: never touched by any tuning decision, looked at exactly once, after model development was finished.

Active model version

sams-scores-router-simulator v1

Trained on 2014–2026 · family: simulator

Graded against 6completed games. Confidence intervals here use a normal approximation (mean ± 1.96 × standard error), not the pipeline's percentile bootstrap — see the methodology page.

Overall accuracy

MAE margin: sample too small

MAE total: sample too small

Straight-up win accuracy

How often the predicted winner was actually the winner — a different question from MAE, which only measures how close the score was.

Overall: sample too small

Confidence level	Accuracy
Toss-up	sample too small
Lean	sample too small
Likely	sample too small
Confident	sample too small

A well-calibrated model should get more accurate as confidence increases — if "confident" calls aren't noticeably better than "toss-up" ones, the confidence label isn't telling you anything real.

Reliability

Predicted win probability vs. observed win rate, 5% bins. Dot size is game count.

MAE trend by week

Win accuracy trend by week

Each week is only ~40-70 games, so individual points are noisier than the segment tables below — this is a trend line, not a table of trustworthy standalone statistics.

Segmented accuracy

Below 100games in a segment, the guard shows "sample too small" instead of a number.

By conference tier

Segment	n	MAE margin	MAE total	Benchmark MAE margin
Power	5	sample too small	sample too small	—
Group of 5	1	sample too small	sample too small	—

By week bucket

Segment	n	MAE margin	MAE total	Benchmark MAE margin
weeks 1-4	6	sample too small	sample too small	—

By predicted margin magnitude

Segment	n	MAE margin	MAE total	Benchmark MAE margin
close	3	sample too small	sample too small	—
blowout	3	sample too small	sample too small	—

Model version history

Name	Version	Trained	Holdout MAE margin	Holdout MAE total	Active
sams-scores-router-simulator	v1	9/3/2026	—	—	active