Sam's Scores

Methodology

What this site does, how the forecasts are built, and where they're weakest — published here rather than left as a footnote.

What this is

Score predictions for FBS college football games outside the top 25, built from historical play-by-play data (2014 onward), not betting-market data. Market lines appear only as a benchmark to check whether the model is adding real information beyond what's already public — this is not a betting product.

How a prediction is built

Every team has an adjusted efficiency rating(offense and defense), recomputed each week using only games played strictly before that week — no information from a game leaks into predictions made before it's played. That rating is blended with pace, rest days, travel distance, elevation, and a heavily regressed turnover margin (turnovers are mostly luck; treating them as skill is one of the most common modeling mistakes in football).

Three different models turn those inputs into a predicted margin and total: a linear ridge regression (simple, and the one every fancier model has to beat), a hierarchical Bayesian model (partial pooling across conferences, useful for thin early-season data), and a gradient-boosted correction layer. A router picks whichever combination has actually performed better in each part of the season — early season leans on the more complex models, mid-to-late season leans on the simple one, because that's what real validation data showed, not a design assumption.

Those point estimates then seed a drive-by-drive Monte Carlo simulator — 20,000 simulated games per matchup — which is where the actual score distribution, win probability, and most-likely-scorelines come from.

How accurate is it, honestly

A football game has roughly 13-14 points of irreducible standard deviation around any honest forecast — no amount of modeling removes that. The goal is to be well-centered and well-calibrated, not precise. Realistic targets set before any of this was built, next to what was actually measured on both the validation data (2023-2024, tuned against) and the sealed test season (2025, never touched by any modeling decision until after the model was finished):

MetricRealistic good outcomeValidation (2023-2024)Test (2025, sealed)
MAE on margin10.5 – 12.0 points12.7312.66
MAE on total9.5 – 11.5 points13.0713.07
Straight-up win accuracyn/a70.9%70.5%

Margin lands inside the target band on both; total is a bit above it on both. The important thing here isn't that either number looks good — it's that the sealed 2025 test season lines up almost exactly with the 2023-2024 validation numbers it was never allowed to influence. That's the honest confirmation that this system generalizes rather than having been quietly overfit to the data it was tuned against.

Where the model is weakest
  • Weeks 1-4 of any season, where the rating system leans hardest on carried-over priors from a roster that may have changed significantly. This is exactly why the router switches to a different model blend for early weeks.
  • Total points more than margin.The drive simulator's intervals for total points are wider than they need to be (over-covered — conservative, not misleadingly narrow) because it doesn't model a shared game-level scoring environment (weather, mutual defensive intensity) that would correlate both teams' scoring in reality.
  • Turnover marginis aggressively regressed toward zero on purpose — a team that's been lucky or unlucky with turnovers will look more average here than its raw record suggests.
  • Weather and preseason priors(recruiting, transfer portal, returning production) are not yet incorporated — both need data sources this project hasn't ingested yet.
  • The Monte Carlo simulator draws each drive's starting field position independently rather than tracking it play-by-play, so it misses real correlations like a short field after a turnover.
The score distribution heatmap, specifically

The heatmap on each game page is a smooth bivariate-normal approximation built from the simulator's margin and total moments, not the raw 20,000-draw simulation — only the summary statistics are stored, not the full joint distribution. The separate "most likely scorelines" table on the same page uses the simulator's real, discrete output instead, so it does respect the actual scoring lattice (multiples of 3 and 7) that the heatmap only approximates.

What the confidence intervals mean

The Performance page's intervals are a normal approximation (mean ± 1.96 × standard error) computed directly from graded predictions, not the percentile bootstrap used in this project's internal backtest reports. Same idea, different computation — the two won't match to the decimal. Any segment with fewer than 100 graded games shows "sample too small" instead of a number, rather than reporting a statistic that isn't trustworthy yet.