A baseball prediction system built against leakage
Jun 29 – Jul 2, 2026
A client wanted a local system that estimates baseball game-outcome probabilities and compares them with market prices, along with an honest evaluation of whether it showed an edge.
What was built
A local pipeline in five milestones: collection of public game statistics and market odds; pregame features for each game; a calibrated model per market, with uncertainty intervals and a breakdown of which inputs moved each prediction; a walk-forward backtest and a daily scoring command; and a dashboard with an audit view and a replay mode showing what the model would have flagged on a past morning.
What made it hold up
The core problem was leakage: a model that sees anything from after the moment it predicts looks better in testing than it will in use. Rolling performance statistics for a game come only from games that started strictly before its scheduled first pitch. Pregame information captured live is frozen, so a later re-harvest cannot overwrite it with what actually happened. Outcome columns are excluded from model inputs at training and checked again at scoring, where a saved model that lists one is refused. Tests pin this down: flipping the recorded outcome for every market on a replayed game leaves its prediction identical, and a companion test confirms that changing a real input does move it.
The season used for evaluation was backfilled after it was played, so its starters are a best-effort reconstruction and its weather is what was recorded at game time rather than the forecast. The documentation names both as remaining look-ahead caveats.
Evaluation was held to the same standard. Before the modeling milestone was delivered, the first version chose its calibrator on the test split, which flatters results; that became fit on one slice, select on another, report once on untouched games. Backtest folds cut on whole dates, and closing odds come only from snapshots taken before the listed start time. Review before each delivery also caught a crash the daily command would have hit on its first real slate, because games not yet played carried a missing value the check did not expect.
On one 2023 season, calibration made every market's probabilities more reliable on held-out games, but in walk-forward testing no market's out-of-sample Brier score beat simply predicting the base rate. That measures probability accuracy against a no-skill guess, not profit against market prices, which needs an odds history the build did not yet have. An edge against real prices is untested rather than disproven. The result is consistent with efficient markets, where calibration improves reliability without creating an edge. The documentation opens with these limits, the dashboard shows the verdict per market, and the backtest can be rerun once odds history accumulates.
Clayton is an exceptional developer, his communication was highly professional, his documentation was flawless, and the entire hand-off process was completely seamless. Highly recommended for any complex data science or software engineering needs!
Client, UpworkJun 29 – Jul 2, 2026
Scope. Five milestones over four days, June 29 to July 2, 2026, each delivered as an installable package verified in a fresh environment, and all accepted before the contract closed. The client holds the delivered system and its source, built to run locally on the client's own machine under the client's own data-provider key, with nothing hosted and nothing for me to operate.