In the previous blog, I compared Squiggle forecasters’ actual margin prediction MAE results with a distribution of potential MAE outcomes from the same forecasts across 10,000 simulated 2024 season as one way of untangling the skill and luck portions of those actual results.

Those simulations require us to select “ground truth” for the underlying expected margin in each game. In the previous blog we used bookmaker data with an added random component of a Normal variable with mean 0 and standard deviation 8 as that ground truth.

We then proceeded by, for each replicate:

• Drawing 216 independent random variables from a Normal distribution with mean 0 and SD of 8

• Adding these to the bookmakers’ fixed expected margins for each of the 216 games

• Drawing 216 new independent random variables from a Normal distribution with mean as determined in the previous step and an SD of 32

• These simulated results are then used for all forecasters for that simulation replicate and allow us to calculate an MAE for each forecaster

This is then repeated 10,000 times, and the results pooled.

Now, as I pointed out in that previous blog, legitimate objections can be made about the choice of bookmaker data as the basis for determining ground truth. It’s possible, for example, that those markets are distorted by uninformed punters driving the line markets’ handicap away from its true expected value.

I also mentioned in that earlier blog a derived forecaster named s10 that Squiggle calculates by averaging the forecasts of the best 10 forecasters from the previous season. Arguably, that s10 forecast might be a better measure of ground truth, albeit likely also with some spread around it.

With that thought in mind, let’s compare the results we get for the 2024 season if we use s10 instead of the bookmaker data.

FORECASTER MAE DISTRIBUTIONS

Whilst the specific numbers are different, the overall pattern of actual MAEs being above or below the means of the MAE distributions from the simulated data, are very similar.

FORECASTER RANK DISTRIBUTIONS

Next we’ll do the same for the actual forecaster ranks versus the distributions of ranks from the simulated seasons.

Whilst, again, there are some differences in the specifics of the underlying rank data, the overall picture is very similar.

CONCLUSION

Choosing s10 over the bookmakers as our source of truth for 2024 seems to have little affect on the broad conclusions we draw about where individual forecaster’s actual results sits within the distribution of the results that might have been.