Setting a Target for Mean Absolute Prediction Error

Until today I've never known why an MAPE of 30 was a sensible target, other than that this seemed to be roughly the figure that the TAB Sportsbet bookmaker achieved each year.

Way back in 2009 in this blog we discovered that Handicap Adjusted Margins were approximately distributed as Normal random variables with a mean of zero and a standard deviation of 37.7 points. What, I wondered, would be the expected MAPE for an unbiased margin predictor making predictions in a competition where the standard deviation of victory margins was 37.7 points per game and those victory margins were distributed as a Normal random variable. The answer, it turns out, is 30.1 points per game.

Even if you assume a mild bias on the part of the predictor, say 2-4 points in favour of the Home team, much as we've seen with the TAB Sportsbet bookmaker, the expected MAPE goes no higher than 30.2 points per game.

This year, however, the TAB Sportsbet bookmaker has an MAPE of a smidge over 28 points per game. Does this mean he's merely eliminated his bias or is it more likely that the standard deviation of victory margins has declined?

Time to act like a real mathematician and generalise the earlier result. If we allow the mean and standard deviation of the underlying Normal distribution to vary, we can express the expected value of the absolute error by the following integral (where phi is the Normal probability density function).

Evaluating this integral for different values of the mean and standard deviation yields the following:

Each line represents a different underlying standard deviation in victory margins and any given line charts what happens to the expected absolute error as we vary the Home team bias.

This chart makes it immediately obvious that there's no way to move from an MAPE of around 30 to one of around 28 merely by eliminating a small bias; this year the variability of victory margins around their expected value has clearly reduced, probably by 2 or 3 points per game.

So, not only have the winners this year been easier to predict than in years past, so too have the margins of their victory.