2014 Margin Predictor Team-by-Team Performance (R1 to R12)

/

There's currently about a 3.4 points per game differential between the Mean Absolute Prediction Error (MAPE) for MatterOfStats' best Margin Predictor, Win_7, and its worst, C_Marg.

Earlier this week I performed a team-by-team analysis of C_Marg's season to date, which revealed quite a large variability in C_Marg's predictive performances across the teams and with a given team depending on whether it was playing at home or away. For example, it showed that C_Marg's MAPE for games involving the Dogs was just 20 points per game, but for games involving the Swans it was 46.5 points per game. As well, for games where Geelong was playing at home C_Marg's MAPE was 26.7 points per game, but where the Cats were away the MAPE was more than 10 points per game worse at 37.9.

I wondered then how C_Marg's team-by-team performances might compare with the other MatterOfStats Margin Predictors.

Here, firstly, are charts for the Predictors' performances for all the teams when they're playing at home shown as box plots. For each box, the heavy line marks the Median Absolute Prediction Error (note the median, which is less affected by outliers, is used here and not the mean) and the box's extent reflects the range of Absolute Prediction Errors from the 25th (the lower hinge) to the 75th percentile (the upper hinge) for that Predictor and that team. The whiskers of each box extend to cover the maximum (minimum) value that is within 1.5 times the Interquartile Range above the upper (below the lower) hinge, and the circles denote outlying observations.

So, for example, the topmost bar in the upper left chart below, which is for Win_7's predictions of games where Adelaide was the home team, reveals that the Median APE for these games was about 32 points per game, and the IQR extends from about 7 to 54 points. The absolute prediction errors for a handful of games were outside this range, which is why there are whiskers on the bar, but their are no outliers.

A great deal can be gleaned from this chart, but I'll constrain myself to a handful of observations:

  • There's a surprising amount of variability in the Median APEs for all Predictors for many teams. As good examples of this refer to the charts for Port Adelaide or Geelong and focus your attention on the dark lines.
  • There's also considerable variability in the Median APEs across all teams. The Melbourne and Western Bulldogs home games, for example, have been particularly accurately predicted, while those of Sydney and Hawthorn have been more difficult.
  • Focussing on C_Marg we see that, relative to other Predictors, it's done most poorly in predicting Kangaroos, Melbourne and Port Adelaide home games, and it's done best in predicting Collingwood, Fremantle and GWS home games.

The Mean Average Prediction Error data is summarised in the table below. It paints a somewhat similar picture to the Median APE data about each Predictor's relative ability in relation to each team, but the presence of outlier APEs for most Predictors for at least some teams does have a significant influence. 

Next, let's look at the Absolute Prediction Errors when we classify games on the basis of the team playing away.

In a similar vein, I'll constrain myself to just a few comments about this chart:

  • There are some remarkably low Median APEs for a number of teams - Adelaide, Gold Coast, Port Adelaide, West Coast and the Western Bulldogs, for example.
  • Again there is considerable variability in Predictor Median APEs for particular teams such as Melbourne, GWS and St Kilda.
  • C_Marg has done relatively poorly predicting games where Carlton, GWS or Port Adelaide were playing away, and relatively well predicting games where Essendon, Geelong or the Western Bulldogs were playing away.

The Mean APE data for this game view appears below. Again we find a very broadly similar picture to the Median APE data but with some significant differences because of outlier APEs.

One component of each Predictor's accuracy for each team is the average bias in its predictions for that team, which we can measure as the average difference between the predicted and the actual victory margin for that team.

Box plots for this metric for each team when it was playing at home appear below. Positive biases reflect the fact that a Predictor has more often predicted too large a victory margin, and negative margins the opposite.

From these charts we can see that:

  • Most Predictors have tended to predict victory margins that were too small for games where Adelaide, Fremantle, Geelong, Hawthorn, Port Adelaide or Sydney were at home, and margins that were too large for games where the Brisbane Lions, Carlton, Richmond, St Kilda, West Coast or the Western Bulldogs were playing at home.
  • C_Marg's biases have been the most positive for the Brisbane Lions, Richmond and St Kilda, and the most negative for Adelaide, Fremantle, Hawthorn, Port Adelaide and Sydney.

The mean bias information is in the table below.

Finally, let's look at the bias data for teams when they are playing away. Note that the biases shown here are from the home team's perspective, so a positive bias indicates that the away team's ability has been underestimated, which is the converse of the interpretation for the data view just discussed. 

Here we can see that:

  • Most Predictors have tended to predict victory margins that were too small for games where Collingwood, the Kangaroos, Melbourne and Sydney were playing away, and margins that were too large for games where the Brisbane Lions, Essendon or Geelong were playing away.
  • C_Marg's biases have been the most positive for the Brisbane Lions, Geelong, Hawthorn and Richmond, and the most negative for the Collingwood, the Kangaroos, Melbourne and Sydney.

Lastly, here's the mean bias data for this view.