Measuring the Surprise in a Season's Results
/In the previous blog we looked at the average level of surprisals generated by teams and by team pairings across all of VFL/AFL history and during the most-recent seasons. Today, as promised in that blog, I'm going to analyse surprisals using the same general methodology, but by season.
Let's start with a boxplot, courtesy of the glorious ggplot2 package.
(You'll probably want to be clicking on this image to access a larger version of it.)
In this chart, for each season:
- the white box extends to cover 50% of surprisal values, from the 1st to the 3rd quartile
- the horizontal line inside the white box denotes the median surprisal value for the season
- the red dot marks the mean surprisal value for the season, which is tracked by the line joining these red dots
- the black circles indicate individual games in the season that generated "unusual" levels of surprisal - upsets if you like
(Upsets in the context of this blog are based on the relative unlikelihood of a victory given the pre-game probabilistic assessments of my binary logit model. This is different from the methodology I used to identify upsets in another recent blog where the difference between the winning and losing team's pre-game MARS Ratings was the sole input. Because the binary logit model used here has season-specific coefficients and because it recognises home team status, upsets determined by it are upsets in the context of both the venue and the season, as well as the differences in the teams' estimated skill-levels. So, for example, it's possible that the binary logit model will assess that a team defeating another with a MARS Rating 50 Ratings Points lower in one season will be more surprising than a team defeating another with a MARS Rating 70 Ratings Point lower in another season where, say, home ground advantage has tended to be greater or where MARS Ratings differences have not translated into victories to the same level.
For anyone who's curious about the coefficients in the binary logit, the summary output from R's glm package is in this PDF.)
The red line in the chart above hints at a modestly increasing trend in the average surprisal level per game since about 1950. Limiting the y-axis to the range (0,2) helps to bring this trend into sharper relief.
One way of distilling the longer-term trend in average surprisals per season is to take a 10-year (uncentered) moving average, which I've done for the following chart. The moving average is the red line, and it shows a clearly increasing trend since about 1950, save for a period of decline and recovery between about 1960 and 1980.
Now, as I pointed out in the previous blog, there are two ways to generate higher levels of surprisals during a season:
- Have more upset wins
- Have a greater proportion of games played between teams of equal or near-equal abilities, which guarantees a result with relative high levels of surprisals regardless of which team wins.
To investigate which of these potential sources best explains the higher levels of surprisals observed in the most-recent seasons I grouped the results for each season on the basis of the estimated pre-game probability of the winning team.
(You'll definitely need to click on this one.)
Using the data from this analysis it turns out that the source of higher surprisal levels since about 1970 has been a reduction in the proportion of games where a strong favourite has prevailed (ie a team priced at about $1.30 or less) and an increase in the proportion of victories recorded by teams with pre-game probability estimates in the 40 to 70% range (ie teams priced from about $1.30 to $2.40 assuming a 5% overround equally levied on both teams).
Specifically, we find that:
- For the period 1960 to 1970 the proportion of games with "Highly Likely" winners was 39%. This fell to 27% in the 1971 to 2012 period.
- For the period 1960 to 1970 the proportion of games described as "Near Equals" or "Little Surprise" was 44%. For the 1971 to 2012 period this rose to 55%.
A reasonable conclusion to draw from these facts, I'd contend, is that the competition is becoming more competitive - at least in the sense that games are tending to generate, on average, more surprising outcomes, mostly because more-closely matched teams are meeting on a week-to-week basis.
This result is completely consistent with the analysis I performed using the NAMSI metric in the 2008 - P3 Newsletter, which assesses competitiveness solely through the level of disparity in team win-loss records at the end of the home-and-away season, somewhat consistent with the analysis I performed in 2012 using winning teams' share of total scoring as the metric, and somewhat different from the conclusion I reached in 2009 using average victory margin as the metric of competitiveness.
What to make of all that then?
Well, to make the obvious point first, the comparative levels of competitiveness in one season versus another depends significantly on your preferred measure. If you define competitiveness as a roughly even distribution of wins and losses across the teams at the end of the home and away season then you'd probably assess the competition as becoming generally more competitive (the last two season aside), but if your concern was with the typical victory margin in a single game or the proportion of total points scored by the victor, then you'd likely assess the competition as becoming less competitive.
And, if you adopted a surprisal approach, equating competitiveness with the level of legitimate surprise that could be associated with game outcomes - either because underdogs win relatively often or games tend to involve teams of roughly equal ability - you'd probably also conclude that the competition was generally becoming more surprising and, by extension, more competitive (again, the last two seasons aside).
To my mind there's no definitive answer but I do feel somewhat swayed by the surprisal analysis in this blog. If more games pit teams of near-equal ability against one another, even if those games don't wind up being close-run affairs, I think that provides reasonable support for the increasing levels of competitiveness hypothesis.
(For what it's worth, perhaps the simplest of all measures of expected game competitiveness - the average pre-game probability attached to the favourite in each game - is correlated -0.98 with the average surprisal measure.)
As one last piece of data for you to consider, here's a chart for which I've extended my previous NAMSI analysis to include the end of home-and-away season results for the years 2008 to 2012 and compared the NAMSI competitiveness estimate for every season with the average surprisal estimate. (Note that I used the ones complement of the NAMSI measure so that larger values of it denote greater competitiveness, making it consistent with the average surprisal measure.)
It's fascinating - and revealing - to me that a season-based measure (NAMSI) and a game-based measure (Average Surprisals) of competitiveness share almost 60% of their variance (ie 0.77 squared) and both come to broadly the same conclusion about the trend in competitiveness in recent times.
(UPDATE : I discovered that ggplot2 offers violin plots as well as box plots, which give a better idea of the underlying distribution of the data. Essentially, the "fatness" of a violin at some level is proportional to the number of values of the underlying distribution that are in the neighbourhood of that level. Here's what a violin plot of the surprisals data by season looks like.)
