At the conclusion of each game in the men’s AFL home and away season, umpires and coaches are asked to vote on who they saw as the best players in the game. Umpires assign 3 votes to the player they rate as best, 2 votes to the next best, 1 vote to the third best, and (implicitly) 0 votes to every other player. It is these votes that are used to award the Brownlow Medal at the end of the season.

Similarly, the coaches of both teams are asked to independently cast 5-4-3-2-1 votes for the players they see as the five best, meaning that each player can end up with anywhere between 0 and 10 Coaches’ votes.

The question for today is: to what extent can available player game statistics data tell us whether and how coaches and umpires differ in how they arrive at their votes.

(Note that we’ll not be getting into the issue of individual umpire or coach quirks, snubs, or biases, and instead be looking at the data across all voting umpires and coaches.)

A few weeks back I analysed the men’s 2025 AFL schedule with a view to determining which teams had secured relatively easier overall fixtures, and which had secured relatively more difficult overall fixtures.

We investigated various approaches there and reached some conclusions about relative team fixture difficulty, but none of the methods provided an intuitive way to interpret the outputs.

On a related note, this week I had a kind email from a reader who suggested that there might be an opportunity to continuously update teams’ ‘fixture difficulty rating’ (which is just another term for strength of schedule) during the season, as this service was frequently provided by various fantasy leagues for English Football and other sports.

All of which got me to revisiting my strength of schedule methodology.

The Situation

We’ve built a model designed to estimate the probability of a binary event (say, for example, the probability that the home team wins on the line market in the AFL).

It’s a good model - very good, in fact, because it is perfectly calibrated. In other words, when the true probability of an event is X% it’s average estimate of the probability of that event is X%.

Those probability estimates, however, are the result of running some simulation replicates with a stochastic element, which means that those estimates will diverge from X% to an extent determined by how many replicates we run.