A Guide to MatterOfStats' Predictions and Team Ratings for 2015

/

With the 2015 season just a few weeks away and an increase in the number of new subscribers and visitors to MatterOfStats, I thought it time to write a blog that describes what goes on here in the Wages and Tips part of the site during the season proper.

Put simply, I do three things in the Wages and Tips section each season:

  1. I rate AFL teams
  2. I assess teams' chances in upcoming AFL games
  3. I make wagers based on some of these assessments and report on the results

1. RATING AFL TEamS

MatterOfStats (MoS) has two team rating systems: MARS and ChiPS. Both are ELO-style rating systems modelled on the original ELO concept, which was created to rate chess players. A key feature of rating systems of this type is that they take into account not just a team's result, but the quality of the opposition they faced in achieving that result.

You can read about MARS in this blog from 2011 (it's the Traditional MARS talked about there. I've not published much about the three other versions discussed in that post), and you can read even more about it in the Newsletter from Round 2 of 2008, which is downloadable via the History & Archives > The Early Years section of the site.

Details on the ChiPS System, one part of which is team rating, are available in this posting from 2014. The process of generating ChiPS team ratings is more complex than that used for MARS in that, for example, ChiPS takes into account the teams' relative recent form when it assesses the expected outcome of a game. It also explicitly includes the benefit to a home team of playing an out-of-state team, and it incorporates variable home ground advantages (HGAs). These HGAs were updated for the start of the 2015 season.

Both Systems rate an "average" team at about 1,000 and, despite the differences in their approaches, produce highly correlated ratings. 

2. Assessing Teams' Chances

MoS has a suite of statistical models, simple and complex, that provide for each game:

  1. Assessments of the probability that the Home team will win (the Probability Predictors)
  2. Assessments of the probability that the Home team will overcome the handicap imposed on it in what's called the Line Betting market (the Line Fund algorithm)
  3. Predictions about what the final margin will be (the Margin Predictors)
  4. Predictions about which team will win (the Head-to-Head Tipsters)

What's different about MoS is that it provides a variety of different tips, predictions and assessments for each game, rather than just a single opinion. That's a reflection of the fact that I'm a Statistician/Data Scientist/Machine Learning Enthusiast by day, so I'm interested in discovering how well different statistical modelling techniques perform in the real world. In a sense, I get to conduct my own tipping competitions every year, but I don't have to harass participants that lose interest once they've fallen a long way behind the leaders.

Head-to-Head Probability Predictors

MoS includes eight different head-to-head probability predictors, three of them based solely on the TAB Bookmaker's prices in the head-to-head market. You can read about the technicalities for two of them, Bookie - OE and Bookie - RE, in this blog post from 2013, and you can read about the creation of the third, Bookie - LPSO, in this post also from 2013 (it's not called Bookie LPSO in that blog, but it's one of the optimal solutions in the second-to-last table). The main thing to recognise about these probability predictors is that they have only two inputs: the TAB Bookmaker's Home Team and Away Team head-to-head prices (usually as at noon on the Wednesday before the game).

WinPred, ProPred and the two H2H predictors, H2H Adjusted and H2H Unadjusted, are fairly garden variety statistical models, the details of which you can access in this blog from 2011. C_Prob is a probability predictor based on ChiPS team ratings, and its derivation is also discussed in the link provided earlier.

Unlike head-to-head predictors for which the goal is accuracy, probability predictors aim for what's called calibration and sharpness. Well-calibrated predictors are those for which teams assessed as X% chances win, on average, about X% of the time. Amongst equally well-calibrated predictors, plaudits accrue to those whose predictions are most divergent from fence-sitting - that is, those whose predictions are, on average, "sharpest" or most distant from 50%.

What to Expect in 2015: The calibration and sharpness of Probability Predictors is measured here on MoS using a slight variant of a probability scoring metric known as the log probability score. The three most directly bookmaker-derived Probability Predictors - Bookie OE, Bookie RE, and Bookie LPSO - are almost certain to return the highest values on this metric. The most interesting question is: in which order?

Line Market Probability Prediction

Only one algorithm struts its stuff in the Line Market, and that's the Line Fund algorithm, which also forms the basis of Line Fund wagering, more on which later. Its creation is described in this blog from 2011, and its accuracy has proven a significant source of profitability for MoS Investors over the period since.

The algorithm provides an assessment of the probability that the Home team in a contest will prevail and its opinions are used to determine whether or not a wager in the Line market is indicated, and what size that wager should be.

What to Expect in 2015: As for the Head-to-Head Probability Predictors, the efficacy of the Line Fund Algorithm's probability assessments are measured using a log probability score. Unlike the Head-to-Head Probability Predictors, however, for which we expect a small, positive average log probability score (say +0.1 to +0.2), we should be thrilled if the Line Fund Algorithm turns in a final average score even fractionally above zero. The reason for this difference in expectation is slightly technical but essentially comes down to the fact that the Line Fund algorithm is assessing 50:50 propositions while the Head-to-Head Probability Predictors usually find themselves with a favourite whose probability exceeds 50%.

Margin Predictors

MoS includes nineteen different margin predictors, eleven of which take as their inputs the outputs of Probability Predictors. These margin predictors were built using Eureqa, a tool that uncovers relationships in data and estimates their complexity. The eleven Margin Predictors created in this way are Bookie_3, Bookie_9, Win_3, Win_7, H2H_Adj_3, H2H_Adj_7, H2H_Unadj_3,H2H_Unadj_10, ProPred_3, ProPred_7 and Combo_7. Each name reflects the Probability Predictor on which it is based (with Combo_7 being based on all of the Probability Predictors) and Eureqa's assessment of that model's complexity, with larger numbers reflecting greater complexity. So, for example, H2H_Unadj_10 is based on the H2H_Unadjusted Probability Predictor and has a complexity rating of 10, the highest of all the Predictors here.

The remaining Predictors include the two Really Simple predictors, whose genesis is described in this post from 2013; two neural networks, Combo_NN1 described in this post from 2011, and Combo_NN2, which is described in this PDF alongside descriptions of a number of other MoS Tipsters and Predictors; C_Marg, which derives from the ChiPS Prediction System linked to earlier in this blog; and two other ensemble predictors, ENS_Linear and ENS_Greedy, whose origin story is chronicled in this 2015 blog post.

What to Expect in 2015: Margin Predictor efficacy is measured on MoS via the Mean Absolute Error (MAE) metric, which rewards Predictors for being close to the final margin, regardless of whether or not the margin they'd predicted implied that the right team won. For example, a prediction of a Swans win by 5 points is as good as a prediction of a Cats win by 1 point if the actual final result is a Swans win by 2 points since each prediction is in error by 3 points.

Any of Combo_7, Bookie_LPSO, Bookie_3 or Bookie_9 might be expected to return the season's lowest MAE, though RSMP_Weighted and RSMP_Simple are also, as they say in racing, "not without chances". Amongst the other Predictors, Combo_NN_1 and Combo_NN_2 are the most enigmatic, and are likely to either do very well or very poorly, while ENS_Linear and ENS_Greedy are completely new to MoS and, as such, have only theoretical prediction records.

Head-to-Head Tipsters

Of the 30 Head-to-Head Tipsters, 19 are based on Margin Predictors, the inferential rule being that a positive margin prediction by a Predictor for the Home team implies a head-to-head prediction of a Home team victory, and conversely for a negative Home team margin prediction.

The remaining 11 Head-to-Head Tipsters are Heuristic Tipsters, whose latest incantation is described in this PDF from 2011. The main conceit of the Heuristic Tipsters is that they can be described by between 1 and 3 simple rules such as "Tip the Home Team" or "Tip the Team Who's Won Most Often When You've Tipped Them", inspired by the work of Gerd Gigerenzer. The Heuristic Tipsters have not proven prescient at the highest levels in recent seasons, but they have been right sufficiently often to both surprise me and to make me more attuned to the base level of tipping accuracy that is possible with minimal inputs.

What to Expect in 2015: Head-to-Head Tipsters have the simplest performance metric of all: accuracy. A correct tip scores 1 point, an incorrect tip 0 points, and a drawn game (or a prediction of a drawn game) scores half a point. 

Any of Bookie_3, Bookie_9, Combo_7, Win_3, Win_7, Win_Pred or BKB would be unsurprising winners of the Most Accurate Head-to-Head Tipster award, and, if recent seasons are any guide, the Heuristic Tipsters (except BKB) can reasonably be expected to fill the bottom section of the MoS leaderboard. Home Sweet Home (HSH) is a fun Tipster to follow as its unswerving belief that the home team will win every contest often sees it alone amongst its peers. 

3. WAGERING ON AFL GAMES 

MoS wagers on games simply as a way of honestly and transparently assessing the real-world quality of some of the predictions it makes. Whilst it would, of course, be possible to "shadow wager" - that is, make only theoretical bets - my view is that it's all too easy for modellers to fool themselves about the efficacy of their models if they follow this approach. They might, for example, think they could theoretically have obtained a price of X for some wager, not knowing that, had they actually been placing bets, they would instead have only obtained Y.

Over the years, MoS has wagered on:

  • The Head-to-Head market, in which the goal is solely to wager on the team that wins
  • The Line Market, in which the goal is to wager on the team that scores the larger number of points after the Bookmaker's handicap has been applied
  • The SuperMargin market, in which the goal is to predict the Home or the Away team's victory margin, most often in a 10-point range.

In 2014, MoS had Funds wagering in all three of these markets, but in 2014 MoS will be wagering in only the first two. The decision to drop out of the SuperMargin market was made based on negative wagering experience and on a statistical analysis that suggested long-term profitability was unlikely.

So, in 2015, MoS will be wagering only in the Head-to-Head and Line Markets. In the Head-to-Head Market it will be fractional Kelly-staking based on the probability assessments of the Head-to-Head Probability Predictor and using a variable cap for different parts of the season based on experience about the accuracy of the Head-to-Head algorithm as the season progresses.

In the Line Market it will be Level-staking based on the probability assessments of the Line Fund algorithm, with bet sizes varying over the course of the season, also based on experience about the accuracy of the relevant algorithm.The Line Fund, which has variously operated under a Level-staking or Kelly-staking regime, has made profits in most of the years in which it's been a part of MoS wagering, especially in recent years. Past performance is though, no indication of future returns.

Overall, across the nine years of wagering that MoS has undertaken, nett profitability has proven elusive, though positive returns in the two most-recent seasons have offered some encouragement.

WRAP UP

This blog post should provide enough information to follow what's going on here on the Wagers & Tips section each week. If you want to know more about any aspect, click on some of the links I've included above and try performing some Site Searches using the search box at the top of the page.

Also, please always feel free to send me an e-mail or to leave me a comment in the Comments section below.