Trialling The Super Smart Model

The best way to trial a potential Fund algorithm, I'm beginning to appreciate, is to publish each week the forecasts that it makes. This forces me to work through the mechanics of how it would be used in practice and, importantly, to set down what restrictions should be applied to its wagering - for example should it, like most of the current Funds, only bet on Home Teams, and in which round of the season should it start wagering.

So, since I'm genuinely considering using the Super Smart Model next year, I thought I should start publishing its forecasts.

The Super Smart Model predicts each team's victory margin using the following equation:

Predicted Victory Margin = 207.811 + 73.9587*Prob + 0.124386*Ave_Res_Last_2 + 8.54291*Interstate_Clash - 0.245292*Opp_MARS

(Note that here I've reverted to using MARS Ratings with a base of 1,000. In earlier versions of this equation I had MARS being divided by 1,000.)

Predictions are formed using this equation for each of the two teams in a game and then averaged in a particular way. If the Predicted Victory Margin for Team A is A and for Team B is B, then Team A's average is (A-B)/2 and Team B's is (B-A)/2. Working through the maths of all this, you can derive a single equation that allows you to calculate the prediction margin in one step:

Predicted Victory Margin for Team A = 36.9794*(Prob Team A - Prob Team B) + 0.062193*(Ave_Res_Last_2 for Team A - Ave_Res_Last_2 for Team B) + 8.54291*Interstate_Clash + 0.122646*(Own_MARS for Team A - Own_MARS for Team B)

The Predicted Victory Margin for Team B will simply be the negative of the margin for Team A.

Here then, is what it thinks about Round 16.

As part of considering SSM's merits and for determining what some of the restrictions I referred to above should be, I've also been taking a closer look at SSM's historical performance.

Its line betting performance is strong and fairly well-calibrated.

In the table above, Predicted Line-Adjusted Margin refers to SSM's prediction of a game's victory margin adjusted for the bookie handicap for that game. So, for example, if the Saints beat the Cats by 20 points and were receiving 12.5 points start, their Line-Adjusted Margin of victory would be 32.5 points.

What you can see in this table is that SSM does a fine job of predicting the correct winner on line betting in those games where its Predicted Line-Adjusted Margin is 3.5 points or more. For these games, across seasons 2006 to 2010, it's been right 58.4% of the time, which is a rate far in excess of what's required for breakeven. For the remaining games, it performs only at a level near chance.

Most Funds algorithms require a training period at the start of each season before they become sufficiently predictive to become profitable, as they learn about and adapt to a new season. To investigate this issue for SSM, I looked at its line betting performance by round, grouping the season into four roughly equal quarters.

It's clear from this table that SSM has, historically, not done well in the first half of the season, barely tipping above breakeven through Rounds 1 to 11. Perhaps the MARS Ratings take time to faithfully reflect the relative strengths of each team.

What if we overlay our earlier finding about SSM's poorer performance in those games where its line-adjusted predicted margin is under 3.5 points and consider SSM's predictions only for those games where the line-adjusted margin prediction is 3.5 points or more?

As you can see from the column poetically named "Chg", applying this filter improves SSM's early-season performance considerably. The filter also improves its performance throughout the remainder of the season. It does this by substantially, and selectively, reducing the proportion of the games for which SSM's tips are considered, especially in Rounds 1 to 6 where only about one-half of the games meet the 3.5 point filter criterion.

All told, if we were to avoid betting on SSM's line predictions whenever they implied a line-adjusted margin under 3.5 points, we'd bet on a little over one half of the games in the home-and-away season. That works for me.

If we were using SSM's predictions for wagering purposes this week, this restriction would mean that we'd be wagering on seven of the eight line contests, with the Saints v Collingwood game the only one on which we'd pass. Last week, the restriction would have precluded our wagering on two of the contests - one of which SSM got right, and the other of which it did not - and we'd have bagged 3 from 6.

To finish, let's take a look at SSM's performance in terms of Absolute Prediction Error and on tipping.

They're good results, especially when you bear in mind that BKB's Mean APE for this season stands at 27.7 points per game and its Median APE at 26.5.

So SSM, for now, has a future (which is more than I can say for the HELP algorithm).