Over the past week or so I've been analysing in detail the performance of ProPred, WinPred and the Head-to-Head Fund across seasons 2007 to 2010. I'll be writing more about what I've found for ProPred and WinPred on the Statistical Analyses blog, but here I want to explain what I've found for the Head-to-Head Fund, which previously I'd evaluated mainly on its 2009 and 2010 performances, and how I've tweaked the Fund's wagering rules as a result.

The most startling aspect of the results for the Head-to-Head model as originally conceived was that, while it returned profits for 2009 and 2010, it did no so such thing in 2007 and 2008. Now it could just be that the TAB Sportsbet bookmaker changed strategy in 2009 and the Conditional Inference Tree approach that underpins the Head-to-Head Fund detected and exploited that, or it could instead be that a weakness in the Head-to-Head model was masked in 2009 and 2010 and laid bare in 2007 and 2008. My instinct is to protect against this second possibility.

I'm always wary about making multiple, arbitrary adjustments to any wagering algorithm on the basis of historical results because, in doing so, I run the risk of overfitting history and poorly describing the future. With the benefit of hindsight and an unlimited licence to make adjustments, it wouldn't be hard to construct a base model to which a few adjustments might be made that rendered it the apparent wagering equivalent of a cash-dispensing machine.

Where there appears to be a systematic deficiency, however, that can be reasonably explained in hindsight and that can be corrected with a few, relatively small changes, I remain generally suspicious of adjustments, but a little less so. 

In the end, I've allowed myself to make three changes to the model described in the earlier blog. These changes, in order of significance, are that the Head-to-Head Fund:

• will not wager on any Home Team priced at over $5.00. To create a vig of around 8%, pricing the undertog at over $5.00 requires that the favourite be priced under $1.15. Once we've wandered into that ballpark there's a far greater likelihood that the bookmaker is being constrained from setting the prices he'd prefer to set, in which case the home team probability implied by the market prices is no longer a true reflection of the bookie's assessment of the home team's chances. The Head-to-Head Fund uses (transformed) implied home team probability as one of its inputs and we all know what follows the phrase "Garbage in" in the popular aphorism. So, the Head-to-Head Fund, unlike many Funds that have gone before it, will avoid Home team longshots. Implementing this rule would have prevented 5 wagers in 2007, 10 in 2008, 10 more in 2009, and 4 more in 2010.
  (You might well ask why I'm not also prohibiting the Head-to-Head Fund from wagering on Home teams priced at under $1.15. After all, the implied probability data for these games is equally as suspect. The reason I'm not doing this is because in these games the evidence is that any distortion tends to work in our favour, probably because it tends to artificially elevate the price of the Home team favourite. Empirically, this elevation in price more than offsets any distortionary effects on the underlying head-to-head algorithm such that the Head-to-Head Fund would have made a profit on teams priced at any price between $1.01 and $1.15 except $1.03, at which price it would have registered a small loss.)
• now uses the mean probability across the 100 replicates for which it is run, not the minimum. Using the mean makes the Fund a little less conservative, but it also makes its wagering less subject to random fluctuations. The minimum probability across 100 replicates can change from run to run by a couple of percentage points or more. This might not seem like much, but it can make a substantial difference to the amount wagered. Consider, for example, the situation where the home team is priced at $1.15 and the away team at $4.75. If, in one run, the algorithm estimated the home team's probability of victory at 88%, the recommended Kelly bet would be 8% of the Fund; if that probability estimate changed to 90% in the next run, the recommended Kelly bet would increase by almost 200% to around 23% of the Fund. That's what a financial analyst would call a material difference.
• will adjust its underlying probability to be no more than 25% points different from the probability implied by the bookmaker's prices. I think it's fair to classify this tweak as a very minor one since it affects only 3 wagers in 2007, 3 more in 2008, none in 2009, and 3 more in 2010, and 2 of those 9 wagers would in any case have been prevented by the $5 price cap. The rationale for this rule is that, if the Head-to-Head model rates a team's chances as being more than 25% points higher than the bookies do - say it rates a team as a 65% chance when the bookies have them as a 35% chance - then the bookies know something that the model doesn't. In these cases the Head-to-Head Fund will still wager, it'll just wager less since its Kelly bet will be based on a probability exactly 25% greater than the bookie's estimate, not on its modelled estimate of the probability.

Together, these rules change the historical performance of the Head-to-Head Fund relative to what was published in the earlier blog, as follows: 

• The Fund becomes slightly more active and aggressive, making 95 wagers in 2009 (up 14) at an average size of just under 5% (up 0.5%), and making 85 wagers in 2010 (up 8) at an average. size of about 7% (also up 0.5%). More wagers of a larger average size lifts the turn in both years, from 3.0 to 4.4 in 2009, and from 5.0 to 6.0 in 2010.
• In 2009 the proportion of bets sized over 10% increases from 9% to 14%, and in 2010 the proportion of bets sized 5-10% increases from 16% to 26%.
• The Fund wins a slightly smaller proportion of wagers (56% in 2009 and 71% in 2010).
• ROI for 2009 falls by just under 5% but is still almost 6% for the season. ROI for 2010 rises by almost 2% to 19.7%.

What the changes also do is turn the losses of 2007 and 2008 into profits, more specifically into ROIs of 3.7% and 2.5% respectively. 

In total, what I think the wagering changes do is reduce the volatility of the Fund's return and generally increase its average return, perhaps at the expense of occasionally reducing the return for a specific year. That's a tradeoff I'm willing to make.