The Chase Australia : Progressively Estimating a Team's Chances of Winning (The Canary’s In Seat 1)
In the last few blogs we’ve built models and created tables to explore various aspects of The Chase game show using data from this Google document from James Spencer, which covers the entire history of Andrew O’Keefe’s reign as host from late 2015 to mid-2021.
Today I want to look at a new aspect: how best to estimate the chances of a team ultimately winning whilst an episode is in progress. In particular, I want to estimate it at five specific points in the contest:
After the contestant in Seat 1 has finished his or her CashBuilder and multiple-choice attempt
After the contestant in Seat 2 has finished his or her CashBuilder and multiple-choice attempt
After the contestant in Seat 3 has finished his or her CashBuilder and multiple-choice attempt
After the contestant in Seat 4 has finished his or her CashBuilder and multiple-choice attempt
After the Final Target has been set
THE DATA
Once again we initially have data for 939 episodes, from which we’ll exclude:
any episode where there were only two contestants
any episode that aired on a Sunday (excluded because these episodes look qualitatively different)
any episode where the high or low offers are not available in the data
any episode where a contestant failed to get at least one Cash Builder question correct
That leaves us with data for 923 episodes.
BUILDING A PREDICTIVE MODEL
For today’s blog I’m again going to build binary logit models, this time where the target variable is 0 or 1 depending on whether or not the team eventually won their bank.
One of the interesting challenges in this particular analysis is coming up with appropriate explanatory variables, or features. After a lot of trial-and-error and thought I have decided on the following:
The Chaser name
The Cash Builder amount for the player in Seat 1, but set to 0 if he or she fails to make it back to the table
The Amount Contributed by the player in Seat 1 (which is also 0 if he or she fails to make it back to the table)
The Cash Builder amount for the player in Seat 2, but set to 0 if he or she fails to make it back to the table
The Amount Contributed by the player in Seat 2
The average Cash Builder amount after Seats 1 and 2 have completed, but including only the Cash Builder amounts for those who made it back to the table
The average Amount Contributed after Seats 1 and 2 have completed, but including only the Amounts Contributed by those who made it back to the table
The Number of Players who’ve made it back to the table after Seats 1 and 2 have completed
Variables 9 to 13 are the same as Variables 4 to 8 but for and including Seat 3, and variables 14 to 18 are the same as Variables 4 to 8 but for and including Seat 4. Variable 19 is the Final Target.
For this analysis we’ll fit all models to the entire data set.
THE MODELS
AFTER SEAT 1 HAS FINISHED
For this model we include as candidates variables 1 to 3 only, and we select the best model by conducting an exhaustive search using the glmulti function from the glmulti package and the AIC metric. In this and in all other models we exclude interaction terms.
The chosen model expressed as a formula in R syntax (where Result is 1 if a team collected money, and 0 if not) is
Result ~ 1 + Seat 1 Cash Builder if Finalist
What’s interesting about this model is that it suggests:
The identity of the Chaser is irrelevant to the estimate
The Amount Contributed by the contestant in Seat 1 is not used. The Cash Builder amount gives this model all it needs to know about the ability of the Seat 1 player. (That said, the Cash Builder and Amount Contributed are identical for Seat 1 in 91% of episodes)
We can interpret the model coefficients as follows:
If Seat 1 is eliminated, the team’s chances are now estimated to be 1/(1+exp(1.94925)), which is a little over 12% or about 1-in-8. That’s roughly half what they were before the episode commenced since the average team success rate is just under 24%.
If Seat 1 gets back to the table, the team’s chances are relatively enhanced. If Seat 1’s Cash Builder was $8,000, the estimated chances rise to just over 22%, and if it was $16,000 they rise to just over 37%. Any Cash Builder of $10,000 or more increases the team’s chances above the pre-episode estimate of 23.6%.
One other thing to note about this model is that 85% of the teams it suggests won’t win don’t win, but only 33% of the teams it suggests will win do win. In other words, failure is much easier to predict than success at this stage of the competition.
(Note that for this and all other models we are calculating accuracy and other statistics using a threshold equal to the pre-episode expected winning rate of 23.6%. Different values would be produced were we to use another threshold.)
AFTER SEAT 2 HAS FINISHED
For this model we include as candidates variables 1 to 8 and we select the best model this time by using glmulti’s genetic algorithm to help navigate the large candidate model space.
The chosen model is:
Result ~ 1 + Seat 1 Cash Builder if Finalist + Number Through to Final After Seats 1 and 2 + Mean Amount Contributed per Finalist After Seats 1 and 2
What we have then is:
A proxy for the ability of the contestant in Seat 1 (his or her Cash Builder amount)
A direct measure of the current size of the final team
A proxy for the combined ability of that final team, here based on the average Amount Contributed per Finalist
The signs on the model coefficients tell us that, as we would expect, the higher the Seat 1 Cash Builder, the higher the number of finalists, and the higher the Average Contribution per Finalist, the higher are a team’s estimated chances.
Here are the estimated probabilities using this model for a few scenarios:
Both contestants eliminated
1/(1+exp(2.757)), which is about 6%
Seat 1 progresses with a Cash Builder and Contribution of $8,000. Seat 2 eliminated
1/(1+exp(-(-2.757 + 8 x 0.046 +1 x 0.552 + 8 x 0.038)), which is about 18%
Seat 1 eliminated. Seat 2 progresses with a Contribution of $16,000.
1/(1+exp(-(-2.757 + 0 x 0.046 +1 x 0.552 + 16 x 0.038)), which is about 17%
Seat 1 progresses with a Cash Builder and Contribution of $8,000. Seat 2 progresses with a Cash Builder and Contribution of $12,000
1/(1+exp(-(-2.757 + 8 x 0.046 +2 x 0.552 + 10 x 0.038)), which is about 29%
Interestingly, if Seat 1 is eliminated, the team’s chances can only be elevated above the original 23.6% if the contestant in Seat 2 takes the Top Offer and it is about $28,000 or more.
Note also that this new model is very slightly more accurate in its predictions both of teams it expects to lose, and of teams it expects to win.
AFTER SEAT 3 HAS FINISHED
For this model we include as candidates variables 1 to 13 and we select the best model again by using glmulti’s genetic algorithm.
The chosen model is:
Result ~ 1 + Seat 1 Cash Builder if Finalist + Mean Cash Builder per Finalist After Seats 1 and 2 + Mean Amount Contributed per Finalist After Seats 1, 2 and 3 + Number Through to Final After Seats 1, 2 and 3
What we have now then is:
A proxy for the ability of the contestant in Seat 1 (his or her Cash Builder amount)
A proxy for the combined ability of the finalists (if any) from Seats 1 and 2 here based on the average Amount Contributed per Finalist from those two Seats
A proxy for the combined ability of all finalists (if any) from Seats 1, 2 and 3 here also based on the average Amount Contributed per Finalist
A direct measure of the current size of the final team
The signs on the model coefficients tell us again that, as we would expect, higher is better in terms of a team’s estimated chances.
The number of possible scenarios to investigate is now vast, but here are the estimated probabilities for a few:
All three contestants eliminated
1/(1+exp(3.4820)), which is just under 3%
Seat 1 progresses with a Cash Builder and Contribution of $8,000. Seats 2 and 3 are eliminated
1/(1+exp(-(-3.4820 + 8 x 0.0433 + 8 x 0.0226 + 8 x 0.0277 + 1 x 0.6537)), which is about 11%
Seat 1 eliminated. Seat 2 progresses with a Contribution of $16,000. Seat 3 eliminated.
1/(1+exp(-(-3.4820 + 0 x 0.0433 + 16 x 0.0226 + 16 x 0.0277 + 1 x 0.6537)), which is about 12%
Seat 1 progresses with a Cash Builder and Contribution of $8,000. Seat 2 progresses with a Cash Builder and Contribution of $12,000. Seat 3 progresses with a Cash Builder and Contribution of $10,000
1/(1+exp(-(-3.4820 + 8 x 0.0433 + 10 x 0.0226 + 10 x 0.0277 + 3 x 0.6537)), which is about 34%
Assuming everyone takes Middle Offer, the team’s chances can only be elevated above the original 23.6% if at least two get back to the table, including Seat 1, with over about $26,000 in the bank. With only Seats 2 and 3 back home, the total bank needs to be nearer $40,000 (which virtually necessitates that someone has taken Top Offer).
Note that this new model is very slightly more accurate in its predictions of teams it expects to win, but slightly less accurate for teams it expects to lose.
AFTER SEAT 4 HAS FINISHED
For this model, we include as candidates variables 1 to 17 and we again select the best model by using glmulti’s genetic algorithm.
The chosen model is:
Result ~ 1 + Seat 1 Cash Builder if Finalist + Mean Cash Builder per Finalist After Seats 1 and 2 + Seat 4 Cash Builder if Finalist + Mean Amount Contributed per Finalist After Seats 1, 2, 3 and 4 + Number Through to Final After Seats 1, 2, 3, and 4
What we have now then is:
A proxy for the ability of the contestant in Seat 1 (his or her Cash Builder amount)
A proxy for the ability of the contestant in Seat 4 (his or her Cash Builder amount)
A proxy for the combined ability of the finalists (if any) from Seats 1 and 2 here based on the average Amount Contributed per Finalist from those two seats
A proxy for the combined ability of all finalists (if any) from Seats 1, 2, 3, and 4 here also based on the average Amount Contributed per Finalist
A direct measure of the size of the final team
The signs on the model coefficients tell us once again that more is better in terms of a team’s estimated chances, which remains eminently sensible.
The number of possible scenarios is the vastest yet. Here are the estimated probabilities for a tiny fraction of them:
All four contestants eliminated
1/(1+exp(4.5559)), which is just over 1% (which is therefore the estimate for a Lazarus player setting a target big enough to win the consolation money. So far, only 1 of 18 such contestants has been successful)
Seat 1 progresses with a Cash Builder and Contribution of $8,000. Seats 2, 3, and 4 are eliminated
1/(1+exp(-(-4.5559 + 8 x 0.0460 + 8 x 0.0329 + 0 x 0.0426 + 8 x 0.0568 + 1 x 0.5659)), which is about 5%
Seat 1 eliminated. Seat 2 progresses with a Contribution of $16,000. Seats 3 and 4 eliminated.
1/(1+exp(-(-4.5559 + 0 x 0.0460 + 16 x 0.0329 + 0 x 0.0426 + 16 x 0.0568 + 1 x 0.5659)), which is about 7%
Seat 1 progresses with a Cash Builder and Contribution of $8,000. Seat 2 progresses with a Cash Builder and Contribution of $12,000. Seat 3 eliminated. Seat 4 progresses with a Cash Builder and Contribution of $10,000
1/(1+exp(-(-4.5559 + 8 x 0.0460 + 10 x 0.0329 + 10 x 0.0277 + 3 x 0.6537)), which is about 24%
According to this model, a team with just Seats 1 and 4 setting the final target would need to have banked about $25,000 to have estimated chances equal to the initial 23.6%. If, instead, the team setting the target was just Seats 2 and 3, they would need to have banked something closer to $50,000.
Note that this new model is the most accurate of all so far in its predictions both of teams it expects to lose, and of teams it expects to win. That said, it is still wrong two-thirds of the time about the teams it forecasts will be successful. We could lift that PPV above 50% by setting a threshold of 0.5, but that would see the Sensitivity fall below 15% (ie we wouldn’t ‘detect’ many of the actual winning teams).
AFTER THE FINAL TARGET HAS BEEN SET
For this model we include as candidates variables 1 to 18, now including the Final Target, and we, for the last time, select the best model by using glmulti’s genetic algorithm.
The chosen model is:
Result ~ 1 + Seat 1 Contribution + Mean Amount Contributed per Finalist After Seats 1, 2, 3 and 4 + Target Set
What we have, finally, then is:
A proxy for the ability of the contestant in Seat 1 (his or her Contribution)
A proxy for the combined ability of all finalists (if any) from Seats 1, 2, 3, and 4 here also based on the average Amount Contributed per Finalist
The Final Target
The model coefficients tell us that:
the stronger the contestant in Seat 1, as measured by his or her Contribution the higher the team’s chance of winning
the larger the Final Target, the higher the team’s chance of winning
interestingly, the larger the Contribution per Finalist (and, so, the larger the potential winnings per player), the lower the team’s chances of winning. This might reflect nervousness in the contestants’ ability to take push backs, the more money that is at stake.
Note also that the preferred model excludes the identity of the Chaser and the size of the final team, the latter of which is somewhat proxied by the size of the target. The average target set by a one contestant team is 12.5, a two contestant team is 14.7, a three contestant team is 16.8, and a four contestant team is 18.8, so each increment is essentially the one added for the additional team member and one more question answered correctly in the time available.
Here are the estimated probabilities for a selected set of scenarios:
Seat 1 only makes the final having put $10,000 in the bank. He or she sets a target of 14.
1/(1+exp(-(-9.5713 + 10 x 0.0410 - 10 x 0.0309 + 14 x 0.4929)), which is about 7%
(The same estimated probability would arise if any or all of Seats 2, 3 and 4 also made the final, with Seat 1 contributing $10,000, and the rest contributing another $30,000 between them.)
Seat 1 only makes the final having put $16,000 in the bank. He or she sets a target of 17.
1/(1+exp(-(-9.5713 + 16 x 0.0410 - 16 x 0.0309 + 17 x 0.4929)), which is about 26%
Seat 1 misses the final, and any or all of Seats 2 to 4 make the final having contributed an average $10,000 each to the bank. They set a target of 14.
1/(1+exp(-(-9.5713 + 0 x 0.0410 - 10 x 0.0309 + 14 x 0.4929)), which is about 5%
Seat 1 misses the final, and any or all of Seats 2 to 4 make the final having contributed an average $12,000 each to the bank. They set a target of 17.
1/(1+exp(-(-9.5713 + 0 x 0.0410 - 12 x 0.0309 + 17 x 0.4929)), which is about 17%
Note that this final model is by far the most accurate of all in its predictions of teams it expects to win (at the selected threshold), but is still only correct half the time. It’s right almost 93% of the time, though, about teams it expects to lose.
In terms of the mix of positive and negative forecasts, this model, and the four others described in this blog, all predict about 44% to 49% of teams to win at the threshold chosen.
IS THE FATE OF SEAT 1 THAT IMPORTANT?
One of the stark features of all five models that we’ve just created is the apparent importance of the fate of whomsoever is sitting in Seat 1 for the ultimate fate of the team, even once we know the Final Target that the team has set.
Does that make any sense?
A quick look at the fate of different team compositions in terms of which Seat occupants are present or absent in the final team confirms the apparent importance of Seat 1’s end-of-show location.
The table above, which is sorted by Win percentage, reveals that seven of the eight most successful teams in terms of winning percentage include the contestant from Seat 1, and only one of the eight least successful teams (and none of the bottom four) does likewise.
Put another way:
of all the final teams of size three, the variations with Seat 1 in them are the most successful
of all the final teams of size two, the variations with Seat 1 in them are the most successful
A solo team comprising the contestant from Seat 1 is second only to a sole team comprising the contestant from Seat 3.
It’s also true that, for final teams of size three, those with Seat 1 in them have the largest average Bank, and the largest average Bank when they win.
Parenthetically, we also see that the individual in Seat 4, who crops up in an individual-based form in the Seat 4 model above, appears in five of the six most successful teams, and in only three of the ten least successful teams, albeit in two of the bottom four.
Another way to analyse the data is to focus solely on whether or not the person in a particular Seat makes or misses the Final, how much he or she contributes if he or she does, and how the team fares under each scenario. We obtain the results shown below.
What we see is that, when the person in Seat 1 puts $12,000 or more into the bank, the team’s chances of winning rise to 34.5%, which is substantially higher than the empirical win percentage associated with the contribution made by any other Seat, be it above or below $12,000.
That, I think, completely explains the models’ inclusion of the fate of the person in Seat 1.
This raises the obvious question: does this suggest that the contestant in Seat 1 is, on average, the strongest?
What’s the evidence we have so far?
In the previous blog we found that, on average, Seat 1 registers only a very slightly higher Cash Builder than the other Seats
We also found that Seat 1 is least likely to get home of any of the Seats once we adjust for the Offer taken
In the previous table in this blog on the composition of final teams we saw that a solo contestant team comprising just Seat 1 fares less well - and sets a Final Target no higher - than one comprising just Seat 3, albeit that the numbers are quite small. More generally, the presence or absence of Seat 1 in a final team seems to have little to no impact on the Final Target set.
The Summary by Individual Seat data also seems to suggest that the presence or absence of the contestant from Seat 1 in the final team has little to no impact on the Final Target set if we adjust for the Amount Contributed
Could it be then that Chasers perform less well for a given target when Seat 1 is in the final compared to when Seat 1 is not, relative to when any other Seat position is in or out of the final?
The answer to that appears to be “yes”. In particular we can see this for the games where the Final Target is between 14 and 22, which represents about 70% of episodes. For these episodes, the difference in winning percentage when Seat 1 makes the final versus when it does not tend to be much higher than the difference for other Seats. Here too, though, we need to be a little cautious about interpreting any single difference because of the small sample sizes in some cases.
Might the differences we’re seeing here be purely down to chance? It’s possible, though it seems unlikely looking at the totality of the evidence. The fact that our final model suggests that there is predictive power in knowing the fate of the contestant in Seat 1 even once we know the Final Target, is highly probative (we would respectfully assert, Your Honour).
Absent a chance-based explanation what we’re left with then are a couple of tentative hypotheses:
The presence of Seat 1 in the Final Chase is enough to slightly put off the Chaser (or is a signal that he or she is not quite on his or her game in that episode)
Contestants in Seat 1 are, on average, better at taking advantage of pushback opportunities
FINAL THOUGHTS
We can construct fairly simple models to estimate the dynamic probability of a given team winning an episode. These models progressively re-estimate the quality and number of contestants who have returned home at any point during the contest, and appear to be better at identifying teams likely to lose rather than teams likely to win.
The fate of the contestant in Seat 1 seems to be especially important, for reasons we’ve speculated about but not definitively determined.
I do wonder if there’s a different way of expressing the explanatory variables that might provide more insight into the Seat 1 phenomenon, but I’ll leave that exploration for another day.
RUN THE MODELS YOURSELF
Here’s a link to a Google Spreadsheet that will let you run these calculations for yourself. Let me know what you think.