Surprisals and Probability Scores

Each week, at round's end, when I comment on the Probability Predictors I find myself stuck for a unit to attach to their performance.

Surprisals, you might recall are a measure of the "surprise" associated with an outcome. Specifically, n bits of surprisal is the amount of surprise associated with an outcome carrying a pre-game probability of 1/2^n. So, for example, using this definition an outcome that was an even money bet pre-game is associated with 1 bit of surprisal, while a 3/1 shot carries 2 bits of surprisal, a 7/1 shot 3 bits of surprisal, and so on. Surprisals are typically measured in bits.

Surprise is a bad thing for a Probability Predictor though, because a surprised Probability Predictor is one that attached a low probability to the outcome that eventuated. We evaluate our Probability Predictors then by assigning a probability score to them which is 1 minus the surprisals associated with the outcome based on the probability they assigned to it. In this way, the more surprised they were by the outcome, the lower the probability score they receive, and vice versa.

A little more formally, if a Probability Predictor attached probability P to the actual outcome of  a game then, from that Predictor's viewpoint, the outcome carried -log(P) bits of surprisal and it would record a probability score of 1+log(P) [note that both logs are base 2]. 

Since surprisals are measured in 'bits' it seems only appropriate to measure the probability score - or the unsurprisals if you like - in 'pieces'.