Updating forecasts

The author would be happy about help on this article.

Bayesian updating

See Bayes Theorem.

Martingale bounds for binary predictions

Consider the evolution of a probabilistic forecast of some binary event <math>A</math>, i.e. a <math>[0,1]</math>-valued function <math>p_t=\mathbb P(A\mid\mathcal F_t)</math> of time, where <math>\mathcal F_t</math> is the sigma algebra encoding the information we have access to at time <math>t</math>. Due to the tower rule of conditional expectation, such (Bayesian) forecasts are necessarily martingales.

What do typical martingales look like?

todo: add pictures of

• "will X happen? (with lots of small updates)" (looks like Brownian motion in <math>[0,1]</math>)
• "will X happen? (with few major updates)" (looks like step function)
• "will X happen? (without new information)" (straight line)
• "will X happen before T? (without new information other than X not happening)" (slightly curved line monotonically going to 0)

Tests for Bayesian updating

From general results about martingales it follows that autocorrelations of increments must vanish in expectation and that the expected quadratic variation equals the expected uncertainty reduction, as defined in Augenblick&Rabin (i.e. moving away from 50% ought to come with a certain amount of "wiggling"/jumps).

Doob's martingale inequality and the optional stopping theorem (using the fact that <math>p_t</math> is bounded) further allow us to reason probabilistically about the evolution of forecasts such as the probability that a forecast currently sitting at 1% ever goes above 10% again.

Connection to Cognitive Biases

Too much quadratic variation suggests that a forecaster is not Bayesian by overreacting or base-rate neglect (or they got extremely unlucky). Confirmation bias and underreaction, on the other hand, always lead to too little quadratic variation. Without access to "true probabilities" (a Bayesian oracle) it is not possible to decide in which way a forecaster is not Bayesian.

Note it is necessary, but not sufficient to pass these tests: Reacting too much at first and not enough later on cannot be told apart from Bayesian updating when relying only on martingale methods.

References

<comments />