How to Write Forecasting Questions
|The author would be happy about help on this article.|
Resources helpful for writing this article: - https://docs.google.com/document/d/1seLTfAv2x76uxrODbUdgG09Y6kYEPdhBwpI5g3zjpX4/edit#
Good forecasting questions are of vital importance. The best forecasting is meaningless if the question is not relevant. Writing a good question is hard and needs to both ask for something interesting and relevant and also be unambiguous in its resolution criteria. Sometimes, there is a tension between these two demands. A question like "Is China becoming more hostile towards the US" is important and interesting, but cannot easily and unambiguously resolved. A question like "Will Xi Jinping say X on national television in year Y" may be easy to resolve, but also much less interesting.
Generally, question development benefits from the input of subject matter experts and it may be prudent to consult experienced forecasters as well as subject matter experts when drafting a question.
Types of questions/markets
Binary questions refer to a yes/no outcome and forecasters are asked to provide a probability. They are the most common type of questions and tend to be easiest to forecast on. For that reason, binary questions usually attract many more forecasters than continuous questions [CITATION].
Range (also called scalar or continuous)
Range questions ask for an outcome such as the number of COVID cases in New York in April 2021. This is a number between 0 and ~8bn (world population) and forecasters are asked for a probability distribution over possible outcomes. Range questions are usually slightly harder to predict on.
Anatomy of a question
The question should have a short and concise title.
The actual question should be one sentence, ending with a question mark.
Questions do not resolve against "reality" they resolve against whatever source is stated. How do you deal with ambiguous cases?
Rather than writing "When will [public figure] die" write "When will [public figure] leave office" or "stop being important in the way we care about"
Questions don't need to be that long. Depending on platform, just a question statement and resolution criteria might be enough for a question which resolves in 3 months. At most you want a paragraph of introduction, a questions statement, resolution criteria and a few ambiguous cases and how to deal with them. The further into the future the more you want clarity as to what happens if the resolution criteria disappears. How might the community disagree over a resolution?
The following checklist can help you think about a forecasting question. Is this question:
- unambiguous? Questions should be “written clearly enough as to leave no room for ex post facto quarrels over what really happened—and who was right”. There should be a high degree of clarity and precision so that if we imagine a real clairvoyant looking into the future they could resolve the question without needing to ask “What what intended by…?”
- following the10-90 "rule"? The rule states that for binary questions you should aim asking questions where the actual probability (or the community prediction) likely falls in between 10% and 90%. The rationale is that forecasters often aren't very well calibrated when it comes to very small or very large probabilities. But of course, there are important exceptions to the rule and questions may be worth asking even if they don't meet the 10-90 criterion.
- likely to reward hard work going into predicting it? Questions should be neither unanswerably hard nor too easy.
- Immune against weird ways in which reality could make it resolve ambiguously? Sometimes odd things happen that may cause a question to resolve ambiguously. Thinking about possible contingencies can improve the question.
- The description should include enough background that anyone with a general education can, through the text you write and the links you provide, understand what the question is about and why anyone might expect it to resolve in the direction of the question (e.g. A question such as “Will there be a human clone in the next 3 years?” should be accompanied by links that tell the story of why this might be likely, such as a change in regulations in a particular jurisdiction, or a new breakthrough paper that reduces the relevant safety concerns). As you write, consider readers from a faraway country, or educated in a different discipline to yourself, or from 10 years in the future. While it is your job to provide context, your tone should be neutral so as not to introduce unnecessary bias that distracts from or negatively impacts the forecasts produced.
- The resolution criteria is often the hardest part.
- The resolution should be unambiguous. Good options are numerical data regularly published by a reliable source. Or a binary event that is sure to be newsworthy if it happens. If it’s hard to pin down, look for someone official collecting statistics related to your topic. Is there some test against their statistics that can be used?
- Compare the resolution criteria to the title question. Is there any possibility where the criteria will go one way but the title question the other? It is unsatisfying when questions resolve on a technicality like that; try to tweak the title question or the resolution criteria to minimize that possibility.
- When you have a resolution that should be easy to check assuming all goes well, try to handle also the case where it doesn't well. What if the data source you specified stops being published? Is there anything else weird that might happen to make the outcome unclear? Consider adding fallback criteria.
I think it would be nice to have the names of a few people here to which others can reach out.
- Metaculus: Question writing and submission guidelines
- Metaculus: Question writing checklist
- Foretold: Best Practices for Question Writing
- AI Forecasting Dictionary: Best Practices for Question Writing