Checking a forecaster

You can check a forecaster yourself. Keep a list of the days they said 30%, and count how many of them it rained where you are. A good forecaster is calibrated: over many 30% days, it rains on about 30 of every 100. If it rained on 60 of them, their 30% means more than they say.

Weather services score this with the Brier score: for each forecast, square the gap between the chance given and what happened (1 if it rained, 0 if not), then average. Saying 30% on a day it stays dry costs 0.09; saying 90% on that day costs 0.81. Lower is better, and hedging every day at 50% scores a steady 0.25.

WeatherForge’s Forecaster’s Confidence Ledger has children forecast, say how sure they are, and see their own Brier score.