Direct answer

Historical forecast validation

You validate a production forecast by hiding production you already know about. Set a cutoff date, withhold everything after it, fit the forecast using only the earlier history, then reveal the withheld months and compare. This is holdout backtesting, and it is the only way to observe a forecast being wrong before a decision depends on it. PronoHorizon scores that comparison with four metrics — WMAPE, bias, cumulative error and P10–P90 coverage — because a single accuracy number cannot tell you whether a forecast is noisy, systematically optimistic, or both.

At a glance

Method: Historical holdout: hide a known period, forecast blind from the remaining history, reveal the actuals.
Demo window: 12-month historical holdout.
WMAPE: Weighted absolute forecast error — how large the misses are.
Bias: Signed forecast error — whether the misses lean high or low.
Cumulative error: Forecast-to-actual volume delta over the window.
P10–P90 coverage: Share of held-out observations that fell inside the scenario envelope.
Published results: None yet. Any published benchmark must name its public dataset and test window, or it is not shown.

The holdout method: fit on the left of the cutoff, forecast blind into the hidden window, then score against the revealed actuals.

The holdout method, step by step

The discipline is entirely in what the model is not allowed to see.

1. Choose a cutoff: Pick a date far enough in the past that a meaningful amount of production followed it, and recent enough that the well was operating under conditions comparable to today.
2. Hide the period after it: Withhold every observation after the cutoff. The fit must not use them, directly or indirectly.
3. Forecast blind: Fit the decline model on the remaining history only and project forward across the hidden window, producing the P50 base case and the P10 and P90 cases around it.
4. Reveal and score: Bring back the withheld actuals and compare them with the blind forecast using WMAPE, bias, cumulative error and P10–P90 coverage.
5. Read the failures: Inspect where the forecast held and where it missed, and decide whether the miss was reservoir behaviour the model could not know or an assumption that should change.

The four metrics, and why one is not enough

Each metric answers a different question, and each is blind to what the others catch.

WMAPE (weighted absolute forecast error): Measures the size of the misses, weighting by volume so that high-rate months count more than low-rate ones. It says how far off you were, but not in which direction.
Bias (signed forecast error): Measures direction. A forecast can have a respectable WMAPE while being consistently high — errors that never change sign compound across a portfolio in a way that random errors do not.
Cumulative error: The volume delta between forecast and actual across the window. This is the number reserves and economics actually inherit, and a forecast can track monthly rates loosely while still landing close on volume, or vice versa.
P10–P90 coverage: The share of held-out observations that fell inside the P10–P90 envelope. Low coverage means the scenario range was too narrow to be useful; very high coverage may mean it was so wide it excluded nothing.

Choosing a cutoff and a window

A holdout test measures the forecast you would have made at the cutoff, so the cutoff choice is part of the result. Too recent and there is not enough revealed production to score against. Too old and you are testing a well that was operating under a different regime — different facilities, different choke policy, different downtime.

The window length has the same trade-off. A short window is dominated by month-to-month noise and operational events; a long window tests the decline model's shape but tells you little about near-term planning accuracy. The public demo uses a 12-month holdout, which is long enough for the decline trend to separate from noise and short enough for the operating basis to stay comparable.

The honest practice is to state both. A validation result without its dataset and window is not a result.

What backtesting cannot tell you

A holdout test measures how well an empirical decline model reproduced a period of history it did not see. It does not certify future accuracy, and it cannot anticipate anything the history contains no information about: a planned workover, a facility expansion, a shut-in, a change in operating strategy.

It also does not distinguish a bad model from bad data. A large miss over months when production was allocated after a facility restart, or when the well was down for compressor work, is telling you about the data, not about the decline fit. That is why operational gaps stay marked rather than being interpolated — the miss has to be interpretable.

Used properly, backtesting is not a stamp of approval. It is a way of finding out which forecasts you are entitled to trust.

Frequently asked questions

How do I validate an oil production forecast?

Use a historical holdout. Choose a cutoff date, withhold all production after it, fit the forecast using only the earlier history, then reveal the withheld months and compare. Score the comparison with WMAPE, bias, cumulative error and P10–P90 coverage so you learn both the size and the direction of the error.

What is backtesting in production forecasting?

Backtesting is running a forecast against a period of history the model was not allowed to use, so the forecast can be checked against what actually happened. In upstream forecasting it is the practical substitute for waiting a year to find out whether a decline fit was right.

What is WMAPE?

WMAPE is the weighted absolute percentage error of a forecast. It aggregates the absolute differences between forecast and actual, weighted by volume, so high-rate months influence the score more than low-rate months. It measures the magnitude of error but not its direction.

What is forecast bias and why does it matter more at portfolio scale?

Bias is the signed error: whether a forecast tends to run high or low. Random errors partly cancel when hundreds of wells are summed, but bias does not — a small consistent optimism per well becomes a large overstatement at the portfolio level, which is exactly where reserves and economics read the number.

How long should the holdout window be?

Long enough that the decline trend separates from month-to-month noise, and short enough that the well's operating conditions during the window are comparable to today's. The public demo uses a 12-month window. Whichever you choose, state it alongside the result — a validation figure without its window and dataset cannot be interpreted.