Skip to content

Evaluation

Ex-Post Evaluation

After each day’s actual measurements are made available, the platform automatically calculates performance metrics for all submitted forecasts. For deterministic forecasts (Q50), we use the Root Mean Squared Error (RMSE). For probabilistic forecasts (Q10, Q90), we use the Mean Winkler Interval (MWI). In both cases, lower values are better.

Requirements:

In order to be eligible for the daily prize pool, forecasters must submit forecasts for all variables (Q10, Q50, Q90) for the entire day.

Ranking and Prizes:

We produce two rankings daily: one for deterministic (based on RMSE) and one for probabilistic (based on MWI).

Ranking:

The ranking available via the platform endpoints is your overall rank (considering everyone) and not your rank in the market (used for incentives distribution, which will not consider non-market forecasters).

  • As example, you might be 5th overall (platform) but 3rd considering only market participants.

For deterministic forecasting (Q50 track), forecasters are evaluated and ranked according to their forecasts Root Mean Squared error (RMSE).

In probabilistic forecasting (Q10, Q90), forecasters are ranked based on the Mean Winkler Interval (MWI), considering both forecasted variables. This is a unique skill score for both variables submitted.

In both cases, a lower value indicates a higher ranking for the forecaster.

Prize Pool:

There is a single daily prize pool which is split 50% for the deterministic track and 50% for the probabilistic track.

Prize Distribution based on Rank (Deterministic Forecast Track):

50% of the prize pool is distributed based on the (daily) RMSE-based ranking over your deterministic (Q50) forecasts. The following distribution is considered:

  • 1st place: 40%
  • 2nd place: 27%
  • 3rd place: 18%
  • 4th place: 10%
  • 5th place: 5%
  • 6th place or lower: 0%

Prize Distribution based on Rank (Probabilistic Forecast Track):

50% of the prize pool is distributed based on the (daily) MWI-based ranking over your probabilistic (Q10 and Q90) forecasts. The following distribution is considered:

  • 1st place: 40%
  • 2nd place: 27%
  • 3rd place: 18%
  • 4th place: 10%
  • 5th place: 5%
  • 6th place or lower: 0%

The purpose of the Winkler score is to balance the sharpness of your forecast (how narrow your intervals are) and the coverage of your interval (whether it contains the actual value).

Daily evaluation:

Assessment is done per session, i.e. per day.

Therefore, if you are 1st for RMSE and 3rd for the Winkler rank, you will receive (for a specific day):

  • 40% x 50% + 18% x 50% = 29% of that day's prize amount.

If there are fewer than five Forecasters, unallocated prize money is distributed proportionally according to their ranking.

Monthly Payment:

A monthly payment will be issued based on the sum of participants daily ranks and respective prize pool distribution.

How can I preview my current scores?

You can check your current scores and rankings via the platform's API endpoints. Please check the available code examples at the Listing Submission Scores section.

Contact us:

If you have any feedback or concerns about this new approach, please let us know. We need to make the process transparent, fair, and understandable.