Skip to content

Evaluation

Ex-Post Evaluation

After each day’s actual measurements are made available, the platform automatically calculates performance metrics for all submitted forecasts. For deterministic forecasts (Q50), we use the Root Mean Squared Error (RMSE). For probabilistic forecasts (Q10, Q90), we use the Mean Winkler Interval (MWI). In both cases, lower values are better.

Scores Updates

Note that, in order to adapt to potential measurement data updates (from the market maker data provision sources) your scores for a specific day might be updated. The following scores calculation routines are considered:

  • Everyday at 06h00 CET, the platform will launch an submission forecast skill evaluation task
  • Tasks launched until the 10th day of each month the platform will recalculate scores for all the days of the previous and current months
  • Tasks launched before the 10th day of each month the platform will recalculate scores for all the days of the current month
  • Before the final prize distribution (i.e., by the market maker), an extra recalculation will be issued for the previous month (to ensure the scores are final)

Requirements

In order to be eligible for the daily prize pool, forecasters must submit forecasts for all variables (Q10, Q50, Q90) for the entire day.

Ranking and Prizes

We produce two rankings daily: one for deterministic (based on RMSE) and one for probabilistic (based on MWI).

Ranking

The ranking available via the platform endpoints is your overall rank (considering everyone) and not your rank in the market (used for incentives distribution, which will not consider non-market forecasters).

  • As example, you might be 5th overall (platform) but 3rd considering only market participants.

For deterministic forecasting (Q50 track), forecasters are evaluated and ranked according to their forecasts Root Mean Squared error (RMSE).

In probabilistic forecasting (Q10, Q90), forecasters are ranked based on the Mean Winkler Interval (MWI), considering both forecasted variables. This is a unique skill score for both variables submitted.

In both cases, a lower value indicates a higher ranking for the forecaster.

Prize Pool

There is a single daily prize pool which is split 50% for the deterministic track and 50% for the probabilistic track.

Prize Distribution based on Rank (Deterministic Forecast Track)

50% of the prize pool is distributed based on the (daily) RMSE-based ranking over your deterministic (Q50) forecasts. The following distribution is considered:

  • 1st place: 40%
  • 2nd place: 27%
  • 3rd place: 18%
  • 4th place: 10%
  • 5th place: 5%
  • 6th place or lower: 0%

Prize Distribution based on Rank (Probabilistic Forecast Track)

50% of the prize pool is distributed based on the (daily) MWI-based ranking over your probabilistic (Q10 and Q90) forecasts. The following distribution is considered:

  • 1st place: 40%
  • 2nd place: 27%
  • 3rd place: 18%
  • 4th place: 10%
  • 5th place: 5%
  • 6th place or lower: 0%

The purpose of the Winkler score is to balance the sharpness of your forecast (how narrow your intervals are) and the coverage of your interval (whether it contains the actual value).

Daily evaluation

Assessment is done per session, i.e. per day.

Therefore, if you are 1st for RMSE and 3rd for the Winkler rank, you will receive (for a specific day):

  • 40% x 50% + 18% x 50% = 29% of that day's prize amount.

If there are fewer than five Forecasters, unallocated prize money is distributed proportionally according to their ranking.

Monthly Payment

A monthly payment will be issued based on the sum of participants daily ranks and respective prize pool distribution.

How can I preview my current scores?

You can check your current scores and rankings via the platform's API endpoints. Please check the available code examples at the Listing Submission Scores section.

Contact us

If you have any feedback or concerns about this new approach, please let us know. We need to make the process transparent, fair, and understandable.