Evaluation

Ex-Post Evaluation

Each day, after actual measurement data becomes available, the platform automatically evaluates all submitted forecasts.

• Deterministic forecasts (Q50) are evaluated using Root Mean Squared Error (RMSE).

• Probabilistic forecasts (Q10, Q90) are evaluated using the Mean Winkler Interval (MWI).

👉 In both cases, lower values are better, as they reflect more accurate forecasts.

Forecast skill scores Updates

Because measurement data may be updated by the market data providers, forecast skill scores are recalculated periodically.

Recalculation logic:

🕕 Every day at 06:00 CET: Automatic evaluation of all forecasts from the previous day.

📅 Until the 7th of each month: Recalculates scores for all days in both the current and previous months.

📅 After the 7th: Recalculates scores for all days in the current month.

🏁 Before monthly payouts: A final recalculation for the previous month ensures all scores are locked before rewards are distributed.

Important

• Forecast skill scores are calculated only when measurement data is available for all time periods of the forecasted day.

Evaluation & Rankings

Forecasters are evaluated separately for:

• Deterministic Track (Q50 forecasts, scored by RMSE)
• Probabilistic Track (Q10 and Q90 forecasts, scored by MWI)

Two types of evaluations are used:

• Daily Rankings: Rank is based on forecast skill for that specific day.
• Monthly Skill Scores: Based on the average of daily scores throughout the month.

These rankings determine your eligibility and share of the prize pool.

Prize Distribution

As of June 2025, a revised incentive mechanism has been introduced to reward forecasters more fairly and consistently. The changes reduce the impact of one-off “lucky” days and promote sustained participation and forecast quality.

Objectives of the New Scheme

🎯 Prioritize consistent, high-quality forecasts

📈 Reward reliable participation across the full month

🏆 Focus prize distribution on top contributors

Key Components

Daily Ranking rewards

Each day, forecasters are ranked based on RMSE (Q50) or MWI (Q10/Q90). The top 5 market forecasters receive a share of the daily reward budget.

Important

• You must submit all three forecast quantiles (Q10, Q50, Q90) to be eligible for daily rewards.
• Daily ranks are based only on market forecasters (i.e., eligible participants).

Monthly Skill Score based rewards (aka League System)

At the end of each month, forecasters are ranked by their average daily skill scores (RMSE for Q50 and MWI for Q10/Q90).

Based on these rankings, forecasters are placed into leagues:

• Elite League: Ranks 1–5 (top 5 with lowest average skill scores)
• Challenger League: Ranks 6–10
• Runner-Up: Rank 11
• Other ranks are not eligible for monthly league rewards.

Important

• You must submit forecasts on at least 85% of the days in the month (max 5 missed days).
• New forecasters joining after the 1st day of the month are not eligible for league-based rewards that month.
• Missing days incur a penalty in monthly score: the 75th percentile score of that day will be assigned for each missed submission.

Payment Distribution

Deterministic Track (Q50)

This track uses RMSE to rank participants. Monthly prize allocation includes:

• Elite League: Shared €1,500 budget + €250 bonus for lowest peak error
• Challenger League: €700 shared equally among positions 6–10
• Runner-Up (11th): Fixed €50 prize
• Daily rewards: Shared daily prize pool of €1,000

You can find more details below.

Probabilistic Track (Q10, Q90)

This track uses MWI to evaluate performance. Monthly prizes include:

• Elite League: Shared €1,750 budget (no extra bonus)
• Challenger League: €700 shared equally among ranks 6–10
• Runner-Up (11th): Fixed €50 prize
• Daily rewards: Shared daily prize pool of €1,000

You can find more details below.

More about the Daily Ranking

Quick reminder that, given that there are fixed payment forecasters (non-market forecasters) and market forecasters, two types of ranks are available:

• Market Rank: This is the rank considering only market forecasters (i.e., those who are eligible for the prize pool).
• Overall Rank: This is the rank considering all forecasters (i.e., market and non-market forecasters).

By downloading the Market Scores report - via the web application - you will be able to differentiate between the two types of participants and calculate your rank in within market forecasters (aka Market Rank).

• As example, you might be 5th overall (platform) but 3rd considering only market participants.

For deterministic forecasting (Q50 track), forecasters are evaluated and ranked according to their forecasts Root Mean Squared error (RMSE).

In probabilistic forecasting (Q10, Q90), forecasters are ranked based on the Mean Winkler Interval (MWI), considering both forecasted variables. This is a unique skill score for both variables submitted.

In both cases, a lower value indicates a higher ranking for the forecaster.

Note

This particular assessment is done per session, i.e. per day (in contrast to the monthly skill score, which is an average of daily scores).

Example

If you are 1st for RMSE and 3rd for the Winkler rank, you will receive (for a specific day):

• 40% x 50% + 18% x 50% = 29% of that day's prize amount.

If there are fewer than five Forecasters, unallocated prize money is distributed proportionally according to their ranking.

FAQ

1. When will I receive the monthly prize?

A monthly payment will be issued within the first weeks of each month based covering the rewards of the previous month.

2. How can I preview my current scores?

You can check your current scores and overall rankings via the platform's API endpoints. Please check the available code examples at the Listing Session Submission Scores section.

Alternatively, and to be able to differentiate between market and non-market forecasters, you can check the official web application of Predico.

3. What happens if there are fewer than 11 forecasters in the platform?

We understand that the number of forecasters can vary, especially in the early stages of new challenges in the platform. In such cases, the prize distribution is adjusted as follows: - If there are fewer than 11 forecasters in a given track, the portion of the budget allocated to the remaining positions without any forecasters is transferred to the daily rewards prize pool.

Practical example - Deterministic Track: If there are only 8 agents in the market:

• €280 of the budget for the challenger league (40% of €700) goes to the daily rewards
• The runner-up position (11th place) budget of €50 is also added to the daily rewards pool

4. What if I have a low number of submissions in a month?

• You’re not eligible for league rewards if you miss forecasts on more than 5 sessions.

  • However, you can still earn daily rewards.

• For missed days, a penalty score is used in your monthly average (75th percentile of other participants for that day).

6. What if I submitted forecasts late or there was a platform issue?

Only forecasts successfully submitted and timestamped before the session cutoff are considered. If you believe a submission was affected by a platform issue, please contact us immediately for review.

7. Can I view my eligibility for the Elite or Challenger League during the month?

Not yet. We are working on figuring out the best way to show this information. Until then, you can manually track your average skill score via the API and reports.

Contact us

If you have any feedback or concerns about this new approach, please let us know. We need to make the process transparent, fair, and understandable.