Skip to content

Downloading Raw Data

After selecting a challenge, the next step is to download the raw observed data associated with the challenge's resource. Market Owners publish time-series data periodically into our platform, which includes raw measurement data for their registered resources.

Important

  • The data is provided at a 15-minute resolution, resulting in a large volume of samples. Therefore, pagination (Limit/Offset Strategy) is required to retrieve all data properly.

What these numbers physically are

This page covers how to fetch a resource's measurements. For what the values mean - the quantity, unit, aggregation boundary, who publishes them and when they stop being revised - see the resource's program page: Elia · 50Hertz · EDP.

API Endpoints

To interact with the Predico API and retrieve information about the raw measurements for a specific challenge target resource, you can use the following endpoints:

Access Token Required

An access token must be included in the Authorization header of your request. If you haven't obtained an access token yet, please refer to the Authentication section.

Markets that do not share their measurements

Some markets may not share their measurements. You can check this before you start: on the Markets page, such a market shows the label Measurements private, and its details say that measurements are used only to score forecasts. Through the API, GET /api/v1/user/resource/ returns share_measurements: false for it. On such a market, the raw-measurements download and the challenge solution answer 403 Forbidden with the message "Measurements for this resource are not shared". Scores and the leaderboard are not affected.

Selecting a Challenge

First, retrieve the information about the challenge you want to submit a forecast for. In the example below, we will select a challenge for a specific market session (open_market_session_id) and target resource name (resource_name).

download_raw_measurements.py
# Get the session id from `/market/session/` endpoint
open_market_session_id = "your_open_market_session_id"

# Resource name, as listed on the Programs pages of the documentation:
resource_name = "germany_solar_50hertz_ca"

# Request the challenges for the open market session:
response = requests.get(
    url=f"{API_URL}/market/challenge",
    params={'market_session': int(open_market_session_id),
            'resource_name': resource_name},
    headers=headers,
    timeout=30
)

# Check if the request was successful
if response.status_code == 200:
    challenges = response.json()
else:
    print("Failed to retrieve challenges.")
    print(f"Status code: {response.status_code}")
    print(response.text)
    exit()

List challenge resource info

This allows you to get more information on the data availability for this challenge target resource raw measurements.

download_raw_measurements.py
# A session can hold several challenges for this resource (one per
# horizon); any of them works here, since we only need the resource UUID.
selected_challenge = challenges["data"][0]
# Unpack selected challenge information
resource_id = selected_challenge['resource'] # challenge resource UUID
print("Selected Challenge:")
print(f"Challenge ID: {selected_challenge['id']}")
print(f"Challenge Resource ID: {resource_id}")

# Get information on the data availability for this resource:
response = requests.get(
    url=f"{API_URL}/user/resource",
    params={"resource": resource_id},
    headers=headers,
    timeout=30
)
print("-"*79)
print("Resource Information:")
print(response.json()["data"])

Retrieving Raw Data for a Challenge

Retrieve the raw data for the selected resource:

download_raw_measurements.py
# Download raw measurements data for this resource:
start_date = response.json()["data"][0]["measurements_metadata"]["start_datetime"]
end_date = response.json()["data"][0]["measurements_metadata"]["end_datetime"]
params = {
    "resource": resource_id,
    "start_date": start_date,
    "end_date": end_date
}
# Download data:
next_url = f"{API_URL}/data/raw-measurements/"
dataset = []

# -- Note: This will stop once all the samples are retrieved.
# -- next_url indicates the URL of the next page (pagination) to be requested)

first_request = True
while next_url is not None:
    print(f"Requesting data...\n{next_url}") # This may take a while
    # The first request needs `params`; subsequent `next_url` values returned
    # by the paginator already contain every query parameter, so we drop them.
    response = requests.get(
        url=next_url,
        params=params if first_request else None,
        headers=headers,
        timeout=30,
    )
    first_request = False
    if response.status_code == 403:
        # This market does not share its measurements (see the note on the
        # "Downloading Raw Data" page). Scores are unaffected.
        print("This market does not share its measurements; nothing to download.")
        print(response.json().get("data", response.text))
        exit(0)
    if response.status_code != 200:
        print(f"Failed to retrieve measurements: HTTP {response.status_code}")
        print(response.text)
        exit(1)
    dataset += response.json()["data"]["results"]
    next_url = response.json()["data"]["next"]
    # -- Note: This will stop once all the samples are retrieved.

Download Full Example

JSON Example Response

After running the example script, you will have access to the full dataset of raw measurements data published by the Market Owner. If no data is received, please confirm if you are requesting data for the right resource identifier.

Click to view Example Response 
{
  "code": 200,
  "data": {
    "count": 2,
    "next": null,
    "previous": null,
    "results": [
      {
        "datetime": "2024-05-20T09:15:00Z",
        "value": 0.182,
        "units": "mw",
        "resource": "b92c96d1-f5ee-4f96-a4cc-216a92acb10b",
        "registered_at": "2024-06-24T09:19:19.428512Z",
        "resource_name": "germany_solar_50hertz_ca"
      },
      {
        "datetime": "2024-05-20T09:30:00Z",
        "value": 0.772,
        "units": "mw",
        "resource": "b92c96d1-f5ee-4f96-a4cc-216a92acb10b",
        "registered_at": "2024-06-24T09:19:19.428512Z",
        "resource_name": "germany_solar_50hertz_ca"
      }
    ]
  }
}