Skip to content

Downloading Raw Data

After selecting a challenge, the next step is to download the raw observed data associated with the challenge's resource. Market Makers publish time-series data periodically into our platform, which includes raw measurement data for their registered resources.

Important

  • The data is provided at a 15-minute resolution, resulting in a large volume of samples. Therefore, pagination (Limit/Offset Strategy) is required to retrieve all data properly.

API Endpoints:

To interact with the Predicto API and retrieve information about the raw measurements for a specific challenge target resource, you can use the following endpoints:

Access Token Required

An access token must be included in the Authorization header of your request. If you haven't obtained an access token yet, please refer to the Authentication section.

Check this out

Check our Useful Links section for additional resources (Jupyter Notebook, API Specifications) to help you get started with the Predico platform.

Selecting a Challenge

First, retrieve the information about the challenge you want to submit a forecast for. In the example below, we will select a challenge for a specific market session (open_market_session_id) and target resource name (resource_name).

download_raw_measurements.py
# Get the session id from `/market/session/` endpoint
open_market_session_id = "your_open_market_session_id"

# Challenge resource identifier (name):
resource_name = "wind_farm_1"

# Request the challenges for the open market session:
response = requests.get(
    url='https://predico-elia.inesctec.pt/api/v1/market/challenge',
    params={'market_session': int(open_market_session_id),
            'resource_name': resource_name},
    headers=headers
)

# Check if the request was successful
if response.status_code == 200:
    challenges = response.json()
else:
    print("Failed to retrieve challenges.")
    print(f"Status code: {response.status_code}")
    exit()

List challenge resource info

This allows you to get more information on the data availability for this challenge target resource raw measurements.

download_raw_measurements.py
# Select the first challenge of the list of challenges previous retrieved
# Note that as we're filtering by resource_name, there should be only a
# single challenge per session
selected_challenge = challenges["data"][0]
# Unpack selected challenge information
resource_id = selected_challenge['resource'] # challenge resource UUID
print("Selected Challenge:")
print(f"Challenge ID: {selected_challenge['id']}")
print(f"Challenge Resource ID: {resource_id}")

# Get information on the data availability for this resource:
response = requests.get(
    url='https://predico-elia.inesctec.pt/api/v1/user/resource',
    params={"resource": resource_id},
    headers=headers
)
print("-"*79)
print("Resource Information:")
print(response.json()["data"])

Retrieving Raw Data for a Challenge

Retrieve the raw data for the selected resource:

download_raw_measurements.py
# Download raw measurements data for this resource:
start_date = response.json()["data"][0]["measurements_metadata"]["start_datetime"]
end_date = response.json()["data"][0]["measurements_metadata"]["end_datetime"]
params = {
    "resource": resource_id,
    "start_date": start_date,
    "end_date": end_date
}
# Download data:
next_url = "https://predico-elia.inesctec.pt/api/v1/data/raw-measurements/"
dataset = []

# -- Note: This will stop once all the samples are retrieved.
# -- next_url indicates the URL of the next page (pagination) to be requested)

while next_url is not None:
    print(f"Requesting data...\n{next_url}") # This may take a while
    response = requests.get(url=next_url, params=params, headers=headers)
    dataset += response.json()["data"]["results"]
    next_url = response.json()["data"]["next"]
    # -- Note: This will stop once all the samples are retrieved.

print("-"*79)
print(f"Retrieved {len(dataset)} records")
print("Challenge observed data (first 10 records preview):")
print(dataset[:10])

Download Full Example

JSON Example Response

After running the example script, you will have access to the full dataset of raw measurements data published by the Market Maker. If no data is received, please confirm if you are requesting data for the right resource identifier.

Click to view Example Response 
{
  "code": 200,
  "data": {
    "count": 2,
    "next": null,
    "previous": null,
    "results": [
      {
        "datetime": "2024-05-20T09:15:00Z",
        "value": 0.182,
        "units": "mw",
        "resource": "b92c96d1-f5ee-4f96-a4cc-216a92acb10b",
        "registered_at": "2024-06-24T09:19:19.428512Z",
        "resource_name": "wind_farm_1"
      },
      {
        "datetime": "2024-05-20T09:30:00Z",
        "value": 0.772,
        "units": "mw",
        "resource": "b92c96d1-f5ee-4f96-a4cc-216a92acb10b",
        "registered_at": "2024-06-24T09:19:19.428512Z",
        "resource_name": "wind_farm_1"
      }
    ]
  }
}

What's next?

Learn how to prepare a forecast submission in the Preparing a Forecast section.