Skip to content

Downloading Raw Data

After selecting a challenge, the next step is to download the raw observed data associated with the challenge's resource. Market Makers send time-series data periodically, which includes raw measurement data for their registered resources (e.g., wind farm #1).

Important

  • The data is provided at a 15-minute resolution, resulting in a large volume of samples. Therefore, pagination (Limit/Offset Strategy) is required to retrieve all data properly.

API Endpoints:

To interact with the Predicto API and retrieve information about the raw measurements for a specific challenge target resource, you can use the following endpoints:

Access Token Required

An access token must be included in the Authorization header of your request. If you haven't obtained an access token yet, please refer to the Authentication section.

Check this out

Check our Useful Links section for additional resources (Jupyter Notebook, API Specifications) to help you get started with the Predico platform.

Selecting a Challenge

First, select a challenge from the list of challenges you have retrieved:

download_raw_measurements.py
# Select the first challenge of the list of challenges previous retrieved
selected_challenge = challenges["data"][0]
resource_id = selected_challenge['resource']
print("Selected Challenge:")
print(f"Challenge ID: {selected_challenge['id']}")
print(f"Challenge Resource ID: {resource_id}")

List challenge resource info

This allows you to get more information on the data availability for this challenge target resource raw measurements.

download_raw_measurements.py
response = requests.get(
    url=f'https://predico-elia.inesctec.pt/api/v1/user/resource',
    params={"resource": resource_id},
    headers=headers
)
print("-"*79)
print("Resource Information:")
print(response.json()["data"])

Retrieving Raw Data for a Challenge

Retrieve the raw data for the selected resource:

download_raw_measurements.py
# Download raw measurements data for this resource:
start_date = response.json()["data"][0]["measurements_metadata"]["start_datetime"]
end_date = response.json()["data"][0]["measurements_metadata"]["end_datetime"]
params = {
    "resource": resource_id,
    "start_date": start_date,
    "end_date": end_date
}
# Download data:
next_url = "https://predico-elia.inesctec.pt/api/v1/data/raw-measurements/"
dataset = []

# -- Note: This will stop once all the samples are retrieved.
# -- next_url indicates the URL of the next page (pagination) to be requested)

while next_url is not None:
    print(f"Requesting data...\n{next_url}") # This may take a while
    response = requests.get(url=next_url, params=params, headers=headers)
    dataset += response.json()["data"]["results"]
    next_url = response.json()["data"]["next"]
    # -- Note: This will stop once all the samples are retrieved.

print("-"*79)
print(f"Retrieved {len(dataset)} records")
print(f"Challenge observed data (first 10 records preview):")
print(dataset[:10])

Download Full Example

JSON Example Response

After running the example script, you will have access to the full dataset of raw measurements data published by the Market Maker. If no data is received, please confirm if you are requesting data for the right resource identifier.

Click to view Example Response 
{
  "code": 200,
  "data": {
    "count": 2,
    "next": null,
    "previous": null,
    "results": [
      {
        "datetime": "2024-05-20T09:15:00Z",
        "value": 0.182,
        "units": "mw",
        "resource": "b92c96d1-f5ee-4f96-a4cc-216a92acb10b",
        "registered_at": "2024-06-24T09:19:19.428512Z",
        "resource_name": "wind_farm_1"
      },
      {
        "datetime": "2024-05-20T09:30:00Z",
        "value": 0.772,
        "units": "mw",
        "resource": "b92c96d1-f5ee-4f96-a4cc-216a92acb10b",
        "registered_at": "2024-06-24T09:19:19.428512Z",
        "resource_name": "wind_farm_1"
      }
    ]
  }
}

What's next?

Learn how to prepare a forecast submission in the Preparing a Forecast section.