{ "cells": [ { "cell_type": "markdown", "id": "9941b0f1-c585-45f6-997a-1dd0ff7c6759", "metadata": { "editable": true, "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "

Predico - Collaborative Forecasting Platform

\n", "\n", "

Forecaster Jupyter NB

\n", "\n", "This notebook illustrates and provides Python code snippets allowing interaction between individual forecasters (aka Market participants) and Predico Platform. \n", "\n", "Below you can find examples for the following functionalities:\n", "\n", "* Authentication\n", "* Listing open sessions & challenges\n", "* Downloading raw data for a resource (published by a market maker)\n", "* Prepare and submit forecast for a challenge\n", "* Preview submissions for a challenge\n", "* Preview forecast skill scores for challenge submissions\n", "* Preview forecast contributions (to the final ensemble models) for challenge submissions\n", "\n", "----\n", "## Important\n", "\n", "### Application:\n", "* Forecasters apply to participate by sending an email to predico@elia.be.\n", "* Upon acceptance (by Elia), they receive a registration link to complete their registration. \n", "\n", "### Useful Links:\n", "\n", "* MainPage: https://predico-elia.inesctec.pt/\n", "\n", "### Contacts:\n", "* Giovanni Buroni (giovanni.buroni@inesctec.pt)\n", "* Carla Gonçalves (carla.s.goncalves@inesctec.pt)\n", "* André Garcia (andre.f.garcia@inesctec.pt)\n", "* José Andrade (jose.r.andrade@inesctec.pt)\n", "* Ricardo Bessa (ricardo.j.bessa@inesctec.pt)\n", "\n", "#### Last update:\n", "2024-10-31, by José Andrade" ] }, { "cell_type": "markdown", "id": "a2415a8d-2f30-49bd-8342-bbabaf736477", "metadata": {}, "source": [ "### Install dependencies" ] }, { "cell_type": "code", "execution_count": null, "id": "a81d5704-6323-4b38-ad78-78764f1cda37", "metadata": {}, "outputs": [], "source": [ "# Note: you just need to run this once - to configure your Python Interpreter:\n", "!pip install requests pandas" ] }, { "cell_type": "markdown", "id": "ca207fe3-63d0-49e5-905c-f4718d54aa66", "metadata": {}, "source": [ "### Imports / Helper functions" ] }, { "cell_type": "code", "execution_count": null, "id": "a55080e799bc521", "metadata": { "jupyter": { "is_executing": true } }, "outputs": [], "source": [ "# Imports\n", "import os\n", "import json\n", "import getpass\n", "import requests\n", "import numpy as np\n", "import pandas as pd\n", "import datetime as dt\n", "\n", "from pprint import pprint" ] }, { "cell_type": "code", "execution_count": null, "id": "ea83124e-8b2f-46d3-803d-47bde5fd5d05", "metadata": {}, "outputs": [], "source": [ "# Helper funcs:\n", "def http_request(request_type, task_msg, verbose=1, **kwargs):\n", " print(\"-\"*79)\n", " print(f\"[{request_type}] {kwargs['url']} {task_msg} ...\")\n", " if request_type == \"GET\":\n", " response = requests.get(**kwargs, verify=\"ca.crt\")\n", " response_codes = [200]\n", " elif request_type == \"POST\":\n", " response = requests.post(**kwargs, verify=\"ca.crt\")\n", " response_codes = [200, 201]\n", " elif request_type == \"PUT\":\n", " response = requests.put(**kwargs, verify=\"ca.crt\")\n", " response_codes = [200, 201]\n", " \n", " if response.status_code in response_codes:\n", " print(\"Success! Response:\")\n", " if verbose > 0:\n", " pprint(response.json())\n", " print(f\"[GET] {task_msg} ... Ok!\")\n", " else:\n", " print(\"Error! Response:\")\n", " if verbose > 0:\n", " pprint(response.json())\n", " print(f\"[GET] {task_msg} ... Failed!\")\n", " \n", " return response\n" ] }, { "cell_type": "markdown", "id": "2320c764-11ee-410b-96a2-0c0ada375f72", "metadata": {}, "source": [ "## Preparation Steps - Set Forecaster Credentials & API URL references" ] }, { "cell_type": "code", "execution_count": null, "id": "71d99746-17b5-4974-9c8b-7d4927cf0ae4", "metadata": {}, "outputs": [], "source": [ "# Settings:\n", "predico_base_url = \"https://predico-elia.inesctec.pt/api/v1\"\n", "\n", "# Credentials:\n", "forecaster_email = input('Email:')\n", "forecaster_pw = getpass.getpass('Password:')" ] }, { "cell_type": "markdown", "id": "b6fb7825-fadb-409c-8fd3-b2d98bb6d9c2", "metadata": {}, "source": [ "***\n", "***" ] }, { "cell_type": "markdown", "id": "8575b63e-5df2-45fc-9ae3-ddb1c6dd595b", "metadata": {}, "source": [ "## Step 1 - Login as Individual Forecaster" ] }, { "cell_type": "code", "execution_count": null, "id": "f7832505-7329-47fb-8461-54641915f372", "metadata": {}, "outputs": [], "source": [ "# Auth request:\n", "login_data = {\"email\": forecaster_email, \n", " \"password\": forecaster_pw}\n", "response = http_request(request_type=\"POST\",\n", " task_msg=\"Authenticating ...\",\n", " url=f\"{predico_base_url}/token\",\n", " data=login_data)\n", "\n", "# Save Token (to be reused in next interactions)\n", "access_token = response.json().get('access', None)\n", "if access_token is None:\n", " print(\"Error! Unable to retrieve access token. Do not forget to validate your account before this step!\")\n", "else:\n", " AUTH_TOKEN = f\"Bearer {access_token}\"" ] }, { "cell_type": "markdown", "id": "f9f6d0e4-b66f-4c46-a0d8-f5ed8f71d87a", "metadata": {}, "source": [ "***\n", "***" ] }, { "cell_type": "markdown", "id": "c317f650-af96-44c4-867b-42537cb5bc77", "metadata": {}, "source": [ "# Step 2 - Forecaster interactions w/ Predico\n", "\n", "\n", "The basic Forecaster work flow is ilustrated in our [Quick Guide](https://predico-elia.inesctec.pt/quick-guide/):\n", " 1. The forecaster lists open market sessions and challenges (created by the market maker)\n", " 2. The forecaster selects a challenge and downloads raw data (to build a forecasting model)\n", " 3. The forecaster prepares and submits its forecast\n", "\n", "__Important__: \n", "***If the forecaster has never submitted a forecast, it is necessary to send at least 1month of historical forecasts data (e.g., which the forecaster used to initially evaluate the forecasting model performance). This is a requirement for the current version of the collaborative forecasting models which, if not fulfiled, will disable the submission of forecasts for a market session.***" ] }, { "cell_type": "markdown", "id": "7adf90e3-ee07-47c6-a0f7-dd69aa32f027", "metadata": {}, "source": [ "### 2.1. List open market session\n", " \n", "Market sessions are specific periods during which Market Makers can create forecasting challenges, and Forecasters can submit forecasts for the open challenges. These are managed (open/closed) by scheduled tasks (e.g., every day or hour).\n", "\n", "**(see the official [documentation](https://predico-elia.inesctec.pt/redoc/#tag/market/operation/get_market_session) for this endpoint)**" ] }, { "cell_type": "code", "execution_count": null, "id": "3479c092-5a46-4270-aa26-d1d771431db9", "metadata": {}, "outputs": [], "source": [ "# Get currently open session\n", "response = http_request(request_type='GET', \n", " task_msg=\"Listing registered sessions ...\",\n", " url=f\"{predico_base_url}/market/session\",\n", " params={\"status\": \"open\"},\n", " headers={\"Authorization\": AUTH_TOKEN})\n", "\n", "if len(response.json()[\"data\"]) > 0:\n", " open_market_session_id = response.json()[\"data\"][0][\"id\"]\n", "else:\n", " raise Exception(\"There are no open market sessions. Please open a session first.\")\n" ] }, { "cell_type": "markdown", "id": "3e9bdbde-f92a-4f5e-8edd-0c244bdf43f8", "metadata": {}, "source": [ "### 2.2 List collab forecasting challenges for open session\n", "\n", "Market Challenges are opportunities, published by the Market Maker (Elia), with meta-data regarding a forecasting challenge that Forecasters can try to submit forecasts for.\n", "\n", "**(see the official [documentation](https://predico-elia.inesctec.pt/redoc/#tag/market/operation/get_market_session_challenge) for this endpoint)**" ] }, { "cell_type": "code", "execution_count": null, "id": "dbf0cbcf-ddc6-4775-a163-405e59664986", "metadata": {}, "outputs": [], "source": [ "# Preview list of registered challenges for the currently open session id:\n", "response = http_request(request_type='GET', \n", " task_msg=f\"Listing registered challenges for session {open_market_session_id} ...\",\n", " url=f\"{predico_base_url}/market/challenge\",\n", " params={\"market_session\": open_market_session_id},\n", " headers={\"Authorization\": AUTH_TOKEN})\n", "\n", "challenges = response.json()[\"data\"]" ] }, { "cell_type": "markdown", "id": "77cd31f3-9caf-4eee-9635-d4aae0d095b3", "metadata": {}, "source": [ "### 2.3. Select a challenge and download raw observed data for the challenge resource\n", "\n", "Market Makers send, periodically, time-series with raw measurement data for their registered resources in the platform (e.g., wind farm #1).\n", "\n", "**(see the official [documentation](https://predico-elia.inesctec.pt/redoc/#tag/data/operation/get_raw_data) for this endpoint)**\n", "\n", "**Note: This step can be replicated for every challenge, we'll just select the first one**\n", "\n", "> __Important__: 15min resolution time-series are expected to be downloaded. Therefore pagination is required to enable proper access to the large volume of samples in historical data. Check the procedures below to know how to properly retrieve the datasets." ] }, { "cell_type": "code", "execution_count": null, "id": "6633168a-058f-4778-b1d8-20bd4efe9259", "metadata": {}, "outputs": [], "source": [ "# Select challenge:\n", "selected_challenge = challenges[0]\n", "print(\"Selected Challenge:\")\n", "pprint(selected_challenge)\n", "\n", "# Download raw measurements data for this resource:\n", "resource_id = selected_challenge[\"resource\"]\n", "start_date = \"2024-01-01T00:00:00Z\"\n", "end_date = \"2025-01-01T00:00:00Z\"\n", "params = {\n", " \"resource\": resource_id,\n", " \"start_date\": start_date,\n", " \"end_date\": end_date\n", "}\n", "# Download data:\n", "next_url = f\"{predico_base_url}/data/raw-measurements/\"\n", "dataset = []\n", "# -- Note: This will stop once all the samples are retrieved.\n", "# -- next_url indicates the URL of the next page (pagination) to be requested)\n", "while next_url is not None:\n", " response = http_request(request_type='GET',\n", " task_msg=f\"Downloading challenge data for resource {resource_id} ...\",\n", " url=next_url,\n", " params=params,\n", " headers={\"Authorization\": AUTH_TOKEN},\n", " verbose=0)\n", " dataset += response.json()[\"data\"][\"results\"]\n", " next_url = response.json()[\"data\"][\"next\"]\n", "\n", "dataset_df = pd.DataFrame(dataset)\n", "\n", "print(\"-\"*79)\n", "print(f\"Challenge observed data for resource {resource_id}\")\n", "dataset_df" ] }, { "cell_type": "markdown", "id": "dc851292-4af9-442b-839f-ac3bf6f93f0e", "metadata": {}, "source": [ "### 2.4. Forecaster prepares forecast\n", "\n", "**Note: In this example we just prepare a 24h submission (15 minute random samples) but it should include the preparation of a forecasting model (e.g., using the dataset retrieved in the previous step combined with any other variables the Forecaster might have access to)**" ] }, { "cell_type": "code", "execution_count": null, "id": "03fbfe63-69ab-4e8a-9ab9-4df49833f062", "metadata": { "scrolled": true }, "outputs": [], "source": [ "# Unpack selected challenge information\n", "resource_id = selected_challenge[\"resource\"]\n", "challenge_id = selected_challenge[\"id\"]\n", "start_datetime = selected_challenge[\"start_datetime\"]\n", "end_datetime = selected_challenge[\"end_datetime\"]\n", "\n", "print(f\"\"\"\n", "Challenge ID: {challenge_id}\n", "Resource ID: {resource_id}\n", "Start DT: {start_datetime}\n", "End DT: {end_datetime}\n", "\"\"\")\n", "\n", "# Create a random 24h submission:\n", "# Generate datetime values for the challenge period:\n", "datetime_range = pd.date_range(start=start_datetime, \n", " end=end_datetime, \n", " freq='15T')\n", "datetime_range = [x.strftime(\"%Y-%m-%dT%H:%M:%SZ\") for x in datetime_range]\n", "# Generate random values for the \"value\" column\n", "values = np.random.uniform(low=0.0, high=1.0, size=len(datetime_range))\n", "values = [round(x, 3) for x in values]\n", "\n", "# Reuse this data to prepare 3 different quantiles submissions Q10, Q50, Q90\n", "submission_list = []\n", "for qt in [\"q10\", \"q50\", \"q90\"]:\n", " qt_forec = pd.DataFrame({\n", " 'datetime': datetime_range,\n", " 'value': values,\n", " })\n", " submission_list.append({\n", " \"variable\": qt, \n", " \"forecasts\": qt_forec.to_dict(orient=\"records\")\n", " })\n", "\n", "# Your submissions:\n", "print(\"Submission List:\")\n", "for i, submission in enumerate(submission_list):\n", " print(\"-\"*79)\n", " print(f\"Submission #{i+1}\")\n", " print(json.dumps(submission, indent=3))" ] }, { "cell_type": "markdown", "id": "35d85cd6-7d99-4d64-b575-02ccb0aee153", "metadata": {}, "source": [ "### 2.5. Forecaster submits forecast\n", "\n", "Forecasters can submit forecasts for open Market Challenges, competing for the prize money available. They may commence or stop contributing to the market at any time.\n", "\n", "**(see the official [documentation](https://predico-elia.inesctec.pt/redoc/#tag/market/operation/post_market_session_submission) for this endpoint)**\n", "\n", "> __Important__: Before forecasters make their first submission, they must fulfil at least 1 month of historical forecast samples submitted to the Predico platform (current requirement). See the step 2.5.1 to find how to do this." ] }, { "cell_type": "code", "execution_count": null, "id": "d29dd90c-fd1c-4da5-8005-6430e000ba97", "metadata": {}, "outputs": [], "source": [ "# Unpack selected challenge information\n", "for submission in submission_list:\n", " response = http_request(request_type='POST',\n", " task_msg=f\"Submiting forecast for challenge '{challenge_id}' resource '{resource_id}' -- variable '{submission['variable']}' ...\",\n", " url=f\"{predico_base_url}/market/challenge/submission/{challenge_id}\",\n", " json=submission,\n", " headers={\"Authorization\": AUTH_TOKEN})" ] }, { "cell_type": "markdown", "id": "fec5132d-b2e9-4ec8-b2cd-01b01a4e3811", "metadata": {}, "source": [ "#### 2.5.1 (first forecast only)\n", "\n", "To assess their potential and define the ensemble methodology, a forecaster must submit a historical of time-series forecasts, with 15-minute time resolution, for a minimum of 30 days before the challenge start date (1st period to forecast). The forecaster should be careful with this step, as providing unrealistic historical forecasts (e.g., overfitting to historical data) might compromise its submission. The forecaster does not stand to gain more money by submitting a better historical forecast, as payment is based on ex-post performance calculations.\n", "\n", " * ***This step is mandatory for the first time a forecaster participates in a collaborative forecasting session and will be optional for the remaining (unless a gap of 30 days passes between participation in challenges).***\n", "\n", "**(see the official [documentation](https://predico-elia.inesctec.pt/redoc/#tag/data/operation/post_individual_forecasts_historical) for this endpoint)**\n", "\n", "> __Note__: The forecasts used to train the ensemble models will be a combination of historical forecasts provided by the Forecaster and forecasts submitted to actual challenges. **The former can be revised (using the same HTTP POST endpoint) until the Forecaster makes their first submission on the platform, the latter cannot be changed once the market session executes.**\n", "\n", "> __Important__: It is recommended to avoid the transfer of high volume of samples, in a single request. Otherwise, your request might fail.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "99f44e73-9b6a-44d6-8cde-c1f46de2bffa", "metadata": {}, "outputs": [], "source": [ "# Unpack selected challenge information\n", "resource_id = selected_challenge[\"resource\"]\n", "\n", "# Create 1month of historical data (sampled from uniform dist):\n", "# Generate datetime values for the challenge period:\n", "N_HISTORICAL_DAYS = 35\n", "\n", "dt_now = pd.to_datetime(dt.datetime.utcnow()).round(\"15min\")\n", "hist_datetime_range = pd.date_range(start=dt_now - pd.DateOffset(days=N_HISTORICAL_DAYS), \n", " end=dt_now.replace(hour=23, minute=45, second=00), \n", " freq='15min')\n", "hist_datetime_range = [x.strftime(\"%Y-%m-%dT%H:%M:%SZ\") for x in hist_datetime_range]\n", "\n", "# Generate random values for the \"value\" column\n", "hist_values = np.random.uniform(low=0.0, high=1.0, size=len(hist_datetime_range))\n", "hist_values = [round(x, 3) for x in hist_values]\n", "\n", "# Prepare historical data for 3 different quantiles submissions Q10, Q50, Q90\n", "hist_submission_list = []\n", "for qt in [\"q10\", \"q50\", \"q90\"]:\n", " qt_forec = pd.DataFrame({\n", " 'datetime': hist_datetime_range,\n", " 'value': hist_values,\n", " })\n", " hist_submission_list.append({\n", " \"resource\": resource_id,\n", " \"variable\": qt, \n", " \"launch_time\": dt_now.strftime(\"%Y-%m-%dT%H:%M:%SZ\"),\n", " \"forecasts\": qt_forec.to_dict(orient=\"records\")\n", " })\n", "\n", "# Submit forecast\n", "for submission in hist_submission_list:\n", " response = http_request(request_type='POST',\n", " task_msg=f\"Submiting forecast for challenge '{challenge_id}' resource '{resource_id}' ...\",\n", " url=f\"{predico_base_url}/data/individual-forecasts/historical\",\n", " json=submission,\n", " headers={\"Authorization\": AUTH_TOKEN})\n" ] }, { "cell_type": "markdown", "id": "69aeef35-b4c7-4336-9f65-85ae4f6f6c01", "metadata": {}, "source": [ "### 2.6. Forecaster previews submitted forecast\n", "\n", "Forecasters can preview their challenge submissions (list) at any point in time.\n", "\n", "**(see the official [documentation](https://predico-elia.inesctec.pt/redoc/#tag/market/operation/get_market_session_submission) for this endpoint)**" ] }, { "cell_type": "code", "execution_count": null, "id": "04001c07-f998-4a33-9ac5-002fd0494fa5", "metadata": {}, "outputs": [], "source": [ "# Preview my submissions\n", "response = http_request(request_type='GET',\n", " task_msg=f\"Retrieving submissions for challenge '{challenge_id}' resource '{resource_id}' ...\",\n", " url=f\"{predico_base_url}/market/challenge/submission\",\n", " params={\"challenge\": challenge_id},\n", " headers={\"Authorization\": AUTH_TOKEN}, \n", " verbose=0)\n", "\n", "pd.DataFrame(response.json()[\"data\"])" ] }, { "cell_type": "markdown", "id": "f181e8ab-a51c-48d1-ae5d-6dde11091152", "metadata": {}, "source": [ "### 2.7. Forecaster downloads submitted forecasts\n", "\n", "Forecasters can also preview their submitted forecast time-series.\n", "\n", "**(see the official [documentation](https://predico-elia.inesctec.pt/redoc/#tag/market/operation/get_market_session_submission_forecasts) for this endpoint)**\n" ] }, { "cell_type": "code", "execution_count": null, "id": "3a0be30f-495d-460a-bd10-b5ed88f952e1", "metadata": {}, "outputs": [], "source": [ "# Preview my submitted forecasts\n", "response = http_request(request_type='GET',\n", " task_msg=f\"Retrieving submissions for challenge '{challenge_id}' resource '{resource_id}' ...\",\n", " url=f\"{predico_base_url}/market/challenge/submission/forecasts\",\n", " params={\"challenge\": challenge_id},\n", " headers={\"Authorization\": AUTH_TOKEN}, \n", " verbose=0)\n", "\n", "pd.DataFrame(response.json()[\"data\"])" ] }, { "cell_type": "markdown", "id": "6e8cd4a2-8768-48f8-8d20-294981c8f605", "metadata": {}, "source": [ "### 2.8. Forecaster previews forecast skill scores\n", "\n", "Every day, a scheduled task calculates the submitted forecasts skill score (error metrics) and relative rank to the remaining forecasters participating in the platform. This information can also be retrieved with a request to the service API.\n", "\n", "**(see the official [documentation](https://predico-elia.inesctec.pt/redoc/#tag/market/operation/get_market_session_submission_scores) for this endpoint)**\n" ] }, { "cell_type": "code", "execution_count": null, "id": "7ea71722-0f2d-4548-855a-6c43aa4b2b99", "metadata": {}, "outputs": [], "source": [ "# Preview my skill scores\n", "response = http_request(request_type='GET',\n", " task_msg=f\"Retrieving submissions scores for challenge '{challenge_id}' resource '{resource_id}' ...\",\n", " url=f\"{predico_base_url}/market/challenge/submission-scores\",\n", " params={\"challenge\": challenge_id},\n", " headers={\"Authorization\": AUTH_TOKEN}, \n", " verbose=1)" ] }, { "cell_type": "markdown", "id": "5dbb73bb-6da5-480e-9049-1b41b63a4676", "metadata": {}, "source": [ "### 2.9. Forecaster previews forecasts contributions\n", "\n", "Every day, a scheduled task calculates the submitted forecasts contribution to the final ensemble and relative rank to the remaining forecasters participating in the platform. This information can also be retrieved with a request to the service API.\n", "\n", "**(see the official [documentation](https://predico-elia.inesctec.pt/redoc/#tag/market/operation/get_market_session_ensemble_weights) for this endpoint)**\n" ] }, { "cell_type": "code", "execution_count": null, "id": "dc4daefe-1e69-43ca-b375-5ae6c20d6921", "metadata": {}, "outputs": [], "source": [ "# Preview my forecast contributions\n", "response = http_request(request_type='GET',\n", " task_msg=f\"Retrieving submissions contributions for challenge '{challenge_id}' resource '{resource_id}' ...\",\n", " url=f\"{predico_base_url}/market/challenge/ensemble-weights\",\n", " params={\"challenge\": challenge_id},\n", " headers={\"Authorization\": AUTH_TOKEN}, \n", " verbose=1)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.1" } }, "nbformat": 4, "nbformat_minor": 5 }