Skip to content

Pipelines Reference

One entry per application under apps/. Each app's own README.md is the authoritative detail — column lists, DDL contracts, algorithm notes. This page is the map: what exists, what it consumes, what it publishes, and when it runs.

Schemas are written with their development defaults (ds_dev_staging, ds_dev_silver, ds_dev_gold). All cron expressions are UTC.

Index

Pipeline Kind Schedule Ingests via
om Open data — weather 4 flows, see below Meltano
mt Open data — weather 5 * * * * Meltano
owm Open data — weather 2 * * * * Meltano
copernicus Open data — climate 0 0,6,12,18 * * * Meltano
dwd Open data — weather paused Meltano
weather Facade 15 * * * * — (dbt only)
osm Open data — geospatial 0 0,6,12,18 * * * Meltano
overture Open data — geospatial 0 2 * * * Meltano
trentino_rooftops Open data — geospatial 0 2 * * * Meltano
pv_estimation Analysis 0 4 * * 1 Python + Meltano
pv_detection Analysis 0 3 * * 0 Python
grid Analysis 0 8 * * *, 20/15 * * * * — (dbt only)
rec_registry Mirror */5 * * * * Python
rec_flexibility_commitments Mirror */15 * * * * Python
rec_metering REC */10 * * * * — (dbt only)
rec_it REC */15 * * * *, 0 0 * * * Meltano
rec_flexibility REC 0 6 * * * — (dbt + Python)

Weather and climate

om — Open-Meteo

Four independent Prefect flows over the free Open-Meteo API (no API key; CC-BY 4.0), sharing one image and separated by dbt tag.

Flow Cron Purpose
om-weather-daily 0 6 * * * 7-day hourly forecast, engineered ML features
om-wind-trentino 0 0,4,8,12,16,20 * * * Wind speed/gust/direction on a 4.4 km Trentino grid (~798 points)
om-heat-trentino 30 7 * * * Heat risk by altitude band (P90)
om-obs 15 1,3,5,7,9,11,13,15,17,19,21,23 * * * 15-minute observations
Layer Datasets
raw raw.om_weather, raw.om_weather_wind, raw.om_heat, raw.om_obs
staging stg_om_weather, stg_om_wind, stg_om_heat, stg_om_obs
silver om_weather_hourly, om_wind_hourly, om_heat_daily, om_obs_15min
gold om_weather_features, om_weather_features_meters, om_weather_hourly_view, om_wind_gusts, om_heat_risk

om_weather_features carries 29 engineered features: temporal and Fourier encodings, rolling statistics, thermal dynamics and interaction terms, for downstream energy forecasting. The wind flow uses POST rather than GET against the API to avoid URL-length limits and per-location rate limiting.

Downstream: apps/grid (grid_wind_risks, grid_heat_risks and their nowcasting variants).

This is the only app in the repository with a packages.yml (dbt_utils), and the one with the broadest test coverage — use it as the reference for how a mature app is tested.

mt — MeteoTrentino

Regional weather from the meteorological service of the Autonomous Province of Trento: station observations, hourly and daily forecasts, weather alerts and reference data.

Layer Datasets
raw MeteoTrentino ASMX / meteo.report / API Manager responses
staging stg_mt_meteo_stations, stg_mt_station_observations, stg_mt_forecasts_hourly, stg_mt_forecasts_daily, stg_mt_alerts, stg_mt_forecast_locations, stg_mt_sky_conditions
silver the same seven, curated
gold mt_stations, mt_observations_current, mt_forecast_hourly, mt_forecast_daily, mt_alerts_active, mt_heat_daily_obs

It also publishes the four provider-neutral contract tables consumed by the weather facade — mt_weather_forecast_hourly, mt_weather_forecast_daily, mt_weather_alerts and mt_weather_current, materialised under the aliases weather__forecast_hourly, weather__forecast_daily, weather__alerts and weather__current.

owm — OpenWeatherMap

One Call API 3.0 observations and forecasts. Requires OWM_API_KEY. Licence constraints apply to redistribution.

Layer Datasets
raw API responses
staging stg_forecast_stream
silver weather_current, weather_minutely, weather_hourly, weather_daily, weather_alerts
gold folgaria_weather_* (per-location views) and an rdf_weather_* family publishing the same observations in a semantic, unpivoted shape

The rdf_weather_observations_unpivoted model is the SOSA-shaped projection: one row per observed property. See the companion's knowledge for why the unpivot is required rather than optional.

copernicus — ERA5 and CAMS

Reanalysis from the Copernicus Climate Change Service and Atmosphere Monitoring Service. Requires CDSAPI_KEY and CDSAPI_URL. Copernicus licence: free, attribution required.

Layer Datasets
staging stg_era5_single_levels, stg_cams_global_reanalysis_eac4_monthly, stg_radiation_raw
silver copernicus_era5_single_levels, copernicus_cams_global_reanalysis_eac4_monthly, copernicus_solar_radiation, copernicus_weather_features
gold copernicus_era5_analysis

dwd — ICON-D2 (paused)

Status: paused. Wind functionality was replaced by the om wind flow, which reads the same ICON-D2 model through the Open-Meteo API. The DWD pipeline had an 8% success rate and a 12+ GB database footprint; om-wind achieves the same coverage at ~90 MB/day. The code is retained because it is the only path to raw GRIB2 fields.

Layer Datasets
staging stg_dwd_icon_d2
silver dwd_icon_d2_silver, dwd_icon_d2_wind, dwd_icon_d2_solar_forecast_interval
gold dwd_icon_d2_gold, dwd_icon_d2_gusts, dwd_icon_d2_solar_energy

weather — multi-provider facade

A dbt-only pipeline with no ingestion of its own. It reads the four provider-neutral weather__* contract tables from the gold schema and produces stable, deduplicated consumer tables: for each entry in the weather_locations seed it keeps the nearest provider location.

Layer Datasets
sources weather__forecast_hourly, weather__forecast_daily, weather__alerts, weather__current (gold, any provider)
staging stg_weather_forecast_hourly, stg_weather_forecast_daily, stg_weather_alerts, stg_weather_current
gold weather_forecast_hourly, weather_forecast_daily, weather_alerts_active, weather_current

Scheduled at 15 * * * *, ten minutes after mt, so each hour's facade reflects that hour's provider run.

Downstream: digital-twin, celine-webapp.

Adding a provider means writing that provider's gold model under the weather__* alias. This pipeline needs no change.

Geospatial

osm — OpenStreetMap

Regional extracts curated into thematic layers for the areas the platform covers. Licence: ODbL-1.0, attribution required.

Layer Datasets
staging openstreetmap_it_alpecimbra, openstreetmap_fi_lappeenranta
silver per-area thematic layers: _base, _amusement, _ev_charging, _hospitality, _parking, _piste_and_lifts, _tourism_culture, _social_buildings

overture — Overture Maps buildings

Building footprints for the Trentino region, the entry point of the PV chain.

Layer Datasets
raw raw.overture_buildings
staging stg_overture_buildings
silver pv_overture_buildings — building_id is md5(ST_AsText(geometry)), stable across runs

Downstream: pv_estimation, pv_detection, trentino_rooftops.

trentino_rooftops — PV siting constraints

Regional open datasets describing where PV may and may not be installed.

Layer Datasets
staging stg_aree_non_idonee, stg_vincoli_diretti, stg_vincoli_indiretti
silver pv_aree_non_idonee, pv_vincoli_diretti, pv_vincoli_indiretti
gold pv_building_suitability — per-building eligibility after constraint overlay

Requires PostGIS.

Photovoltaic analysis

pv_estimation — rooftop PV ROI

Estimates the return on investment of a rooftop PV installation for each eligible building, using the celine-roi library. Includes a Streamlit dashboard for policy makers and REC managers at province and primary-substation level.

Layer Datasets
raw raw.pv_roi_estimates — written by the Prefect flow, one row per building (NPV, IRR, payback, tasso_autoconsumo, regime)
staging stg_pv_roi_estimates
silver pv_roi_estimates
gold pv_rooftop_opportunities, pv_installation_ranking, pv_installation_plan, pv_installation_plan_summary, rec_building_cabina, rec_cabina_opportunities, rec_cabina_plan, rec_cabina_summary

Sources: pv_overture_buildings (overture, silver) and gse_cabine_primarie (rec_it, gold). The ROI computation is a Python task, not dbt — apps/pv_estimation/flows/roi_estimator.py, run incrementally with a --full-refresh escape hatch.

pv_detection — existing PV from imagery

Detects PV already installed on rooftops from aerial orthophotos using a vision-language model (Ollama, e.g. Qwen2.5-VL 7B). Building-driven: it queries pv_overture_buildings, fetches the aerial tile covering each footprint, crops the rooftop and classifies it, skipping crops where less than half the building is visible.

Layer Datasets
raw raw.pv_predictions — has_pv, confidence, model_name, reasoning, centroid
staging stg_pv_predictions
silver pv_detections
gold pv_detected_installations

Complements pv_estimation: that one answers should a building have PV, this one answers does it already.

Imagery licensing is not uniform. Tile sources differ in what they permit for derived-data publication. Check the licence of the provider you configure before exposing any output.

Grid

grid — grid resilience

CIM-inspired wind and heat risk overlays for the distribution network, joining grid topology with Open-Meteo forecasts. Requires PostGIS. dbt-only; no ingestion of its own.

Flow Cron Purpose
grid-resilience-daily 0 8 * * * Daily risk overlay, after the om wind and heat flows
grid-nowcasting-15min 20/15 * * * * Nowcast refresh from 15-minute observations
Kind Datasets
gold grid_network_topology, grid_substations, grid_shapes, grid_tiles, grid_tile_index, grid_wind_risks, grid_wind_risks_now, grid_heat_risks, grid_heat_risks_now, grid_risks, grid_risks_now, grid_risks_trendline, v_superset_grid

Upstream (private): two CIM-normalised silver tables produced by a DSO-specific ingestion pipeline that is not part of this repository — silver_grid_ac_line_segment and the substation table. The expected columns are declared in apps/grid/dbt/models/sources.yml, which is the contract; README.md states the DDL so the pipeline can be run against locally-created tables.

A column "strike_tree_*" does not exist error means the local silver table predates an upstream change, not a bug in this pipeline.

REC and community

These four pipelines form a chain. rec_metering publishes the metering interface; rec_it and rec_flexibility both read it; the two mirrors feed the raw tables those need.

[private metering ingestion]        [rec-registry API]      [flexibility-api]
   silver.meters_data_normalized          |                        |
              |                     rec_registry            rec_flexibility_commitments
              v                            |                        |
        rec_metering                raw.rec_registry_mirror  raw.flexibility_commitments_mirror
   gold.meters_data_15m / _1h              |                        |
              |______________________ ____ | ______________________ |
                                     |                              |
                                     v                              v
                                  rec_it                      rec_flexibility

rec_metering — metering interface

Promotes normalised 15-minute meter readings from silver into the gold tables every other REC pipeline consumes. dbt-only.

Kind Datasets
source {{CELINE_SILVER_SCHEMA}}.meters_data_normalized (private upstream)
gold meters_data_15m, meters_data_1h, meters_data_15m_missing_intervals, meters_measurements_15m
  • meters_data_15m — deduplicated and summed per (device_id, ts), incremental merge on md5(device_id || ts). Energy in kWh per 15-minute bucket.
  • meters_data_1h — hourly rollup of the above.
  • meters_data_15m_missing_intervals — quality model. Generates the complete expected 15-minute grid per device over the last 7 days and left-joins actuals, surfacing gaps.
  • meters_measurements_15m — the SOSA-shaped unpivot of meters_data_15m. It shares its governance block with the parent by YAML anchor, deliberately: a semantic sibling that quietly lost its rec_registry row filter would serve every device's readings to every caller.

This is the smallest and most load-bearing pipeline in the repository. Its gold tables are the contract for rec_it and rec_flexibility, so a change here is a change to both.

rec_it — Italian CER settlement

Virtual self-consumption allocation per device under Italian GSE rules, plus the primary substation reference layer.

Flow Cron Purpose
rec_it_flow */15 * * * * Settlement models
rec_it_batches_flow 0 0 * * * Daily batch reprocessing
Kind Datasets
sources meters_data_15m (rec_metering, gold); rec_device_membership and raw.rec_registry_mirror (rec_registry); raw.gse_cabine_primarie (self-contained Meltano extractor)
silver silver_gse_cabine_primarie
gold gse_cabine_primarie, rec_virtual_consumption_15m, rec_virtual_consumption_hourly, rec_virtual_consumption_per_device_15m, rec_virtual_consumption_per_device_hourly, rec_measurements_15m, rec_measurements_per_device_15m, rec_member_supply_points

Per-substation netting is correct and must not be "simplified". The community figure is least() per (ts, community_id, substation_id) and then summed. Sharing cannot cross an unconnected cabina primaria; netting community-wide would invent energy that physically cannot flow. See the companion's knowledge, which also records the known defect in the hourly model.

The netting is only as correct as the substation attribution. rec_device_membership (rec_registry) takes topology_ids[1] of the member's area as its substation_id. That is right when an area lists exactly one primary_substation node, whose id is the cod_ac of the area's boundary; an area listing several nodes puts all its members under the first. The registry enforces it from schema v0.7 on area writes, topology node writes and bundle import. The mirror carries the area's boundary_id, and the singular test rec_registry_mirror_substation_is_area_boundary (rec_registry) fails on any area breaking it. See apps/rec_registry/README.md.

A membership change (meter attached or detached, role or area changed) reaches only the rows computed after it: the settlement models are incremental over a short lookback, and per-device keys include substation_id. Aligning history needs a bounded recompute by an operator; see apps/rec_it/README.md.

Gold gse_cabine_primarie is read at runtime by the Digital Twin (through dataset-api) to resolve community areas, on cod_ac and geometry; its merge never deletes a substation.

rec_flexibility — flexibility and gamification

Opportunity windows, per-commitment settlement with proportional redistribution, three-layer gamification scoring, anti-gaming flags and CO2 impact. Mixes dbt models with Python tasks (baselines, streaks, auto-commit). 0 6 * * *.

Kind Datasets
sources meters_data_15m (rec_metering, gold); meters_energy_forecast and total_meters_forecast (meter forecasting, not in this repository); raw.flexibility_commitments_mirror
silver rec_meters_15m, silver_flexibility_commitments
gold rec_flexibility_windows, rec_flexibility_windows_community, rec_device_baselines, rec_device_class, rec_device_streaks, rec_settlement_15m, rec_settlement_1h, rec_settlement_points, rec_commitment_settlement, rec_flexibility_bonus, rec_participant_points, rec_points_leaderboard, rec_gamification_summary, rec_anti_gaming_flags, rec_co2_savings, rec_co2_savings_community

Without the forecast tables the windows model produces no output — no surplus is detected, and the pipeline succeeds with an empty result. That is the expected local behaviour, not a failure.

The points fleet is the registry's membership. Points, settlement, streaks and the leaderboard cover every device of rec_device_membership (every role), and every figure computed across devices is computed within one community.

The flexibility signal is netted across substations. There is no join path from this app to substation_id at all; a deficit on one cabina cancels a surplus on another.

rec_registry — registry mirror

Full-replace mirror of the CELINE REC Registry API into raw.rec_registry_mirror, every 5 minutes. One row per user/community pair of an active member: area, role, member type, topology nodes, the area's boundary id, delivery points in service (a point flagged active: false is left out), meter sensors. Python-only; no dbt, no Meltano. OIDC-authenticated. Each run logs the areas whose topology_ids[1] is not their boundary id (it flags them; it does not refuse the export).

The mirror always equals the export's active members, the empty set included. When the export holds no active member the table is emptied, so a suspended last member stops feeding rec_it. A fetch that fails or returns an empty export raises before the mirror is touched, and the previous rows stay.

rec_flexibility_commitments — commitments mirror

90-day sliding mirror of the CELINE Flexibility API into raw.flexibility_commitments_mirror, every 15 minutes, tracking status evolution (accepted, settled, rejected, cancelled). Python-only. OIDC-authenticated.

Pipelines with no README.md

overture and trentino_rooftops have no per-pipeline README yet; their entries above are derived from the models and flow configuration. Adding one is the first thing to do when next working in either.