Guarantees of Origin

Solback Data Pipeline

This service collects European energy certificate auction results, standardizes them, and exposes clean analytics-ready data for market insights. Built to support analysts working with renewable energy producers.

What it does

Solback pulls the latest auction ZIP files, extracts structured data, and stores normalized results for fast, filtered access. It helps answer questions about price trends, technology performance, and market direction.

Technology

Go + Gin for the API, GORM for persistence, PostgreSQL for storage, and a pipeline service to orchestrate ingestion. OpenAI assists with HTML and spreadsheet parsing where needed.

Why it matters

Renewable producers can benchmark their latest price against market averages, compare across technologies, and understand long-term trends to improve bidding strategies.

Data flow

1

Load source URLs from the database and fetch HTML pages.

2

Extract the newest ZIP link, download the file, and unpack XLSX data in memory.

3

Normalize auction rows, compute aggregates, and store results in Postgres.

4

Expose the data through GET endpoints for filtering, grouping, and analysis.

GET endpoints

GET /health service status
GET /sources configured data sources
GET /logs ingestion logs
GET /data auction data with filters
GET /refresh trigger a refresh run

Product + DevOps notes

Product notes: Solback powers the Sol frontend by delivering structured auction data and filters for year, technology, and aggregation. The UI focuses on quick trend insight with interactive charts while keeping the raw dataset accessible.

DevOps overview: Kubernetes runs Postgres as a StatefulSet with a 1Gi PVC, the Go API as a Deployment with config + secret mounts, and the React frontend as a separate Deployment. Traefik + Lets Encrypt terminate TLS for both domains, and images are pulled from GHCR via a registry secret.

Known issues: The refresh cron lives inside the API pod, so scaling the deployment would duplicate refresh runs. The UI does not surface a "last data received" timestamp yet.