Hi, I'm Sarun Saji

Danish Power Data Pipeline

A production data platform for the Danish electricity market: every night it ingests day-ahead prices, wind & solar forecasts, CO₂ intensity and household consumption for the whole country — 31.8 million rows spanning 5+ years. Every morning a machine-learning model predicts the next day's 24 hourly prices before the market auction closes, scored publicly as the official prices land. Runs 24/7 and deploys itself on every git push.

• ⚙️ Orchestrated ingestion (Dagster) — 8 assets across 3 data providers with nightly self-healing re-materialization; the initial 5-year backfill ran as 85 resumable, state-tracked jobs that survived mid-run network failures.
• 🗄️ Time-series storage (TimescaleDB) — hypertables with columnar compression (6 GB → 563 MB, 10.7×) and continuous aggregates that cut the heaviest dashboard query from 2,195 ms to 19 ms (116×).
• 📈 ML price forecasting (LightGBM) — predicts tomorrow's hourly prices each morning before the 12:00 auction from leak-free features (price history, point-in-time weather, gas settlements). Walk-forward backtested: mean error 0.19 DKK/kWh vs 0.29 naive (−32%), scored live on the dashboard.
• 🛡️ Idempotent & defensive by design — batched upserts make every partition safely re-runnable; the fetch layer retries rate limits and network failures, and stitches the hourly and 15-minute market eras across the 2025 Nordic settlement switch.
• 🚀 CI/CD to production — every push runs the test suite, then deploys both the pipeline daemon and the dashboard container to a self-managed VPS; the dashboard reads through a least-privilege read-only database role.