Here is the case for declarative data pipelines. The standard advice for years has been to answer every data problem with more PySpark and more custom Python boilerplate. What that buys you is a bottleneck, where an analyst waits weeks on an engineer to ship a single metric.
I have watched this play out on plenty of projects. A “simple” data request comes in, and three weeks later you are still tuning Spark resource allocation and untangling Airflow DAG dependencies. Moving data around should not require a background in distributed systems.
Where the PySpark bottleneck comes from
On an architecture review a while back, a client was paying $40k a month for a cluster whose main job was counting Monthly Active Users (MAU). Adding one event source meant a developer wrote a custom PySpark script, waited on code review, then hoped no “Out of Memory” error fired at 3 AM. That does not hold up over time. If any of it sounds familiar, it may be worth rethinking how you are scaling real-time data pipelines.
The problem is not the code, it is ownership. Logic trapped in Python locks the analyst out of production, and the analyst is the person who understands the business requirements best. What you want instead are declarative data pipelines, where the config file doubles as the documentation.
The stack: dlt, dbt and Trino
We swapped the procedural code for a declarative setup built on three tools. Anyone who knows SQL and enough YAML to edit a config file can ship a production pipeline in about 24 hours.
- dlt (data load tool): pulls data out of APIs and databases from a YAML config.
- dbt (data build tool): runs on top of Trino and does the transformations in plain SQL.
- Airflow + Cosmos: generates the DAGs from your dbt project and runs them.
Step 1: ingestion in YAML
Instead of a 200-line Python script that handles pagination and retries, the source goes in a dlt.yaml file. Ingesting a billing API looks like this:
source:
type: rest_api
client:
base_url: "https://api.example.com"
auth:
type: bearer
token: billing-token
resources:
- name: charges_raw
endpoint:
path: /data/charges
method: POST
write_disposition: replaceStep 2: transformation in SQL
Once the data has landed in storage, dbt models transform it. Trino supports federated queries, so you can join across different stores without moving anything first, which keeps the transformation logic readable. Bear in mind that optimization matters in SQL too.
-- int_mau_events.sql
{{ config(materialized='table') }}
SELECT
customer_id,
'visit' AS event_type,
endpoint
FROM {{ source('raw', 'visits') }}
WHERE customer_id IS NOT NULLWhere this approach falls short
Trino’s fault tolerance does not match Spark’s on terabyte-scale shuffles. Lose a worker mid-query and the whole query can fail. dlt’s support for some formats, Delta among them, is also still maturing. So do not migrate everything on principle. Keep PySpark for the 10% of jobs that need heavy custom Python logic SQL cannot reach.
If pipeline work like this is eating your dev hours, I can take it off your hands. I have been wrestling with WordPress and awkward backend integrations since the 4.x days.
Scaling comes down to ownership
The declarative model moved pipeline building off the engineering team and onto the analysts who wanted the data. Delivery went from weeks to a single day, and nobody had to write boilerplate to get there.
