We need to talk about Declarative Data Pipelines. For some reason, the standard advice for years has been to throw more PySpark and custom Python boilerplate at every data problem. Consequently, we’ve created a massive bottleneck where analysts are stuck waiting weeks for engineers to ship a single metric.
I’ve seen this play out in dozens of projects. You start with a “simple” data request, and three weeks later, you’re still debugging Spark resource allocations and Airflow DAG dependencies. It’s a mess. However, there is a better way to handle modern data movement without needing a Ph.D. in distributed systems.
The PySpark Bottleneck: A Senior Dev’s War Story
In one of my previous architectural reviews, a client was spending $40k a month on a cluster just to calculate Monthly Active Users (MAU). Every time they wanted to add a new event source, it required a developer to write a custom PySpark script, go through code review, and hope the “Out of Memory” errors didn’t hit at 3 AM. It wasn’t sustainable. If you’re dealing with similar issues, you might want to look at scaling real-time data pipelines differently.
The core problem isn’t the code itself; it’s the ownership. When the logic is trapped in Python, the analyst—who understands the business requirements best—is effectively locked out of the production environment. We need a way to build Declarative Data Pipelines where the configuration is the documentation.
The New Stack: dlt, dbt, and Trino
To fix this, we replaced the procedural nightmare with a declarative approach using three specific tools. This setup allows anyone who knows SQL and basic YAML to ship a production-grade pipeline in 24 hours.
- dlt (data load tool): Handles the extraction from APIs and databases using simple YAML configs.
- dbt (data build tool): Runs on top of Trino to handle transformations using pure SQL.
- Airflow + Cosmos: Orchestrates the whole thing by auto-generating DAGs from your dbt project.
Step 1: Ingestion via YAML
Instead of writing a 200-line Python script to handle pagination and retries, you define the source in a dlt.yaml file. Here is what a real-world billing API ingestion looks like:
source:
type: rest_api
client:
base_url: "https://api.example.com"
auth:
type: bearer
token: billing-token
resources:
- name: charges_raw
endpoint:
path: /data/charges
method: POST
write_disposition: replaceStep 2: Transformation via SQL
Once the data is in your storage, you use dbt models to transform it. Since Trino supports federated queries, you can join data across different stores with zero movement. Therefore, the transformation logic remains clean and readable. Just remember that optimization matters even in SQL.
-- int_mau_events.sql
{{ config(materialized='table') }}
SELECT
customer_id,
'visit' AS event_type,
endpoint
FROM {{ source('raw', 'visits') }}
WHERE customer_id IS NOT NULLWhy This Isn’t a Silver Bullet
Specifically, there are limitations. Trino’s fault tolerance isn’t as robust as Spark’s for massive terabyte-scale shuffles. If a worker goes down, the whole query might fail. Furthermore, dlt’s support for certain formats like Delta is still maturing. You shouldn’t blindly migrate everything; keep PySpark for the 10% of cases where you need heavy custom Python logic that SQL just can’t touch.
Look, if this Declarative Data Pipelines stuff is eating up your dev hours, let me handle it. I’ve been wrestling with WordPress and complex backend integrations since the 4.x days.
The Only Way to Scale is Ownership
By moving to a declarative model, we shifted the burden of pipeline creation from the engineering team to the analysts. The result? Development time dropped from weeks to a single day. Stop writing boilerplate and start shipping value.
