I keep running into beefy clusters sitting idle because the data layout underneath them is a mess. Last week I looked at a 420-core cluster that spent close to 10 hours grinding through 18 partitions. If you are a developer or a business owner paying those Databricks bills, that number should bother you. It is not a compute problem, it is a data engineering failure.
The standard advice for scaling ML inference usually starts and ends with “add more workers.” That falls apart the moment your data is skewed, meaning one product or category carries 100x more rows than another. Spark’s default partitioning hands one executor 50 million rows to sweat over while the other 419 cores do nothing, and no amount of RAM fixes that.
Why AQE alone will not fix skew
Most devs lean on Spark’s Adaptive Query Execution to handle the heavy lifting. AQE is genuinely good at standard SQL queries, but it was never built to optimize model inference runtimes. In a partitioned table with no salt you end up with fat files. Product D in our case study accounted for nearly 80% of the 550M rows. With nothing breaking that up, you get long-running skewed tasks that block the whole pipeline.
For the wider context on how data structures ripple through the rest of the stack, I wrote up my thoughts on modern data stack consolidation. How data reaches your applications matters as much as what happens once it gets there.
Option 1: adding salt to your datasets
Salting is an old hack for forcing Spark to spread data more evenly. You append a random salt key to your high-cardinality columns, which breaks one massive partition into hundreds of smaller ones and lets you actually saturate the cluster.
# bbioon_dynamic_salting_example
from pyspark.sql import functions as F
# Calculate percentages to determine bucket counts
total_count = df.count()
product_percents = df.groupBy("ProductLine").count().withColumn(
"percent", F.col("count") / F.lit(total_count)
)
# Set min/max buckets for parallelism
min_buckets = 10
max_buckets = 1160
# Generate the salt based on product volume
df = df.withColumn("salt", (F.rand(seed=42) * F.lit(max_buckets)).cast("int"))
# Repartition to unlock parallelism
df_final = df.repartition(1200, "ProductLine", "salt").drop("salt")With the salty and partitioned setup, Product D went from never-ending to roughly 3 hours. Capping partitions at 1 million rows with maxRecordsPerFile kept any single executor from getting buried.
Option 2: liquid clustering
Databricks added Liquid Clustering fairly recently, and it changes the math on scaling ML inference. Traditional partitioning is rigid: partition by date and product and you are married to that folder structure. Liquid clustering organizes data around clustering keys with no folder hierarchy, so it behaves like a more flexible, automated Z-Ordering.
Per the official Databricks documentation, liquid clustering adapts as query patterns change. Pair it with salting and you get parallelism plus data skipping.
Combining salting with liquid clustering
The salty and liquid setup gave us the most stable task distribution in testing. The runtime window per task was much tighter, so fewer outliers. Running hundreds of models in an ensemble, you cannot afford a long tail of tasks finishing at 2x the time of everything else. Liquid clustering also preserves data locality, which matters when you filter for specific products or dates before inference.
It is the same point I made in a post about why your environment dictates ML success. Your data layout is your environment. Get the layout wrong and the pipeline fails regardless of how good the models are.
What to check before you add more workers
- Spark’s default partitioning is a performance killer on skewed datasets, so do not trust it.
- If one product holds 80% of your data, salt that key and push the rows across more cores.
- Liquid clustering holds up better than traditional partitioning and absorbs data growth without a full refactor.
- Watch the Spark UI. Ten tasks at 5 hours sitting next to 400 tasks at 2 minutes is skew.
If this kind of tuning is eating your dev hours, I can take it off your plate. I have been wrestling with WordPress and high-performance backend systems since the 4.x days.
Ship it or fix it?
Scaling ML inference on Databricks comes down to cluster utilization. If you pay for 420 cores, make sure 420 cores are working. Salting gives you the control, liquid clustering gives you room to grow, and skewed data stops quietly burning your budget.
