The usual advice for handling large datasets in Python is to write the logic and let it run. That holds up until the script becomes the reason a page hangs. It does not matter much whether you are building a custom WooCommerce integration or a standalone scraper: code that is slow enough is broken. Most developers still go digging through logs when a sampling profiler like py-spy would point straight at the bottleneck.
In 14 years of working on complex systems I have watched plenty of projects sink on silent bottlenecks, meaning code that returns the right answer but takes three minutes to do half a second of work. If you read what I wrote about WordPress core performance, you already know how I feel about scripts that sit on server resources for no reason.
Naive loops and where the overhead comes from
The trap I run into most often is Pandas iterrows(). It reads well, and it is slow, because it builds a fresh Series object for every single row. Take a real case: calculating the Haversine distance across 3.5 million flight records. The naive version looks like this.
# The Slow Approach: Using iterrows()
haversine_dists = []
for i, row in flights_df.iterrows():
haversine_dists.append(haversine(
lat_1=row["LATITUDE_ORIGIN"],
lon_1=row["LONGITUDE_ORIGIN"],
lat_2=row["LATITUDE_DEST"],
lon_2=row["LONGITUDE_DEST"]
))
flights_df["Distance"] = haversine_distsOn a dataset that size the script needs close to 170 seconds, so nearly three minutes of the process doing nothing useful. In production that is long enough to hit a timeout and take the worker down with it.
Why py-spy for Python profiling
Tracers like cProfile add enough overhead of their own to skew the numbers they report. py-spy works differently. It is a sampling profiler: it sits outside your process and reads the call stack 100 times a second. Your program runs at normal speed and you still get an honest picture of where the CPU time goes.
Install it with pip and run the recorder:
pip install py-spy
py-spy record -o profile.svg -r 100 -- python main.pyOpen the resulting SVG in a browser and you get an icicle graph. If the bar for iterrows() covers 70% of the width, that is the thing to refactor.
Vectorize instead of iterating
Once the profile tells you where the time goes, the fix is usually to get out of Python-level loops and into C-level vectorized operations with NumPy. That means rewriting the function so it takes arrays instead of one pair of coordinates at a time.
# The Optimized Approach: Vectorized NumPy
import numpy as np
def haversine_vectorized(lat_1, lon_1, lat_2, lon_2):
lat_1_rad, lon_1_rad = np.radians(lat_1), np.radians(lon_1)
lat_2_rad, lon_2_rad = np.radians(lat_2), np.radians(lon_2)
# Logic remains the same, but operates on arrays
delta_lat = lat_2_rad - lat_1_rad
# ... calculation ...
return result
flights_df["Distance"] = haversine_vectorized(
flights_df["LATITUDE_ORIGIN"],
flights_df["LONGITUDE_ORIGIN"],
flights_df["LATITUDE_DEST"],
flights_df["LONGITUDE_DEST"]
)Execution time drops from 169 seconds to 0.56 seconds, about 300 times faster. The Py-Spy documentation on GitHub covers the internals if you want them.
If performance work like this is eating your dev hours, I take it on. I have been wrestling with WordPress and high-load backend scripts since the 4.x days.
Measure before you refactor
Guessing where the lag lives costs more time than profiling does. Run py-spy, find the widest bar in the graph, rewrite that part, then measure again. The routine is the same whether the slow thing is a WordPress build or a large data migration.
