Every week I see a developer trying to shave milliseconds off a Python loop with increasingly unreadable list comprehensions or some obscure NumPy trick. The advice is always “just vectorize it,” but plenty of logic does not fit the matrix mold. Once you hit that wall, the move is to call Rust from Python instead of fighting the interpreter’s overhead.
After 14 years wrestling with high-traffic backends, my view is that Python is fantastic for orchestration and a terrible engine for character-by-character text processing or complex branching logic. You do not rewrite the whole site for that. You swap the engine. Rust gives you memory safety with no garbage collector and performance that rivals C++, and your Python ecosystem stays intact.
Why your loops are slow
Python’s Global Interpreter Lock (GIL) and dynamic typing are the primary culprits. Libraries like Polars are great, but every so often you hit a hot loop that refuses to vectorize: cleaning messy JSON strings, or parsing a custom binary format. My earlier take on Pandas performance optimization makes the same point, which is that loops are usually where data pipelines go to die.
The Maturin + PyO3 stack
Two tools bridge the worlds: PyO3 for the Rust bindings and Maturin for the build. Maturin compiles your Rust crate into a shared library (.so or .pyd) that Python imports like any other module.
# Quick setup using uv
uv init pyrust && cd pyrust
uv venv && source .venv/bin/activate
pip install maturin pyo3A real example: text normalization
Take a common case, processing 500,000 strings. Walking them character by character in pure Python is painfully slow, because every single operation carries Python object overhead. The bridge itself gets defined in Rust with the #[pyfunction] macro.
// src/lib.rs
use pyo3::prelude::*;
#[pyfunction]
fn bbioon_batch_process(texts: Vec<String>) -> PyResult<Vec<Vec<String>>> {
let mut results = Vec::new();
for text in texts {
let normalized: Vec<String> = text.to_lowercase()
.split_whitespace()
.map(|s| s.replace(|c: char| !c.is_alphanumeric(), ""))
.collect();
results.push(normalized);
}
Ok(results)
}
#[pymodule]
fn rust_text(_py: Python, m: &Bound<'_, PyModule>) -> PyResult<()> {
m.add_function(wrap_pyfunction!(bbioon_batch_process, m)?)?;
Ok(())
}Compiling for production is one command: maturin develop --release. It builds the optimized binary and installs it straight into your virtualenv. After that, when you call Rust from Python you are running machine code rather than bytecode.
Scaling with Rayon
Using more than one CPU core is the real gotcha in Python. Multiprocessing is clunky and pays for expensive data serialization. Rust has Rayon, which turns a sequential iterator into a parallel one in a single line. For as long as the Rust code runs, the GIL is out of the picture.
// Adding parallelism is trivial
use rayon::prelude::*;
#[pyfunction]
fn bbioon_process_parallel(texts: Vec<String>) -> PyResult<Vec<Vec<String>>> {
Ok(texts.par_iter().map(|t| {
// Logic remains the same, but runs on all cores
t.to_lowercase().split_whitespace().map(|s| s.to_string()).collect()
}).collect())
}The benchmark numbers move a long way. Bare Python might handle 96k texts/sec; a single-threaded Rust bridge hits 160k. Turn Rayon on and you are at 220k+ texts/sec. That gap is the difference between a server timing out and one that keeps up.
If performance work is eating your dev hours, hand it over. I have been wrestling with WordPress and high-load backend architecture since the 4.x days.
The takeaway
Do not abandon Python for being “slow.” It is the best glue language there is. Keep it for your API endpoints, your database migrations, and your high-level logic. The heavy lifting is what gets a bridge. Once you can call Rust from Python, hot loops stop being something you dread. The codebase stays maintainable too, because only the performance-critical 1% is written in Rust and the other 99% stays in the flexible Python world we all love.
