Most developers I talk to use AI for boilerplate and mediocre unit tests. That is fine, but it is not where the interesting work is. When you are trying to shave milliseconds off something like matrix multiplication in Rust, the big cloud models usually miss the hardware details that matter. Local LLMs running on your own machine, chewing on low-level code, turn out to be better suited to that particular job.
I have seen plenty of “AI-optimized” code that looks great and falls over under load, so I paid attention when Stefano Bosisio wrote up his experiment: a MacBook Pro M3, the open-source Mixtral 8x7B, and a hunt for a better matrix multiplication (matmul) algorithm. He did not do it with one clever prompt. He wired up several agents that kept rewriting the code and rechecking it until it beat the standard implementation.
Building a multi-agent roundtable with local LLMs
The setup runs on Microsoft Autogen. Instead of asking one model for the answer, you seat several around a table with jobs: a Proposer for the theory, a Coder to implement it, a Tester to benchmark it. Running local models in that loop is really just automating the refine-and-iterate habit any experienced dev already has, except the loop keeps going without you.
It also sidesteps the token limit problem. State and context go into a vector database, Chroma in this case, so the agents can look up what worked on earlier runs instead of walking into the same dead end again. Anyone who has spent an evening debugging knows that failure mode.
Where the generated code went wrong
The runs were not clean. Stefano hit what he calls “diagonal fallacies”, where the generated code computed only the diagonal blocks and quietly skipped the rest of the matrix. That category of gap shows up constantly; I wrote about it in Technical Debt in AI Development. The math reads as sound and the answer is still wrong.
Early iterations tripped over buffer overwrites and cache misses. By the third or fourth pass, though, the model had reached for NEON SIMD intrinsics and Rayon parallelism on its own. The baseline was 760ms. The version it landed on ran in 359ms, a 50% speedup found by a model sitting on a consumer laptop.
From naive Rust to NEON intrinsics
The size of the jump is easiest to see in the code. First the naive Rust matmul, then what the agent settled on with SIMD (Single Instruction, Multiple Data).
// The Naive Approach (Slow, lacks vectorization)
fn naive_matmul(a: &[f32], b: &[f32], c: &mut [f32], size: usize) {
for i in 0..size {
for j in 0..size {
for k in 0..size {
c[i * size + j] += a[i * size + k] * b[k * size + j];
}
}
}
}
// The Optimized Approach (Rayon + NEON intrinsics)
// Discovered by Local LLMs during iteration
use std::arch::aarch64::*;
use rayon::prelude::*;
fn optimized_matmul(a: &[f32], b: &[f32], c: &mut [f32], size: usize) {
c.par_chunks_mut(size).enumerate().for_each(|(i, row)| {
for k in 0..size {
let va = unsafe { vdupq_n_f32(a[i * size + k]) };
for j in (0..size).step_by(4) {
unsafe {
let vb = vld1q_f32(&b[k * size + j]);
let mut vc = vld1q_f32(&row[j]);
vc = vfmaq_f32(vc, va, vb);
vst1q_f32(&mut row[j], vc);
}
}
}
});
}Note vfmaq_f32 and vld1q_f32. Those are ARM-specific instructions, which is why this particular win only applies to hardware like the M3. I do not have them memorized and neither does anyone I work with, but a well-tuned local model pulled them out of its training data once the Tester agent reported a bad baseline. A specialist would know to try those intrinsics. A generalist would have to go looking, and the model already had them.
If this kind of AI work is eating your dev hours, I can take it off your plate. I have been wrestling with WordPress and custom integrations since the 4.x days.
Refactor and ship it
The small models on our laptops can now work out optimizations that used to need a cluster. You do not have to reach for a BLAS-level library on every custom need if an agent loop can find the specific hack your bottleneck wants. Worth pointing one at your slowest function before you go shopping for a dependency.
