For most developers, scaling means another EC2 instance or a bigger VPS. Step inside the Torre Girona chapel in Barcelona and that definition stops being useful. The 19th-century hall now holds MareNostrum V, a 200M€ machine that runs on rules of distributed computing most WordPress developers have never had to think about.
Fourteen years of WooCommerce race conditions and database bottlenecks did not prepare me for HPC. High-performance computing is messier, far more restricted, and it will kill your job without apology if you ignore the architecture. If you have ever watched expensive GPUs sit idle waiting on data, you already know that here the network is the computer.
The architecture of distributed computing: fat trees and InfiniBand
The idea that trips people up is picturing a supercomputer as one enormous machine. It isn’t. It is a cluster of more than 8,000 independent nodes wired together by a fabric fast enough to break your intuitions about networking. MareNostrum V uses InfiniBand NDR200 arranged in a fat-tree topology.
On an office network, bandwidth chokes at the main switch. In a fat tree the branches get thicker as you move toward the trunk, which keeps bandwidth non-blocking: node 1 and node 8,000 see the same latency. That property is what makes distributed computing workable at this size, because at this scale the overhead of nodes talking to each other costs you more than raw CPU speed ever will.
The partitions: GPP vs. ACC
- General Purpose Partition (GPP): 6,408 nodes on Intel Sapphire Rapids, where the heavily parallel CPU work runs.
- Accelerated Partition (ACC): 1,120 nodes carrying NVIDIA H100 GPUs. At $25,000 a card, the GPUs in that room come to more than $110 million.
The hard truths: airgaps and wall-time
In WordPress work you run pip install or composer update whenever you feel like it. Not here. MareNostrum V is airgapped, and the compute nodes have no outbound internet at all. Everything, and I do mean everything, has to be downloaded, compiled and sitting in your storage directory before you submit a job.
Then there is wall-time. You don’t run code so much as book a slot for it. Ask for two hours, finish in two hours and one second, and SLURM kills the process. There is no warning, and nothing lands in your logs unless you flushed the buffer first. It teaches performance optimization faster than any blog post will.
How to talk to the machine: SLURM
You also don’t just type commands into a terminal here. Start a heavy Python script on the login node and an admin will email you, politely, to ask you to stop breaking the front door. Work goes through SLURM, the Simple Linux Utility for Resource Management.
Here is a SLURM job script of the kind you would actually submit, requesting resources for a distributed computing pipeline:
#!/bin/bash
#SBATCH --job-name=bbioon_simulation
#SBATCH --output=logs/sim_%j.out
#SBATCH --error=logs/sim_%j.err
#SBATCH --time=00:30:00
#SBATCH --nodes=1
#SBATCH --ntasks=6
#SBATCH --account=nct_XXX
# Clean environment
module purge
# Load specific libraries provided by admins
module load OpenFOAM/11-foss-2023a
# Execute using the MPI launcher
srun --mpi=pmix simpleFoam -parallelWith 50 simulations to get through, you don’t submit them one at a time. You chain them with dependencies so the scheduler never floods:
#!/bin/bash
PREV_JOB_ID=""
for CASE_DIR in cases/case_*; do
if [ -z "$PREV_JOB_ID" ]; then
OUT=$(sbatch run_sim.sh)
else
# Only start this job if the previous one finished
OUT=$(sbatch --dependency=afterany:$PREV_JOB_ID run_sim.sh)
fi
PREV_JOB_ID=$(echo $OUT | awk '{print $4}')
doneThe ceiling: Amdahl’s Law
More cores does not mean more speed. Amdahl’s Law puts a ceiling on your speedup, set by the serial part of your code. If 5% of your program is fundamentally sequential, 20x is the best you will ever see, whether you use one node or all 8,000 of MareNostrum V.
It is the same lesson we keep relearning while tuning WooCommerce performance or a large PHP application. Throwing hardware at a serial bottleneck is an expensive way to fail.
If distributed computing or high-end performance scaling is eating your dev hours, I can take it off your hands. I’ve been wrestling with WordPress and complex infrastructure since the 4.x days.
What carries over to ordinary servers
There is no such thing as the single powerful computer any more. Performance work is mostly about managing communication overhead and accepting that the network is where things slow down. That holds whether you are running CFD simulations in a chapel or scaling a global store. Cut the serial parts, stay inside what the hardware will actually give you, and watch your logs, because nothing else is going to tell you when the job died.
