Task 2 — HPL Cluster Benchmark

Goal: measure the sustained double-precision floating-point performance (GFLOPS) of the cluster using the industry-standard High Performance LINPACK (HPL) benchmark — the same workload used to rank the world's fastest supercomputers in the Top500 list — and identify the bottlenecks.

---

Executive summary

A 9-node Raspberry 8 Pi 3s and 1 Pi 5 (master) cluster was benchmarked using HPL. The single-node configuration (master alone, 4 cores) achieved a verified 29.52 GFLOPS with HPL's residual check PASSED.

After applying power-delivery fixes and updated the configuration in .dat file the full 9-node / 36-rank cluster was re-run and this time the residual check PASSED (4.23131034e-03, well inside the 16.0 threshold). As an additional safety margin on top of the power fixes, this retest also lowered the problem size (N=10240 → 6912) and kept the run at 1 MPI rank per process/core (no oversubscription), P×Q=6×6, so no worker was pushed into a sustained peak-current load while the power fix was being validated. The whole cluster now passes. This confirms the undervoltage diagnosis was correct and that, once power is fixed, the cluster computes correctly end-to-end across all 9 nodes. Verification screenshots (pass1.jpeg, pass2.jpeg, pass3.jpeg) are above.

---

Key definitions

HPL (High Performance LINPACK) A portable benchmark that solves a large dense system of linear equations Ax = b in double precision using LU factorisation with partial pivoting. Because the operation count is known (~2/3 × N³ for matrix size N), HPL converts wall-clock time directly into floating-point performance. Critically, HPL ends every run with a residual check — if ||Ax - b|| / threshold > 16, the result is marked FAILED and the GFLOPS number is invalid.

GFLOPS Giga FLoating-point Operations Per Second. One GFLOPS = 10⁹ arithmetic operations per second on real numbers.

MPI / MPICH Message Passing Interface, the protocol HPL uses to coordinate work across nodes. We used MPICH after Open MPI 5.0.7 hit a launcher bug on this hardware.

BLAS / OpenBLAS Basic Linear Algebra Subprograms — the library that does HPL's heavy arithmetic. We linked against OpenBLAS, configured to run single-threaded per MPI rank.

MPI Rank, P, Q A rank is a single MPI process. P × Q is the 2D grid into which ranks are arranged; each rank owns a tile of the matrix. P × Q must equal the number of ranks (-np). Grids closer to square are more efficient. Our cluster ran 36 ranks (one per core across 9 nodes) in a 6×6 grid.

---

Result summary

Metric	Value
GFLOPS	29.52
Wall-clock time	24.26 s
Problem size N	10 240
Block size NB	128
Process grid P×Q	2×2
MPI ranks	4
Scaled residual	4.6 × 10⁻³
Outcome	PASSED
Parallel efficiency	62 %

Note: this is a single-node result (master only, 4 ranks on 4 cores) — "parallel efficiency" here measures how well the 4 cores were utilised, not efficiency across cluster nodes. It is calculated as measured GFLOPS ÷ theoretical peak GFLOPS:

Peak = cores × clock × FLOPs/cycle
     = 4 × 1.5 GHz × 8 (NEON double-precision FMA)
     = 48 GFLOPS

Efficiency = Measured / Peak = 29.52 / 48 = 61.5% ≈ 62%

See Peak GFLOPS comparison below for why 8 FLOPs/cycle is the right basis.

Raw HPL summary line:

WR11C2C4   10240   128   2   2   24.26   2.9518e+01

Residual check

||Ax-b||_oo / (eps * (||A||_oo * ||x||_oo + ||b||_oo) * N) = 4.59905368e-03 ...... PASSED

---

Cluster specification

Role	Hostname	IP	Cores	RAM
Master	master	192.168.137.10	4	7.9 GiB
Worker 1–8	worker1–worker8	.101–.108	4 each	~900 MiB each

Total: 36 cores across 9 nodes. Unmanaged gigabit Ethernet switch; internet via macOS host with Internet Sharing. Memory is heterogeneous (master 8 GB, workers ~1 GB), so HPL's even partitioning means the workers cap the achievable matrix size.

---

Peak GFLOPS comparison

Estimate	GFLOPS	Basis
Conservative peak	24	4 cores × 1.5 GHz × 4 FLOPs/cycle
Measured	29.52	Verified (residual PASSED)
Optimistic peak	48	8 FLOPs/cycle via FMA pairing

Our single-node measurement is 23% above the conservative theoretical peak of 24 GFLOPS. This is because the Cortex-A72 supports double-precision fused multiply-add (FMA) via NEON: a single instruction performs one multiply and one add — counted as 2 FLOPs but completing in 1 cycle. With FMA pairing the effective rate can reach 8 FLOPs/cycle, giving an optimistic peak of 48 GFLOPS, against which our 29.52 GFLOPS is a healthy 62% efficiency — very respectable for an SBC.

---

Why multi-node runs failed

All multi-node HPL runs at the original problem sizes returned FAILED residuals ≥10⁸, until the problem size was lowered and the worker power supplies were fixed:

Test	Ranks	Grid	Residual	Outcome
Master only	4	2×2	4.6×10⁻³	PASSED
Master + worker1	8	2×4	1.67×10⁹	FAILED
Worker1 only	4	2×2	5.8×10⁹	FAILED
All 9 nodes, N=10240	36	6×6	1.32×10⁹	FAILED
All 9 nodes, N=16896	36	6×6	6.35×10⁸	FAILED
All 9 nodes, N=6912 (lower N, 1 rank/process, after all fixes and config update)	36	6×6	4.23×10⁻³	PASSED

Reading the last row: this is the same all-9-node, 36-rank configuration that previously FAILED. It only passed once (a) the problem size was lowered from N=10240/16896 down to N=6912

What P and Q mean

P and Q define the MPI process grid — how ranks are logically arranged to partition the matrix. P = process rows, Q = process columns, and P × Q must equal -np. Each rank owns one rectangular tile of the matrix; ranks in the same row/column exchange row/column data during factorisation, so grids closer to square minimise communication. For the Master + worker1 run (8 ranks) we used P=2, Q=4:

        Q=0    Q=1    Q=2    Q=3
      ┌─────┬─────┬─────┬─────┐
  P=0 │  0  │  1  │  2  │  3  │
      ├─────┼─────┼─────┼─────┤
  P=1 │  4  │  5  │  6  │  7  │
      └─────┴─────┴─────┴─────┘

Each numbered cell is one MPI rank. For 8 ranks, 2×4 (or 4×2) is the best available shape — 1×8/8×1 would be far less efficient.

HPL.dat and launch command per test

All runs shared the same HPL.dat template (NB=128, threshold 16.0, PFACT/RFACT=Crout, NBMIN=4, NDIV=2, BCAST=1ringM, DEPTH=1, SWAP=Mix(64), EQUIL=yes) — only N, P, Q and the hostfile changed. HPL.dat lived at /opt/hpl/bin/rpi/HPL.dat on the master and was rsync'd out to whichever worker joined a run.

Master only — N=10240, P×Q=2×2:

cd /opt/hpl/bin/rpi

sed -i 's/^[0-9]\+\s\+Ns$/10240        Ns/' HPL.dat
sed -i 's/^[0-9]\+\s\+Ps$/2            Ps/' HPL.dat
sed -i 's/^[0-9]\+\s\+Qs$/2            Qs/' HPL.dat

echo "master:4" > hostfile.solo

mpirun -iface eth0 -f hostfile.solo -wdir /opt/hpl/bin/rpi -np 4 \
  -genv OPENBLAS_NUM_THREADS 1 \
  -genv OMP_NUM_THREADS 1 \
  ./xhpl | tee ~/hpl_solo.txt

Master + worker1 — N=10240, P×Q=2×4 (grid widened from 2×2 by bumping Qs):

cd /opt/hpl/bin/rpi

sed -i 's/^2            Qs/4            Qs/' HPL.dat
rsync -a HPL.dat worker1:/home/worker1/hpl-2.3/bin/rpi/HPL.dat

cat > hostfile.test <<EOF
master:4
worker1:4
EOF

mpirun -iface eth0 -f hostfile.test -wdir /opt/hpl/bin/rpi -np 8 \
  -genv OPENBLAS_NUM_THREADS 1 \
  -genv OMP_NUM_THREADS 1 \
  ./xhpl | tee ~/hpl_w1.txt

Worker1 only — N=6912, P×Q=2×2 (matrix size dropped to keep the run short; this is the same 67 s run referenced in Root cause below, where vcgencmd get_throttled flipped from 0x0 to 0x50000 immediately after):

cd /opt/hpl/bin/rpi

sed -i 's/^10240\s*Ns$/6912         Ns/' HPL.dat
sed -i 's/^4            Qs/2            Qs/' HPL.dat
rsync -a HPL.dat worker1:/home/worker1/hpl-2.3/bin/rpi/HPL.dat

echo "worker1:4" > hostfile.w1only

mpirun -iface eth0 -f hostfile.w1only -wdir /opt/hpl/bin/rpi -np 4 \
  -genv OPENBLAS_NUM_THREADS 1 \
  -genv OMP_NUM_THREADS 1 \
  ./xhpl | tee ~/hpl_w1_alone.txt

All 9 nodes, N=10240 — P×Q=6×6 :

cd /opt/hpl/bin/rpi

sed -i 's/^6912\s*Ns$/10240        Ns/' HPL.dat
sed -i 's/^2            Ps/6            Ps/' HPL.dat
sed -i 's/^2            Qs/6            Qs/' HPL.dat

# Push to all workers
for i in 1 2 3 4 5 6 7 8; do
  rsync -a HPL.dat worker$i:/home/worker$i/hpl-2.3/bin/rpi/HPL.dat
done

# hostfile (9 nodes)
cat > hostfile <<EOF
master:4
worker1:4
worker2:4
worker3:4
worker4:4
worker5:4
worker6:4
worker7:4
worker8:4
EOF

mpirun -iface eth0 -f hostfile -wdir /opt/hpl/bin/rpi -np 36 \
  -genv OPENBLAS_NUM_THREADS 1 \
  -genv OMP_NUM_THREADS 1 \
  ./xhpl | tee ~/hpl_result.txt

All 9 nodes, N=16896 — P×Q=6×6, Ns raised to 16896:

cd /opt/hpl/bin/rpi

sed -i 's/^10240\s*Ns$/16896        Ns/' HPL.dat

for i in 1 2 3 4 5 6 7 8; do
  rsync -a HPL.dat worker$i:/home/worker$i/hpl-2.3/bin/rpi/HPL.dat
done

# Same hostfile as Test 4
mpirun -iface eth0 -f hostfile -wdir /opt/hpl/bin/rpi -np 36 \
  -genv OPENBLAS_NUM_THREADS 1 \
  -genv OMP_NUM_THREADS 1 \
  ./xhpl | tee ~/hpl_result_16896.txt

Critical observation: master alone PASSED, worker1 alone FAILED. Same software, same network (none, in the worker1-alone case), same RAM type, same OpenBLAS, same binary. The difference lies in the hardware itself.

Hypotheses ruled out

Hypothesis	Test performed	Verdict
Network corruption (UCX/MPI bug)	Worker1 alone (no network) also FAILED	Ruled out
OpenBLAS install mismatch	md5sum libopenblas.so.0: master = worker1	Ruled out
Binary corruption in transit	md5sum xhpl: master = worker1	Ruled out
SD card silent corruption	Repeated md5sum of same file matched	Ruled out
Worker RAM corruption	dd + md5 of 400 MB random data matched	Ruled out
OpenBLAS thread oversubscription	OPENBLAS_NUM_THREADS=1 did not help	Ruled out

Root cause: undervoltage throttling

The Raspberry Pi firmware exposes a real-time throttle/undervoltage status via vcgencmd get_throttled. The return value is a bitmask where bits 16/17/18 indicate "this condition has occurred since boot":

Bit	Hex	Meaning
16	0x10000	Under-voltage has occurred since boot
17	0x20000	ARM frequency cap has occurred since boot
18	0x40000	Throttling has occurred since boot

Every worker reported CPU throttling after the cluster-scale runs:

Node	Flag	Undervoltage	ARM freq cap	Throttling
worker1	0x50000	✓	—	✓
worker2	0x50000	✓	—	✓
worker3	0x70000	✓	✓	✓
worker4	0x50000	✓	—	✓
worker5	0x70000	✓	✓	✓
worker6	0x70000	✓	✓	✓
worker7	0x70000	✓	✓	✓
worker8	0x70000	✓	✓	✓

Controlled experiment: all workers were rebooted to clear the sticky throttle flags. After reboot all 8 workers reported throttled=0x0. Worker1 then ran HPL alone for 67 seconds (N = 6912). Immediately after the run:

$ ssh worker1 "vcgencmd get_throttled"
throttled=0x50000
(undervoltage and throttling occurred during HPL load)

This is a direct cause-effect observation: the HPL workload itself draws enough current to push worker1's supply below 4.63 V, triggering the firmware's undervoltage protection and CPU throttling. The accompanying computational error (||x|| = 8.65 in HPL's output, vs the expected ~1 for a correctly solved system) is the silent data corruption characteristic of unstable CPU operation under under-voltage.

Mitigations:

1. Use the official Raspberry Pi 3 USB-C power supply (15.3 W) for each worker, with its hardwired thicker cable.
2. Avoid powering Pis through USB hubs — even powered hubs typically deliver only 1–1.5 A per downstream port, insufficient for Pi 3 peak.
3. Use short, thick USB-C cables. Long thin cables can drop 0.3 V at 3 A — enough to trigger the threshold.
4. Verify by checking vcgencmd get_throttled = 0x0 after running HPL for > 1 minute.
5. As a temporary software mitigation, capping each Pi's CPU frequency reduces current draw at the cost of ~15% performance:

   echo 1200000 | sudo tee /sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq
   

Verification after power fixes

After applying the power-delivery fixes above, the full 9-node / 36-rank run (N=6912, P×Q=6×6) was re-executed and passed the residual check end-to-end:

---

Analysis: theoretical full-cluster performance

With adequately powered workers, the cluster's projected ceiling is:

Per-node measured:                29.52 GFLOPS
9 nodes × 29.52 GFLOPS =           265 GFLOPS  (naive ceiling)
Realistic cluster efficiency over gigabit Ethernet:  ~30 - 50 %
Expected verified cluster result:  ~80 - 130 GFLOPS

---

Why the residual check matters

HPL reported up to 11.9 GFLOPS in the failed multi-node runs above. Without the residual check, these figures would have looked like plausible benchmark numbers. The check exists precisely to prevent this: any system that produces fast-but-wrong answers is not actually doing the work the benchmark claims to measure. In our case, the residual check detected silent data corruption caused by CPU undervoltage, exposing a hardware infrastructure problem that would otherwise have gone unnoticed and propagated into any subsequent computational use of the cluster.

This is a real engineering finding, not a coursework artefact. In production HPC sites, the residual check (and analogous correctness checks in other benchmarks) is the primary line of defence against silent data corruption, a serious and well-documented issue in large-scale computing.

---

Installing HPL

sudo apt install -y libopenmpi-dev openmpi-bin libopenblas-dev liblapack-dev
wget https://www.netlib.org/benchmark/hpl/hpl-2.3.tar.gz
tar xzf hpl-2.3.tar.gz
cd hpl-2.3
./configure --prefix=$(pwd)/bin/rpi \
  LDFLAGS="-L/usr/lib/aarch64-linux-gnu/openblas-pthread" \
  LIBS="-lopenblas"
make -j4 && make install

---

Launch command

mpirun -iface eth0 -f hostfile -wdir /opt/hpl/bin/rpi -np 36 ./xhpl | tee ~/hpl_result.txt

Config based run

mpirun -iface eth0 -f hostfile -wdir /opt/hpl/bin/rpi -np 36 \
  -genv OPENBLAS_NUM_THREADS 1 \
  -genv OMP_NUM_THREADS 1 \
  -genv GOTO_NUM_THREADS 1 \
  ./xhpl | tee ~/hpl_result.txt

---

Conclusion

HPL 2.3 was successfully built, deployed and executed across a 9-node Raspberry Pi cluster. The verified single-node throughput is 29.52 GFLOPS (PASSED), achieving 62% efficiency against the Cortex-A72's theoretical peak. Cluster-scale runs failed HPL's residual check; systematic diagnosis identified worker undervoltage under load as the root cause, with direct evidence from the firmware's throttle status. The cluster is software-ready: with adequate power delivery to the workers, a verified multi-node result on the order of 100 GFLOPS is expected.

References:

• HPL home page
• vcgencmd documentation
• Full HPL report (PDF)

---