Skip to content

Task 1 — Infrastructure Setup

Lead: Muhammad Ghayoor Ali - Cluster Setup · Object Detection · PXE Investigation

1. Network Architecture & Node Configuration

All nodes are connected via a dedicated network switch. The laptop acts as the internet gateway using Windows Internet Connection Sharing (ICS), assigning itself the IP 192.168.137.1 on Ethernet and sharing its WiFi internet to all Pi nodes over ethernet. A wired network was chosen to ensure the cluster remains stable and portable regardless of WiFi availability at different locations.

1.1 Node Overview

Node Hardware OS Hostname Static IP
Master Raspberry Pi 5 + AI HAT+ (Hailo-8L) RPi OS Lite 64-bit master 192.168.137.10
Sensor Raspberry Pi 4 + AI Camera (imx500) RPi OS Lite 64-bit sensor 192.168.137.20
Worker 1–8 Raspberry Pi 3 (×8) RPi OS Lite 32-bit worker1–worker8 192.168.137.101–108
Gateway Windows Laptop (ICS) Windows — 192.168.137.1

1.2 OS Installation

Each SD card was flashed using Raspberry Pi Imager with the following settings configured before writing:

  • SSH enabled — required for headless cluster management
  • WiFi left blank — ethernet only, ensures consistent connectivity
  • Hostname set per node (master, sensor, worker1–worker8)
  • Username and password set per node
  • Timezone: Europe/Berlin, Keyboard: de

Note: Pi 3 workers use 32-bit Raspberry Pi OS (armhf) for maximum compatibility with the Pi 3B hardware. Master and sensor use 64-bit OS to support their respective AI accelerators.

2. Static IP Assignment

After first boot each Pi receives a temporary DHCP IP from Windows ICS. A permanent static IP is then assigned via NetworkManager. Static IPs are essential for MPI hostfiles, k3s node registration, Prometheus metric scraping, and all inter-node communication — any IP change after reboot would break these services.

Command run on each node (IP address changed per node):

sudo nmcli con mod "Wired connection 1" \
  ipv4.addresses 192.168.137.10/24 \
  ipv4.gateway 192.168.137.1 \
  ipv4.dns "8.8.8.8" \
  ipv4.method manual
sudo nmcli con up "Wired connection 1"

The SSH session disconnects after applying the static IP. Reconnection is done immediately using the new static address. Verified with:

ip addr show eth0
ping -c 4 8.8.8.8   # confirms internet via laptop gateway

3. SSH Key Distribution

Passwordless SSH access was configured from the master to all 8 worker nodes. This is a prerequisite for MPI in Task 3 (mpirun distributes jobs over SSH), for Ansible-based cluster management, and for k3s node joining in Task 7.

# On master — generate key pair
ssh-keygen -t ed25519

# Copy public key to each worker
ssh-copy-id worker1@192.168.137.101
ssh-copy-id worker2@192.168.137.102
ssh-copy-id worker3@192.168.137.103
ssh-copy-id worker4@192.168.137.104
ssh-copy-id worker5@192.168.137.105
ssh-copy-id worker6@192.168.137.106
ssh-copy-id worker7@192.168.137.107
ssh-copy-id worker8@192.168.137.108

Verified by SSHing from master to each worker without a password prompt:

ssh worker1@192.168.137.101   # must connect without password

4. Hostname Configuration (/etc/hosts)

The /etc/hosts file was updated on all 10 nodes so that every node can reach every other node by hostname instead of IP. This is required by MPI (hostfile uses hostnames), Prometheus (scrapes by hostname), and k3s (nodes register by hostname).

The following lines were appended to /etc/hosts on every node:

192.168.137.10  master
192.168.137.20  sensor
192.168.137.101 worker1
192.168.137.102 worker2
192.168.137.103 worker3
192.168.137.104 worker4
192.168.137.105 worker5
192.168.137.106 worker6
192.168.137.107 worker7
192.168.137.108 worker8

A bash loop was run from master to push the update to all workers simultaneously:

for i in 1 2 3 4 5 6 7 8; do
  ssh worker$i@192.168.137.10$i "echo '192.168.137.10 master' | sudo tee -a /etc/hosts"
done

5. Object Detection Software Deployment

5.1 Sensor Node — Raspberry Pi 4 + AI Camera (imx500)

The Raspberry Pi AI Camera (imx500) contains a built-in NPU (Neural Processing Unit) chip that runs AI inference directly on the camera hardware. This means object detection runs at 30 FPS on the camera itself with less than 5% CPU usage on the Pi 4 — far superior to running YOLO on the CPU which causes an Illegal Instruction error on Pi 4 ARM hardware.

The imx500 approach is the officially documented method for this exact hardware as referenced in the professor's task link [2] (raspberrypi.com/documentation/accessories/ai-camera.html).

Installation

sudo apt update && sudo apt full-upgrade -y
sudo apt install -y imx500-all
sudo apt install -y python3-picamera2 python3-opencv python3-munkres
git clone https://github.com/raspberrypi/picamera2
sudo reboot

Camera Detection Verified

rpicam-hello --list-cameras
# Available cameras
# 0 : imx500 [4056x3040 10-bit RGGB] (/base/soc/i2c0mux/i2c@1/imx500@1a)
#     Modes: 2028x1520 [30.02 fps], 4056x3040 [10.00 fps]

Object Detection Running and Confirmed

rpicam-hello -t 30000 \
  --post-process-file /usr/share/rpi-camera-assets/imx500_mobilenet_ssd.json \
  --lores-width 400 --lores-height 300 --nopreview --verbose 2

Detection output confirmed (camera pointed at a person):

Network Firmware Upload: 100% (3872/3872 KB)
Number of objects detected: 1
[0] : person[0] (0.85) @ 22,226  811x1293
Number of objects detected: 1
[0] : person[0] (0.82) @ 18,210  853x1308

Why imx500 NPU Instead of YOLO on CPU

Approach Speed on Pi 4 CPU Usage Decision
YOLO on Pi 4 CPU (PyTorch) ~1 FPS 100% ❌ Illegal Instruction error on ARM
NCNN format on Pi 4 CPU ~4 FPS 90% ❌ Not optimal for this hardware
imx500 NPU — MobileNet SSD 30 FPS <5% ✅ Official approach — used

5.2 Master Node — Raspberry Pi 5 + AI HAT+ (Hailo-8L)

The AI HAT+ contains a Hailo-8L NPU with 13 TOPS (Tera Operations Per Second) of AI processing power. On the master node it receives detection events from the sensor and runs additional inference for verification — reducing false positives by confirming sensor detections with a higher-accuracy model.

Installation

sudo apt update && sudo apt full-upgrade -y
sudo apt install -y hailo-all
sudo reboot

AI HAT+ Verified

hailortcli fw-control identify
# Executing on device: 0001:01:00.0
# Control Protocol Version: 2
# Firmware Version: 4.23.0 (release,app,extended context switch buffer)
# Board Name: Hailo-8
# Device Architecture: HAILO8L

Note: The AI HAT+ does not have a camera attached directly. It receives detection events from the sensor node over the network and runs additional AI processing on flagged images. This is the correct architecture — sensor captures and detects, master processes and stores.

6. PXE Boot Investigation

The task required investigating how Pi 3 OS images can be consolidated on the master (Pi 5) so that worker nodes boot via PXE over the LAN. A complete PXE infrastructure was built and tested. The investigation revealed both what works and the specific technical reason for the failure.

6.1 How PXE Boot Works

In a PXE setup the Pi 3 powers on with no full OS on its SD card. It broadcasts a request on the network asking for boot instructions. The master responds with three services working together:

  • DHCP — assigns an IP address and tells the Pi 3 where the boot server is
  • TFTP — transfers the kernel and boot files to the Pi 3 over the network
  • NFS — shares the complete root filesystem (full OS) so the Pi 3 can run it

The Pi 3 then boots entirely from the master's disk — its SD card only needs bootcode.bin (52KB) to trigger the process.

6.2 PXE Server Setup on Master

Step 1 — Install Required Services

Three services are needed: dnsmasq handles both DHCP and TFTP, nfs-kernel-server shares the OS filesystem over the network, and kpartx allows mounting partitions inside a disk image file so we can extract the OS without a physical SD card.

sudo apt install -y dnsmasq nfs-kernel-server kpartx wget

Step 2 — Download and Extract Pi 3 OS Image on Master

Instead of flashing an SD card and copying from it, the Pi 3 OS image was downloaded directly on the master. This is the core of the consolidation — the OS image lives on master, not on the worker nodes. The 32-bit armhf version is required because Pi 3B is not fully compatible with 64-bit OS for network booting.

wget https://downloads.raspberrypi.org/raspios_lite_armhf/images/\
raspios_lite_armhf-2023-12-11/2023-12-11-raspios-bookworm-armhf-lite.img.xz
xz -d 2023-12-11-raspios-bookworm-armhf-lite.img.xz

The image file contains two partitions — a boot partition and a root filesystem partition. kpartx maps these as virtual block devices so they can be mounted and copied:

sudo kpartx -av 2023-12-11-raspios-bookworm-armhf-lite.img
# add map loop1p1 (boot partition — kernel, config files)
# add map loop1p2 (root partition — full OS filesystem)

Boot partition copied to /tftpboot/ — this is what the Pi 3 downloads first via TFTP. It contains the kernel, firmware files, and boot configuration:

sudo mount /dev/mapper/loop1p1 /mnt
sudo cp -r /mnt/* /tftpboot/
sudo umount /mnt

Root filesystem copied to /nfs/worker/ — the complete OS that the Pi 3 mounts and runs from over the network. It contains everything: /bin, /etc, /home, /usr, /var:

sudo mount /dev/mapper/loop1p2 /mnt
sudo cp -ra /mnt/* /nfs/worker/
sudo umount /mnt

Step 3 — Configure NFS Server to Share OS Filesystem

NFS (Network File System) allows the Pi 3 to mount the /nfs/worker/ directory from master as its own root filesystem over the network. Without NFS, the Pi 3 would have nowhere to run the OS from after downloading the kernel.

/etc/exports: 

/nfs/worker  192.168.137.0/24(rw,sync,no_subtree_check,no_root_squash,insecure)
/tftpboot    192.168.137.0/24(rw,sync,no_subtree_check,no_root_squash,insecure)


sudo exportfs -ra
sudo systemctl enable nfs-kernel-server
sudo systemctl restart nfs-kernel-server

NFS exports verified:

showmount -e localhost
# Export list for localhost:
# /tftpboot    192.168.137.0/24
# /nfs/worker  192.168.137.0/24

Step 4 — Configure cmdline.txt to Point Pi 3 to NFS Root

cmdline.txt is the kernel command line — it tells the Linux kernel where to find the root filesystem when booting. By setting root=/dev/nfs and providing the NFS server address, the kernel mounts its root filesystem from master over the network instead of from a local SD card.

console=serial0,115200 console=tty1 root=/dev/nfs \
  nfsroot=192.168.137.10:/nfs/worker,vers=3,proto=tcp \
  rw ip=dhcp rootwait rootdelay=5 elevator=deadline

Step 5 — Configure dnsmasq for PXE and TFTP

dnsmasq was configured in proxy DHCP mode — it works alongside the existing Windows ICS DHCP server without replacing it. In proxy mode, dnsmasq does not assign IP addresses itself; it intercepts DHCP requests and adds PXE boot instructions to the response. The enable-tftp and tftp-root settings activate the built-in TFTP server.

/etc/dnsmasq.conf: 

# Proxy DHCP — adds PXE info without replacing Windows ICS DHCP
dhcp-range=192.168.137.0,proxy

# Announces this server as a Raspberry Pi PXE boot server
pxe-service=0,"Raspberry Pi Boot"

# Activates built-in TFTP server in dnsmasq
enable-tftp
tftp-root=/tftpboot

# Tells Pi 3 to download bootcode.bin from master (192.168.137.10)
dhcp-boot=bootcode.bin,,192.168.137.10

# Assigns worker1 a fixed IP based on its MAC address
dhcp-host=b8:27:eb:6e:8c:72,192.168.137.101

Step 6 — Enable OTP Network Boot on Worker1

By default Pi 3 always boots from SD card first. To enable network boot, it must be permanently enabled in the OTP (One Time Programmable) memory — a special chip inside the Pi 3. Writing program_usb_boot_mode=1 triggers the OTP write on next reboot. Once written, this cannot be undone.

echo program_usb_boot_mode=1 | sudo tee -a /boot/firmware/config.txt
sudo reboot

Verified after reboot — 17:3020000a confirms network boot is permanently enabled:

vcgencmd otp_dump | grep 17:
# 17:3020000a   ← network boot permanently enabled in Pi 3 hardware chip

Step 7 — Create Serial Number Boot Folder

When Pi 3 starts network boot, it requests files from a folder named after its own hardware serial number from the TFTP server. This allows different Pi 3 nodes to have different boot configurations. Without this folder the Pi 3 cannot find its boot files.

Worker1 serial number retrieved:

cat /proc/cpuinfo | grep Serial
# Serial: 00000000786e8c72  → folder name: 786e8c72

sudo mkdir /tftpboot/786e8c72
sudo cp /tftpboot/*.elf  /tftpboot/786e8c72/   # GPU firmware files
sudo cp /tftpboot/*.dat  /tftpboot/786e8c72/   # memory split config
sudo cp /tftpboot/cmdline.txt /tftpboot/786e8c72/  # kernel command line
sudo cp /tftpboot/config.txt  /tftpboot/786e8c72/  # Pi boot config
sudo cp /tftpboot/kernel*     /tftpboot/786e8c72/  # Linux kernel
sudo cp /tftpboot/initramfs*  /tftpboot/786e8c72/  # initial RAM disk

6.3 What Worked

Component Result Evidence from Logs/Output
OS image on master ✅ Working Files confirmed in /tftpboot/ and /nfs/worker/
TFTP server running ✅ Working ss -ulnp \| grep 69 → dnsmasq listening on port 69
NFS exporting correctly ✅ Working showmount -e localhost → both paths exported
Pi 3 sending PXE requests ✅ Working dnsmasq log: PXE(eth0) b8:27:eb:6e:8c:72 proxy
Master responding with boot info ✅ Working dnsmasq log: bootfile name: bootcode.bin, next server: 192.168.137.10
OTP network boot enabled ✅ Working vcgencmd otp_dump: 17:3020000a on worker1
Serial number folder created ✅ Working /tftpboot/786e8c72/ with all boot files
NFS versions supported ✅ Working sudo cat /proc/fs/nfsd/versions → -2 +3 +4 +4.1 +4.2

6.4 What Did Not Work and Why

Critical Failure — TFTP File Transfer Never Completed

Even though the master correctly responded to every PXE request with the boot server address, and the Pi 3 received this information, the actual TFTP download of boot files never started. No TFTP transfer entries ever appeared in the dnsmasq logs. The Pi 3 bootloader entered a retry loop without progressing to the file download stage.

Root Cause — Two DHCP Servers on the Same Subnet

The network had two DHCP servers running simultaneously which caused the Pi 3 to receive conflicting responses:

DHCP Server Source Speed What It Provides to Pi 3
Server 1 Windows ICS (laptop) Fast — runs locally on laptop IP address only — NO PXE boot instructions
Server 2 dnsmasq (master) Slower — response travels over switch IP address + PXE boot server location

When Pi 3 powered on and broadcast "I need an IP address and boot instructions", both servers responded. Windows ICS always won because it runs locally on the laptop and responds faster than a response travelling over the switch. Pi 3 accepted the first response (Windows ICS), received only an IP address with no PXE context, and therefore never knew where to find the boot server.

Additional Hardware Limitation — Pi 3B

Model PXE without SD card Reason
Raspberry Pi 5 ✅ Fully possible Complete bootloader stored in firmware chip on board
Raspberry Pi 4 ✅ Fully possible Complete bootloader stored in firmware chip on board
Raspberry Pi 3B ❌ Not possible Bootloader NOT in firmware — requires bootcode.bin on SD card to start

This means even with a perfectly configured PXE server and no DHCP conflict, the Pi 3B cannot boot from pure network without any SD card. The SD card must contain at least bootcode.bin (52KB) to trigger the network boot process.

6.5 All Troubleshooting Attempts

# What Was Tried Result
1 Disabled Windows ICS completely All Pis lost internet — not viable for project
2 dnsmasq full DHCP mode (removed proxy) Pi 3 received DHCP from wrong server due to race condition
3 Proxy DHCP mode (dhcp-range=subnet,proxy) Pi 3 received PXE response but TFTP download never started
4 bootcode.bin only on SD card (no full OS) Pi 3 not visible on network at all
5 Full OS on SD + OTP enabled PXE requests sent and confirmed, master responded, TFTP incomplete
6 Serial number folder /tftpboot/786e8c72/ Folder created with all files — TFTP still did not transfer
7 Changed NFS version: vers=4.1 → vers=3 in cmdline.txt No change — TFTP stage was never reached to use NFS

6.7 Benefits and Drawbacks of PXE Boot

Benefits ✅ Drawbacks ❌
Single OS image — update once on master, all workers updated on next boot Master is single point of failure — if master goes down, all workers lose OS
Workers need only bootcode.bin (52KB) on SD card instead of full OS (2–4GB) Network required to boot — network failure means workers cannot start
Adding new worker only requires OTP enable + MAC in dnsmasq — no OS flashing Requires dedicated DHCP server — conflicts with existing DHCP (e.g. Windows ICS)
All workers run identical OS — no configuration drift between nodes NFS filesystem access is slower than local SD card for I/O operations
Centralized administration — one place to manage all worker configurations Complex initial setup compared to simply flashing individual SD cards

7. Task 1 — Final Status

7.1 Cluster Status

Node IP OS Special Hardware Status
master 192.168.137.10 RPi OS Lite 64-bit AI HAT+ Hailo-8L (13 TOPS) ✅ Online
sensor 192.168.137.20 RPi OS Lite 64-bit AI Camera imx500 NPU ✅ Online
worker1 192.168.137.101 RPi OS Lite 32-bit — ✅ Online
worker2 192.168.137.102 RPi OS Lite 32-bit — ✅ Online
worker3 192.168.137.103 RPi OS Lite 32-bit — ✅ Online
worker4 192.168.137.104 RPi OS Lite 32-bit — ✅ Online
worker5 192.168.137.105 RPi OS Lite 32-bit — ✅ Online
worker6 192.168.137.106 RPi OS Lite 32-bit — ✅ Online
worker7 192.168.137.107 RPi OS Lite 32-bit — ✅ Online
worker8 192.168.137.108 RPi OS Lite 32-bit — ✅ Online

7.2 Task Requirements Completion

Requirement Status Notes
OS deployed on all 10 nodes ✅ Complete Raspberry Pi OS Lite on all nodes
Object detection on sensor ✅ Complete 30 FPS on imx500 NPU, person detected at 0.85 confidence
AI HAT+ configured on master ✅ Complete Hailo-8L confirmed, firmware 4.23.0
Static IPs on all nodes ✅ Complete All nodes permanently assigned on 192.168.137.x
SSH keys distributed ✅ Complete Passwordless access from master to all workers
/etc/hosts on all nodes ✅ Complete All nodes reachable by hostname
PXE infrastructure on master ✅ Complete TFTP, NFS, dnsmasq all configured and running
PXE boot investigated ✅ Complete Communication confirmed, failure root cause identified
PXE boot fully working ⚠️ Not achieved DHCP conflict (Windows ICS vs dnsmasq) prevented TFTP transfer. Pi 3B also requires bootcode.bin on SD card.

Cloud Computing SS2026 — Task 1 Technical Documentation