Infrastructure Setup

Overview

We assembled a heterogeneous 10-node cluster from three types of Raspberry Pi hardware. The Pi 5 serves as the master node running all control-plane services. The Pi 4 is a dedicated sensor node with an AI Camera Module. Eight Pi 3B+ nodes act as compute workers for Kubernetes, MPI benchmarks, and HPL — 7 of 8 booting entirely over the network via PXE with no SD cards required.

All nodes communicate over a dedicated 100Mbit/s switched LAN (10.10.10.0/24). The Pi 5 provides DHCP, TFTP, NFS, and DNS for the entire subnet — making it the single point of administration for the cluster.

Hardware

Accessories

PXE Diskless Boot

Seven Pi 3 nodes boot entirely over the network — no SD cards required. The Pi 5 serves DHCP and TFTP (via dnsmasq) for the initial boot firmware, and NFS for the root filesystem. Each Pi 3 gets its own NFS root directory exported from the Pi 5 at /nfs/clients/pi3-0X.

Setup Steps

• 1

  Enable USB/Network Boot on Pi 5

  Configure dnsmasq on Pi 5 to serve DHCP on eth0 (10.10.10.0/24) and TFTP boot files. Each Pi 3 MAC address gets a static IP assignment.

• 2

  Prepare Root Filesystems on Pi 5

  Copy a base Raspberry Pi OS image for each Pi 3 into /nfs/clients/pi3-0X/. Each node gets its own copy so hostname, SSH keys, and k3s tokens can differ.

• 3

  Export NFS Shares

  Add each Pi 3 root filesystem to /etc/exports on Pi 5. Also export /nfs/shared — the shared storage used by MPI and Task Distributor.

• 4

  Configure TFTP Boot Files

  Each Pi 3 serial number gets a directory under /tftpboot/ containing the firmware files. Pi 3 reads these on power-on before mounting NFS root.

• 5

  Configure cmdline.txt

  Set the kernel command line in each node's TFTP boot directory to mount the NFS root: root=/dev/nfs nfsroot=10.10.10.1:/nfs/clients/pi3-0X

• 6

  Verify Boot

  Power on Pi 3 — it broadcasts DHCP request, Pi 5 assigns IP and points to TFTP, firmware loads, kernel mounts NFS root, system boots. Check with ssh admin@10.10.10.21

Key Configuration Files

# DHCP for cluster LAN
interface=eth0
dhcp-range=10.10.10.21,10.10.10.28,12h

# Static IP by MAC address
dhcp-host=<pi3-01-MAC>,pi3-01,10.10.10.21
dhcp-host=<pi3-02-MAC>,pi3-02,10.10.10.22
# ... repeat for all 8 Pi3 nodes

# TFTP for PXE boot
enable-tftp
tftp-root=/tftpboot
pxe-service=0,"Raspberry Pi Boot"

# Per-node root filesystems
/nfs/clients/pi3-01  10.10.10.21(rw,sync,no_subtree_check,no_root_squash)
/nfs/clients/pi3-02  10.10.10.22(rw,sync,no_subtree_check,no_root_squash)
# ... repeat for pi3-03 through pi3-08

# Shared storage for MPI and Task Distributor
/nfs/shared  10.10.10.0/24(rw,sync,no_subtree_check,no_root_squash)

# Ping all Pi3 nodes
for ip in 10.10.10.21 10.10.10.22 10.10.10.23 10.10.10.24 \
          10.10.10.25 10.10.10.26 10.10.10.27 10.10.10.28; do
    echo -n "$ip: "
    ping -c 1 -W 1 $ip >/dev/null && echo "UP" || echo "DOWN"
done

# Check uptime on all nodes
for ip in 10.10.10.21 10.10.10.22 10.10.10.23 10.10.10.24 \
          10.10.10.25 10.10.10.26 10.10.10.27 10.10.10.28; do
    echo -n "$ip: "
    ssh admin@$ip "uptime" 2>/dev/null
done

Benefits & Drawbacks

Network Configuration

The cluster uses a dedicated 100Mbit/s LAN on subnet 10.10.10.0/24 via two TP-Link TL-SG108E managed switches. Pi 5 also connects to external WiFi (wlan0) for internet access and remote SSH — the hotspot IP changes per session.

External WiFi (wlan0)
    │  IP: 192.168.x.x (changes per hotspot session)
    ▼
┌─ Raspberry Pi 5 — master-node (eth0: 10.10.10.1) ──────┐
│  DHCP · TFTP · NFS · k3s master · Hailo inference       │
│  PostgreSQL · MinIO · Prometheus · Grafana               │
└──────────────────────────┬──────────────────────────────┘
                           │ 100Mbit/s
        ┌──────────────────┴──────────────────────┐
        │         Switch 1 (TP-Link)              │
        │  Pi5 · pi3-01 · pi3-02 · pi3-03        │
        │  pi3-04 · pi3-05 · pi3-06              │
        └──────────────────┬──────────────────────┘
                           │ uplink
        ┌──────────────────┴──────────────────────┐
        │         Switch 2 (TP-Link)              │
        │  pi3-07 · pi3-08 · sensor-node (Pi4)   │
        └─────────────────────────────────────────┘

# Required before running MPI or Task Distributor benchmarks
for ip in 10.10.10.21 10.10.10.22 10.10.10.23 10.10.10.24 \
          10.10.10.25 10.10.10.26 10.10.10.27 10.10.10.28; do
    ssh admin@$ip "sudo mkdir -p /nfs/shared && \
        sudo mount -t nfs 10.10.10.1:/nfs/shared /nfs/shared" &
done
wait
echo "NFS mounted on all Pi3 nodes"

# Verify
for ip in 10.10.10.21 10.10.10.22 10.10.10.23 10.10.10.24 \
          10.10.10.25 10.10.10.26 10.10.10.27 10.10.10.28; do
    echo -n "$ip: "
    ssh admin@$ip "ls /nfs/shared/ 2>/dev/null | wc -l || echo 'NOT MOUNTED'"
done

k3s Kubernetes

k3s is deployed across all 10 nodes with the Pi 5 as the control plane. Pod anti-affinity rules ensure replicated services (backend, frontend) never land on the same node — providing genuine high availability. Before running benchmarks, k3s-agent must be stopped on all Pi 3 nodes to free CPU and memory.

# Stop k3s-agent to free CPU and RAM for benchmarks
for ip in 10.10.10.21 10.10.10.22 10.10.10.23 10.10.10.24 \
          10.10.10.25 10.10.10.26 10.10.10.27 10.10.10.28; do
    ssh admin@$ip "sudo systemctl stop k3s-agent" &
done
wait
echo "k3s-agent stopped on all Pi3 nodes"

# Verify all stopped
for ip in 10.10.10.21 10.10.10.22 10.10.10.23 10.10.10.24 \
          10.10.10.25 10.10.10.26 10.10.10.27 10.10.10.28; do
    echo -n "$ip: "
    ssh admin@$ip "systemctl is-active k3s-agent"
done

# Restart k3s-agent after benchmarks
for ip in 10.10.10.21 10.10.10.22 10.10.10.23 10.10.10.24 \
          10.10.10.25 10.10.10.26 10.10.10.27 10.10.10.28; do
    ssh admin@$ip "sudo systemctl start k3s-agent" &
done
wait

# View all nodes and their status
kubectl get nodes -o wide

# View all running pods
kubectl get pods -A

# Check services
kubectl get svc -A

Camera & AI Inference Pipeline

The Pi 4 streams video from the AI Camera Module to the Pi 5 via ZMQ PubSub. The Pi 5 runs YOLOv8n on the Hailo AI HAT+ for face coverage detection — classifying each detected person as "face visible" (safe) or "face covered" (suspicious).

Model Pipeline: PyTorch → Hailo HEF

• 1

  Dataset

  Roboflow Universe + custom raw data. Augmented inside Roboflow (horizontal flip, rotation, brightness variation).

• 2

  Training

  YOLOv8n trained in Google Colab (free GPU tier). Output: PyTorch .pt weights file.

• 3

  Conversion Challenge

  PyTorch .pt cannot run directly on the Hailo AI HAT+. Conversion chain: PT → ONNX → HAR → HEF via Hailo Model Zoo. This required Hailo's dataflow compiler running on cloud GPUs (Massive Compute).

• 4

  Streaming

  Original HTTP MJPEG stream had 2–4s latency. Replaced with ZMQ PubSub achieving ~50ms latency. Pi 4 publishes frames, Pi 5 subscribes and runs inference.

• 5

  Result

  14–17ms per frame on Hailo NPU vs 2–3 minutes per frame on CPU — approximately 8,000× speedup. Sustained at 20 FPS @ 1080p. Inference service starts automatically via systemd on boot.

Key Findings