Object Detection & AI Inference

Overview

The goal of Task 6 is to run real-time AI object detection on the cluster. EdgeGuard detects face coverage as a threat indicator — a person with a covered face (mask, hood, scarf covering features) triggers a suspicious alert, while a visible face is classified as safe. This is a binary classification per detected person in the camera frame.

The system consists of a Raspberry Pi 4 with an AI Camera Module capturing video, streaming it via ZMQ to the Raspberry Pi 5 which runs inference on the Hailo AI HAT+ neural processing unit. Inference results feed into the alert pipeline — suspicious detections are saved with a snapshot, stored in PostgreSQL, and pushed to Telegram.

Hardware

Hailo AI HAT+ (on Pi 5)

The Hailo AI HAT+ is an M.2 AI accelerator that sits on top of the Raspberry Pi 5. It contains the Hailo-8L NPU (Neural Processing Unit) with 13 TOPS (Tera Operations Per Second) of AI compute — dedicated hardware designed specifically for running neural networks efficiently.

The key advantage over running on the Pi 5 CPU: the NPU handles matrix multiplications (the core operation in neural networks) in dedicated silicon at orders of magnitude higher speed and lower power than a general-purpose CPU core.

Raspberry Pi AI Camera (on Pi 4)

The Raspberry Pi AI Camera Module contains the IMX500 sensor — a Sony imaging sensor with an onboard ISP. The Pi 4 captures 1080p video from this camera and streams frames to Pi 5 for inference. The camera runs on a dedicated Pi 4 (sensor-node, 10.10.10.40) to keep camera capture isolated from inference workload.

Dataset & Model Training

Dataset

We used Roboflow Universe as the primary data source — a public repository of annotated computer vision datasets. We found existing datasets for face mask detection and combined them with custom raw images captured by our own camera setup.

All data was processed and augmented inside Roboflow's online platform before export:

Classes:       face_visible, face_covered
Augmentations: Horizontal flip
               Rotation (±15°)
               Brightness variation (±25%)
               Blur (up to 1.5px)
               Mosaic augmentation
Split:         Train 80% / Validation 15% / Test 5%

Training — YOLOv8n

We chose YOLOv8n (nano variant) from Ultralytics. The "n" (nano) variant was specifically chosen because it is the smallest and fastest YOLOv8 model — optimized for edge devices with limited compute. It trades some accuracy for significantly lower inference time, which is the right tradeoff for real-time surveillance at 20 FPS.

Training was done in Google Colab using the free GPU tier (T4 GPU). Output: a best.pt PyTorch weights file.

from ultralytics import YOLO

model = YOLO('yolov8n.pt')  # start from pretrained nano weights

model.train(
    data='dataset.yaml',     # Roboflow exported dataset config
    epochs=100,
    imgsz=640,
    batch=16,
    device='cuda'            # Colab T4 GPU
)
# Output: runs/detect/train/weights/best.pt

Model Evaluation

After training, we evaluated the model on the held-out test set to measure how well it detects face coverage. The key metrics for object detection are:

Training Configuration

Hardware:    Google Colab — Tesla T4 GPU (14.9GB VRAM)
Framework:   Ultralytics YOLOv8 8.4.63 · PyTorch 2.11 · CUDA 12.8
Model:       YOLOv8n (nano) — 73 layers · 3,006,038 parameters · 8.1 GFLOPs
Epochs:      20
Image size:  640×640
Batch size:  16
Dataset:     326 validation images · 1,018 instances
Classes:     face_covered · non_covered_face
Training time: 0.300 hours (~18 minutes) on T4
Output:      best.pt — 6.2MB

Evaluation Results — best.pt (Final Model)

Training Progress — How the Model Improved

Epoch  1:  mAP50=0.789  mAP50-95=0.460  ← starting point
Epoch  5:  mAP50=0.890  mAP50-95=0.587  ← rapid improvement
Epoch 10:  mAP50=0.923  mAP50-95=0.628  ← strong performance
Epoch 15:  mAP50=0.937  mAP50-95=0.653  ← approaching plateau
Epoch 20:  mAP50=0.951  mAP50-95=0.673  ← final model (best.pt)

All losses decreasing consistently:
  box_loss:  1.499 → 1.007  ← localization improving
  cls_loss:  2.107 → 0.543  ← classification improving
  dfl_loss:  1.172 → 0.971  ← distribution focal loss improving

📈 BoxPR_curve.png Precision-Recall curve per class at different confidence thresholds

Figure 1 — PR curve showing precision vs recall tradeoff for face_covered and non_covered_face at varying confidence thresholds.

🔢 confusion_matrix_normalized.png Normalized confusion matrix — % correct vs misclassified per class

Figure 2 — Normalized confusion matrix showing classification accuracy per class. Diagonal = correct predictions.

The Conversion Challenge: PT → HEF

This was the most technically challenging part of Task 6. The Hailo AI HAT+ does not run standard PyTorch or ONNX models — it requires its own proprietary .HEF (Hailo Executable Format) file. Converting from PyTorch to HEF is a multi-step process that requires significant GPU compute.

Why Not Use RPK?

The Raspberry Pi AI Camera comes with RPK (Raspberry Pi Kit) — a simpler way to deploy models. However RPK only works with the IMX500 sensor's onboard processing, not with the Hailo AI HAT+. Since we specifically wanted to use the Hailo AI HAT+ for its superior compute (13 TOPS vs IMX500's limited onboard processing), we had to go through the full HEF conversion pipeline.

The Conversion Pipeline

The GPU Problem — and How We Solved It

Steps 2 and 3 require a powerful GPU to run the Hailo Dataflow Compiler. None of the team members had access to a local GPU machine, and the Pi 5 itself cannot run the compiler. This blocked deployment.

# Step 1: Export PyTorch to ONNX
python3 -c "
from ultralytics import YOLO
model = YOLO('best.pt')
model.export(format='onnx', imgsz=640)
"

# Step 2: Parse ONNX to HAR using Hailo Model Zoo
hailomz parse --hw-arch hailo8l \
    --ckpt best.onnx \
    yolov8n

# Step 3: Optimize and compile HAR to HEF
hailomz optimize --hw-arch hailo8l \
    --har yolov8n.har \
    --calib-path calibration_images/

hailomz compile --hw-arch hailo8l \
    --har yolov8n_optimized.har

# Output: yolov8n.hef — deploy this to Pi 5

Video Streaming — From Pi 4 to Pi 5

Problem with HTTP MJPEG

Our first approach was streaming via HTTP MJPEG — the Pi 4 serves camera frames as a continuous HTTP response. This worked but had a 2–4 second latency due to HTTP connection overhead, buffering, and TCP retransmission. A 2–4 second delay between what the camera sees and what the AI processes makes real-time surveillance impossible.

Solution: ZMQ PubSub

We replaced HTTP MJPEG with ZeroMQ (ZMQ) PubSub — a high-performance asynchronous messaging library. The Pi 4 publishes compressed JPEG frames on a ZMQ socket. The Pi 5 subscribes and receives frames as fast as the network allows — with no HTTP overhead, no buffering, and no connection management. Latency dropped from 2–4 seconds to ~50ms.

# Pi 4 — Publisher (det.py)
import zmq, cv2

context = zmq.Context()
socket = context.socket(zmq.PUB)
socket.bind("tcp://0.0.0.0:5555")

while True:
    ret, frame = camera.read()
    _, buffer = cv2.imencode('.jpg', frame, [cv2.IMWRITE_JPEG_QUALITY, 80])
    socket.send(buffer.tobytes())   # fire and forget — no ACK needed

# Pi 5 — Subscriber (inference service)
socket = context.socket(zmq.SUB)
socket.connect("tcp://10.10.10.40:5555")
socket.setsockopt(zmq.SUBSCRIBE, b"")

while True:
    frame_bytes = socket.recv()     # ~50ms latency
    frame = cv2.imdecode(np.frombuffer(frame_bytes, np.uint8), cv2.IMREAD_COLOR)
    # → run Hailo inference on frame

Detection Logic & Alert Pipeline

What counts as suspicious?

The model outputs a bounding box and class label for each detected person in the frame. We defined two classes during training:

A detection only triggers an alert if the model's confidence score is above 0.5 (50%). This threshold was chosen to balance false alarms (too low) against missed detections (too high).

Alert Pipeline

Each alert stored in PostgreSQL contains: event type, confidence score, timestamp, camera ID, and MinIO image URL. The EdgeGuard frontend displays all alerts with pagination and the snapshot image.

Screenshots

📷 detection_live.png Live camera feed with detection overlay and bounding boxes

Figure 1 — Live detection in EdgeGuard dashboard showing bounding boxes and confidence scores.

🔔 alert_history.png Alert history page showing suspicious event snapshots

Figure 2 — Alert history with MinIO snapshot images and confidence scores per alert.

Key Findings