A custom-built Prometheus-driven auto-scaler that dynamically adds and removes the Raspberry Pi 4 as a k3s compute node based on real-time cluster CPU load.
Auto-scaling is one of the defining features of modern cloud computing — the ability for a system to automatically add or remove compute resources based on demand, without human intervention. In production cloud environments (AWS, GCP, Azure), this is handled by managed services. We built our own from scratch on a Raspberry Pi cluster.
The EdgeGuard auto-scaler monitors cluster CPU utilization via Prometheus every 10 seconds. When average CPU across all Pi3 nodes exceeds a configurable threshold, it automatically executes the k3s node join sequence on the Raspberry Pi 4 — adding 4 additional ARM cores to the cluster within ~90 seconds. When CPU drops back below the threshold for a sustained period, the Pi4 gracefully drains its workloads and leaves the cluster.
Most student projects deploy a static cluster and leave it running. We implemented a dynamic elastic cluster — the compute capacity literally changes in real time based on load. This is exactly how cloud providers charge by the hour — resources exist only when needed. We demonstrated this concept on physical hardware with real Kubernetes orchestration, not a simulation.
AWS Auto Scaling, GCP Managed Instance Groups, Azure VMSS are all managed services. We wrote every line of the scaling logic ourselves in Python — Prometheus query, kubectl commands, state machine, cooldown timers.
This is not a virtual machine spawning in seconds. A real physical Raspberry Pi 4 board is added to the Kubernetes cluster — it boots k3s-agent, registers with the control plane, and becomes schedulable. You can see it with kubectl get nodes.
Scaling decisions are based on actual CPU data collected by Node Exporter from all 8 Pi3 nodes and averaged by Prometheus. Not a synthetic trigger — real cluster load from inference, benchmarks, or k3s pod workloads.
The scaling threshold is not hardcoded. Any authorized user can change it live from the EdgeGuard frontend — the new value takes effect on the next polling cycle with no restart required.
When Pi4 needs to leave, the auto-scaler first cordons the node (marks it unschedulable), drains all pods to other nodes, waits for completion, then removes it from the cluster. No workload disruption.
Every scale-out and scale-in event sends a Telegram notification to the team group with the trigger reason, current CPU, and Pi4 join/leave status.
Every 10 seconds the auto-scaler queries Prometheus for the average CPU across all Pi3 nodes. If the average exceeds the threshold (default 80%) for 3 consecutive checks (~30 seconds), scale-out is triggered. The 3-check requirement prevents false triggers from momentary CPU spikes.
The auto-scaler checks if sensor-node (Pi4) is already registered in k3s. If it is, no action is taken. If not, the join sequence begins.
The Pi5 backend SSHs to Pi4 (10.10.10.40) and starts the k3s-agent service with the cluster token and Pi5 server address. The k3s-agent on Pi4 contacts the Pi5 control plane and requests to join.
Poll kubectl get node sensor-node every 5 seconds until status is Ready. Typically takes 60–90 seconds for the Pi4 to fully join the cluster and pass health checks.
Once Pi4 is Ready, send a Telegram message to the team and log the event with timestamp and trigger CPU value. The auto-scaler then enters a cooldown period (5 minutes) before it can scale again.
When cluster CPU drops below the threshold for 3 consecutive checks after Pi4 has joined, scale-in is triggered. Same sustained-check requirement to avoid flapping.
kubectl cordon sensor-node — prevents any new pods from being scheduled onto Pi4. Existing pods continue running until drained.
kubectl drain sensor-node --ignore-daemonsets --delete-emptydir-data — evicts all pods running on Pi4 and reschedules them on Pi3 nodes. k3s waits until all pods have migrated before completing.
kubectl delete node sensor-node — removes Pi4 from the k3s node registry. Then SSH to Pi4 and stop k3s-agent so it doesn't try to rejoin automatically.
Notify team that Pi4 has left the cluster, with downtime duration and reason. Enter cooldown period before next possible scale event.
import threading import subprocess import requests import time from datetime import datetime PROMETHEUS_URL = "http://prometheus-service:9090" PI4_IP = "10.10.10.40" PI4_NODE_NAME = "sensor-node" K3S_TOKEN = "your_k3s_token" K3S_SERVER = "https://10.10.10.1:6443" POLL_INTERVAL = 10 # seconds between Prometheus checks COOLDOWN_SECONDS = 300 # 5 min cooldown after any scale event SUSTAINED_CHECKS = 3 # consecutive checks before scaling # Shared state — readable by /api/autoscaler/status endpoint autoscaler_state = { "threshold": 80.0, # CPU % to trigger scale-out "current_cpu": 0.0, "pi4_joined": False, "last_event": None, "event_log": [], "in_cooldown": False, "scale_out_count": 0, "scale_in_count": 0, } high_cpu_streak = 0 low_cpu_streak = 0 def get_cluster_cpu(): # PromQL: average CPU usage across all Pi3 nodes query = ('100 - (avg(rate(node_cpu_seconds_total' '{mode="idle",job="node-exporter"}' '[1m])) * 100)') r = requests.get( f"{PROMETHEUS_URL}/api/v1/query", params={"query": query}, timeout=5 ) result = r.json()["data"]["result"] return float(result[0]["value"][1]) if result else 0.0 def pi4_in_cluster(): result = subprocess.run( ["kubectl", "get", "node", PI4_NODE_NAME, "--no-headers"], capture_output=True, text=True ) return result.returncode == 0 def scale_out(): global high_cpu_streak log_event(f"⚡ Scale-out triggered — CPU at {autoscaler_state['current_cpu']:.1f}%") # SSH to Pi4 and start k3s-agent subprocess.run([ "ssh", f"admin@{PI4_IP}", f"sudo K3S_URL={K3S_SERVER} K3S_TOKEN={K3S_TOKEN} " f"sh -c 'systemctl start k3s-agent'" ], timeout=30) # Wait for Pi4 to become Ready (up to 120 seconds) for _ in range(24): time.sleep(5) result = subprocess.run( ["kubectl", "get", "node", PI4_NODE_NAME, "--no-headers", "-o", "custom-columns=STATUS:.status.conditions[-1].type"], capture_output=True, text=True ) if "Ready" in result.stdout: autoscaler_state["pi4_joined"] = True autoscaler_state["scale_out_count"] += 1 log_event("✅ Pi4 joined cluster — Ready") send_telegram( f"🔥 <b>AUTO-SCALER: Scale Out</b>\n\n" f"📊 Cluster CPU was {autoscaler_state['current_cpu']:.1f}%\n" f"➕ Pi4 sensor-node joined k3s cluster\n" f"🖥️ Cluster now has {8+1} active nodes" ) high_cpu_streak = 0 enter_cooldown() return log_event("❌ Pi4 join timed out after 120s") def scale_in(): global low_cpu_streak log_event(f"📉 Scale-in triggered — CPU at {autoscaler_state['current_cpu']:.1f}%") # Cordon — prevent new pods from scheduling on Pi4 subprocess.run(["kubectl", "cordon", PI4_NODE_NAME], timeout=30) # Drain — evict all pods gracefully subprocess.run([ "kubectl", "drain", PI4_NODE_NAME, "--ignore-daemonsets", "--delete-emptydir-data", "--timeout=120s" ], timeout=150) # Remove from cluster subprocess.run(["kubectl", "delete", "node", PI4_NODE_NAME], timeout=30) # Stop k3s-agent on Pi4 subprocess.run([ "ssh", f"admin@{PI4_IP}", "sudo systemctl stop k3s-agent" ], timeout=15) autoscaler_state["pi4_joined"] = False autoscaler_state["scale_in_count"] += 1 log_event("✅ Pi4 left cluster gracefully") send_telegram( f"📉 <b>AUTO-SCALER: Scale In</b>\n\n" f"📊 Cluster CPU dropped to {autoscaler_state['current_cpu']:.1f}%\n" f"➖ Pi4 sensor-node left k3s cluster\n" f"🖥️ Cluster back to 8 Pi3 nodes" ) low_cpu_streak = 0 enter_cooldown() def enter_cooldown(): autoscaler_state["in_cooldown"] = True threading.Timer(COOLDOWN_SECONDS, lambda: autoscaler_state.update({"in_cooldown": False}) ).start() def log_event(message): entry = {"time": datetime.now().isoformat(), "message": message} autoscaler_state["event_log"].insert(0, entry) autoscaler_state["event_log"] = autoscaler_state["event_log"][:50] # keep last 50 print(f"[AutoScaler] {message}") def autoscaler_loop(): global high_cpu_streak, low_cpu_streak while True: try: cpu = get_cluster_cpu() autoscaler_state["current_cpu"] = cpu threshold = autoscaler_state["threshold"] in_cooldown = autoscaler_state["in_cooldown"] pi4_joined = autoscaler_state["pi4_joined"] if not in_cooldown: if cpu > threshold and not pi4_joined: high_cpu_streak += 1 low_cpu_streak = 0 if high_cpu_streak >= SUSTAINED_CHECKS: scale_out() elif cpu < (threshold - 10) and pi4_joined: # 10% hysteresis — scale in at threshold-10 to avoid flapping low_cpu_streak += 1 high_cpu_streak = 0 if low_cpu_streak >= SUSTAINED_CHECKS: scale_in() else: high_cpu_streak = 0 low_cpu_streak = 0 except Exception as e: print(f"[AutoScaler] Error: {e}") time.sleep(POLL_INTERVAL) # Start auto-scaler when Flask app starts scaler_thread = threading.Thread(target=autoscaler_loop, daemon=True) scaler_thread.start()
@app.route('/api/autoscaler/status', methods=['GET']) @token_required def get_autoscaler_status(): return jsonify(autoscaler_state) @app.route('/api/autoscaler/threshold', methods=['POST']) @token_required def set_threshold(): data = request.get_json() new_threshold = float(data.get('threshold', 80)) if 10 <= new_threshold <= 95: autoscaler_state['threshold'] = new_threshold log_event(f"⚙️ Threshold changed to {new_threshold}%") return jsonify({'threshold': new_threshold}) return jsonify({'error': 'Threshold must be 10–95'}), 400
| UI Element | Data Source | Description |
|---|---|---|
| CPU Gauge | autoscaler_state.current_cpu | Live cluster average CPU — updates every 10s |
| Threshold Slider | autoscaler_state.threshold | Adjustable 10–95% — POST to /api/autoscaler/threshold |
| Pi4 Status Badge | autoscaler_state.pi4_joined | Shows "In Cluster" (green) or "Standby" (grey) |
| Scale Event Log | autoscaler_state.event_log | Last 50 events with timestamps — scale-out, scale-in, errors |
| Scale Count | scale_out_count / scale_in_count | Total number of scale events since backend started |
Without hysteresis, the system would flap — Pi4 joins at 80% CPU, load drops to 79%, Pi4 leaves, load rises to 80%, Pi4 joins again, in a rapid loop. We scale out at 80% but only scale in when CPU drops to 70% (threshold - 10%). This 10% hysteresis band prevents flapping and gives the newly joined Pi4 time to absorb workload and genuinely reduce cluster CPU.
A single CPU spike to 85% does not trigger scale-out. Three consecutive readings above threshold (~30 seconds of sustained high load) are required. This prevents unnecessary scale events from brief benchmark spikes, page refreshes, or momentary inference bursts.
After any scale event (in or out), the auto-scaler enters a 5-minute cooldown. This gives the cluster time to stabilize after the topology change before making another scaling decision. Without cooldown, the system could thrash between 8-node and 9-node configurations continuously.
The Pi4 is a physical board that needs to boot k3s-agent, register with the control plane, pass readiness checks, and receive pod scheduling. This takes 60–90 seconds. During this window the cluster is still running at high CPU on 8 nodes — the auto-scaler waits for Ready before logging success.
Tested multiple times under real benchmark load. The k3s-agent on Pi4 consistently reaches Ready state within 90 seconds of the SSH trigger command.
kubectl drain successfully migrates all non-daemonset pods to Pi3 nodes before Pi4 leaves. No request errors observed in the backend during scale-in events due to the anti-affinity rules ensuring the backend replica was never on Pi4.
This implementation covers: metrics-driven scaling decisions, hysteresis to prevent flapping, graceful node drain, cooldown periods, and event logging — the same concepts used in production auto-scalers at every major cloud provider. Built entirely with open-source tools on physical hardware.