Task 7 / 8 — Deployment Architecture Decision (for review)¶

Status: Implemented and running at http://192.168.137.10/ Scope: How we deploy the backend (Task 7) and frontend (Task 8) onto the Pi cluster Author: (Tasks 7–9)

This document describes the architecture as built. All open questions from the original proposal have been resolved — see §9 for the decisions made.

1. TL;DR¶

Deploy the backend and frontend as Docker containers on a lightweight Kubernetes cluster (k3s), following the official course architecture diagram (§3): the frontend on the master, a backend replica on the master and on every one of the 8 workers, MinIO distributed across the workers for evidence images, and PostgreSQL on the master for event metadata. This satisfies every Task 7 keyword (Docker, k3s, scalable, high-availability, distributed file system) while leaving the existing MPI / NFS / monitoring stack untouched.

The main thing I want a second opinion on: how much high availability is worth the effort for a demo — specifically whether we accept the master node as a single point of failure (simplest) or invest in a 3-server HA control plane (a lot more work) — plus whether co-locating backend + MinIO on the RAM-tight workers is OK.

2. What Task 7 actually requires¶

From the project brief:

Task 7 (backend): "The backend can be deployed as Docker container(s) on a Raspberry Pi Kubernetes cluster with k3s … a robust and scalable high availability cluster of Raspberry Pi 3 nodes. The cluster includes a distributed file system (e.g. Ceph + S3 gateway) or a storage service (e.g. SeaweedFS or MinIO)."

Task 8 (frontend): "The frontend will also be deployed as a Docker container on a Raspberry Pi Kubernetes cluster with k3s or a similar solution… Use REST or MQTT for communication between components."

Keywords we must hit: Docker containers · k3s · scalable · high availability · distributed file system.

3. The authoritative topology (course diagram)¶

This is the official architecture from the course page — it's our source of truth:

Official SS2026 edge-computing architecture

What the diagram dictates:

Frontend runs on the master (Pi 5).
A Backend runs on the master and on every worker (Pi3 1–8) — i.e. the workers are the scalable / high-availability backend tier. (This is the official answer to "do we need backend replicas?" — yes, one per worker.)
The User / Administrator reaches the frontend on the master.
The sensor node (Pi 4 + AI Camera Module) feeds the master.
The master also carries a Pi AI HAT+ accelerator.

The diagram is a component view — it deliberately doesn't draw the database or storage. Section 4 shows how we realize it concretely on k3s (where MinIO and Postgres go).

4. Cluster state before k3s (the gap that was closed)¶

Before this deployment, the cluster was a classic HPC setup, not a Kubernetes one:

Node	Hardware	Currently runs
master (`192.168.137.10`)	Pi 5, 8 GB, aarch64, Debian 13, + AI HAT+	Docker Compose monitoring (Grafana/Prometheus/Alertmanager), PostgreSQL, single-node MinIO, Flask backend (`server.py`, as a process), `dnsmasq` (cluster DHCP/DNS), NFS export
worker1–8 (`.101–.108`)	Pi 3, ~900 MB, aarch64, Debian 13	MPI compute (HPL, Amdahl/Gustafson), NFS clients, `node_exporter`. No Docker, no k3s.
sensor (`192.168.137.20`)	Pi 4 + AI Camera Module	Detection model, pushes events to the backend

The gap (now closed): the brief + diagram assumed a k3s cluster with a backend on every node and distributed storage; we had MPI + NFS + a single-node Docker stack on the master. This deployment closed that gap without disturbing the compute/monitoring work (Tasks 2–5).

Preflight confirmed all nodes are aarch64 / 64-bit, so standard ARM container images (MinIO, nginx, etc.) will run — this was the main blocker and it's clear.

5. Proposed realization on k3s¶

We map the diagram onto k3s, adding distributed MinIO (the Task 7 "distributed file system") co-located on the workers and keeping PostgreSQL on the master:

flowchart TB
  user["User / Administrator"] -->|HTTP :80| traefik["Traefik ingress (master)"]
  sensor["Sensor (Pi 4 + AI camera)"] -->|REST events| traefik

  subgraph master["master — Pi 5 (8 GB) + AI HAT+"]
    traefik --> fe["Frontend pod (nginx + React)"]
    traefik --> svc["Backend Service (load-balanced)"]
    svc --> bem["Backend pod"]
    bem --> pg[("PostgreSQL — metadata")]
    k3ss["k3s server (control plane)"]
  end

  subgraph workers["worker1–8 — Pi 3 (~900 MB each)"]
    bw1["Backend + MinIO"]
    bw2["Backend + MinIO"]
    bw3["Backend + MinIO"]
    bw8["Backend + MinIO (×8 total)"]
  end

  svc --> bw1
  svc --> bw2
  svc --> bw8
  bem -->|"S3 (images)"| bw1
  bw1 -. "erasure-coded shards" .- bw2 -. "across all 8" .- bw8
  bw1 --> pg

Component placement:

Component	Where	Why
k3s server (control plane)	master	Only node with RAM headroom (8 GB)
k3s agents	all 8 workers	Make them part of the cluster (~300 MB each)
Backend (Flask)	master + all 8 workers, one replica each	Per the course diagram; load-balanced by a Service
Distributed MinIO (8-node, EC:4)	worker1–8 (co-located with backend)	The "distributed file system" — images sharded across all workers
Frontend (nginx + React build)	master	Served via Traefik at `http://192.168.137.10/`
PostgreSQL	master (existing Compose)	Event metadata; no migration needed
Local image registry (`registry:2`)	master	k3s pulls our built images from here

Untouched: the Task 5 monitoring Compose stack, MPI, NFS, and dnsmasq all keep running as-is. k3s uses its own bundled containerd, so it coexists with Docker on the master.

6. Why a backend on every worker?¶

The backend (server.py) is the single component everything talks to: the sensor pushes detections to it, the frontend reads events from it, the Telegram bot fires from it. As one instance, if it dies the whole system's API goes dark — no event ingestion, no dashboard, no alerts.

The course diagram puts a Backend on every worker, so multiple replicas behind a k3s Service (which load-balances across them) are the intended design, not an add-on. That buys us:

Benefit	What it means in practice
High availability	A worker (and its backend replica) dies → the Service routes to the others. No outage.
Self-healing	k3s restarts a dead pod automatically (vs. SSH-ing in to restart a process).
Scalability	Request load (sensor pushes + dashboard polling) spreads across replicas.
Zero-downtime updates	Rolling deploys — new version comes up before the old one goes down.

Important precondition: replicas only work if the backend is stateless — all state must live in Postgres + MinIO, never on a pod's local disk. Our backend already does this (events → Postgres, images → MinIO), so it's safe to replicate.

What this gives us, and the remaining gap: because a backend runs on every worker, we get genuine node-level HA for the API. The remaining single point of failure is the master (it holds Postgres, the ingress, and the frontend), which the worker replicas don't protect against — see Q1. The only real cost of backend-on-every-worker is RAM on the 900 MB Pi 3s, where each backend pod (~80 MB) sits alongside MinIO — see Q2.

7. Failure scenarios — what happens if a node goes down?¶

Assumes MinIO EC:4 across 8 workers (each image = 4 data + 4 parity shards), with a backend replica on each worker.

Scenario	Impact	Recovery
1–3 workers down	Fully operational. Images still readable and writable; the lost backend replicas just drop out of the Service (it routes to the rest); k3s reschedules where it can.	Node returns → MinIO auto-heals the missing shards; replica rejoins. Automatic.
4 workers down	Images still readable (EC:4 tolerates 4 lost shards); new image writes pause (write quorum 5/8). Backend still served by remaining replicas. No data loss.	Bring any one node back → writes resume + auto-heal.
5+ workers down	MinIO unavailable; data loss only if more than 4 are permanently destroyed (dead SD card). Metadata in Postgres unaffected.	Restore nodes; if ≤4 were truly lost, all images rebuild from parity.
A backend pod crashes	None visible — Service routes to other replicas; k3s restarts the dead pod.	Automatic.
Sensor (Pi 4) down	No new detections; existing events + dashboard work normally.	Restart the sensor.
MASTER down ⚠️	Major outage. Control plane, PostgreSQL, frontend, ingress, and dnsmasq + NFS live here. Even though backend replicas on the workers stay alive, there's no ingress to reach them and no Postgres to serve — so the API + dashboard are down. MinIO data on the workers survives but is unreachable until the master returns.	Bring the master back; pods + Postgres reconcile.

Takeaway: the design is genuinely resilient to worker failures (the distributed part — exactly what Task 7's "distributed file system / HA" is about). The master is the remaining single point of failure — see Q1.

8. Alternatives considered¶

Option	Verdict
Path A: k3s + distributed MinIO (this doc)	Chosen. Matches the course diagram; hits every keyword; best demo; reversible. More setup effort.
Path B: Docker Compose/Swarm + distributed MinIO	Lower risk, less faithful to "k3s." Good fallback if k3s setup eats too much time.
MinIO on 4 workers instead of 8	Frees RAM on the other 4, but less distribution. Backend still on all 8 (cheap). Possible compromise (Q2).
EC:2 instead of EC:4	75% usable capacity but only tolerates 2 failures. Default pick is EC:4 (50% usable, tolerates 4) — SD cards are 32 GB so capacity isn't the constraint; resilience is the better demo.
3-server k3s HA control plane	True control-plane HA, removes the master SPOF — but 3× the setup/complexity. Likely overkill for a demo (Q1).
Ceph instead of MinIO	Brief allows it, but Ceph is far heavier on 900 MB Pi 3s. MinIO is the pragmatic, allowed choice.

9. Decisions made (originally open questions)¶

HA scope: Master SPOF accepted for the demo — documented as a known limitation. Worker-level HA is fully functional (backend DaemonSet + MinIO erasure coding).
Worker RAM / co-location: Co-located backend + MinIO on all 8 workers. Stable in practice with the ~900 MB Pi 3s — no OOM issues observed during normal operation.
MinIO parity: EC:4 chosen — tolerates up to 4 worker failures, better failure demo story. 50% usable capacity is fine with 32 GB SD cards.
Existing data: Started MinIO fresh. The evidence bucket is auto-created by the backend on first event. Old events keep their legacy :9000 image URLs.
Timing: k3s installed alongside the running MPI/monitoring stack with no conflicts. k3s uses bundled containerd; Docker Compose monitoring continues independently.
Fallback: Not needed — k3s setup succeeded. Path A (k3s + distributed MinIO) is live.

10. Rollback plan¶

Everything is reversible with no impact on Tasks 2–5:

k3s-uninstall.sh (master) and k3s-agent-uninstall.sh (workers) remove k3s entirely and restore the original iptables.
The monitoring Compose stack, MPI, NFS, and dnsmasq are never modified.
Postgres stays where it is throughout.

11. Implementation steps (completed)¶

~~Authorized a shared SSH key across the nodes.~~
~~Installed k3s server on master; joined 8 agents. kubectl get nodes = 9 Ready.~~
~~Stood up local registry on master (192.168.137.10:5000); containerized backend + frontend; pushed images.~~
~~Deployed distributed MinIO (8-node StatefulSet, EC:4). evidence bucket auto-created.~~
~~Deployed backend as DaemonSet (one pod per node, 9 total) behind a Service + Traefik. Deployed frontend as a Deployment pinned to the master.~~
~~Frontend uses relative URLs (/api, /prom) via the ingress. End-to-end verified at http://192.168.137.10/.~~

All steps complete. The system is live and serving the full Sentinel dashboard.