SS2026 Edge Threat Monitoring — System Documentation & Test Guide¶

Project: Cloud Computing (SS2026) — edge-AI threat monitoring on a Raspberry Pi cluster Product name in the UI: Sentinel — Edge Threat Monitoring Status: Built and running. Reachable at http://192.168.137.10/

This document explains how the whole system works (every moving part and how data flows through it) and how to test that the cluster / distributed system is actually working — including deliberate failure tests. For why we chose this deployment (trade-offs, alternatives, open questions), see the companion task7-deployment-architecture.md.

1. What the system does (in one paragraph)¶

A sensor node (Pi 4 + AI Camera) runs an object/threat-detection model on a live camera feed. When it detects something, it POSTs an event (metadata + a snapshot image) to a backend running on the Pi cluster. The backend stores the metadata in PostgreSQL and the image in a distributed MinIO object store, and fires a Telegram alert. A React dashboard ("Sentinel") reads those events over REST and shows live detections, an events table, evidence images, cluster health (from Prometheus/Grafana), the live k3s topology, HPC results (HPL benchmark with live re-run, MPI scalability benchmarks, and Amdahl/Gustafson scaling with a live experiment runner). The frontend supports light and dark themes with a sticky navigation layout. The backend, frontend, and storage all run as Docker containers on a k3s Kubernetes cluster spread across 9 Raspberry Pis, behind a single Traefik ingress.

2. Hardware inventory¶

Node	Address	Hardware	Role in the system
master	`192.168.137.10`	Pi 5, 8 GB, + AI HAT+	k3s server (control plane), PostgreSQL, frontend pod, a backend pod, Traefik ingress, local image registry, Prometheus/Grafana stack, DHCP/DNS (`dnsmasq`), NFS export
worker1–8	`.101`–`.108`	Pi 3, ~0.9 GB each	k3s agents: one backend pod + one MinIO pod each (the distributed tier)
sensor	`192.168.137.20`	Pi 4 + AI Camera	runs the detection model + live MJPEG stream; POSTs events to the master
(client / gateway)	`192.168.137.1`	Your laptop (macOS or Windows)	views the dashboard; acts as the cluster's internet gateway at `.1` — macOS Internet Sharing or Windows ICS (see §6 and the Windows guide)

All nodes are aarch64 / 64-bit Debian 13, so they run standard ARM container images.

3. Architecture at a glance (as built)¶

flowchart TB
  user["You (browser / Mac)"] -->|"HTTP :80"| traefik
  sensor["Sensor (Pi4 + AI camera)"] -->|"POST /events :8080"| bemaster

  subgraph master["master — Pi 5 (control plane)"]
    traefik["Traefik ingress :80"]
    fe["Frontend pod (nginx + React)"]
    bemaster["Backend pod"]
    pg[("PostgreSQL :5432 — event metadata")]
    prom[("Prometheus :9090 / Grafana :3000")]
    reg[["registry :5000"]]
    traefik -->|"/"| fe
    traefik -->|"/api"| besvc["Backend Service (load-balanced)"]
    traefik -->|"/prom"| prom
    traefik -->|"/evidence"| miniosvc["MinIO Service"]
  end

  subgraph workers["worker1–8 — Pi 3 (distributed tier)"]
    bw["Backend pod ×8"]
    mw["MinIO pod ×8 (EC:4)"]
  end

  besvc --> bemaster
  besvc --> bw
  bemaster -->|"metadata"| pg
  bemaster -->|"S3 images"| miniosvc
  miniosvc --> mw
  mw <-->|"erasure-coded shards across all 8"| mw

One sentence per layer:

Ingress (Traefik, master:80): the single front door. Routes / → frontend, /api → backend, /evidence → MinIO images, /prom → Prometheus.
Backend (DaemonSet): one Flask replica on every node (master + 8 workers), load-balanced by a Kubernetes Service. Stateless.
PostgreSQL (master): relational store for event metadata (id, time, type, confidence, status, image key…).
MinIO (StatefulSet, 8 workers): S3-compatible object store for the evidence images, erasure-coded across all 8 workers (the "distributed file system").
Frontend (master): the React single-page app, served by nginx.

4. End-to-end data flow¶

4a. Detection → storage → alert (write path)¶

The sensor detects an object and POSTs JSON to http://192.168.137.10:8080/events (the backend pod on the master, via hostPort 8080). Payload includes sensor_id, location, threat_level, detections[] (each with class/score/box), and a base64 snapshot_b64 image.
The backend (server.py → event_store.store_event):
uploads the image to MinIO (bucket evidence, key like 2026/06/29/HHMMSS_sensor.jpg) — sharded across the 8 workers;
inserts a row into PostgreSQL with the metadata + the image URL;
sends a Telegram message (if the bot token is configured).
The dashboard's "active threats" and "beep" react to the new event on its next poll.

4b. Dashboard (read path)¶

The SPA calls /api/events, /api/events/:id, /api/stores (same-origin, via the ingress → backend Service, load-balanced across all 9 backend pods).
Evidence images load from /evidence/<key> (ingress → MinIO).
Cluster health comes from /prom/api/v1/query?... (ingress → Prometheus).

Why this is robust: every read is served by any backend pod (the Service picks one), and images are reconstructed from erasure shards on any 4-of-8 workers. No single worker is required for the dashboard to work.

5. Components in detail¶

5.1 Sensor (`sensor/`)¶

detect_stream.py / start_stream.py — detection model + live MJPEG stream.
Pushes events to the backend over REST. Not containerized (runs directly on the Pi 4).

5.2 Backend (`master/server/`, image `sentinel-backend`)¶

Flask app, deployed as a DaemonSet (deploy/k8s/20-backend.yaml) — one pod per node, hostPort 8080, behind a Service. Stateless: all state is in Postgres + MinIO.

Method & path	Purpose
`POST /events`	Ingest a detection from the sensor (writes Postgres + MinIO + Telegram). Returns `{"status":"ignored"}` when detection is paused.
`GET /health`	Liveness/readiness probe (`{"status":"running"}`)
`GET /api/events?limit=N`	List events (dashboard)
`GET /api/events/<id>`	One event
`POST /api/events/<id>/status`	Acknowledge / resolve an event
`DELETE /api/events/<id>`	Delete an event row and its MinIO image (the "Delete event" button)
`GET /api/detection`	Read the global detection on/off switch (`{"enabled":true\\|false}`)
`POST /api/detection`	Start / stop detection cluster-wide (`{"enabled":bool}`)
`GET /api/stores`	Store health probe (`{"postgres":true,"minio":true}`)

Telegram alerts include the snapshot image (sent via sendPhoto), not just text. Detection on/off is a shared switch persisted in a Postgres app_settings table — so it survives restarts and reads identically across all 9 backend pods (a short in-memory cache avoids a DB hit per request). The sensor polls GET /api/detection; when off it keeps streaming MJPEG but stops running the model / posting events and overlays "DETECTION PAUSED".

Key env (set in the manifest / backend-secrets): PGHOST=192.168.137.10, MINIO_ENDPOINT=minio.sentinel.svc.cluster.local:9000, MINIO_PUBLIC_URL=http://192.168.137.10 (so image URLs go through the ingress), MINIO_BUCKET=evidence, BOT_TELEGRAM_TOKEN/CHAT_ID.

5.3 PostgreSQL (master, Docker Compose `master/server/docker-compose.yml`)¶

events-postgres container, events DB, table events (metadata only).
Reused by the k3s backend pods — not migrated. Demo creds: events / events_pw.

5.4 MinIO — distributed object storage (`deploy/k8s/10-minio.yaml`, image `minio`)¶

StatefulSet, 8 replicas, one drive (local-path PVC) per worker, pod anti-affinity (one per node) + node-affinity (workers only).
Erasure coding EC:4: each object = 4 data + 4 parity shards across the 8 nodes.
Reads tolerate up to 4 workers down (any 4-of-8 shards reconstruct the image) — no data loss.
Writes need a quorum of 5-of-8 drives, so new images stop being stored once 4+ workers go offline (existing images still read fine). This is exactly the failure mode behind events that have no image: restore the workers to Running and writes + auto-heal resume. (See the full failure table in the architecture doc §7.)
Bucket evidence (auto-created by the backend on first event, public-read so images load).
Demo creds: minioadmin / minioadmin123.

5.5 Frontend (`frontend/`, image `ss2026-frontend`)¶

React 19 + Vite + Tailwind SPA, served by nginx, pinned to the master.
Production build uses relative URLs (/api, /prom) so everything is same-origin behind the ingress (frontend/.env.production).
Light / dark theme toggle (sidebar + topbar) with localStorage persistence.
Sticky layout: sidebar and topbar stay fixed; only the main content area scrolls.
Pages (sidebar): Dashboard, Events (+ detail), Monitoring (Task 5), Cluster (live k3s topology), Performance (HPL benchmark + live re-run, Task 2), MPI Clustering (Monte Carlo + matrix multiplication benchmarks, Amdahl/Gustafson analysis, Task 3), Non MPI (POV-Ray scaling + live experiment runner, Task 4), Logs.

5.6 Monitoring — Task 5 (Docker stack on the master)¶

prometheus (:9090), grafana (:3000), alertmanager, blackbox exporter, and node_exporter on each node. Scrapes CPU/RAM/temp/disk + a camera probe.
The dashboard's Monitoring page reads Prometheus through the ingress at /prom.
Grafana UI: http://192.168.137.10:3000.

5.7 HPC / compute — Tasks 2–4 (`master/benchmark/`)¶

HPL (High-Performance LINPACK) benchmark → Performance page. Includes a live re-run panel (bench_server.py, port 8090) that executes HPL on-demand (single node, all 9 nodes, or worker1) and reports fresh GFLOPS + residual verdict.
MPI benchmarks (Monte Carlo π estimation, parallel matrix multiplication) → MPI Clustering page. Shows execution time, speedup, efficiency, Amdahl/Gustafson analysis, and bottleneck analysis from hardcoded PDF report data.
Amdahl's & Gustafson's law + Task-Distributor / POV-Ray scaling → Non MPI page. Includes a live experiment runner (task4_api.py, port 8081) that triggers new POV-Ray scaling runs and updates charts in real time.
These run over MPI/NFS (the original HPC cluster), independent of k3s.
Known issue: multi-node MPI runs currently fail because worker nodes lose their /etc/hosts entries on reboot and can't resolve master. Single-node runs work fine.

5.8 k3s + ingress + registry (the distribution layer)¶

k3s server on master, agents on the 8 workers (kubectl get nodes → 9 Ready).
Traefik (bundled with k3s) is the single ingress on :80 (deploy/k8s/40-ingress.yaml).
Local registry 192.168.137.10:5000 holds the 3 ARM images; every node trusts it via /etc/rancher/k3s/registries.yaml so pulls stay on the LAN.

6. Networking & how you reach it¶

What	URL	Notes
Dashboard + API + images + Prometheus	http://192.168.137.10/	single origin via Traefik
Sensor → backend ingest	`http://192.168.137.10:8080/events`	`hostPort` on the master backend pod
Grafana	`http://192.168.137.10:3000`	host service
MinIO console (optional)	`kubectl -n sentinel port-forward svc/minio 9001:9001`	not exposed publicly

Production = the cluster. The browser only ever talks to http://192.168.137.10/. Because it's one origin, the earlier macOS ERR_ADDRESS_UNREACHABLE problem is gone, and the dev-only frontend/cluster-proxy.py + .env.local workaround is not needed for the deployed app (they remain only for npm run dev against the cluster from the Mac).
From Windows (or any non-Mac): the deployed dashboard is just a URL — open http://192.168.137.10/ from any device on the cluster LAN. To make a Windows PC the cluster gateway/dev box (replacing the Mac), run npm run dev, or redeploy from Windows, follow the Windows guide in the deploy runbook.

7. How work is actually distributed¶

Compute / API: the backend DaemonSet puts a replica on all 9 nodes; the Service load-balances dashboard requests across them. A dead replica is auto-recreated.
Storage: MinIO erasure-codes every image across all 8 workers — the data itself is split and spread, not just copied. Reads/writes succeed as long as a quorum of shards is up.
HPC: HPL/POV-Ray work is split across nodes via MPI (Tasks 2–4).

Honest caveat (single point of failure): the master holds Postgres, the ingress, and the frontend. Worker failures are fully tolerated; a master outage takes the API + dashboard down (worker data survives but is unreachable until it returns). This is the documented demo trade-off — see §6/§9 of the architecture doc.

8. Task status map¶

Task	What	Status	Where to see it
1	Infrastructure (manual OS install, networking)	Done	this doc §2
2	HPL benchmark	Done	Performance page (results + live re-run)
3	MPI scalability	Done	MPI Clustering page (Monte Carlo, matrix mult, Amdahl/Gustafson)
4	Amdahl / Gustafson + Task Distributor	Done	Non MPI page (POV-Ray scaling + live experiment runner)
5	Monitoring (Prometheus/Grafana)	Done	Monitoring page + Grafana
6	Threat detection → DB + storage + bot	Done	Events / Telegram
7	Backend on k3s, HA, distributed FS	Done	Cluster page (live k3s topology)
8	Frontend on k3s	Done	http://192.168.137.10/
9	REST/MQTT comms	Done (REST everywhere)	§4 data flow
10	Docs / poster / demo	Done	this doc + deploy runbook + frontend README

9. Testing & verification¶

Before you start: SSH to the master and point kubectl at the cluster config.
ssh master@192.168.137.10            # password: 12345678
export KUBECONFIG=$HOME/.kube/config # (add to ~/.bashrc to make permanent)
Commands prefixed “(Mac)” run from your laptop instead.

A. 30-second smoke test (Mac)¶

curl -s -o /dev/null -w "UI        %{http_code}\n" http://192.168.137.10/
curl -s -w  "stores    "            http://192.168.137.10/api/stores; echo
curl -s -o /dev/null -w "prom up   %{http_code}\n" "http://192.168.137.10/prom/api/v1/query?query=up"

Expected: UI 200, stores {"postgres":true,"minio":true}, prom up 200.

B. Is the k3s cluster healthy?¶

kubectl get nodes -o wide
kubectl -n sentinel get pods -o wide

Pass = 9 nodes Ready; minio-0..7 Running (one per worker), backend-* Running on all 9, frontend-* Running on master.

C. Is the backend distributed + highly available?¶

# One replica per node (DaemonSet), and the Service load-balances across all of them:
kubectl -n sentinel get ds backend
kubectl -n sentinel get endpoints backend     # should list ~9 pod IPs

# HA test — kill a worker's backend pod; the API must stay up and the pod must respawn:
kubectl -n sentinel delete pod -l app=backend --field-selector spec.nodeName=worker3
curl -s http://192.168.137.10/api/stores       # still {"postgres":true,"minio":true}
kubectl -n sentinel get pods -l app=backend -o wide   # worker3 pod recreated within seconds

D. Is storage distributed + fault-tolerant?¶

# 8 drives, one per worker:
kubectl -n sentinel get pods -l app=minio -o wide

# Cluster has write-quorum (200 = healthy). port-forward avoids needing a client:
kubectl -n sentinel port-forward svc/minio 9000:9000 >/dev/null 2>&1 &
sleep 2; curl -s -o /dev/null -w "minio cluster health %{http_code}\n" \
  http://localhost:9000/minio/health/cluster
kill %1 2>/dev/null

Pass = 8 MinIO pods Running on 8 distinct workers; cluster health 200. (8/8 pods 1/1 Ready already implies erasure-set quorum.)

E. Is the ingress (single origin) routing correctly?¶

for p in /  /api/stores  "/api/events?limit=1"  "/prom/api/v1/query?query=up" ; do
  printf "%-28s " "$p"; curl -s -o /dev/null -w "%{http_code}\n" "http://192.168.137.10$p"
done
# /evidence/ returns 403 (bucket listing denied) — that's expected; object GETs are public.

F. End-to-end pipeline test (the convincing one)¶

Best: trigger a real detection — show the target object to the sensor camera, then watch a new row appear in the Events page with its evidence image loading.

No camera? Inject a synthetic event (writes through backend → distributed MinIO + Postgres):

# (Mac or master) post a fake detection as if from the sensor
python3 - <<'PY'
import base64, json, urllib.request
payload = {
  "sensor_id": "test", "location": "test-bench", "threat_level": "low",
  "detections": [{"class": "test", "score": 0.99, "box": [0,0,10,10]}],
  "snapshot_b64": base64.b64encode(b"SENTINEL-STORAGE-TEST").decode(),
}
req = urllib.request.Request("http://192.168.137.10:8080/events",
    data=json.dumps(payload).encode(), headers={"Content-Type": "application/json"})
print(urllib.request.urlopen(req, timeout=10).read().decode())
PY

# read it back + confirm the image is served from distributed MinIO via the ingress
curl -s "http://192.168.137.10/api/events?limit=1"        # note the image_url (/evidence/...)
# copy the image_key from above into KEY, then:
KEY="2026/06/29/REPLACE_ME.jpg"
curl -s -o /dev/null -w "evidence GET %{http_code}\n" "http://192.168.137.10/evidence/$KEY"

Pass = the POST returns {"status":"received",...}, the event appears in /api/events with an image_url under /evidence/..., and that URL returns 200.

Cleanup (optional): remove the test row — docker exec events-postgres psql -U events -d events -c "DELETE FROM events WHERE sensor_id='test';"

G. Failure-injection: kill a whole worker (the headline HA demo)¶

KEY="<image_key from test F>"
kubectl drain worker7 --ignore-daemonsets --delete-emptydir-data --force
#   - backend pod on worker7 leaves the Service → API still served by the other 8
#   - MinIO pod on worker7 goes offline → EC:4 reconstructs images from the remaining shards
curl -s http://192.168.137.10/api/stores                              # still true/true
curl -s -o /dev/null -w "image during outage %{http_code}\n" "http://192.168.137.10/evidence/$KEY"  # 200
kubectl get nodes                                                     # worker7 Ready,SchedulingDisabled

# restore — MinIO auto-heals the missing shards, pods reschedule:
kubectl uncordon worker7
kubectl -n sentinel get pods -o wide

Note: the drained worker's MinIO pod stays Pending until you uncordon, because its drive is local to that node (storage is pinned to nodes; redundancy comes from erasure coding across nodes, not from moving data). The system stays fully usable the whole time — that is the distributed-storage guarantee.

H. Monitoring (Task 5)¶

# which Prometheus targets are up?
curl -s "http://192.168.137.10/prom/api/v1/query?query=up" \
  | python3 -c 'import sys,json;[print(r["metric"].get("job"),r["metric"].get("instance"),r["value"][1]) for r in json.load(sys.stdin)["data"]["result"]]'

Then open Grafana at http://192.168.137.10:3000. (If the dashboard shows fewer "nodes online" than expected, a node_exporter likely didn't restart after a reboot — restart it on the affected node.)

I. Compute distribution (Tasks 2–4)¶

Re-running the HPL benchmark / POV-Ray task across MPI nodes is the compute-side distribution test; results render on the Performance and Scaling pages.

10. Operations runbook¶

Where things live - k8s manifests: deploy/k8s/*.yaml · registry trust: deploy/registries.yaml · runbook: deploy/README.md · images: 192.168.137.10:5000 · build contexts on master: ~/build/{backend,frontend} · manifests on master: ~/deploy/k8s.

Redeploy after a code change

# backend: (Mac) ship context -> (master) build + push -> rollout
tar czf - -C master/server . | ssh master@192.168.137.10 'tar xzf - -C ~/build/backend'
ssh master@192.168.137.10 'cd ~/build/backend && docker build -t localhost:5000/sentinel-backend:latest . && docker push localhost:5000/sentinel-backend:latest && KUBECONFIG=~/.kube/config kubectl -n sentinel rollout restart ds/backend'

# frontend: (Mac) build SPA, ship Dockerfile+nginx then dist, build + push -> rollout
( cd frontend && mv .env.local .env.local.bak 2>/dev/null; npm run build; mv .env.local.bak .env.local 2>/dev/null )
tar czf - -C deploy/frontend Dockerfile nginx.conf | ssh master@192.168.137.10 'rm -rf ~/build/frontend && mkdir -p ~/build/frontend && tar xzf - -C ~/build/frontend'
tar czf - -C frontend dist | ssh master@192.168.137.10 'tar xzf - -C ~/build/frontend'
ssh master@192.168.137.10 'cd ~/build/frontend && docker build -t localhost:5000/sentinel-frontend:latest . && docker push localhost:5000/sentinel-frontend:latest && KUBECONFIG=~/.kube/config kubectl -n sentinel rollout restart deploy/frontend'

Push target: building on the master and pushing to localhost:5000 (as above) needs no extra config — Docker always trusts localhost. If you instead tag with the IP 192.168.137.10:5000, that registry is plain-HTTP, so the daemon needs /etc/docker/daemon.json → {"insecure-registries":["192.168.137.10:5000"]} + a restart, and newer Docker/BuildKit needs docker build --provenance=false --sbom=false (otherwise it pushes an OCI image index that k3s can't pull).

:latest rollout gotcha: since the tag is :latest, a rollout restart can leave some nodes on a stale cached digest. If kubectl -n sentinel get pods -l app=backend -o wide plus a digest check shows pods disagreeing, pin the exact digest printed by docker push:
kubectl -n sentinel set image ds/backend backend=192.168.137.10:5000/sentinel-backend@sha256:<digest>

Restart / inspect

kubectl -n sentinel rollout restart ds/backend          # restart all backends
kubectl -n sentinel logs -l app=backend --tail=50       # backend logs
kubectl -n sentinel logs minio-0 --tail=50              # a storage node's logs
kubectl -n sentinel describe pod <pod>                  # why is a pod not starting?

Host helper services on the master (NOT in k3s — relaunch after a master reboot) These are plain Python processes reached directly or via the ingress. They do not auto-start yet:

ssh master@192.168.137.10 'bash ~/bench/run_bench_server.sh'        # HPL benchmark     (:8090, feeds Performance page)
ssh master@192.168.137.10 'bash ~/cluster/run_cluster_server.sh'    # k3s topology      (:8091, feeds Cluster page)
ssh master@192.168.137.10 'python3 ~/task4/task4_api.py &'          # Task 4 scaling    (:8081, feeds Non MPI page)

Tear down k3s (reversible; never touches Postgres/monitoring/MPI/NFS)

# workers:  sudo /usr/local/bin/k3s-agent-uninstall.sh
# master:   sudo /usr/local/bin/k3s-uninstall.sh

Credentials (demo defaults): Postgres events/events_pw · MinIO minioadmin/minioadmin123 · node login <user>/12345678. Telegram bot token lives in the backend-secrets Secret (sourced from ~/server/server.env).

11. Known limitations / follow-ups¶

Master is a single point of failure (control plane + Postgres + ingress + frontend). Acceptable for the demo; a 3-server HA control plane + Postgres replication would remove it.
Sensor ingest targets the master's :8080 only; adding an ingress route for /events would load-balance writes too.
Old events keep their original :9000 image URLs (served by the legacy host MinIO); only new events use the distributed store. A one-off migration could unify them.
node_exporter on one or two nodes may need a restart after a reboot for the Monitoring page to show full node count.
Multi-node MPI runs currently fail because /etc/hosts on the workers resets on reboot, so they can't resolve master. Single-node HPL runs work fine. Fix: add persistent hostname entries or set up proper DNS.
Host helper services (benchmark server, cluster topology server, Task 4 API) must be manually restarted after a master reboot — they are not managed by k3s or systemd.