local-ci-cd-system/plans/web-ui-redesign-plan.md

# Web UI Redesign — Implementation Plan (Phase 1)

**Status:** Stages A, B, C complete + gates approved (989 tests pass, cov 94.12%, ruff+mypy clean). Stage D (service management, SECURITY-CRITICAL) not started. Uncommitted.
**Owner / executor:** Sonnet (implementation) under Opus supervision (review gate after every stage).
**Scope:** Presentation redesign + new read-only panels + service management. **No** time-series store, **no** light theme, **no** rebuild of Gitea job/log views.
**Target files root:** `src/ci_orchestrator/web/`, `src/ci_orchestrator/operations/`.

This plan is written to be executed by a Sonnet-class model. Each stage is
self-contained, ends with a hard **STOP — supervisor review** gate, and lists
exact files, signatures, tests, and acceptance criteria. Do not start a stage
before the previous stage's gate is approved. Keep the checklist below current
(tick `- [x]` as tasks land); the supervisor or a subagent updates it.

---

## Progress checklist (keep this updated)

Legend: `[ ]` todo · `[~]` in progress · `[x]` done · `[!]` blocked

### Stage A — Restyle foundation + bug fixes
- [x] A1. Swap CSS tokens to real Gitea dark palette (accent → blue `#4183c4`)
- [x] A2. `web/filters.py` — number/format Jinja filters + registration in `app.py`
- [x] A3. Status header partial (`partials/status_header.html`) + health thresholds
- [x] A4. Fix unthemed action pages (htmx swap + full-page `HX-Request` fallback)
- [x] A5. Fix polling-wipes-form-input bug (`hx-preserve` on inputs)
- [x] A6. Inline SVG icon sprite (`partials/icons.html`) + apply to card headers
- [x] A7. Stage A tests green · ruff · mypy --strict · coverage ≥90
- [x] **A-GATE — supervisor review** (Opus: gates green 895 pass / cov 94.15% / ruff+mypy clean; theming fix + action-slot verified)

### Stage B — Read-only panels
- [x] B1. `operations/host.py` — `host_resources()` (/proc, 2-sample CPU%)
- [x] B2. `operations/host.py` — `vmware_module_health()` (lsmod + vmware-host.service)
- [x] B3. `operations/inventory.py` — `list_templates()` (+ `.vmem` health, last backup)
- [x] B4. `operations/inventory.py` — `list_backups()` / `list_artifacts()` (path-contained)
- [x] B5. GET partials + routes: `/resources`, `/vmware`, `/templates`, `/files`
- [x] B6. Backup card → template **dropdown** (validated against `list_templates`)
- [x] B7. Slow-poll / on-demand refresh for expensive panels (dir-size cache)
- [x] B8. Stage B tests green · ruff · mypy --strict · coverage ≥90
- [x] **B-GATE — supervisor review** (Opus: gates green 958 pass / cov 94.10% / ruff+mypy clean; containment + 2-sample CPU + backup-name 400 verified)

### Stage C — VM enrichment + Gitea link-out + audit polish
- [x] C1. `parse_clone_name()` + `repo/job` (job_id + started) column in VM table
- [x] C2. Gitea base URL in config/context; nav `Gitea ↗` + contextual links
- [x] C3. Audit page restyle (relative time, action icons, simple filter)
- [x] C4. Stage C tests green · ruff · mypy --strict · coverage ≥90
- [x] **C-GATE — supervisor review** (Opus: gates green 989 pass / cov 94.12% / ruff+mypy clean; subagent cut off mid-C2 by session limit — Opus finished wiring + fixed 2 left-over test bugs: empty-VM header assert, OIDC keyring stub)

### Stage D — Service management (SECURITY-CRITICAL, last)
- [ ] D1. `operations/services.py` — allow-list, `service_states()`, `control_service()`
- [ ] D2. POST `/services/{service_id}/{action}` — confirm + CSRF + audit + refusals
- [ ] D3. `ci-web` self-restart via detached `systemd-run` (response returns first)
- [ ] D4. Services panel partial (state-aware buttons, active-VM warning on stop)
- [ ] D5. `sudoers.d/ci-web` (exact-command NOPASSWD) + installer `visudo -c` validate
- [ ] D6. Config `[web].managed_services` override + `config.example.toml` + `WEB-UI.md`
- [ ] D7. Security regression tests (unknown id → 400 & systemctl never called; etc.)
- [ ] D8. Stage D tests green · ruff · mypy --strict · coverage ≥90
- [ ] **D-GATE — supervisor review + deploy note**

---

## Locked design decisions

| Decision | Value |
|---|---|
| Theme | **Dark only**, mirroring Gitea `gitea-auto` dark values |
| CSS approach | Hand-written, extend existing tokens. **No framework, no CDN** (LAN offline) |
| Data over time | **None** — all panels point-in-time snapshots |
| Job history / logs | Not rebuilt — link out to Gitea |
| New runtime dep | **None** — host metrics via `/proc`, not psutil |
| Service control | Full start/stop/restart over a **fixed allow-list** |
| Read routes | `web/app.py` · Mutating routes | `web/routes.py` (existing split, keep it) |

### Real Gitea dark palette (source: `gitea.emulab.it` theme `gitea-auto`)

```
--color-primary        #4183c4   (accent — replaces green #3fb950)
--color-primary-contrast #ffffff
--color-body           #1e1f20   (page bg)
--color-box-body       #161718   (card bg)
--color-box-header     #1b1c1e   (card header bg)
--color-nav-bg         #18191b   (navbar bg)
--color-secondary-bg   #2e3033   (input / subtle panel)
--color-text           #d2d4d8
--color-text-light     #c0c2c7   (dim)
--color-secondary      #3f4248   (border-ish)
--color-success-text   #3fb950   --color-error-text #f85149   --color-warning-text #d29922
--color-info-text      #2f81f7
```

### Health thresholds (status header + row colors)
- Disk `used_pct` > 85 → **red**, > 70 → **amber**, else green.
- Any runner unit not `active` → **red**.
- Orphan VM present → **amber**.
- VMware `vmmon` or `vmnet` not loaded, or `vmware-host.service` not active → **red**.

---

## Architecture changes (file map)

New files:
- `web/filters.py` — Jinja formatting filters (pure functions, unit-tested).
- `operations/services.py` — managed-service allow-list + status + control.
- `operations/inventory.py` — template discovery + directory listing (path-contained).
- `web/templates/partials/`: `status_header.html`, `resources.html`, `vmware.html`,
  `templates.html`, `files.html`, `services.html`, `icons.html`.

Edited files:
- `web/static/app.css` — token swap + new components (gauge, stat tile, service controls).
- `web/app.py` — register filters; new GET routes/partials; status-header context.
- `web/routes.py` — service-control POST; backup route → dropdown validation.
- `operations/host.py` — add `host_resources()`, `vmware_module_health()`.
- `web/templates/base.html` — nav `Gitea ↗`; `#action-slot`.
- `web/templates/partials/{vms,cache,host,audit,confirm,result}.html` — restyle + fixes.
- `config.py` / `config.example.toml` — `[web].gitea_url`, `[web].managed_services`.
- `docs/WEB-UI.md` — sudoers, service panel, new panels.

**Isolation rule (why the split):** `operations/host.py` already carries
disk+runner+backup. Adding resources+modules+templates+files+service-control
would bloat it past what fits in one reviewer's head. Keep host-level probes in
`host.py`, inventory/listing in `inventory.py`, service control in
`services.py`. Each is independently testable and imported by both CLI and web.

---

## Stage A — Restyle foundation + bug fixes

**Goal:** the dashboard *looks* professional and the two known UI bugs die.
No new data sources. This stage alone fixes "look grezzo / colpo d'occhio /
layout piatto" and the unthemed-action-page complaint.

### A1 — CSS tokens
Edit `web/static/app.css` `:root`: replace the palette with the Gitea dark
values above (accent `#4183c4`; `--bg`→`#1e1f20`; `--panel`→`#161718`;
`--panel-2`→`#2e3033`; `--border`→`#3f4248`; `--text`→`#d2d4d8`;
`--text-dim`→`#c0c2c7`; `--accent`→`#4183c4`; `--accent-hover` a lighter blue;
keep `--danger`/`--warn` as-is). Update `.btn-primary` contrast text to
`#ffffff`. No structural HTML change in A1.

### A2 — Jinja filters (`web/filters.py`)
Pure functions + a `register(templates: Jinja2Templates)` helper called from
`app.py` `build_app`. Signatures:
```python
def humanbytes(n: int | float) -> str          # 12_345_678 -> "11.8 MB"
def duration(seconds: int | float | None) -> str # 3783 -> "1h 03m"; None -> "-"
def relative_time(value: str | datetime | None) -> str  # "2m ago"; None -> "-"
def localtime(value: str | datetime | None) -> str      # ISO -> host-local "YYYY-MM-DD HH:MM"
def shorten_path(p: str | Path, keep: int = 2) -> str    # ".../Templates/LinuxBuild2404"
def thousands(n: int | float) -> str            # 812 -> "812"; 12345 -> "12,345"
```
Apply in templates: `cache.html` (`total_bytes|humanbytes`, `entry_count|thousands`),
`vms.html` (`uptime_sec|duration`), `host.html` (disk GB formatting tidy),
`audit.html` (`ts|relative_time` with `ts|localtime` in `title=`).

### A3 — Status header (`partials/status_header.html`)
A row of stat tiles fed from the existing dashboard context (no new fetch):
VM count (+orphan count), runner `active/total`, disk `used_pct` gauge, cache
size. Apply health-threshold classes. Included at top of `dashboard.html`,
above `.dashboard-grid`. Add a `_health_summary(...)` helper in `app.py` that
derives the tile values/colors from already-fetched context (keep it pure).

### A4 — Fix unthemed action pages
Root cause: mutating `<form method="post">` do full-page navigation and the
routes return bare partials (`confirm/result/lock_conflict.html`) with no
`base.html` wrapper. Fix both paths:
1. Convert the mutating forms to **htmx** (`hx-post`, `hx-target="#action-slot"`,
   `hx-swap="innerHTML"`). Confirm→execute stays two-step but renders inline.
2. In the render helpers (`_confirm_partial`, `_result_partial`,
   `_lock_conflict_partial`), when `request.headers.get("HX-Request")` is
   **absent**, render the partial wrapped in a themed full page (a new
   `action_page.html` that `extends base.html` and drops the partial into
   `#action-slot`). This keeps no-JS browsers themed too.
Add `<div id="action-slot"></div>` near the top of `base.html` content block.

### A5 — Polling wipes form input
`vms.html` cleanup form (`max_age_hours`) and the backup form live inside cards
that self-replace every 5s via `hx-swap="outerHTML"` → typed values vanish.
Add `hx-preserve="true"` + stable `id` to those inputs, **or** move the forms
into a non-polled sibling container. Prefer `hx-preserve` (smaller diff).

### A6 — Icon sprite (`partials/icons.html`)
One inline SVG `<symbol>` sprite (server, disk, cpu, box/template, gear/service,
file, external-link, check, alert). Reference via `<svg class="icon"><use href="#i-…"></use></svg>`.
No external requests. Add `.icon` sizing to CSS.

### A7 — Verify
```
python -m pytest tests/python -q --cov=ci_orchestrator --cov-fail-under=90
python -m ruff check src tests/python
python -m mypy --strict src
```
New tests: `tests/python/test_web_filters.py` (each filter, incl. `None`/edge),
`test_web_theming.py` (HX-Request present → bare partial; absent → contains
`<link rel="stylesheet"`).

### A-GATE
STOP. Show the supervisor the rendered dashboard (describe or screenshot) and
the green test/lint/type output. Do not proceed to Stage B until approved.

---

## Stage B — Read-only panels

**Goal:** four new information panels. All degrade to an "unavailable" notice on
any error (never 500), matching the existing `_fetch_*` pattern in `app.py`.

### B1 — `host_resources()` (`operations/host.py`)
```python
@dataclass(frozen=True)
class HostResources:
    cpu_pct: float          # 0..100
    ncpu: int
    mem_total_gb: float
    mem_used_gb: float
    uptime_sec: int
    load1: float
    load5: float
    load15: float

def host_resources(*, cpu_sample_ms: int = 150) -> HostResources: ...
```
CPU% **requires two `/proc/stat` samples** `cpu_sample_ms` apart; compute
`1 - idle_delta/total_delta`. Read `/proc/meminfo` (MemTotal, MemAvailable →
used = total-available), `/proc/uptime` (field 0), `/proc/loadavg`,
`os.cpu_count()`. Degrade to zeros on `OSError` (never raise).

### B2 — `vmware_module_health()` (`operations/host.py`)
```python
@dataclass(frozen=True)
class VmwareModuleHealth:
    vmmon_loaded: bool
    vmnet_loaded: bool
    host_service_active: bool
    host_service_sub_state: str
```
`vmmon`/`vmnet` via reading `/proc/modules` (dependency-free; no `lsmod`
subprocess needed) or `lsmod` if simpler — prefer `/proc/modules`. Service
state reuses the existing `runner_status(["vmware-host.service"])` machinery.
Ties to the documented "VMs won't start" gotcha.

### B3 — `list_templates()` (`operations/inventory.py`)
```python
@dataclass(frozen=True)
class TemplateInfo:
    name: str
    vmx: Path | None
    guest_os: str | None       # parsed from vmx guestOS
    snapshot: str | None       # best-effort; may be None
    transport: str             # "ssh" | "winrm" | "unknown" (from guest_os)
    dir_bytes: int
    has_vmem: bool             # True -> snapshot captured running RAM (warn)
    last_backup: FileEntry | None

def list_templates(config: Config) -> list[TemplateInfo]: ...
```
Scan `config.paths.templates` subdirs. `guest_os` from the `.vmx` `guestOS`
line; `transport` = ssh if `ubuntu`/`linux` else winrm. `has_vmem` = any
`*.vmem` in the dir. `dir_bytes` via `os.walk` sum (**cached** — see B7).
`last_backup` = newest `Template_<name>_*.7z` under `<root>/backups`.

### B4 — Directory listings (`operations/inventory.py`)
```python
@dataclass(frozen=True)
class FileEntry:
    name: str
    size_bytes: int
    mtime: float

def list_dir(base: Path, *, root: Path, pattern: str = "*") -> list[FileEntry]: ...
def list_backups(config: Config) -> list[FileEntry]: ...
def list_artifacts(config: Config) -> list[FileEntry]: ...
```
**Containment (security):** `list_dir` must resolve `base` and assert it is
`root` or under `root` (`base.resolve().is_relative_to(root.resolve())`); never
follow symlinks that escape `root`; return `[]` if `base` missing. No user input
reaches these in Phase 1 (fixed dirs), but enforce anyway.

### B5 — Routes + partials
In `app.py` add GET routes (behind `require_user`), each rendering its partial:
`/resources`→`resources.html`, `/vmware`→`vmware.html`,
`/templates`→`templates.html`, `/files`→`files.html`. Add matching `_fetch_*`
wrappers (try/except → `None`/`[]`). Include panels in `dashboard.html`.

### B6 — Backup dropdown
Replace the free-text `template` input in the backup form with a `<select>`
populated from `list_templates()` names. In `routes.py` `host_backup`, validate
the submitted `template` is in the discovered set → else 400 (no arbitrary path
to `backup_template`). Add a one-line description above the control ("7z archive
of the template dir under `<root>/backups`").

### B7 — Poll cadence / caching
`resources` may poll 5s (cheap). `vmware` poll 15s. `templates`/`files` poll 30s
**or** render once with a manual "Refresh" button (`hx-get` on click). Template
`dir_bytes` (`os.walk` over multi-GB dirs) must be **cached** with a short TTL
(e.g. module-level `{path: (mtime_ns, bytes, ts)}`, recompute if older than 60s)
so a 30s poll doesn't walk the tree every time.

### B8 — Verify (same commands as A7)
New tests: `test_operations_inventory.py` (tmpdir templates incl. `.vmem`
present/absent, containment reject `../escape` and out-of-root symlink),
`test_host_resources.py` (feed fixture `/proc` strings via a small reader seam;
two-sample CPU math). Coverage ≥90.

### B-GATE
STOP for supervisor review.

---

## Stage C — VM enrichment + Gitea link-out + audit polish

### C1 — Clone-name parsing
```python
@dataclass(frozen=True)
class ParsedClone:
    job_id: str | None
    started: datetime | None

def parse_clone_name(name: str) -> ParsedClone: ...
```
Format is `Clone_<job_id>_<YYYYmmdd_HHMMSS>_<suffix>` (see
`commands/job.py::_make_clone_name`). Repo is **not** in the name — do not
invent it. VM table gains a `Job` column showing `job_id` (+ `started` relative
time in a tooltip). Put the helper in `operations/vms.py` (shared) and expose
`ParsedClone` on the render-ready `LiveVm` or compute in the view layer.

### C2 — Gitea link-out
Config: add optional `[web].gitea_url`; default = origin of `[web.oidc].issuer`
(`https://gitea.emulab.it`). Expose in template context. `base.html` nav gets
`Gitea ↗` (opens new tab, `rel="noopener"`). If `job_id` looks like a Gitea run
id, link the Job cell to `<gitea_url>` (top-level; do **not** fabricate a
run-URL path unless the exact Actions URL scheme is confirmed with the
supervisor first).

### C3 — Audit page polish
Restyle `audit.html`: `ts|relative_time` (full ts in `title`), an action icon
per row (reuse sprite), result badge (already present), and a simple
client-side text filter over rows (a text input + tiny inline JS, or an
`hx-get` with a query param — prefer no-JS `hx-get?q=` if cheap). "Basta un bel
log": keep it clean, monospace targets, zebra rows optional.

### C4 — Verify + **C-GATE** (STOP).

---

## Stage D — Service management (SECURITY-CRITICAL — do last)

Treat this stage with the same rigor as the existing mutating routes. Read the
`routes.py` module docstring first; mirror its confirm-gate + CSRF + audit shape
exactly.

### D1 — `operations/services.py`
```python
@dataclass(frozen=True)
class ManagedService:
    id: str            # stable slug, e.g. "act-runner"
    unit: str          # "act-runner.service"
    label: str
    allow_stop: bool   # ci-web -> False (restart only; stop would lock out the UI)

MANAGED_SERVICES: tuple[ManagedService, ...] = (
    ManagedService("act-runner", "act-runner.service", "Runner", True),
    ManagedService("act-runner-rust", "act-runner-rust.service", "Runner (Rust)", True),
    ManagedService("vmware-host", "vmware-host.service", "VMware host", True),
    ManagedService("ci-web", "ci-web.service", "Web UI", False),
)
ALLOWED_ACTIONS = ("start", "stop", "restart")

@dataclass(frozen=True)
class ServiceState:
    service: ManagedService
    active: bool
    sub_state: str

@dataclass(frozen=True)
class ServiceActionResult:
    service_id: str
    action: str
    ok: bool
    detail: str

def find_service(service_id: str) -> ManagedService | None: ...
def service_states(services: tuple[ManagedService, ...] = MANAGED_SERVICES) -> list[ServiceState]: ...
def control_service(service: ManagedService, action: str, *,
                    dry_run: bool = False) -> ServiceActionResult: ...
```
`control_service` rules:
- `action` **must** be in `ALLOWED_ACTIONS` else raise `ValueError` (caller maps to 400).
- `action == "stop"` and `not service.allow_stop` → refuse (400).
- Build an **argument list** `["sudo", "-n", "systemctl", action, service.unit]`
  — never a shell string, never a user-supplied unit name. `service` comes only
  from `find_service`, which only returns allow-list members.
- `dry_run` returns the would-run command without executing.
- Timeout + `check=False`; non-zero → `ok=False` with stderr tail in `detail`.

### D2 — Route `POST /services/{service_id}/{action}`
In `routes.py`, mirror `vms_remove`: `require_user` + `require_csrf` +
two-stage confirm gate + exactly one `AuditEntry` per stage
(`action="service.<action>"`, `target=unit`). `service_id` → `find_service`;
unknown → 400 **before** any subprocess. `action` not allowed → 400. Register
in `register_routes`.

### D3 — `ci-web` self-restart
Restarting `ci-web.service` from within its own worker kills the response
mid-flight. For `service.id == "ci-web"` and `action == "restart"`, run a
**detached** deferred restart so the HTTP response returns first:
`["sudo","-n","systemd-run","--on-active=2","systemctl","restart","ci-web.service"]`.
Audit as `dry-run`→`ok` with detail "scheduled in 2s". Document that the UI
will briefly drop.

### D4 — Services panel (`partials/services.html`)
Table of `service_states()`: label, unit, active badge, sub_state, and
**state-aware** action buttons (hide "start" when active, hide "stop" when
inactive or `allow_stop=False`). Stop/restart of `vmware-host.service` confirm
step shows **"N active VMs — this interrupts running jobs"** (count from the VM
context). Buttons are htmx → `#action-slot` (same pattern as Stage A4).

### D5 — sudoers + installer
Create `deploy/sudoers.d/ci-web` (or wherever the systemd installer lives — find
it first; referenced by `docs/WEB-UI.md` / the "systemd installer" commit):
```
# Exact-command NOPASSWD — no wildcards, no bare `systemctl`.
Cmnd_Alias CI_WEB_SVC = \
  /usr/bin/systemctl start act-runner.service, /usr/bin/systemctl stop act-runner.service, /usr/bin/systemctl restart act-runner.service, \
  /usr/bin/systemctl start act-runner-rust.service, /usr/bin/systemctl stop act-runner-rust.service, /usr/bin/systemctl restart act-runner-rust.service, \
  /usr/bin/systemctl start vmware-host.service, /usr/bin/systemctl stop vmware-host.service, /usr/bin/systemctl restart vmware-host.service, \
  /usr/bin/systemd-run --on-active=2 systemctl restart ci-web.service
ci-runner ALL=(root) NOPASSWD: CI_WEB_SVC
```
Installer must (a) write this to `/etc/sudoers.d/ci-web` mode `0440`, (b) run
`visudo -cf /etc/sudoers.d/ci-web` and abort/rollback on non-zero (a broken
sudoers file can lock out sudo). Verify the real `systemctl`/`systemd-run`
absolute paths on the host (`command -v`) — adjust if not `/usr/bin`.
Note: `ci-web` has **no** `stop` entry (restart-only by design).

### D6 — Config + docs
`config.py`: optional `[web].managed_services` to override the default
allow-list (list of `{id, unit, label, allow_stop}`); default = `MANAGED_SERVICES`.
Update `config.example.toml` and `docs/WEB-UI.md` (new panels, sudoers install,
self-restart caveat, security model).

### D7 — Security regression tests (`test_web_services_security.py`)
- Unknown `service_id` → 400 **and** `control_service`/subprocess never called (assert via mock).
- `action` not in `ALLOWED_ACTIONS` → 400, no subprocess.
- `stop` on `ci-web` (`allow_stop=False`) → refused, no subprocess.
- Missing/invalid CSRF → 403 (reuse existing security test pattern).
- `control_service` builds an arg list (never `shell=True`); `ci-web` restart uses `systemd-run`.
- Confirm-gate: POST without `stage=execute` → dry-run + confirm partial, no real systemctl.

### D8 — Verify + **D-GATE**
Full suite green, coverage ≥90, ruff, mypy --strict. Supervisor review +
**deploy note**: after merge, re-run `pip install .` into the production venv,
install the sudoers file, `sudo systemctl restart ci-web.service`. This stage
changes privilege boundaries — supervisor confirms the sudoers file on the live
host with `visudo -c` before enabling the panel.

---

## Cross-cutting rules (all stages)

- **Never** `subprocess.run(..., shell=True)`; always arg lists. No user input in argv.
- Every route stays behind `require_user`; every mutating POST behind `require_csrf` + confirm gate + audit.
- Read routes must never 500 on a backend/probe hiccup — degrade to an "unavailable" notice.
- `click.echo` only in `commands/*`; `operations/*` and `web/*` return dataclasses / render — no `print`.
- Match existing code style, comment density, and the `operations/` "pure logic, zero click" boundary.
- Coverage gate 90% is non-negotiable; add tests in the same stage as the code.

## Non-goals (explicit)
- No light theme. No `prefers-color-scheme`.
- No metrics-over-time, sparklines, or sampler/store (that would be a separate Phase 2 spec).
- No file **download** (Phase 1 lists only).
- No reimplementation of Gitea job/run/log views — link out only.
- No new mutating VM actions beyond existing remove/cleanup (service control is the only new mutation).

## Open risks / de-risk notes
- **Gitea run-URL scheme** (C2): confirm the exact Actions URL path before linking a Job cell to a specific run; default to a top-level Gitea link.
- **systemd unit names** (D): `act-runner.service` / `act-runner-rust.service` are from `web/app.py::RUNNER_SERVICE_NAMES`; confirm they exist on the live host (`systemctl status`) before wiring buttons.
- **systemctl/systemd-run paths** (D5): confirm absolute paths on host; sudoers matches literal paths.
- **CPU% correctness** (B1): must use two `/proc/stat` samples — single-read % is a classic bug.