# Web UI — Management Dashboard

Server-rendered dashboard (`ci web serve`) for the local CI/CD system: recent
CI executions with their live VMs (§9), cargo cache, disk, and act_runner
status, plus confirm-gated destructive actions (VM remove/cleanup, cache
purge, template backup) with an audit trail. In-process only — it imports
`operations/` and `backends.load_backend` directly and never shells out to
the CLI.

See `plans/web-ui-management-plan.md` (steps W0–W7) for the design history
and `config.example.toml`'s `[web]` / `[web.oidc]` section for every
config key referenced below.

---

## 0. Current deployment on this host

Live since 2026-07-23: `ci-web.service` is enabled + running, bound to the
LAN (`192.168.10.11:8800`), authenticating against `https://gitea.emulab.it`
via OIDC. Only the allowlisted Gitea user(s) in `/var/lib/ci/config.toml`'s
`[web].allowed_users` can log in. Manage it exactly as described in §4 below
(`systemctl status/restart ci-web.service`, `journalctl -u ci-web.service -f`).
`WebSessionSecret` and `WebOidcClientSecret` are already stored in the
`ci-runner` keyring — no action needed unless they're rotated (§3/§6).

---

## 1. Prerequisites

- The Linux host already bootstrapped by `setup-host-linux.sh` (`ci-runner`
  service user, `/opt/ci/venv`, `/var/lib/ci/config.toml`).
- A reachable Gitea instance to authenticate against (OIDC/OAuth2) — only
  required if you intend to bind the dashboard on the LAN; a loopback-only
  install can skip Gitea entirely (see §5, "open mode").
- Outbound network from the host to Gitea's `/.well-known/openid-configuration`
  endpoint, if OIDC is enabled.

Nothing here needs internet access — `ci_orchestrator[web]`'s dependencies
(FastAPI, Jinja2, Authlib, uvicorn, ...) install from the same package index
already used for the core install, and every frontend asset
(`web/static/app.css`, `web/static/htmx.min.js`) is vendored in the repo, no
CDN.

---

## 2. Running the installer

```bash
cd /opt/ci/local-ci-cd-system
sudo bash setup-web-linux.sh --host 127.0.0.1        # loopback-only, no OIDC needed
# or, once OIDC is registered (see §3):
sudo bash setup-web-linux.sh --host 192.168.1.50      # LAN
```

`setup-web-linux.sh` is idempotent — every mutating step checks its own
state first and skips (with a message) when already satisfied, so re-running
it (e.g. after `git pull` + a code change) is always safe:

1. `pip install '<repo>[web]'` into `/opt/ci/venv`, **non-editable** — per
   `CLAUDE.md`, the production venv can't resolve editable installs when
   `act_runner`/systemd launches Python from a transient working directory.
2. Renders `deploy/systemd/ci-web.service` into `/etc/systemd/system/`,
   substituting the bind IP you passed via `--host`.
3. Appends a **commented** `[web]` / `[web.oidc]` template to
   `/var/lib/ci/config.toml` if no such section exists yet — it never
   overwrites an existing one, and never writes a secret value.
4. Prompts (interactively, one at a time) to store `WebSessionSecret` and
   `WebOidcClientSecret` in the keyring, if they aren't already set. The
   secret is only ever typed at the terminal prompt — the installer never
   accepts it as a command-line argument and never echoes or logs it.
5. Prints the manual Gitea OIDC application registration steps (see §3).
6. `systemctl daemon-reload` then enables + starts (or restarts, if already
   running) `ci-web.service`.

Run `setup-web-linux.sh --help` for the full flag list (`--skip-pip`,
`--skip-secrets`, `--config-path`).

After step 3/4, edit `/var/lib/ci/config.toml`: uncomment the appended
`[web]` / `[web.oidc]` block, fill in `issuer`, `client_id`,
`allowed_users`, then restart the service (`systemctl restart ci-web.service`).

---

## 3. Gitea OAuth2 application registration

The dashboard authenticates against Gitea via OIDC (Authlib + Gitea's
`/.well-known/openid-configuration` discovery document). Register the
application **before** enabling `[web.oidc]`:

1. In Gitea: **Settings → Applications → "Manage OAuth2 Applications" →
   Create a new OAuth2 Application**.
2. **Application Name**: anything recognizable, e.g. `ci-web`.
3. **Redirect URI**: `http://<LAN-IP>:8800/auth/callback` — this **must
   match `[web.oidc].redirect_url` exactly** (scheme, host, port, and path).
   A mismatch is the #1 cause of a login redirect loop (see §6).
4. Gitea shows a **Client ID** and **Client Secret** exactly once:
   - Client ID → `[web.oidc].client_id` in `config.toml` (not a secret, plain text is fine).
   - Client Secret → **never put it in `config.toml`**. Store it in the
     keyring instead:
     ```bash
     sudo -u ci-runner /opt/ci/venv/bin/python -m ci_orchestrator creds set \
         --target WebOidcClientSecret --user web-oidc-client
     ```
     `config.toml` only ever holds the keyring *key name*
     (`client_secret_cred = "WebOidcClientSecret"`), never the value.
5. Add the Gitea username(s) allowed to log in to `[web].allowed_users`.
   **The allowlist is fail-closed**: an empty list denies everyone, it
   never means "allow all".
6. For the Executions card (§9), add an API read scope to
   `[web.oidc].scopes` — `read:repository`, plus `read:admin` if the account
   is a Gitea admin and you want runs from every repo on the instance. New
   scopes take effect after one fresh login (Gitea shows the consent screen
   again); existing sessions keep working with the old scopes until then.

---

## 4. Operating

```bash
# Start / stop / restart
sudo systemctl start ci-web.service
sudo systemctl stop ci-web.service
sudo systemctl restart ci-web.service     # after a config.toml edit or a code upgrade

# Status
systemctl status ci-web.service

# Logs (uvicorn access log + app stdout/stderr go to the journal)
journalctl -u ci-web.service -f
journalctl -u ci-web.service -n 100 --no-pager

# Health check (unauthenticated, always 200). Use the address the service is
# bound to -- uvicorn listens on `[web].bind_host` only, so on this host
# (192.168.10.11) a loopback request gets connection-refused, not a 200.
curl -s http://192.168.10.11:8800/healthz
```

Once code changes: `sudo bash setup-web-linux.sh --host <same-as-before>`
reinstalls the `[web]` extra and restarts the service in one step (steps
2–6 no-op if nothing else changed).

---

## 5. Firewall / security

- **Bind only a specific LAN IP, never `0.0.0.0`.** Neither `ci web serve`
  nor `setup-web-linux.sh` will ever write `0.0.0.0` — the installer prompts
  for (or accepts via `--host`) one explicit address, defaulting to
  `127.0.0.1`.
- **Open mode (no `[web.oidc]` configured) is localhost-only, enforced at
  startup.** `ci web serve` refuses to start if you bind a non-loopback
  address while `[web.oidc]` is absent — an unauthenticated dashboard that
  can destroy VMs and purge caches must never be reachable from the LAN.
  Configure `[web.oidc]` first, or bind `127.0.0.1`.
- **`/docs`, `/redoc`, `/openapi.json` are gated behind login too.** FastAPI
  auto-registers these with no auth of their own; the app disables its
  built-in ones (`docs_url=None` etc.) and re-adds them behind
  `require_user`, same as every data route — otherwise anyone who can reach
  the port gets a full map of every route (including every destructive one)
  with no session needed.
- Every mutating route (VM remove/cleanup, cache purge, template backup,
  template delete, backup delete) is behind a synchronizer-token CSRF check
  and a two-step confirm gate (a preview/dry-run, then an explicit "Confirm"
  that actually executes) — there is no one-click destroy.
- **Template delete has no backup-required safety rail.** It removes the
  entire template directory — the golden image VM clones are made from, not
  a single backup archive — behind the same confirm gate as everything else,
  but nothing stops you from deleting a template with zero backups. Use the
  adjacent Backup button first if you want a way back.
- Every mutating action appends exactly one entry to the audit log
  (`[web].audit_log`, default `/var/lib/ci/logs/web-audit.jsonl`), viewable
  at `/audit` in the dashboard.
- If you run a host firewall (ufw/nftables/iptables), only open the chosen
  port (default `8800`) to the specific LAN/VLAN that needs dashboard
  access, not the whole network.

---

## 6. Keyring / headless note

`ci-web.service` runs as `ci-runner` under systemd — there is no desktop
session and no D-Bus, so the GNOME Secret Service backend (`secret-tool`)
does not work here. This host uses **`keyrings.alt.file.PlaintextKeyring`**
instead (already the credential backend used for guest/Gitea credentials —
see `plans/archived/2026-05-23/B3-closeout.md`):

- Config: `/home/ci-runner/.local/share/python_keyring/keyringrc.cfg`
  (`[backend] default-keyring=keyrings.alt.file.PlaintextKeyring`)
- Store: `/home/ci-runner/.local/share/python_keyring/keyring_pass.cfg` —
  plain text, readable only by `ci-runner` (file permissions), no D-Bus
  session required. Verified to work both from `sudo -u ci-runner ...` and
  from inside a plain systemd service.
- This is why `ci-web.service`'s hardening uses `ProtectHome=read-only`
  and **not** `ProtectHome=yes`/`true`: the latter hides `/home` entirely
  from the unit's mount namespace, which would break every OIDC/session
  secret lookup and make login fail even though the keyring itself is
  fine. `read-only` keeps `/home/ci-runner` readable (the app never writes
  to the keyring at request time — secrets are set once, out-of-band, by
  `ci creds set`) while still blocking writes.
- `WebSessionSecret` / `WebOidcClientSecret` are keyring key *names* in
  `config.toml` (`session_secret_cred`, `client_secret_cred`) — the actual
  values live only in the keyring, set via `ci creds set --target <name>
  --user <label>` (interactive prompt, `hide_input` + confirmation; never
  a CLI argument, never logged).

---

## 7. Service management (Stage D)

The dashboard's **Services** panel lets an authenticated web user start, stop,
or restart a fixed allow-list of systemd units on the host — the same
confirm-gated, CSRF-protected, audited mutating-action shape as VM
remove/cleanup, cache purge, and template backup (§5), applied to `sudo
systemctl`.

**Allow-list (default, `operations/services.py::MANAGED_SERVICES`):**

| Slug | Unit | Stop allowed? |
| --- | --- | --- |
| `act-runner` | `act-runner.service` | yes |
| `act-runner-rust` | `act-runner-rust.service` | yes |
| `vmware-host` | `vmware-host.service` | yes |
| `ci-web` | `ci-web.service` | **no** — restart-only |

Override it with `[web].managed_services` in `config.toml` (see
`config.example.toml`'s commented example) — a list of `{id, unit, label,
allow_stop}` tables. A malformed row is skipped, never crashes startup.

### Security model

- **Allow-list by slug, not by unit name.** The URL is
  `POST /services/{service_id}/{action}` where `service_id` is one of the
  slugs above (`find_service()`), never a raw systemd unit name — there is no
  code path that builds a controllable unit from unvalidated request input.
- **Exact-command sudoers, no wildcards.** `deploy/sudoers.d/ci-web` grants
  `ci-runner` passwordless `sudo` for exactly nine literal command lines (one
  per unit × action, plus the `ci-web` restart special case) — no bare
  `systemctl`, no `*`, no `ALL` in the command position. Even if the web app
  itself were compromised, `ci-runner`'s sudo grant cannot be used to run any
  command beyond those nine lines.
- **Arg list, never a shell.** `operations/services.py::control_service`
  builds `["sudo", "-n", "systemctl", action, unit]` (or the `systemd-run`
  variant below) as a Python list handed straight to `subprocess.run` —
  `shell=True` is never used anywhere in this path.
- **`ci-web` is restart-only.** `allow_stop=False` — a `stop` request against
  it is refused with a 400 *before* any subprocess or audit entry, both at
  the route (`web/routes.py::services_control`) and defensively again inside
  `control_service` itself. Rationale: stopping the service that renders this
  very panel would lock the operator out with no way to start it again from
  the UI.
- **`ci-web` restart is deferred.** Restarting `ci-web.service` from inside
  its own worker would kill the HTTP response before it could be sent, so
  that one case runs `sudo -n systemd-run --on-active=2 systemctl restart
  ci-web.service` instead of a direct `systemctl restart` — the
  `systemd-run` call itself returns almost immediately (it only schedules
  the restart ~2s out), so the confirming request completes normally and the
  dashboard briefly drops a couple of seconds later.
- **`vmware-host` stop/restart interrupts running jobs.** The confirm step
  shows the current count of active VMs ("N active VM(s) — this interrupts
  running jobs") when stopping or restarting `vmware-host.service`, computed
  from the same VM backend the Executions card uses — best-effort, degrades
  to no warning if the backend is unavailable, never blocks the action.
- **Every action is audited exactly once per stage** (`service.start` /
  `service.stop` / `service.restart`, `target` = the unit name), same as
  every other mutating route — visible at `/audit`.

### Sudoers install

`setup-web-linux.sh` installs `deploy/sudoers.d/ci-web` to
`/etc/sudoers.d/ci-web` (mode `0440`, owner `root:root`) as step 6/7,
validating it with `visudo -cf` and **removing it again if validation
fails** — a broken `sudoers.d` fragment can lock out `sudo` entirely, so this
step never leaves an unvalidated file in place. Before installing on a new
host, confirm `systemctl`/`systemd-run` are actually at `/usr/bin` on that
host (`command -v systemctl systemd-run`) — edit both
`deploy/sudoers.d/ci-web` and the installed copy if not.

To install just the sudoers grant (e.g. after editing
`deploy/sudoers.d/ci-web`) without touching anything else:

```bash
sudo bash setup-web-linux.sh --skip-pip --skip-secrets
```

### Deploy note

After merging a Stage D change: re-run `pip install .` into the production
venv (per `CLAUDE.md`), re-run `setup-web-linux.sh` (installs/updates the
sudoers grant and restarts `ci-web.service`), then confirm the sudoers file
on the live host with `sudo visudo -c` before relying on the panel.

---

## 8. Dashboard layout — one card per subject, no tile row

The dashboard opens with four compact info cards, each the **only** place its
subject is reported:

| Card | Route | Refresh | Contents |
|---|---|---|---|
| **Host** | `GET /host` | 5s | `System` (hostname · kernel), `CPU` (model · cores), CPU / RAM / disk with gauges; uptime in the header bar, load average under the CPU gauge |
| **VMware modules** | `GET /vmware` | 15s | `vmmon`, `vmnet`, `vmware-host.service` |
| **Runners** | `GET /runners` | 5s | each `act-runner*` unit with its sub-state |
| **Cargo cache** | `GET /cache` | 5s | root, total size, entries, oldest entry, retention |

Then the wide tables: Executions (§9), Templates, Artifacts, Services,
Maintenance.

All four info cards render the same `<dl>` grammar at the page's plain body
size — one `<dt>`/`<dd>` pair per fact. VMware and Runners used to be a
bordered `<table>` at a smaller font (`0.9rem` body text, uppercase `0.75rem`
column headers), which read as a visibly different style sitting next to
Host and Cargo cache's plain label/value lists. They carry the same two data
points as before per row (a status badge, a sub-state) — just as `<dt>{{
unit }}</dt><dd>{{ badge }} {{ sub_state }}</dd>` instead of a table row —
so no reading was dropped, and no card in this row uses `<table>` any more.

There is **no status-header tile row**. It used to sit above these cards
showing VMs / Runners / CPU / RAM / Disk / Cache, which meant runner state was
displayed in three places at once (tile, Runners card, Services card) and the
core count, RAM total, cache size and entry count in two. The readouts that
only the tiles had — CPU%, RAM used, disk used/free — moved into the Host card,
where the rest of the host data already was. The VM count went away with the
tiles; running VMs are listed in the Executions card, and the orphan sweep is
in Maintenance.

The Host card carries the most data of the four, so it is packed to match
their height: the near-static facts pair up (`System`, `CPU`), uptime rides the
header bar (spelled out — it has no `<dt>` to label it), and the load average
is the CPU gauge's sub-line. Five `<dt>` rows total. The CPU model is trimmed
by the `cpu_short` filter (`Intel(R) Core(TM) i9-13900K CPU @ 3.00GHz` →
`Intel Core i9-13900K`); the untrimmed name is the `title` of that row.

**Health does not depend on color.** Each gauge states its value as text
directly above the bar (`42%`, `20.0 / 62 GB`, `40% · 60 GB free`), and the
level word — `ok` / `elevated` / `critical` / `unknown` — is part of the
gauge's `aria-label`, so a screen reader announces it. There is no visible
level strip: the words repeated what the numbers already said on every row.
Thresholds behind the fill colour and the announced word: CPU and RAM `> 75%`
amber, `> 90%` red; disk `> 70%` amber, `> 85%` red; a source that could not
be read is a neutral `unknown`, never a false green — and when the disk probe
fails outright, `unknown` is printed as the row's value, since there is no
gauge left to carry it.

---

## 9. Executions card (recent Gitea Actions runs)

The dashboard's top wide table is **Executions**: the most recent workflow
runs, the live ones included, with the build VM(s) that run spawned (with
their `Remove` button). It replaced the old read-only "Live VMs" table — a VM
alone could never answer "did the last build pass?".

Every fact is its own column — Run, Repo, Workflow, Branch, Commit, Trigger,
Actor, Status, Duration, Started, VM, IP / Uptime, Actions. No cell stacks two
facts on separate lines with `<br>`; a run's repo, the commit hash, who
triggered it, and when it started each get their own column instead. The
table is wide as a result — 13 columns — and scrolls horizontally inside the
card's own `.table-wrap` rather than the page, same as the other wide tables
(Templates, Services).

**Run shows `run_id`, not the per-repo `#N`.** Gitea's `run_number` only
counts within its own repo/workflow, so it is meaningless once the admin
endpoint mixes every repo on the instance into one list; `run_id` is Gitea's
own instance-wide, strictly-increasing id, which is what "global recency"
above is sorted on too, so the number in the column and the row order agree.
The cell still links out to Gitea's own run page (`html_url`, which Gitea
builds from the per-repo number internally) — only the *displayed* text
changed.

**Started is click-to-toggle.** It renders relative ("2h ago") by default;
clicking swaps to the absolute local timestamp, and clicking again swaps
back. Both strings are pre-rendered server-side into the button's
`data-relative`/`data-absolute` attributes, so the swap (`app.js`) is
instant and needs no request. The card's own 5s poll re-renders the whole
table from the server, always starting back at relative — so which format
was showing is remembered client-side, keyed by the button's stable `id`
(`started-{run_id}-{attempt}`, or `started-vm-{vmx}` for an unmatched VM
row), and re-applied to the fresh button after every poll rather than
frozen onto the old node (which would also freeze its relative-time text,
badge, duration — everything else in that row).

Runs come from Gitea's Actions API: `/api/v1/admin/actions/runs` (every repo
on the instance, admin token) with `/api/v1/user/actions/runs` as the
non-admin fallback. Results are cached for 10s, so the card's 5s refresh costs
Gitea at most one call per 10s window; the VM columns are re-read on every
refresh.

The Status pill carries a check/✕ icon for the two outcomes that are a real
verdict (`success`/`failure`, via `run_badge_icon`); `queued`/`running`/
`cancelled`/`skipped` stay plain text since they are not a pass/fail on the
run. Every pill in the column — outcome pills and the VM-only `orphan`/
`running` pills on unmatched-VM rows — is forced to the same width
(`#executions-card td:nth-child(8)` — Status is the 8th column, see above —
sized to fit `cancelled`, the longest word) with left-aligned content, so the
column reads as a clean list instead of ragged pill widths.

Rows are ordered **newest first by actual start time, globally** —
`run.started` (a run still queued reads as "now", since it was only just
created), with `run_id` then `attempt` as tiebreakers for runs sharing the
exact same timestamp. `run.number` is deliberately *not* the sort key: it is
a per-repo/workflow counter, and the admin endpoint mixes every repo/workflow
on the instance into one feed, so a rarely-triggered workflow's #53 that just
fired legitimately outranks a #142 from a busier workflow that ran hours
earlier — sorting by `number` would bury it. A manual re-run bumping
`started_at` and jumping its row back to the top is not a bug either: it
really is the most recent activity on that run.
The row-count dropdown at
the right end of the card's header bar picks how many to list — 10 / 25 / 50 /
100, default from `[web].executions_limit` (10). The choice rides on the card's own polling URL
(`/executions?limit=N`), so it survives every 5s refresh with no client-side
state; a value set in the config outside those four is offered as an extra
option rather than dropped.

**Where the API token comes from.** Normally nowhere extra: the dashboard
reuses the access token from your own OIDC login, so what the card shows is
exactly what your Gitea account can see. That token is kept **server-side
only** (an in-process map keyed by username) and dropped on logout — it is
never written into the session cookie, which is signed but *not* encrypted and
travels over plain HTTP on the LAN. An expired token is refreshed once
automatically; a refusal that refresh cannot fix shows a "log in again" link
in the card.

Optional fallback: store a Gitea PAT in the keyring and name it in
`[web].gitea_token_cred`. It is used when no logged-in token is available —
after a service restart (which empties the in-process map while session cookies
stay valid) — **and** when Gitea refuses the login token (`401`, or `403` for
insufficient scope), which is what an OIDC token limited to `openid profile`
gets. Set it exactly like the other secrets:

```bash
sudo -u ci-runner /opt/ci/venv/bin/python -m ci_orchestrator creds set \
    --target GiteaApiToken --user gitea-api
```

**Degraded states** never take the card down: no token, an expired token, or
an unreachable Gitea each render a warning notice *and still list every
running VM* below it, so the dashboard is never less useful than the VM table
it replaced. The notice includes whatever the server said (Gitea names the
missing scope in its own body), which is the fastest way to tell a token
problem from something in front of Gitea.

**Gotcha — Cloudflare in front of Gitea.** `gitea.emulab.it` is proxied by
Cloudflare, which rejects urllib's default `Python-urllib/3.x` user agent with
`403` + `error code: 1010` *before the request reaches Gitea* — indistinguishable
from a token refusal if you only look at the status code. `operations/executions.py`
therefore always sends an explicit `User-Agent`; any ordinary agent string
passes. Do not remove that header.

**Orphaned VMs are not handled in this card.** A powered-on VM whose run is no
longer in the window (or that no run ever claimed) keeps a row of its own
flagged `no run`, with its `Remove` button — but the orphan *sweep* lives in
the **Maintenance** card ("Cleanup orphaned VMs"). That is the place to act on
orphans; the Executions card only avoids hiding them.

---

## 10. Troubleshooting

**Login loop (bounces back to `/login` after authenticating with Gitea)**
: The registered Redirect URI in Gitea does not exactly match
  `[web.oidc].redirect_url` in `config.toml` — check scheme (`http` vs
  `https`), host/IP, port, and the `/auth/callback` path character-for-character.

**403 after a successful Gitea login**
: Your Gitea username is not in `[web].allowed_users` (or the list is
  empty — the allowlist is fail-closed by design). Add it and restart the
  service.

**"keyring key '...' not found -- run `ci creds set ...`" in the journal /
on `/login`**
: The session secret or the OIDC client secret has not been stored yet
  (or was stored for a different Unix user than `ci-runner`, or under a
  different `_cred` name than the one in `config.toml`). Re-run:
  ```bash
  sudo -u ci-runner /opt/ci/venv/bin/python -m ci_orchestrator creds set \
      --target WebSessionSecret --user web-session
  ```
  substituting the exact key name printed in the error. See §6 for why this
  must run as `ci-runner`, not as your own login.

**Dashboard is blank / cards show "unavailable"**
: The VM backend (`vmrun`) or `shutil.disk_usage`/`systemctl` could not be
  reached from the host — the dashboard degrades a card to an "unavailable"
  notice instead of a 500. Check `vmware-host.service` is up
  (`systemctl is-active vmware-host.service`) and that `/healthz` still
  returns `200`.

**`ci web serve` refuses to start with "refusing to bind ... without
authentication"**
: You passed a non-loopback `--host` (or `[web].bind_host` in
  `config.toml`) while `[web.oidc]` is unset. This is the guard described
  in §5 — configure `[web.oidc]` (see §3) or bind `127.0.0.1`.

**Confirm/Cleanup/Purge form returns 403**
: The CSRF token expired (the session cookie was cleared, e.g. by
  `/logout`, or the browser tab was left open across a service restart
  that rotated the session secret in open mode). Reload the page and retry
  — the confirm form always mints a fresh token.