Simone
af7dee60e5
The dashboard could only show what was powered on this instant, so it could
not answer "did the last build pass?" — and no local execution history exists
to answer it from (`report job` reads a PowerShell-era JSONL path that nothing
writes on this host, and the recent artifact dirs are empty).
Recent runs now come from Gitea's Actions API (`operations/executions.py`):
`/api/v1/admin/actions/runs` for every repo on the instance, falling back to
`/api/v1/user/actions/runs` when the token is not admin-scoped. One list call
per refresh, cached 10s, so the card's 5s htmx poll costs Gitea at most one
call per window. Live VMs are grouped under the run that spawned them — the
job id is `{run_id}-{run_attempt}[-suffix]` and the clone name carries it — so
a matrix run collects several VMs on one row. Every failure mode (no token,
401, unreachable Gitea, malformed payload) degrades to a notice plus the plain
VM rows, each keeping its Remove button, so the card is never less useful than
the table it replaces. Orphans stay where they were: the sweep in Maintenance,
the count in the status-header tile.
Auth reuses the access token from the operator's own OIDC login, which
`auth.py` previously discarded. SECURITY: that token is kept server-side only
(`app.state.gitea_tokens`, dropped on logout) and never written into the
session — `SessionMiddleware` signs the session cookie but does not encrypt
it, and this service is deployed over plain HTTP on the LAN, so a bearer token
in there would be readable in the browser and on the wire. An expired token is
refreshed once; `[web] gitea_token_cred` names an optional keyring PAT for the
cases a login token cannot cover (service restart, non-admin operator).
`GET /vms` and `partials/vms.html` are gone; `POST /vms/{id}/remove` and
`POST /vms/cleanup` are untouched.
Design: docs/superpowers/specs/2026-07-25-executions-card-design.md
373 lines
18 KiB
Markdown
373 lines
18 KiB
Markdown
# Web UI — Management Dashboard
|
||
|
||
Server-rendered dashboard (`ci web serve`) for the local CI/CD system: recent
|
||
CI executions with their live VMs (§8), cargo cache, disk, and act_runner
|
||
status, plus confirm-gated destructive actions (VM remove/cleanup, cache
|
||
purge, template backup) with an audit trail. In-process only — it imports
|
||
`operations/` and `backends.load_backend` directly and never shells out to
|
||
the CLI.
|
||
|
||
See `plans/web-ui-management-plan.md` (steps W0–W7) for the design history
|
||
and `config.example.toml`'s `[web]` / `[web.oidc]` section for every
|
||
config key referenced below.
|
||
|
||
---
|
||
|
||
## 0. Current deployment on this host
|
||
|
||
Live since 2026-07-23: `ci-web.service` is enabled + running, bound to the
|
||
LAN (`192.168.10.11:8800`), authenticating against `https://gitea.emulab.it`
|
||
via OIDC. Only the allowlisted Gitea user(s) in `/var/lib/ci/config.toml`'s
|
||
`[web].allowed_users` can log in. Manage it exactly as described in §4 below
|
||
(`systemctl status/restart ci-web.service`, `journalctl -u ci-web.service -f`).
|
||
`WebSessionSecret` and `WebOidcClientSecret` are already stored in the
|
||
`ci-runner` keyring — no action needed unless they're rotated (§3/§6).
|
||
|
||
---
|
||
|
||
## 1. Prerequisites
|
||
|
||
- The Linux host already bootstrapped by `setup-host-linux.sh` (`ci-runner`
|
||
service user, `/opt/ci/venv`, `/var/lib/ci/config.toml`).
|
||
- A reachable Gitea instance to authenticate against (OIDC/OAuth2) — only
|
||
required if you intend to bind the dashboard on the LAN; a loopback-only
|
||
install can skip Gitea entirely (see §5, "open mode").
|
||
- Outbound network from the host to Gitea's `/.well-known/openid-configuration`
|
||
endpoint, if OIDC is enabled.
|
||
|
||
Nothing here needs internet access — `ci_orchestrator[web]`'s dependencies
|
||
(FastAPI, Jinja2, Authlib, uvicorn, ...) install from the same package index
|
||
already used for the core install, and every frontend asset
|
||
(`web/static/app.css`, `web/static/htmx.min.js`) is vendored in the repo, no
|
||
CDN.
|
||
|
||
---
|
||
|
||
## 2. Running the installer
|
||
|
||
```bash
|
||
cd /opt/ci/local-ci-cd-system
|
||
sudo bash setup-web-linux.sh --host 127.0.0.1 # loopback-only, no OIDC needed
|
||
# or, once OIDC is registered (see §3):
|
||
sudo bash setup-web-linux.sh --host 192.168.1.50 # LAN
|
||
```
|
||
|
||
`setup-web-linux.sh` is idempotent — every mutating step checks its own
|
||
state first and skips (with a message) when already satisfied, so re-running
|
||
it (e.g. after `git pull` + a code change) is always safe:
|
||
|
||
1. `pip install '<repo>[web]'` into `/opt/ci/venv`, **non-editable** — per
|
||
`CLAUDE.md`, the production venv can't resolve editable installs when
|
||
`act_runner`/systemd launches Python from a transient working directory.
|
||
2. Renders `deploy/systemd/ci-web.service` into `/etc/systemd/system/`,
|
||
substituting the bind IP you passed via `--host`.
|
||
3. Appends a **commented** `[web]` / `[web.oidc]` template to
|
||
`/var/lib/ci/config.toml` if no such section exists yet — it never
|
||
overwrites an existing one, and never writes a secret value.
|
||
4. Prompts (interactively, one at a time) to store `WebSessionSecret` and
|
||
`WebOidcClientSecret` in the keyring, if they aren't already set. The
|
||
secret is only ever typed at the terminal prompt — the installer never
|
||
accepts it as a command-line argument and never echoes or logs it.
|
||
5. Prints the manual Gitea OIDC application registration steps (see §3).
|
||
6. `systemctl daemon-reload` then enables + starts (or restarts, if already
|
||
running) `ci-web.service`.
|
||
|
||
Run `setup-web-linux.sh --help` for the full flag list (`--skip-pip`,
|
||
`--skip-secrets`, `--config-path`).
|
||
|
||
After step 3/4, edit `/var/lib/ci/config.toml`: uncomment the appended
|
||
`[web]` / `[web.oidc]` block, fill in `issuer`, `client_id`,
|
||
`allowed_users`, then restart the service (`systemctl restart ci-web.service`).
|
||
|
||
---
|
||
|
||
## 3. Gitea OAuth2 application registration
|
||
|
||
The dashboard authenticates against Gitea via OIDC (Authlib + Gitea's
|
||
`/.well-known/openid-configuration` discovery document). Register the
|
||
application **before** enabling `[web.oidc]`:
|
||
|
||
1. In Gitea: **Settings → Applications → "Manage OAuth2 Applications" →
|
||
Create a new OAuth2 Application**.
|
||
2. **Application Name**: anything recognizable, e.g. `ci-web`.
|
||
3. **Redirect URI**: `http://<LAN-IP>:8800/auth/callback` — this **must
|
||
match `[web.oidc].redirect_url` exactly** (scheme, host, port, and path).
|
||
A mismatch is the #1 cause of a login redirect loop (see §6).
|
||
4. Gitea shows a **Client ID** and **Client Secret** exactly once:
|
||
- Client ID → `[web.oidc].client_id` in `config.toml` (not a secret, plain text is fine).
|
||
- Client Secret → **never put it in `config.toml`**. Store it in the
|
||
keyring instead:
|
||
```bash
|
||
sudo -u ci-runner /opt/ci/venv/bin/python -m ci_orchestrator creds set \
|
||
--target WebOidcClientSecret --user web-oidc-client
|
||
```
|
||
`config.toml` only ever holds the keyring *key name*
|
||
(`client_secret_cred = "WebOidcClientSecret"`), never the value.
|
||
5. Add the Gitea username(s) allowed to log in to `[web].allowed_users`.
|
||
**The allowlist is fail-closed**: an empty list denies everyone, it
|
||
never means "allow all".
|
||
6. For the Executions card (§8), add an API read scope to
|
||
`[web.oidc].scopes` — `read:repository`, plus `read:admin` if the account
|
||
is a Gitea admin and you want runs from every repo on the instance. New
|
||
scopes take effect after one fresh login (Gitea shows the consent screen
|
||
again); existing sessions keep working with the old scopes until then.
|
||
|
||
---
|
||
|
||
## 4. Operating
|
||
|
||
```bash
|
||
# Start / stop / restart
|
||
sudo systemctl start ci-web.service
|
||
sudo systemctl stop ci-web.service
|
||
sudo systemctl restart ci-web.service # after a config.toml edit or a code upgrade
|
||
|
||
# Status
|
||
systemctl status ci-web.service
|
||
|
||
# Logs (uvicorn access log + app stdout/stderr go to the journal)
|
||
journalctl -u ci-web.service -f
|
||
journalctl -u ci-web.service -n 100 --no-pager
|
||
|
||
# Health check (unauthenticated, always 200). Use the address the service is
|
||
# bound to -- uvicorn listens on `[web].bind_host` only, so on this host
|
||
# (192.168.10.11) a loopback request gets connection-refused, not a 200.
|
||
curl -s http://192.168.10.11:8800/healthz
|
||
```
|
||
|
||
Once code changes: `sudo bash setup-web-linux.sh --host <same-as-before>`
|
||
reinstalls the `[web]` extra and restarts the service in one step (steps
|
||
2–6 no-op if nothing else changed).
|
||
|
||
---
|
||
|
||
## 5. Firewall / security
|
||
|
||
- **Bind only a specific LAN IP, never `0.0.0.0`.** Neither `ci web serve`
|
||
nor `setup-web-linux.sh` will ever write `0.0.0.0` — the installer prompts
|
||
for (or accepts via `--host`) one explicit address, defaulting to
|
||
`127.0.0.1`.
|
||
- **Open mode (no `[web.oidc]` configured) is localhost-only, enforced at
|
||
startup.** `ci web serve` refuses to start if you bind a non-loopback
|
||
address while `[web.oidc]` is absent — an unauthenticated dashboard that
|
||
can destroy VMs and purge caches must never be reachable from the LAN.
|
||
Configure `[web.oidc]` first, or bind `127.0.0.1`.
|
||
- Every mutating route (VM remove/cleanup, cache purge, template backup) is
|
||
behind a synchronizer-token CSRF check and a two-step confirm gate (a
|
||
preview/dry-run, then an explicit "Confirm" that actually executes) —
|
||
there is no one-click destroy.
|
||
- Every mutating action appends exactly one entry to the audit log
|
||
(`[web].audit_log`, default `/var/lib/ci/logs/web-audit.jsonl`), viewable
|
||
at `/audit` in the dashboard.
|
||
- If you run a host firewall (ufw/nftables/iptables), only open the chosen
|
||
port (default `8800`) to the specific LAN/VLAN that needs dashboard
|
||
access, not the whole network.
|
||
|
||
---
|
||
|
||
## 6. Keyring / headless note
|
||
|
||
`ci-web.service` runs as `ci-runner` under systemd — there is no desktop
|
||
session and no D-Bus, so the GNOME Secret Service backend (`secret-tool`)
|
||
does not work here. This host uses **`keyrings.alt.file.PlaintextKeyring`**
|
||
instead (already the credential backend used for guest/Gitea credentials —
|
||
see `plans/archived/2026-05-23/B3-closeout.md`):
|
||
|
||
- Config: `/home/ci-runner/.local/share/python_keyring/keyringrc.cfg`
|
||
(`[backend] default-keyring=keyrings.alt.file.PlaintextKeyring`)
|
||
- Store: `/home/ci-runner/.local/share/python_keyring/keyring_pass.cfg` —
|
||
plain text, readable only by `ci-runner` (file permissions), no D-Bus
|
||
session required. Verified to work both from `sudo -u ci-runner ...` and
|
||
from inside a plain systemd service.
|
||
- This is why `ci-web.service`'s hardening uses `ProtectHome=read-only`
|
||
and **not** `ProtectHome=yes`/`true`: the latter hides `/home` entirely
|
||
from the unit's mount namespace, which would break every OIDC/session
|
||
secret lookup and make login fail even though the keyring itself is
|
||
fine. `read-only` keeps `/home/ci-runner` readable (the app never writes
|
||
to the keyring at request time — secrets are set once, out-of-band, by
|
||
`ci creds set`) while still blocking writes.
|
||
- `WebSessionSecret` / `WebOidcClientSecret` are keyring key *names* in
|
||
`config.toml` (`session_secret_cred`, `client_secret_cred`) — the actual
|
||
values live only in the keyring, set via `ci creds set --target <name>
|
||
--user <label>` (interactive prompt, `hide_input` + confirmation; never
|
||
a CLI argument, never logged).
|
||
|
||
---
|
||
|
||
## 7. Service management (Stage D)
|
||
|
||
The dashboard's **Services** panel lets an authenticated web user start, stop,
|
||
or restart a fixed allow-list of systemd units on the host — the same
|
||
confirm-gated, CSRF-protected, audited mutating-action shape as VM
|
||
remove/cleanup, cache purge, and template backup (§5), applied to `sudo
|
||
systemctl`.
|
||
|
||
**Allow-list (default, `operations/services.py::MANAGED_SERVICES`):**
|
||
|
||
| Slug | Unit | Stop allowed? |
|
||
| --- | --- | --- |
|
||
| `act-runner` | `act-runner.service` | yes |
|
||
| `act-runner-rust` | `act-runner-rust.service` | yes |
|
||
| `vmware-host` | `vmware-host.service` | yes |
|
||
| `ci-web` | `ci-web.service` | **no** — restart-only |
|
||
|
||
Override it with `[web].managed_services` in `config.toml` (see
|
||
`config.example.toml`'s commented example) — a list of `{id, unit, label,
|
||
allow_stop}` tables. A malformed row is skipped, never crashes startup.
|
||
|
||
### Security model
|
||
|
||
- **Allow-list by slug, not by unit name.** The URL is
|
||
`POST /services/{service_id}/{action}` where `service_id` is one of the
|
||
slugs above (`find_service()`), never a raw systemd unit name — there is no
|
||
code path that builds a controllable unit from unvalidated request input.
|
||
- **Exact-command sudoers, no wildcards.** `deploy/sudoers.d/ci-web` grants
|
||
`ci-runner` passwordless `sudo` for exactly nine literal command lines (one
|
||
per unit × action, plus the `ci-web` restart special case) — no bare
|
||
`systemctl`, no `*`, no `ALL` in the command position. Even if the web app
|
||
itself were compromised, `ci-runner`'s sudo grant cannot be used to run any
|
||
command beyond those nine lines.
|
||
- **Arg list, never a shell.** `operations/services.py::control_service`
|
||
builds `["sudo", "-n", "systemctl", action, unit]` (or the `systemd-run`
|
||
variant below) as a Python list handed straight to `subprocess.run` —
|
||
`shell=True` is never used anywhere in this path.
|
||
- **`ci-web` is restart-only.** `allow_stop=False` — a `stop` request against
|
||
it is refused with a 400 *before* any subprocess or audit entry, both at
|
||
the route (`web/routes.py::services_control`) and defensively again inside
|
||
`control_service` itself. Rationale: stopping the service that renders this
|
||
very panel would lock the operator out with no way to start it again from
|
||
the UI.
|
||
- **`ci-web` restart is deferred.** Restarting `ci-web.service` from inside
|
||
its own worker would kill the HTTP response before it could be sent, so
|
||
that one case runs `sudo -n systemd-run --on-active=2 systemctl restart
|
||
ci-web.service` instead of a direct `systemctl restart` — the
|
||
`systemd-run` call itself returns almost immediately (it only schedules
|
||
the restart ~2s out), so the confirming request completes normally and the
|
||
dashboard briefly drops a couple of seconds later.
|
||
- **`vmware-host` stop/restart interrupts running jobs.** The confirm step
|
||
shows the current count of active VMs ("N active VM(s) — this interrupts
|
||
running jobs") when stopping or restarting `vmware-host.service`, computed
|
||
from the same VM backend the Executions card uses — best-effort, degrades
|
||
to no warning if the backend is unavailable, never blocks the action.
|
||
- **Every action is audited exactly once per stage** (`service.start` /
|
||
`service.stop` / `service.restart`, `target` = the unit name), same as
|
||
every other mutating route — visible at `/audit`.
|
||
|
||
### Sudoers install
|
||
|
||
`setup-web-linux.sh` installs `deploy/sudoers.d/ci-web` to
|
||
`/etc/sudoers.d/ci-web` (mode `0440`, owner `root:root`) as step 6/7,
|
||
validating it with `visudo -cf` and **removing it again if validation
|
||
fails** — a broken `sudoers.d` fragment can lock out `sudo` entirely, so this
|
||
step never leaves an unvalidated file in place. Before installing on a new
|
||
host, confirm `systemctl`/`systemd-run` are actually at `/usr/bin` on that
|
||
host (`command -v systemctl systemd-run`) — edit both
|
||
`deploy/sudoers.d/ci-web` and the installed copy if not.
|
||
|
||
To install just the sudoers grant (e.g. after editing
|
||
`deploy/sudoers.d/ci-web`) without touching anything else:
|
||
|
||
```bash
|
||
sudo bash setup-web-linux.sh --skip-pip --skip-secrets
|
||
```
|
||
|
||
### Deploy note
|
||
|
||
After merging a Stage D change: re-run `pip install .` into the production
|
||
venv (per `CLAUDE.md`), re-run `setup-web-linux.sh` (installs/updates the
|
||
sudoers grant and restarts `ci-web.service`), then confirm the sudoers file
|
||
on the live host with `sudo visudo -c` before relying on the panel.
|
||
|
||
---
|
||
|
||
## 8. Executions card (recent Gitea Actions runs)
|
||
|
||
The dashboard's top wide table is **Executions**: the most recent workflow
|
||
runs, the live ones included, each with its status, workflow, branch/commit,
|
||
trigger, duration, and the build VM(s) that run spawned (with their `Remove`
|
||
button). It replaced the old read-only "Live VMs" table — a VM alone could
|
||
never answer "did the last build pass?".
|
||
|
||
Runs come from Gitea's Actions API: `/api/v1/admin/actions/runs` (every repo
|
||
on the instance, admin token) with `/api/v1/user/actions/runs` as the
|
||
non-admin fallback. `[web].executions_limit` (default 15) caps how many.
|
||
Results are cached for 10s, so the card's 5s refresh costs Gitea at most one
|
||
call per 10s window; the VM columns are re-read on every refresh.
|
||
|
||
**Where the API token comes from.** Normally nowhere extra: the dashboard
|
||
reuses the access token from your own OIDC login, so what the card shows is
|
||
exactly what your Gitea account can see. That token is kept **server-side
|
||
only** (an in-process map keyed by username) and dropped on logout — it is
|
||
never written into the session cookie, which is signed but *not* encrypted and
|
||
travels over plain HTTP on the LAN. An expired token is refreshed once
|
||
automatically; a refusal that refresh cannot fix shows a "log in again" link
|
||
in the card.
|
||
|
||
Optional fallback: store a Gitea PAT in the keyring and name it in
|
||
`[web].gitea_token_cred`. It is used only when no logged-in token is
|
||
available — after a service restart (which empties the in-process map while
|
||
session cookies stay valid), or for an operator whose own token lacks the
|
||
scope. Set it exactly like the other secrets:
|
||
|
||
```bash
|
||
sudo -u ci-runner /opt/ci/venv/bin/python -m ci_orchestrator creds set \
|
||
--target GiteaApiToken --user gitea-api
|
||
```
|
||
|
||
**Degraded states** never take the card down: no token, an expired token, or
|
||
an unreachable Gitea each render a warning notice *and still list every
|
||
running VM* below it, so the dashboard is never less useful than the VM table
|
||
it replaced.
|
||
|
||
**Orphaned VMs are not handled in this card.** A powered-on VM whose run is no
|
||
longer in the window (or that no run ever claimed) keeps a row of its own
|
||
flagged `no run`, with its `Remove` button — but the orphan *sweep* lives in
|
||
the **Maintenance** card ("Cleanup orphaned VMs") and the orphan *count* in
|
||
the status-header VMs tile. Those two are the places to look at orphans; the
|
||
Executions card only avoids hiding them.
|
||
|
||
---
|
||
|
||
## 9. Troubleshooting
|
||
|
||
**Login loop (bounces back to `/login` after authenticating with Gitea)**
|
||
: The registered Redirect URI in Gitea does not exactly match
|
||
`[web.oidc].redirect_url` in `config.toml` — check scheme (`http` vs
|
||
`https`), host/IP, port, and the `/auth/callback` path character-for-character.
|
||
|
||
**403 after a successful Gitea login**
|
||
: Your Gitea username is not in `[web].allowed_users` (or the list is
|
||
empty — the allowlist is fail-closed by design). Add it and restart the
|
||
service.
|
||
|
||
**"keyring key '...' not found -- run `ci creds set ...`" in the journal /
|
||
on `/login`**
|
||
: The session secret or the OIDC client secret has not been stored yet
|
||
(or was stored for a different Unix user than `ci-runner`, or under a
|
||
different `_cred` name than the one in `config.toml`). Re-run:
|
||
```bash
|
||
sudo -u ci-runner /opt/ci/venv/bin/python -m ci_orchestrator creds set \
|
||
--target WebSessionSecret --user web-session
|
||
```
|
||
substituting the exact key name printed in the error. See §6 for why this
|
||
must run as `ci-runner`, not as your own login.
|
||
|
||
**Dashboard is blank / cards show "unavailable"**
|
||
: The VM backend (`vmrun`) or `shutil.disk_usage`/`systemctl` could not be
|
||
reached from the host — the dashboard degrades a card to an "unavailable"
|
||
notice instead of a 500. Check `vmware-host.service` is up
|
||
(`systemctl is-active vmware-host.service`) and that `/healthz` still
|
||
returns `200`.
|
||
|
||
**`ci web serve` refuses to start with "refusing to bind ... without
|
||
authentication"**
|
||
: You passed a non-loopback `--host` (or `[web].bind_host` in
|
||
`config.toml`) while `[web.oidc]` is unset. This is the guard described
|
||
in §5 — configure `[web.oidc]` (see §3) or bind `127.0.0.1`.
|
||
|
||
**Confirm/Cleanup/Purge form returns 403**
|
||
: The CSRF token expired (the session cookie was cleared, e.g. by
|
||
`/logout`, or the browser tab was left open across a service restart
|
||
that rotated the session secret in open mode). Reload the page and retry
|
||
— the confirm form always mints a fresh token.
|