local-ci-cd-system/docs/WEB-UI.md

Web UI — Management Dashboard

Server-rendered dashboard (ci web serve) for the local CI/CD system: recent CI executions with their live VMs (§8), cargo cache, disk, and act_runner status, plus confirm-gated destructive actions (VM remove/cleanup, cache purge, template backup) with an audit trail. In-process only — it imports operations/ and backends.load_backend directly and never shells out to the CLI.

See plans/web-ui-management-plan.md (steps W0–W7) for the design history and config.example.toml's [web] / [web.oidc] section for every config key referenced below.

---

0. Current deployment on this host

Live since 2026-07-23: ci-web.service is enabled + running, bound to the LAN (192.168.10.11:8800), authenticating against https://gitea.emulab.it via OIDC. Only the allowlisted Gitea user(s) in /var/lib/ci/config.toml's [web].allowed_users can log in. Manage it exactly as described in §4 below (systemctl status/restart ci-web.service, journalctl -u ci-web.service -f). WebSessionSecret and WebOidcClientSecret are already stored in the ci-runner keyring — no action needed unless they're rotated (§3/§6).

---

1. Prerequisites

• The Linux host already bootstrapped by setup-host-linux.sh (ci-runner service user, /opt/ci/venv, /var/lib/ci/config.toml).
• A reachable Gitea instance to authenticate against (OIDC/OAuth2) — only required if you intend to bind the dashboard on the LAN; a loopback-only install can skip Gitea entirely (see §5, "open mode").
• Outbound network from the host to Gitea's /.well-known/openid-configuration endpoint, if OIDC is enabled.

Nothing here needs internet access — ci_orchestrator[web]'s dependencies (FastAPI, Jinja2, Authlib, uvicorn, ...) install from the same package index already used for the core install, and every frontend asset (web/static/app.css, web/static/htmx.min.js) is vendored in the repo, no CDN.

---

2. Running the installer

cd /opt/ci/local-ci-cd-system
sudo bash setup-web-linux.sh --host 127.0.0.1        # loopback-only, no OIDC needed
# or, once OIDC is registered (see §3):
sudo bash setup-web-linux.sh --host 192.168.1.50      # LAN

setup-web-linux.sh is idempotent — every mutating step checks its own state first and skips (with a message) when already satisfied, so re-running it (e.g. after git pull + a code change) is always safe:

1. pip install '<repo>[web]' into /opt/ci/venv, non-editable — per CLAUDE.md, the production venv can't resolve editable installs when act_runner/systemd launches Python from a transient working directory.
2. Renders deploy/systemd/ci-web.service into /etc/systemd/system/, substituting the bind IP you passed via --host.
3. Appends a commented [web] / [web.oidc] template to /var/lib/ci/config.toml if no such section exists yet — it never overwrites an existing one, and never writes a secret value.
4. Prompts (interactively, one at a time) to store WebSessionSecret and WebOidcClientSecret in the keyring, if they aren't already set. The secret is only ever typed at the terminal prompt — the installer never accepts it as a command-line argument and never echoes or logs it.
5. Prints the manual Gitea OIDC application registration steps (see §3).
6. systemctl daemon-reload then enables + starts (or restarts, if already running) ci-web.service.

Run setup-web-linux.sh --help for the full flag list (--skip-pip, --skip-secrets, --config-path).

After step 3/4, edit /var/lib/ci/config.toml: uncomment the appended [web] / [web.oidc] block, fill in issuer, client_id, allowed_users, then restart the service (systemctl restart ci-web.service).

---

3. Gitea OAuth2 application registration

The dashboard authenticates against Gitea via OIDC (Authlib + Gitea's /.well-known/openid-configuration discovery document). Register the application before enabling [web.oidc]:

1. In Gitea: Settings → Applications → "Manage OAuth2 Applications" → Create a new OAuth2 Application.
2. Application Name: anything recognizable, e.g. ci-web.
3. Redirect URI: http://<LAN-IP>:8800/auth/callback — this must match [web.oidc].redirect_url exactly (scheme, host, port, and path). A mismatch is the #1 cause of a login redirect loop (see §6).
4. Gitea shows a Client ID and Client Secret exactly once:
   • Client ID → [web.oidc].client_id in config.toml (not a secret, plain text is fine).
   • Client Secret → never put it in config.toml. Store it in the keyring instead:

     sudo -u ci-runner /opt/ci/venv/bin/python -m ci_orchestrator creds set \
         --target WebOidcClientSecret --user web-oidc-client
     

     config.toml only ever holds the keyring key name (client_secret_cred = "WebOidcClientSecret"), never the value.
5. Add the Gitea username(s) allowed to log in to [web].allowed_users. The allowlist is fail-closed: an empty list denies everyone, it never means "allow all".
6. For the Executions card (§8), add an API read scope to [web.oidc].scopes — read:repository, plus read:admin if the account is a Gitea admin and you want runs from every repo on the instance. New scopes take effect after one fresh login (Gitea shows the consent screen again); existing sessions keep working with the old scopes until then.

---

4. Operating

# Start / stop / restart
sudo systemctl start ci-web.service
sudo systemctl stop ci-web.service
sudo systemctl restart ci-web.service     # after a config.toml edit or a code upgrade

# Status
systemctl status ci-web.service

# Logs (uvicorn access log + app stdout/stderr go to the journal)
journalctl -u ci-web.service -f
journalctl -u ci-web.service -n 100 --no-pager

# Health check (unauthenticated, always 200). Use the address the service is
# bound to -- uvicorn listens on `[web].bind_host` only, so on this host
# (192.168.10.11) a loopback request gets connection-refused, not a 200.
curl -s http://192.168.10.11:8800/healthz

Once code changes: sudo bash setup-web-linux.sh --host <same-as-before> reinstalls the [web] extra and restarts the service in one step (steps 2–6 no-op if nothing else changed).

---

5. Firewall / security

• Bind only a specific LAN IP, never 0.0.0.0. Neither ci web serve nor setup-web-linux.sh will ever write 0.0.0.0 — the installer prompts for (or accepts via --host) one explicit address, defaulting to 127.0.0.1.
• Open mode (no [web.oidc] configured) is localhost-only, enforced at startup. ci web serve refuses to start if you bind a non-loopback address while [web.oidc] is absent — an unauthenticated dashboard that can destroy VMs and purge caches must never be reachable from the LAN. Configure [web.oidc] first, or bind 127.0.0.1.
• Every mutating route (VM remove/cleanup, cache purge, template backup) is behind a synchronizer-token CSRF check and a two-step confirm gate (a preview/dry-run, then an explicit "Confirm" that actually executes) — there is no one-click destroy.
• Every mutating action appends exactly one entry to the audit log ([web].audit_log, default /var/lib/ci/logs/web-audit.jsonl), viewable at /audit in the dashboard.
• If you run a host firewall (ufw/nftables/iptables), only open the chosen port (default 8800) to the specific LAN/VLAN that needs dashboard access, not the whole network.

---

6. Keyring / headless note

ci-web.service runs as ci-runner under systemd — there is no desktop session and no D-Bus, so the GNOME Secret Service backend (secret-tool) does not work here. This host uses keyrings.alt.file.PlaintextKeyring instead (already the credential backend used for guest/Gitea credentials — see plans/archived/2026-05-23/B3-closeout.md):

• Config: /home/ci-runner/.local/share/python_keyring/keyringrc.cfg ([backend] default-keyring=keyrings.alt.file.PlaintextKeyring)
• Store: /home/ci-runner/.local/share/python_keyring/keyring_pass.cfg — plain text, readable only by ci-runner (file permissions), no D-Bus session required. Verified to work both from sudo -u ci-runner ... and from inside a plain systemd service.
• This is why ci-web.service's hardening uses ProtectHome=read-only and not ProtectHome=yes/true: the latter hides /home entirely from the unit's mount namespace, which would break every OIDC/session secret lookup and make login fail even though the keyring itself is fine. read-only keeps /home/ci-runner readable (the app never writes to the keyring at request time — secrets are set once, out-of-band, by ci creds set) while still blocking writes.
• WebSessionSecret / WebOidcClientSecret are keyring key names in config.toml (session_secret_cred, client_secret_cred) — the actual values live only in the keyring, set via ci creds set --target <name> --user <label> (interactive prompt, hide_input + confirmation; never a CLI argument, never logged).

---

7. Service management (Stage D)

The dashboard's Services panel lets an authenticated web user start, stop, or restart a fixed allow-list of systemd units on the host — the same confirm-gated, CSRF-protected, audited mutating-action shape as VM remove/cleanup, cache purge, and template backup (§5), applied to sudo systemctl.

Allow-list (default, operations/services.py::MANAGED_SERVICES):

Slug	Unit	Stop allowed?
act-runner	act-runner.service	yes
act-runner-rust	act-runner-rust.service	yes
vmware-host	vmware-host.service	yes
ci-web	ci-web.service	no — restart-only

Override it with [web].managed_services in config.toml (see config.example.toml's commented example) — a list of {id, unit, label, allow_stop} tables. A malformed row is skipped, never crashes startup.

Security model

• Allow-list by slug, not by unit name. The URL is POST /services/{service_id}/{action} where service_id is one of the slugs above (find_service()), never a raw systemd unit name — there is no code path that builds a controllable unit from unvalidated request input.
• Exact-command sudoers, no wildcards. deploy/sudoers.d/ci-web grants ci-runner passwordless sudo for exactly nine literal command lines (one per unit × action, plus the ci-web restart special case) — no bare systemctl, no *, no ALL in the command position. Even if the web app itself were compromised, ci-runner's sudo grant cannot be used to run any command beyond those nine lines.
• Arg list, never a shell. operations/services.py::control_service builds ["sudo", "-n", "systemctl", action, unit] (or the systemd-run variant below) as a Python list handed straight to subprocess.run — shell=True is never used anywhere in this path.
• ci-web is restart-only. allow_stop=False — a stop request against it is refused with a 400 before any subprocess or audit entry, both at the route (web/routes.py::services_control) and defensively again inside control_service itself. Rationale: stopping the service that renders this very panel would lock the operator out with no way to start it again from the UI.
• ci-web restart is deferred. Restarting ci-web.service from inside its own worker would kill the HTTP response before it could be sent, so that one case runs sudo -n systemd-run --on-active=2 systemctl restart ci-web.service instead of a direct systemctl restart — the systemd-run call itself returns almost immediately (it only schedules the restart ~2s out), so the confirming request completes normally and the dashboard briefly drops a couple of seconds later.
• vmware-host stop/restart interrupts running jobs. The confirm step shows the current count of active VMs ("N active VM(s) — this interrupts running jobs") when stopping or restarting vmware-host.service, computed from the same VM backend the Executions card uses — best-effort, degrades to no warning if the backend is unavailable, never blocks the action.
• Every action is audited exactly once per stage (service.start / service.stop / service.restart, target = the unit name), same as every other mutating route — visible at /audit.

Sudoers install

setup-web-linux.sh installs deploy/sudoers.d/ci-web to /etc/sudoers.d/ci-web (mode 0440, owner root:root) as step 6/7, validating it with visudo -cf and removing it again if validation fails — a broken sudoers.d fragment can lock out sudo entirely, so this step never leaves an unvalidated file in place. Before installing on a new host, confirm systemctl/systemd-run are actually at /usr/bin on that host (command -v systemctl systemd-run) — edit both deploy/sudoers.d/ci-web and the installed copy if not.

To install just the sudoers grant (e.g. after editing deploy/sudoers.d/ci-web) without touching anything else:

sudo bash setup-web-linux.sh --skip-pip --skip-secrets

Deploy note

After merging a Stage D change: re-run pip install . into the production venv (per CLAUDE.md), re-run setup-web-linux.sh (installs/updates the sudoers grant and restarts ci-web.service), then confirm the sudoers file on the live host with sudo visudo -c before relying on the panel.

---

8. Executions card (recent Gitea Actions runs)

The dashboard's top wide table is Executions: the most recent workflow runs, the live ones included, each with its status, workflow, branch/commit, trigger, duration, and the build VM(s) that run spawned (with their Remove button). It replaced the old read-only "Live VMs" table — a VM alone could never answer "did the last build pass?".

Runs come from Gitea's Actions API: /api/v1/admin/actions/runs (every repo on the instance, admin token) with /api/v1/user/actions/runs as the non-admin fallback. Results are cached for 10s, so the card's 5s refresh costs Gitea at most one call per 10s window; the VM columns are re-read on every refresh.

Rows are ordered newest first (a queued run, which has no start time yet, sorts above everything; ties break on Gitea's run id). The Show selector in the card's header picks how many to list — 10 / 25 / 50 / 100, default from [web].executions_limit (10). The choice rides on the card's own polling URL (/executions?limit=N), so it survives every 5s refresh with no client-side state; a value set in the config outside those four is offered as an extra option rather than dropped.

Where the API token comes from. Normally nowhere extra: the dashboard reuses the access token from your own OIDC login, so what the card shows is exactly what your Gitea account can see. That token is kept server-side only (an in-process map keyed by username) and dropped on logout — it is never written into the session cookie, which is signed but not encrypted and travels over plain HTTP on the LAN. An expired token is refreshed once automatically; a refusal that refresh cannot fix shows a "log in again" link in the card.

Optional fallback: store a Gitea PAT in the keyring and name it in [web].gitea_token_cred. It is used when no logged-in token is available — after a service restart (which empties the in-process map while session cookies stay valid) — and when Gitea refuses the login token (401, or 403 for insufficient scope), which is what an OIDC token limited to openid profile gets. Set it exactly like the other secrets:

sudo -u ci-runner /opt/ci/venv/bin/python -m ci_orchestrator creds set \
    --target GiteaApiToken --user gitea-api

Degraded states never take the card down: no token, an expired token, or an unreachable Gitea each render a warning notice and still list every running VM below it, so the dashboard is never less useful than the VM table it replaced. The notice includes whatever the server said (Gitea names the missing scope in its own body), which is the fastest way to tell a token problem from something in front of Gitea.

Gotcha — Cloudflare in front of Gitea. gitea.emulab.it is proxied by Cloudflare, which rejects urllib's default Python-urllib/3.x user agent with 403 + error code: 1010 before the request reaches Gitea — indistinguishable from a token refusal if you only look at the status code. operations/executions.py therefore always sends an explicit User-Agent; any ordinary agent string passes. Do not remove that header.

Orphaned VMs are not handled in this card. A powered-on VM whose run is no longer in the window (or that no run ever claimed) keeps a row of its own flagged no run, with its Remove button — but the orphan sweep lives in the Maintenance card ("Cleanup orphaned VMs") and the orphan count in the status-header VMs tile. Those two are the places to look at orphans; the Executions card only avoids hiding them.

---

9. Troubleshooting

Login loop (bounces back to /login after authenticating with Gitea)

The registered Redirect URI in Gitea does not exactly match [web.oidc].redirect_url in config.toml — check scheme (http vs https), host/IP, port, and the /auth/callback path character-for-character.

403 after a successful Gitea login

Your Gitea username is not in [web].allowed_users (or the list is empty — the allowlist is fail-closed by design). Add it and restart the service.

**"keyring key '...' not found -- run ci creds set ..." in the journal /

on /login**

The session secret or the OIDC client secret has not been stored yet (or was stored for a different Unix user than ci-runner, or under a different _cred name than the one in config.toml). Re-run:

sudo -u ci-runner /opt/ci/venv/bin/python -m ci_orchestrator creds set \
    --target WebSessionSecret --user web-session

substituting the exact key name printed in the error. See §6 for why this must run as ci-runner, not as your own login.

Dashboard is blank / cards show "unavailable"

The VM backend (vmrun) or shutil.disk_usage/systemctl could not be reached from the host — the dashboard degrades a card to an "unavailable" notice instead of a 500. Check vmware-host.service is up (systemctl is-active vmware-host.service) and that /healthz still returns 200.

**ci web serve refuses to start with "refusing to bind ... without

authentication"**

You passed a non-loopback --host (or [web].bind_host in config.toml) while [web.oidc] is unset. This is the guard described in §5 — configure [web.oidc] (see §3) or bind 127.0.0.1.

Confirm/Cleanup/Purge form returns 403

The CSRF token expired (the session cookie was cleared, e.g. by /logout, or the browser tab was left open across a service restart that rotated the session secret in open mode). Reload the page and retry — the confirm form always mints a fresh token.