Simone
e56eae95c7
Adds operations/services.py: a fixed ManagedService allow-list (act-runner, act-runner-rust, vmware-host, ci-web) with find_service/service_states/control_service. control_service builds an argument list only (never shell=True), validates the action against ALLOWED_ACTIONS (ValueError->400), refuses stop on restart-only services (ci-web, allow_stop=False) without building a command, and defers ci-web's own restart via 'sudo -n systemd-run --on-active=2 systemctl restart ci-web.service' so the HTTP response returns first.
Route POST /services/{service_id}/{action} in web/routes.py mirrors vms_remove: require_user + require_csrf + two-step confirm gate + one audit entry per stage; unknown id / bad action / disallowed-stop short-circuit to 400 before any subprocess or audit; the vmware-host confirm step warns with the active-VM count. New /services read partial with state-aware Start/Stop/Restart buttons, polled every 15s.
deploy/sudoers.d/ci-web grants ci-runner exact-command NOPASSWD for exactly these nine systemctl lines plus the systemd-run deferral — no wildcards, no bare systemctl, no ci-web stop; setup-web-linux.sh installs it mode 0440 and validates with 'visudo -cf', rolling back on failure. Config gains [web].managed_services override; docs/WEB-UI.md documents the panel + security model.
Gates: 1044 tests pass (46 new security/unit tests), coverage 94.20%, ruff + mypy --strict clean. Not yet deployed: sudoers install + prod-venv reinstall + ci-web restart are a human privilege-boundary step.
317 lines
15 KiB
Markdown
317 lines
15 KiB
Markdown
# Web UI — Management Dashboard
|
||
|
||
Server-rendered dashboard (`ci web serve`) for the local CI/CD system: live
|
||
VMs, cargo cache, disk, and act_runner status, plus confirm-gated destructive
|
||
actions (VM remove/cleanup, cache purge, template backup) with an audit
|
||
trail. In-process only — it imports `operations/` and `backends.load_backend`
|
||
directly and never shells out to the CLI.
|
||
|
||
See `plans/web-ui-management-plan.md` (steps W0–W7) for the design history
|
||
and `config.example.toml`'s `[web]` / `[web.oidc]` section for every
|
||
config key referenced below.
|
||
|
||
---
|
||
|
||
## 0. Current deployment on this host
|
||
|
||
Live since 2026-07-23: `ci-web.service` is enabled + running, bound to the
|
||
LAN (`192.168.10.11:8800`), authenticating against `https://gitea.emulab.it`
|
||
via OIDC. Only the allowlisted Gitea user(s) in `/var/lib/ci/config.toml`'s
|
||
`[web].allowed_users` can log in. Manage it exactly as described in §4 below
|
||
(`systemctl status/restart ci-web.service`, `journalctl -u ci-web.service -f`).
|
||
`WebSessionSecret` and `WebOidcClientSecret` are already stored in the
|
||
`ci-runner` keyring — no action needed unless they're rotated (§3/§6).
|
||
|
||
---
|
||
|
||
## 1. Prerequisites
|
||
|
||
- The Linux host already bootstrapped by `setup-host-linux.sh` (`ci-runner`
|
||
service user, `/opt/ci/venv`, `/var/lib/ci/config.toml`).
|
||
- A reachable Gitea instance to authenticate against (OIDC/OAuth2) — only
|
||
required if you intend to bind the dashboard on the LAN; a loopback-only
|
||
install can skip Gitea entirely (see §5, "open mode").
|
||
- Outbound network from the host to Gitea's `/.well-known/openid-configuration`
|
||
endpoint, if OIDC is enabled.
|
||
|
||
Nothing here needs internet access — `ci_orchestrator[web]`'s dependencies
|
||
(FastAPI, Jinja2, Authlib, uvicorn, ...) install from the same package index
|
||
already used for the core install, and every frontend asset
|
||
(`web/static/app.css`, `web/static/htmx.min.js`) is vendored in the repo, no
|
||
CDN.
|
||
|
||
---
|
||
|
||
## 2. Running the installer
|
||
|
||
```bash
|
||
cd /opt/ci/local-ci-cd-system
|
||
sudo bash setup-web-linux.sh --host 127.0.0.1 # loopback-only, no OIDC needed
|
||
# or, once OIDC is registered (see §3):
|
||
sudo bash setup-web-linux.sh --host 192.168.1.50 # LAN
|
||
```
|
||
|
||
`setup-web-linux.sh` is idempotent — every mutating step checks its own
|
||
state first and skips (with a message) when already satisfied, so re-running
|
||
it (e.g. after `git pull` + a code change) is always safe:
|
||
|
||
1. `pip install '<repo>[web]'` into `/opt/ci/venv`, **non-editable** — per
|
||
`CLAUDE.md`, the production venv can't resolve editable installs when
|
||
`act_runner`/systemd launches Python from a transient working directory.
|
||
2. Renders `deploy/systemd/ci-web.service` into `/etc/systemd/system/`,
|
||
substituting the bind IP you passed via `--host`.
|
||
3. Appends a **commented** `[web]` / `[web.oidc]` template to
|
||
`/var/lib/ci/config.toml` if no such section exists yet — it never
|
||
overwrites an existing one, and never writes a secret value.
|
||
4. Prompts (interactively, one at a time) to store `WebSessionSecret` and
|
||
`WebOidcClientSecret` in the keyring, if they aren't already set. The
|
||
secret is only ever typed at the terminal prompt — the installer never
|
||
accepts it as a command-line argument and never echoes or logs it.
|
||
5. Prints the manual Gitea OIDC application registration steps (see §3).
|
||
6. `systemctl daemon-reload` then enables + starts (or restarts, if already
|
||
running) `ci-web.service`.
|
||
|
||
Run `setup-web-linux.sh --help` for the full flag list (`--skip-pip`,
|
||
`--skip-secrets`, `--config-path`).
|
||
|
||
After step 3/4, edit `/var/lib/ci/config.toml`: uncomment the appended
|
||
`[web]` / `[web.oidc]` block, fill in `issuer`, `client_id`,
|
||
`allowed_users`, then restart the service (`systemctl restart ci-web.service`).
|
||
|
||
---
|
||
|
||
## 3. Gitea OAuth2 application registration
|
||
|
||
The dashboard authenticates against Gitea via OIDC (Authlib + Gitea's
|
||
`/.well-known/openid-configuration` discovery document). Register the
|
||
application **before** enabling `[web.oidc]`:
|
||
|
||
1. In Gitea: **Settings → Applications → "Manage OAuth2 Applications" →
|
||
Create a new OAuth2 Application**.
|
||
2. **Application Name**: anything recognizable, e.g. `ci-web`.
|
||
3. **Redirect URI**: `http://<LAN-IP>:8800/auth/callback` — this **must
|
||
match `[web.oidc].redirect_url` exactly** (scheme, host, port, and path).
|
||
A mismatch is the #1 cause of a login redirect loop (see §6).
|
||
4. Gitea shows a **Client ID** and **Client Secret** exactly once:
|
||
- Client ID → `[web.oidc].client_id` in `config.toml` (not a secret, plain text is fine).
|
||
- Client Secret → **never put it in `config.toml`**. Store it in the
|
||
keyring instead:
|
||
```bash
|
||
sudo -u ci-runner /opt/ci/venv/bin/python -m ci_orchestrator creds set \
|
||
--target WebOidcClientSecret --user web-oidc-client
|
||
```
|
||
`config.toml` only ever holds the keyring *key name*
|
||
(`client_secret_cred = "WebOidcClientSecret"`), never the value.
|
||
5. Add the Gitea username(s) allowed to log in to `[web].allowed_users`.
|
||
**The allowlist is fail-closed**: an empty list denies everyone, it
|
||
never means "allow all".
|
||
|
||
---
|
||
|
||
## 4. Operating
|
||
|
||
```bash
|
||
# Start / stop / restart
|
||
sudo systemctl start ci-web.service
|
||
sudo systemctl stop ci-web.service
|
||
sudo systemctl restart ci-web.service # after a config.toml edit or a code upgrade
|
||
|
||
# Status
|
||
systemctl status ci-web.service
|
||
|
||
# Logs (uvicorn access log + app stdout/stderr go to the journal)
|
||
journalctl -u ci-web.service -f
|
||
journalctl -u ci-web.service -n 100 --no-pager
|
||
|
||
# Health check (unauthenticated, always 200)
|
||
curl -s http://127.0.0.1:8800/healthz
|
||
```
|
||
|
||
Once code changes: `sudo bash setup-web-linux.sh --host <same-as-before>`
|
||
reinstalls the `[web]` extra and restarts the service in one step (steps
|
||
2–6 no-op if nothing else changed).
|
||
|
||
---
|
||
|
||
## 5. Firewall / security
|
||
|
||
- **Bind only a specific LAN IP, never `0.0.0.0`.** Neither `ci web serve`
|
||
nor `setup-web-linux.sh` will ever write `0.0.0.0` — the installer prompts
|
||
for (or accepts via `--host`) one explicit address, defaulting to
|
||
`127.0.0.1`.
|
||
- **Open mode (no `[web.oidc]` configured) is localhost-only, enforced at
|
||
startup.** `ci web serve` refuses to start if you bind a non-loopback
|
||
address while `[web.oidc]` is absent — an unauthenticated dashboard that
|
||
can destroy VMs and purge caches must never be reachable from the LAN.
|
||
Configure `[web.oidc]` first, or bind `127.0.0.1`.
|
||
- Every mutating route (VM remove/cleanup, cache purge, template backup) is
|
||
behind a synchronizer-token CSRF check and a two-step confirm gate (a
|
||
preview/dry-run, then an explicit "Confirm" that actually executes) —
|
||
there is no one-click destroy.
|
||
- Every mutating action appends exactly one entry to the audit log
|
||
(`[web].audit_log`, default `/var/lib/ci/logs/web-audit.jsonl`), viewable
|
||
at `/audit` in the dashboard.
|
||
- If you run a host firewall (ufw/nftables/iptables), only open the chosen
|
||
port (default `8800`) to the specific LAN/VLAN that needs dashboard
|
||
access, not the whole network.
|
||
|
||
---
|
||
|
||
## 6. Keyring / headless note
|
||
|
||
`ci-web.service` runs as `ci-runner` under systemd — there is no desktop
|
||
session and no D-Bus, so the GNOME Secret Service backend (`secret-tool`)
|
||
does not work here. This host uses **`keyrings.alt.file.PlaintextKeyring`**
|
||
instead (already the credential backend used for guest/Gitea credentials —
|
||
see `plans/archived/2026-05-23/B3-closeout.md`):
|
||
|
||
- Config: `/home/ci-runner/.local/share/python_keyring/keyringrc.cfg`
|
||
(`[backend] default-keyring=keyrings.alt.file.PlaintextKeyring`)
|
||
- Store: `/home/ci-runner/.local/share/python_keyring/keyring_pass.cfg` —
|
||
plain text, readable only by `ci-runner` (file permissions), no D-Bus
|
||
session required. Verified to work both from `sudo -u ci-runner ...` and
|
||
from inside a plain systemd service.
|
||
- This is why `ci-web.service`'s hardening uses `ProtectHome=read-only`
|
||
and **not** `ProtectHome=yes`/`true`: the latter hides `/home` entirely
|
||
from the unit's mount namespace, which would break every OIDC/session
|
||
secret lookup and make login fail even though the keyring itself is
|
||
fine. `read-only` keeps `/home/ci-runner` readable (the app never writes
|
||
to the keyring at request time — secrets are set once, out-of-band, by
|
||
`ci creds set`) while still blocking writes.
|
||
- `WebSessionSecret` / `WebOidcClientSecret` are keyring key *names* in
|
||
`config.toml` (`session_secret_cred`, `client_secret_cred`) — the actual
|
||
values live only in the keyring, set via `ci creds set --target <name>
|
||
--user <label>` (interactive prompt, `hide_input` + confirmation; never
|
||
a CLI argument, never logged).
|
||
|
||
---
|
||
|
||
## 7. Service management (Stage D)
|
||
|
||
The dashboard's **Services** panel lets an authenticated web user start, stop,
|
||
or restart a fixed allow-list of systemd units on the host — the same
|
||
confirm-gated, CSRF-protected, audited mutating-action shape as VM
|
||
remove/cleanup, cache purge, and template backup (§5), applied to `sudo
|
||
systemctl`.
|
||
|
||
**Allow-list (default, `operations/services.py::MANAGED_SERVICES`):**
|
||
|
||
| Slug | Unit | Stop allowed? |
|
||
| --- | --- | --- |
|
||
| `act-runner` | `act-runner.service` | yes |
|
||
| `act-runner-rust` | `act-runner-rust.service` | yes |
|
||
| `vmware-host` | `vmware-host.service` | yes |
|
||
| `ci-web` | `ci-web.service` | **no** — restart-only |
|
||
|
||
Override it with `[web].managed_services` in `config.toml` (see
|
||
`config.example.toml`'s commented example) — a list of `{id, unit, label,
|
||
allow_stop}` tables. A malformed row is skipped, never crashes startup.
|
||
|
||
### Security model
|
||
|
||
- **Allow-list by slug, not by unit name.** The URL is
|
||
`POST /services/{service_id}/{action}` where `service_id` is one of the
|
||
slugs above (`find_service()`), never a raw systemd unit name — there is no
|
||
code path that builds a controllable unit from unvalidated request input.
|
||
- **Exact-command sudoers, no wildcards.** `deploy/sudoers.d/ci-web` grants
|
||
`ci-runner` passwordless `sudo` for exactly nine literal command lines (one
|
||
per unit × action, plus the `ci-web` restart special case) — no bare
|
||
`systemctl`, no `*`, no `ALL` in the command position. Even if the web app
|
||
itself were compromised, `ci-runner`'s sudo grant cannot be used to run any
|
||
command beyond those nine lines.
|
||
- **Arg list, never a shell.** `operations/services.py::control_service`
|
||
builds `["sudo", "-n", "systemctl", action, unit]` (or the `systemd-run`
|
||
variant below) as a Python list handed straight to `subprocess.run` —
|
||
`shell=True` is never used anywhere in this path.
|
||
- **`ci-web` is restart-only.** `allow_stop=False` — a `stop` request against
|
||
it is refused with a 400 *before* any subprocess or audit entry, both at
|
||
the route (`web/routes.py::services_control`) and defensively again inside
|
||
`control_service` itself. Rationale: stopping the service that renders this
|
||
very panel would lock the operator out with no way to start it again from
|
||
the UI.
|
||
- **`ci-web` restart is deferred.** Restarting `ci-web.service` from inside
|
||
its own worker would kill the HTTP response before it could be sent, so
|
||
that one case runs `sudo -n systemd-run --on-active=2 systemctl restart
|
||
ci-web.service` instead of a direct `systemctl restart` — the
|
||
`systemd-run` call itself returns almost immediately (it only schedules
|
||
the restart ~2s out), so the confirming request completes normally and the
|
||
dashboard briefly drops a couple of seconds later.
|
||
- **`vmware-host` stop/restart interrupts running jobs.** The confirm step
|
||
shows the current count of active VMs ("N active VM(s) — this interrupts
|
||
running jobs") when stopping or restarting `vmware-host.service`, computed
|
||
from the same VM backend the Live VMs panel uses — best-effort, degrades
|
||
to no warning if the backend is unavailable, never blocks the action.
|
||
- **Every action is audited exactly once per stage** (`service.start` /
|
||
`service.stop` / `service.restart`, `target` = the unit name), same as
|
||
every other mutating route — visible at `/audit`.
|
||
|
||
### Sudoers install
|
||
|
||
`setup-web-linux.sh` installs `deploy/sudoers.d/ci-web` to
|
||
`/etc/sudoers.d/ci-web` (mode `0440`, owner `root:root`) as step 6/7,
|
||
validating it with `visudo -cf` and **removing it again if validation
|
||
fails** — a broken `sudoers.d` fragment can lock out `sudo` entirely, so this
|
||
step never leaves an unvalidated file in place. Before installing on a new
|
||
host, confirm `systemctl`/`systemd-run` are actually at `/usr/bin` on that
|
||
host (`command -v systemctl systemd-run`) — edit both
|
||
`deploy/sudoers.d/ci-web` and the installed copy if not.
|
||
|
||
To install just the sudoers grant (e.g. after editing
|
||
`deploy/sudoers.d/ci-web`) without touching anything else:
|
||
|
||
```bash
|
||
sudo bash setup-web-linux.sh --skip-pip --skip-secrets
|
||
```
|
||
|
||
### Deploy note
|
||
|
||
After merging a Stage D change: re-run `pip install .` into the production
|
||
venv (per `CLAUDE.md`), re-run `setup-web-linux.sh` (installs/updates the
|
||
sudoers grant and restarts `ci-web.service`), then confirm the sudoers file
|
||
on the live host with `sudo visudo -c` before relying on the panel.
|
||
|
||
---
|
||
|
||
## 8. Troubleshooting
|
||
|
||
**Login loop (bounces back to `/login` after authenticating with Gitea)**
|
||
: The registered Redirect URI in Gitea does not exactly match
|
||
`[web.oidc].redirect_url` in `config.toml` — check scheme (`http` vs
|
||
`https`), host/IP, port, and the `/auth/callback` path character-for-character.
|
||
|
||
**403 after a successful Gitea login**
|
||
: Your Gitea username is not in `[web].allowed_users` (or the list is
|
||
empty — the allowlist is fail-closed by design). Add it and restart the
|
||
service.
|
||
|
||
**"keyring key '...' not found -- run `ci creds set ...`" in the journal /
|
||
on `/login`**
|
||
: The session secret or the OIDC client secret has not been stored yet
|
||
(or was stored for a different Unix user than `ci-runner`, or under a
|
||
different `_cred` name than the one in `config.toml`). Re-run:
|
||
```bash
|
||
sudo -u ci-runner /opt/ci/venv/bin/python -m ci_orchestrator creds set \
|
||
--target WebSessionSecret --user web-session
|
||
```
|
||
substituting the exact key name printed in the error. See §6 for why this
|
||
must run as `ci-runner`, not as your own login.
|
||
|
||
**Dashboard is blank / cards show "unavailable"**
|
||
: The VM backend (`vmrun`) or `shutil.disk_usage`/`systemctl` could not be
|
||
reached from the host — the dashboard degrades a card to an "unavailable"
|
||
notice instead of a 500. Check `vmware-host.service` is up
|
||
(`systemctl is-active vmware-host.service`) and that `/healthz` still
|
||
returns `200`.
|
||
|
||
**`ci web serve` refuses to start with "refusing to bind ... without
|
||
authentication"**
|
||
: You passed a non-loopback `--host` (or `[web].bind_host` in
|
||
`config.toml`) while `[web.oidc]` is unset. This is the guard described
|
||
in §5 — configure `[web.oidc]` (see §3) or bind `127.0.0.1`.
|
||
|
||
**Confirm/Cleanup/Purge form returns 403**
|
||
: The CSRF token expired (the session cookie was cleared, e.g. by
|
||
`/logout`, or the browser tab was left open across a service restart
|
||
that rotated the session secret in open mode). Reload the page and retry
|
||
— the confirm form always mints a fresh token.
|