The backups <details> carried hx-preserve="true" to survive the card's
30s poll without collapsing if the operator had it open -- but hx-preserve
freezes the whole node, so "newest X ago" inside it never updated either,
same class of bug as the Started column before it got its own JS-side
toggle state. Drop hx-preserve; track open/closed the same way (stable id,
state re-applied after every swap) so the count/age text stays live.
uptime_sec came from the clone directory's own mtime, but a running VM's
hypervisor can rewrite lock files under that same directory at any point,
resetting the mtime and pinning "up" near 0s for the VM's entire life --
worst exactly while a build is actively running, since that's when the VM
is busiest. The clone name already carries its own creation timestamp
(_make_clone_name, parsed back via parse_clone_name), set once and never
disturbed after -- prefer it, falling back to the old mtime read only for
a name that doesn't parse.
template backup ran a full multi-GB, multi-minute 7z pass over every
template every night regardless of whether anything under it had actually
changed -- the systemd unit's own comment already noted the resulting
archives were byte-identical run to run. Now compares the template dir's
newest file mtime against its most recent existing backup and skips the
compress step (still honoring --keep-count via prune) when nothing is
newer. --force opts back into an unconditional backup.
run.number only counts within its own repo/workflow -- meaningless once
the admin endpoint mixes every repo into one list, and already not what
the row order is sorted by. run_id is Gitea's actual instance-wide,
strictly-increasing id, so the displayed number now agrees with "most
recent globally". The link still goes to Gitea's own per-repo run page
(html_url); only the label text changed.
number is a per-repo/workflow counter with no meaning once the admin
endpoint mixes every repo on the instance into one feed -- a rarely-run
workflow's low #53 that just fired outranked a busier workflow's #142 from
hours earlier, but sorting by number buried it instead. Sort by started
(globally comparable) with run_id/attempt as tiebreaks; a re-run bumping
started_at back to the top is correct, not a bug, under "most recent
globally".
htmx's outerHTML swap (every polling card, including Executions) removes
the original target node from the DOM as part of the swap, but
htmx:afterSwap's eventInfo.target still points at that dead node -- so
restoreTimeToggles() was querying a detached subtree every poll and never
touching the live buttons. Fall back to the native event target, which
htmx dispatches on the freshly-inserted content, whenever detail.target
is no longer connected.
base.html stamps the same asset_version() value onto both app.css and
app.js's <link>/<script> tags, but the value was computed from app.css's
mtime alone. A JS-only deploy left it unchanged, so app.js's URL stayed
byte-identical to the previous deploy -- nothing forced a browser holding
that URL cached to refetch it.
Extracted into _static_asset_version(web_dir), keyed to the newer of the
two files' mtimes, and unit-tested directly (a JS-only mtime bump now
changes the value; both-missing degrades to "0").
The card's own htmx poll re-renders the whole table from the server every
5s, always starting fresh at "relative" -- so a toggled-to-absolute cell
snapped back on the next poll. Each button now has a stable id
(started-{run_id}-{attempt}, or started-vm-{vmx} on an unmatched-VM row);
app.js remembers which format was showing per id and re-applies it to the
fresh button after every swap, rather than freezing the node itself (which
would also freeze its relative-time text, badge, duration, everything else
in that row).
Same affordance, cleaner line: solid underline tinted to the border gray at
rest (subtle, doesn't read as a broken link) and to the current text colour
on hover (more emphasis).
Both renderings are pre-rendered server-side into a <button>'s
data-relative/data-absolute attributes; app.js swaps textContent (and the
hover title, to the opposite value) on click, delegated on document.body
so it survives the card's own 5s htmx poll without needing to re-wire
after every swap. Click again to swap back.
FastAPI auto-registers these with no auth of its own -- every other data
route is individually wrapped in Depends(require_user), but these three
are wired up internally by the FastAPI() constructor and were never
covered by that. Confirmed live: all three returned 200 with zero session,
handing anyone who could reach the port a full map of every route --
VM remove, cache purge, template delete, service control included --
without logging in.
Disabled FastAPI's built-in docs_url/redoc_url/openapi_url and re-added all
three as ordinary routes behind the same require_user dependency every
other route uses. Verified against the live deployment: all three now 303
to /login instead of 200.
The previous fix (sort by run_id/attempt) traded one bug for another:
run_id is Gitea's instance-wide id, shared across every repo/workflow the
admin endpoint pulls from, not a per-workflow counter. A rarely-triggered
workflow's #53 could land a run_id between two ids from a busier workflow's
#141/#142, putting #53's row between them in the Run column even though 53
isn't close to either -- exactly the reported symptom, now reproduced as a
regression test before the fix and passing after it.
run.number is the value the Run column actually displays; sort by that,
then attempt, then run_id only to keep a genuine same-number tie (two
different workflows landing on the same number independently) stable.
Sorting by started_at let a manually re-run old workflow jump above a
genuinely newer run: the re-run's started_at bumps to now, but its run
number never changes, so the Run column stopped reading strictly
descending. run_id is assigned once at creation and never changes across
attempts, so (run_id, attempt) descending is both correct and simpler --
it also handles a queued run (no started_at yet) for free, no datetime.max
special case needed.
mypy --strict (files = ["src"]) and pytest --cov=ci_orchestrator both walk
the whole src tree, web/ included, but the lint job's venv only ever
installed .[dev] -- fastapi/starlette/uvicorn/jinja2/authlib were never
resolvable, so every symbol they touch degraded to Any under mypy (masking
real type errors, and producing a cascading "Returning Any" on
_is_htmx_request) and every tests/python/test_web_*.py would fail to import
under pytest. Install .[dev,web] instead. Verified against a fresh venv:
ruff/mypy --strict/pytest --cov all pass.
CLAUDE.md's web/app.py summary, WEB-UI.md's dashboard-layout and Executions
sections, and a new design-history spec covering the tile removal, the
compacted Host card, the dropped level-word strip, and the four cards'
shared <dl> grammar.
- Two filled button kinds, no neutral/outline third state: every action is
either "normal" (#4382c2) or destructive (#cb4b4c); border is the same
neutral gray as every table/card border, not the fill colour.
- Icons: bolder stroke site-wide (2 -> 2.5) and +1px, except inside buttons
(2.25, no size bump) where a bold icon next to now-unbolded button text
was too heavy.
- tbody row hover: flat gray -> a transparent wash of the link/accent blue.
- The Actions <td> in Services/Templates was itself a flex container; some
engines size that box's content slightly differently from a plain-text
sibling cell, which showed up as the row's collapsed bottom border sitting
1px higher under Actions than its neighbours. The flex row now lives on an
inner <div class="actions-row"> instead, with its <form> children pinned
to content width (a bare <form> is display:flex by the site's own rule,
which was letting one form claim the whole row and push the next to wrap).
- Navbar brand gets an icon (a server rack -- this dashboard orchestrates
hosts/VMs, not generic settings).
POST /host/delete-template: same two-step confirm gate, CSRF check, and
known-template containment as /host/backup, offloaded to a thread the same
way (rmtree on a multi-GB template dir is not instant). No backup-required
safety rail -- same confirm-gate as every other destructive action, nothing
more; the confirm summary spells out that clones can no longer be made from
the template until one is restored.
Also cleaned up the per-backup Delete button it sits next to: name+meta
grouped into one shrinkable unit so the row wraps as a block instead of the
delete button drifting away from a name that wrapped, and each backup is
now its own bordered line instead of a run-on flex list.
No cell stacks two facts on separate lines with <br> any more (Run/Repo,
Branch/Commit, Trigger/Actor, Duration/Started, VM/IP-Uptime each get their
own column, 8 -> 13 total). The card is already .card--wide with its own
.table-wrap, so the extra width scrolls horizontally rather than the page.
Status pills carry a check/x/play icon for the three outcomes that are a
real verdict or an in-progress state (success/failure/running); queued,
cancelled, skipped stay plain text. Every pill in the column -- including
the VM-only orphan/running pills on unmatched-VM rows -- is forced to the
same width with left-aligned content instead of a ragged column.
The dashboard used to open with six status-header tiles *and* four compact
cards restating them (runner state lived in three places at once). The tiles
are gone; each card is now the single owner of its subject, enriched with
what only the tiles had (CPU%/RAM/disk with gauges) plus what neither had
(hostname, kernel, full CPU model, the vmware-host.service row).
All four cards share the same <dl> grammar and font scale now -- VMware and
Runners were a bordered <table> at a different size/case, which read as a
visibly different style sitting next to Host and Cargo cache's plain
label/value lists.
operations/host.py also gains delete_template()/TemplateDeleteResult, used
by the template-delete route in the next commit.
It was sitting in a strip below the heading with a "Show" label and a "newest
first" hint — three lines of chrome for one control. The select now lives at the
right end of the card's own header bar (`margin-left:auto` inside the flex h2),
compact and case-normal so it reads as a control and does not grow the bar.
Label and hint are gone; the header text names the card, the values speak for
themselves, and screen readers keep the `aria-label`.
`--link` was still the brighter GitHub blue while everything else moved to
Gitea's #4183c4, so links read as a second, slightly different accent. One blue
now.
Contrast on this palette: 4.50:1 on --panel (#161718), 4.14:1 on --bg
(#1e1f20), 3.77:1 on --panel-2 (#22272e) — down from 4.79 / 4.41 / 4.01. Body
links sit on --panel and stay above the 4.5:1 AA threshold; the ones on the
darker surfaces (nav hover, table cells) land between AA-large and AA, as the
rest of this palette's dim text already does.
Adds a "Show" selector (10 / 25 / 50 / 100) to the card header and drops the
default row count from 15 to 10. The chosen value rides on the card's own
polling URL (`/executions?limit=N`), so it survives the 5s htmx refresh with no
client-side state; a config default outside those four is rendered as an extra
option instead of disappearing. The query is parsed permissively — a stale or
hand-edited polling URL clamps or falls back to the config default rather than
422-ing a card that would otherwise render.
Ordering is now explicit rather than inherited from whatever Gitea returns:
newest first, a not-yet-started (queued) run above everything, ties broken on
Gitea's run id so matrix runs sharing a timestamp keep a stable order.
Every Gitea API call from Python 403s with "error code: 1010" unless a
User-Agent is set, including endpoints that need no auth — and it looks like a
token/scope problem, which is where this went wrong once. Registered next to
the other environment traps, with the two-command check that tells them apart.
Three bugs the real API surfaced, none visible against a stubbed payload:
- Gitea here is proxied by Cloudflare, which answers urllib's default
`Python-urllib/3.x` agent with 403 + "error code: 1010" before the request
reaches Gitea — every endpoint 403s, including unauthenticated `/version`,
so it reads exactly like a token problem. Send an explicit User-Agent.
- Gitea reports `path` as either `.gitea/workflows/lint.yml` or the
`lint.yml@refs/heads/main` form; splitting on "/" alone rendered the latter
as the workflow name "main". Drop the ref first.
- Gitea reports `run_attempt: 0` for a never-retried run while the job id the
workflow builds uses github.run_attempt, which starts at 1 — so 0 must read
as 1 or no VM would ever correlate to its run. (The existing `or 1` already
did this; it now has a test naming why, against the real payload shape.)
HTTP failures now carry the server's own message (Gitea names the missing
scope; a Cloudflare interception says something else entirely) instead of a
bare status code that sent this investigation the wrong way once already.
The login token took precedence unconditionally, so a configured keyring PAT
could never help while an operator was logged in — and on this instance the
OIDC token (scopes `openid profile`) is exactly what Gitea refuses with a 403
for insufficient API scope. `ExecutionsView` now carries `refused` (401 or a
final 403) and the web layer retries once with the PAT when the token it used
was not already the PAT. A refused PAT is never sent twice, and a 401 still
gets its single refresh attempt first.
Both listing endpoints answering 403 means Gitea authenticated the token and
then rejected its scope -- exactly what an OIDC login token with only
`openid profile` gets. The card showed a bare "Gitea API returned HTTP 403",
which sends the operator hunting the wrong thing (admin rights, allowlist,
URL). Say what it is and name both fixes: add an API read scope to
`[web.oidc] scopes`, or configure the keyring PAT fallback. A 403 on the admin
listing alone still falls through to the user listing as before, and a 404
there keeps its plain message.
The dashboard could only show what was powered on this instant, so it could
not answer "did the last build pass?" — and no local execution history exists
to answer it from (`report job` reads a PowerShell-era JSONL path that nothing
writes on this host, and the recent artifact dirs are empty).
Recent runs now come from Gitea's Actions API (`operations/executions.py`):
`/api/v1/admin/actions/runs` for every repo on the instance, falling back to
`/api/v1/user/actions/runs` when the token is not admin-scoped. One list call
per refresh, cached 10s, so the card's 5s htmx poll costs Gitea at most one
call per window. Live VMs are grouped under the run that spawned them — the
job id is `{run_id}-{run_attempt}[-suffix]` and the clone name carries it — so
a matrix run collects several VMs on one row. Every failure mode (no token,
401, unreachable Gitea, malformed payload) degrades to a notice plus the plain
VM rows, each keeping its Remove button, so the card is never less useful than
the table it replaces. Orphans stay where they were: the sweep in Maintenance,
the count in the status-header tile.
Auth reuses the access token from the operator's own OIDC login, which
`auth.py` previously discarded. SECURITY: that token is kept server-side only
(`app.state.gitea_tokens`, dropped on logout) and never written into the
session — `SessionMiddleware` signs the session cookie but does not encrypt
it, and this service is deployed over plain HTTP on the LAN, so a bearer token
in there would be readable in the browser and on the wire. An expired token is
refreshed once; `[web] gitea_token_cred` names an optional keyring PAT for the
cases a login token cannot cover (service restart, non-admin operator).
`GET /vms` and `partials/vms.html` are gone; `POST /vms/{id}/remove` and
`POST /vms/cleanup` are untouched.
Design: docs/superpowers/specs/2026-07-25-executions-card-design.md
Replaces the Live VMs card with a run-centric Executions card fed by the
Gitea Actions API, reusing the OIDC login token (kept server-side, never in
the unencrypted session cookie) with an optional keyring PAT fallback.
The Maintenance forms inherit the global block-`form` flex row, so the
max-age field and the button row are siblings on one line. Flex centring
uses each item's margin box, so the field's 0.5rem bottom margin lifted the
number input and its stepper arrows ~4px above the buttons' centre line, and
the global `form` top margin offset the whole control block inside the row.
Zero both; the stale comment claiming the field sits on its own row is
replaced by what actually renders.
Three dashboard polish fixes:
- `showModal()` was called straight from the htmx response handler, outside
the user's own click, so the first click on the dialog's Confirm button was
swallowed while the top-layer/focus handoff settled. Defer it one animation
frame and focus Confirm, so a single click confirms.
- Reorder the grid so infrastructure health (Host/VMware/Runners/Cache) reads
before the VM table instead of after it.
- Drop the Maintenance purge-semantics hint duplicated in the Cargo cache
card; it already sits next to the buttons it describes.
A large, cohesive redesign of the web dashboard (all pieces depend on
each other, so this lands as one commit):
Layout / display:
- Host/Runners/VMware/Cache split into a Host-VMware-Runners-Cache info
row; Host card keeps uptime/load/cores/mem-total now that CPU%/RAM
are promoted to the status-header tiles (new gauge tiles, red/amber/
green thresholds shared with the existing disk tile).
- VMware modules rendered as a table; Live VMs/Templates/Artifacts/
Services/Maintenance span the full grid width (card--wide).
- Backups moved from a standalone Files-panel table into an expandable
per-template row on the Templates panel (backups_by_template), with
a per-backup Delete action and a per-row Backup button replacing the
old host-card template dropdown.
- New Maintenance card: cleanup-orphans and cache-purge forms
consolidated into one danger-zone card (red rule + row separators),
each op's max-age control on its own row so it reads as the op's
parameter, not decoration; cache purge gets an explicit "Empty all"
button alongside the age-based "Purge".
- Artifacts gain a per-row Delete action.
Action/feedback layer:
- Every mutating route now sets HX-Trigger: ci:refresh on its result,
so polled panels reload immediately instead of waiting for their
next poll interval; _result_partial takes an explicit level
(ok/info/error) independent of the ok bool, so e.g. an age-based
purge that finds nothing to remove reads as neutral "info" instead
of a misleading green "ok" or a red "error".
- Results render as toasts appended to a fixed #toast-stack (base.html)
instead of swapping a single #action-slot -- concurrent actions (two
backups running at once) now each keep their own toast instead of
the later one clobbering the earlier. Per-state colour is a left
stripe + tint (result-ok/info/error), fixing a cascade bug where the
plain .toast background silently beat the state class.
- The confirm dialog is a real modal <dialog> (app.js: showModal(),
Cancel/ESC clear the slot); confirming closes it immediately and
shows a busy spinner toast in the stack (since a real backup can run
for minutes) instead of leaving the dialog open with no feedback.
- host_backup now runs backup_template via asyncio.to_thread so a
multi-GB 7z run no longer blocks the event loop (and every other
dashboard poll) for its whole duration.
- Static assets are served with a mtime-based ?v= query param so a
redeploy is picked up without a hard refresh.
purge_cargo_cache gains force: bool. The default age-based purge only
ever removed entries past the configured retention window, which read
as a lie when a user expected "purge" to clear everything -- force=True
sets the cutoff a day in the future so every existing entry is treated
as expired, backing the web UI's explicit "Empty all" action. A
disabled cache stays untouched either way.
_template_of_backup only stripped one "_"-chunk off the archive name,
so real Template_<name>_%Y%m%d_%H%M%S.7z files (date *and* time) never
matched their template -- backups silently failed to show up grouped
under it. Replaced with a regex that strips the full optional
date+time suffix and tolerates underscores in template names.
Also adds delete_artifact/delete_backup (mirrored, containment-checked
single-file deletion) and backups_by_template (groups list_backups()
output by parsed template name, newest-first) for the web dashboard's
per-item delete actions and the Templates panel's per-row backup list.
/var/lib/ci/backups was root:root because ci-backup-template.service
runs as root and mkdir(exist_ok=True) created it before ci-runner ever
touched it, so every 7z backup failed with errno=13. Chown to
ci-runner:ci-runner + 750, and stop the drift from coming back: the
setup script now creates+chmods the dir, and the systemd unit gets an
ExecStartPre that (re)installs it with the right owner on every start.
Adds operations/services.py: a fixed ManagedService allow-list (act-runner, act-runner-rust, vmware-host, ci-web) with find_service/service_states/control_service. control_service builds an argument list only (never shell=True), validates the action against ALLOWED_ACTIONS (ValueError->400), refuses stop on restart-only services (ci-web, allow_stop=False) without building a command, and defers ci-web's own restart via 'sudo -n systemd-run --on-active=2 systemctl restart ci-web.service' so the HTTP response returns first.
Route POST /services/{service_id}/{action} in web/routes.py mirrors vms_remove: require_user + require_csrf + two-step confirm gate + one audit entry per stage; unknown id / bad action / disallowed-stop short-circuit to 400 before any subprocess or audit; the vmware-host confirm step warns with the active-VM count. New /services read partial with state-aware Start/Stop/Restart buttons, polled every 15s.
deploy/sudoers.d/ci-web grants ci-runner exact-command NOPASSWD for exactly these nine systemctl lines plus the systemd-run deferral — no wildcards, no bare systemctl, no ci-web stop; setup-web-linux.sh installs it mode 0440 and validates with 'visudo -cf', rolling back on failure. Config gains [web].managed_services override; docs/WEB-UI.md documents the panel + security model.
Gates: 1044 tests pass (46 new security/unit tests), coverage 94.20%, ruff + mypy --strict clean. Not yet deployed: sudoers install + prod-venv reinstall + ci-web restart are a human privilege-boundary step.
Stage A: swap CSS tokens to the real Gitea dark palette (accent blue #4183c4); add Jinja formatting filters (humanbytes/duration/relative_time/localtime/shorten_path/thousands), a health status-header, and an inline SVG icon sprite; fix two UI bugs — unthemed action pages (htmx swap into #action-slot with a full-page HX-Request fallback) and polling wiping form input (hx-preserve).
Stage B: four read-only panels that degrade to notices (never 500) — host resources (/proc, two-sample CPU%), VMware module health, template inventory (.vmem warn + last backup), and a path-contained file browser; backup card becomes a validated template dropdown; expensive dir-size walk is TTL-cached. Adds operations/inventory.py plus host_resources()/vmware_module_health() in operations/host.py.
Stage C: parse_clone_name() adds a Job column (job_id + started) to the VM table; nav gains a Gitea link-out derived from the OIDC issuer origin (no fabricated run URLs); audit page gets action icons, relative time, and a client-side filter.
Gates: 989 tests pass, coverage 94.12%, ruff + mypy --strict clean. Stage D (service management) intentionally deferred.
CLAUDE.md and docs/ARCHITECTURE.md were still missing the operations/ and web/ packages added by the web-UI plan (W0-W7). Also records that ci-web.service is now live on this host (LAN + Gitea OIDC) in docs/WEB-UI.md and closes plans/web-ui-management-plan.md with a status header.
Restyles every template with a hand-written dark Gitea-style theme
(web/static/app.css, ~270 lines: badges, buttons, cards, focus rings,
prefers-reduced-motion) and completes the htmx polling wiring for all
four dashboard cards. Adds the repeatable setup-web-linux.sh installer
(idempotent: pip install '[web]' non-editable, render ci-web.service,
template [web]/[web.oidc] into config.toml, prompt for keyring secrets,
print Gitea OIDC registration steps, enable/restart the unit) plus
deploy/systemd/ci-web.service and docs/WEB-UI.md.
remove_vm() deletes vmx.parent, so containment must check the clone dir, not just the vmx: a crafted vmx_id of build_vms/x.vmx passed the old check and would rmtree the entire clone base. Now the parent must be strictly inside build_vms.
Adds the four destructive routes (vm.remove, vm.cleanup, cache.purge,
host.backup) plus a read-only /audit view, all behind require_user and
a synchronizer-token CSRF dependency (web/security.py). Every mutating
POST is a two-step confirm gate: a first POST always dry-runs the
underlying operations/ call and renders a confirm partial; only a
stage=execute POST performs the real action. Both stages write one
audit entry via operations/audit.py.
VM remove additionally validates that the target VMX resolves inside
config.paths.build_vms before touching anything (400 otherwise), and
vm.remove/vm.cleanup refuse to run (409) when a fresh vm-start.lock is
present, guarding against racing a live job.
SessionMiddleware is now installed unconditionally in create_app (was
previously only added when OIDC auth was active) since CSRF needs
request.session in both auth modes; the session secret stays
keyring-sourced when OIDC is active and falls back to a per-process
ephemeral secret in open mode. Adds python-multipart to the [web]
extra (required by FastAPI/Starlette form parsing, including plain
url-encoded forms, in the installed starlette version).
In open mode ([web.oidc] unset) the dashboard has no login. Binding a LAN/any address there would expose host-management unauthenticated, so ci web serve now aborts unless the bind host is loopback. Auth-configured deployments bind LAN as before.
backup_template used a hardcoded _BACKUP_BASE_DIR=/var/lib/ci/backups; now derives <config.paths.root>/backups (matches the CLI --backup-base-dir default) per the plan's no-hardcoded-path gotcha. Tests no longer need to monkeypatch the constant.
Extracts pure, click-free cargo-cache status/purge and host disk/runner/
backup-template logic into operations/, mirroring the W1 pattern. Routes
commands/retention.py::_purge_cargo_cache through operations.cache while
preserving identical CLI output. commands/monitor.py is left untouched
(its refactor is optional per the plan; the Windows/Linux polymorphic
_service_status and the disk not-found detection don't map cleanly onto
the new pure functions without risking behavioural drift).
A job killed mid-flight never released its IP lease; with only 4 pool addresses a single leak exhausted the pool ("IP pool exhausted after 60s"). IpSlotPool.reconcile() releases leases whose owner clone-dir is gone; vm cleanup runs it only against the real build_vms base. Closes TODO §2.8.
Simone