Sorting by started_at let a manually re-run old workflow jump above a
genuinely newer run: the re-run's started_at bumps to now, but its run
number never changes, so the Run column stopped reading strictly
descending. run_id is assigned once at creation and never changes across
attempts, so (run_id, attempt) descending is both correct and simpler --
it also handles a queued run (no started_at yet) for free, no datetime.max
special case needed.
CLAUDE.md's web/app.py summary, WEB-UI.md's dashboard-layout and Executions
sections, and a new design-history spec covering the tile removal, the
compacted Host card, the dropped level-word strip, and the four cards'
shared <dl> grammar.
It was sitting in a strip below the heading with a "Show" label and a "newest
first" hint — three lines of chrome for one control. The select now lives at the
right end of the card's own header bar (`margin-left:auto` inside the flex h2),
compact and case-normal so it reads as a control and does not grow the bar.
Label and hint are gone; the header text names the card, the values speak for
themselves, and screen readers keep the `aria-label`.
Adds a "Show" selector (10 / 25 / 50 / 100) to the card header and drops the
default row count from 15 to 10. The chosen value rides on the card's own
polling URL (`/executions?limit=N`), so it survives the 5s htmx refresh with no
client-side state; a config default outside those four is rendered as an extra
option instead of disappearing. The query is parsed permissively — a stale or
hand-edited polling URL clamps or falls back to the config default rather than
422-ing a card that would otherwise render.
Ordering is now explicit rather than inherited from whatever Gitea returns:
newest first, a not-yet-started (queued) run above everything, ties broken on
Gitea's run id so matrix runs sharing a timestamp keep a stable order.
Three bugs the real API surfaced, none visible against a stubbed payload:
- Gitea here is proxied by Cloudflare, which answers urllib's default
`Python-urllib/3.x` agent with 403 + "error code: 1010" before the request
reaches Gitea — every endpoint 403s, including unauthenticated `/version`,
so it reads exactly like a token problem. Send an explicit User-Agent.
- Gitea reports `path` as either `.gitea/workflows/lint.yml` or the
`lint.yml@refs/heads/main` form; splitting on "/" alone rendered the latter
as the workflow name "main". Drop the ref first.
- Gitea reports `run_attempt: 0` for a never-retried run while the job id the
workflow builds uses github.run_attempt, which starts at 1 — so 0 must read
as 1 or no VM would ever correlate to its run. (The existing `or 1` already
did this; it now has a test naming why, against the real payload shape.)
HTTP failures now carry the server's own message (Gitea names the missing
scope; a Cloudflare interception says something else entirely) instead of a
bare status code that sent this investigation the wrong way once already.
The login token took precedence unconditionally, so a configured keyring PAT
could never help while an operator was logged in — and on this instance the
OIDC token (scopes `openid profile`) is exactly what Gitea refuses with a 403
for insufficient API scope. `ExecutionsView` now carries `refused` (401 or a
final 403) and the web layer retries once with the PAT when the token it used
was not already the PAT. A refused PAT is never sent twice, and a 401 still
gets its single refresh attempt first.
The dashboard could only show what was powered on this instant, so it could
not answer "did the last build pass?" — and no local execution history exists
to answer it from (`report job` reads a PowerShell-era JSONL path that nothing
writes on this host, and the recent artifact dirs are empty).
Recent runs now come from Gitea's Actions API (`operations/executions.py`):
`/api/v1/admin/actions/runs` for every repo on the instance, falling back to
`/api/v1/user/actions/runs` when the token is not admin-scoped. One list call
per refresh, cached 10s, so the card's 5s htmx poll costs Gitea at most one
call per window. Live VMs are grouped under the run that spawned them — the
job id is `{run_id}-{run_attempt}[-suffix]` and the clone name carries it — so
a matrix run collects several VMs on one row. Every failure mode (no token,
401, unreachable Gitea, malformed payload) degrades to a notice plus the plain
VM rows, each keeping its Remove button, so the card is never less useful than
the table it replaces. Orphans stay where they were: the sweep in Maintenance,
the count in the status-header tile.
Auth reuses the access token from the operator's own OIDC login, which
`auth.py` previously discarded. SECURITY: that token is kept server-side only
(`app.state.gitea_tokens`, dropped on logout) and never written into the
session — `SessionMiddleware` signs the session cookie but does not encrypt
it, and this service is deployed over plain HTTP on the LAN, so a bearer token
in there would be readable in the browser and on the wire. An expired token is
refreshed once; `[web] gitea_token_cred` names an optional keyring PAT for the
cases a login token cannot cover (service restart, non-admin operator).
`GET /vms` and `partials/vms.html` are gone; `POST /vms/{id}/remove` and
`POST /vms/cleanup` are untouched.
Design: docs/superpowers/specs/2026-07-25-executions-card-design.md
Replaces the Live VMs card with a run-centric Executions card fed by the
Gitea Actions API, reusing the OIDC login token (kept server-side, never in
the unencrypted session cookie) with an optional keyring PAT fallback.
Adds operations/services.py: a fixed ManagedService allow-list (act-runner, act-runner-rust, vmware-host, ci-web) with find_service/service_states/control_service. control_service builds an argument list only (never shell=True), validates the action against ALLOWED_ACTIONS (ValueError->400), refuses stop on restart-only services (ci-web, allow_stop=False) without building a command, and defers ci-web's own restart via 'sudo -n systemd-run --on-active=2 systemctl restart ci-web.service' so the HTTP response returns first.
Route POST /services/{service_id}/{action} in web/routes.py mirrors vms_remove: require_user + require_csrf + two-step confirm gate + one audit entry per stage; unknown id / bad action / disallowed-stop short-circuit to 400 before any subprocess or audit; the vmware-host confirm step warns with the active-VM count. New /services read partial with state-aware Start/Stop/Restart buttons, polled every 15s.
deploy/sudoers.d/ci-web grants ci-runner exact-command NOPASSWD for exactly these nine systemctl lines plus the systemd-run deferral — no wildcards, no bare systemctl, no ci-web stop; setup-web-linux.sh installs it mode 0440 and validates with 'visudo -cf', rolling back on failure. Config gains [web].managed_services override; docs/WEB-UI.md documents the panel + security model.
Gates: 1044 tests pass (46 new security/unit tests), coverage 94.20%, ruff + mypy --strict clean. Not yet deployed: sudoers install + prod-venv reinstall + ci-web restart are a human privilege-boundary step.
CLAUDE.md and docs/ARCHITECTURE.md were still missing the operations/ and web/ packages added by the web-UI plan (W0-W7). Also records that ci-web.service is now live on this host (LAN + Gitea OIDC) in docs/WEB-UI.md and closes plans/web-ui-management-plan.md with a status header.
Restyles every template with a hand-written dark Gitea-style theme
(web/static/app.css, ~270 lines: badges, buttons, cards, focus rings,
prefers-reduced-motion) and completes the htmx polling wiring for all
four dashboard cards. Adds the repeatable setup-web-linux.sh installer
(idempotent: pip install '[web]' non-editable, render ci-web.service,
template [web]/[web.oidc] into config.toml, prompt for keyring secrets,
print Gitea OIDC registration steps, enable/restart the unit) plus
deploy/systemd/ci-web.service and docs/WEB-UI.md.
Measured through the real pipeline, not the unit tests: a Rust job goes from
186s to 84s (build phase 150s -> 47s), and 75s with the source modified, which
is the scenario CI actually runs. The entry is 645 MB and costs 8s to restore.
The test that decides whether this ships is row 4: a deliberate type error in
fury-core, built warm. The cache restored 645 MB of dependencies and the build
still went red with E0308, exit 1, publishing nothing. A cache that can turn a
red build green is the worst failure a CI system has because it is silent --
re-run that row whenever the prune rules change.
AGENTS.md #17 records something found while validating: `vmrun list` is
per-user. Run as the wrong user it answers "Total running VMs: 0" while a clone
is very much alive, which makes the "no VMs running" check before a
deploy --force (#14) unreliable. `ps | grep vmware-vmx` is user-independent and
was already the recommended diagnosis.
Documented with `sudo -u ci-runner`, copied from the neighbouring tools that
call the Gitea API. Wrong here: the guard fetches, fetching writes
.git/FETCH_HEAD, and that file belongs to the committing user -- so as
ci-runner it died with "impossibile aprire '.git/FETCH_HEAD': Permesso negato"
before comparing anything.
A guard that cannot run is worse than no guard, since the failure looks like
infrastructure noise rather than an answer. The question it asks is "does what
I have match what is published", so it belongs to whoever holds the working
tree.
act_runner never runs this working copy. Every job re-clones the repository
from the Gitea server at the ref the workflow names, so a fix under .gitea/
is inert until it is pushed -- and nothing here says so. `git log` shows the
fix while CI keeps running the old file.
That gap cost a downstream repo a day: 59 commits sat unpushed while they
debugged a snapshot-name bug we had already fixed and announced as done.
ci-action-published.py compares the working tree against the remote ref,
which catches both committed-but-unpushed and edited-but-uncommitted in one
diff. It fetches first, so it compares against the server rather than a
remote-tracking ref that can itself be stale -- the same failure mode one
level down. Exit 1 on drift, 2 when the check could not run.
- AGENTS.md: LinuxRust2404 reference, plus errors #14-#16 -- deploy --force
against a powered-on VM, the three distinct causes behind the identical
"VMware Tools are not running" message, and the orphan snapshot deltas
--force leaves behind after deleting the .vmsd.
- docs/RUNBOOK.md §16: triage for a Step 7 IP timeout, ordered cheapest check
first, with a Quick Reference entry.
- docs/RUNBOOK-PhaseC.md: deploy-rust/prepare-rust, the ownership and non-login
shell constraints on /opt/rust, --then-prepare, and how to route a label to a
template (a second runner instance, with the verification step).
- TODO.md §2.9: retention run covers only artifacts, logs and IP leases, so it
warned about low disk while freeing 0.0 GB two runs in a row. Mitigated by
hand; the automated policy is still blind to what actually weighs.
- TODO.md §6.5: act_runner v1.0.4 -> v2.x evaluated and deferred, with the
trigger for reopening. v2.1.0 exposes no RUNNER_LABELS either (checked
against the binary), so the two-runner design is not a stopgap.
Build VMs stopped starting after a reboot into a new kernel: vmmon/vmnet
were left unloaded and vmnet1/vmnet8 (DHCP/NAT) never came up, so
vmware-vmx died (status=237/KEYRING) and clones got no IP. Root cause:
the LSB->systemd sysv-generated vmware.service intermittently fails to
generate at early boot ("systemd-sysv-generator: Cannot find unit
vmware.service"), and the existing module-rebuild guard only builds the
modules -- it never loads them or starts vmnet networking.
Add a native vmware-host.service that runs `/etc/init.d/vmware start`
(loads vmmon/vmci + brings up vmnet), ordered after the rebuild guard,
and mask the flaky sysv vmware.service so the native unit is the sole
owner. Provisioned by scripts/install-vmware-host-services.sh and wired
into setup-host-linux.sh (new step 8/9). VMware ships no native unit and
no kernel postinst hook, so this native wrapper around VMware's own
tools is the reliable boot path.
Verified: linked clone boots and gets a vmnet8 DHCP IP.
Add `ci template check-win-iso`: parse the massgrave windows-server-links
markdown source on GitHub for the newest Windows Server 2025 build, file
name and link, and report whether it is already in <CI_ROOT>/iso/. The
download itself stays manual — the massgrave link is behind a Cloudflare
Turnstile file host, not scriptable.
Add `ci template patch-win-iso --in ISO [--out ISO]`: standalone equivalent
of deploy-win Step 4 — inject autounattend.xml + post-install.ps1 and a
no-prompt boot image into a manually downloaded ISO, producing a
ready-to-deploy `.patched.iso`.
winimage/massgrave.py does the parse (no auto-download). RUNBOOK §5
documents the check → download → patch → deploy workflow. Coverage 93.6%.
On a Linux host the UUP converter only produces the RTM install.wim (build
26100.1), so Windows Update had to install the entire cumulative anyway —
the UUP download/convert machinery added a slow ~5 GB step and several host
tools (aria2/wimtools/cabextract/chntpw) for no benefit.
Replace it: deploy-win now takes the install ISO directly via `--iso-url`
(urllib download + cache by file name under <CI_ROOT>/iso/, optional
--iso-sha256) or a local `--win-iso` (exactly one required). Patch + deploy
flow is unchanged. Supply an already-updated ISO (e.g. a massgrave link) to
skip the WU step entirely.
- Remove winimage/uup.py and its tests.
- Trim setup-host-linux.sh deps to p7zip-full + xorriso + genisoimage.
- Update RUNBOOK §5 and the Windows-host port plan. Coverage 93.6%.
Verified the whole §4 chain end-to-end on the live host and fixed every
break:
- Capture the clone VMX from `vm new` stdout into $VMX and the guest IP
via `vmrun getGuestIPAddress` into $IP, then reuse both downstream.
Removes the fragile `Clone_dbg-1_*` globs that broke when more than one
clone shared a job-id ("unexpected extra argument").
- §4.3: `build run` does not inject the Gitea PAT (only `job` does), so the
private burnin-dummy repo can't be cloned in-guest. Use --host-source-dir
against the local copy instead.
- §4.4: collect from /opt/ci/output (where build run stages the artifact),
not the `dist` source dir — the old path failed with "tar: dist: Cannot
open".
- §4.1 now boots with --gui (manual debugging); §4.7 reuses $VMX.
Confirmed green: vm new → wait-ready → build → collect (build-info.txt +
manifest.json) → vm remove, and the §3 `job` pipeline (SUCCESS in 66s).
Mirror the `job` flag on the manual lifecycle: `vm new --start` was
hardcoded headless; add --gui/--nogui (default --nogui) so a clone can be
booted into the Workstation GUI window for interactive debugging. The
start banner reflects the mode.
Also update the runbook: §4.1 shows the --gui variant, and §3 notes that
the burnin-dummy build.sh is a ~30s fake build writing dist/build-info.txt
so the examples exercise the full pipeline.
Switch the §3 and §4.3 examples to Simone/burnin-dummy (the canonical
burn-in test repo): `bash build.sh` (fake ~30s build) producing a `dist/`
artifact, so the examples exercise a real build + artifact collection end
to end.
The Phase-2 VM start was hardcoded to headless. Expose a --gui/--nogui
toggle (default --nogui, preserving current behaviour); --gui starts the
clone with the Workstation GUI window (needs an X display, see §4.7). The
phase banner now reflects the chosen mode.
Replace the non-existent myrepo.git placeholder (which fails with
"repository not found") with this repo's URL and a harmless
`echo build-ok` build command, so §3 and §4.3 run as pasted.
The `job` command enforces --template-path (job.py:447) and defaults
--snapshot-name to BaseClean, which is wrong for Linux. The §3 example
omitted both and failed with "--template-path is required". Add both
flags (BaseClean-Linux) and note them as required.
Replace the terse, abbreviation-heavy style with clean prose and
consistent formatting. Make command blocks copy-paste-ready on this host:
capture the guest IP into a shell variable for `validate guest` / `build
run`, expand globbed VMX paths, and group the `job` flags into readable
lists. Add intra-doc anchor links throughout. Point the pwsh-rollback note
at scripts/archived/. No commands or flags changed.
Document `vmrun getGuestIPAddress` (run as ci-runner) for templates and
clones as the prerequisite for `validate guest`, with the AGENTS.md #1
"Error: on stdout" gotcha. Update the validate-guest Linux example to the
live template IP.
§4.7 now explains the grant is per-session and points to setup Step 8 /
the standalone install-vmware-gui-access.sh, keeping the manual one-off
xhost command as a fallback.
The GUI connects via the X host ACL (xhost), not the inherited XAUTHORITY
cookie (unreadable by ci-runner). On a normal session the grant is already
present and `ci vmgui` works with no --display/xhost dance — sudo forwards
DISPLAY. xhost is only needed if you hit "cannot open X display". Drop the
overstated "--display :0 required" framing from §4.7.
`vmgui` sent the child's stderr to /dev/null, so when the X display was
unreachable (the common case: sudo -u ci-runner loses the desktop user's
XAUTHORITY and xhost wasn't granted) the GUI died with "cannot open display"
and the command still printed success — "nothing opens".
Now it:
- pre-flights the display with xdpyinfo and aborts with the exact
`xhost +SI:localuser:ci-runner` hint instead of spawning a doomed process;
- drops an inherited XAUTHORITY that this user can't read (falls back to
host-based xhost auth);
- logs the GUI's stdio to /tmp/vmgui-*.log and grace-polls 1.5s, surfacing the
log if the GUI exits immediately;
- errors (non-zero) when no DISPLAY is resolvable.
Runbook §4.7 updated with the "if nothing opens" guidance. 13 tests; green ≥90%.
Add top-level `vmgui` to open the VMware Workstation GUI (renamed from the
earlier `vm open`). --vmx is optional: with it the GUI opens that VM, without
it the GUI starts bare. Power-on/fullscreen flags ignored (with a notice) when
no VMX is given. Needs an X display — sudo -u ci-runner strips $DISPLAY, so
--display / xhost; see runbook §4.7. 9 tests; suite green ≥90%.
Rewrite docs/RUNBOOK-PhaseC.md as a full manual-command reference: every
ci_orchestrator command with a working example as the ci-runner user, the `ci`
alias setup, quick-reference table, manual VM lifecycle (incl. vm open GUI
launch with the xhost/--display procedure), templating, maintenance, Phase C
testing, cutover checklist and a troubleshooting map.
- CLAUDE.md: Linux host no longer runs pwsh; PS 5.1 constraints scoped to
Windows guests + Windows runner.
- RUNBOOK: add "Phase C — Python equivalents" table; §6-§13 benchmark records
kept as historical provenance.
- TODO §3.7: close $IsWindows shim bug (resolved-by-replacement — bench run
bypasses the shim).
- plans: mark C1-C8 done + docs C9; flag operational cutover/uninstall pending.
RUNBOOK §15: dual-boot operation — Linux⇄Windows switch procedure, per-OS
CI-stack table (roots/runner/vmrun), current GRUB behaviour (DEFAULT=0,
TIMEOUT=0 hidden → boots straight to Linux), post-boot verification commands
for both OSes, and an optional GRUB tweak (left unapplied — boot-to-Linux is
intentional so the box returns as the Linux CI host after any unattended
reboot).
Checklist: Passo 9 done; fixed the stale global tracking table (Passi 6–9
were still [ ] despite being complete). Phase B is now fully closed (1–9 [x]).
§10: single-job Measure-CIBenchmark on LinuxBuild2404 (DHCP, 10 iter) on the
Linux host — boot total 8.82s avg, near-deterministic even on DHCP (IP-acquire
7.45s, σ≈0.01). Guest-vs-guest table: Linux guest ~63% faster boot total than
the Windows static-IP guest (§8) on the same host.
plans/benchmark-windows-host.md: dual-boot plan to collect the symmetric
Windows-host numbers (§11 Win static, §12 Linux DHCP, §13 concurrent burn-in)
so the host×guest×IP-mode matrix is complete and Linux-vs-Windows is a
column-for-column comparison. Includes Windows paths, Credential Manager
prereqs, and the BaseClean-Linux snapshot caveat.
Concurrent capacity burn-in run on the Linux host as ci-runner with the
static-IP pool: WinBuild2025 40/40 PASS (~78.6s/round) and LinuxBuild2404
40/40 PASS (~70.2s/round) — 80/80 jobs, 20/20 rounds, near-deterministic
(static IP eliminates the DHCP IP-acquire variance of the §6/§7 baselines).
- RUNBOOK §9: concurrent burn-in result + ip-pool reset/recovery note.
- PhaseB-user-checklist Passo 7/8 marked done; fixed two checklist bugs:
run as ci-runner (the simone path needs g+w on the /var/lib/ci root for
ip-pool.tmp, not just the sub-dirs), and the Linux command was missing
-SnapshotName BaseClean-Linux (default BaseClean is Windows-only).
- idea-2-linux-host §8 DoD: Phase B complete (B1–B7 + >2 weeks stability).
- TODO §2.8 [P2]: ip-pool is not reconciled against live clones; a killed
job leaks its lease permanently and exhausts the 4-IP pool. Proposed fix.
Postmortem of the Linux template deploy/prepare 'no IP' investigation:
separates the three distinct causes (apt index stale from --skip-update,
deploy --force snapshot-chain, wedged host vmnet-dhcpd) and documents the
recovery (restart vmware-networks). Changelog entry summarising the
toolchain, xvfb, --force cleanup and cloud-init seal changes.
Add an --xvfb switch to 'build run' and 'job' that wraps the Linux build
command in 'xvfb-run -a sh -c ...', giving GUI toolkits (GTK4/PyGObject)
a headless $DISPLAY. Exported env vars precede xvfb-run so they
propagate into the virtual-display session. No-op (with a notice) for
Windows guests.
The local-ci-build composite action exposes it as the 'xvfb' input
(default false), forwarded as INPUT_XVFB -> --xvfb, mirroring the
use-shared-cache/skip-artifact wiring.
Docs: WORKFLOW-AUTHORING.md parameter table + Linux example note;
LINUX-TEMPLATE-SETUP.md Step 4a (the template ships python3-gi
gir1.2-gtk-4.0 xvfb). Adds build-run unit tests (wrap, plain, Windows
no-op).
Replaces the abandoned PSWSMan approach (New-PSSession hangs on Linux)
with a Python-native implementation using the proven pypsrp WinRmTransport.
- New `template prepare-win` command in commands/template.py: starts VM,
waits for WinRM, uploads and runs Install-CIToolchain-*.ps1, handles
Windows Update 3010 reboot loop (max 10 iterations), installs CI-StaticIp
scheduled task, runs post-setup validation, graceful shutdown, snapshot.
- Helpers: _ps_escape, _wait_tcp, _parse_exit_marker, _wait_winrm.
- 40 new tests in test_commands_template.py covering all major branches.
- Fix test_commands_job.py: patch load_config in _patch_common so tests
don't pick up live /var/lib/ci/config.toml ip_pool and fail on lock.
- Coverage gate: 90.06% (was 88.95% before fix).
4-iteration run on Linux Mint host, WinBuild2025/BaseClean.
Clone/Start/Ready/Destroy within ±20% of Windows baseline.
IP-acquire avg 99.7 s vs 58.2 s Windows — high variance (39–177 s),
attributed to VMware Tools reporting latency, not orchestrator regression.
Data from Measure-CIBenchmark.ps1 × 4 (2026-05-17, commit 5b46bba):
avg boot 60.6s, IP-acquire dominant (20–85s, σ≈26s). Adds ±20% ranges
for B7 comparison and guidance on sample size given IP-phase variance.
Simone
Simone