/var/lib/ci/backups was root:root because ci-backup-template.service
runs as root and mkdir(exist_ok=True) created it before ci-runner ever
touched it, so every 7z backup failed with errno=13. Chown to
ci-runner:ci-runner + 750, and stop the drift from coming back: the
setup script now creates+chmods the dir, and the systemd unit gets an
ExecStartPre that (re)installs it with the right owner on every start.
Restyles every template with a hand-written dark Gitea-style theme
(web/static/app.css, ~270 lines: badges, buttons, cards, focus rings,
prefers-reduced-motion) and completes the htmx polling wiring for all
four dashboard cards. Adds the repeatable setup-web-linux.sh installer
(idempotent: pip install '[web]' non-editable, render ci-web.service,
template [web]/[web.oidc] into config.toml, prompt for keyring secrets,
print Gitea OIDC registration steps, enable/restart the unit) plus
deploy/systemd/ci-web.service and docs/WEB-UI.md.
`runs-on: rust-build` needs a second act_runner instance, not another label on
the existing one: act_runner resolves a single runner.envs block per process
and never tells the workflow which label picked the job. A label can select a
template only by selecting a runner whose envs name that template. Registered
as ci-linux-rust (id 3), online, announcing rust-build.
- deploy/runner-rust-config.yaml: label rust-build:host only, envs pointing at
LinuxRust2404 + BaseClean-LinuxRust, its own cache dir (two processes must
not share one actcache database), capacity 1 on top of the main runner's 2.
runner.file is absolute: act_runner resolves a relative `file:` against the
process CWD, so registering from elsewhere drops the registration state --
which holds the runner token -- wherever it was invoked, and the daemon then
fails with "registration file not found".
- scripts/install-rust-runner.sh: idempotent, --dry-run, refuses to touch the
main runner directory, requires the template to exist first. Takes
--token-file rather than --token so the secret stays out of the process
argument list; the dry-run echo redacts it.
- tools/gitea-runner-token.py: mints a registration token from the stored
GiteaPAT and writes it to a 0600 file. Neither secret is ever printed. Sets
an explicit User-Agent because Cloudflare rejects urllib's default with
error 1010.
- tools/gitea-runner-list.py: --label NAME exits non-zero unless an ONLINE
runner announces it. A runs-on value no runner advertises leaves jobs
unschedulable with no error anywhere, which is exactly the failure the
consumer repo warned about.
Backups had grown to 195 GB -- half the disk, which was at 94% -- because
retention run covers only artifacts, logs and IP leases, none of which weigh
anything here. The three generations were byte-identical per template: the
templates rarely change, so the extra copies bought nothing. WinBuildTest alone
cost 26 GB per generation and has been unused since 2026-06-10.
Build VMs stopped starting after a reboot into a new kernel: vmmon/vmnet
were left unloaded and vmnet1/vmnet8 (DHCP/NAT) never came up, so
vmware-vmx died (status=237/KEYRING) and clones got no IP. Root cause:
the LSB->systemd sysv-generated vmware.service intermittently fails to
generate at early boot ("systemd-sysv-generator: Cannot find unit
vmware.service"), and the existing module-rebuild guard only builds the
modules -- it never loads them or starts vmnet networking.
Add a native vmware-host.service that runs `/etc/init.d/vmware start`
(loads vmmon/vmci + brings up vmnet), ordered after the rebuild guard,
and mask the flaky sysv vmware.service so the native unit is the sole
owner. Provisioned by scripts/install-vmware-host-services.sh and wired
into setup-host-linux.sh (new step 8/9). VMware ships no native unit and
no kernel postinst hook, so this native wrapper around VMware's own
tools is the reliable boot path.
Verified: linked clone boots and gets a vmnet8 DHCP IP.
If ci-backup-template.service is killed mid-run (SIGTERM), the Python
finally block that restarts act-runner.service is skipped. A subsequent
backup then finds the runner already inactive, _stop_runner returns
False, and the finally restart is gated off — leaving the runner down
indefinitely (incident: 21 mag 2026, runner stayed off ~80 min).
Add ExecStopPost=-/bin/systemctl start act-runner.service so systemd
guarantees the runner is restarted regardless of how the unit exits
(success, failure, signal). The `-` prefix preserves the unit's own
exit status.
- Add `retention run` command (ports Invoke-RetentionPolicy.ps1): purges
old artifact/log dirs with aggressive mode when disk space is low.
- Add `template backup` command (ports Backup-CITemplate.ps1): 7z -mx=1
compressed archives in /var/lib/ci/backups/, prunes to keep-count=3,
stops/restarts act-runner.service around the copy.
- Update ci-retention-policy.service and ci-backup-template.service to use
Python; pwsh is no longer required on the Linux host.
- Fix ci-watch-runner-health.service: pass --service-name act-runner
(Linux service name, not Windows act_runner default).
- Fix _list_orphans in vm.py: wrap is_dir() inside the OSError try block
so a stat() failure on an entry is silently skipped rather than raised.
- Mark B5 complete in PhaseB-user-checklist.md.
Simone