The backups <details> carried hx-preserve="true" to survive the card's
30s poll without collapsing if the operator had it open -- but hx-preserve
freezes the whole node, so "newest X ago" inside it never updated either,
same class of bug as the Started column before it got its own JS-side
toggle state. Drop hx-preserve; track open/closed the same way (stable id,
state re-applied after every swap) so the count/age text stays live.
uptime_sec came from the clone directory's own mtime, but a running VM's
hypervisor can rewrite lock files under that same directory at any point,
resetting the mtime and pinning "up" near 0s for the VM's entire life --
worst exactly while a build is actively running, since that's when the VM
is busiest. The clone name already carries its own creation timestamp
(_make_clone_name, parsed back via parse_clone_name), set once and never
disturbed after -- prefer it, falling back to the old mtime read only for
a name that doesn't parse.
template backup ran a full multi-GB, multi-minute 7z pass over every
template every night regardless of whether anything under it had actually
changed -- the systemd unit's own comment already noted the resulting
archives were byte-identical run to run. Now compares the template dir's
newest file mtime against its most recent existing backup and skips the
compress step (still honoring --keep-count via prune) when nothing is
newer. --force opts back into an unconditional backup.
run.number only counts within its own repo/workflow -- meaningless once
the admin endpoint mixes every repo into one list, and already not what
the row order is sorted by. run_id is Gitea's actual instance-wide,
strictly-increasing id, so the displayed number now agrees with "most
recent globally". The link still goes to Gitea's own per-repo run page
(html_url); only the label text changed.
number is a per-repo/workflow counter with no meaning once the admin
endpoint mixes every repo on the instance into one feed -- a rarely-run
workflow's low #53 that just fired outranked a busier workflow's #142 from
hours earlier, but sorting by number buried it instead. Sort by started
(globally comparable) with run_id/attempt as tiebreaks; a re-run bumping
started_at back to the top is correct, not a bug, under "most recent
globally".
htmx's outerHTML swap (every polling card, including Executions) removes
the original target node from the DOM as part of the swap, but
htmx:afterSwap's eventInfo.target still points at that dead node -- so
restoreTimeToggles() was querying a detached subtree every poll and never
touching the live buttons. Fall back to the native event target, which
htmx dispatches on the freshly-inserted content, whenever detail.target
is no longer connected.
base.html stamps the same asset_version() value onto both app.css and
app.js's <link>/<script> tags, but the value was computed from app.css's
mtime alone. A JS-only deploy left it unchanged, so app.js's URL stayed
byte-identical to the previous deploy -- nothing forced a browser holding
that URL cached to refetch it.
Extracted into _static_asset_version(web_dir), keyed to the newer of the
two files' mtimes, and unit-tested directly (a JS-only mtime bump now
changes the value; both-missing degrades to "0").
The card's own htmx poll re-renders the whole table from the server every
5s, always starting fresh at "relative" -- so a toggled-to-absolute cell
snapped back on the next poll. Each button now has a stable id
(started-{run_id}-{attempt}, or started-vm-{vmx} on an unmatched-VM row);
app.js remembers which format was showing per id and re-applies it to the
fresh button after every swap, rather than freezing the node itself (which
would also freeze its relative-time text, badge, duration, everything else
in that row).
Same affordance, cleaner line: solid underline tinted to the border gray at
rest (subtle, doesn't read as a broken link) and to the current text colour
on hover (more emphasis).
Both renderings are pre-rendered server-side into a <button>'s
data-relative/data-absolute attributes; app.js swaps textContent (and the
hover title, to the opposite value) on click, delegated on document.body
so it survives the card's own 5s htmx poll without needing to re-wire
after every swap. Click again to swap back.
FastAPI auto-registers these with no auth of its own -- every other data
route is individually wrapped in Depends(require_user), but these three
are wired up internally by the FastAPI() constructor and were never
covered by that. Confirmed live: all three returned 200 with zero session,
handing anyone who could reach the port a full map of every route --
VM remove, cache purge, template delete, service control included --
without logging in.
Disabled FastAPI's built-in docs_url/redoc_url/openapi_url and re-added all
three as ordinary routes behind the same require_user dependency every
other route uses. Verified against the live deployment: all three now 303
to /login instead of 200.
The previous fix (sort by run_id/attempt) traded one bug for another:
run_id is Gitea's instance-wide id, shared across every repo/workflow the
admin endpoint pulls from, not a per-workflow counter. A rarely-triggered
workflow's #53 could land a run_id between two ids from a busier workflow's
#141/#142, putting #53's row between them in the Run column even though 53
isn't close to either -- exactly the reported symptom, now reproduced as a
regression test before the fix and passing after it.
run.number is the value the Run column actually displays; sort by that,
then attempt, then run_id only to keep a genuine same-number tie (two
different workflows landing on the same number independently) stable.
Sorting by started_at let a manually re-run old workflow jump above a
genuinely newer run: the re-run's started_at bumps to now, but its run
number never changes, so the Run column stopped reading strictly
descending. run_id is assigned once at creation and never changes across
attempts, so (run_id, attempt) descending is both correct and simpler --
it also handles a queued run (no started_at yet) for free, no datetime.max
special case needed.
mypy --strict (files = ["src"]) and pytest --cov=ci_orchestrator both walk
the whole src tree, web/ included, but the lint job's venv only ever
installed .[dev] -- fastapi/starlette/uvicorn/jinja2/authlib were never
resolvable, so every symbol they touch degraded to Any under mypy (masking
real type errors, and producing a cascading "Returning Any" on
_is_htmx_request) and every tests/python/test_web_*.py would fail to import
under pytest. Install .[dev,web] instead. Verified against a fresh venv:
ruff/mypy --strict/pytest --cov all pass.
CLAUDE.md's web/app.py summary, WEB-UI.md's dashboard-layout and Executions
sections, and a new design-history spec covering the tile removal, the
compacted Host card, the dropped level-word strip, and the four cards'
shared <dl> grammar.
- Two filled button kinds, no neutral/outline third state: every action is
either "normal" (#4382c2) or destructive (#cb4b4c); border is the same
neutral gray as every table/card border, not the fill colour.
- Icons: bolder stroke site-wide (2 -> 2.5) and +1px, except inside buttons
(2.25, no size bump) where a bold icon next to now-unbolded button text
was too heavy.
- tbody row hover: flat gray -> a transparent wash of the link/accent blue.
- The Actions <td> in Services/Templates was itself a flex container; some
engines size that box's content slightly differently from a plain-text
sibling cell, which showed up as the row's collapsed bottom border sitting
1px higher under Actions than its neighbours. The flex row now lives on an
inner <div class="actions-row"> instead, with its <form> children pinned
to content width (a bare <form> is display:flex by the site's own rule,
which was letting one form claim the whole row and push the next to wrap).
- Navbar brand gets an icon (a server rack -- this dashboard orchestrates
hosts/VMs, not generic settings).
POST /host/delete-template: same two-step confirm gate, CSRF check, and
known-template containment as /host/backup, offloaded to a thread the same
way (rmtree on a multi-GB template dir is not instant). No backup-required
safety rail -- same confirm-gate as every other destructive action, nothing
more; the confirm summary spells out that clones can no longer be made from
the template until one is restored.
Also cleaned up the per-backup Delete button it sits next to: name+meta
grouped into one shrinkable unit so the row wraps as a block instead of the
delete button drifting away from a name that wrapped, and each backup is
now its own bordered line instead of a run-on flex list.
No cell stacks two facts on separate lines with <br> any more (Run/Repo,
Branch/Commit, Trigger/Actor, Duration/Started, VM/IP-Uptime each get their
own column, 8 -> 13 total). The card is already .card--wide with its own
.table-wrap, so the extra width scrolls horizontally rather than the page.
Status pills carry a check/x/play icon for the three outcomes that are a
real verdict or an in-progress state (success/failure/running); queued,
cancelled, skipped stay plain text. Every pill in the column -- including
the VM-only orphan/running pills on unmatched-VM rows -- is forced to the
same width with left-aligned content instead of a ragged column.
The dashboard used to open with six status-header tiles *and* four compact
cards restating them (runner state lived in three places at once). The tiles
are gone; each card is now the single owner of its subject, enriched with
what only the tiles had (CPU%/RAM/disk with gauges) plus what neither had
(hostname, kernel, full CPU model, the vmware-host.service row).
All four cards share the same <dl> grammar and font scale now -- VMware and
Runners were a bordered <table> at a different size/case, which read as a
visibly different style sitting next to Host and Cargo cache's plain
label/value lists.
operations/host.py also gains delete_template()/TemplateDeleteResult, used
by the template-delete route in the next commit.
It was sitting in a strip below the heading with a "Show" label and a "newest
first" hint — three lines of chrome for one control. The select now lives at the
right end of the card's own header bar (`margin-left:auto` inside the flex h2),
compact and case-normal so it reads as a control and does not grow the bar.
Label and hint are gone; the header text names the card, the values speak for
themselves, and screen readers keep the `aria-label`.
`--link` was still the brighter GitHub blue while everything else moved to
Gitea's #4183c4, so links read as a second, slightly different accent. One blue
now.
Contrast on this palette: 4.50:1 on --panel (#161718), 4.14:1 on --bg
(#1e1f20), 3.77:1 on --panel-2 (#22272e) — down from 4.79 / 4.41 / 4.01. Body
links sit on --panel and stay above the 4.5:1 AA threshold; the ones on the
darker surfaces (nav hover, table cells) land between AA-large and AA, as the
rest of this palette's dim text already does.
Adds a "Show" selector (10 / 25 / 50 / 100) to the card header and drops the
default row count from 15 to 10. The chosen value rides on the card's own
polling URL (`/executions?limit=N`), so it survives the 5s htmx refresh with no
client-side state; a config default outside those four is rendered as an extra
option instead of disappearing. The query is parsed permissively — a stale or
hand-edited polling URL clamps or falls back to the config default rather than
422-ing a card that would otherwise render.
Ordering is now explicit rather than inherited from whatever Gitea returns:
newest first, a not-yet-started (queued) run above everything, ties broken on
Gitea's run id so matrix runs sharing a timestamp keep a stable order.
Every Gitea API call from Python 403s with "error code: 1010" unless a
User-Agent is set, including endpoints that need no auth — and it looks like a
token/scope problem, which is where this went wrong once. Registered next to
the other environment traps, with the two-command check that tells them apart.
Three bugs the real API surfaced, none visible against a stubbed payload:
- Gitea here is proxied by Cloudflare, which answers urllib's default
`Python-urllib/3.x` agent with 403 + "error code: 1010" before the request
reaches Gitea — every endpoint 403s, including unauthenticated `/version`,
so it reads exactly like a token problem. Send an explicit User-Agent.
- Gitea reports `path` as either `.gitea/workflows/lint.yml` or the
`lint.yml@refs/heads/main` form; splitting on "/" alone rendered the latter
as the workflow name "main". Drop the ref first.
- Gitea reports `run_attempt: 0` for a never-retried run while the job id the
workflow builds uses github.run_attempt, which starts at 1 — so 0 must read
as 1 or no VM would ever correlate to its run. (The existing `or 1` already
did this; it now has a test naming why, against the real payload shape.)
HTTP failures now carry the server's own message (Gitea names the missing
scope; a Cloudflare interception says something else entirely) instead of a
bare status code that sent this investigation the wrong way once already.
The login token took precedence unconditionally, so a configured keyring PAT
could never help while an operator was logged in — and on this instance the
OIDC token (scopes `openid profile`) is exactly what Gitea refuses with a 403
for insufficient API scope. `ExecutionsView` now carries `refused` (401 or a
final 403) and the web layer retries once with the PAT when the token it used
was not already the PAT. A refused PAT is never sent twice, and a 401 still
gets its single refresh attempt first.
Both listing endpoints answering 403 means Gitea authenticated the token and
then rejected its scope -- exactly what an OIDC login token with only
`openid profile` gets. The card showed a bare "Gitea API returned HTTP 403",
which sends the operator hunting the wrong thing (admin rights, allowlist,
URL). Say what it is and name both fixes: add an API read scope to
`[web.oidc] scopes`, or configure the keyring PAT fallback. A 403 on the admin
listing alone still falls through to the user listing as before, and a 404
there keeps its plain message.
The dashboard could only show what was powered on this instant, so it could
not answer "did the last build pass?" — and no local execution history exists
to answer it from (`report job` reads a PowerShell-era JSONL path that nothing
writes on this host, and the recent artifact dirs are empty).
Recent runs now come from Gitea's Actions API (`operations/executions.py`):
`/api/v1/admin/actions/runs` for every repo on the instance, falling back to
`/api/v1/user/actions/runs` when the token is not admin-scoped. One list call
per refresh, cached 10s, so the card's 5s htmx poll costs Gitea at most one
call per window. Live VMs are grouped under the run that spawned them — the
job id is `{run_id}-{run_attempt}[-suffix]` and the clone name carries it — so
a matrix run collects several VMs on one row. Every failure mode (no token,
401, unreachable Gitea, malformed payload) degrades to a notice plus the plain
VM rows, each keeping its Remove button, so the card is never less useful than
the table it replaces. Orphans stay where they were: the sweep in Maintenance,
the count in the status-header tile.
Auth reuses the access token from the operator's own OIDC login, which
`auth.py` previously discarded. SECURITY: that token is kept server-side only
(`app.state.gitea_tokens`, dropped on logout) and never written into the
session — `SessionMiddleware` signs the session cookie but does not encrypt
it, and this service is deployed over plain HTTP on the LAN, so a bearer token
in there would be readable in the browser and on the wire. An expired token is
refreshed once; `[web] gitea_token_cred` names an optional keyring PAT for the
cases a login token cannot cover (service restart, non-admin operator).
`GET /vms` and `partials/vms.html` are gone; `POST /vms/{id}/remove` and
`POST /vms/cleanup` are untouched.
Design: docs/superpowers/specs/2026-07-25-executions-card-design.md
Replaces the Live VMs card with a run-centric Executions card fed by the
Gitea Actions API, reusing the OIDC login token (kept server-side, never in
the unencrypted session cookie) with an optional keyring PAT fallback.
The Maintenance forms inherit the global block-`form` flex row, so the
max-age field and the button row are siblings on one line. Flex centring
uses each item's margin box, so the field's 0.5rem bottom margin lifted the
number input and its stepper arrows ~4px above the buttons' centre line, and
the global `form` top margin offset the whole control block inside the row.
Zero both; the stale comment claiming the field sits on its own row is
replaced by what actually renders.
Three dashboard polish fixes:
- `showModal()` was called straight from the htmx response handler, outside
the user's own click, so the first click on the dialog's Confirm button was
swallowed while the top-layer/focus handoff settled. Defer it one animation
frame and focus Confirm, so a single click confirms.
- Reorder the grid so infrastructure health (Host/VMware/Runners/Cache) reads
before the VM table instead of after it.
- Drop the Maintenance purge-semantics hint duplicated in the Cargo cache
card; it already sits next to the buttons it describes.
A large, cohesive redesign of the web dashboard (all pieces depend on
each other, so this lands as one commit):
Layout / display:
- Host/Runners/VMware/Cache split into a Host-VMware-Runners-Cache info
row; Host card keeps uptime/load/cores/mem-total now that CPU%/RAM
are promoted to the status-header tiles (new gauge tiles, red/amber/
green thresholds shared with the existing disk tile).
- VMware modules rendered as a table; Live VMs/Templates/Artifacts/
Services/Maintenance span the full grid width (card--wide).
- Backups moved from a standalone Files-panel table into an expandable
per-template row on the Templates panel (backups_by_template), with
a per-backup Delete action and a per-row Backup button replacing the
old host-card template dropdown.
- New Maintenance card: cleanup-orphans and cache-purge forms
consolidated into one danger-zone card (red rule + row separators),
each op's max-age control on its own row so it reads as the op's
parameter, not decoration; cache purge gets an explicit "Empty all"
button alongside the age-based "Purge".
- Artifacts gain a per-row Delete action.
Action/feedback layer:
- Every mutating route now sets HX-Trigger: ci:refresh on its result,
so polled panels reload immediately instead of waiting for their
next poll interval; _result_partial takes an explicit level
(ok/info/error) independent of the ok bool, so e.g. an age-based
purge that finds nothing to remove reads as neutral "info" instead
of a misleading green "ok" or a red "error".
- Results render as toasts appended to a fixed #toast-stack (base.html)
instead of swapping a single #action-slot -- concurrent actions (two
backups running at once) now each keep their own toast instead of
the later one clobbering the earlier. Per-state colour is a left
stripe + tint (result-ok/info/error), fixing a cascade bug where the
plain .toast background silently beat the state class.
- The confirm dialog is a real modal <dialog> (app.js: showModal(),
Cancel/ESC clear the slot); confirming closes it immediately and
shows a busy spinner toast in the stack (since a real backup can run
for minutes) instead of leaving the dialog open with no feedback.
- host_backup now runs backup_template via asyncio.to_thread so a
multi-GB 7z run no longer blocks the event loop (and every other
dashboard poll) for its whole duration.
- Static assets are served with a mtime-based ?v= query param so a
redeploy is picked up without a hard refresh.
purge_cargo_cache gains force: bool. The default age-based purge only
ever removed entries past the configured retention window, which read
as a lie when a user expected "purge" to clear everything -- force=True
sets the cutoff a day in the future so every existing entry is treated
as expired, backing the web UI's explicit "Empty all" action. A
disabled cache stays untouched either way.
_template_of_backup only stripped one "_"-chunk off the archive name,
so real Template_<name>_%Y%m%d_%H%M%S.7z files (date *and* time) never
matched their template -- backups silently failed to show up grouped
under it. Replaced with a regex that strips the full optional
date+time suffix and tolerates underscores in template names.
Also adds delete_artifact/delete_backup (mirrored, containment-checked
single-file deletion) and backups_by_template (groups list_backups()
output by parsed template name, newest-first) for the web dashboard's
per-item delete actions and the Templates panel's per-row backup list.
/var/lib/ci/backups was root:root because ci-backup-template.service
runs as root and mkdir(exist_ok=True) created it before ci-runner ever
touched it, so every 7z backup failed with errno=13. Chown to
ci-runner:ci-runner + 750, and stop the drift from coming back: the
setup script now creates+chmods the dir, and the systemd unit gets an
ExecStartPre that (re)installs it with the right owner on every start.
Adds operations/services.py: a fixed ManagedService allow-list (act-runner, act-runner-rust, vmware-host, ci-web) with find_service/service_states/control_service. control_service builds an argument list only (never shell=True), validates the action against ALLOWED_ACTIONS (ValueError->400), refuses stop on restart-only services (ci-web, allow_stop=False) without building a command, and defers ci-web's own restart via 'sudo -n systemd-run --on-active=2 systemctl restart ci-web.service' so the HTTP response returns first.
Route POST /services/{service_id}/{action} in web/routes.py mirrors vms_remove: require_user + require_csrf + two-step confirm gate + one audit entry per stage; unknown id / bad action / disallowed-stop short-circuit to 400 before any subprocess or audit; the vmware-host confirm step warns with the active-VM count. New /services read partial with state-aware Start/Stop/Restart buttons, polled every 15s.
deploy/sudoers.d/ci-web grants ci-runner exact-command NOPASSWD for exactly these nine systemctl lines plus the systemd-run deferral — no wildcards, no bare systemctl, no ci-web stop; setup-web-linux.sh installs it mode 0440 and validates with 'visudo -cf', rolling back on failure. Config gains [web].managed_services override; docs/WEB-UI.md documents the panel + security model.
Gates: 1044 tests pass (46 new security/unit tests), coverage 94.20%, ruff + mypy --strict clean. Not yet deployed: sudoers install + prod-venv reinstall + ci-web restart are a human privilege-boundary step.
Stage A: swap CSS tokens to the real Gitea dark palette (accent blue #4183c4); add Jinja formatting filters (humanbytes/duration/relative_time/localtime/shorten_path/thousands), a health status-header, and an inline SVG icon sprite; fix two UI bugs — unthemed action pages (htmx swap into #action-slot with a full-page HX-Request fallback) and polling wiping form input (hx-preserve).
Stage B: four read-only panels that degrade to notices (never 500) — host resources (/proc, two-sample CPU%), VMware module health, template inventory (.vmem warn + last backup), and a path-contained file browser; backup card becomes a validated template dropdown; expensive dir-size walk is TTL-cached. Adds operations/inventory.py plus host_resources()/vmware_module_health() in operations/host.py.
Stage C: parse_clone_name() adds a Job column (job_id + started) to the VM table; nav gains a Gitea link-out derived from the OIDC issuer origin (no fabricated run URLs); audit page gets action icons, relative time, and a client-side filter.
Gates: 989 tests pass, coverage 94.12%, ruff + mypy --strict clean. Stage D (service management) intentionally deferred.
CLAUDE.md and docs/ARCHITECTURE.md were still missing the operations/ and web/ packages added by the web-UI plan (W0-W7). Also records that ci-web.service is now live on this host (LAN + Gitea OIDC) in docs/WEB-UI.md and closes plans/web-ui-management-plan.md with a status header.
Restyles every template with a hand-written dark Gitea-style theme
(web/static/app.css, ~270 lines: badges, buttons, cards, focus rings,
prefers-reduced-motion) and completes the htmx polling wiring for all
four dashboard cards. Adds the repeatable setup-web-linux.sh installer
(idempotent: pip install '[web]' non-editable, render ci-web.service,
template [web]/[web.oidc] into config.toml, prompt for keyring secrets,
print Gitea OIDC registration steps, enable/restart the unit) plus
deploy/systemd/ci-web.service and docs/WEB-UI.md.
remove_vm() deletes vmx.parent, so containment must check the clone dir, not just the vmx: a crafted vmx_id of build_vms/x.vmx passed the old check and would rmtree the entire clone base. Now the parent must be strictly inside build_vms.
Adds the four destructive routes (vm.remove, vm.cleanup, cache.purge,
host.backup) plus a read-only /audit view, all behind require_user and
a synchronizer-token CSRF dependency (web/security.py). Every mutating
POST is a two-step confirm gate: a first POST always dry-runs the
underlying operations/ call and renders a confirm partial; only a
stage=execute POST performs the real action. Both stages write one
audit entry via operations/audit.py.
VM remove additionally validates that the target VMX resolves inside
config.paths.build_vms before touching anything (400 otherwise), and
vm.remove/vm.cleanup refuse to run (409) when a fresh vm-start.lock is
present, guarding against racing a live job.
SessionMiddleware is now installed unconditionally in create_app (was
previously only added when OIDC auth was active) since CSRF needs
request.session in both auth modes; the session secret stays
keyring-sourced when OIDC is active and falls back to a per-process
ephemeral secret in open mode. Adds python-multipart to the [web]
extra (required by FastAPI/Starlette form parsing, including plain
url-encoded forms, in the installed starlette version).
In open mode ([web.oidc] unset) the dashboard has no login. Binding a LAN/any address there would expose host-management unauthenticated, so ci web serve now aborts unless the bind host is loopback. Auth-configured deployments bind LAN as before.
backup_template used a hardcoded _BACKUP_BASE_DIR=/var/lib/ci/backups; now derives <config.paths.root>/backups (matches the CLI --backup-base-dir default) per the plan's no-hardcoded-path gotcha. Tests no longer need to monkeypatch the constant.
Extracts pure, click-free cargo-cache status/purge and host disk/runner/
backup-template logic into operations/, mirroring the W1 pattern. Routes
commands/retention.py::_purge_cargo_cache through operations.cache while
preserving identical CLI output. commands/monitor.py is left untouched
(its refactor is optional per the plan; the Windows/Linux polymorphic
_service_status and the disk not-found detection don't map cleanly onto
the new pure functions without risking behavioural drift).
A job killed mid-flight never released its IP lease; with only 4 pool addresses a single leak exhausted the pool ("IP pool exhausted after 60s"). IpSlotPool.reconcile() releases leases whose owner clone-dir is gone; vm cleanup runs it only against the real build_vms base. Closes TODO §2.8.
Server-rendered management UI (FastAPI + HTMX + Jinja2), Gitea OIDC auth,
bare-systemd deploy, in-process operations/ seam reused by both CLI and web
(no subprocess, load_backend kept for the ESXi path). Includes the
implementer recipe (TDD, mypy --strict, coverage >=90, no CDN, secrets in
keyring only).
Added two implementer notes tying the refactor to open TODO items:
- W1: fold the TODO §2.8 IP-pool leak fix into operations/vms.cleanup_orphans
while moving it (same function the web "cleanup orphans" action calls).
- W2: TODO §2.9 (retention blind to backups/templates/ISO) is orthogonal —
operations/cache.py wraps only cargo-cache purge; do not widen W2 scope.
report.py's residual F:\CI\Logs default is already flagged under Rischi.
The shared composite action ran 3 steps under `shell: pwsh`, making every
CI job depend on PowerShell on the Linux host. Rewrote them in bash and
dropped the dead $IsLinux / F:\CI branches (Phase B left a single Linux
host serving both windows-build and linux-build runners).
Closes Phase C / idea-3: no CI job or systemd timer requires pwsh on the
Linux host anymore. Ticked all C1-C8 subtask checkboxes and set the plans
to "C chiusa". pwsh is deliberately left installed (operator choice, not a
technical blocker).
Cargo dependency cache for Rust jobs on LinuxRust2404: 186s -> 84s measured
end to end, 75s with the source modified. Dependency artifacts are reused
between runs; workspace crates never are, and the job refuses to publish an
entry the prune assertion cannot vouch for.
Closes thread entries furyrs#7 and furyrs#4. TODO 3.8.
Measured through the real pipeline, not the unit tests: a Rust job goes from
186s to 84s (build phase 150s -> 47s), and 75s with the source modified, which
is the scenario CI actually runs. The entry is 645 MB and costs 8s to restore.
The test that decides whether this ships is row 4: a deliberate type error in
fury-core, built warm. The cache restored 645 MB of dependencies and the build
still went red with E0308, exit 1, publishing nothing. A cache that can turn a
red build green is the worst failure a CI system has because it is silent --
re-run that row whenever the prune rules change.
AGENTS.md #17 records something found while validating: `vmrun list` is
per-user. Run as the wrong user it answers "Total running VMs: 0" while a clone
is very much alive, which makes the "no VMs running" check before a
deploy --force (#14) unreliable. `ps | grep vmware-vmx` is user-independent and
was already the recommended diagnosis.
TODO 2.9 recorded a retention policy that warned about low disk while freeing
0.0 GB twice in a row, because it governed artifacts, logs and IP leases -- 52
KB in total on this host -- while 195 G of backups sat outside its reach.
Adding an unbounded directory of ~645 MB entries, one per Cargo.lock change,
without extending retention in the same change would repeat that at a larger
scale.
The window is the tighter of the global retention and the cache's own, so the
aggressive low-disk mode reaches the cache rather than stopping at the light
directories.
Sizes are read before deleting, not after. Summing st_size over files that have
already been unlinked yields 0 -- which is exactly how a policy ends up
reporting that it freed nothing while doing real work. A test asserts the
reported figure.
auto/on/off, defaulting to auto: branches and pull requests cache, refs/tags/v*
builds cold. furyrs asked for exactly that in thread #13 -- a tag run verifies
a release artifact rather than a change, so the clean room is worth two
minutes there and not elsewhere.
An unrecognised value throws instead of silently picking a side, the same
posture as the guest-os: 'Auto' fix: a cache setting that quietly means its
opposite is worse than a failed job.
Verified by running the shipped PowerShell block itself under pwsh across all
seven mode/ref combinations plus the invalid-mode throw, rather than by
reimplementing the logic in the test -- a reimplementation only checks the
model, not the code that ships.
Default on, because the cache is safe by construction rather than by
convention: it engages only where a Cargo.lock exists, and refuses to publish
anything the prune assertion cannot vouch for. Two independent off switches --
the flag for one job, `enabled = false` for the host.
The store root is resolved after the environment overrides are applied, so
CI_ROOT relocates it along with everything else, while an explicit root in the
config still wins. Both directions are covered by tests.
Also asserts that the Windows branch never receives the argument at all: the
cache is Linux-only and the two branches stay separate, per AGENTS.md.
Restore runs after the source is in place and before the build; publish runs
only after a successful build and only on a miss, so the steady state uploads
nothing and a failed build can never become an entry. The whole path is
best-effort: a cache problem degrades to a cold build with a message rather
than turning a good build red.
Gated on Cargo.lock being present, so non-Rust jobs are untouched with no flag
to set. The Windows branch is not touched at all.
The prune script is read through importlib.resources rather than a path
relative to the repo root: the production venv is installed non-editable, so a
parents[] walk lands in site-packages where scripts/ does not exist.
The fake transport gained substring-matched canned results; indexing by call
position broke as soon as the cache added calls mid-flow.
Mutation-tested rather than assumed. Three guards were each disabled in turn --
the publish call, the prune-refusal check, the packing-failure check -- and a
test went red for each. The prune-refusal test initially survived its mutant:
it passed because the fake's fetch left no file behind, so publishing failed
for an unrelated reason. Fixed by making fetch write a real file, so the only
thing preventing publication is the guard under test.
Measured first, as thread entry #12 requires: a cold furyrs pipeline is 134.7s,
target/ is 3.0 GiB raw and 2.5 GiB pruned, and the stored entry is 645 MiB. A
warm job pays 8.1s (upload + extract) to skip ~120s of dependency compilation.
Publishing costs 15.7s but only on a miss. Build it.
The assertion in the plan turned out to be worthless and is replaced. It
re-scanned for */.fingerprint/<member>-* after deleting exactly those paths, so
asked to catch a planted fury-core fingerprint it exited 0 -- the prune had
removed the evidence before the check looked. An assertion sharing its globs
with the deletion can only prove that rm ran; it cannot see the failure that
matters, which is a glob that never matched the artifact in the first place.
The replacement asks a different oracle. Cargo records in each .d depfile the
sources an artifact was built from: dependencies point into the registry,
anything built from workspace source points back into the workspace. After a
correct prune, zero surviving depfiles reference workspace source outside
target/. Verified negatively too -- an artifact named totallyunrelated-abc123.d
that no prune glob matches, declaring a dependency on fury-core/src/lib.rs,
makes the script exit 2 and refuse to publish.
zstd is already present on the template, so no rebuild is needed; the gzip
fallback stays for the plain LinuxBuild2404.
One immutable file per key, published by rename(2) inside the store. Two jobs
publishing the same key both produce a valid entry and the last wins; a reader
holding the previous one keeps reading it. No advisory locking, which is the
property a VMware shared folder cannot offer with several jobs in flight --
the objection raised in the furyrs thread.
The key joins its parts with NUL so ("ab","c") and ("a","bc") cannot collide,
and carries a format version so the whole store can be invalidated at once
when the prune rules change and older entries stop being trustworthy.
The feature set is deliberately absent from the key: cargo already encodes it
in each artifact's own hash suffix, so several feature graphs coexist in one
entry rather than fragmenting the store into one entry per variant.
Documented with `sudo -u ci-runner`, copied from the neighbouring tools that
call the Gitea API. Wrong here: the guard fetches, fetching writes
.git/FETCH_HEAD, and that file belongs to the committing user -- so as
ci-runner it died with "impossibile aprire '.git/FETCH_HEAD': Permesso negato"
before comparing anything.
A guard that cannot run is worse than no guard, since the failure looks like
infrastructure noise rather than an answer. The question it asks is "does what
I have match what is published", so it belongs to whoever holds the working
tree.
Thread entry furyrs#12 measured where a 133s Rust job actually goes: ~120s is
compiling the ~490 external dependencies, which a committed Cargo.lock keeps
bit-identical between runs. The workspace crates cost 7s. #13 settled the
policy -- reuse dependency artifacts, never workspace ones, and keep v* tags
cold.
The plan departs from #12's cargo-chef shape for a concrete reason: the
orchestrator receives build-command as an opaque string (furyrs runs six cargo
invocations across three feature sets) and `cargo chef cook` must be handed the
same flags as the real build. Rather than guess them, invert the order -- build
normally, then prune the workspace members' artifacts out of target/ and
publish the remainder. No knowledge of the command, one entry holding every
feature graph, no new tool in the template.
That makes the safety property checkable rather than argued: after the prune,
an assertion looks for any surviving workspace fingerprint and the job refuses
to publish if it finds one. A missing entry costs 120s; a poisoned one costs a
wrong verdict, which is the worst failure a build system has because it is
silent.
Store is one immutable file per key with an atomic rename, so there is no
shared mutable directory and no advisory locking -- the objection furyrs raised
about VMware Shared Folders under concurrency. Retention of the store is in the
same plan: adding an unbounded directory without it would repeat TODO 2.9 at a
larger scale.
act_runner never runs this working copy. Every job re-clones the repository
from the Gitea server at the ref the workflow names, so a fix under .gitea/
is inert until it is pushed -- and nothing here says so. `git log` shows the
fix while CI keeps running the old file.
That gap cost a downstream repo a day: 59 commits sat unpushed while they
debugged a snapshot-name bug we had already fixed and announced as done.
ci-action-published.py compares the working tree against the remote ref,
which catches both committed-but-unpushed and edited-but-uncommitted in one
diff. It fetches first, so it compares against the server rather than a
remote-tracking ref that can itself be stale -- the same failure mode one
level down. Exit 1 on drift, 2 when the check could not run.
Committed by accident in 3680f07 while working on the massgrave ISO
resolver: a saved ZeroFS search page, 52 KB, referenced by nothing in
src/, tests/, tools/, scripts/, template/ or deploy/. It carries a
Google Analytics tag and a third-party ad-injector script, neither of
which belongs in this repository.
Removed from HEAD rather than rebased out: furyrs cites 7cca938 by SHA
in their coordination thread, and rewriting history would dangle that
reference. The blob stays reachable in history.
- AGENTS.md: LinuxRust2404 reference, plus errors #14-#16 -- deploy --force
against a powered-on VM, the three distinct causes behind the identical
"VMware Tools are not running" message, and the orphan snapshot deltas
--force leaves behind after deleting the .vmsd.
- docs/RUNBOOK.md §16: triage for a Step 7 IP timeout, ordered cheapest check
first, with a Quick Reference entry.
- docs/RUNBOOK-PhaseC.md: deploy-rust/prepare-rust, the ownership and non-login
shell constraints on /opt/rust, --then-prepare, and how to route a label to a
template (a second runner instance, with the verification step).
- TODO.md §2.9: retention run covers only artifacts, logs and IP leases, so it
warned about low disk while freeing 0.0 GB two runs in a row. Mitigated by
hand; the automated policy is still blind to what actually weighs.
- TODO.md §6.5: act_runner v1.0.4 -> v2.x evaluated and deferred, with the
trigger for reopening. v2.1.0 exposes no RUNNER_LABELS either (checked
against the binary), so the two-runner design is not a stopgap.
`runs-on: rust-build` needs a second act_runner instance, not another label on
the existing one: act_runner resolves a single runner.envs block per process
and never tells the workflow which label picked the job. A label can select a
template only by selecting a runner whose envs name that template. Registered
as ci-linux-rust (id 3), online, announcing rust-build.
- deploy/runner-rust-config.yaml: label rust-build:host only, envs pointing at
LinuxRust2404 + BaseClean-LinuxRust, its own cache dir (two processes must
not share one actcache database), capacity 1 on top of the main runner's 2.
runner.file is absolute: act_runner resolves a relative `file:` against the
process CWD, so registering from elsewhere drops the registration state --
which holds the runner token -- wherever it was invoked, and the daemon then
fails with "registration file not found".
- scripts/install-rust-runner.sh: idempotent, --dry-run, refuses to touch the
main runner directory, requires the template to exist first. Takes
--token-file rather than --token so the secret stays out of the process
argument list; the dry-run echo redacts it.
- tools/gitea-runner-token.py: mints a registration token from the stored
GiteaPAT and writes it to a 0600 file. Neither secret is ever printed. Sets
an explicit User-Agent because Cloudflare rejects urllib's default with
error 1010.
- tools/gitea-runner-list.py: --label NAME exits non-zero unless an ONLINE
runner announces it. A runs-on value no runner advertises leaves jobs
unschedulable with no error anywhere, which is exactly the failure the
consumer repo warned about.
Backups had grown to 195 GB -- half the disk, which was at 94% -- because
retention run covers only artifacts, logs and IP leases, none of which weigh
anything here. The three generations were byte-identical per template: the
templates rarely change, so the extra copies bought nothing. WinBuildTest alone
cost 26 GB per generation and has been unused since 2026-06-10.
act_runner does not expose the job's runs-on label to the workflow: v1.0.4
provides only RUNNER_OS/ARCH/TEMP/TOOL_CACHE/DEBUG/VERSION, verified against
the binary. The `$env:RUNNER_LABELS -match 'linux'` check was therefore always
false, so guest-os: 'Auto' silently resolved to Windows and selected the
Windows template for Linux jobs. It was masked only because every caller passes
guest-os explicitly or overrides template-path.
'Auto' cannot pick a template on its own -- the VMX guestOS field is only
readable after the clone -- so it now requires template-path and snapshot-name
and errors otherwise, instead of falling back to the wrong template.
Add GITEA_CI_LINUX_SNAPSHOT_NAME so a runner instance dedicated to a Linux
variant can declare its own snapshot instead of every workflow repeating it;
the Linux snapshot was hardcoded to BaseClean-Linux.
BREAKING: a workflow using the composite with neither guest-os nor
template-path now fails instead of silently using the Windows template.
Four defects, each of which could waste a full template rebuild:
1. getGuestIPAddress was called without -wait, so vmrun answered "The VMware
Tools are not running" unless the Tools happened to have registered the IP
at that exact instant. A guest that was booted, on the network and answering
SSH could still burn the entire 10-minute Step 7 budget. This race is the
likely cause of every intermittent "no guest IP" incident so far, including
ones previously attributed to wedged vmnet DHCP. AGENTS.md #10 already
prescribed -wait; the code did not follow it.
2. --force did not check whether the target VM was powered on. It deleted the
live .vmdk.lck as a stale artifact and Step 4 rewrote the base VMDK under
the running guest, killing it; the damage surfaced ten minutes later as an
IP timeout because `vmrun list` still showed the OLD instance.
3. Step 6 accepted "VM appears in vmrun list" as proof of power-on, which a
pre-existing instance satisfies. Check the exit code of `vmrun start`.
4. The cloud-init seed ISO was detached while the VM was still running, and
VMware rewrote the VMX from its in-memory copy at power-off, restoring
ide1:0.present = "TRUE". Every clone would boot with the seed attached
(AGENTS.md #6). Detach it in prepare, after power-off and before the
snapshot.
Also in this change:
- --then-prepare chains deploy into the matching prepare in one run, reusing
the VM deploy leaves running. The two commands stay separately invocable.
The guest toolchain script is resolved and verified BEFORE any VM work, so a
wrong working directory costs two seconds instead of failing the hand-off
after a 45-minute deploy; --toolchain-script allows running from anywhere.
- Corrected the final messages: deploy-rust told the user to run prepare-linux,
and both deploy commands reported a snapshot name (PostInstall-Linux) that no
command ever creates. The prepare hand-off hint passed a bare filename to
--template and omitted --snapshot, so copy-pasting it cloned BaseClean, the
Windows snapshot.
- template backup --exclude, repeatable, for retired templates. An unknown name
or --exclude without --all-templates is an error rather than a silent no-op.
test_deploy_linux_full_happy_path declared the VM already running before Step 6,
an impossible state that masked the new --force guard; its vmrun mock is now
stateful.
The chown ran before rustup and cargo-audit were installed, so everything
those created under RUSTUP_HOME/CARGO_HOME was left root-owned: only the two
empty directories existing at chown time belonged to ci_build. Measured on the
previous snapshot: 18350 root-owned paths, including cargo/registry. Every Rust
build with external dependencies failed with "Permission denied (os error 13)".
Re-assert ownership after the installs, and assert writability rather than mere
existence -- test -x on the binaries could not catch this, so a broken template
passed its own build and failed in consumers' jobs instead.
Also wire the toolchain for non-login shells. The orchestrator runs build
commands via paramiko exec_command + `sh -c`, which never sources
/etc/profile.d, so profile.d alone was not enough: add RUSTUP_HOME/CARGO_HOME
to /etc/environment (read by pam_env at SSH session open) and symlink the
rustup proxies into /usr/local/bin.
Build VMs stopped starting after a reboot into a new kernel: vmmon/vmnet
were left unloaded and vmnet1/vmnet8 (DHCP/NAT) never came up, so
vmware-vmx died (status=237/KEYRING) and clones got no IP. Root cause:
the LSB->systemd sysv-generated vmware.service intermittently fails to
generate at early boot ("systemd-sysv-generator: Cannot find unit
vmware.service"), and the existing module-rebuild guard only builds the
modules -- it never loads them or starts vmnet networking.
Add a native vmware-host.service that runs `/etc/init.d/vmware start`
(loads vmmon/vmci + brings up vmnet), ordered after the rebuild guard,
and mask the flaky sysv vmware.service so the native unit is the sole
owner. Provisioned by scripts/install-vmware-host-services.sh and wired
into setup-host-linux.sh (new step 8/9). VMware ships no native unit and
no kernel postinst hook, so this native wrapper around VMware's own
tools is the reliable boot path.
Verified: linked clone boots and gets a vmnet8 DHCP IP.
Setting DelegationConsole/DelegationTerminal to the Windows Terminal GUID made
WT the console host for the cmd.exe shell. With Shell=cmd.exe and
AutoRestartShell=1, closing the WT-hosted shell window breaks the cmd<->WT host
handshake: Winlogon's relaunch leaves a bare grey desktop with no shell instead
of a fresh cmd. Classic conhost restarts cleanly and is ~82 MB lighter (no
WindowsTerminal process in the idle baseline).
Comment out the four delegation reg adds (HKCU + Default hive) so the shell
falls back to conhost. Kept commented (not deleted) to document the rationale.
schtasks.exe /Change on a non-existent task writes to stderr and exits 1.
Inside the disk-cleanup function (EAP=Stop, wrapped by the exit-code-sensitive
prepare-win cleanup invoker), that leaked $LASTEXITCODE=1 all the way to the
result marker — DISM runs via Start-Process and does not reset it — so the
step failed with "Disk cleanup failed (exit 1)" even though everything worked.
On a freshly-deployed WS2025 the UpdateOrchestrator "Schedule Scan" task does
not exist, so this fired every run.
Replace the schtasks loop with Get-ScheduledTask | Disable-ScheduledTask,
which no-ops silently on a missing task and sets no process exit code.
Follow-up to fb5de6f; applied to both toolchains.
The disk-cleanup function stopped wuauserv before DISM /StartComponentCleanup
/ResetBase but only re-enforced Start=4 AFTER it. /ResetBase drives heavy CBS
servicing (TiWorker); during that window WaaSMedicSvc woke, flipped wuauserv
Start back to Manual, and a "Schedule Scan" task started it. If the pass was
interrupted there (reboot/crash), the template was left with WU running and
Manual startup — a half-prepared state where Windows Update keeps firing on a
supposedly-locked-down build template.
Move the lockdown ahead of DISM: set wuauserv Start=4, apply the
WaaSMedicSvc registry write-deny ACL (the only reliable lever — the service
is protected and cannot itself be disabled), and disable the WindowsUpdate /
UpdateOrchestrator scan tasks, all BEFORE /ResetBase rather than only after.
The post-DISM re-enforce stays as a belt-and-suspenders check.
Applied identically to the WinBuild2025 and WinBuild2022 toolchains.
An ephemeral headless build VM carries a lot of dead weight. Split the trim
by cost so heavy servicing never runs under the live explorer shell (which
crash-loops during CBS churn at first logon):
post-install.win.ps1 (light, runs under explorer at first logon):
- disable waste services: audio, camera, CDP/sync, push (Wpn), telemetry
(DiagTrack/InventorySvc), search index (WSearch), DPS/WDI, MSDTC, TrkWks,
Themes, MapsBroker, RetailDemo, PcaSvc, WSAIFabricSvc; BITS -> on-demand
- disable per-user service templates (CDPUserSvc/OneSyncSvc/WpnUserService/...)
so no per-session clone spawns
- remove in-box apps: FeedbackHub/GetHelp/Getstarted/OfficeHub (appx +
provisioned), Snip & Sketch, Paint, RDP client (mstsc) — all best-effort
prepare-win (heavy, runs under cmd shell, before the final reboot so the
following DISM cleanup compacts the freed component store):
- Uninstall-WindowsFeature WindowsAdminCenterSetup, XPS-Viewer,
Wireless-Networking, Windows-Defender (-NoRestart)
- Remove-WindowsCapability AzureArcSetup (payload; auto-start already off)
- uninstall Edge + EdgeWebView2 and disable their leftover updater services
- gated by --skip-trim
Server Manager auto-launch was already disabled (task + reg + GPO policy);
its exe is part of the OS and cannot be uninstalled. Defender was already
runtime-off; removing the feature drops the AV stack entirely.
run_streaming used index-based _drain, which read new records from
ps.output / ps.streams.{information,error} but never released the consumed
ones. pypsrp retains every record (each a heavy deserialised PSObject) for
the whole invocation, so a long prepare-win (Windows Update + toolchain,
thousands of lines) grew the host python process RAM without bound.
Replace _drain with _drain_clear, which emits each pending record then
deletes the consumed prefix (`del seq[:n]`). Safe because draining runs on
the same thread as poll_invoke — nothing appends while we mutate. Track an
explicit had_error flag since the error stream is now cleared as we go.
PowerShell 5.1 strips an empty-string argument ('') when invoking a native
exe, so `reg add ... /d '' /f` reached reg.exe as `reg add ... /d /f`: reg
consumed /f as the data value, lost the force flag, and prompted
"overwrite? (Yes/No)" on stdin. FirstLogon runs headless with no stdin, so
post-install hung forever right after the "Explorer UX tweaks applied" log
line. Route both empty-value reg adds (HKCU + Default hive) through
`cmd /c`, which preserves the empty "" so the value is set and /f stays.
Regression from 8095f00 (wallpaper removal feature).
Prepend a guard that runs `autoload -Uz compinit && compinit` when compdef
isn't defined, so the generated zsh completion works without oh-my-zsh or if
sourced before compinit.
Generate a click completion file for bash/zsh and source it from the alias
block. Click's generated script queries completions by invoking the command
— but `ci` is a sudo shell function (env vars don't survive sudo, and `env
VAR=x ci` can't call a function), so rewrite the query to call the venv
python directly (read-only, no sudo) and drop zsh's `$+commands[ci]` guard.
Writes ~/.ci-completion.<shell>; removed cleanly on --uninstall (block strip).
Invoking `python -m ci_orchestrator` now reports prog_name "ci" — matching
the shell alias — so `--help` reads naturally and click shell completion
keys off the _CI_COMPLETE env var.
The base ISO's volume label (e.g. "26100.1_X64_EN-US") contains characters
outside strict ISO 9660 d-characters, so xorriso warned "-volid text does
not comply to ISO 9660 / ECMA 119 rules" on every repack. Map the label to
[A-Z0-9_] (cap 16). Cosmetic only — Windows boots from the El Torito images,
not the volid — but keeps the output clean.
Remove only the wallpaper image (Wallpaper="" + WallpaperStyle=0) and leave
Control Panel\Colors\Background at the WS2025 default, so the desktop shows
the default grey instead of black.
A headless CI build VM needs no wallpaper. post-install now clears the
Wallpaper value + sets a black background for the autologin Administrator
(HKCU) and the Default user hive (so any future profile inherits it).
autounattend FirstLogonCommands — which run post-install.ps1 — are triggered
by explorer starting as the shell. Setting Winlogon\Shell=cmd.exe in the
specialize pass stopped explorer, so post-install never ran on a fresh
deploy.
Remove the shell switch from specialize; explorer runs once for the first
logon (deploy-win does no heavy servicing, so no crash-loop risk) and
post-install.ps1 sets Shell=cmd.exe at its end — subsequent boots and all of
prepare-win run under cmd. Keep the (shell-independent) Server Manager
auto-launch disable in specialize.
Add specialize-pass RunSynchronousCommands to disable the ServerManager
scheduled task (the actual auto-launcher on WS2025) plus the HKLM/policy
DoNotOpen* keys, so Server Manager never pops up — not even at the first
logon, before post-install runs. Redundant with post-install.
Add a Microsoft-Windows-Deployment RunSynchronousCommand in the specialize
pass that sets Winlogon\Shell=cmd.exe before the first logon. A freshly
deployed template then never starts explorer — not even the transient
desktop during OOBE/install — closing the last window where the shell could
crash-loop during servicing. post-install.ps1 and prepare-win still
re-affirm it for redundancy / existing templates.
Up front, read Winlogon\Shell; if it isn't already cmd.exe, set it and do a
clean reboot before running the toolchain — so the shell is genuinely cmd
(not a killed-explorer half-state) for the whole servicing run. Idempotent:
when the shell is already cmd (re-runs), skip the reboot. Replaces the
earlier kill-explorer approach.
Set Winlogon\Shell=cmd.exe and drop the live explorer at the very start of
prepare-win, before the toolchain runs, so every reboot in the Windows
Update 3010 loop already boots without explorer — eliminating the shell
crash-loop during heavy servicing, not just at the final reboot. Still
re-affirmed before the final reboot and baked into post-install.
Setting Winlogon\Shell=cmd.exe only in the install-time post-install meant
existing templates (deployed earlier, or reverted to a pre-cmd snapshot)
kept explorer as the shell — re-running prepare-win never changed it, since
prepare runs the toolchain, not post-install. Set Shell=cmd.exe in the
prepare-win autologin re-affirm (before the final reboot) so any template
adopts it on the next prepare.
A headless CI build VM never needs the interactive desktop (builds run over
WinRM in Session 0), and a running explorer is fragile: heavy servicing
(Windows Update / DISM component cleanup) destabilises the live shell and
explorer crash-loops until the next reboot, spiking CPU and tripping the
pre-snapshot idle check.
Set Winlogon\Shell = cmd.exe in post-install so explorer never auto-starts
— removing that whole failure class at the root. Autologin is unchanged;
explorer remains launchable on demand (run explorer.exe, or via RDP).
Verified on the live WinBuildTest VM: post-reboot no explorer, 0 shell-stop
events, ~2% idle CPU.
Heavy servicing (DISM StartComponentCleanup in the -CleanupOnly step, like
WU finalization) destabilises the running shell session — explorer
crash-loops until the next reboot. Cleanup ran *after* the final reboot, so
the template ended on a crash-looping desktop and the idle check aborted.
Reorder: disk cleanup + autologin re-affirm now run before the final
reboot, making that reboot the last action before the idle check — the
shell comes up against fully-settled components and the snapshot captures a
clean, idle desktop.
The one-shot idle sample fired ~30s after the final reboot, but Windows
Update / CBS (TiWorker) keeps finalizing for minutes — so a still-busy
template was wrongly rejected. Poll until CPU and RAM are both under
threshold (up to --idle-timeout, default 600s) instead of sampling once,
and report the busiest process to aid diagnosis. Adds --idle-timeout.
A fast reboot never showed TCP/5986 "down" between polls, so the old
down→up wait could hang on "waiting for VM to go down" until timeout.
Replace it: record the guest LastBootUpTime (FileTime ticks) before the
reboot, then poll until the port is back AND the boot time is newer — a
signal immune to reboot speed. Applies to both the 3010 loop and the final
reboot. Drops _wait_tcp_down in favour of _tcp_open + _guest_boot_ticks.
After the final reboot and just before the graceful shutdown for the
snapshot, sample the guest's idle CPU and used-RAM percentages (CIM
Win32_PerfFormattedData — locale-independent — averaged over ~5s after a
settle delay). If either exceeds its threshold, abort prepare without
taking the snapshot, so a busy/leaky template isn't frozen into every clone.
New options: --max-idle-cpu (10%), --max-idle-ram (50%), --idle-settle
(30s), --skip-idle-check.
Microsoft rotated the unversioned https://dot.net/v1/dotnet-install.ps1, so
the pinned SHA256 was stale and the toolchain aborted ("Possible MITM or
wrong installer version"). Independently re-downloaded the official script
(builds.dotnet.microsoft.com) — its hash matches the guest's observed
value, confirming it is the legitimate current script, not tampering.
Update the pin in both WinBuild2025 and WinBuild2022 toolchains to
6585899AED55FF6AE13DBE1E8C3B878F2D00433520E7EFBE250B75DB948B7DA9 and note
that this pin must be refreshed whenever MS updates the script in place.
pypsrp's execute_ps is fully blocking, so Install-CIToolchain / Windows
Update output only appeared at the end behind a "still running (Ns)"
heartbeat. Add WinRmTransport.run_streaming: drive the script via pypsrp's
incremental begin_invoke/poll_invoke, emitting output, Write-Host
(information) and error stream lines as they arrive over the same WinRM
channel.
_run_with_heartbeat now prefers run_streaming (suppressing the CI_EXITCODE
marker line) and falls back to the blocking heartbeat only on an unexpected
pypsrp error — TransportError still propagates so the 3010 reboot handling
is unaffected.
Simone