The backups <details> carried hx-preserve="true" to survive the card's
30s poll without collapsing if the operator had it open -- but hx-preserve
freezes the whole node, so "newest X ago" inside it never updated either,
same class of bug as the Started column before it got its own JS-side
toggle state. Drop hx-preserve; track open/closed the same way (stable id,
state re-applied after every swap) so the count/age text stays live.
uptime_sec came from the clone directory's own mtime, but a running VM's
hypervisor can rewrite lock files under that same directory at any point,
resetting the mtime and pinning "up" near 0s for the VM's entire life --
worst exactly while a build is actively running, since that's when the VM
is busiest. The clone name already carries its own creation timestamp
(_make_clone_name, parsed back via parse_clone_name), set once and never
disturbed after -- prefer it, falling back to the old mtime read only for
a name that doesn't parse.
template backup ran a full multi-GB, multi-minute 7z pass over every
template every night regardless of whether anything under it had actually
changed -- the systemd unit's own comment already noted the resulting
archives were byte-identical run to run. Now compares the template dir's
newest file mtime against its most recent existing backup and skips the
compress step (still honoring --keep-count via prune) when nothing is
newer. --force opts back into an unconditional backup.
run.number only counts within its own repo/workflow -- meaningless once
the admin endpoint mixes every repo into one list, and already not what
the row order is sorted by. run_id is Gitea's actual instance-wide,
strictly-increasing id, so the displayed number now agrees with "most
recent globally". The link still goes to Gitea's own per-repo run page
(html_url); only the label text changed.
number is a per-repo/workflow counter with no meaning once the admin
endpoint mixes every repo on the instance into one feed -- a rarely-run
workflow's low #53 that just fired outranked a busier workflow's #142 from
hours earlier, but sorting by number buried it instead. Sort by started
(globally comparable) with run_id/attempt as tiebreaks; a re-run bumping
started_at back to the top is correct, not a bug, under "most recent
globally".
base.html stamps the same asset_version() value onto both app.css and
app.js's <link>/<script> tags, but the value was computed from app.css's
mtime alone. A JS-only deploy left it unchanged, so app.js's URL stayed
byte-identical to the previous deploy -- nothing forced a browser holding
that URL cached to refetch it.
Extracted into _static_asset_version(web_dir), keyed to the newer of the
two files' mtimes, and unit-tested directly (a JS-only mtime bump now
changes the value; both-missing degrades to "0").
The card's own htmx poll re-renders the whole table from the server every
5s, always starting fresh at "relative" -- so a toggled-to-absolute cell
snapped back on the next poll. Each button now has a stable id
(started-{run_id}-{attempt}, or started-vm-{vmx} on an unmatched-VM row);
app.js remembers which format was showing per id and re-applies it to the
fresh button after every swap, rather than freezing the node itself (which
would also freeze its relative-time text, badge, duration, everything else
in that row).
Both renderings are pre-rendered server-side into a <button>'s
data-relative/data-absolute attributes; app.js swaps textContent (and the
hover title, to the opposite value) on click, delegated on document.body
so it survives the card's own 5s htmx poll without needing to re-wire
after every swap. Click again to swap back.
FastAPI auto-registers these with no auth of its own -- every other data
route is individually wrapped in Depends(require_user), but these three
are wired up internally by the FastAPI() constructor and were never
covered by that. Confirmed live: all three returned 200 with zero session,
handing anyone who could reach the port a full map of every route --
VM remove, cache purge, template delete, service control included --
without logging in.
Disabled FastAPI's built-in docs_url/redoc_url/openapi_url and re-added all
three as ordinary routes behind the same require_user dependency every
other route uses. Verified against the live deployment: all three now 303
to /login instead of 200.
The previous fix (sort by run_id/attempt) traded one bug for another:
run_id is Gitea's instance-wide id, shared across every repo/workflow the
admin endpoint pulls from, not a per-workflow counter. A rarely-triggered
workflow's #53 could land a run_id between two ids from a busier workflow's
#141/#142, putting #53's row between them in the Run column even though 53
isn't close to either -- exactly the reported symptom, now reproduced as a
regression test before the fix and passing after it.
run.number is the value the Run column actually displays; sort by that,
then attempt, then run_id only to keep a genuine same-number tie (two
different workflows landing on the same number independently) stable.
Sorting by started_at let a manually re-run old workflow jump above a
genuinely newer run: the re-run's started_at bumps to now, but its run
number never changes, so the Run column stopped reading strictly
descending. run_id is assigned once at creation and never changes across
attempts, so (run_id, attempt) descending is both correct and simpler --
it also handles a queued run (no started_at yet) for free, no datetime.max
special case needed.
POST /host/delete-template: same two-step confirm gate, CSRF check, and
known-template containment as /host/backup, offloaded to a thread the same
way (rmtree on a multi-GB template dir is not instant). No backup-required
safety rail -- same confirm-gate as every other destructive action, nothing
more; the confirm summary spells out that clones can no longer be made from
the template until one is restored.
Also cleaned up the per-backup Delete button it sits next to: name+meta
grouped into one shrinkable unit so the row wraps as a block instead of the
delete button drifting away from a name that wrapped, and each backup is
now its own bordered line instead of a run-on flex list.
No cell stacks two facts on separate lines with <br> any more (Run/Repo,
Branch/Commit, Trigger/Actor, Duration/Started, VM/IP-Uptime each get their
own column, 8 -> 13 total). The card is already .card--wide with its own
.table-wrap, so the extra width scrolls horizontally rather than the page.
Status pills carry a check/x/play icon for the three outcomes that are a
real verdict or an in-progress state (success/failure/running); queued,
cancelled, skipped stay plain text. Every pill in the column -- including
the VM-only orphan/running pills on unmatched-VM rows -- is forced to the
same width with left-aligned content instead of a ragged column.
The dashboard used to open with six status-header tiles *and* four compact
cards restating them (runner state lived in three places at once). The tiles
are gone; each card is now the single owner of its subject, enriched with
what only the tiles had (CPU%/RAM/disk with gauges) plus what neither had
(hostname, kernel, full CPU model, the vmware-host.service row).
All four cards share the same <dl> grammar and font scale now -- VMware and
Runners were a bordered <table> at a different size/case, which read as a
visibly different style sitting next to Host and Cargo cache's plain
label/value lists.
operations/host.py also gains delete_template()/TemplateDeleteResult, used
by the template-delete route in the next commit.
It was sitting in a strip below the heading with a "Show" label and a "newest
first" hint — three lines of chrome for one control. The select now lives at the
right end of the card's own header bar (`margin-left:auto` inside the flex h2),
compact and case-normal so it reads as a control and does not grow the bar.
Label and hint are gone; the header text names the card, the values speak for
themselves, and screen readers keep the `aria-label`.
Adds a "Show" selector (10 / 25 / 50 / 100) to the card header and drops the
default row count from 15 to 10. The chosen value rides on the card's own
polling URL (`/executions?limit=N`), so it survives the 5s htmx refresh with no
client-side state; a config default outside those four is rendered as an extra
option instead of disappearing. The query is parsed permissively — a stale or
hand-edited polling URL clamps or falls back to the config default rather than
422-ing a card that would otherwise render.
Ordering is now explicit rather than inherited from whatever Gitea returns:
newest first, a not-yet-started (queued) run above everything, ties broken on
Gitea's run id so matrix runs sharing a timestamp keep a stable order.
Three bugs the real API surfaced, none visible against a stubbed payload:
- Gitea here is proxied by Cloudflare, which answers urllib's default
`Python-urllib/3.x` agent with 403 + "error code: 1010" before the request
reaches Gitea — every endpoint 403s, including unauthenticated `/version`,
so it reads exactly like a token problem. Send an explicit User-Agent.
- Gitea reports `path` as either `.gitea/workflows/lint.yml` or the
`lint.yml@refs/heads/main` form; splitting on "/" alone rendered the latter
as the workflow name "main". Drop the ref first.
- Gitea reports `run_attempt: 0` for a never-retried run while the job id the
workflow builds uses github.run_attempt, which starts at 1 — so 0 must read
as 1 or no VM would ever correlate to its run. (The existing `or 1` already
did this; it now has a test naming why, against the real payload shape.)
HTTP failures now carry the server's own message (Gitea names the missing
scope; a Cloudflare interception says something else entirely) instead of a
bare status code that sent this investigation the wrong way once already.
The login token took precedence unconditionally, so a configured keyring PAT
could never help while an operator was logged in — and on this instance the
OIDC token (scopes `openid profile`) is exactly what Gitea refuses with a 403
for insufficient API scope. `ExecutionsView` now carries `refused` (401 or a
final 403) and the web layer retries once with the PAT when the token it used
was not already the PAT. A refused PAT is never sent twice, and a 401 still
gets its single refresh attempt first.
Both listing endpoints answering 403 means Gitea authenticated the token and
then rejected its scope -- exactly what an OIDC login token with only
`openid profile` gets. The card showed a bare "Gitea API returned HTTP 403",
which sends the operator hunting the wrong thing (admin rights, allowlist,
URL). Say what it is and name both fixes: add an API read scope to
`[web.oidc] scopes`, or configure the keyring PAT fallback. A 403 on the admin
listing alone still falls through to the user listing as before, and a 404
there keeps its plain message.
The dashboard could only show what was powered on this instant, so it could
not answer "did the last build pass?" — and no local execution history exists
to answer it from (`report job` reads a PowerShell-era JSONL path that nothing
writes on this host, and the recent artifact dirs are empty).
Recent runs now come from Gitea's Actions API (`operations/executions.py`):
`/api/v1/admin/actions/runs` for every repo on the instance, falling back to
`/api/v1/user/actions/runs` when the token is not admin-scoped. One list call
per refresh, cached 10s, so the card's 5s htmx poll costs Gitea at most one
call per window. Live VMs are grouped under the run that spawned them — the
job id is `{run_id}-{run_attempt}[-suffix]` and the clone name carries it — so
a matrix run collects several VMs on one row. Every failure mode (no token,
401, unreachable Gitea, malformed payload) degrades to a notice plus the plain
VM rows, each keeping its Remove button, so the card is never less useful than
the table it replaces. Orphans stay where they were: the sweep in Maintenance,
the count in the status-header tile.
Auth reuses the access token from the operator's own OIDC login, which
`auth.py` previously discarded. SECURITY: that token is kept server-side only
(`app.state.gitea_tokens`, dropped on logout) and never written into the
session — `SessionMiddleware` signs the session cookie but does not encrypt
it, and this service is deployed over plain HTTP on the LAN, so a bearer token
in there would be readable in the browser and on the wire. An expired token is
refreshed once; `[web] gitea_token_cred` names an optional keyring PAT for the
cases a login token cannot cover (service restart, non-admin operator).
`GET /vms` and `partials/vms.html` are gone; `POST /vms/{id}/remove` and
`POST /vms/cleanup` are untouched.
Design: docs/superpowers/specs/2026-07-25-executions-card-design.md
A large, cohesive redesign of the web dashboard (all pieces depend on
each other, so this lands as one commit):
Layout / display:
- Host/Runners/VMware/Cache split into a Host-VMware-Runners-Cache info
row; Host card keeps uptime/load/cores/mem-total now that CPU%/RAM
are promoted to the status-header tiles (new gauge tiles, red/amber/
green thresholds shared with the existing disk tile).
- VMware modules rendered as a table; Live VMs/Templates/Artifacts/
Services/Maintenance span the full grid width (card--wide).
- Backups moved from a standalone Files-panel table into an expandable
per-template row on the Templates panel (backups_by_template), with
a per-backup Delete action and a per-row Backup button replacing the
old host-card template dropdown.
- New Maintenance card: cleanup-orphans and cache-purge forms
consolidated into one danger-zone card (red rule + row separators),
each op's max-age control on its own row so it reads as the op's
parameter, not decoration; cache purge gets an explicit "Empty all"
button alongside the age-based "Purge".
- Artifacts gain a per-row Delete action.
Action/feedback layer:
- Every mutating route now sets HX-Trigger: ci:refresh on its result,
so polled panels reload immediately instead of waiting for their
next poll interval; _result_partial takes an explicit level
(ok/info/error) independent of the ok bool, so e.g. an age-based
purge that finds nothing to remove reads as neutral "info" instead
of a misleading green "ok" or a red "error".
- Results render as toasts appended to a fixed #toast-stack (base.html)
instead of swapping a single #action-slot -- concurrent actions (two
backups running at once) now each keep their own toast instead of
the later one clobbering the earlier. Per-state colour is a left
stripe + tint (result-ok/info/error), fixing a cascade bug where the
plain .toast background silently beat the state class.
- The confirm dialog is a real modal <dialog> (app.js: showModal(),
Cancel/ESC clear the slot); confirming closes it immediately and
shows a busy spinner toast in the stack (since a real backup can run
for minutes) instead of leaving the dialog open with no feedback.
- host_backup now runs backup_template via asyncio.to_thread so a
multi-GB 7z run no longer blocks the event loop (and every other
dashboard poll) for its whole duration.
- Static assets are served with a mtime-based ?v= query param so a
redeploy is picked up without a hard refresh.
purge_cargo_cache gains force: bool. The default age-based purge only
ever removed entries past the configured retention window, which read
as a lie when a user expected "purge" to clear everything -- force=True
sets the cutoff a day in the future so every existing entry is treated
as expired, backing the web UI's explicit "Empty all" action. A
disabled cache stays untouched either way.
_template_of_backup only stripped one "_"-chunk off the archive name,
so real Template_<name>_%Y%m%d_%H%M%S.7z files (date *and* time) never
matched their template -- backups silently failed to show up grouped
under it. Replaced with a regex that strips the full optional
date+time suffix and tolerates underscores in template names.
Also adds delete_artifact/delete_backup (mirrored, containment-checked
single-file deletion) and backups_by_template (groups list_backups()
output by parsed template name, newest-first) for the web dashboard's
per-item delete actions and the Templates panel's per-row backup list.
Adds operations/services.py: a fixed ManagedService allow-list (act-runner, act-runner-rust, vmware-host, ci-web) with find_service/service_states/control_service. control_service builds an argument list only (never shell=True), validates the action against ALLOWED_ACTIONS (ValueError->400), refuses stop on restart-only services (ci-web, allow_stop=False) without building a command, and defers ci-web's own restart via 'sudo -n systemd-run --on-active=2 systemctl restart ci-web.service' so the HTTP response returns first.
Route POST /services/{service_id}/{action} in web/routes.py mirrors vms_remove: require_user + require_csrf + two-step confirm gate + one audit entry per stage; unknown id / bad action / disallowed-stop short-circuit to 400 before any subprocess or audit; the vmware-host confirm step warns with the active-VM count. New /services read partial with state-aware Start/Stop/Restart buttons, polled every 15s.
deploy/sudoers.d/ci-web grants ci-runner exact-command NOPASSWD for exactly these nine systemctl lines plus the systemd-run deferral — no wildcards, no bare systemctl, no ci-web stop; setup-web-linux.sh installs it mode 0440 and validates with 'visudo -cf', rolling back on failure. Config gains [web].managed_services override; docs/WEB-UI.md documents the panel + security model.
Gates: 1044 tests pass (46 new security/unit tests), coverage 94.20%, ruff + mypy --strict clean. Not yet deployed: sudoers install + prod-venv reinstall + ci-web restart are a human privilege-boundary step.
Stage A: swap CSS tokens to the real Gitea dark palette (accent blue #4183c4); add Jinja formatting filters (humanbytes/duration/relative_time/localtime/shorten_path/thousands), a health status-header, and an inline SVG icon sprite; fix two UI bugs — unthemed action pages (htmx swap into #action-slot with a full-page HX-Request fallback) and polling wiping form input (hx-preserve).
Stage B: four read-only panels that degrade to notices (never 500) — host resources (/proc, two-sample CPU%), VMware module health, template inventory (.vmem warn + last backup), and a path-contained file browser; backup card becomes a validated template dropdown; expensive dir-size walk is TTL-cached. Adds operations/inventory.py plus host_resources()/vmware_module_health() in operations/host.py.
Stage C: parse_clone_name() adds a Job column (job_id + started) to the VM table; nav gains a Gitea link-out derived from the OIDC issuer origin (no fabricated run URLs); audit page gets action icons, relative time, and a client-side filter.
Gates: 989 tests pass, coverage 94.12%, ruff + mypy --strict clean. Stage D (service management) intentionally deferred.
remove_vm() deletes vmx.parent, so containment must check the clone dir, not just the vmx: a crafted vmx_id of build_vms/x.vmx passed the old check and would rmtree the entire clone base. Now the parent must be strictly inside build_vms.
Adds the four destructive routes (vm.remove, vm.cleanup, cache.purge,
host.backup) plus a read-only /audit view, all behind require_user and
a synchronizer-token CSRF dependency (web/security.py). Every mutating
POST is a two-step confirm gate: a first POST always dry-runs the
underlying operations/ call and renders a confirm partial; only a
stage=execute POST performs the real action. Both stages write one
audit entry via operations/audit.py.
VM remove additionally validates that the target VMX resolves inside
config.paths.build_vms before touching anything (400 otherwise), and
vm.remove/vm.cleanup refuse to run (409) when a fresh vm-start.lock is
present, guarding against racing a live job.
SessionMiddleware is now installed unconditionally in create_app (was
previously only added when OIDC auth was active) since CSRF needs
request.session in both auth modes; the session secret stays
keyring-sourced when OIDC is active and falls back to a per-process
ephemeral secret in open mode. Adds python-multipart to the [web]
extra (required by FastAPI/Starlette form parsing, including plain
url-encoded forms, in the installed starlette version).
In open mode ([web.oidc] unset) the dashboard has no login. Binding a LAN/any address there would expose host-management unauthenticated, so ci web serve now aborts unless the bind host is loopback. Auth-configured deployments bind LAN as before.
backup_template used a hardcoded _BACKUP_BASE_DIR=/var/lib/ci/backups; now derives <config.paths.root>/backups (matches the CLI --backup-base-dir default) per the plan's no-hardcoded-path gotcha. Tests no longer need to monkeypatch the constant.
Extracts pure, click-free cargo-cache status/purge and host disk/runner/
backup-template logic into operations/, mirroring the W1 pattern. Routes
commands/retention.py::_purge_cargo_cache through operations.cache while
preserving identical CLI output. commands/monitor.py is left untouched
(its refactor is optional per the plan; the Windows/Linux polymorphic
_service_status and the disk not-found detection don't map cleanly onto
the new pure functions without risking behavioural drift).
A job killed mid-flight never released its IP lease; with only 4 pool addresses a single leak exhausted the pool ("IP pool exhausted after 60s"). IpSlotPool.reconcile() releases leases whose owner clone-dir is gone; vm cleanup runs it only against the real build_vms base. Closes TODO §2.8.
TODO 2.9 recorded a retention policy that warned about low disk while freeing
0.0 GB twice in a row, because it governed artifacts, logs and IP leases -- 52
KB in total on this host -- while 195 G of backups sat outside its reach.
Adding an unbounded directory of ~645 MB entries, one per Cargo.lock change,
without extending retention in the same change would repeat that at a larger
scale.
The window is the tighter of the global retention and the cache's own, so the
aggressive low-disk mode reaches the cache rather than stopping at the light
directories.
Sizes are read before deleting, not after. Summing st_size over files that have
already been unlinked yields 0 -- which is exactly how a policy ends up
reporting that it freed nothing while doing real work. A test asserts the
reported figure.
Default on, because the cache is safe by construction rather than by
convention: it engages only where a Cargo.lock exists, and refuses to publish
anything the prune assertion cannot vouch for. Two independent off switches --
the flag for one job, `enabled = false` for the host.
The store root is resolved after the environment overrides are applied, so
CI_ROOT relocates it along with everything else, while an explicit root in the
config still wins. Both directions are covered by tests.
Also asserts that the Windows branch never receives the argument at all: the
cache is Linux-only and the two branches stay separate, per AGENTS.md.
Restore runs after the source is in place and before the build; publish runs
only after a successful build and only on a miss, so the steady state uploads
nothing and a failed build can never become an entry. The whole path is
best-effort: a cache problem degrades to a cold build with a message rather
than turning a good build red.
Gated on Cargo.lock being present, so non-Rust jobs are untouched with no flag
to set. The Windows branch is not touched at all.
The prune script is read through importlib.resources rather than a path
relative to the repo root: the production venv is installed non-editable, so a
parents[] walk lands in site-packages where scripts/ does not exist.
The fake transport gained substring-matched canned results; indexing by call
position broke as soon as the cache added calls mid-flow.
Mutation-tested rather than assumed. Three guards were each disabled in turn --
the publish call, the prune-refusal check, the packing-failure check -- and a
test went red for each. The prune-refusal test initially survived its mutant:
it passed because the fake's fetch left no file behind, so publishing failed
for an unrelated reason. Fixed by making fetch write a real file, so the only
thing preventing publication is the guard under test.
One immutable file per key, published by rename(2) inside the store. Two jobs
publishing the same key both produce a valid entry and the last wins; a reader
holding the previous one keeps reading it. No advisory locking, which is the
property a VMware shared folder cannot offer with several jobs in flight --
the objection raised in the furyrs thread.
The key joins its parts with NUL so ("ab","c") and ("a","bc") cannot collide,
and carries a format version so the whole store can be invalidated at once
when the prune rules change and older entries stop being trustworthy.
The feature set is deliberately absent from the key: cargo already encodes it
in each artifact's own hash suffix, so several feature graphs coexist in one
entry rather than fragmenting the store into one entry per variant.
Four defects, each of which could waste a full template rebuild:
1. getGuestIPAddress was called without -wait, so vmrun answered "The VMware
Tools are not running" unless the Tools happened to have registered the IP
at that exact instant. A guest that was booted, on the network and answering
SSH could still burn the entire 10-minute Step 7 budget. This race is the
likely cause of every intermittent "no guest IP" incident so far, including
ones previously attributed to wedged vmnet DHCP. AGENTS.md #10 already
prescribed -wait; the code did not follow it.
2. --force did not check whether the target VM was powered on. It deleted the
live .vmdk.lck as a stale artifact and Step 4 rewrote the base VMDK under
the running guest, killing it; the damage surfaced ten minutes later as an
IP timeout because `vmrun list` still showed the OLD instance.
3. Step 6 accepted "VM appears in vmrun list" as proof of power-on, which a
pre-existing instance satisfies. Check the exit code of `vmrun start`.
4. The cloud-init seed ISO was detached while the VM was still running, and
VMware rewrote the VMX from its in-memory copy at power-off, restoring
ide1:0.present = "TRUE". Every clone would boot with the seed attached
(AGENTS.md #6). Detach it in prepare, after power-off and before the
snapshot.
Also in this change:
- --then-prepare chains deploy into the matching prepare in one run, reusing
the VM deploy leaves running. The two commands stay separately invocable.
The guest toolchain script is resolved and verified BEFORE any VM work, so a
wrong working directory costs two seconds instead of failing the hand-off
after a 45-minute deploy; --toolchain-script allows running from anywhere.
- Corrected the final messages: deploy-rust told the user to run prepare-linux,
and both deploy commands reported a snapshot name (PostInstall-Linux) that no
command ever creates. The prepare hand-off hint passed a bare filename to
--template and omitted --snapshot, so copy-pasting it cloned BaseClean, the
Windows snapshot.
- template backup --exclude, repeatable, for retired templates. An unknown name
or --exclude without --all-templates is an error rather than a silent no-op.
test_deploy_linux_full_happy_path declared the VM already running before Step 6,
an impossible state that masked the new --force guard; its vmrun mock is now
stateful.
Up front, read Winlogon\Shell; if it isn't already cmd.exe, set it and do a
clean reboot before running the toolchain — so the shell is genuinely cmd
(not a killed-explorer half-state) for the whole servicing run. Idempotent:
when the shell is already cmd (re-runs), skip the reboot. Replaces the
earlier kill-explorer approach.
The one-shot idle sample fired ~30s after the final reboot, but Windows
Update / CBS (TiWorker) keeps finalizing for minutes — so a still-busy
template was wrongly rejected. Poll until CPU and RAM are both under
threshold (up to --idle-timeout, default 600s) instead of sampling once,
and report the busiest process to aid diagnosis. Adds --idle-timeout.
A fast reboot never showed TCP/5986 "down" between polls, so the old
down→up wait could hang on "waiting for VM to go down" until timeout.
Replace it: record the guest LastBootUpTime (FileTime ticks) before the
reboot, then poll until the port is back AND the boot time is newer — a
signal immune to reboot speed. Applies to both the 3010 loop and the final
reboot. Drops _wait_tcp_down in favour of _tcp_open + _guest_boot_ticks.
After the final reboot and just before the graceful shutdown for the
snapshot, sample the guest's idle CPU and used-RAM percentages (CIM
Win32_PerfFormattedData — locale-independent — averaged over ~5s after a
settle delay). If either exceeds its threshold, abort prepare without
taking the snapshot, so a busy/leaky template isn't frozen into every clone.
New options: --max-idle-cpu (10%), --max-idle-ram (50%), --idle-settle
(30s), --skip-idle-check.
pypsrp's execute_ps is fully blocking, so Install-CIToolchain / Windows
Update output only appeared at the end behind a "still running (Ns)"
heartbeat. Add WinRmTransport.run_streaming: drive the script via pypsrp's
incremental begin_invoke/poll_invoke, emitting output, Write-Host
(information) and error stream lines as they arrive over the same WinRM
channel.
_run_with_heartbeat now prefers run_streaming (suppressing the CI_EXITCODE
marker line) and falls back to the blocking heartbeat only on an unexpected
pypsrp error — TransportError still propagates so the 3010 reboot handling
is unaffected.
On exit 3010 the old path issued Restart-Computer, slept 30s, then probed
WinRM. With a pending WU/CBS install the restart is delayed minutes while
the WinRM session stays alive, so the probe saw the still-up pre-reboot
session as "ready" and re-ran the toolchain — which then died mid-run when
the real reboot kicked in (looping "treating as 3010").
Add `_reboot_and_wait`: issue the reboot, then wait for TCP/5986 to go
*down* (`_wait_tcp_down`, up to 300s) before waiting for it to come back
and re-probing WinRM. Use it for both the 3010 loop and the final reboot.
deploy-win patched its input into vm_dir/install-noprompt.iso and only its
own sidecar cache could skip the repack — so feeding an already-patched ISO
(e.g. from `template patch-win-iso`, or via --win-iso) triggered a wasteful
second extract+repack that double-injected the answer files.
deploy-win now detects a patched input (autounattend.xml present at the ISO
root via `iso_patch.iso_has_root_file`) and uses it as-is, skipping Step 4.
Raw ISOs are still patched once and cached as before.
Add `ci template check-win-iso`: parse the massgrave windows-server-links
markdown source on GitHub for the newest Windows Server 2025 build, file
name and link, and report whether it is already in <CI_ROOT>/iso/. The
download itself stays manual — the massgrave link is behind a Cloudflare
Turnstile file host, not scriptable.
Add `ci template patch-win-iso --in ISO [--out ISO]`: standalone equivalent
of deploy-win Step 4 — inject autounattend.xml + post-install.ps1 and a
no-prompt boot image into a manually downloaded ISO, producing a
ready-to-deploy `.patched.iso`.
winimage/massgrave.py does the parse (no auto-download). RUNBOOK §5
documents the check → download → patch → deploy workflow. Coverage 93.6%.
Simone