docs: record the LinuxRust2404 template, its failure modes and open decisions
- AGENTS.md: LinuxRust2404 reference, plus errors #14-#16 -- deploy --force against a powered-on VM, the three distinct causes behind the identical "VMware Tools are not running" message, and the orphan snapshot deltas --force leaves behind after deleting the .vmsd. - docs/RUNBOOK.md §16: triage for a Step 7 IP timeout, ordered cheapest check first, with a Quick Reference entry. - docs/RUNBOOK-PhaseC.md: deploy-rust/prepare-rust, the ownership and non-login shell constraints on /opt/rust, --then-prepare, and how to route a label to a template (a second runner instance, with the verification step). - TODO.md §2.9: retention run covers only artifacts, logs and IP leases, so it warned about low disk while freeing 0.0 GB two runs in a row. Mitigated by hand; the automated policy is still blind to what actually weighs. - TODO.md §6.5: act_runner v1.0.4 -> v2.x evaluated and deferred, with the trigger for reopening. v2.1.0 exposes no RUNNER_LABELS either (checked against the binary), so the two-runner design is not a stopgap.
This commit is contained in:
@@ -77,10 +77,12 @@ sudo); `--uninstall` removes the block. The script defines two helpers:
|
||||
| `ci vmware --vmx ...` | Open a VMX in the VMware Workstation GUI | [§4](#4-manual-vm-lifecycle) |
|
||||
| `ci creds set --user ...` | Store guest credentials in the keyring | [§5](#5-credentials--template-provisioning) |
|
||||
| `ci template deploy-linux` | Build the Linux template VM | [§5](#5-credentials--template-provisioning) |
|
||||
| `ci template deploy-rust` | Build the Rust Linux template VM (LinuxRust2404) | [§5](#5-credentials--template-provisioning) |
|
||||
| `ci template check-win-iso` | Report the newest WS2025 ISO from massgrave + cache status | [§5](#5-credentials--template-provisioning) |
|
||||
| `ci template patch-win-iso` | Inject autounattend + no-prompt boot into a Windows ISO | [§5](#5-credentials--template-provisioning) |
|
||||
| `ci template deploy-win` | Build the Windows Server 2025 template VM (download→patch→deploy) | [§5](#5-credentials--template-provisioning) |
|
||||
| `ci template prepare-linux` | Provision the Linux template over SSH | [§5](#5-credentials--template-provisioning) |
|
||||
| `ci template prepare-rust` | Provision the Rust Linux template over SSH | [§5](#5-credentials--template-provisioning) |
|
||||
| `ci template prepare-win` | Provision a Windows template over WinRM | [§5](#5-credentials--template-provisioning) |
|
||||
| `ci template backup` | Timestamped copy of the template directories | [§5](#5-credentials--template-provisioning) |
|
||||
| `ci retention run` | Purge old artifacts and logs | [§6](#6-maintenance) |
|
||||
@@ -412,12 +414,139 @@ Run `ci template <sub> --help` for the (many) per-template options.
|
||||
|
||||
```bash
|
||||
ci template deploy-linux # build the Linux template VM from scratch (Ubuntu 24.04)
|
||||
ci template deploy-rust # build the Rust Linux template VM (LinuxRust2404)
|
||||
ci template deploy-win # build the Windows Server 2025 template VM (see below)
|
||||
ci template prepare-linux # provision an existing Linux template over SSH
|
||||
ci template prepare-rust # provision the Rust Linux template over SSH
|
||||
ci template prepare-win # provision a Windows template over WinRM (Linux-host driven)
|
||||
ci template backup # timestamped backup of the template directories
|
||||
```
|
||||
|
||||
### `deploy-rust` / `prepare-rust` — the Rust Linux template (LinuxRust2404)
|
||||
|
||||
A separate template for Rust workspaces (e.g. Tauri apps): same Ubuntu 24.04
|
||||
cloud image and guest layout as `LinuxBuild2404`, but `prepare-rust` runs
|
||||
`template/Install-CIToolchain-LinuxRust2404.sh`, which additionally bakes:
|
||||
|
||||
- the Tauri v2 webview/GTK development libraries
|
||||
(`libwebkit2gtk-4.1-dev`, `libgtk-3-dev`, `libayatana-appindicator3-dev`,
|
||||
`librsvg2-dev`);
|
||||
- a system-wide Rust toolchain via rustup under `/opt/rust`
|
||||
(`RUSTUP_HOME=/opt/rust/rustup`, `CARGO_HOME=/opt/rust/cargo`) with the
|
||||
`rustfmt` and `clippy` components. The tree is `chown`ed to `ci_build`
|
||||
**after** the installs, not only before — otherwise every write the build
|
||||
user makes (toolchain sync, dependency download into
|
||||
`CARGO_HOME/registry`, `cargo install`) fails with `Permission denied`.
|
||||
Exposure is threefold because the orchestrator runs build commands in a
|
||||
**non-login** shell (`exec_command` + `sh -c`), which never sources
|
||||
`profile.d`: `/etc/profile.d/rust.sh` for interactive shells,
|
||||
`/etc/environment` (read by `pam_env` at SSH session open) for the two
|
||||
home variables, and `/usr/local/bin` symlinks for the rustup proxies;
|
||||
- the `cargo-audit` binary (skip with `--no-cargo-audit`). The audit
|
||||
**advisory DB is never baked** — `cargo audit` fetches it at runtime.
|
||||
|
||||
```bash
|
||||
ci template deploy-rust # same flags as deploy-linux; 60 GB disk default
|
||||
ci template prepare-rust # default VMX: <CI_TEMPLATES>/LinuxRust2404/...
|
||||
# snapshot: BaseClean-LinuxRust
|
||||
ci template prepare-rust --rust-channel 1.97.1 # pin an exact toolchain
|
||||
|
||||
# Both phases in one run, unattended. deploy and prepare stay separate
|
||||
# commands — --then-prepare only chains them, reusing the VM deploy left
|
||||
# running (no second power-on, no second IP detection).
|
||||
#
|
||||
# Run from the repo root: the guest toolchain script is looked up under
|
||||
# <cwd>/template/. With --then-prepare this is checked BEFORE the deploy
|
||||
# starts, so a wrong directory costs two seconds instead of failing the
|
||||
# hand-off after a ~45-minute deploy. Pass --toolchain-script to run from
|
||||
# anywhere.
|
||||
cd /opt/ci/local-ci-cd-system
|
||||
ci template deploy-rust --then-prepare --rust-channel 1.97.1
|
||||
ci template deploy-linux --then-prepare
|
||||
|
||||
# from any directory
|
||||
ci template deploy-rust --then-prepare --rust-channel 1.97.1 \
|
||||
--toolchain-script /opt/ci/local-ci-cd-system/template/Install-CIToolchain-LinuxRust2404.sh
|
||||
```
|
||||
|
||||
`--snapshot-name` on `deploy-linux`/`deploy-rust` names the snapshot that
|
||||
`prepare-*` will create (`BaseClean-Linux` / `BaseClean-LinuxRust`) — the
|
||||
deploy commands take no snapshot of their own, unlike `deploy-win`. With
|
||||
`--then-prepare` the value is handed to `prepare-*`, so overriding it on the
|
||||
deploy side changes what actually gets created.
|
||||
|
||||
Both `prepare-linux` and `prepare-rust` default `--vmx` to their template VMX
|
||||
under `<CI_TEMPLATES>` (`LinuxBuild2404` / `LinuxRust2404`) when neither
|
||||
`--vmx` nor `--ip` is passed — so the plain `ci template prepare-rust` just
|
||||
works after `deploy-rust`. Pass `--ip` to provision a running guest without
|
||||
VM lifecycle (no auto power-on/shutdown/snapshot).
|
||||
|
||||
`--rust-channel` defaults to `stable` — a moving target: when a new stable
|
||||
ships, `rustup show` in a job re-downloads the toolchain at runtime. Pin an
|
||||
exact version for a fully deterministic snapshot (the consumer repo decides
|
||||
this via its `rust-toolchain.toml`). Refresh the snapshot when the pinned
|
||||
channel or the apt security baseline moves.
|
||||
|
||||
Two ways to route jobs to it:
|
||||
|
||||
1. **Per job (no host change).** Pass `template-path` + `snapshot-name` to the
|
||||
`local-ci-build` composite action. This cannot leave a job unschedulable,
|
||||
which is why consumer repos default to it.
|
||||
2. **By label** — `runs-on: rust-build`. This needs a **second act_runner
|
||||
instance**, not just an extra label on the existing one: act_runner reads a
|
||||
single `runner.envs` block per process and never tells the workflow which
|
||||
label picked the job (there is no `RUNNER_LABELS` variable — v1.0.4 exposes
|
||||
only `RUNNER_OS/ARCH/TEMP/TOOL_CACHE/DEBUG/VERSION`). A label can select a
|
||||
template only by selecting a runner whose envs name that template.
|
||||
|
||||
```bash
|
||||
# 1. mint a registration token from the stored GiteaPAT (0600 file, never
|
||||
# printed — pass it by path so it stays out of `ps` and shell history)
|
||||
sudo -u ci-runner /opt/ci/venv/bin/python tools/gitea-runner-token.py \
|
||||
--out /var/lib/ci/.rust-runner-regtoken
|
||||
|
||||
# 2. provision + register + enable the unit
|
||||
sudo ./scripts/install-rust-runner.sh \
|
||||
--token-file /var/lib/ci/.rust-runner-regtoken
|
||||
|
||||
# 3. the token is single-use — destroy it
|
||||
sudo shred -u /var/lib/ci/.rust-runner-regtoken
|
||||
```
|
||||
|
||||
Provisions `/var/lib/ci/runner-rust/` from `deploy/runner-rust-config.yaml`
|
||||
(label `rust-build:host`, `GITEA_CI_LINUX_TEMPLATE_PATH` →
|
||||
`LinuxRust2404`, `GITEA_CI_LINUX_SNAPSHOT_NAME` → `BaseClean-LinuxRust`),
|
||||
registers it, and installs `act-runner-rust.service`. Its `capacity` adds
|
||||
to the main runner's — keep the sum inside the host RAM budget (currently
|
||||
2 + 1 = 3 concurrent build VMs).
|
||||
|
||||
**Verify the label is announced before any workflow depends on it**: a
|
||||
`runs-on` value that no runner advertises leaves the job unschedulable with
|
||||
no error anywhere — silently zero runs.
|
||||
|
||||
```bash
|
||||
sudo -u ci-runner /opt/ci/venv/bin/python tools/gitea-runner-list.py \
|
||||
--label rust-build # exit 1 unless an ONLINE runner announces it
|
||||
```
|
||||
|
||||
Note that `runner.file` in the config is an **absolute** path. act_runner
|
||||
resolves a relative `file:` against the process CWD, so registering from
|
||||
anywhere else drops the registration state (which contains the runner
|
||||
token) into that directory and the daemon then fails with
|
||||
`registration file not found`.
|
||||
|
||||
Both runner services share **one binary**, `/opt/ci/act_runner/act_runner`.
|
||||
Replacing it upgrades `act-runner.service` and `act-runner-rust.service`
|
||||
together — a staged rollout needs a second binary alongside it and the
|
||||
`ExecStart` of one unit pointed at it. Current version is v1.0.4; the
|
||||
upgrade to v2.x was evaluated and deferred (TODO §6.5).
|
||||
|
||||
Note that `guest-os: 'Auto'` cannot pick a template by itself, for the same
|
||||
reason: the label is invisible to the action, and the VMX `guestOS` field is
|
||||
only readable *after* the clone. With `Auto` the action requires
|
||||
`template-path` **and** `snapshot-name` and fails fast otherwise, instead of
|
||||
silently falling back to the Windows template as it did before.
|
||||
|
||||
### `deploy-win` — build the Windows Server 2025 template
|
||||
|
||||
Takes a Windows Server 2025 install ISO — downloaded from `--iso-url` (cached
|
||||
|
||||
@@ -250,6 +250,7 @@ Estimated time: 2-4 hours including Windows Update.
|
||||
| Template corrupt | Remove `*.lck` files; if persistent, restore from `F:\CI\Backups\` |
|
||||
| Snapshot missing | `vmrun listSnapshots <vmx>`; update `GITEA_CI_SNAPSHOT_NAME` |
|
||||
| IP collision | `Remove-Item F:\CI\State\ip-leases\*.lease`; lower `capacity` |
|
||||
| Step 7: no guest IP | `ps -eo pid,lstart,args \| grep vmware-vmx` — see §16 |
|
||||
|
||||
---
|
||||
|
||||
@@ -868,3 +869,92 @@ This is **optional** — the default boots-to-Linux behaviour is intentional so
|
||||
the machine comes back as the Linux CI host after any unattended reboot
|
||||
(power-loss, kernel update). Leave it as-is unless Windows becomes the daily
|
||||
driver.
|
||||
|
||||
---
|
||||
|
||||
## 16. `deploy-*` step 7 in timeout — nessun IP dal guest
|
||||
|
||||
**Sintomo.** `deploy-linux` / `deploy-rust` cicla su
|
||||
`... waiting for guest IP (open-vm-tools) ...` per 10 minuti e chiude con
|
||||
`Error: [Deploy Step 7] Timed out waiting for guest IP.` Interrogando a mano:
|
||||
|
||||
```
|
||||
$ vmrun -T ws getGuestIPAddress <vmx>
|
||||
Error: The VMware Tools are not running in the virtual machine
|
||||
```
|
||||
|
||||
Quel messaggio è ambiguo: esce identico sia quando il guest **non ha** un IP,
|
||||
sia quando ce l'ha ma i Tools non rispondono. Tre cause distinte, in ordine di
|
||||
verifica dal più veloce al più lento.
|
||||
|
||||
### 16.1 Collisione con una VM già accesa (la più insidiosa)
|
||||
|
||||
Succede quando si lancia `deploy-* --force` mentre la stessa VMX è in uso —
|
||||
tipicamente un `prepare-*` ancora in corso. Occorso il 2026-07-22 su
|
||||
`LinuxRust2404`.
|
||||
|
||||
```bash
|
||||
ps -eo pid,lstart,etime,args | grep vmware-vmx | grep -v grep
|
||||
```
|
||||
|
||||
Se il `vmware-vmx` vivo è **più vecchio** del deploy appena lanciato, è questo
|
||||
il caso: il deploy ha cancellato il `.vmdk.lck` della VM viva credendolo stale,
|
||||
ha riscritto il VMDK base sotto il guest in esecuzione, e il suo power-on è
|
||||
morto subito perché la VMX era occupata. Il controllo
|
||||
`[OK] VM appears in vmrun list` **non protegge**: vede l'istanza vecchia.
|
||||
|
||||
Recupero:
|
||||
|
||||
```bash
|
||||
sudo -u ci-runner vmrun -T ws stop <vmx> hard
|
||||
sudo rm -f <vm-dir>/<disk>-00000*.vmdk # delta orfani, vedi 16.4
|
||||
sudo -u ci-runner vmrun -T ws list # deve dire 0
|
||||
# poi rilanciare il deploy, con nient'altro in esecuzione sulla stessa VMX
|
||||
```
|
||||
|
||||
Non risolvere con un retry: riavviare una VM già corrotta ripete il fallimento
|
||||
con la causa ormai illeggibile.
|
||||
|
||||
### 16.2 DHCP dell'host incastrato
|
||||
|
||||
```bash
|
||||
ip -4 addr show vmnet8 # atteso: inet 192.168.79.1/24
|
||||
ps -ef | grep vmnet-dhcpd # atteso: due istanze (vmnet1 + vmnet8)
|
||||
```
|
||||
|
||||
Se `vmnet8` manca o `vmnet-dhcpd` non c'è:
|
||||
|
||||
```bash
|
||||
sudo systemctl restart vmware-networks # oppure vmware-host.service
|
||||
```
|
||||
|
||||
### 16.3 Guest senza lease, oppure Tools non ancora pronti
|
||||
|
||||
```bash
|
||||
grep -A1 '^lease' /etc/vmware/vmnet8/dhcpd/dhcpd.leases | tail -20
|
||||
ip neigh show dev vmnet8
|
||||
```
|
||||
|
||||
Se compare un lease per il MAC della VM, **il guest è vivo e ha rete**: mancano
|
||||
solo i Tools. Sondare direttamente la porta 22 invece di aspettare
|
||||
`getGuestIPAddress`:
|
||||
|
||||
```bash
|
||||
timeout 5 bash -c 'cat < /dev/null > /dev/tcp/<ip>/22' && echo "SSH aperta"
|
||||
```
|
||||
|
||||
Se invece non c'è nessun lease e le due verifiche precedenti sono pulite, il
|
||||
guest non sta completando il boot: cloud-init fallito o immagine corrotta.
|
||||
|
||||
### 16.4 Delta di snapshot orfani
|
||||
|
||||
`deploy-*` cancella `<vm>.vmsd` per evitare l'errore di resize sulla catena
|
||||
snapshot, ma **non** rimuove i file delta `<disk>-00000N.vmdk`. Restano con un
|
||||
`parentCID` che non corrisponde più al base ricreato. Sono innocui — la VMX
|
||||
punta al base, riconoscibile da `parentCID=ffffffff` — ma confondono la
|
||||
diagnosi. Verifica della catena:
|
||||
|
||||
```bash
|
||||
grep -i 'scsi0:0.fileName' <vmx>
|
||||
head -c 2000 <disk>.vmdk | strings | grep -E '^CID|parentCID'
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user