Simone
14a67ead79
- AGENTS.md: LinuxRust2404 reference, plus errors #14-#16 -- deploy --force against a powered-on VM, the three distinct causes behind the identical "VMware Tools are not running" message, and the orphan snapshot deltas --force leaves behind after deleting the .vmsd. - docs/RUNBOOK.md §16: triage for a Step 7 IP timeout, ordered cheapest check first, with a Quick Reference entry. - docs/RUNBOOK-PhaseC.md: deploy-rust/prepare-rust, the ownership and non-login shell constraints on /opt/rust, --then-prepare, and how to route a label to a template (a second runner instance, with the verification step). - TODO.md §2.9: retention run covers only artifacts, logs and IP leases, so it warned about low disk while freeing 0.0 GB two runs in a row. Mitigated by hand; the automated policy is still blind to what actually weighs. - TODO.md §6.5: act_runner v1.0.4 -> v2.x evaluated and deferred, with the trigger for reopening. v2.1.0 exposes no RUNNER_LABELS either (checked against the binary), so the two-runner design is not a stopgap.
184 lines
9.1 KiB
Markdown
184 lines
9.1 KiB
Markdown
# CLAUDE.md
|
||
|
||
This file provides guidance to Claude Code (claude.ai/code) when working with this repository.
|
||
|
||
---
|
||
|
||
## Project Overview
|
||
|
||
Self-hosted CI/CD system running on a Windows 11 host with VMware Workstation. Each build job runs inside an ephemeral VM (linked clone from a frozen template snapshot), communicates via WinRM (Windows guests) or SSH (Linux guests), and is destroyed after the job.
|
||
|
||
**Current state**: The Python orchestrator (`src/ci_orchestrator/`) is the production entry point. The PowerShell scripts in `scripts/` are 3-line shims that forward to the Python CLI. All new logic goes in Python.
|
||
|
||
**Phase C (PowerShell-free Linux host)**: the manual burn-in / benchmark / smoke / host-validation / credential-setup tooling that used to run on the Linux host via `pwsh` is now native Python: `bench run`, `bench measure`, `smoke run`, `validate host`, `validate guest`, `creds set`. The Linux host no longer needs `pwsh` for any operation — it may be uninstalled (`setup-host-linux.sh` no longer installs it by default; opt in with `--with-pwsh`). PSScriptAnalyzer lint runs on a Windows runner on-demand, not on the Linux runner.
|
||
|
||
---
|
||
|
||
## Development Commands
|
||
|
||
### Python (primary)
|
||
|
||
The production venv is at `/opt/ci/venv` (Linux host). To invoke the CLI as the service user:
|
||
|
||
```bash
|
||
sudo -u ci-runner /opt/ci/venv/bin/python -m ci_orchestrator --help
|
||
```
|
||
|
||
```bash
|
||
# Dev venv setup (one-time)
|
||
python -m venv .venv
|
||
.\.venv\Scripts\python.exe -m pip install -e ".[dev]"
|
||
|
||
# Run all tests with coverage (gate: ≥90%)
|
||
python -m pytest tests/python -q --cov=ci_orchestrator --cov-fail-under=90
|
||
|
||
# Run a single test file
|
||
python -m pytest tests/python/test_commands_job.py -q
|
||
|
||
# Lint
|
||
python -m ruff check src tests/python
|
||
|
||
# Type-check (strict)
|
||
python -m mypy --strict src
|
||
|
||
# CLI
|
||
python -m ci_orchestrator --help
|
||
```
|
||
|
||
**Important**: The production venv at `F:\CI\python\venv` must be installed **non-editable** (`pip install .`, not `-e`), because act_runner runs as LocalSystem and cannot resolve editable `.pth` files from a transient RunnerWork path. Re-run `pip install .` into the production venv after every code change. The dev `.venv` may use `-e`.
|
||
|
||
### PowerShell linting
|
||
|
||
```powershell
|
||
Invoke-ScriptAnalyzer -Path scripts\ -Recurse -Settings PSScriptAnalyzerSettings.psd1 -Severity Error,Warning
|
||
# Auto-fix formatting:
|
||
Invoke-ScriptAnalyzer -Path scripts\ -Recurse -Fix
|
||
```
|
||
|
||
---
|
||
|
||
## Architecture
|
||
|
||
### Pipeline flow
|
||
|
||
```
|
||
git push → Gitea Actions → act_runner (Windows service) → python -m ci_orchestrator job
|
||
→ vm new (linked clone)
|
||
→ wait-ready (WinRM poll or SSH poll)
|
||
→ build run (WinRM Invoke-Command or SSH+SFTP)
|
||
→ artifacts collect (WinRM Copy-Item or SCP)
|
||
→ vm remove (vmrun stop + deleteVM)
|
||
```
|
||
|
||
The host **never executes build tools directly** — it only orchestrates VM lifecycle.
|
||
|
||
### Python package layout (`src/ci_orchestrator/`)
|
||
|
||
| Module | Role |
|
||
|--------|------|
|
||
| `__main__.py` | click entry point; registers all sub-command groups |
|
||
| `config.py` | `Config` dataclass + TOML/env loader; resolution: OS defaults → `config.toml` → env vars |
|
||
| `credentials.py` | `CredentialStore` Protocol + `KeyringCredentialStore` |
|
||
| `backends/protocol.py` | `VmBackend` Protocol + `VmHandle` + `VmState` (hypervisor-agnostic) |
|
||
| `backends/workstation.py` | `WorkstationVmrunBackend` — wraps `vmrun.exe` subprocesses |
|
||
| `backends/__init__.py` | `load_backend(config)` factory — the only import application code should use |
|
||
| `transport/winrm.py` | WinRM transport via `pypsrp` |
|
||
| `transport/ssh.py` | SSH/SFTP transport via `paramiko` |
|
||
| `transport/errors.py` | `TransportConnectError`, `TransportCommandError` |
|
||
| `commands/job.py` | Full end-to-end orchestrator (auto-detects guest OS from VMX `guestOS` field) |
|
||
| `commands/vm.py` | `vm new / remove / cleanup` |
|
||
| `commands/build.py` | `build run` |
|
||
| `commands/artifacts.py` | `artifacts collect` |
|
||
| `commands/wait.py` | `wait-ready` |
|
||
| `commands/monitor.py` | `monitor disk / runner` |
|
||
| `commands/report.py` | `report job` |
|
||
|
||
**Key design rule**: Application code (especially `commands/job.py`) must import `backends.load_backend` only — never `WorkstationVmrunBackend` directly. This keeps the Phase D ESXi extension path open.
|
||
|
||
### Transport selection
|
||
|
||
`commands/job.py` reads `guestOS = "..."` from the cloned VMX. `ubuntu-*` → Linux/SSH (`transport/ssh.py`); anything else → Windows/WinRM (`transport/winrm.py`).
|
||
|
||
### VM templates and snapshots
|
||
|
||
| Template | VMX path | Snapshot | Transport |
|
||
|----------|----------|----------|-----------|
|
||
| WinBuild2025 | `F:\CI\Templates\WinBuild2025\WinBuild2025.vmx` | `BaseClean` | WinRM HTTPS/5986 |
|
||
| WinBuild2022 | `F:\CI\Templates\WinBuild2022\WinBuild2022.vmx` | `BaseClean` | WinRM HTTPS/5986 |
|
||
| LinuxBuild2404 | `F:\CI\Templates\LinuxBuild2404\LinuxBuild2404.vmx` | `BaseClean-Linux` | SSH/22 (`ci_build`, key `F:\CI\keys\ci_linux`) |
|
||
| LinuxRust2404 | `LinuxRust2404/LinuxRust2404.vmx` (templates dir) | `BaseClean-LinuxRust` | SSH/22 (same user/key as LinuxBuild2404) |
|
||
|
||
`LinuxRust2404` is a LinuxBuild2404 variant for Rust workspaces (Tauri):
|
||
`ci template deploy-rust` / `ci template prepare-rust` provision it via
|
||
`template/Install-CIToolchain-LinuxRust2404.sh` (webview/GTK libs + system-wide
|
||
rustup under `/opt/rust` + cargo-audit binary; the audit advisory DB is fetched
|
||
at runtime, never baked). `--rust-channel` pins the baked toolchain (default
|
||
`stable`, a moving target).
|
||
|
||
Snapshots **must be taken from powered-off state** — presence of `.vmem` in the template dir means the snapshot captured running memory and `vmrun clone ... linked` will fail.
|
||
|
||
### Configuration
|
||
|
||
The config file is auto-discovered at `$CI_ROOT/config.toml` (Linux default: `/var/lib/ci/config.toml`; Windows: `F:\CI\config.toml`). Set `$CI_CONFIG` to override. The file at `/var/lib/ci/config.toml` is the live config on the Linux host — keep its `[paths]` section pointing to `/var/lib/ci/...` (not the old Windows `F:/CI/...` paths). Environment variables `CI_ROOT`, `CI_TEMPLATES`, `CI_BUILD_VMS`, `CI_ARTIFACTS`, `CI_KEYS` override the file. `CI_PYTHON_LAUNCHER` in `runner/config.yaml` is the single source of truth for the system Python path used to bootstrap the production venv.
|
||
|
||
---
|
||
|
||
## PowerShell 5.1 Constraints
|
||
|
||
These constraints apply to the `.ps1` / `.psm1` scripts that still run **inside Windows guests** and on the optional Windows runner. The **Linux host itself no longer runs any `.ps1`** (Phase C) — do not add PowerShell to the Linux host orchestration path.
|
||
|
||
All `.ps1` / `.psm1` scripts run on **Windows PowerShell 5.1** (not Core/7). Every script must start with:
|
||
|
||
```powershell
|
||
#Requires -Version 5.1
|
||
Set-StrictMode -Version Latest
|
||
$ErrorActionPreference = 'Stop'
|
||
```
|
||
|
||
Forbidden PS 7+ syntax (use the PS 5.1 alternatives):
|
||
|
||
| Forbidden | Use instead |
|
||
|-----------|-------------|
|
||
| `$x ??= 'val'` | `if ($null -eq $x) { $x = 'val' }` |
|
||
| `$a ?? $b` | `if ($null -ne $a) { $a } else { $b }` |
|
||
| `cmd1 && cmd2` | `cmd1; if ($LASTEXITCODE -eq 0) { cmd2 }` |
|
||
| Ternary `$a ? $b : $c` | `if ($a) { $b } else { $c }` |
|
||
| `ForEach-Object -Parallel` | `foreach` or `Start-Job` |
|
||
|
||
Always check `$LASTEXITCODE` after `vmrun`, `ssh`, `scp` and `throw` if non-zero.
|
||
|
||
---
|
||
|
||
## Python Conventions
|
||
|
||
- `click.echo` everywhere — no bare `print()`.
|
||
- No `subprocess.run(..., shell=True)`. Use `paramiko` for SSH, `pypsrp` for WinRM, `keyring` for credentials.
|
||
- No global variables — state lives in `Config` and objects passed explicitly.
|
||
- All public functions and exported helpers must have type hints. Avoid `Any` except in pytest fixtures/mocks.
|
||
- `ruff` + `mypy --strict` are enforced in CI. The coverage gate is **90%**.
|
||
|
||
---
|
||
|
||
## Known Gotchas (from `AGENTS.md`)
|
||
|
||
- **`vmrun getGuestIPAddress`** can return exit code 0 with `"Error: The VMware Tools are not running..."` on stdout. Filter lines starting with `"Error:"`.
|
||
- **Linux machine-id**: Ubuntu cloud images ship with a fixed `/etc/machine-id`. VMware DHCP assigns IPs by machine-id — clones get the same IP and collide. The `BaseClean-Linux` snapshot must be taken after truncating machine-id.
|
||
- **`& nativecmd 2>$null` under `$ErrorActionPreference='Stop'`**: native stderr is converted to an ErrorRecord before `2>$null` discards it, causing a terminating error. Wrap with `$ErrorActionPreference = 'SilentlyContinue'` or `2>&1 | Out-Null`.
|
||
- **Snapshot with VM running**: never take a template snapshot while the VM is powered on.
|
||
- **WinRM vs SSH separation**: do not modify the Windows WinRM branch and the Linux SSH branch in the same commit/PR. Keep `if ($GuestOS -eq 'Linux')` guards clean.
|
||
|
||
---
|
||
|
||
## Key Files Reference
|
||
|
||
| File | Purpose |
|
||
|------|---------|
|
||
| `AGENTS.md` | Environment reference + error registry — read before writing any script |
|
||
| `docs/ARCHITECTURE.md` | Full system diagram and component descriptions |
|
||
| `docs/CI-FLOW.md` | Step-by-step pipeline description |
|
||
| `docs/RUNBOOK.md` | Common incident triage |
|
||
| `TODO.md` | Roadmap, audit trail, backlog |
|
||
| `plans/` | Phase closeout docs (A1–A5, B5) and implementation plans |
|
||
| `config.example.toml` | Annotated config template |
|
||
| `tests/python/test_agents_errors.py` | Regression tests for `AGENTS.md` errors #9–#12 — do not remove |
|