# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with this repository.

---

## Project Overview

Self-hosted CI/CD system running on a Windows 11 host with VMware Workstation. Each build job runs inside an ephemeral VM (linked clone from a frozen template snapshot), communicates via WinRM (Windows guests) or SSH (Linux guests), and is destroyed after the job.

**Current state**: The Python orchestrator (`src/ci_orchestrator/`) is the production entry point. The PowerShell scripts in `scripts/` are 3-line shims that forward to the Python CLI. All new logic goes in Python.

**Phase C (PowerShell-free Linux host)**: the manual burn-in / benchmark / smoke / host-validation / credential-setup tooling that used to run on the Linux host via `pwsh` is now native Python: `bench run`, `bench measure`, `smoke run`, `validate host`, `validate guest`, `creds set`. The Linux host no longer needs `pwsh` for any operation — it may be uninstalled (`setup-host-linux.sh` no longer installs it by default; opt in with `--with-pwsh`). PSScriptAnalyzer lint runs on a Windows runner on-demand, not on the Linux runner.

---

## Development Commands

### Python (primary)

The production venv is at `/opt/ci/venv` (Linux host). To invoke the CLI as the service user:

```bash
sudo -u ci-runner /opt/ci/venv/bin/python -m ci_orchestrator --help
```

```bash
# Dev venv setup (one-time)
python -m venv .venv
.\.venv\Scripts\python.exe -m pip install -e ".[dev]"

# Run all tests with coverage (gate: ≥90%)
python -m pytest tests/python -q --cov=ci_orchestrator --cov-fail-under=90

# Run a single test file
python -m pytest tests/python/test_commands_job.py -q

# Lint
python -m ruff check src tests/python

# Type-check (strict)
python -m mypy --strict src

# CLI
python -m ci_orchestrator --help
```

**Important**: The production venv at `F:\CI\python\venv` must be installed **non-editable** (`pip install .`, not `-e`), because act_runner runs as LocalSystem and cannot resolve editable `.pth` files from a transient RunnerWork path. Re-run `pip install .` into the production venv after every code change. The dev `.venv` may use `-e`.

### PowerShell linting

```powershell
Invoke-ScriptAnalyzer -Path scripts\ -Recurse -Settings PSScriptAnalyzerSettings.psd1 -Severity Error,Warning
# Auto-fix formatting:
Invoke-ScriptAnalyzer -Path scripts\ -Recurse -Fix
```

---

## Architecture

### Pipeline flow

```
git push → Gitea Actions → act_runner (Windows service) → python -m ci_orchestrator job
    → vm new (linked clone)
    → wait-ready (WinRM poll or SSH poll)
    → build run (WinRM Invoke-Command or SSH+SFTP)
    → artifacts collect (WinRM Copy-Item or SCP)
    → vm remove (vmrun stop + deleteVM)
```

The host **never executes build tools directly** — it only orchestrates VM lifecycle.

### Python package layout (`src/ci_orchestrator/`)

| Module | Role |
|--------|------|
| `__main__.py` | click entry point; registers all sub-command groups |
| `config.py` | `Config` dataclass + TOML/env loader; resolution: OS defaults → `config.toml` → env vars |
| `credentials.py` | `CredentialStore` Protocol + `KeyringCredentialStore` |
| `backends/protocol.py` | `VmBackend` Protocol + `VmHandle` + `VmState` (hypervisor-agnostic) |
| `backends/workstation.py` | `WorkstationVmrunBackend` — wraps `vmrun.exe` subprocesses |
| `backends/__init__.py` | `load_backend(config)` factory — the only import application code should use |
| `transport/winrm.py` | WinRM transport via `pypsrp` |
| `transport/ssh.py` | SSH/SFTP transport via `paramiko` |
| `transport/errors.py` | `TransportConnectError`, `TransportCommandError` |
| `commands/job.py` | Full end-to-end orchestrator (auto-detects guest OS from VMX `guestOS` field) |
| `commands/vm.py` | `vm new / remove / cleanup` |
| `commands/build.py` | `build run` |
| `commands/artifacts.py` | `artifacts collect` |
| `commands/wait.py` | `wait-ready` |
| `commands/monitor.py` | `monitor disk / runner` |
| `commands/report.py` | `report job` |
| `operations/vms.py`, `cache.py`, `host.py`, `audit.py`, `executions.py` | Pure logic layer shared by the CLI and the web UI — dataclasses only, zero `click`/`print`. `commands/*` and `web/routes.py` both call these; never duplicate logic in either caller. |
| `commands/web.py` | `web serve` — `ci web serve --host --port` launches the dashboard (`uvicorn`/`fastapi` imported lazily so the core CLI stays dep-light without the `[web]` extra) |
| `web/app.py`, `auth.py`, `routes.py`, `security.py` | FastAPI dashboard: four info cards (Host with CPU/RAM/disk gauges, VMware modules, Runners, Cargo cache — no status-header tile row) + recent-executions view, Gitea OIDC login (Authlib, fail-closed allowlist), CSRF-protected confirm-gated mutating actions (VM remove/cleanup, cache purge, template backup), JSONL audit trail. In-process only — imports `operations/` + `backends.load_backend` directly, never shells the CLI. See `docs/WEB-UI.md`. |

**Key design rule**: Application code (especially `commands/job.py`) must import `backends.load_backend` only — never `WorkstationVmrunBackend` directly. This keeps the Phase D ESXi extension path open.

### Web management dashboard

`ci web serve` (package `web/`, optional `[web]` extra) is a server-rendered
FastAPI dashboard for VM/host management — see `docs/WEB-UI.md` for install,
Gitea OIDC setup, and operations, and `plans/web-ui-management-plan.md` for
the design history (steps W0–W7, all closed). Deployed live on this host as
`ci-web.service` (systemd, `ci-runner` user), bound to the LAN with Gitea
OIDC auth active — never bind `0.0.0.0`, and never bind non-loopback without
`[web.oidc]` configured (enforced at startup by `commands/web.py`).

The dashboard's main table is the **Executions** card (`operations/executions.py`,
`GET /executions`): recent Gitea Actions runs, live ones included, with the
build VMs grouped under the run that spawned them (job id
`{run_id}-{run_attempt}` ↔ clone name). It reuses the operator's OIDC access
token for the Gitea API — held **server-side only** in `app.state.gitea_tokens`,
never in the session cookie, which is signed but not encrypted and travels over
plain HTTP on the LAN. Optional keyring PAT fallback: `[web] gitea_token_cred`.

### Transport selection

`commands/job.py` reads `guestOS = "..."` from the cloned VMX. `ubuntu-*` → Linux/SSH (`transport/ssh.py`); anything else → Windows/WinRM (`transport/winrm.py`).

### VM templates and snapshots

| Template | VMX path | Snapshot | Transport |
|----------|----------|----------|-----------|
| WinBuild2025 | `F:\CI\Templates\WinBuild2025\WinBuild2025.vmx` | `BaseClean` | WinRM HTTPS/5986 |
| WinBuild2022 | `F:\CI\Templates\WinBuild2022\WinBuild2022.vmx` | `BaseClean` | WinRM HTTPS/5986 |
| LinuxBuild2404 | `F:\CI\Templates\LinuxBuild2404\LinuxBuild2404.vmx` | `BaseClean-Linux` | SSH/22 (`ci_build`, key `F:\CI\keys\ci_linux`) |
| LinuxRust2404 | `LinuxRust2404/LinuxRust2404.vmx` (templates dir) | `BaseClean-LinuxRust` | SSH/22 (same user/key as LinuxBuild2404) |

`LinuxRust2404` is a LinuxBuild2404 variant for Rust workspaces (Tauri):
`ci template deploy-rust` / `ci template prepare-rust` provision it via
`template/Install-CIToolchain-LinuxRust2404.sh` (webview/GTK libs + system-wide
rustup under `/opt/rust` + cargo-audit binary; the audit advisory DB is fetched
at runtime, never baked). `--rust-channel` pins the baked toolchain (default
`stable`, a moving target).

Snapshots **must be taken from powered-off state** — presence of `.vmem` in the template dir means the snapshot captured running memory and `vmrun clone ... linked` will fail.

### Configuration

The config file is auto-discovered at `$CI_ROOT/config.toml` (Linux default: `/var/lib/ci/config.toml`; Windows: `F:\CI\config.toml`). Set `$CI_CONFIG` to override. The file at `/var/lib/ci/config.toml` is the live config on the Linux host — keep its `[paths]` section pointing to `/var/lib/ci/...` (not the old Windows `F:/CI/...` paths). Environment variables `CI_ROOT`, `CI_TEMPLATES`, `CI_BUILD_VMS`, `CI_ARTIFACTS`, `CI_KEYS` override the file. `CI_PYTHON_LAUNCHER` in `runner/config.yaml` is the single source of truth for the system Python path used to bootstrap the production venv.

---

## PowerShell 5.1 Constraints

These constraints apply to the `.ps1` / `.psm1` scripts that still run **inside Windows guests** and on the optional Windows runner. The **Linux host itself no longer runs any `.ps1`** (Phase C) — do not add PowerShell to the Linux host orchestration path.

All `.ps1` / `.psm1` scripts run on **Windows PowerShell 5.1** (not Core/7). Every script must start with:

```powershell
#Requires -Version 5.1
Set-StrictMode -Version Latest
$ErrorActionPreference = 'Stop'
```

Forbidden PS 7+ syntax (use the PS 5.1 alternatives):

| Forbidden | Use instead |
|-----------|-------------|
| `$x ??= 'val'` | `if ($null -eq $x) { $x = 'val' }` |
| `$a ?? $b` | `if ($null -ne $a) { $a } else { $b }` |
| `cmd1 && cmd2` | `cmd1; if ($LASTEXITCODE -eq 0) { cmd2 }` |
| Ternary `$a ? $b : $c` | `if ($a) { $b } else { $c }` |
| `ForEach-Object -Parallel` | `foreach` or `Start-Job` |

Always check `$LASTEXITCODE` after `vmrun`, `ssh`, `scp` and `throw` if non-zero.

---

## Python Conventions

- `click.echo` everywhere — no bare `print()`.
- No `subprocess.run(..., shell=True)`. Use `paramiko` for SSH, `pypsrp` for WinRM, `keyring` for credentials.
- No global variables — state lives in `Config` and objects passed explicitly.
- All public functions and exported helpers must have type hints. Avoid `Any` except in pytest fixtures/mocks.
- `ruff` + `mypy --strict` are enforced in CI. The coverage gate is **90%**.

---

## Known Gotchas (from `AGENTS.md`)

- **`vmrun getGuestIPAddress`** can return exit code 0 with `"Error: The VMware Tools are not running..."` on stdout. Filter lines starting with `"Error:"`.
- **Linux machine-id**: Ubuntu cloud images ship with a fixed `/etc/machine-id`. VMware DHCP assigns IPs by machine-id — clones get the same IP and collide. The `BaseClean-Linux` snapshot must be taken after truncating machine-id.
- **`& nativecmd 2>$null` under `$ErrorActionPreference='Stop'`**: native stderr is converted to an ErrorRecord before `2>$null` discards it, causing a terminating error. Wrap with `$ErrorActionPreference = 'SilentlyContinue'` or `2>&1 | Out-Null`.
- **Snapshot with VM running**: never take a template snapshot while the VM is powered on.
- **WinRM vs SSH separation**: do not modify the Windows WinRM branch and the Linux SSH branch in the same commit/PR. Keep `if ($GuestOS -eq 'Linux')` guards clean.

---

## Key Files Reference

| File | Purpose |
|------|---------|
| `AGENTS.md` | Environment reference + error registry — read before writing any script |
| `docs/ARCHITECTURE.md` | Full system diagram and component descriptions |
| `docs/CI-FLOW.md` | Step-by-step pipeline description |
| `docs/RUNBOOK.md` | Common incident triage |
| `TODO.md` | Roadmap, audit trail, backlog |
| `plans/` | Phase closeout docs (A1–A5, B5) and implementation plans |
| `config.example.toml` | Annotated config template |
| `tests/python/test_agents_errors.py` | Regression tests for `AGENTS.md` errors #9–#12 — do not remove |