feat: update action.yml to use pwsh shell and enhance vm.py with start option; add credential management scripts
This commit is contained in:
@@ -0,0 +1,171 @@
|
||||
# CLAUDE.md
|
||||
|
||||
This file provides guidance to Claude Code (claude.ai/code) when working with this repository.
|
||||
|
||||
---
|
||||
|
||||
## Project Overview
|
||||
|
||||
Self-hosted CI/CD system running on a Windows 11 host with VMware Workstation. Each build job runs inside an ephemeral VM (linked clone from a frozen template snapshot), communicates via WinRM (Windows guests) or SSH (Linux guests), and is destroyed after the job.
|
||||
|
||||
**Current state**: The Python orchestrator (`src/ci_orchestrator/`) is the production entry point. The PowerShell scripts in `scripts/` are 3-line shims that forward to the Python CLI. All new logic goes in Python.
|
||||
|
||||
---
|
||||
|
||||
## Development Commands
|
||||
|
||||
### Python (primary)
|
||||
|
||||
The production venv is at `/opt/ci/venv` (Linux host). To invoke the CLI as the service user:
|
||||
|
||||
```bash
|
||||
sudo -u ci-runner /opt/ci/venv/bin/python -m ci_orchestrator --help
|
||||
```
|
||||
|
||||
```bash
|
||||
# Dev venv setup (one-time)
|
||||
python -m venv .venv
|
||||
.\.venv\Scripts\python.exe -m pip install -e ".[dev]"
|
||||
|
||||
# Run all tests with coverage (gate: ≥90%)
|
||||
python -m pytest tests/python -q --cov=ci_orchestrator --cov-fail-under=90
|
||||
|
||||
# Run a single test file
|
||||
python -m pytest tests/python/test_commands_job.py -q
|
||||
|
||||
# Lint
|
||||
python -m ruff check src tests/python
|
||||
|
||||
# Type-check (strict)
|
||||
python -m mypy --strict src
|
||||
|
||||
# CLI
|
||||
python -m ci_orchestrator --help
|
||||
```
|
||||
|
||||
**Important**: The production venv at `F:\CI\python\venv` must be installed **non-editable** (`pip install .`, not `-e`), because act_runner runs as LocalSystem and cannot resolve editable `.pth` files from a transient RunnerWork path. Re-run `pip install .` into the production venv after every code change. The dev `.venv` may use `-e`.
|
||||
|
||||
### PowerShell linting
|
||||
|
||||
```powershell
|
||||
Invoke-ScriptAnalyzer -Path scripts\ -Recurse -Settings PSScriptAnalyzerSettings.psd1 -Severity Error,Warning
|
||||
# Auto-fix formatting:
|
||||
Invoke-ScriptAnalyzer -Path scripts\ -Recurse -Fix
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
### Pipeline flow
|
||||
|
||||
```
|
||||
git push → Gitea Actions → act_runner (Windows service) → python -m ci_orchestrator job
|
||||
→ vm new (linked clone)
|
||||
→ wait-ready (WinRM poll or SSH poll)
|
||||
→ build run (WinRM Invoke-Command or SSH+SFTP)
|
||||
→ artifacts collect (WinRM Copy-Item or SCP)
|
||||
→ vm remove (vmrun stop + deleteVM)
|
||||
```
|
||||
|
||||
The host **never executes build tools directly** — it only orchestrates VM lifecycle.
|
||||
|
||||
### Python package layout (`src/ci_orchestrator/`)
|
||||
|
||||
| Module | Role |
|
||||
|--------|------|
|
||||
| `__main__.py` | click entry point; registers all sub-command groups |
|
||||
| `config.py` | `Config` dataclass + TOML/env loader; resolution: OS defaults → `config.toml` → env vars |
|
||||
| `credentials.py` | `CredentialStore` Protocol + `KeyringCredentialStore` |
|
||||
| `backends/protocol.py` | `VmBackend` Protocol + `VmHandle` + `VmState` (hypervisor-agnostic) |
|
||||
| `backends/workstation.py` | `WorkstationVmrunBackend` — wraps `vmrun.exe` subprocesses |
|
||||
| `backends/__init__.py` | `load_backend(config)` factory — the only import application code should use |
|
||||
| `transport/winrm.py` | WinRM transport via `pypsrp` |
|
||||
| `transport/ssh.py` | SSH/SFTP transport via `paramiko` |
|
||||
| `transport/errors.py` | `TransportConnectError`, `TransportCommandError` |
|
||||
| `commands/job.py` | Full end-to-end orchestrator (auto-detects guest OS from VMX `guestOS` field) |
|
||||
| `commands/vm.py` | `vm new / remove / cleanup` |
|
||||
| `commands/build.py` | `build run` |
|
||||
| `commands/artifacts.py` | `artifacts collect` |
|
||||
| `commands/wait.py` | `wait-ready` |
|
||||
| `commands/monitor.py` | `monitor disk / runner` |
|
||||
| `commands/report.py` | `report job` |
|
||||
|
||||
**Key design rule**: Application code (especially `commands/job.py`) must import `backends.load_backend` only — never `WorkstationVmrunBackend` directly. This keeps the Phase C ESXi extension path open.
|
||||
|
||||
### Transport selection
|
||||
|
||||
`commands/job.py` reads `guestOS = "..."` from the cloned VMX. `ubuntu-*` → Linux/SSH (`transport/ssh.py`); anything else → Windows/WinRM (`transport/winrm.py`).
|
||||
|
||||
### VM templates and snapshots
|
||||
|
||||
| Template | VMX path | Snapshot | Transport |
|
||||
|----------|----------|----------|-----------|
|
||||
| WinBuild2025 | `F:\CI\Templates\WinBuild2025\WinBuild2025.vmx` | `BaseClean` | WinRM HTTPS/5986 |
|
||||
| WinBuild2022 | `F:\CI\Templates\WinBuild2022\WinBuild2022.vmx` | `BaseClean` | WinRM HTTPS/5986 |
|
||||
| LinuxBuild2404 | `F:\CI\Templates\LinuxBuild2404\LinuxBuild2404.vmx` | `BaseClean-Linux` | SSH/22 (`ci_build`, key `F:\CI\keys\ci_linux`) |
|
||||
|
||||
Snapshots **must be taken from powered-off state** — presence of `.vmem` in the template dir means the snapshot captured running memory and `vmrun clone ... linked` will fail.
|
||||
|
||||
### Configuration
|
||||
|
||||
The config file is auto-discovered at `$CI_ROOT/config.toml` (Linux default: `/var/lib/ci/config.toml`; Windows: `F:\CI\config.toml`). Set `$CI_CONFIG` to override. The file at `/var/lib/ci/config.toml` is the live config on the Linux host — keep its `[paths]` section pointing to `/var/lib/ci/...` (not the old Windows `F:/CI/...` paths). Environment variables `CI_ROOT`, `CI_TEMPLATES`, `CI_BUILD_VMS`, `CI_ARTIFACTS`, `CI_KEYS` override the file. `CI_PYTHON_LAUNCHER` in `runner/config.yaml` is the single source of truth for the system Python path used to bootstrap the production venv.
|
||||
|
||||
---
|
||||
|
||||
## PowerShell 5.1 Constraints
|
||||
|
||||
All `.ps1` / `.psm1` scripts run on **Windows PowerShell 5.1** (not Core/7). Every script must start with:
|
||||
|
||||
```powershell
|
||||
#Requires -Version 5.1
|
||||
Set-StrictMode -Version Latest
|
||||
$ErrorActionPreference = 'Stop'
|
||||
```
|
||||
|
||||
Forbidden PS 7+ syntax (use the PS 5.1 alternatives):
|
||||
|
||||
| Forbidden | Use instead |
|
||||
|-----------|-------------|
|
||||
| `$x ??= 'val'` | `if ($null -eq $x) { $x = 'val' }` |
|
||||
| `$a ?? $b` | `if ($null -ne $a) { $a } else { $b }` |
|
||||
| `cmd1 && cmd2` | `cmd1; if ($LASTEXITCODE -eq 0) { cmd2 }` |
|
||||
| Ternary `$a ? $b : $c` | `if ($a) { $b } else { $c }` |
|
||||
| `ForEach-Object -Parallel` | `foreach` or `Start-Job` |
|
||||
|
||||
Always check `$LASTEXITCODE` after `vmrun`, `ssh`, `scp` and `throw` if non-zero.
|
||||
|
||||
---
|
||||
|
||||
## Python Conventions
|
||||
|
||||
- `click.echo` everywhere — no bare `print()`.
|
||||
- No `subprocess.run(..., shell=True)`. Use `paramiko` for SSH, `pypsrp` for WinRM, `keyring` for credentials.
|
||||
- No global variables — state lives in `Config` and objects passed explicitly.
|
||||
- All public functions and exported helpers must have type hints. Avoid `Any` except in pytest fixtures/mocks.
|
||||
- `ruff` + `mypy --strict` are enforced in CI. The coverage gate is **90%**.
|
||||
|
||||
---
|
||||
|
||||
## Known Gotchas (from `AGENTS.md`)
|
||||
|
||||
- **`vmrun getGuestIPAddress`** can return exit code 0 with `"Error: The VMware Tools are not running..."` on stdout. Filter lines starting with `"Error:"`.
|
||||
- **Linux machine-id**: Ubuntu cloud images ship with a fixed `/etc/machine-id`. VMware DHCP assigns IPs by machine-id — clones get the same IP and collide. The `BaseClean-Linux` snapshot must be taken after truncating machine-id.
|
||||
- **`& nativecmd 2>$null` under `$ErrorActionPreference='Stop'`**: native stderr is converted to an ErrorRecord before `2>$null` discards it, causing a terminating error. Wrap with `$ErrorActionPreference = 'SilentlyContinue'` or `2>&1 | Out-Null`.
|
||||
- **Snapshot with VM running**: never take a template snapshot while the VM is powered on.
|
||||
- **WinRM vs SSH separation**: do not modify the Windows WinRM branch and the Linux SSH branch in the same commit/PR. Keep `if ($GuestOS -eq 'Linux')` guards clean.
|
||||
|
||||
---
|
||||
|
||||
## Key Files Reference
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `AGENTS.md` | Environment reference + error registry — read before writing any script |
|
||||
| `docs/ARCHITECTURE.md` | Full system diagram and component descriptions |
|
||||
| `docs/CI-FLOW.md` | Step-by-step pipeline description |
|
||||
| `docs/RUNBOOK.md` | Common incident triage |
|
||||
| `TODO.md` | Roadmap, audit trail, backlog |
|
||||
| `plans/` | Phase closeout docs (A1–A5, B5) and implementation plans |
|
||||
| `config.example.toml` | Annotated config template |
|
||||
| `tests/python/test_agents_errors.py` | Regression tests for `AGENTS.md` errors #9–#12 — do not remove |
|
||||
Reference in New Issue
Block a user