Commit Graph

188 Commits

Author SHA1 Message Date
Simone 0c7c9272ee ci(lint): detect broken venv (stale base interpreter) and rebuild
Lint / python (push) Failing after 8s
Lint / pssa (push) Successful in 24s
The venv at F:\CI\python\venv may exist but be broken if its pyvenv.cfg points to a deleted base interpreter (e.g. a stale actions/setup-python tool_cache). Detect this by running 'python --version' and rebuild the venv with py.exe -3.11 if it fails.
2026-05-14 22:48:46 +02:00
Simone 406c0e31a9 ci(lint): bootstrap venv via py.exe launcher; bump coverage to 80%
Lint / python (push) Failing after 18s
Lint / pssa (push) Successful in 24s
act_runner runs as SYSTEM and does not have plain 'python' in PATH; use py.exe launcher (-3.11) to bootstrap the venv when missing. The production venv at F:\CI\python\venv normally already exists, so the bootstrap is just a fallback. Also align coverage threshold to 80% (raised in A5, see AGENTS.md).
2026-05-14 22:46:13 +02:00
Simone 632d668327 ci(lint): tweak comment header to trigger workflow re-run
Lint / python (push) Failing after 9s
Lint / pssa (push) Successful in 25s
2026-05-14 22:43:55 +02:00
Simone ebc53b5fb9 ci: trigger lint workflow re-run 2026-05-14 22:39:30 +02:00
Simone 137287176e docs: rename branch references to feature/python-rewrite-and-linux-migration
Update active docs (PhaseA/B checklists, README) after rename of feature/python-rewrite-phase-a -> feature/python-rewrite-and-linux-migration. Historical A1-A5 closeouts left untouched as they document state at the time of writing.
2026-05-14 18:34:04 +02:00
Simone 28523375b9 docs(plans): add A->B bridge checklist (windows prereqs + baseline + logistics)
Bridges the gap between PhaseA-user-checklist.md (Windows validation) and PhaseB-user-checklist.md (Linux setup):

  Section 1: capture pre-migration benchmark to docs/RUNBOOK.md (referenced by B7)

  Section 2: 7 Windows prerequisites (SSH keys, BuildVMGuest password, Gitea PAT, OpenSSH server, template power-off check per AGENTS.md #9, F:\CI\ backup, scheduled task cadence audit)

  Section 3: logistics (hardware, DNS, maintenance window, user comms)
2026-05-14 18:34:04 +02:00
Simone bd61258a42 feat(b6): add Phase B user checklist for CI migration to Linux Mint 2026-05-14 18:34:04 +02:00
Simone 0bd3b52fb5 feat(b5): add systemd unit files for CI periodic tasks (Linux host)
Implements the repo-side artifacts of Phase B5 (plans/implementation-plan-A-B.md). Creates 5 .service + .timer pairs under deploy/systemd/ that replicate the periodic tasks registered on Windows by scripts/Register-CIScheduledTasks.ps1:

  - ci-cleanup-orphans       -> python -m ci_orchestrator vm cleanup --max-age-hours 6 (every 6h + boot)

  - ci-retention-policy      -> pwsh Invoke-RetentionPolicy.ps1 (daily 03:00)

  - ci-watch-disk-space      -> python -m ci_orchestrator monitor disk (every 15min)

  - ci-watch-runner-health   -> python -m ci_orchestrator monitor runner (every 15min)

  - ci-backup-template       -> pwsh Backup-CITemplate.ps1 (weekly Sun 02:00, NEW)

Includes deploy/systemd/README.md with mapping table, install/test/rollback instructions, and PowerShell Core install note for the two PS-only scripts. Plan checklist updated; B5-closeout.md added. Hardware install steps (systemctl enable + smoke) remain user responsibility on the Linux host.
2026-05-14 18:34:04 +02:00
Simone 4e0b796f81 docs: add Fase A user checklist for hardware validation 2026-05-14 18:34:04 +02:00
Simone a658f4ed3b docs(a5): mark A5 as code-complete; add A5-closeout.md
Master checklist + step A5 attivita + 'definizione di fatto' updated; A5-closeout.md added (116 pytest, coverage 80.10% total, gate raised to 80%). Hardware validation (burn-in 4x10 + workflow matrix end-to-end) deferred to user. Shim PS removal deferred to B5 (systemd timers): all 10 shims still referenced by Register-CIScheduledTasks.ps1 / docs / Test-*.ps1.
2026-05-14 18:34:04 +02:00
Simone 4d9c6d07ba docs(a5): document Python orchestrator in AGENTS.md, ARCHITECTURE.md, README.md
AGENTS.md: new 'Python development' section (venv paths, install/lint/typecheck/test commands, conventions, full PS->Python sub-command mapping, list of PS scripts that stay PowerShell, link to phase closeouts). README.md: expand 'Python orchestrator' with examples for all 10 sub-commands (wait-ready, vm new/remove/cleanup, build run, artifacts collect, monitor disk/runner, report job, job), local validation gates with 80% coverage gate, link to test_agents_errors.py. docs/ARCHITECTURE.md: new 'Python orchestrator (Phase A)' section with package layout tree, VmBackend Protocol contract, load_backend(config) factory, Phase C ESXi extension recipe, transport selection note, and CI validation gates.
2026-05-14 18:34:04 +02:00
Simone 5459672b42 test(a5): add pytest regression tests for AGENTS.md errors #9-#12
12 dedicated tests covering: (#9) clone_linked surfaces 'should not be powered on' as typed BackendError; (#10) is_running uses 'vmrun list' only and never falls back to getGuestIPAddress (incl. on vmrun list failure); (#11) two consecutive 'vm new' invocations with identical inputs MUST produce distinct clone names/destinations (timestamp suffix), guarding the orchestrator-side safety net against the machine-id duplicate-IP failure mode; (#12) SshTransport defaults to paramiko AutoAddPolicy with no known_hosts, never shells out to ssh-keygen/ssh.exe/scp.exe, wraps paramiko import/exec/SFTP errors as TransportConnectError. Coverage: 80.10% (gate 80%).
2026-05-14 18:34:04 +02:00
Simone 1bcd40e483 docs(a4): mark A4 as code-complete; add A4-closeout.md
Master checklist + step A4 attivita + 'definizione di fatto' updated; A4-closeout.md added (104 pytest, coverage 79.25% total / 83% commands/job.py). Hardware validation (matrix workflow + benchmark) deferred to user.
2026-05-14 18:34:04 +02:00
Simone e5cd749ea1 ci(a4): switch local-ci-build action.yml to Python CLI; add PYTHONIOENCODING=utf-8
The composite action now invokes & $venvPython -m ci_orchestrator job ... directly instead of splatting a hashtable into Invoke-CIJob.ps1, removing the only mismatch between the new shim's positional args contract and the action's previous PascalCase splatting. runner/config.yaml exports PYTHONIOENCODING=utf-8 so non-ASCII build output survives the act_runner -> Gitea round-trip.
2026-05-14 18:34:03 +02:00
Simone f5091d0903 feat(a4): port Invoke-CIJob.ps1 to ci_orchestrator job
Phase A4 of plans/implementation-plan-A-B.md. Implements the full job orchestrator (clone -> start -> wait -> probe -> build -> collect -> guaranteed cleanup) as a new commands/job.py click command, registered under python -m ci_orchestrator job. Backend selection goes through backends.load_backend(config) so Phase C can swap in remote drivers without touching the command. The legacy scripts/Invoke-CIJob.ps1 is replaced by a thin PS 5.1 shim that delegates to the Python CLI; tests/python/test_commands_job.py adds 13 cases covering Linux/Windows happy paths, override application, skip-artifact, and cleanup on every failure mode.
2026-05-14 18:34:03 +02:00
Simone 816a15503e feat(a3): port build pipeline (vm new, build run, artifacts collect)
Implements Phase A3 of plans/implementation-plan-A-B.md:

- commands/vm.py: vm new (replaces New-BuildVM.ps1)

- commands/build.py: build run (replaces Invoke-RemoteBuild.ps1) with WinRM/SSH dispatch and stdout streaming for act_runner

- commands/artifacts.py: artifacts collect (replaces Get-BuildArtifacts.ps1) using transport.fetch()

- 3 PS scripts reduced to shims preserving \0

- Pester tests/{New-BuildVM,Wait-VMReady,Remove-BuildVM}.Tests.ps1 removed; equivalent cases covered in pytest

- Phase C hook: build/artifacts accept opaque VmHandle/vmx; no Windows path assumptions

Tests: 91 pytest, ruff clean, mypy --strict clean, coverage 78.27% (>=70 gate).

Master checklist + step A3 attivita + 'definizione di fatto' updated; A3-closeout.md added.
2026-05-14 18:34:03 +02:00
Simone 096ba7fe16 docs(a1): sync master checklist + A1 'definizione di fatto' to actual state
A1 items were completed but only A1-closeout.md was updated. Mark all A1 checkboxes done in implementation-plan-A-B.md (master checklist + step-A1 attivita + definizione di fatto), leaving only the real-VM PoC pending (hardware validation deferred).
2026-05-14 18:34:03 +02:00
Simone ad9e6e9673 docs(a2): mark A2 as code-complete; add A2-closeout.md 2026-05-14 18:34:03 +02:00
Simone 80f6661ad5 feat(a2): port leaf PS scripts to ci_orchestrator CLI
Implements Phase A2 of plans/implementation-plan-A-B.md:

- commands/wait.py    -> wait-ready (replaces Wait-VMReady.ps1)

- commands/vm.py      -> vm remove + vm cleanup (replaces Remove-BuildVM.ps1, Cleanup-OrphanedBuildVMs.ps1); cleanup accepts injected VmBackend (Phase C ESXi hook preserved)

- commands/monitor.py -> monitor disk + monitor runner (replaces Watch-DiskSpace.ps1, Watch-RunnerHealth.ps1)

- commands/report.py  -> report job (replaces Get-CIJobSummary.ps1)

Each PS script reduced to a 3-line shim that delegates to the Python CLI and preserves \0.

Tests: 69 pytest cases across new test_commands_*.py modules; ruff clean, mypy --strict clean, coverage 74.5% (>=70 gate).
2026-05-14 18:34:03 +02:00
Simone 10da8f4e81 docs(A1): mark lint.yml python job as PASS on act_runner 2026-05-14 18:34:03 +02:00
Simone 484a18efd7 ci(lint): drop actions/setup-python; use host Python in machine PATH
The runner's act_runner runs as SYSTEM, which only sees the Machine PATH.
Host has Python 3.14 installed at C:\Program Files\Python314\; once that
path is added to the machine PATH (one-time admin op) the simple
Get-Command python lookup works without network downloads.
2026-05-14 18:34:03 +02:00
Simone 3f05e89362 ci(lint): add CmdletBinding and ShouldProcess support to ISO creation functions for safer execution 2026-05-14 18:34:03 +02:00
Simone 2ac26dbd00 ci(lint): enhance PSScriptAnalyzer settings and suppress false positives; improve error handling in scripts 2026-05-14 18:34:03 +02:00
Simone 032d6f200e ci(lint): add rule to suppress false positives for param() in remote script blocks 2026-05-14 18:34:03 +02:00
Simone 7e6b65db18 ci(lint): update PSScriptAnalyzer settings and suppress warnings for plain-text passwords in templates 2026-05-14 18:34:03 +02:00
Simone 366ff0ce46 ci(lint): update PSScriptAnalyzer settings to include additional formatting rules 2026-05-14 18:34:02 +02:00
Simone e09c0cf5a5 ci(lint): update PSScriptAnalyzer invocation to use custom settings file 2026-05-14 18:34:02 +02:00
Simone 81c85beb81 ci: remove Python setup step from lint workflow 2026-05-14 18:34:02 +02:00
Simone 8a2727ccf7 ci(lint): use actions/setup-python to provision Python on runner
The act_runner service account doesn't have python in PATH. Delegating
to actions/setup-python@v5 which downloads/caches Python 3.11 reliably.

Also bump timeout-minutes 15 -> 20 to absorb first-run pip install.
2026-05-14 18:34:02 +02:00
Simone 305ee6fa8b ci: move lint.yml to .gitea/workflows so Gitea Actions discovers it
Gitea Actions only scans .gitea/workflows/ and .github/workflows/. The
file under gitea/workflows/ (no leading dot) was never picked up. Other
files in gitea/workflows/ remain as templates copied into consumer repos.
2026-05-14 18:34:02 +02:00
Simone 44a99af14f feat: add closeout checklist for Phase A1 implementation 2026-05-14 18:34:02 +02:00
Simone bd31a3f2f3 Phase A1: bootstrap Python ci_orchestrator package
- pyproject.toml (hatchling) with ruff/mypy/pytest dev deps
- src/ci_orchestrator/: config, credentials, backends (Protocol + WorkstationVmrunBackend), transport (WinRM via pypsrp, SSH via paramiko)
- CLI entry point with PoC 'wait-ready' subcommand (Windows + Linux guests)
- tests/python/: 35 unit tests, 83% coverage, all backends/transport mocked
- gitea/workflows/lint.yml: new 'python' job (ruff + mypy --strict + pytest --cov-fail-under=70)
- config.example.toml + README setup section + .gitignore Python entries

Neutral VmBackend Protocol prepared for Phase C ESXi extension.
See plans/implementation-plan-A-B.md step A1.
2026-05-14 18:34:02 +02:00
Simone 4d957d34b2 fix: correct table formatting in ideas-overview.md for better readability 2026-05-13 11:12:28 +02:00
Simone 4ac6dd85f9 refactor: restructure implementation plan for clarity and readability 2026-05-13 11:08:01 +02:00
Simone 770ac1edf8 Add unified implementation plan for Phase A and Phase B in Markdown format
- Create a new document `plans/implementation-plan-A-B.md` that consolidates the implementation plans for the Python rewrite (Phase A) and the migration to Linux Mint (Phase B).
- The document includes detailed sections such as executive summary, checklist, prerequisites, step-by-step activities, architectural hooks for future phases, risk matrix, and references.
- Ensure the plan is self-contained, adhering to specified structure and style guidelines, and does not introduce new technologies or modify existing files.
2026-05-13 10:49:10 +02:00
Simone 081f7e1d16 archive: move completed plans and sprint test plan to dated folders
- plans/archived/2026-05-13/: final-master-plan (done), 4 AI analysis/review
  docs (gpt55-analysis, gpt55-review-of-opus, opus47-analysis,
  opus47-review-of-gpt55)
- docs/archived/2026-05-13/: TEST-PLAN-v1.3-to-HEAD (sprint complete)
2026-05-13 10:47:06 +02:00
Simone 1dbeec0f8e feat: update installer SHA256 hashes for Python, DotNetInstallScript, SevenZip, and NodeJS 2026-05-13 10:41:42 +02:00
Simone 6cd6bff882 nice-to-have: implement all 6 tier items
- Get-CIJobSummary.ps1: already present with full implementation (marked done)
- Get-BuildArtifacts.ps1: add -JobId/-Commit params; Write-ArtifactManifest writes
  manifest.json (name, size, sha256, commit, jobId) after both Linux and Windows
  artifact collection branches
- Invoke-CIJob.ps1: add -GuestCPU/-GuestMemoryMB params; insert post-clone VMX
  override block (numvcpus/memsize rewrite) before vmrun start; pass -JobId/-Commit
  to Get-BuildArtifacts calls
- Watch-RunnerHealth.ps1: add -GiteaUrl/-GiteaCredentialTarget params; optional
  GET /api/v1/admin/runners check in Running branch warns if 0 online runners
- Install-CIToolchain-WinBuild2025.ps1: fill SHA256 hashes for Python 3.13.3,
  7-Zip 26.01, Node.js 22.14.0, dotnet-install.ps1
- Measure-CIBenchmark.ps1: add deltaKB field (linked-clone disk footprint in KB)
  measured post-clone; added to result object and summary Format-Table
2026-05-13 10:38:49 +02:00
Simone 8ca3e530c5 feat(low): sprint 4 Low items tutti 5 completati
codice:
- template/Validate-SetupState.ps1: docblock HTTP/Basic -> HTTPS/Basic, port 5986
- scripts/Validate-HostState.ps1 (nuovo): check ACL SSH key ci_linux, Credential Manager
  target BuildVMGuest, struttura dir F:\CI\, vmrun.exe raggiungibile

docs:
- docs/TEST-PLAN-v1.3-to-HEAD.md sezione 3.6: campi JSONL corretti (cloneSec/startSec/
  ipSec/winrmSec/destroySec/totalBootSec) per riflettere output reale di Measure-CIBenchmark.ps1
- docs/CI-FLOW.md tabella failure: 'partial artifacts' -> Get-GuestDiagnostics a diagnostics/
- docs/CHANGELOG.md (nuovo): log reverse-cronologico di tutti gli sprint completati
- plans/final-master-plan.md: tutti i Low marcati done
2026-05-13 10:26:08 +02:00
Simone e9d0f38f9f feat(medium): sprint 3 Medium items 6 implementati, 6 confermati gia' presenti
codice:
- Watch-DiskSpace.ps1: Event Log source CI-DiskAlert -> CI-DiskSpaceAlert
  (allineato con task scheduler CI-DiskSpaceAlert)
- Watch-RunnerHealth.ps1: maintenance flag  salta restart se
  F:\CI\State\runner-maintenance.flag esiste
- Invoke-CIJob.ps1:
  - param WebhookUrl (default '') per alert webhook
  - sanity check TemplatePath: Write-Warning se non sotto F:\CI\Templates\
  - 90-min duration warning: Start-Job che posta [WARNING] dopo 5400s
  - finally: Remove-Job per cancellare il warn se build finisce prima

docs:
- BEST-PRACTICES.md sezione 11: VMware HGFS write semantics e cache-poisoning risk
- final-master-plan.md: marcati done  post-failure diagnostics, UseSharedCache,
  VMDK SHA256, SSH host-key, template-refresh runbook, 90-min warn, HGFS,
  Event Log names, TemplatePath sanity, maintenance flag, Setup-Host pass docs,
  emoji-free webhooks
2026-05-13 10:19:55 +02:00
Simone 91f2305bb0 docs(plan): sprint 3 verification pass tutti gli High restanti gia' implementati
- H9 (ci-report-ip.service check): gia' in Validate-DeployState.ps1 branch Linux
- H10 (smoke test workflow): gia' in gitea/workflows/self-test.yml + scripts/Test-Smoke.ps1
- H12 (Get-GuestIPAddress): gia' estratto in _Common.psm1 e usato da Measure-CIBenchmark
- Medium: disk gate (L339), JobId ValidatePattern (L77), ExtraGuestEnv key validation
  (L184), phase-duration summary (L541), TEST-PLAN PS7 syntax gia' corretta
- Low: Install-Runner.ps1 gia' assente dal repo
- Aggiornato header Last updated + Status
2026-05-13 10:10:55 +02:00
Simone 344cdf7993 refactor(plans): update Linux host adaptation details for VMware Workstation Pro and KVM options 2026-05-13 10:04:09 +02:00
Simone c7cc2c7c38 feat(plans): add proposals for Python rewrite, Linux host adaptation, and ESXi support 2026-05-13 09:52:44 +02:00
Simone 3592dcab78 Refactor documentation and analysis in final master plan, opus47 analysis, and opus47 review of GPT-5.5
- Updated the final master plan to improve clarity and consistency in the status of various capabilities, including fixes and enhancements made as of 2026-05-12.
- Revised the opus47 analysis to provide a more structured overview of the system's production readiness, including detailed coverage of OWASP Top 10 risks and operational gaps.
- Enhanced the opus47 review of GPT-5.5 by clarifying severity ratings and rationales for various items, while adding new high-severity items that were previously overlooked.
2026-05-13 00:08:19 +02:00
Simone 5d1b47e943 feat(workflow): rinomina build-nsis.yml -> build-nsInnoUnp.yml, aggiunge job release
- Rinominato gitea/workflows/build-nsis.yml in build-nsInnoUnp.yml
- Aggiunto job release: scarica artifact Windows+Linux, crea release Gitea
  via API REST, comprime e uploada gli asset come zip (richiede secret GITEA_TOKEN)
- Aggiornati tutti i riferimenti in README.md, TODO.md, plans/*.md
2026-05-12 23:37:00 +02:00
Simone 075d1c5dcf docs(readme): sync con stato attuale (Linux support, burn-in, transport, action.yml)
- Stato: aggiornato (Windows+Linux e2e PASS, burn-in 4 job concorrenti PASS)
- Panoramica: diagram aggiornato con Linux/SSH path e dual runner label
- Template VM: aggiunta sezione LinuxBuild2404 (BaseClean-Linux, SSH, ci_build)
- Struttura: aggiunti _Transport.psm1, Test-CapacityBurnIn, Start-BurnInTest-Linux,
  Get-CIJobSummary, template Linux (Deploy/Prepare/Install-CIToolchain), actions/
- Setup rapido: separato in 1a Windows / 1b Linux con Prepare-LinuxBuild2404 -TakeSnapshot
- act_runner: tabella env vars completa (LINUX_TEMPLATE_PATH, SSH_KEY_PATH, ...)
  + labels windows-build:host e linux-build:host
- Uso build manuale: aggiunto esempio Linux guest + note -UseGitClone/-SkipArtifact
- Workflow: composite action documentata con tabella input; matrix Windows+Linux
- Docs: LINUX-TEMPLATE-SETUP.md non piu' 'non implementato'
2026-05-12 23:23:50 +02:00
Simone fe4439b95f fix(ssh): switch CI SSH to permissive mode (no known_hosts)
Root cause: ssh-keygen -R prints 'Host X not found in F:\CI\State\known_hosts'
to stderr when the IP has no prior entry. In PS5.1 with ErrorActionPreference=Stop,
native stderr is converted to ErrorRecords before 2>$null can discard them,
making the command a terminating error caught by the job's catch block.

This caused all 4 concurrent burn-in jobs to fail after ~15s with:
  Error: Host 192.168.79.XXX not found in F:\CI\State\known_hosts

Additionally, 4 concurrent SSH connections writing to the same known_hosts file
with no advisory locking risk file corruption under concurrent load.

Fix: set SshKnownHostsFile default to '' in Invoke-CIJob, Invoke-RemoteBuild,
and Get-BuildArtifacts. When KnownHostsFile='' _Transport.psm1 uses
StrictHostKeyChecking=no + UserKnownHostsFile=NUL (permissive mode), consistent
with Wait-VMReady and Prepare-LinuxBuild2404.ps1. Ephemeral CI VMs on a
local NAT network have no meaningful MITM risk.

Remove the ssh-keygen purge block from Invoke-CIJob.ps1 (no longer needed).

AGENTS.md: document PS5.1 native stderr / 2>$null interaction (error #12).
2026-05-12 22:51:20 +02:00
Simone 985e546f2d fix(template): clear machine-id in Prepare-LinuxBuild2404 (Step 7b)
Each Ubuntu clone derived from the same snapshot shared an identical
machine-id, causing systemd-networkd to use the same DHCP client-id
and receive the same NAT IP  triggering IP collision failures.

Step 7b truncates /etc/machine-id and removes /var/lib/dbus/machine-id
before the snapshot is taken. On first boot each clone generates a
fresh machine-id from /dev/urandom -> unique DHCP client-id -> unique IP.
2026-05-12 22:33:55 +02:00
Simone 435ccba06f docs(AGENTS): add machine-id Linux DHCP collision as known error #11 2026-05-12 22:30:18 +02:00
Simone 74331ac301 fix(diagnostics): use StrictHostKeyChecking=no for Linux guest-diag SSH
Diagnostics is best-effort. Using accept-new without UserKnownHostsFile
caused the system known_hosts (~/.ssh/known_hosts) to be consulted,
which holds a stale key from a previous clone of the same IP.
Switch to no+NUL (same as Wait-VMReady) to avoid false host-key errors.
2026-05-12 22:29:57 +02:00