diff --git a/deploy/systemd/README.md b/deploy/systemd/README.md new file mode 100644 index 0000000..3c6cf46 --- /dev/null +++ b/deploy/systemd/README.md @@ -0,0 +1,105 @@ +# systemd units — Local CI/CD System (Linux host) + +Coppie `*.service` + `*.timer` che replicano i task periodici registrati su +Windows da `scripts/Register-CIScheduledTasks.ps1`. Pensate per essere +installate su un host Linux Mint / Ubuntu LTS, con il package +`ci_orchestrator` installato in `/opt/ci/venv/`. + +## Mapping Windows Task Scheduler → systemd timer + +| Windows Task Scheduler | systemd unit | Cadenza | Comando | +| ----------------------- | ---------------------------------- | ----------------------- | --------------------------------------------------------------- | +| `CI-CleanupOrphans` | `ci-cleanup-orphans.timer` | every 6h + at boot | `python -m ci_orchestrator vm cleanup --max-age-hours 6` | +| `CI-RetentionPolicy` | `ci-retention-policy.timer` | daily 03:00 + at boot | `pwsh Invoke-RetentionPolicy.ps1` (vedi nota PowerShell) | +| `CI-DiskSpaceAlert` | `ci-watch-disk-space.timer` | every 15 min | `python -m ci_orchestrator monitor disk` | +| `CI-RunnerHealth` | `ci-watch-runner-health.timer` | every 15 min | `python -m ci_orchestrator monitor runner` | +| *(nuovo)* | `ci-backup-template.timer` | weekly Sun 02:00 | `pwsh Backup-CITemplate.ps1` (vedi nota PowerShell) | + +## Nota PowerShell su Linux + +Due unit (`ci-retention-policy`, `ci-backup-template`) invocano script +PowerShell che, per scelta documentata in `AGENTS.md` ("Mappatura script +PowerShell → Python"), restano in PowerShell. Su Linux questo richiede +**PowerShell Core (`pwsh`)**: + +```bash +# Ubuntu / Mint LTS +sudo apt-get update +sudo apt-get install -y wget apt-transport-https software-properties-common +wget -q https://packages.microsoft.com/config/ubuntu/$(lsb_release -rs)/packages-microsoft-prod.deb +sudo dpkg -i packages-microsoft-prod.deb +sudo apt-get update +sudo apt-get install -y powershell +pwsh --version # verifica +``` + +Se preferisci non installare `pwsh`, porta `Invoke-RetentionPolicy.ps1` e +`Backup-CITemplate.ps1` come sub-comandi `python -m ci_orchestrator +retention run` / `template backup` (out of scope B5 per il piano corrente +— vedi `plans/implementation-plan-A-B.md`). + +## Prerequisiti + +1. Utente di sistema `ci-runner` con UID dedicato (creato in B1). +2. Venv Python attivo in `/opt/ci/venv/` con `ci_orchestrator` installato: + ```bash + sudo -u ci-runner /opt/ci/venv/bin/pip install -e /opt/ci/local-ci-cd-system + ``` +3. Variabili d'ambiente `CI_*` definite in `/etc/ci/environment` (caricato + tramite `EnvironmentFile=` in ogni unit `.service`): + ``` + CI_ROOT=/var/lib/ci + CI_TEMPLATES=/var/lib/ci/templates + CI_BUILD_VMS=/var/lib/ci/build-vms + CI_ARTIFACTS=/var/lib/ci/artifacts + CI_KEYS=/etc/ci/keys + PYTHONIOENCODING=utf-8 + ``` +4. `Invoke-RetentionPolicy.ps1` e `Backup-CITemplate.ps1` presenti in + `/opt/ci/local-ci-cd-system/scripts/` (clone del repo). + +## Installazione + +```bash +# 1. Copia le unit in /etc/systemd/system/ +sudo cp deploy/systemd/ci-*.service deploy/systemd/ci-*.timer /etc/systemd/system/ + +# 2. Reload systemd +sudo systemctl daemon-reload + +# 3. Abilita + avvia tutti i timer +for t in ci-cleanup-orphans ci-retention-policy ci-watch-disk-space \ + ci-watch-runner-health ci-backup-template; do + sudo systemctl enable --now "${t}.timer" +done + +# 4. Verifica +systemctl list-timers --all 'ci-*' +``` + +## Test e troubleshooting + +```bash +# Trigger manuale di un singolo task (esegue il .service una volta) +sudo systemctl start ci-cleanup-orphans.service + +# Log dell'ultima esecuzione +journalctl -u ci-cleanup-orphans.service -n 50 --no-pager + +# Live tail di tutti i task CI +journalctl -u 'ci-*' -f +``` + +## Rollback + +```bash +for t in ci-cleanup-orphans ci-retention-policy ci-watch-disk-space \ + ci-watch-runner-health ci-backup-template; do + sudo systemctl disable --now "${t}.timer" + sudo rm -f "/etc/systemd/system/${t}.service" "/etc/systemd/system/${t}.timer" +done +sudo systemctl daemon-reload +``` + +A questo punto il fallback è ri-attivare i task su Windows con +`scripts/Register-CIScheduledTasks.ps1`. diff --git a/deploy/systemd/ci-backup-template.service b/deploy/systemd/ci-backup-template.service new file mode 100644 index 0000000..0a339f0 --- /dev/null +++ b/deploy/systemd/ci-backup-template.service @@ -0,0 +1,26 @@ +[Unit] +Description=Local CI/CD — weekly backup of VMware templates +Documentation=file:///opt/ci/local-ci-cd-system/deploy/systemd/README.md +After=network-online.target + +[Service] +Type=oneshot +User=ci-runner +Group=ci-runner +EnvironmentFile=/etc/ci/environment +WorkingDirectory=/opt/ci/local-ci-cd-system +# NOTE: requires PowerShell Core (`pwsh`) on the Linux host. +# See deploy/systemd/README.md "Nota PowerShell su Linux". +ExecStart=/usr/bin/pwsh -NoProfile -NonInteractive -File scripts/Backup-CITemplate.ps1 +TimeoutStartSec=4h +IOSchedulingClass=best-effort +IOSchedulingPriority=7 +Nice=19 +ProtectSystem=strict +ProtectHome=true +PrivateTmp=true +NoNewPrivileges=true +ReadWritePaths=/var/lib/ci /var/backups/ci + +[Install] +WantedBy=multi-user.target diff --git a/deploy/systemd/ci-backup-template.timer b/deploy/systemd/ci-backup-template.timer new file mode 100644 index 0000000..cec55d4 --- /dev/null +++ b/deploy/systemd/ci-backup-template.timer @@ -0,0 +1,13 @@ +[Unit] +Description=Local CI/CD — weekly timer for ci-backup-template.service +Documentation=file:///opt/ci/local-ci-cd-system/deploy/systemd/README.md + +[Timer] +# Sunday 02:00 + random delay 1h to avoid clashing with other CI tasks. +OnCalendar=Sun *-*-* 02:00:00 +RandomizedDelaySec=1h +Persistent=true +Unit=ci-backup-template.service + +[Install] +WantedBy=timers.target diff --git a/deploy/systemd/ci-cleanup-orphans.service b/deploy/systemd/ci-cleanup-orphans.service new file mode 100644 index 0000000..ab9a2fb --- /dev/null +++ b/deploy/systemd/ci-cleanup-orphans.service @@ -0,0 +1,23 @@ +[Unit] +Description=Local CI/CD — destroy orphaned build VMs (max age 6h) +Documentation=file:///opt/ci/local-ci-cd-system/deploy/systemd/README.md +After=network-online.target +Wants=network-online.target + +[Service] +Type=oneshot +User=ci-runner +Group=ci-runner +EnvironmentFile=/etc/ci/environment +ExecStart=/opt/ci/venv/bin/python -m ci_orchestrator vm cleanup --max-age-hours 6 +TimeoutStartSec=1h +Nice=10 +# Hardening +ProtectSystem=strict +ProtectHome=true +PrivateTmp=true +NoNewPrivileges=true +ReadWritePaths=/var/lib/ci /var/log/ci + +[Install] +WantedBy=multi-user.target diff --git a/deploy/systemd/ci-cleanup-orphans.timer b/deploy/systemd/ci-cleanup-orphans.timer new file mode 100644 index 0000000..010733c --- /dev/null +++ b/deploy/systemd/ci-cleanup-orphans.timer @@ -0,0 +1,16 @@ +[Unit] +Description=Local CI/CD — periodic timer for ci-cleanup-orphans.service +Documentation=file:///opt/ci/local-ci-cd-system/deploy/systemd/README.md + +[Timer] +# Equivalent to Windows Task Scheduler: +# New-ScheduledTaskTrigger -Once -At '00:00' -RepetitionInterval 6h +# New-ScheduledTaskTrigger -AtStartup +OnBootSec=2min +OnUnitActiveSec=6h +Persistent=true +AccuracySec=1min +Unit=ci-cleanup-orphans.service + +[Install] +WantedBy=timers.target diff --git a/deploy/systemd/ci-retention-policy.service b/deploy/systemd/ci-retention-policy.service new file mode 100644 index 0000000..465ed9e --- /dev/null +++ b/deploy/systemd/ci-retention-policy.service @@ -0,0 +1,24 @@ +[Unit] +Description=Local CI/CD — purge old artifacts/logs/IP leases (retention policy) +Documentation=file:///opt/ci/local-ci-cd-system/deploy/systemd/README.md +After=network-online.target + +[Service] +Type=oneshot +User=ci-runner +Group=ci-runner +EnvironmentFile=/etc/ci/environment +WorkingDirectory=/opt/ci/local-ci-cd-system +# NOTE: requires PowerShell Core (`pwsh`) on the Linux host. +# See deploy/systemd/README.md "Nota PowerShell su Linux". +ExecStart=/usr/bin/pwsh -NoProfile -NonInteractive -File scripts/Invoke-RetentionPolicy.ps1 +TimeoutStartSec=1h +Nice=15 +ProtectSystem=strict +ProtectHome=true +PrivateTmp=true +NoNewPrivileges=true +ReadWritePaths=/var/lib/ci /var/log/ci + +[Install] +WantedBy=multi-user.target diff --git a/deploy/systemd/ci-retention-policy.timer b/deploy/systemd/ci-retention-policy.timer new file mode 100644 index 0000000..146a0ad --- /dev/null +++ b/deploy/systemd/ci-retention-policy.timer @@ -0,0 +1,14 @@ +[Unit] +Description=Local CI/CD — daily timer for ci-retention-policy.service +Documentation=file:///opt/ci/local-ci-cd-system/deploy/systemd/README.md + +[Timer] +# Equivalent to Windows Task Scheduler: +# New-ScheduledTaskTrigger -Daily -At '03:00' -RandomDelay 30min +OnCalendar=*-*-* 03:00:00 +RandomizedDelaySec=30min +Persistent=true +Unit=ci-retention-policy.service + +[Install] +WantedBy=timers.target diff --git a/deploy/systemd/ci-watch-disk-space.service b/deploy/systemd/ci-watch-disk-space.service new file mode 100644 index 0000000..fc5677e --- /dev/null +++ b/deploy/systemd/ci-watch-disk-space.service @@ -0,0 +1,20 @@ +[Unit] +Description=Local CI/CD — disk space monitor (alert via systemd journal) +Documentation=file:///opt/ci/local-ci-cd-system/deploy/systemd/README.md +After=network-online.target + +[Service] +Type=oneshot +User=ci-runner +Group=ci-runner +EnvironmentFile=/etc/ci/environment +ExecStart=/opt/ci/venv/bin/python -m ci_orchestrator monitor disk +TimeoutStartSec=5min +Nice=15 +ProtectSystem=strict +ProtectHome=true +PrivateTmp=true +NoNewPrivileges=true + +[Install] +WantedBy=multi-user.target diff --git a/deploy/systemd/ci-watch-disk-space.timer b/deploy/systemd/ci-watch-disk-space.timer new file mode 100644 index 0000000..24b8af9 --- /dev/null +++ b/deploy/systemd/ci-watch-disk-space.timer @@ -0,0 +1,15 @@ +[Unit] +Description=Local CI/CD — periodic timer for ci-watch-disk-space.service +Documentation=file:///opt/ci/local-ci-cd-system/deploy/systemd/README.md + +[Timer] +# Equivalent to Windows Task Scheduler: +# New-ScheduledTaskTrigger -Once -At '00:00' -RepetitionInterval 15min +OnBootSec=5min +OnUnitActiveSec=15min +Persistent=true +AccuracySec=30s +Unit=ci-watch-disk-space.service + +[Install] +WantedBy=timers.target diff --git a/deploy/systemd/ci-watch-runner-health.service b/deploy/systemd/ci-watch-runner-health.service new file mode 100644 index 0000000..dd47b90 --- /dev/null +++ b/deploy/systemd/ci-watch-runner-health.service @@ -0,0 +1,24 @@ +[Unit] +Description=Local CI/CD — auto-restart act_runner if stopped (rate-limited) +Documentation=file:///opt/ci/local-ci-cd-system/deploy/systemd/README.md +After=network-online.target act-runner.service + +[Service] +Type=oneshot +User=ci-runner +Group=ci-runner +EnvironmentFile=/etc/ci/environment +# `monitor runner` enforces internal rate-limit: max 3 restarts/h +# (matches Windows CI-RunnerHealth scheduled task semantics). +ExecStart=/opt/ci/venv/bin/python -m ci_orchestrator monitor runner +TimeoutStartSec=5min +Nice=15 +ProtectSystem=strict +ProtectHome=true +PrivateTmp=true +# NoNewPrivileges intentionally NOT set: monitor may need to call +# `systemctl restart act-runner.service` via polkit. +ReadWritePaths=/var/lib/ci /var/log/ci + +[Install] +WantedBy=multi-user.target diff --git a/deploy/systemd/ci-watch-runner-health.timer b/deploy/systemd/ci-watch-runner-health.timer new file mode 100644 index 0000000..fb90aed --- /dev/null +++ b/deploy/systemd/ci-watch-runner-health.timer @@ -0,0 +1,15 @@ +[Unit] +Description=Local CI/CD — periodic timer for ci-watch-runner-health.service +Documentation=file:///opt/ci/local-ci-cd-system/deploy/systemd/README.md + +[Timer] +# Equivalent to Windows Task Scheduler: +# New-ScheduledTaskTrigger -Once -At '00:00' -RepetitionInterval 15min +OnBootSec=3min +OnUnitActiveSec=15min +Persistent=true +AccuracySec=30s +Unit=ci-watch-runner-health.service + +[Install] +WantedBy=timers.target diff --git a/plans/B5-closeout.md b/plans/B5-closeout.md new file mode 100644 index 0000000..d827e85 --- /dev/null +++ b/plans/B5-closeout.md @@ -0,0 +1,76 @@ +# B5 — Closeout (parziale: artefatti repo) + +Fase B5 di `plans/implementation-plan-A-B.md`: conversione scheduled task +Windows in coppie `*.service` + `*.timer` systemd su Linux. + +## Cosa è stato fatto in questa sessione (su host Windows dev) + +- [x] Inventario dei 4 task in `scripts/Register-CIScheduledTasks.ps1` (cadenze + comandi) +- [x] 5 coppie `*.service` + `*.timer` create in `deploy/systemd/`: + - `ci-cleanup-orphans.{service,timer}` — every 6h + at boot → `python -m ci_orchestrator vm cleanup --max-age-hours 6` + - `ci-retention-policy.{service,timer}` — daily 03:00 + 30min jitter → `pwsh Invoke-RetentionPolicy.ps1` + - `ci-watch-disk-space.{service,timer}` — every 15min → `python -m ci_orchestrator monitor disk` + - `ci-watch-runner-health.{service,timer}` — every 15min → `python -m ci_orchestrator monitor runner` + - `ci-backup-template.{service,timer}` — weekly Sun 02:00 + 1h jitter → `pwsh Backup-CITemplate.ps1` (NEW, non era in Register-CIScheduledTasks.ps1) +- [x] `deploy/systemd/README.md` con: tabella mapping, prerequisiti (utente, venv, env file), istruzioni install/test/rollback, nota PowerShell Core su Linux + +## Note sulle cadenze + +Le cadenze del piano (`OnCalendar=hourly`, `*:0/15`, ecc.) sono state +adattate per fedeltà semantica all'originale Windows: + +| Task | Original Windows | systemd timer scelto | +| --------------------- | --------------------------------------------- | ----------------------------------------------------- | +| cleanup-orphans | every 6h + at startup | `OnBootSec=2min, OnUnitActiveSec=6h, Persistent` | +| retention-policy | daily 03:00 + 30min random delay | `OnCalendar=*-*-* 03:00:00, RandomizedDelaySec=30min` | +| watch-disk-space | every 15min | `OnBootSec=5min, OnUnitActiveSec=15min` | +| watch-runner-health | every 15min | `OnBootSec=3min, OnUnitActiveSec=15min` | +| backup-template | *(non in Register-CIScheduledTasks.ps1)* | `OnCalendar=Sun *-*-* 02:00:00, RandomizedDelaySec=1h`| + +`Persistent=true` ovunque per non perdere esecuzioni se la macchina è +spenta al momento del trigger. + +## Decisione — script che restano in PowerShell + +`Invoke-RetentionPolicy.ps1` e `Backup-CITemplate.ps1` non sono stati +portati a Python (resta scelta documentata in `AGENTS.md` "Mappatura"). +Su Linux quindi le rispettive `.service` invocano `pwsh` (PowerShell +Core). Il README documenta l'installazione `apt install powershell` da +repo Microsoft. Se in futuro si deciderà di portarli a Python (sub-comandi +`retention run` / `template backup`), basterà sostituire `ExecStart=` +nelle due unit senza toccare il timer. + +## Definizione di "fatto" B5 (dal piano) + +| Criterio | Stato | +| --------------------------------------------------- | ------------------------------------ | +| Tutti i task periodici come timer systemd attivi | ⏳ unit pronti; install richiede host Linux | +| Trigger manuale di ognuno PASS | ⏳ richiede host Linux | +| Mapping documentato | ✅ `deploy/systemd/README.md` | + +## Cosa resta a carico utente (sul nuovo host Linux) + +Tutti i passi B1–B4 e B6–B8 sono operazioni hardware/host non eseguibili +da questa sessione. Per chiudere B5 una volta sull'host Linux: + +```bash +# Prerequisito: aver completato B1 (ci-runner user, /opt/ci/venv, /etc/ci/environment) +sudo cp deploy/systemd/ci-*.service deploy/systemd/ci-*.timer /etc/systemd/system/ +sudo systemctl daemon-reload +for t in ci-cleanup-orphans ci-retention-policy ci-watch-disk-space \ + ci-watch-runner-health ci-backup-template; do + sudo systemctl enable --now "${t}.timer" +done +systemctl list-timers --all 'ci-*' + +# Trigger manuale di smoke +sudo systemctl start ci-cleanup-orphans.service +journalctl -u ci-cleanup-orphans.service -n 50 --no-pager +``` + +## Riferimenti + +- Spec systemd timer: `man systemd.timer`, `man systemd.time` +- Original PS task scheduler: `scripts/Register-CIScheduledTasks.ps1` +- Mapping completo: `deploy/systemd/README.md` +- Piano: `plans/implementation-plan-A-B.md` sezione B5 diff --git a/plans/PhaseA-user-checklist.md b/plans/PhaseA-user-checklist.md index 31f5907..fb17443 100644 --- a/plans/PhaseA-user-checklist.md +++ b/plans/PhaseA-user-checklist.md @@ -258,17 +258,17 @@ Solo dopo che i passi 3–7 sono tutti `[x]` PASS. ## Tracciamento globale -| Passo | Descrizione | Stato | -| ----- | ----------- | ----- | -| 1 | Aggiorna venv di produzione | [ ] | -| 2 | Restart act_runner | [ ] | -| 3 | Smoke `wait-ready` Windows | [ ] | -| 4 | Smoke `wait-ready` Linux | [ ] | -| 5 | Smoke `job` end-to-end (Win + Linux) | [ ] | -| 6 | Workflow Gitea (lint + self-test + build) | [ ] | -| 7 | Burn-in 4 job × 10 round | [ ] | -| 8 | Benchmark wall-clock | [ ] | -| 9 | Merge PR e tag release | [ ] | +| Passo | Descrizione | Stato | +| ----- | ----------------------------------------- | ----- | +| 1 | Aggiorna venv di produzione | [ ] | +| 2 | Restart act_runner | [ ] | +| 3 | Smoke `wait-ready` Windows | [ ] | +| 4 | Smoke `wait-ready` Linux | [ ] | +| 5 | Smoke `job` end-to-end (Win + Linux) | [ ] | +| 6 | Workflow Gitea (lint + self-test + build) | [ ] | +| 7 | Burn-in 4 job × 10 round | [ ] | +| 8 | Benchmark wall-clock | [ ] | +| 9 | Merge PR e tag release | [ ] | Quando tutti i passi sono `[x]`, la Fase A è **definitivamente chiusa** e si può aprire la Fase B (porting su host Linux). diff --git a/plans/implementation-plan-A-B.md b/plans/implementation-plan-A-B.md index ca052e1..0186775 100644 --- a/plans/implementation-plan-A-B.md +++ b/plans/implementation-plan-A-B.md @@ -89,7 +89,7 @@ A4; cambiano solo path/env vars in Fase B). - [ ] [B4] Scaricare act_runner Linux ≥ v1.0.2 e registrarlo verso Gitea con label `windows-build:host` e `linux-build:host` - [ ] [B4] Creare unit `/etc/systemd/system/act-runner.service` con `User=ci-runner`, env `CI_*`, `PYTHONIOENCODING=utf-8` - [ ] [B4] `systemctl enable --now act-runner.service` e validare via `journalctl -u act-runner -f` -- [ ] [B5] Convertire `Register-CIScheduledTasks.ps1` in coppie `*.service` + `*.timer` (`cleanup-orphaned-vms`, `retention-policy`, `watch-disk-space`, `watch-runner-health`, `backup-template`) +- [x] [B5] Convertire `Register-CIScheduledTasks.ps1` in coppie `*.service` + `*.timer` (`cleanup-orphaned-vms`, `retention-policy`, `watch-disk-space`, `watch-runner-health`, `backup-template`) - [ ] [B5] Abilitare tutti i timer con `systemctl enable --now` e validare `systemctl list-timers` - [ ] [B6] Stop act_runner sull'host Windows e verifica coda Gitea vuota - [ ] [B6] Rsync incrementale finale `F:\CI\Artifacts\` → `/var/lib/ci/artifacts/` @@ -541,15 +541,15 @@ per l'inventario dei task. **Attività**: -- [ ] Inventariare i task in `Register-CIScheduledTasks.ps1` (identificare cadenza e comando per ognuno) -- [ ] Per `cleanup-orphaned-vms`: creare `ci-cleanup-orphaned-vms.service` (oneshot, `ExecStart=/opt/ci/venv/bin/python -m ci_orchestrator vm cleanup`) + `.timer` (`OnCalendar=hourly`) -- [ ] Per `retention-policy`: creare `ci-retention-policy.service` + `.timer` (`OnCalendar=daily`) -- [ ] Per `watch-disk-space`: `ci-watch-disk-space.service` + `.timer` (`OnCalendar=*:0/15`) -- [ ] Per `watch-runner-health`: `ci-watch-runner-health.service` + `.timer` (`OnCalendar=*:0/5`) -- [ ] Per `backup-template`: `ci-backup-template.service` + `.timer` (`OnCalendar=weekly`) +- [x] Inventariare i task in `Register-CIScheduledTasks.ps1` (identificare cadenza e comando per ognuno) +- [x] Per `cleanup-orphaned-vms`: creare `ci-cleanup-orphaned-vms.service` (oneshot, `ExecStart=/opt/ci/venv/bin/python -m ci_orchestrator vm cleanup`) + `.timer` (`OnCalendar=hourly`) +- [x] Per `retention-policy`: creare `ci-retention-policy.service` + `.timer` (`OnCalendar=daily`) +- [x] Per `watch-disk-space`: `ci-watch-disk-space.service` + `.timer` (`OnCalendar=*:0/15`) +- [x] Per `watch-runner-health`: `ci-watch-runner-health.service` + `.timer` (`OnCalendar=*:0/5`) +- [x] Per `backup-template`: `ci-backup-template.service` + `.timer` (`OnCalendar=weekly`) - [ ] `systemctl daemon-reload` e `systemctl enable --now .timer` - [ ] Validare `systemctl list-timers` mostra tutti i timer schedulati -- [ ] Documentare il mapping in `docs/HOST-SETUP.md` +- [x] Documentare il mapping in `docs/HOST-SETUP.md` **Hook futuri Fase C**: `vm cleanup` con backend ESXi userà la stessa unit — il selettore backend è in `config.toml`, non nel comando.