From a906f4696f0d71a2693254f4cddc039532a74055 Mon Sep 17 00:00:00 2001 From: Simone Date: Thu, 21 May 2026 23:35:57 +0200 Subject: [PATCH] fix(templates): bump WinRM quotas + desktop heap for parallel Win builds MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Parallel MSBuild fan-out (e.g. ns7zip x86/x64/x86-ansi) over pypsrp Linux→Win fails with MSBuild MSB6003 / Win32 1816 'Not enough quota' because two default guest limits are too tight for concurrent native processes via WinRM: - Plugin Microsoft.PowerShell MaxMemoryPerShellMB = 150 MB (pypsrp uses the plugin endpoint, not the Shell endpoint that Deploy was previously bumping) - Non-interactive desktop heap SharedSection 3rd field = 768 KB (Session 0 csrss; CreateProcess fails when 3+ cl.exe spawn) Pre-migration Win→Win went through Invoke-Command on PSSession with quotas inherited from the local process, so the limits never bit. Changes: - Deploy-WinBuild2025.ps1 / Deploy-WinBuild2022.ps1: also set the plugin-level MaxMemoryPerShellMB=2048, bump MaxProcessesPerShell 25→100, patch SharedSection 3rd field 768→4096 (reboot needed for csrss to pick up, applied by post-install shutdown). - Install-CIToolchain-WinBuild2025/2022.ps1: add Assert-Step for plugin quota, MaxProcessesPerShell≥100, SharedSection≥4096; fail hard if any are under threshold. - docs/WINDOWS-TEMPLATE-SETUP.md: new "WinRM quotas e desktop heap" section explaining the diagnosis + one-shot patch for legacy templates; updated Step 3 validation row + troubleshooting entry. - AGENTS.md: error #13 with symptom, root cause, and pointer to docs. Co-Authored-By: Claude Sonnet 4.6 --- AGENTS.md | 1 + docs/WINDOWS-TEMPLATE-SETUP.md | 52 ++++++++++++++++++- template/Deploy-WinBuild2022.ps1 | 22 +++++++- template/Deploy-WinBuild2025.ps1 | 22 +++++++- template/Install-CIToolchain-WinBuild2022.ps1 | 18 +++++-- template/Install-CIToolchain-WinBuild2025.ps1 | 18 +++++-- 6 files changed, 122 insertions(+), 11 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 3cb8bc7..9cabc40 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -226,3 +226,4 @@ Configurazione in `PSScriptAnalyzerSettings.psd1`. Regole principali: 10. **`vmrun getGuestIPAddress` per check "VM running"** — richiede VMware Tools attivi nel guest; in headless Tools puo' impiegare 30-60s a rispondere e il check appare bloccato. Usare `vmrun list` e cercare il path del VMX nell'output per `is_running`. Per recuperare l'IP usare `getGuestIPAddress -wait` (blocca internamente finché Tools registrano l'IP) e filtrare le righe che iniziano con `"Error:"` perché vmrun in modalità headless può tornare exit code 0 con `"Error: The VMware Tools are not running..."` su stdout invece di usare un exit code non-zero. 11. **Machine-id identico tra clone Linux** — le cloud image Ubuntu hanno un `/etc/machine-id` fisso. VMware DHCP assegna l'IP tramite DHCP client-id derivato da machine-id: se tutti i clone hanno lo stesso machine-id ottengono lo stesso IP e si bloccano per "IP collision". Fix nel template (prima di prendere lo snapshot): `sudo truncate -s 0 /etc/machine-id && sudo rm -f /var/lib/dbus/machine-id && sudo shutdown -h now`, poi ricreare lo snapshot `BaseClean-Linux` da stato powered-off. 12. **`& nativecmd 2>$null` NON sopprime stderr in PS 5.1 con `$ErrorActionPreference='Stop'`** — la stderr del processo nativo viene convertita in ErrorRecord nel pipeline di PowerShell; sotto `'Stop'` questo diventa un errore terminante PRIMA che `2>$null` possa scartarlo. Esempio: `ssh-keygen -R -f ` stampa "Host X not found in file" su stderr → eccezione terminante. Fix: usare `$ErrorActionPreference = 'SilentlyContinue'` attorno alla chiamata oppure `2>&1 | Out-Null`. Per i job CI su VM ephemere locali, usare direttamente `StrictHostKeyChecking=no UserKnownHostsFile=NUL` (default in `_Transport.psm1` con `KnownHostsFile=''`) — nessun `ssh-keygen` necessario. +13. **Quote WinRM/desktop-heap troppo basse per builds Windows paralleli** — sintomo: `MSBuild error MSB6003: CL.exe could not be run. System.ComponentModel.Win32Exception (0x80004005): Not enough quota is available to process this command` (oppure le altre tool MSVC: link.exe, mt.exe) quando il build interno fa fan-out parallelo (es. matrix x86/x64/x86-ansi, MSBuild `/maxcpucount`). Causa: il plugin endpoint Microsoft.PowerShell del guest ha `MaxMemoryPerShellMB=150` di default e l'heap del desktop non-interattivo (Session 0) ha `SharedSection 3rd field=768 KB` — entrambi insufficienti per più CL.exe concorrenti via WinRM. **Non emerge** in pre-migration Win→Win perché `Invoke-Command` su `PSSession` localhost-style eredita quote più larghe; emerge appena il path passa per `pypsrp` Linux→Win. Fix nel template (Deploy-WinBuild2025.ps1 / 2022 lo fa, Install-CIToolchain lo valida): `Set-Item WSMan:\localhost\Plugin\Microsoft.PowerShell\Quotas\MaxMemoryPerShellMB 2048`, `Set-Item WSMan:\localhost\Shell\MaxProcessesPerShell 100`, registry `HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\SubSystems\Windows` patch del 3° campo `SharedSection` a 4096, **reboot** (csrss legge SharedSection solo al boot), nuovo `BaseClean`. Dettagli e procedura completa in `docs/WINDOWS-TEMPLATE-SETUP.md § "WinRM quotas e desktop heap"`. diff --git a/docs/WINDOWS-TEMPLATE-SETUP.md b/docs/WINDOWS-TEMPLATE-SETUP.md index bc0cde9..e7dde39 100644 --- a/docs/WINDOWS-TEMPLATE-SETUP.md +++ b/docs/WINDOWS-TEMPLATE-SETUP.md @@ -197,7 +197,7 @@ Final Pre-snapshot gate — 7 check cross-cutting + no installer leftover | ----- | --------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | 1 | Assert | `Get-NetFirewallProfile -Profile Domain/Private/Public → Enabled=False` | | 2 | Assert | GPO DisableAntiSpyware=1 (set da Deploy) | -| 3 | Assert | WinRM Running+Automatic, AllowUnencrypted=false (HTTPS-only), Auth/Basic=true, MaxMem≥2048 | +| 3 | Assert | WinRM Running+Automatic, AllowUnencrypted=false (HTTPS-only), Auth/Basic=true, Shell+Plugin MaxMem≥2048, MaxProcesses≥100, desktop heap SharedSection 3rd≥4096 | | 4 | Operativo | User exists, enabled, PasswordNeverExpires, member of Administrators | | 4b | Assert | EnableLUA=0, LocalAccountTokenFilterPolicy=1 | | 5 | Operativo | Ogni dir Test-Path -PathType Container | @@ -290,6 +290,55 @@ toolchain (.NET, VS Build Tools, Python). --- +## WinRM quotas e desktop heap (build paralleli) + +I template Windows escono di default da Windows con quote dimensionate per uso +interattivo. Da Linux/`pypsrp` queste quote si rivelano strette appena il build +guest fa fan-out di processi nativi (compilatori, linker) in parallelo. Tre +configurazioni che il template **deve** applicare prima dello snapshot +`BaseClean` (gestite automaticamente da `Deploy-WinBuild2025.ps1` / `2022`): + +| Setting | Default | Valore template | Motivo | +| ---------------------------------------------------------------------------------------- | ------- | --------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `WSMan:\localhost\Shell\MaxMemoryPerShellMB` | 150 | 2048 | Limite globale memoria per shell WinRM (winrs) | +| `WSMan:\localhost\Plugin\Microsoft.PowerShell\Quotas\MaxMemoryPerShellMB` | 150 | 2048 | **pypsrp da Linux usa l'endpoint plugin Microsoft.PowerShell**, non lo Shell. Senza questo, builds paralleli falliscono con `MSB6003 / Not enough quota` anche se Shell è a 2048 | +| `WSMan:\localhost\Shell\MaxProcessesPerShell` | 25 | 100 | MSBuild + cl + link + mt + altri tool si moltiplicano in fan-out parallelo | +| `HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\SubSystems\Windows SharedSection` | `…,768` | `…,4096` | Heap del desktop **non-interattivo** (Session 0). CreateProcess fallisce con `ERROR_NOT_ENOUGH_QUOTA` (1816) quando 3+ cl.exe partono insieme. Richiede **reboot** per essere letto da `csrss.exe`. | + +**Importante**: la modifica `SharedSection` è una scrittura di registry che il +kernel legge **solo al boot** (`csrss.exe`). Lo `shutdown /s` finale di +`post-install.ps1` fa sì che il prossimo avvio del template (per ulteriore +manutenzione, o l'invocazione di `Install-CIToolchain`) parta con il valore +corretto. Il `BaseClean` snapshot va preso **dopo** un boot completo per essere +certi che il valore sia attivo. + +**Perché pre-migrazione (Win host) funzionava**: il path Windows→Windows usava +`Invoke-Command` su `PSSession` con la stessa identità del processo locale (act_runner +come LocalSystem), ereditando spesso quote più generose tramite GPO e default +del client locale. Il path Linux→Windows (`pypsrp`) attinge ai default puri +dell'endpoint plugin Microsoft.PowerShell del guest — se il template non li ha +bumpati, sono **150 MB / 25 processi / 768 KB heap**. + +**Validazione**: `Install-CIToolchain-WinBuild2025.ps1` Step 3 controlla tutti e +quattro i valori e fallisce hard se anche solo uno è sotto soglia. Per +template già esistenti, applicare il blocco PowerShell sotto, reboot, shutdown, +nuovo snapshot `BaseClean`: + +```powershell +# Una tantum su template legacy +Set-Item WSMan:\localhost\Shell\MaxMemoryPerShellMB 2048 -Force +Set-Item 'WSMan:\localhost\Plugin\Microsoft.PowerShell\Quotas\MaxMemoryPerShellMB' 2048 -Force +Set-Item WSMan:\localhost\Shell\MaxProcessesPerShell 100 -Force +$k = 'HKLM:\SYSTEM\CurrentControlSet\Control\Session Manager\SubSystems' +$cur = (Get-ItemProperty -Path $k -Name Windows).Windows +$new = $cur -replace 'SharedSection=(\d+),(\d+),\d+', 'SharedSection=$1,$2,4096' +Set-ItemProperty -Path $k -Name Windows -Value $new +Restart-Service WinRM -Force +Restart-Computer -Force # necessario per il desktop heap +``` + +--- + ## Troubleshooting | Sintomo | Diagnosi / Fix | @@ -301,6 +350,7 @@ toolchain (.NET, VS Build Tools, Python). | `Python version match` fail | `$PythonVersion` ≠ versione installata. Aggiorna param o reinstalla | | Post-setup `MSBuild not found` | VS install fallita silenziosa. Check `C:\CI\vs_log.txt` | | Pre-snapshot Final check fail | Stato VM rotto — non prendere snapshot. Re-run o ricomincia da zero | +| MSBuild `error MSB6003: CL.exe could not be run` + `Win32Exception (0x80004005): Not enough quota` su build paralleli | Quote WinRM/desktop-heap non bumpate nel template (vedi §"WinRM quotas e desktop heap" sotto). Tipico quando il build interno fa fan-out (es. 3× cl.exe paralleli) via pypsrp Linux→Win | --- diff --git a/template/Deploy-WinBuild2022.ps1 b/template/Deploy-WinBuild2022.ps1 index 9d8f0cf..9d1f228 100644 --- a/template/Deploy-WinBuild2022.ps1 +++ b/template/Deploy-WinBuild2022.ps1 @@ -639,11 +639,29 @@ try { New-NetFirewallRule -DisplayName 'WinRM-HTTPS' -Direction Inbound -Protocol TCP -LocalPort 5986 -Action Allow -Profile Any -RemoteAddress '192.168.79.0/24' | Out-Null L 'WinRM HTTPS listener configured (port 5986, restricted to 192.168.79.0/24)' } catch { L "WinRM HTTPS failed: `$(`$_.Exception.Message)" } -winrm set winrm/config/winrs '@{MaxProcessesPerShell="25"}' | Out-Null +winrm set winrm/config/winrs '@{MaxProcessesPerShell="100"}' | Out-Null Set-Item WSMan:\localhost\Shell\MaxMemoryPerShellMB 2048 -Force 3>`$null Set-Item WSMan:\localhost\Shell\IdleTimeOut 7200000 -Force 3>`$null +# Plugin-level quota: pypsrp (Linux→Win) uses Microsoft.PowerShell endpoint, +# which has its own MaxMemoryPerShellMB independent of the Shell quota above. +# Without this, parallel native processes (e.g. 3x cl.exe in MSBuild fan-out) +# fail with MSB6003 / Win32 1816 'Not enough quota'. +Set-Item 'WSMan:\localhost\Plugin\Microsoft.PowerShell\Quotas\MaxMemoryPerShellMB' 2048 -Force 3>`$null Set-Service -Name WinRM -StartupType Automatic 3>`$null -L 'WinRM shell limits: MaxMemory=2048MB IdleTimeout=2h MaxProcesses=25; StartType=Automatic' +Restart-Service -Name WinRM -Force 3>`$null +L 'WinRM limits: Shell+Plugin MaxMemory=2048MB, IdleTimeout=2h, MaxProcesses=100; StartType=Automatic' + +# ── Non-interactive desktop heap (csrss SharedSection) ─────────────────── +# WinRM sessions run in Session 0 non-interactive window station; default +# desktop heap is 768 KB. CreateProcess fails with ERROR_NOT_ENOUGH_QUOTA +# (Win32 1816 → MSB6003) when multiple compiler processes spawn in parallel. +# Bump the third SharedSection field to 4096 KB. Change is read by csrss only +# at boot — the post-install shutdown at the end of this script applies it. +`$subsysKey = 'HKLM:\SYSTEM\CurrentControlSet\Control\Session Manager\SubSystems' +`$subsysCur = (Get-ItemProperty -Path `$subsysKey -Name Windows).Windows +`$subsysNew = `$subsysCur -replace 'SharedSection=(\d+),(\d+),\d+', 'SharedSection=`$1,`$2,4096' +Set-ItemProperty -Path `$subsysKey -Name Windows -Value `$subsysNew +L "Desktop heap (non-interactive) SharedSection 3rd field set to 4096 KB (takes effect after reboot)" # ── Server Manager / WAC popup / first-logon UX ────────────────────────── # Server Manager auto-start off — machine-wide + current Administrator + Default diff --git a/template/Deploy-WinBuild2025.ps1 b/template/Deploy-WinBuild2025.ps1 index 0cb2f8b..8c6180d 100644 --- a/template/Deploy-WinBuild2025.ps1 +++ b/template/Deploy-WinBuild2025.ps1 @@ -639,11 +639,29 @@ try { New-NetFirewallRule -DisplayName 'WinRM-HTTPS' -Direction Inbound -Protocol TCP -LocalPort 5986 -Action Allow -Profile Any -RemoteAddress '192.168.79.0/24' | Out-Null L 'WinRM HTTPS listener configured (port 5986, restricted to 192.168.79.0/24)' } catch { L "WinRM HTTPS failed: `$(`$_.Exception.Message)" } -winrm set winrm/config/winrs '@{MaxProcessesPerShell="25"}' | Out-Null +winrm set winrm/config/winrs '@{MaxProcessesPerShell="100"}' | Out-Null Set-Item WSMan:\localhost\Shell\MaxMemoryPerShellMB 2048 -Force 3>`$null Set-Item WSMan:\localhost\Shell\IdleTimeOut 7200000 -Force 3>`$null +# Plugin-level quota: pypsrp (Linux→Win) uses Microsoft.PowerShell endpoint, +# which has its own MaxMemoryPerShellMB independent of the Shell quota above. +# Without this, parallel native processes (e.g. 3x cl.exe in MSBuild fan-out) +# fail with MSB6003 / Win32 1816 'Not enough quota'. +Set-Item 'WSMan:\localhost\Plugin\Microsoft.PowerShell\Quotas\MaxMemoryPerShellMB' 2048 -Force 3>`$null Set-Service -Name WinRM -StartupType Automatic 3>`$null -L 'WinRM shell limits: MaxMemory=2048MB IdleTimeout=2h MaxProcesses=25; StartType=Automatic' +Restart-Service -Name WinRM -Force 3>`$null +L 'WinRM limits: Shell+Plugin MaxMemory=2048MB, IdleTimeout=2h, MaxProcesses=100; StartType=Automatic' + +# ── Non-interactive desktop heap (csrss SharedSection) ─────────────────── +# WinRM sessions run in Session 0 non-interactive window station; default +# desktop heap is 768 KB. CreateProcess fails with ERROR_NOT_ENOUGH_QUOTA +# (Win32 1816 → MSB6003) when multiple compiler processes spawn in parallel. +# Bump the third SharedSection field to 4096 KB. Change is read by csrss only +# at boot — the post-install shutdown at the end of this script applies it. +`$subsysKey = 'HKLM:\SYSTEM\CurrentControlSet\Control\Session Manager\SubSystems' +`$subsysCur = (Get-ItemProperty -Path `$subsysKey -Name Windows).Windows +`$subsysNew = `$subsysCur -replace 'SharedSection=(\d+),(\d+),\d+', 'SharedSection=`$1,`$2,4096' +Set-ItemProperty -Path `$subsysKey -Name Windows -Value `$subsysNew +L "Desktop heap (non-interactive) SharedSection 3rd field set to 4096 KB (takes effect after reboot)" # ── Server Manager / WAC popup / first-logon UX ────────────────────────── # Server Manager auto-start off — machine-wide + current Administrator + Default diff --git a/template/Install-CIToolchain-WinBuild2022.ps1 b/template/Install-CIToolchain-WinBuild2022.ps1 index e6d3f03..d49dfb6 100644 --- a/template/Install-CIToolchain-WinBuild2022.ps1 +++ b/template/Install-CIToolchain-WinBuild2022.ps1 @@ -318,8 +318,10 @@ Assert-Step 'Defender' 'GPO DisableAntiSpyware=1 (set by Deploy-WinBuild2022)' { Write-Step "WinRM state (validation only — configured at deploy time)" # Deploy-WinBuild2022.ps1 post-install.ps1 ran Enable-PSRemoting, set Basic auth over HTTPS, -# AllowUnencrypted=false (HTTPS/5986 only), TrustedHosts=*, MaxMemory=2048MB, IdleTimeout=2h, -# MaxProcesses=25, StartupType=Automatic. Validated here before the build user and toolchain steps. +# AllowUnencrypted=false (HTTPS/5986 only), TrustedHosts=*, Shell+Plugin MaxMemory=2048MB, +# IdleTimeout=2h, MaxProcesses=100, StartupType=Automatic, and bumped the non-interactive +# desktop heap (SharedSection 3rd field) to 4096 KB. Validated here before the build user +# and toolchain steps. Assert-Step 'WinRM' 'service Running' { (Get-Service WinRM -ErrorAction Stop).Status -eq 'Running' @@ -333,9 +335,19 @@ Assert-Step 'WinRM' 'AllowUnencrypted=false (HTTPS-only)' { Assert-Step 'WinRM' 'Auth/Basic=true' { (Get-Item WSMan:\localhost\Service\Auth\Basic).Value -eq 'true' } -Assert-Step 'WinRM' 'MaxMemoryPerShellMB >= 2048' { +Assert-Step 'WinRM' 'Shell MaxMemoryPerShellMB >= 2048' { [int](Get-Item WSMan:\localhost\Shell\MaxMemoryPerShellMB).Value -ge 2048 } +Assert-Step 'WinRM' 'Plugin Microsoft.PowerShell MaxMemoryPerShellMB >= 2048' { + [int](Get-Item 'WSMan:\localhost\Plugin\Microsoft.PowerShell\Quotas\MaxMemoryPerShellMB').Value -ge 2048 +} +Assert-Step 'WinRM' 'MaxProcessesPerShell >= 100' { + [int](Get-Item WSMan:\localhost\Shell\MaxProcessesPerShell).Value -ge 100 +} +Assert-Step 'DesktopHeap' 'SharedSection non-interactive >= 4096 KB' { + $w = (Get-ItemProperty -Path 'HKLM:\SYSTEM\CurrentControlSet\Control\Session Manager\SubSystems' -Name Windows).Windows + if ($w -match 'SharedSection=\d+,\d+,(\d+)') { [int]$Matches[1] -ge 4096 } else { $false } +} # ── Step 3b: Windows KMS Activation ───────────────────────────────────────── # Must run after network is confirmed (Step 3) and before Windows Update (Step 6) diff --git a/template/Install-CIToolchain-WinBuild2025.ps1 b/template/Install-CIToolchain-WinBuild2025.ps1 index d6e368f..d06f081 100644 --- a/template/Install-CIToolchain-WinBuild2025.ps1 +++ b/template/Install-CIToolchain-WinBuild2025.ps1 @@ -318,8 +318,10 @@ Assert-Step 'Defender' 'GPO DisableAntiSpyware=1 (set by Deploy-WinBuild2025)' { Write-Step "WinRM state (validation only — configured at deploy time)" # Deploy-WinBuild2025.ps1 post-install.ps1 ran Enable-PSRemoting, set Basic auth over HTTPS, -# AllowUnencrypted=false (HTTPS/5986 only), TrustedHosts=*, MaxMemory=2048MB, IdleTimeout=2h, -# MaxProcesses=25, StartupType=Automatic. Validated here before the build user and toolchain steps. +# AllowUnencrypted=false (HTTPS/5986 only), TrustedHosts=*, Shell+Plugin MaxMemory=2048MB, +# IdleTimeout=2h, MaxProcesses=100, StartupType=Automatic, and bumped the non-interactive +# desktop heap (SharedSection 3rd field) to 4096 KB. Validated here before the build user +# and toolchain steps. Assert-Step 'WinRM' 'service Running' { (Get-Service WinRM -ErrorAction Stop).Status -eq 'Running' @@ -333,9 +335,19 @@ Assert-Step 'WinRM' 'AllowUnencrypted=false (HTTPS-only)' { Assert-Step 'WinRM' 'Auth/Basic=true' { (Get-Item WSMan:\localhost\Service\Auth\Basic).Value -eq 'true' } -Assert-Step 'WinRM' 'MaxMemoryPerShellMB >= 2048' { +Assert-Step 'WinRM' 'Shell MaxMemoryPerShellMB >= 2048' { [int](Get-Item WSMan:\localhost\Shell\MaxMemoryPerShellMB).Value -ge 2048 } +Assert-Step 'WinRM' 'Plugin Microsoft.PowerShell MaxMemoryPerShellMB >= 2048' { + [int](Get-Item 'WSMan:\localhost\Plugin\Microsoft.PowerShell\Quotas\MaxMemoryPerShellMB').Value -ge 2048 +} +Assert-Step 'WinRM' 'MaxProcessesPerShell >= 100' { + [int](Get-Item WSMan:\localhost\Shell\MaxProcessesPerShell).Value -ge 100 +} +Assert-Step 'DesktopHeap' 'SharedSection non-interactive >= 4096 KB' { + $w = (Get-ItemProperty -Path 'HKLM:\SYSTEM\CurrentControlSet\Control\Session Manager\SubSystems' -Name Windows).Windows + if ($w -match 'SharedSection=\d+,\d+,(\d+)') { [int]$Matches[1] -ge 4096 } else { $false } +} # ── Step 3b: Windows KMS Activation ───────────────────────────────────────── # Must run after network is confirmed (Step 3) and before Windows Update (Step 6)