feat(runner): dedicated rust-build runner instance for LinuxRust2404

`runs-on: rust-build` needs a second act_runner instance, not another label on
the existing one: act_runner resolves a single runner.envs block per process
and never tells the workflow which label picked the job. A label can select a
template only by selecting a runner whose envs name that template. Registered
as ci-linux-rust (id 3), online, announcing rust-build.

- deploy/runner-rust-config.yaml: label rust-build:host only, envs pointing at
  LinuxRust2404 + BaseClean-LinuxRust, its own cache dir (two processes must
  not share one actcache database), capacity 1 on top of the main runner's 2.
  runner.file is absolute: act_runner resolves a relative `file:` against the
  process CWD, so registering from elsewhere drops the registration state --
  which holds the runner token -- wherever it was invoked, and the daemon then
  fails with "registration file not found".
- scripts/install-rust-runner.sh: idempotent, --dry-run, refuses to touch the
  main runner directory, requires the template to exist first. Takes
  --token-file rather than --token so the secret stays out of the process
  argument list; the dry-run echo redacts it.
- tools/gitea-runner-token.py: mints a registration token from the stored
  GiteaPAT and writes it to a 0600 file. Neither secret is ever printed. Sets
  an explicit User-Agent because Cloudflare rejects urllib's default with
  error 1010.
- tools/gitea-runner-list.py: --label NAME exits non-zero unless an ONLINE
  runner announces it. A runs-on value no runner advertises leaves jobs
  unschedulable with no error anywhere, which is exactly the failure the
  consumer repo warned about.

deploy/runner-rust-config.yaml

@@ -0,0 +1,84 @@
+# act_runner configuration — rust-build instance (LinuxRust2404 template).
+#
+# Installed to /var/lib/ci/runner-rust/config.yaml by
+# scripts/install-rust-runner.sh. Keep this file and the live one in sync.
+#
+# This instance exists only to give `runs-on: rust-build` its own template.
+# act_runner resolves one `runner.envs` block per process and never tells the
+# workflow which label picked the job, so routing a label to a template means
+# routing it to a runner. Everything else (host paths, capacity budget) mirrors
+# the main runner at /var/lib/ci/runner/config.yaml.
+
+log:
+  level: info
+
+runner:
+  # Absolute on purpose: act_runner resolves a relative `file` against the
+  # process CWD, so `register` run from anywhere but this directory would drop
+  # the registration state (which holds the runner token) wherever it happened
+  # to be invoked, and the daemon would then start with "registration file not
+  # found".
+  file: /var/lib/ci/runner-rust/.runner
+  # Concurrent jobs. This is ON TOP of the main runner's capacity — both
+  # instances clone VMs onto the same host. Keep main.capacity +
+  # rust.capacity within the host RAM budget (~8 GB per build VM).
+  capacity: 1
+  envs:
+    CI_PYTHON_LAUNCHER: /opt/ci/venv/bin/python
+    # The Rust template. `guest-os: 'Linux'` in the workflow resolves to this.
+    GITEA_CI_LINUX_TEMPLATE_PATH: /var/lib/ci/templates/LinuxRust2404/LinuxRust2404.vmx
+    # Read by the local-ci-build composite action; without it the action falls
+    # back to 'BaseClean-Linux', which does not exist on this template.
+    GITEA_CI_LINUX_SNAPSHOT_NAME: BaseClean-LinuxRust
+    # Windows template kept out on purpose: this runner announces no Windows
+    # label, so a Windows job can never be scheduled here.
+  env_file: .env
+  timeout: 3h
+  shutdown_timeout: 0s
+  insecure: false
+  fetch_timeout: 5s
+  fetch_interval: 2s
+  fetch_interval_max: 5s
+  workdir_cleanup_age: 24h
+  idle_cleanup_interval: 10m
+  log_report_interval: 5s
+  log_report_max_latency: 3s
+  log_report_batch_size: 100
+  state_report_interval: 5s
+  github_mirror: ''
+  # Exactly one label. Adding linux-build here would make this instance
+  # eligible for generic Linux jobs and hand them the Rust template.
+  labels:
+    - "rust-build:host"
+  allocate_pty: false
+
+cache:
+  # Separate cache dir from the main runner: two processes must not share one
+  # actcache database.
+  enabled: true
+  dir: "/var/lib/ci/runner-rust/cache"
+  host: ""
+  port: 0
+  external_server: ""
+  external_secret: ""
+
+container:
+  # All builds run in VMware VMs, never containers.
+  network: ""
+  privileged: false
+  options:
+  workdir_parent:
+  valid_volumes: []
+  docker_host: "-"
+  force_pull: false
+  force_rebuild: false
+  require_docker: false
+  docker_timeout: 0s
+  bind_workdir: false
+
+host:
+  workdir_parent:
+
+metrics:
+  enabled: false
+  addr: "127.0.0.1:9102"

deploy/systemd/act-runner-rust.service

@@ -0,0 +1,31 @@
+[Unit]
+# Second act_runner instance, dedicated to the Rust build template.
+#
+# Why a second instance instead of one more label on the existing runner:
+# act_runner has a single `runner.envs` block per process, and it does not
+# expose the job's runs-on label to the workflow (no RUNNER_LABELS variable).
+# So a label can only select a *template* if it selects a *runner* whose envs
+# name that template. One label per runner instance is the only wiring that
+# actually routes.
+#
+# Provisioned by scripts/install-rust-runner.sh.
+Description=Gitea Act Runner (rust-build)
+After=network-online.target vmware-host.service
+Wants=network-online.target
+
+[Service]
+Type=simple
+User=ci-runner
+WorkingDirectory=/var/lib/ci/runner-rust
+Environment="PYTHONIOENCODING=utf-8"
+Environment="CI_ROOT=/var/lib/ci"
+Environment="CI_TEMPLATES=/var/lib/ci/templates"
+Environment="CI_BUILD_VMS=/var/lib/ci/build-vms"
+Environment="CI_ARTIFACTS=/var/lib/ci/artifacts"
+Environment="CI_KEYS=/etc/ci/keys"
+ExecStart=/opt/ci/act_runner/act_runner daemon --config /var/lib/ci/runner-rust/config.yaml
+Restart=on-failure
+RestartSec=10
+
+[Install]
+WantedBy=multi-user.target

scripts/install-rust-runner.sh

@@ -0,0 +1,192 @@
+#!/usr/bin/env bash
+# install-rust-runner.sh
+#
+# Provision the second act_runner instance that owns the `rust-build:host`
+# label and the LinuxRust2404 template.
+#
+# Why a second instance: act_runner has one `runner.envs` block per process and
+# does not expose the job's runs-on label to the workflow (there is no
+# RUNNER_LABELS variable — v1.0.4 provides only RUNNER_OS/ARCH/TEMP/
+# TOOL_CACHE/DEBUG/VERSION). A label can therefore only select a template by
+# selecting a runner whose envs name that template.
+#
+# Usage:
+#   sudo ./scripts/install-rust-runner.sh --token-file <PATH>
+#        [--instance https://gitea.emulab.it] [--name ci-linux-rust]
+#        [--capacity 1] [--dry-run]
+#
+# The registration token is single-use and short-lived. Get one either from
+# Gitea (Site Administration -> Actions -> Runners -> Create new Runner) or,
+# using the stored GiteaPAT, with:
+#
+#   sudo -u ci-runner /opt/ci/venv/bin/python tools/gitea-runner-token.py \
+#       --out /var/lib/ci/.rust-runner-regtoken
+#
+# Prefer --token-file over --token: an argument passed on the command line is
+# visible to any user via `ps` for the lifetime of the process.
+#
+# Idempotent: re-running with an existing .runner file skips registration and
+# only refreshes the config and the unit.
+
+set -euo pipefail
+
+INSTANCE="https://gitea.emulab.it"
+RUNNER_NAME="ci-linux-rust"
+TOKEN=""
+TOKEN_FILE=""
+CAPACITY=""
+DRY_RUN=0
+
+RUNNER_DIR="/var/lib/ci/runner-rust"
+RUNNER_USER="ci-runner"
+ACT_RUNNER_BIN="/opt/ci/act_runner/act_runner"
+UNIT_NAME="act-runner-rust.service"
+REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+CONFIG_SRC="${REPO_ROOT}/deploy/runner-rust-config.yaml"
+UNIT_SRC="${REPO_ROOT}/deploy/systemd/${UNIT_NAME}"
+
+while [[ $# -gt 0 ]]; do
+    case "$1" in
+        --token)    TOKEN="${2:?--token requires a value}"; shift ;;
+        # Preferred over --token: keeps the secret out of the process argument
+        # list (visible in `ps`) and out of shell history.
+        --token-file)
+            TOKEN_FILE="${2:?--token-file requires a value}"
+            [ -r "$TOKEN_FILE" ] || { echo "Token file not readable: $TOKEN_FILE" >&2; exit 1; }
+            TOKEN="$(tr -d '[:space:]' < "$TOKEN_FILE")"
+            [ -n "$TOKEN" ] || { echo "Token file is empty: $TOKEN_FILE" >&2; exit 1; }
+            shift
+            ;;
+        --instance) INSTANCE="${2:?--instance requires a value}"; shift ;;
+        --name)     RUNNER_NAME="${2:?--name requires a value}"; shift ;;
+        --capacity) CAPACITY="${2:?--capacity requires a value}"; shift ;;
+        --dry-run)  DRY_RUN=1 ;;
+        *) echo "Unknown option: $1" >&2; exit 1 ;;
+    esac
+    shift
+done
+
+run() {
+    if [ "$DRY_RUN" = "1" ]; then
+        echo "  [DRY-RUN] $*"
+    else
+        "$@"
+    fi
+}
+
+echo "=== install-rust-runner.sh ==="
+echo "  instance : ${INSTANCE}"
+echo "  name     : ${RUNNER_NAME}"
+echo "  dir      : ${RUNNER_DIR}"
+echo "  unit     : ${UNIT_NAME}"
+echo ""
+
+# --- Pre-flight -------------------------------------------------------------
+echo "--- Pre-flight ---"
+[ "$(id -u)" -eq 0 ] || { echo "Must run as root (sudo)." >&2; exit 1; }
+[ -x "$ACT_RUNNER_BIN" ] || { echo "act_runner not found: $ACT_RUNNER_BIN" >&2; exit 1; }
+[ -f "$CONFIG_SRC" ] || { echo "Config template not found: $CONFIG_SRC" >&2; exit 1; }
+[ -f "$UNIT_SRC" ] || { echo "Unit file not found: $UNIT_SRC" >&2; exit 1; }
+id "$RUNNER_USER" >/dev/null 2>&1 || { echo "User $RUNNER_USER does not exist." >&2; exit 1; }
+
+# The template this runner is dedicated to must exist, otherwise the first job
+# fails at clone time with a much less obvious error.
+RUST_VMX="/var/lib/ci/templates/LinuxRust2404/LinuxRust2404.vmx"
+if [ ! -f "$RUST_VMX" ]; then
+    echo "Rust template not found: $RUST_VMX" >&2
+    echo "Run 'ci-orchestrator template deploy-rust --then-prepare' first." >&2
+    exit 1
+fi
+echo "  [OK] act_runner: $ACT_RUNNER_BIN"
+echo "  [OK] template:   $RUST_VMX"
+
+# A second instance must not reuse the main runner's state directory.
+if [ "$RUNNER_DIR" = "/var/lib/ci/runner" ]; then
+    echo "Refusing to overwrite the main runner directory." >&2
+    exit 1
+fi
+
+# --- Directory + config -----------------------------------------------------
+echo ""
+echo "--- Directory + config ---"
+run install -d -o "$RUNNER_USER" -g "$RUNNER_USER" -m 0750 "$RUNNER_DIR"
+run install -d -o "$RUNNER_USER" -g "$RUNNER_USER" -m 0750 "${RUNNER_DIR}/cache"
+run install -o "$RUNNER_USER" -g "$RUNNER_USER" -m 0644 "$CONFIG_SRC" "${RUNNER_DIR}/config.yaml"
+if [ -n "$CAPACITY" ]; then
+    run sed -i "s/^  capacity: .*/  capacity: ${CAPACITY}/" "${RUNNER_DIR}/config.yaml"
+fi
+echo "  [OK] ${RUNNER_DIR}/config.yaml"
+
+# --- Registration -----------------------------------------------------------
+echo ""
+echo "--- Registration ---"
+if [ -f "${RUNNER_DIR}/.runner" ]; then
+    echo "  [SKIP] ${RUNNER_DIR}/.runner exists — already registered."
+    echo "         Delete it and re-run to register as a new runner."
+elif [ -z "$TOKEN" ]; then
+    echo "  No --token given and no existing .runner." >&2
+    echo "  Get a registration token from Gitea (Actions -> Runners ->" >&2
+    echo "  Create new Runner) and re-run with --token <token>." >&2
+    exit 1
+else
+    # Never let the token reach stdout: `run` echoes its arguments in dry-run
+    # mode, so the register call is handled separately with a redacted echo.
+    if [ "$DRY_RUN" = "1" ]; then
+        echo "  [DRY-RUN] ${ACT_RUNNER_BIN} register --no-interactive" \
+             "--instance ${INSTANCE} --token <redacted:${#TOKEN} chars>" \
+             "--name ${RUNNER_NAME} --labels rust-build:host"
+    else
+        # cd into the runner dir: act_runner resolves `runner.file` relative to
+        # the CWD. The shipped config uses an absolute path, but staying here
+        # keeps it correct even if that is ever edited back to `.runner`.
+        # --labels is accepted but ignored by `register` when the config file
+        # declares labels; the config is the single source of truth.
+        ( cd "$RUNNER_DIR" && sudo -u "$RUNNER_USER" env HOME="$RUNNER_DIR" \
+            "$ACT_RUNNER_BIN" register \
+            --no-interactive \
+            --instance "$INSTANCE" \
+            --token "$TOKEN" \
+            --name "$RUNNER_NAME" \
+            --config "${RUNNER_DIR}/config.yaml" )
+        chown "$RUNNER_USER:$RUNNER_USER" "${RUNNER_DIR}/.runner"
+        chmod 0600 "${RUNNER_DIR}/.runner"
+        echo "  [OK] registered as ${RUNNER_NAME}"
+    fi
+fi
+
+# --- systemd unit -----------------------------------------------------------
+echo ""
+echo "--- systemd unit ---"
+run install -o root -g root -m 0644 "$UNIT_SRC" "/etc/systemd/system/${UNIT_NAME}"
+run systemctl daemon-reload
+run systemctl enable --now "$UNIT_NAME"
+echo "  [OK] ${UNIT_NAME} enabled and started"
+
+# --- Verification -----------------------------------------------------------
+echo ""
+echo "--- Verification ---"
+if [ "$DRY_RUN" = "1" ]; then
+    echo "  [DRY-RUN] skipped"
+else
+    sleep 3
+    systemctl is-enabled "$UNIT_NAME"
+    systemctl is-active "$UNIT_NAME"
+    echo ""
+    echo "  Labels registered:"
+    sudo -u "$RUNNER_USER" grep -A4 '"labels"' "${RUNNER_DIR}/.runner" || true
+    echo ""
+    echo "  Recent log:"
+    journalctl -u "$UNIT_NAME" -n 15 --no-pager || true
+fi
+
+echo ""
+echo "=== done ==="
+echo "Confirm the runner shows Online in Gitea with label 'rust-build', then"
+echo "a workflow can use:"
+echo ""
+echo "    runs-on: rust-build"
+echo "    # guest-os: 'Linux'  -> LinuxRust2404 + BaseClean-LinuxRust"
+echo ""
+echo "Check that the label is actually announced BEFORE relying on it: a"
+echo "runs-on value no runner announces leaves the job unschedulable with no"
+echo "error anywhere."

tools/gitea-runner-list.py

@@ -0,0 +1,106 @@
+#!/opt/ci/venv/bin/python
+"""List the act_runners Gitea knows about, with their status and labels.
+
+Answers the question that matters before writing a `runs-on:` value: is that
+label actually announced by an online runner? A runs-on that no runner
+advertises leaves the job unschedulable with no error anywhere — the CI just
+goes quiet.
+
+Usage:
+    sudo -u ci-runner /opt/ci/venv/bin/python tools/gitea-runner-list.py
+    ... --label rust-build      # exit 1 unless an online runner announces it
+
+Reads the PAT from the keyring and never prints it.
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+import urllib.error
+import urllib.request
+
+DEFAULT_INSTANCE = "https://gitea.emulab.it"
+CRED_TARGET = "GiteaPAT"
+# Cloudflare in front of the instance rejects the default "Python-urllib/x.y"
+# agent with error 1010; any explicit agent passes.
+USER_AGENT = "ci-orchestrator-gitea-runner-list/1.0"
+
+
+def _read_pat(target: str) -> str:
+    try:
+        import keyring
+    except ImportError:  # pragma: no cover - environment problem, not logic
+        sys.exit("keyring not installed")
+
+    cred = keyring.get_credential(target, None)
+    if cred and cred.password:
+        return str(cred.password)
+    username = keyring.get_password(f"{target}:meta", "username")
+    if username:
+        password = keyring.get_password(target, username)
+        if password:
+            return str(password)
+    sys.exit(f"credential {target!r} not found in keyring")
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser(description=__doc__)
+    ap.add_argument("--instance", default=DEFAULT_INSTANCE)
+    ap.add_argument("--target", default=CRED_TARGET)
+    ap.add_argument("--label", default="", help="require this label on an online runner")
+    args = ap.parse_args()
+
+    url = args.instance.rstrip("/") + "/api/v1/admin/actions/runners"
+    req = urllib.request.Request(
+        url,
+        headers={
+            "Authorization": f"token {_read_pat(args.target)}",
+            "Accept": "application/json",
+            "User-Agent": USER_AGENT,
+        },
+    )
+    try:
+        with urllib.request.urlopen(req, timeout=30) as resp:
+            payload = json.loads(resp.read().decode("utf-8"))
+    except urllib.error.HTTPError as exc:
+        print(f"HTTP {exc.code} from {url}", file=sys.stderr)
+        return 1
+    except urllib.error.URLError as exc:
+        print(f"cannot reach {url}: {exc.reason}", file=sys.stderr)
+        return 1
+
+    runners = payload.get("runners", payload) if isinstance(payload, dict) else payload
+    if not isinstance(runners, list):
+        print(f"unexpected response shape: {type(runners).__name__}", file=sys.stderr)
+        return 1
+
+    print(f"{'ID':<5} {'NAME':<20} {'STATUS':<10} LABELS")
+    matched = False
+    for r in runners:
+        # Gitea reports labels either as plain strings or as {name: ...} objects.
+        labels = [
+            str(x.get("name", x)) if isinstance(x, dict) else str(x)
+            for x in (r.get("labels") or [])
+        ]
+        status = str(r.get("status", "?"))
+        name = str(r.get("name", "?"))
+        print(f"{r.get('id', '?'):<5} {name:<20} {status:<10} {','.join(labels)}")
+        if args.label and args.label in labels and status.lower() == "online":
+            matched = True
+
+    if args.label:
+        if matched:
+            print(f"\nOK — an online runner announces {args.label!r}.")
+            return 0
+        print(
+            f"\nFAIL — no ONLINE runner announces {args.label!r}. "
+            "A job with that runs-on would sit unschedulable, silently.",
+            file=sys.stderr,
+        )
+        return 1
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

tools/gitea-runner-token.py

@@ -0,0 +1,122 @@
+#!/opt/ci/venv/bin/python
+"""Mint a Gitea act_runner registration token using the stored GiteaPAT.
+
+The PAT never leaves this process and the registration token is written to a
+0600 file instead of stdout, so neither secret lands in a shell history, a
+process argument list, or a terminal log.
+
+Usage:
+    sudo -u ci-runner /opt/ci/venv/bin/python tools/gitea-runner-token.py \
+        --out /var/lib/ci/.rust-runner-regtoken
+
+    # non-instance scopes
+    ... --scope user
+    ... --scope org  --org  myorg
+    ... --scope repo --repo Simone/local-ci-cd-system
+
+Instance scope (the default) requires the PAT to belong to an admin user and
+produces a runner visible to every repository — which is what a shared build
+host wants. Registration tokens are short-lived and single-use.
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import sys
+import urllib.error
+import urllib.request
+
+DEFAULT_INSTANCE = "https://gitea.emulab.it"
+CRED_TARGET = "GiteaPAT"
+USER_AGENT = "ci-orchestrator-gitea-runner-token/1.0"
+
+
+def _read_pat(target: str) -> str:
+    """Read the PAT via the same two-tier scheme as KeyringCredentialStore."""
+    try:
+        import keyring
+    except ImportError:  # pragma: no cover - environment problem, not logic
+        sys.exit("keyring not installed")
+
+    cred = keyring.get_credential(target, None)
+    if cred and cred.password:
+        return str(cred.password)
+
+    username = keyring.get_password(f"{target}:meta", "username")
+    if username:
+        password = keyring.get_password(target, username)
+        if password:
+            return str(password)
+    sys.exit(f"credential {target!r} not found in keyring")
+
+
+def _endpoint(scope: str, org: str, repo: str) -> str:
+    if scope == "admin":
+        return "/api/v1/admin/actions/runners/registration-token"
+    if scope == "user":
+        return "/api/v1/user/actions/runners/registration-token"
+    if scope == "org":
+        if not org:
+            sys.exit("--scope org requires --org")
+        return f"/api/v1/orgs/{org}/actions/runners/registration-token"
+    if not repo or "/" not in repo:
+        sys.exit("--scope repo requires --repo owner/name")
+    owner, name = repo.split("/", 1)
+    return f"/api/v1/repos/{owner}/{name}/actions/runners/registration-token"
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser(description=__doc__)
+    ap.add_argument("--instance", default=DEFAULT_INSTANCE)
+    ap.add_argument("--scope", default="admin", choices=("admin", "user", "org", "repo"))
+    ap.add_argument("--org", default="")
+    ap.add_argument("--repo", default="")
+    ap.add_argument("--target", default=CRED_TARGET, help="keyring target holding the PAT")
+    ap.add_argument("--out", required=True, help="file to write the token to (mode 0600)")
+    args = ap.parse_args()
+
+    pat = _read_pat(args.target)
+    url = args.instance.rstrip("/") + _endpoint(args.scope, args.org, args.repo)
+
+    req = urllib.request.Request(
+        url,
+        method="POST",
+        data=b"",
+        headers={
+            "Authorization": f"token {pat}",
+            "Accept": "application/json",
+            "Content-Length": "0",
+            # Cloudflare in front of the instance rejects the default
+            # "Python-urllib/x.y" agent with error 1010; any explicit agent
+            # passes. Identify the caller rather than impersonate a browser.
+            "User-Agent": USER_AGENT,
+        },
+    )
+    try:
+        with urllib.request.urlopen(req, timeout=30) as resp:
+            payload = json.loads(resp.read().decode("utf-8"))
+    except urllib.error.HTTPError as exc:
+        body = exc.read().decode("utf-8", "replace")[:300]
+        # 403 on the admin scope usually means the PAT owner is not an admin.
+        print(f"HTTP {exc.code} from {url}\n{body}", file=sys.stderr)
+        return 1
+    except urllib.error.URLError as exc:
+        print(f"cannot reach {url}: {exc.reason}", file=sys.stderr)
+        return 1
+
+    token = payload.get("token") or payload.get("data", {}).get("token")
+    if not token:
+        print(f"no token in response: {sorted(payload)}", file=sys.stderr)
+        return 1
+
+    fd = os.open(args.out, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
+    with os.fdopen(fd, "w", encoding="ascii") as fh:
+        fh.write(token)
+    print(f"OK — registration token ({len(token)} chars) written to {args.out}")
+    print("Single-use and short-lived: delete the file once the runner is registered.")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())