#51 · Fix GitLab runner: EACCES / ENOTCONN / readStream errors breaking CI deploys

Context

The self-hosted GitLab CE runner (H2eHrgbtW) is randomly killing pnpm-spawned processes with three different node errors, breaking CI for at least three jobs:

• typecheck — Error: readStream must be readable (pnpm internal, pre-TypeScript)
• deploy-workers — Error: spawn /usr/bin/node EACCES (wrangler can't spawn node)
• deploy-wiki — Error: read ENOTCONN (socket teardown mid-spawn)

All three are runner-side, not code-side. They started happening on GitLab CE 18.8.4 (per the project_gitlab_runner_issue.md memory). The current symptom: deploy jobs report "✓ deployed successfully" because apps/scoreboard/deploy.sh (and the root orchestrator) don't check wrangler's exit code — silent failures.

This blocks the entire community-bounty workflow because contributors won't see their merged PRs deploy. As of 2026-05-04, scoreboard and wiki had to be deployed manually from local.

Acceptance criteria

• [ ] All three jobs pass on a green pipeline run with no retries (typecheck, deploy-workers, deploy-wiki)
• [ ] Root cause documented in docs/runbooks/gitlab-runner-recovery.md (or wherever your runbooks live)
• [ ] apps/scoreboard/deploy.sh (and any sibling app deploy scripts) hard-fail on wrangler non-zero exit — log line ✓ deployed successfully only prints when the deploy actually succeeded
• [ ] At least one CI run shows the deploy script exiting with status 1 when wrangler is artificially broken (test the failure path)
• [ ] Pipeline 670 retry succeeds on the same main SHA (or equivalent backfill)

Technical hints

• The runner is on the Proxmox box (Tailscale 100.87.20.88). SSH via your usual path.
• gitlab-runner --version should be checked first; CE 18.8.4 had a known regression in scheduler signal handling
• EACCES on /usr/bin/node is suspicious — check the runner's concurrent_id user and whether /usr/bin/node is mode 755. Container runner? executor type matters
• ENOTCONN on socket spawn often means the runner hit an OOM-killer or prlimit cap mid-job — dmesg and journalctl -u gitlab-runner -n 200
• Memory project_gitlab_runner_issue.md may have more from prior debugging
• .gitlab-ci.yml deploy-workers calls apps/scoreboard/deploy.sh (or root deploy.sh workers) — both need exit-code checks added
• Use set -euo pipefail at the top of each deploy script if not already present

Related

• Pipeline 670: https://git.irregularchat.com/irregulars/irregularchat-monorepo/-/pipelines/670 (3 jobs failed)
• Memory note (operator's prior context): project_gitlab_runner_issue.md

--- *Submit a merge request that says Closes #<this iid> in the description. On merge to main, the GitLab webhook auto-awards the L bounty + standard merge_pr (40 pts) to your scoreboard account. This is operator/infra work — needs Proxmox access for the runner-side fix.*