Support execution on self-hosted without sudo

[dhrapson]

When executing gh aw compile Workflows on self-hosted GitHub Runners fails currently on the Install GitHub Copilot CLI task of the agent Job, if the Runners do not permit escalation to root user.
This is because the script at https://github.com/github/gh-aw/blob/main/actions/setup/sh/install_copilot_cli.sh blindly runs sudo commands without checking existing directory permissions or user, and copies binaries into root-owned directories such as /usr/local/bin.

Please change the GH AW workflow compilation to support execution on a self-hosted Runner that does not permit sudo commands e.g. Actions Runner Controller (ARC) with allowPrivilegeEscalation: false

[dsyme]

/scout research the codebase and determine all the requirements we would have of a self hosted runner to make it viable to use it to run GH-AW workflows, and what would need to be done to also allow "allowPrivilegeEscalation: false" for those runners. Be very precise and do a detailed, careful analysis. Also explain how we might test that we don't pick up additional requirements unexpectedly

[github-actions]

🔍 Scout Research Report

Triggered by @dsyme

Executive Summary

A detailed codebase analysis reveals four distinct categories of privilege escalation in GH-AW workflows, plus a fundamental architectural dependency on Docker and iptables via the Agentic Workflow Firewall (AWF). Supporting allowPrivilegeEscalation: false on ARC runners requires addressing all four, with the AWF iptables dependency being the deepest architectural challenge. A testing strategy using static analysis and CI gating can prevent new requirements from accumulating undetected.

Full Research Findings

Findings: Self-Hosted Runner Requirements

A. Baseline Requirements (No Privilege Issues)

Every GH-AW agent workflow job requires these capabilities from the runner regardless of privilege posture:

Requirement	Evidence	Notes
Linux (amd64 or arm64)	AWF uses iptables explicitly; macOS note in install_awf_binary.sh:121	AWF is Linux-only
Docker daemon accessible	download_docker_images.sh, start_mcp_gateway.sh, constants.go (ghcr.io images referenced throughout)	Both the MCP gateway container (gh-aw-mcpg) and AWF firewall containers are pulled via docker pull and run via docker run
bash, curl, jq, git	Used in every setup script	Standard tools
Node.js (LTS)	setup.sh deploys .cjs scripts; check_workflow_timestamp_api.cjs, setup_globals.cjs etc. are invoked in steps	No specific version pinned in generated workflows; whatever node is on PATH
sha256sum or shasum	install_copilot_cli.sh:74-81, install_awf_binary.sh:46-56	Checksum verification during binary install
Writable /tmp/gh-aw/	create_gh_aw_tmp_dir.sh, all MCP log paths	Workspace for logs, config, payloads
Writable /opt/gh-aw/	setup.sh DESTINATION default (/opt/gh-aw/actions)	Scripts and MCP server files are deployed here
Network egress to ghcr.io	Docker image pulls for gh-aw-mcpg, gh-aw-firewall, github-mcp-server, serena-mcp-server etc.	Images are pre-pulled in the "Download container images" step
Network egress to github.com	Binary downloads in install_copilot_cli.sh, install_awf_binary.sh	Release downloads with SHA256 verification
GITHUB_TOKEN / secrets	Nearly every generated step	Standard Actions requirement

B. Privilege Escalation Points — Exact Code Locations

There are four distinct sudo / privilege escalation patterns in the codebase:

---

1. Copilot CLI installation to /usr/local/bin — actions/setup/sh/install_copilot_cli.sh:119-120

sudo tar -xz -C "${INSTALL_DIR}" -f "${TEMP_DIR}/${TARBALL_NAME}"   # INSTALL_DIR=/usr/local/bin
sudo chmod +x "${INSTALL_DIR}/copilot"

And a separate ownership fix for a root-created .copilot dir — install_copilot_cli.sh:32:

sudo chown -R runner:runner "$COPILOT_DIR"   # COPILOT_DIR=/home/runner/.copilot

The chown is needed because a previous AWF run creates /home/runner/.copilot as root (see issue #12066 referenced in the comment).

---

2. AWF binary installation to /usr/local/bin — actions/setup/sh/install_awf_binary.sh:109,134

sudo mv "${TEMP_DIR}/${awf_binary}" "${AWF_INSTALL_DIR}/${AWF_INSTALL_NAME}"   # AWF_INSTALL_DIR=/usr/local/bin

---

3. AWF runtime execution requires root — pkg/constants/constants.go:407

const AWFDefaultCommand = "sudo -E awf"

Used in pkg/workflow/awf_helpers.go:233 as the default AWF command prefix for every agent engine (Copilot, Claude, Codex, Gemini). AWF uses iptables to redirect HTTP/HTTPS traffic through a Squid proxy container for L7 egress control. The AWF documentation states: "AWF establishes an isolated network with a Squid proxy that enforces the workflow network permissions…uses iptables to redirect HTTP/HTTPS traffic." iptables manipulation requires CAP_NET_ADMIN, which is why sudo is needed.

---

4. Post-run firewall log permission fix — pkg/workflow/copilot_installer.go:69

fmt.Sprintf("          sudo chmod -R a+r %s 2>/dev/null || true", firewallLogsDir)

This is a downstream consequence of #3: because AWF runs as root, the firewall logs at /tmp/gh-aw/sandbox/firewall/logs are owned by root, so a subsequent sudo chmod is needed to make them readable for artifact upload.

---

5. macOS-only /opt directory creation — actions/setup/setup.sh:19-21

sudo mkdir -p "$1"
sudo chown -R "$(whoami)" "$1"

This only triggers on Darwin. Linux self-hosted runners are not affected.

---

C. Architectural Dependency: Docker

Beyond sudo, the workflows have a hard dependency on Docker that runners must satisfy:

• MCP Gateway (ghcr.io/github/gh-aw-mcpg) is run via docker run with --network host
• AWF firewall (ghcr.io/github/gh-aw-firewall) is run via sudo -E awf, which itself orchestrates Docker containers
• MCP servers (GitHub MCP server, Serena, etc.) are run as Docker containers
• start_mcp_gateway.sh validates that MCP_GATEWAY_DOCKER_COMMAND starts with docker run

The runner user must have access to the Docker socket. On GitHub-hosted runners this is pre-configured. On self-hosted runners, the user must be in the docker group, or Docker must be accessible via socket with appropriate permissions.

---

Findings: allowPrivilegeEscalation: false on ARC

allowPrivilegeEscalation: false in Kubernetes pod securityContext prevents any process in the container from gaining more privileges than its parent via setuid, setgid, or sudo. This directly blocks all four privilege escalation points listed above.

The key insight: the difficulty is layered — items 1, 2, and 4 are "convenience sudo" (writing to system-owned directories), but item 3 (sudo -E awf) is a deep architectural requirement because iptables manipulation requires a Linux capability (CAP_NET_ADMIN) that cannot be obtained via sudo when allowPrivilegeEscalation: false.

What Would Need to Change

Tier 1 — Easy fixes (convenience sudo only):

Issue	Current code	Proposed change
Copilot CLI install location	INSTALL_DIR="/usr/local/bin" (install_copilot_cli.sh:22)	Change to ${HOME}/.local/bin, ensure it's on PATH
AWF binary install location	AWF_INSTALL_DIR="/usr/local/bin" (install_awf_binary.sh:27)	Change to ${HOME}/.local/bin, ensure it's on PATH
.copilot ownership fix	sudo chown -R runner:runner "$COPILOT_DIR" (install_copilot_cli.sh:32)	The root cause is AWF creating the dir as root (item 3). If AWF no longer runs as root, this step becomes unnecessary.
Firewall log permissions	sudo chmod -R a+r $firewallLogsDir (copilot_installer.go:69)	Downstream effect of AWF running as root; also becomes unnecessary if AWF is run without sudo

The ~/.local/bin approach is standard for rootless tool installation. These scripts would also need to verify ~/.local/bin is in $PATH or prepend it, since GitHub-hosted runners do not have it by default.

Tier 2 — Architectural change (AWF iptables):

sudo -E awf is the core challenge. There are three viable paths:

Path A: Grant CAP_NET_ADMIN at the Kubernetes pod level (no sudo needed)

# In ARC RunnerDeployment or AutoscalingRunnerSet spec
spec:
  template:
    spec:
      containers:
      - name: runner
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            add: ["NET_ADMIN"]   # ← permits iptables without sudo

Then use agent.command: awf (without sudo) in the GH-AW workflow frontmatter. The agent.command override is already supported — GetAWFCommandPrefix in pkg/workflow/awf_helpers.go:225-234 checks for agentConfig.Command first:

if agentConfig != nil && agentConfig.Command != "" {
    return agentConfig.Command   // returns custom command instead of "sudo -E awf"
}

The compiled binary would still need to be installed to a location accessible to the non-root runner user, and AWF itself may internally invoke docker commands that need socket access.

Limitation: NET_ADMIN is still a significant capability; many Kubernetes cluster policies (e.g., OPA Gatekeeper, Kyverno) and PSS "Restricted" profile will reject pods with NET_ADMIN. It may also conflict with the organisation's security posture.

Path B: Disable AWF (no network isolation)

Set agent.command to invoke the engine binary directly (e.g., agent.command: "copilot") or set agent.command: "". This is already supported by the override mechanism above.

Implication: network egress from the AI agent is not firewalled. This is a security trade-off — acceptable in environments where network policies are enforced at the cluster/namespace level (Kubernetes NetworkPolicy or a service mesh).

This path is the most immediately practical for environments that enforce allowPrivilegeEscalation: false and do not grant NET_ADMIN.

Path C: Rootless iptables / eBPF-based firewall (future work)

AWF could be modified (as a separate project, gh-aw-firewall) to support a mode using eBPF-based network control (e.g., Cilium's per-pod egress policy) instead of iptables. This would enable network isolation without CAP_NET_ADMIN. This is architecturally significant work in the AWF project, not in gh-aw itself.

---

Findings: Testing Strategy for Unexpected New Requirements

The question is: how do we ensure that no new privilege requirements are introduced over time without explicit awareness?

Strategy 1 — Static Analysis Gate in CI (Recommended)

Add a job to CI that scans for privilege patterns across all scripts and compiled workflows:

# Check 1: No new sudo calls in scripts
if grep -rn '\bsudo\b' actions/setup/sh/ | grep -v '# ALLOWED:'; then
  echo "ERROR: Unexpected sudo usage found in setup scripts"
  exit 1
fi

# Check 2: Compiled lock.yml workflows must not contain sudo
# (except in known-approved locations)
if grep -rn '\bsudo\b' pkg/cli/workflows/*.lock.yml | \
   grep -v 'sudo chmod -R a+r.*firewall'; then
  echo "ERROR: Unexpected sudo usage in compiled workflows"
  exit 1
fi

By using an explicit allowlist comment (e.g., # ALLOWED: sudo - installs to user dir), approved uses are documented in-code. Unanticipated additions fail the check.

Strategy 2 — Compiled Workflow Diff Testing

When gh aw compile is run (e.g., on recompile Makefile target), the existing golden-file test pattern should assert the compiled output is byte-for-byte identical to the committed .lock.yml. This already appears to be the intent of recompile in the Makefile. Extending this to assert no sudo appears in the output catches cases where a new Go template introduces a privileged step.

Strategy 3 — Run Test Workflow on a Privilege-Restricted Container in CI

Add an integration test job with a container that explicitly blocks privilege escalation:

jobs:
  test-nonroot:
    runs-on: ubuntu-latest
    container:
      image: ubuntu:latest
      options: --security-opt no-new-privileges   # blocks sudo escalation
    steps:
      - name: Test setup scripts run without sudo
        run: bash actions/setup/sh/install_copilot_cli.sh
      - name: Test AWF binary install runs without sudo
        run: bash actions/setup/sh/install_awf_binary.sh v0.13.1

--security-opt no-new-privileges is the Docker equivalent of allowPrivilegeEscalation: false. Any step that calls sudo will fail with a clear error, making the test self-documenting.

Strategy 4 — Dependency Manifest / Capability Declaration

Document all runner capabilities as a machine-readable manifest (e.g., a YAML file in the repo):

# .github/aw/runner-requirements.yml
runner:
  sudo:
    - install_copilot_cli.sh: "writes to /usr/local/bin"
    - install_awf_binary.sh: "writes to /usr/local/bin"
    - awf_default_command: "iptables via CAP_NET_ADMIN"
  docker: required
  linux_capabilities:
    - NET_ADMIN: "required for AWF iptables"

A CI script validates the manifest is complete by diffing it against grep results. This serves as living documentation and a change-detection mechanism simultaneously.

---

Recommendations

1. Immediate (unblocks the reported issue): Modify install_copilot_cli.sh and install_awf_binary.sh to install to ${HOME}/.local/bin instead of /usr/local/bin. Add ~/.local/bin to PATH if not already present. Remove the sudo chown workaround in install_copilot_cli.sh—the root cause (AWF creating dirs as root) will also need addressing.

2. Short term (unblocks allowPrivilegeEscalation: false when AWF is desired): Document that agent.command: awf (without sudo) can be used when the pod is granted CAP_NET_ADMIN at the Kubernetes level. Update the documentation with an ARC configuration example showing the pod spec with capabilities.add: ["NET_ADMIN"] and allowPrivilegeEscalation: false.

3. Short term (unblocks allowPrivilegeEscalation: false without NET_ADMIN): Add a documented option to run without AWF (set agent.command to bypass AWF). Document the security trade-off clearly. This is already architecturally supported—it just needs documentation and possibly a first-class workflow option.

4. Medium term (address log permissions): The sudo chmod -R a+r in copilot_installer.go:69 should be conditioned on whether AWF ran with sudo. If agent.command does not include sudo, the files won't be root-owned and the chmod step is a no-op that still fails if sudo is unavailable.

5. Testing: Add the --security-opt no-new-privileges integration test job to CI and a static analysis grep gate, as described in Strategy 1 and 3 above.

Key Sources

• actions/setup/sh/install_copilot_cli.sh — primary source of reported issue (sudo at lines 32, 119, 120)
• actions/setup/sh/install_awf_binary.sh — sudo at lines 109, 134
• pkg/constants/constants.go:407 — AWFDefaultCommand = "sudo -E awf"
• pkg/workflow/awf_helpers.go:225-234 — GetAWFCommandPrefix() override mechanism
• pkg/workflow/copilot_installer.go:69 — sudo chmod on firewall logs
• AWF Architecture — confirms iptables requirement
• ARC PodSpec customization — Kubernetes security context options

References:

• [§22363017209]

🔭 Intelligence gathered by Scout

[Mossaka]

Unfortunately, AW require sudo because the Agentic Workflow Firewall (AWF) manipulates the host kernel's iptables DOCKER-USER chain to enforce L7 egress filtering on all agent containers. Because this is a deliberate security design requiring root UID, self-hosted runners must permit privilege escalation.

[dsyme]

@dhrapson Thanks for reporting this, I should have said that upfront. I do think this is out of scope for us for where we're at at the moment.