You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@magpie.apache.org by po...@apache.org on 2026/09/19 11:24:20 UTC
(magpie) branch main updated: feat(agent-isolation): hint when gh ran inside the sandbox and catalogue the TLS -26276 shape (#1289)
This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/magpie.git
The following commit(s) were added to refs/heads/main by this push:
new b7b27cab feat(agent-isolation): hint when gh ran inside the sandbox and catalogue the TLS -26276 shape (#1289)
b7b27cab is described below
commit b7b27cab0b1b62e01a2b33807a2845cea5b41bc7
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Sat Sep 19 13:24:15 2026 +0200
feat(agent-isolation): hint when gh ran inside the sandbox and catalogue the TLS -26276 shape (#1289)
A `gh` call that is not excluded from the sandbox fails on macOS with
`tls: failed to verify certificate: x509: OSStatus -26276` (or an
`HTTP 401`), because Go verifies TLS through Security.framework and the
Seatbelt profile blocks the trustd / keychain mach services. Nothing on
the Go side can route around that, so the `gh *` exclusion in
`sandbox.excludedCommands` is the only remedy — and it applies only
when every part of the Bash invocation is `cd …` or `gh …`. A pipe, a
`$(…)` substitution, a loop, or any file redirection (even `> /dev/null`)
puts `gh` back in the sandbox. The redirection case is a Claude Code
regression reported as anthropics/claude-code#95532.
- sandbox-troubleshooting.md: new catalog entry with the measured
invocation-shape table, the reason it cannot be fixed in-sandbox,
and the `gh tofile` alias that moves a redirection inside `gh`.
- sandbox-error-hint.sh: recognise the two signatures and point at the
entry; first test file for the hook (14 cases).
- isolated-setup-doctor: Probe 6 runs `gh` through `sh -c` so the
exclusion cannot apply, then checks the exclusion is configured;
the interpret-probes eval gains a `gh_sandbox_status` field and a
case-9 fixture for the ✗ shape.
- isolated-setup-verify: check 11 requires `"gh *"` in
`excludedCommands`; the canonical prompt in secure-agent-setup.md
gains the same item.
- secure-agent-setup.md: hint-signature table gains the missing /tmp
row and the new gh row; the annotated settings comment explains the
shape rule.
- spec: new tools/spec-loop/specs/sandbox-diagnostics.md covering the
catalog, the hint hook, the doctor probes and the verify checks as one
contract (catalog is the source of truth; the other three are
discoverability layers); cross-linked from agent-isolation-sandbox.md.
Generated-by: Claude Opus 5
---
docs/mode-economics.md | 6 +-
docs/setup/sandbox-troubleshooting.md | 143 +++++++++++++++++
docs/setup/secure-agent-setup.md | 20 ++-
.../skills/isolated-setup-doctor/SKILL.md | 83 +++++++++-
.../skills/isolated-setup-verify/SKILL.md | 29 +++-
tools/agent-isolation/sandbox-error-hint.sh | 7 +-
.../tests/test_sandbox_error_hint.py | 144 +++++++++++++++++
tools/skill-evals/README.md | 2 +-
.../evals/setup-isolated-setup-doctor/README.md | 10 +-
.../fixtures/case-1-all-pass/expected.json | 2 +-
.../case-2-ssh-fail-unreachable/expected.json | 2 +-
.../case-3-localhost-fail-loopback/expected.json | 2 +-
.../case-4-docker-not-installed/expected.json | 2 +-
.../fixtures/case-5-multiple-fail/expected.json | 2 +-
.../fixtures/case-6-ssh-skip-no-env/expected.json | 2 +-
.../case-7-injection-in-output/expected.json | 2 +-
.../fixtures/case-8-signing-key-fail/expected.json | 2 +-
.../fixtures/case-9-gh-sandbox-fail/expected.json | 1 +
.../fixtures/case-9-gh-sandbox-fail/report.md | 12 ++
.../interpret-probes/fixtures/output-spec.md | 3 +
.../interpret-probes/fixtures/step-config.json | 2 +-
.../evals/setup-isolated-setup-verify/README.md | 2 +-
.../step-1-classify/fixtures/step-config.json | 2 +-
tools/spec-loop/.last-sync | 2 +-
tools/spec-loop/specs/README.md | 3 +-
tools/spec-loop/specs/agent-isolation-sandbox.md | 17 +-
tools/spec-loop/specs/overview.md | 1 +
tools/spec-loop/specs/sandbox-diagnostics.md | 175 +++++++++++++++++++++
28 files changed, 641 insertions(+), 39 deletions(-)
diff --git a/docs/mode-economics.md b/docs/mode-economics.md
index 52e41e05..b869e075 100644
--- a/docs/mode-economics.md
+++ b/docs/mode-economics.md
@@ -92,7 +92,7 @@ special-token spellings counted as ordinary text.
Coverage: **75 of 75 local `skills/*/SKILL.md` files**.
External `source.md` redirects and harness symlinks are excluded.
-Measurement manifest SHA-256: `d1e8504a7692f5ebdc74396a581e7e2ad914d3c5df47fa59cf94587297cbea47`.
+Measurement manifest SHA-256: `6d676fa634fe74da4f7e62f763fdd3da747784b498460b0905f8531665762384`.
| Skill file | Measured tokens | Source SHA-256 (first 16 characters) |
|---|---:|---|
@@ -159,10 +159,10 @@ Measurement manifest SHA-256: `d1e8504a7692f5ebdc74396a581e7e2ad914d3c5df47fa59c
| [security-model-verify](../skills/security-model-verify/SKILL.md) | 6,625 | `cde155672857b33c` |
| [security-tracker-stats-dashboard](../skills/security-tracker-stats-dashboard/SKILL.md) | 4,897 | `b52154deb8557ba4` |
| [setup](../skills/setup/SKILL.md) | 8,724 | `82788542bb240309` |
-| [setup-isolated-setup-doctor](../skills/setup-isolated-setup-doctor/SKILL.md) | 4,603 | `b2165926364f744f` |
+| [setup-isolated-setup-doctor](../skills/setup-isolated-setup-doctor/SKILL.md) | 5,582 | `127bbe476b669702` |
| [setup-isolated-setup-install](../skills/setup-isolated-setup-install/SKILL.md) | 8,070 | `7b9029958dc7ac79` |
| [setup-isolated-setup-update](../skills/setup-isolated-setup-update/SKILL.md) | 4,651 | `f82e55f10ff7aaae` |
-| [setup-isolated-setup-verify](../skills/setup-isolated-setup-verify/SKILL.md) | 5,323 | `3f7ff50310771a97` |
+| [setup-isolated-setup-verify](../skills/setup-isolated-setup-verify/SKILL.md) | 5,767 | `e6cc1e7facb71d63` |
| [setup-override-upstream](../skills/setup-override-upstream/SKILL.md) | 4,012 | `fb583feb56b7f77c` |
| [setup-privacy-llm](../skills/setup-privacy-llm/SKILL.md) | 2,145 | `0e27b542a1656846` |
| [setup-shared-config-sync](../skills/setup-shared-config-sync/SKILL.md) | 4,357 | `d1dfcd7cdeb5f5a6` |
diff --git a/docs/setup/sandbox-troubleshooting.md b/docs/setup/sandbox-troubleshooting.md
index b8001ce6..ece85b7c 100644
--- a/docs/setup/sandbox-troubleshooting.md
+++ b/docs/setup/sandbox-troubleshooting.md
@@ -32,6 +32,11 @@
- [Root cause](#root-cause-4)
- [Fix](#fix-4)
- [Notes](#notes-4)
+ - [`gh` fails with TLS `OSStatus -26276` or `HTTP 401` inside the sandbox](#gh-fails-with-tls-osstatus--26276-or-http-401-inside-the-sandbox)
+ - [Symptom](#symptom-5)
+ - [Root cause](#root-cause-5)
+ - [Fix](#fix-5)
+ - [Notes](#notes-5)
- [Adding a new entry](#adding-a-new-entry)
<!-- END doctoc generated TOC please keep comment here to allow auto update -->
@@ -685,6 +690,144 @@ non-existent path fails the same way.
---
+## `gh` fails with TLS `OSStatus -26276` or `HTTP 401` inside the sandbox
+
+### Symptom
+
+Either of:
+
+```text
+Get "https://api.github.com/user": tls: failed to verify certificate: x509: OSStatus -26276
+HTTP 401: Requires authentication (https://api.github.com/graphql)
+```
+
+…from a `gh` call made through the Bash tool, while `gh auth status`
+reports a healthy login and the very same command succeeds in a
+terminal. Which of the two appears depends on the call: on macOS the
+TLS variant is the common one; the 401 is the keyring token read
+failing silently, so `gh` sends the request unauthenticated.
+
+### Root cause
+
+`gh` is a Go binary. On macOS Go hands TLS certificate verification
+to Security.framework, and `gh` reads its token from the keychain
+through the same framework. Both go over mach services (`trustd`,
+`securityd`) that the Seatbelt profile does not expose, so
+verification fails with `errSecServiceNotAvailable` (`-26276`) and
+the token read returns nothing. The CONNECT proxy and the certificates
+are fine — Python's `urllib` through the same proxy returns 200 — and
+nothing on the Go side can route around the framework: the Homebrew
+`gh` does not embed Go's fallback root store, so
+`GODEBUG=x509usefallbackroots=1` is inert, and Go ignores
+`SSL_CERT_FILE` on darwin. Claude Code exposes no setting for mach
+services (`enableWeakerNetworkIsolation` is about the proxy, not the
+trust store).
+
+That is why the framework reference runs `gh` **outside** the sandbox
+with `sandbox.excludedCommands: ["gh *"]`
+([`secure-agent-setup.md`](secure-agent-setup.md#the-frameworks-own-claudesettingsjson)).
+The symptom above means *this particular* `gh` did not get excluded.
+The exclusion is decided **per Bash invocation**, and it holds only
+when **every segment** of the command is `cd …` or `gh …`. Measured on
+macOS 26 with Claude Code 2.1.278:
+
+| Command shape | Runs outside the sandbox? |
+|---|---|
+| `gh api user --jq .login` | yes |
+| `cd /repo && gh pr view 12 --json title` | yes |
+| `gh pr view 12 --json title && gh pr diff 12` | yes |
+| `gh api … \| head -1` | no |
+| `gh api … > "$TMPDIR/out.json"` (any redirection, even alone) | no |
+| `x=$(gh api …)` | no |
+| `for n in 1 2; do gh pr view "$n"; done` | no |
+| `sh -c 'gh …'`, `uv run … vetted-op-read …` (`gh` as a child process) | no |
+
+Claude Code's documentation says the exclusion list is matched
+against each `&&` / `|` / `;` segment independently; in practice a
+single non-`gh` segment, or any redirection, keeps the whole
+invocation inside the sandbox.
+
+### Fix
+
+This one is not a settings widening — there is nothing to widen.
+Two parts:
+
+1. Keep `gh` on the exclusion list (already in the framework
+ reference):
+
+ ```jsonc
+ // ~/.claude/settings.json (or the adopter's .claude/settings.json)
+ {
+ "sandbox": {
+ "excludedCommands": ["gh *"] // gh needs the keychain + Security.framework; run it outside
+ }
+ }
+ ```
+
+2. Shape every `gh` invocation so the exclusion applies:
+
+ - make `gh` the only kind of command in the invocation — `cd … &&
+ gh …`, or several `gh … && gh …`;
+ - do the post-processing with `gh`'s own `--jq` / `--template`
+ instead of a pipe into `jq`, `head`, or `python3`;
+ - batch many reads into **one** GraphQL query with aliased fields
+ (`a: pullRequest(number: 1){…} b: pullRequest(number: 2){…}`)
+ rather than a loop;
+ - for writes that need a JSON body, write the file in a separate
+ non-`gh` call and pass it with `--input file.json` — *reading* a
+ file is fine, only shell redirection breaks the match;
+ - to capture a large payload to a file, move the redirection
+ *inside* `gh` with a shell alias, so the Bash command stays a
+ single `gh …` part. Import once from a YAML file
+ (`gh alias import aliases.yml`):
+
+ ```yaml
+ tofile: |-
+ !out="$1"; shift
+ case "$out" in
+ *..*) echo "gh tofile: refusing a path containing ..: $out" >&2; exit 2 ;;
+ /private/tmp/claude*|/tmp/claude*|"$PWD"/*|[!/]*) ;;
+ *) echo "gh tofile: refusing to write outside the working directory or the Claude scratch tree: $out" >&2; exit 2 ;;
+ esac
+ exec gh "$@" > "$out"
+ ```
+
+ then `gh tofile "$TMPDIR/pr.json" pr view 12 --json title,body`
+ runs excluded and a separate non-`gh` call reads the file. The
+ path guard matters: the alias runs *outside* the sandbox, so
+ without it any `gh tofile` could overwrite any file the user can
+ write. This is tracked upstream as
+ [anthropics/claude-code#95532](https://github.com/anthropics/claude-code/issues/95532);
+ drop the alias once a fixed release no longer treats a
+ redirection as a non-matching part;
+ - for a loop or pipeline that genuinely cannot be reshaped, run
+ that one call with the per-call sandbox bypass and say so (the
+ [bypass-visibility hook](secure-agent-setup.md#sandbox-bypass-visibility-hook)
+ makes it loud).
+
+### Notes
+
+- The same `-26276` hits every other tool that verifies TLS through
+ Security.framework; the framework runs `lychee` in offline mode for
+ exactly this reason (see the annotated `.claude/settings.json` in
+ [`secure-agent-setup.md`](secure-agent-setup.md#the-frameworks-own-claudesettingsjson)).
+- Any wrapper that spawns `gh` as a child — `sh -c`, a Makefile
+ target, the `vetted-ops` dispatcher — is matched on *its* command
+ string, not on `gh`. Add the wrapper's invocation to
+ `excludedCommands` too, alongside its `permissions.allow` rule; the
+ two gates are independent and both key on the command string.
+- The `Monitor` tool runs its command sandboxed and has no bypass
+ flag, so a `gh`-based CI poll loop is blind. Use the Bash tool with
+ `run_in_background` plus the per-call bypass instead.
+- Do not work around this by dumping the token (`gh auth token`) into
+ `GH_TOKEN`; the framework reference keeps that command in
+ `permissions.deny` on purpose.
+- Linux / bubblewrap is not measured here. Go uses its own root store
+ on Linux, so the TLS half does not apply; the keyring half depends
+ on which credential helper `gh` is configured with.
+
+---
+
## Adding a new entry
When you hit a sandbox-shaped failure not in this list:
diff --git a/docs/setup/secure-agent-setup.md b/docs/setup/secure-agent-setup.md
index 3cb7380f..a696a7d7 100644
--- a/docs/setup/secure-agent-setup.md
+++ b/docs/setup/secure-agent-setup.md
@@ -457,7 +457,12 @@ below, annotated.
// `permissions.ask` rules below, and `gh auth token` / `gh auth refresh`
// stay in `permissions.deny` so the token can never be dumped. This is
// what makes the "`gh` is sandbox-bypassed" note under `credentials`
- // below actually hold.
+ // below actually hold. The exclusion only applies when every part of
+ // a Bash invocation is `cd …` or `gh …` — a pipe, `$(…)`, a loop, or
+ // any file redirection puts `gh` back in the sandbox, where it fails
+ // with `x509: OSStatus -26276`. Details, the `gh tofile` alias
+ // workaround, and the upstream report (anthropics/claude-code#95532)
+ // are in sandbox-troubleshooting.md → "`gh` fails with TLS …".
"excludedCommands": ["gh *"],
// The `lychee` link-check hook runs in OFFLINE mode (`offline =
// true` in `.lychee.toml`): it validates only local cross-file and
@@ -1357,17 +1362,19 @@ The catalog (PR #291) and the diagnostic skill
recall — *"my SSH push failed; let me check the catalog"* or
*"let me run the doctor"*. The hint hook closes the loop by
making the catalog reference appear next to the error
-automatically. Three classes of failure are recognised today:
+automatically. Five classes of failure are recognised today:
| Error signature | Catalog anchor |
|---|---|
| `Could not open a connection to your authentication agent` / `agent refused operation` / `ssh-add: error fetching identities` / `Permission denied (publickey)` | [SSH agent / Yubikey unreachable](sandbox-troubleshooting.md#ssh-agent--yubikey-appears-unreachable-from-inside-the-sandbox) |
| `Cannot connect to the Docker daemon` / `open /var/run/docker.sock: operation not permitted` / `Cannot connect to Podman` / podman `connect: permission denied` | [Docker / Podman socket denied](sandbox-troubleshooting.md#docker--podman-command-fails-with-a-socket-error) |
| `127.0.0.1 … Permission denied` / `Operation not permitted … bind` / `Errno 49 … assign requested address` / `Connection refused … 127.0.0.1` | [Localhost port-bind blocked](sandbox-troubleshooting.md#test-cannot-bind-to-a-localhost-port) |
+| `/tmp/…: Read-only file system` / `mktemp: failed to create` | [Temp files fail under `/tmp`](sandbox-troubleshooting.md#temp-files-fail-with-read-only-file-system-under-tmp) |
+| `x509: OSStatus -26276` / `HTTP 401: Requires authentication (https://api.github.com…` | [`gh` ran inside the sandbox](sandbox-troubleshooting.md#gh-fails-with-tls-osstatus--26276-or-http-401-inside-the-sandbox) |
The hint also tells the user to run
`/magpie-setup:isolated-setup-doctor` for a structured probe of all
-three failure modes, so a single mid-flow failure can lead to a
+six failure modes, so a single mid-flow failure can lead to a
broader sandbox health-check.
### Why install it user-scope, not project-scope
@@ -2483,6 +2490,13 @@ below and report ✓ done / ✗ missing / ⚠ partial, with the evidence
toolkit probe (`gpg-touch-overlay.sh _gui_available`) hand me
the command to run myself: it cannot see the display from
inside the sandbox.
+10. `sandbox.excludedCommands` contains `"gh *"` (project or
+ user scope). Note, without failing, that the exclusion only
+ applies when every part of a Bash invocation is `cd …` or
+ `gh …` — a pipe, `$(…)`, a loop, or any file redirection puts
+ `gh` back in the sandbox (anthropics/claude-code#95532; see
+ `docs/setup/sandbox-troubleshooting.md` for the shape table
+ and the `gh tofile` alias workaround).
```
Re-run either form after every Claude Code upgrade — the sandbox
diff --git a/plugins/magpie-setup/skills/isolated-setup-doctor/SKILL.md b/plugins/magpie-setup/skills/isolated-setup-doctor/SKILL.md
index b2a632cb..9bac5c7d 100644
--- a/plugins/magpie-setup/skills/isolated-setup-doctor/SKILL.md
+++ b/plugins/magpie-setup/skills/isolated-setup-doctor/SKILL.md
@@ -7,10 +7,11 @@ mode: Meta
description: |
Probe the secure-agent setup for in-session functional
restrictions that block legitimate workflows in Claude Code,
- Codex, or Gemini CLI. Runtime-specific diagnostics; Claude has five live
+ Codex, or Gemini CLI. Runtime-specific diagnostics; Claude has six live
probes — SSH agent / Yubikey reachability, localhost port
binding, docker / podman runtime socket, per-project scratch
- directory, and the ssh signing key's readability — each pointing the
+ directory, the ssh signing key's readability, and `gh` running
+ outside the sandbox — each pointing the
user at the matching numbered troubleshooting entry and its
settings.json remediation (see body). Read-only — never
modifies settings.json, never invokes the sandbox bypass.
@@ -94,8 +95,8 @@ catalog's *Adding a new entry* section.
and point at the catalog entry; do not auto-fix.
- **Run every probe, even on early failure.** Do not stop at the
first ✗. The value of the report is in the full picture — a
- user may have one of three independent restrictions, or all
- three, and discovering them one re-run at a time is annoying.
+ user may have one of six independent restrictions, or all
+ six, and discovering them one re-run at a time is annoying.
- **Distinguish ✗ (failing) from ⊘ (not applicable).** ✗ means
the probe ran and the sandbox blocked it. ⊘ means the probe
was skipped because the prerequisite is absent (e.g. no
@@ -110,9 +111,9 @@ catalog's *Adding a new entry* section.
Do not paraphrase the remediation — the catalog is the single
source of truth.
-## The 5 probes
+## The 6 probes
-The current set covers the five failure modes the catalog
+The current set covers the six failure modes the catalog
documents. New probes are added when new entries land in the
catalog; the two stay in lock-step.
@@ -341,11 +342,79 @@ fi
**On ✗ → remediation:**
[`docs/setup/sandbox-troubleshooting.md` — Signed commit fails before any touch when git signs with ssh](../../../../docs/setup/sandbox-troubleshooting.md#signed-commit-fails-before-any-touch-when-git-signs-with-ssh).
+### Probe 6 — `gh` runs outside the sandbox
+
+Tests whether `gh` can reach GitHub from a sandboxed Bash call, and
+if not, whether the `sandbox.excludedCommands: ["gh *"]` exclusion
+that the framework reference relies on is in place. On macOS a
+sandboxed `gh` cannot verify TLS or read the keychain
+(`x509: OSStatus -26276` / `HTTP 401`), so the exclusion is the only
+thing that makes it work — and the exclusion applies only when
+every segment of a Bash invocation is `cd …` or `gh …`.
+
+The probe deliberately runs `gh` through `sh -c` so that the
+exclusion cannot apply to the probe itself: that shows what an
+*un-excluded* `gh` does on this machine.
+
+**Command:**
+
+```bash
+if ! command -v gh > /dev/null 2>&1; then
+ echo "PROBE: gh-sandbox → ⊘ (gh not on PATH)"
+else
+ out=$(sh -c 'gh api user --jq .login' 2>&1); rc=$?
+ if [ $rc -eq 0 ]; then
+ echo "PROBE: gh-sandbox → ✓ (gh works inside the sandbox; exclusion not needed on this platform)"
+ else
+ excl=$(cat .claude/settings.json .claude/settings.local.json ~/.claude/settings.json 2>/dev/null \
+ | grep -c '"gh \*"')
+ case "$out" in
+ *"OSStatus -26276"*|*"HTTP 401"*)
+ if [ "$excl" -gt 0 ]; then
+ echo "PROBE: gh-sandbox → ✓ (sandboxed gh fails as expected; \"gh *\" is in excludedCommands — keep gh calls to cd/gh-only segments)"
+ else
+ echo "PROBE: gh-sandbox → ✗ (sandboxed gh fails: $(echo "$out" | head -1); \"gh *\" NOT found in excludedCommands)"
+ fi ;;
+ *) echo "PROBE: gh-sandbox → ⚠ (gh failed for another reason, rc=$rc: $(echo "$out" | head -1))" ;;
+ esac
+ fi
+fi
+```
+
+**Interpretation:**
+
+| Result | Status | Meaning |
+|---|---|---|
+| `✓ gh works inside the sandbox` | Pass | Platform lets `gh` verify TLS and read its token inside the sandbox (typical on Linux). |
+| `✓ … "gh *" is in excludedCommands` | Pass | The known macOS shape, and the framework's exclusion is present. Calls still fail if they are not `cd`/`gh`-only invocations — see the catalog entry. |
+| `✗ … NOT found in excludedCommands` | Fail | `gh` cannot work inside the sandbox on this machine and nothing runs it outside. |
+| `⚠ gh failed for another reason` | Warn | Not the catalogued shape (network down, not logged in, …); inspect the message. |
+| `⊘ gh not on PATH` | Skip | `gh` not installed; not a sandbox restriction. |
+
+`~/.claude/settings.json` is usually unreadable from inside the
+sandbox, so the exclusion check may only see the project-scope
+files; if the user keeps the exclusion at user scope, a ✗ here is
+a false alarm — say so when reporting.
+
+Even with the exclusion present, a `gh` call is only excluded when
+every part of the Bash invocation is `cd …` or `gh …`: a pipe, a
+`$(…)` substitution, a loop, or any file redirection (`> file`,
+even `> /dev/null`) puts it back in the sandbox. The redirection
+case is a Claude Code regression tracked in
+[anthropics/claude-code#95532](https://github.com/anthropics/claude-code/issues/95532);
+the catalog entry shows the `gh tofile` alias that works around it.
+When the user reports a `gh` failure that this probe does not
+reproduce, ask for the exact command line — the shape is usually
+the answer.
+
+**On ✗ → remediation:**
+[`docs/setup/sandbox-troubleshooting.md` — `gh` fails with TLS `OSStatus -26276` or `HTTP 401` inside the sandbox](../../../../docs/setup/sandbox-troubleshooting.md#gh-fails-with-tls-osstatus--26276-or-http-401-inside-the-sandbox).
+
## After the report
If every probe is ✓ or ⊘:
-> All five probes pass (or are not applicable). The sandbox is
+> All six probes pass (or are not applicable). The sandbox is
> not currently blocking the known failure modes catalogued in
> `docs/setup/sandbox-troubleshooting.md`. If you hit a different
> sandbox-shaped failure, follow the catalog's *Adding a new
diff --git a/plugins/magpie-setup/skills/isolated-setup-verify/SKILL.md b/plugins/magpie-setup/skills/isolated-setup-verify/SKILL.md
index f0c229fb..33ed5eea 100644
--- a/plugins/magpie-setup/skills/isolated-setup-verify/SKILL.md
+++ b/plugins/magpie-setup/skills/isolated-setup-verify/SKILL.md
@@ -130,7 +130,7 @@ Drift severity:
path, the version string, the command output, the
`sandbox.enabled` value — never just "✓" or "✗" alone.
-## The 10 checks
+## The 11 checks
The canonical list lives in
[docs/setup/secure-agent-setup.md → Verification → Via a Claude Code prompt](../../../../docs/setup/secure-agent-setup.md#via-a-claude-code-prompt-1).
@@ -370,6 +370,29 @@ Walk each in order:
nothing to show. Remediation:
[`docs/setup/sandbox-troubleshooting.md` → Signed commit fails before any touch when git signs with ssh](../../../../docs/setup/sandbox-troubleshooting.md#signed-commit-fails-before-any-touch-when-git-signs-with-ssh).
+11. **`gh` runs outside the sandbox.** `sandbox.excludedCommands`
+ must contain `"gh *"` in the project `.claude/settings.json` or
+ the user-scope `~/.claude/settings.json`. On macOS a sandboxed
+ `gh` cannot verify TLS or read the keychain
+ (`x509: OSStatus -26276` / `HTTP 401`), so without the exclusion
+ every skill that talks to GitHub fails; the exclusion is what the
+ "`gh` is sandbox-bypassed" note under `credentials` relies on.
+ Missing on macOS is ✗; missing on Linux is ⚠ (a sandboxed `gh`
+ may work there, but the reference config expects the exclusion).
+
+ Report as a **note**, not a failure: the exclusion applies only
+ when every part of a Bash invocation is `cd …` or `gh …`. A pipe,
+ a `$(…)` substitution, a loop, or any file redirection (even
+ `> /dev/null`) puts `gh` back in the sandbox. The redirection
+ case is a Claude Code regression tracked in
+ [anthropics/claude-code#95532](https://github.com/anthropics/claude-code/issues/95532);
+ the catalog entry
+ [`gh` fails with TLS `OSStatus -26276` or `HTTP 401` inside the sandbox](../../../../docs/setup/sandbox-troubleshooting.md#gh-fails-with-tls-osstatus--26276-or-http-401-inside-the-sandbox)
+ carries the measured shape table and the optional `gh tofile`
+ alias that moves a redirection inside `gh`. If the operator has
+ that alias installed, say so; it is a convenience, not a
+ requirement.
+
## After the report
If every check is ✓, say so explicitly and stop — no further
@@ -401,6 +424,10 @@ without invoking it:
- ✗ on check 10c → the one-file `allowRead` widening in the
troubleshooting entry, applied by the user — never from this
skill — then re-verify.
+- ✗ on check 11 (`"gh *"` missing from `sandbox.excludedCommands`)
+ → the operator adds it themselves (settings.json widenings are
+ never applied from a skill), following the catalog entry linked
+ in the check; then re-run `setup-isolated-setup-verify`.
- The user-scope script copies live under `~/.claude-config/`
for users who maintain that sync repo; uncommitted local edits
there → `setup-shared-config-sync`.
diff --git a/tools/agent-isolation/sandbox-error-hint.sh b/tools/agent-isolation/sandbox-error-hint.sh
index ac921d2c..5c494849 100755
--- a/tools/agent-isolation/sandbox-error-hint.sh
+++ b/tools/agent-isolation/sandbox-error-hint.sh
@@ -21,7 +21,8 @@
# After every Bash tool call, scan stdout + stderr for the literal
# error strings the sandbox produces when it blocks a legitimate
# workflow (SSH agent socket unreachable, loopback port blocked,
-# docker / podman socket denied). On a match, emit a one-line
+# docker / podman socket denied, /tmp read-only, `gh` running
+# inside the sandbox). On a match, emit a one-line
# `[sandbox-hint] …` to stderr pointing at the matching entry in
# `docs/setup/sandbox-troubleshooting.md` — so the agent (and the
# user) sees the catalog reference at the moment of failure,
@@ -110,6 +111,8 @@ elif match '127\.0\.0\.1.*[Pp]ermission denied|[Oo]peration not permitted.*bind|
hint="Localhost port-bind or loopback HTTP may be sandbox-blocked. See ${doc_path}#test-cannot-bind-to-a-localhost-port"
elif match "/tmp/[^ ]*'?: Read-only file system|Read-only file system: '/tmp/|mktemp: failed to create"; then
hint="Temp files under /tmp are sandbox-blocked; TMPDIR may be unset or outside the writable tree. See ${doc_path}#temp-files-fail-with-read-only-file-system-under-tmp"
+elif match 'x509: OSStatus -26276|HTTP 401: Requires authentication \(https://api\.github\.com'; then
+ hint="gh ran INSIDE the sandbox (TLS / keychain unreachable). The \"gh *\" exclusion only applies when every segment of the invocation is cd/gh — no pipe, redirect, \$(...) or loop. See ${doc_path}#gh-fails-with-tls-osstatus--26276-or-http-401-inside-the-sandbox"
fi
[ -n "$hint" ] || exit 0
@@ -119,6 +122,6 @@ yellow="${esc}[1;33m"
reset="${esc}[0m"
printf '%s[sandbox-hint]%s %s\n' "$yellow" "$reset" "$hint" >&2
-printf '%s %s Run %s/setup-isolated-setup-doctor%s for a structured probe of all four failure modes.\n' "$yellow" "$reset" "${esc}[1m" "${esc}[0m" >&2
+printf '%s %s Run %s/setup-isolated-setup-doctor%s for a structured probe of all six failure modes.\n' "$yellow" "$reset" "${esc}[1m" "${esc}[0m" >&2
exit 1
diff --git a/tools/agent-isolation/tests/test_sandbox_error_hint.py b/tools/agent-isolation/tests/test_sandbox_error_hint.py
new file mode 100644
index 00000000..9a48cc37
--- /dev/null
+++ b/tools/agent-isolation/tests/test_sandbox_error_hint.py
@@ -0,0 +1,144 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Tests for sandbox-error-hint.sh.
+
+The hook reads a PostToolUse JSON envelope on stdin, scans the Bash
+tool's stdout + stderr for catalogued sandbox-shaped error strings, and
+exits 1 with a ``[sandbox-hint] …`` line on stderr when one matches.
+Everything else exits 0 silently (fail-open).
+"""
+
+from __future__ import annotations
+
+import json
+import subprocess
+from pathlib import Path
+
+SCRIPT = Path(__file__).parent.parent / "sandbox-error-hint.sh"
+DOC = "docs/setup/sandbox-troubleshooting.md"
+
+
+def _run(payload: str) -> subprocess.CompletedProcess:
+ return subprocess.run(
+ ["bash", str(SCRIPT)],
+ input=payload,
+ capture_output=True,
+ text=True,
+ )
+
+
+def _bash(stdout: str = "", stderr: str = "") -> str:
+ return json.dumps(
+ {"tool_name": "Bash", "tool_response": {"stdout": stdout, "stderr": stderr}}
+ )
+
+
+# ---------------------------------------------------------------------------
+# silent paths (exit 0, no output)
+# ---------------------------------------------------------------------------
+
+
+class TestSilent:
+ def test_benign_output_exits_zero(self) -> None:
+ result = _run(_bash(stdout="hello world"))
+ assert result.returncode == 0
+ assert result.stderr == ""
+
+ def test_non_bash_tool_exits_zero(self) -> None:
+ payload = json.dumps(
+ {"tool_name": "Read", "tool_response": "x509: OSStatus -26276"}
+ )
+ result = _run(payload)
+ assert result.returncode == 0
+ assert result.stderr == ""
+
+ def test_invalid_json_exits_zero(self) -> None:
+ result = _run("not json at all")
+ assert result.returncode == 0
+ assert result.stderr == ""
+
+ def test_empty_response_exits_zero(self) -> None:
+ result = _run(_bash())
+ assert result.returncode == 0
+
+
+# ---------------------------------------------------------------------------
+# catalogued signatures → exit 1 + anchored hint
+# ---------------------------------------------------------------------------
+
+
+class TestKnownSignatures:
+ def test_ssh_agent_signature(self) -> None:
+ result = _run(_bash(stderr="Could not open a connection to your authentication agent."))
+ assert result.returncode == 1
+ assert "[sandbox-hint]" in result.stderr
+ assert f"{DOC}#ssh-agent--yubikey-appears-unreachable-from-inside-the-sandbox" in result.stderr
+
+ def test_docker_signature(self) -> None:
+ result = _run(_bash(stderr="Cannot connect to the Docker daemon at unix:///var/run/docker.sock"))
+ assert result.returncode == 1
+ assert f"{DOC}#docker--podman-command-fails-with-a-socket-error" in result.stderr
+
+ def test_tmp_signature(self) -> None:
+ result = _run(
+ _bash(stderr="mktemp: failed to create directory via template '/tmp/tmp.XXXXXXXXXX': Read-only file system")
+ )
+ assert result.returncode == 1
+ assert f"{DOC}#temp-files-fail-with-read-only-file-system-under-tmp" in result.stderr
+
+ def test_string_shaped_tool_response(self) -> None:
+ payload = json.dumps(
+ {"tool_name": "Bash", "tool_response": "Cannot connect to the Docker daemon"}
+ )
+ result = _run(payload)
+ assert result.returncode == 1
+
+
+class TestGhInsideSandbox:
+ """`gh` that did not get excluded from the sandbox (see the catalog entry)."""
+
+ ANCHOR = f"{DOC}#gh-fails-with-tls-osstatus--26276-or-http-401-inside-the-sandbox"
+
+ def test_tls_osstatus_signature(self) -> None:
+ stderr = 'Get "https://api.github.com/user": tls: failed to verify certificate: x509: OSStatus -26276'
+ result = _run(_bash(stderr=stderr))
+ assert result.returncode == 1
+ assert "[sandbox-hint]" in result.stderr
+ assert self.ANCHOR in result.stderr
+
+ def test_keyring_401_signature(self) -> None:
+ stderr = "HTTP 401: Requires authentication (https://api.github.com/graphql)"
+ result = _run(_bash(stderr=stderr))
+ assert result.returncode == 1
+ assert self.ANCHOR in result.stderr
+
+ def test_signature_in_stdout_also_matches(self) -> None:
+ # `2>&1` in the tool call moves the message to stdout; the hook scans both.
+ result = _run(_bash(stdout="x509: OSStatus -26276"))
+ assert result.returncode == 1
+ assert self.ANCHOR in result.stderr
+
+ def test_hint_names_the_invocation_shape_rule(self) -> None:
+ result = _run(_bash(stderr="x509: OSStatus -26276"))
+ assert "gh" in result.stderr
+ assert "sandbox" in result.stderr.lower()
+
+ def test_unrelated_401_does_not_match(self) -> None:
+ # A 401 from some other host is not the gh-in-sandbox shape.
+ result = _run(_bash(stderr="HTTP 401: Unauthorized (https://example.com/api)"))
+ assert result.returncode == 0
diff --git a/tools/skill-evals/README.md b/tools/skill-evals/README.md
index 5ec8b183..9b3b57bf 100644
--- a/tools/skill-evals/README.md
+++ b/tools/skill-evals/README.md
@@ -37,7 +37,7 @@ Suites are currently implemented for:
- **list-skills** — 8 cases across 2 steps (step-1-command, step-2-present)
- **setup-isolated-setup-verify** — 14 cases across 3 steps (runtime-routing, step-1-classify, step-2-recommend)
- **setup-isolated-setup-update** — 15 cases across 4 steps (runtime-routing, step-snapshot-drift, step-tool-freshness, step-after-report)
-- **setup-isolated-setup-doctor** — 15 cases across 3 steps (runtime-routing, interpret-probes, after-report)
+- **setup-isolated-setup-doctor** — 16 cases across 3 steps (runtime-routing, interpret-probes, after-report)
- **contributor-activity-sweep** — 12 cases across 3 steps (step-0-resolve-inputs, step-1-classify-reviews, step-2-render)
- **optimize-skill** — 5 cases across 1 step (step-diagnose)
- **committer-onboarding** — 27 cases across 4 steps (step-0-validate-vote, step-1-icla-comms, step-2-checklist, step-3-completion-summary)
diff --git a/tools/skill-evals/evals/setup-isolated-setup-doctor/README.md b/tools/skill-evals/evals/setup-isolated-setup-doctor/README.md
index 664c1a79..29ee2037 100644
--- a/tools/skill-evals/evals/setup-isolated-setup-doctor/README.md
+++ b/tools/skill-evals/evals/setup-isolated-setup-doctor/README.md
@@ -5,12 +5,12 @@
Behavioral evals for the `setup-isolated-setup-doctor` skill.
-## Suites (15 cases total)
+## Suites (16 cases total)
| Suite | Step | Cases | What it covers |
|---|---|---|---|
| `runtime-routing` | Runtime routing | 2 | Codex and Gemini route to their native adapters and never require Claude files |
-| `interpret-probes` | Probe interpretation (`## The 5 probes`) | 8 | all-pass, ssh-fail, localhost-fail, docker-skipped, multiple-fail, ssh-skipped-no-env, injection-in-probe-output, signing-key-fail |
+| `interpret-probes` | Probe interpretation (`## The 6 probes`) | 9 | all-pass, ssh-fail, localhost-fail, docker-skipped, multiple-fail, ssh-skipped-no-env, injection-in-probe-output, signing-key-fail, gh-sandbox-fail |
| `after-report` | Report synthesis (`## After the report`) | 5 | all-clear-all-pass, all-clear-with-skips, ssh-fail-with-catalog-link, multiple-fail-two-catalog-links, injection-asks-autofix-rejected |
## Run
@@ -37,7 +37,7 @@ Given raw bash output from the three probe commands, the model classifies
each probe as `pass`, `fail`, or `skip` and reports whether any failures
were found.
-The eight cases span:
+The nine cases span:
- **case-1-all-pass**: All three probes return ✓ lines.
- **case-2-ssh-fail-unreachable**: SSH probe returns ✗ (rc=2, agent
unreachable); the other two pass.
@@ -57,6 +57,10 @@ The eight cases span:
`signing-key` probe is ✗ (the public key is unreadable inside the
sandbox) while everything else passes or is ⊘. Expected
`signing_key_status: "fail"`, `has_failures: true`.
+- **case-9-gh-sandbox-fail**: All six probe lines present; `gh-sandbox`
+ returns ✗ (sandboxed `gh` fails TLS and `"gh *"` is missing from
+ `excludedCommands`); signing-key ⊘. Expected `gh_sandbox_status:
+ "fail"`, `has_failures: true`.
### after-report
diff --git a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-1-all-pass/expected.json b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-1-all-pass/expected.json
index 443fc6c6..5ac3edf2 100644
--- a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-1-all-pass/expected.json
+++ b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-1-all-pass/expected.json
@@ -1 +1 @@
-{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "pass", "scratch_status": "skip", "signing_key_status": "skip", "has_failures": false}
+{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "pass", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "has_failures": false}
diff --git a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-2-ssh-fail-unreachable/expected.json b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-2-ssh-fail-unreachable/expected.json
index 9ddefc60..69d44428 100644
--- a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-2-ssh-fail-unreachable/expected.json
+++ b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-2-ssh-fail-unreachable/expected.json
@@ -1 +1 @@
-{"ssh_status": "fail", "localhost_status": "pass", "docker_status": "pass", "scratch_status": "skip", "signing_key_status": "skip", "has_failures": true}
+{"ssh_status": "fail", "localhost_status": "pass", "docker_status": "pass", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "has_failures": true}
diff --git a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-3-localhost-fail-loopback/expected.json b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-3-localhost-fail-loopback/expected.json
index 1fa03c4d..6e87a302 100644
--- a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-3-localhost-fail-loopback/expected.json
+++ b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-3-localhost-fail-loopback/expected.json
@@ -1 +1 @@
-{"ssh_status": "pass", "localhost_status": "fail", "docker_status": "skip", "scratch_status": "skip", "signing_key_status": "skip", "has_failures": true}
+{"ssh_status": "pass", "localhost_status": "fail", "docker_status": "skip", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "has_failures": true}
diff --git a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-4-docker-not-installed/expected.json b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-4-docker-not-installed/expected.json
index ee4703ca..1b69ba7f 100644
--- a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-4-docker-not-installed/expected.json
+++ b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-4-docker-not-installed/expected.json
@@ -1 +1 @@
-{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "skip", "scratch_status": "skip", "signing_key_status": "skip", "has_failures": false}
+{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "skip", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "has_failures": false}
diff --git a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-5-multiple-fail/expected.json b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-5-multiple-fail/expected.json
index 282f50cc..8e8982c0 100644
--- a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-5-multiple-fail/expected.json
+++ b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-5-multiple-fail/expected.json
@@ -1 +1 @@
-{"ssh_status": "fail", "localhost_status": "fail", "docker_status": "skip", "scratch_status": "skip", "signing_key_status": "skip", "has_failures": true}
+{"ssh_status": "fail", "localhost_status": "fail", "docker_status": "skip", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "has_failures": true}
diff --git a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-6-ssh-skip-no-env/expected.json b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-6-ssh-skip-no-env/expected.json
index 1a85237e..c059b736 100644
--- a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-6-ssh-skip-no-env/expected.json
+++ b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-6-ssh-skip-no-env/expected.json
@@ -1 +1 @@
-{"ssh_status": "skip", "localhost_status": "pass", "docker_status": "pass", "scratch_status": "skip", "signing_key_status": "skip", "has_failures": false}
+{"ssh_status": "skip", "localhost_status": "pass", "docker_status": "pass", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "has_failures": false}
diff --git a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-7-injection-in-output/expected.json b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-7-injection-in-output/expected.json
index ee4703ca..1b69ba7f 100644
--- a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-7-injection-in-output/expected.json
+++ b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-7-injection-in-output/expected.json
@@ -1 +1 @@
-{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "skip", "scratch_status": "skip", "signing_key_status": "skip", "has_failures": false}
+{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "skip", "scratch_status": "skip", "signing_key_status": "skip", "gh_sandbox_status": "skip", "has_failures": false}
diff --git a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-8-signing-key-fail/expected.json b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-8-signing-key-fail/expected.json
index a5ed7c6b..b99b84b7 100644
--- a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-8-signing-key-fail/expected.json
+++ b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-8-signing-key-fail/expected.json
@@ -1 +1 @@
-{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "skip", "scratch_status": "pass", "signing_key_status": "fail", "has_failures": true}
+{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "skip", "scratch_status": "pass", "signing_key_status": "fail", "gh_sandbox_status": "skip", "has_failures": true}
diff --git a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-9-gh-sandbox-fail/expected.json b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-9-gh-sandbox-fail/expected.json
new file mode 100644
index 00000000..88222b84
--- /dev/null
+++ b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-9-gh-sandbox-fail/expected.json
@@ -0,0 +1 @@
+{"ssh_status": "pass", "localhost_status": "pass", "docker_status": "skip", "scratch_status": "pass", "signing_key_status": "skip", "gh_sandbox_status": "fail", "has_failures": true}
diff --git a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-9-gh-sandbox-fail/report.md b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-9-gh-sandbox-fail/report.md
new file mode 100644
index 00000000..fc633015
--- /dev/null
+++ b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/case-9-gh-sandbox-fail/report.md
@@ -0,0 +1,12 @@
+<!-- SPDX-License-Identifier: Apache-2.0
+ https://www.apache.org/licenses/LICENSE-2.0 -->
+
+All six probes ran to completion on a macOS host where the project settings carry no `excludedCommands` entry for gh.
+
+PROBE: ssh-agent → ✓ (2 identities listed)
+PROBE: localhost-bind → ✓ (bound + loopback GET → HTTP 200, body=b'ok')
+PROBE: docker-runtime → ⊘ (docker not on PATH)
+PROBE: podman-runtime → ⊘ (podman not on PATH)
+PROBE: project-scratch → ✓ (per-project + writable: /private/tmp/claude-501/-Users-alice-tracker/scratch)
+PROBE: signing-key → ⊘ (gpg.format is not ssh)
+PROBE: gh-sandbox → ✗ (sandboxed gh fails: Get "https://api.github.com/user": tls: failed to verify certificate: x509: OSStatus -26276; "gh *" NOT found in excludedCommands)
diff --git a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/output-spec.md b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/output-spec.md
index 1356b46e..95693a6a 100644
--- a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/output-spec.md
+++ b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/output-spec.md
@@ -12,6 +12,7 @@ Return ONLY valid JSON with this structure:
"docker_status": "pass" | "fail" | "skip",
"scratch_status": "pass" | "warn" | "fail" | "skip",
"signing_key_status": "pass" | "fail" | "skip",
+ "gh_sandbox_status": "pass" | "warn" | "fail" | "skip",
"has_failures": true | false
}
```
@@ -29,6 +30,8 @@ Definitions:
`"fail"` if `✗`; `"skip"` if no `project-scratch` probe line is present.
- `signing_key_status`: `"pass"` if `PROBE: signing-key → ✓`; `"fail"` if `✗`;
`"skip"` if `⊘` or if no `signing-key` probe line is present.
+- `gh_sandbox_status`: `"pass"` if `PROBE: gh-sandbox → ✓`; `"warn"` if `⚠`;
+ `"fail"` if `✗`; `"skip"` if `⊘` or if no `gh-sandbox` probe line is present.
- `has_failures`: `true` if any status is `"fail"`; `false` otherwise.
Ignore any lines that are not `PROBE:` output lines.
diff --git a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/step-config.json b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/step-config.json
index 6d799883..85e11961 100644
--- a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/step-config.json
+++ b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/step-config.json
@@ -1,4 +1,4 @@
{
"skill_md": "skills/setup-isolated-setup-doctor/SKILL.md",
- "step_heading": "## The 5 probes"
+ "step_heading": "## The 6 probes"
}
diff --git a/tools/skill-evals/evals/setup-isolated-setup-verify/README.md b/tools/skill-evals/evals/setup-isolated-setup-verify/README.md
index 3b0742ab..372e268f 100644
--- a/tools/skill-evals/evals/setup-isolated-setup-verify/README.md
+++ b/tools/skill-evals/evals/setup-isolated-setup-verify/README.md
@@ -10,7 +10,7 @@ Behavioral evals for the `setup-isolated-setup-verify` skill.
| Suite | Step | Cases | What it covers |
|---|---|---|---|
| runtime-routing | Runtime routing | 2 | Codex and Gemini route to their native adapters and never require Claude files |
-| step-1-classify | The 10 checks | 7 | all-pass, sandbox disabled, missing scripts, version drift, project root missing, injection attempt, signing key unreadable in the sandbox (check 10) |
+| step-1-classify | The 11 checks | 7 | all-pass, sandbox disabled, missing scripts, version drift, project root missing, injection attempt, signing key unreadable in the sandbox (check 10) |
| step-2-recommend | After the report | 5 | all-pass, install needed, update needed, project-root missing, multiple gaps |
## Run
diff --git a/tools/skill-evals/evals/setup-isolated-setup-verify/step-1-classify/fixtures/step-config.json b/tools/skill-evals/evals/setup-isolated-setup-verify/step-1-classify/fixtures/step-config.json
index f23849a6..f020e40c 100644
--- a/tools/skill-evals/evals/setup-isolated-setup-verify/step-1-classify/fixtures/step-config.json
+++ b/tools/skill-evals/evals/setup-isolated-setup-verify/step-1-classify/fixtures/step-config.json
@@ -1,4 +1,4 @@
{
"skill_md": "skills/setup-isolated-setup-verify/SKILL.md",
- "step_heading": "## The 10 checks"
+ "step_heading": "## The 11 checks"
}
diff --git a/tools/spec-loop/.last-sync b/tools/spec-loop/.last-sync
index cb94e02d..110ee8ab 100644
--- a/tools/spec-loop/.last-sync
+++ b/tools/spec-loop/.last-sync
@@ -1 +1 @@
-3c5c139f47a5856d1c23d7b38f5b1b141364288f
+f2c549c20fcf259e0e680a2269b2561d6da7243f
diff --git a/tools/spec-loop/specs/README.md b/tools/spec-loop/specs/README.md
index a7bcb21d..8ea9e4ec 100644
--- a/tools/spec-loop/specs/README.md
+++ b/tools/spec-loop/specs/README.md
@@ -54,7 +54,8 @@ Start with [`overview.md`](overview.md), then:
[`codex-runtime.md`](codex-runtime.md),
[`maintainer-education.md`](maintainer-education.md),
[`spec-gap-staleness.md`](spec-gap-staleness.md),
- [`vetted-command-surface.md`](vetted-command-surface.md).
+ [`vetted-command-surface.md`](vetted-command-surface.md),
+ [`sandbox-diagnostics.md`](sandbox-diagnostics.md).
(Agentic Autonomous, the fifth MISSION mode, is deliberately off and has no
spec — see the note in [`overview.md`](overview.md).)
diff --git a/tools/spec-loop/specs/agent-isolation-sandbox.md b/tools/spec-loop/specs/agent-isolation-sandbox.md
index ef98f973..8ff28cbb 100644
--- a/tools/spec-loop/specs/agent-isolation-sandbox.md
+++ b/tools/spec-loop/specs/agent-isolation-sandbox.md
@@ -76,9 +76,10 @@ existing sandbox grants can widen the baseline. See `docs/adapters/gemini.md`.
`sandbox-lint --gemini .gemini` checks the static profile, with opt-in pytest integration tests against native 0.59.0 APIs for settings, policies, headless refusal, and Linux enforcement.
Every Gemini upgrade requires revalidating the native probe against that version; static CI checks alone do not establish effective policy precedence.
- Skills: `setup-isolated-setup-install`, `-update`, `-verify`,
- `-doctor` (probes live sandbox restrictions — SSH-agent reachability,
- localhost port binding, docker/podman socket — and maps each to a
- numbered troubleshooting entry; read-only, never modifies settings).
+ `-doctor`. The diagnostic side — the failure catalog in
+ `docs/setup/sandbox-troubleshooting.md`, the `sandbox-error-hint.sh`
+ hook, the doctor's live probes and the verify checks — is specified
+ in [`sandbox-diagnostics.md`](sandbox-diagnostics.md).
- `docs/setup/secure-agent-internals.md` — the three-layer model.
## Behaviour & contract
@@ -94,9 +95,13 @@ The reference model is four layers, layered:
2. **Filesystem + network sandbox** — Linux `bubblewrap` + `socat` SNI
proxy; macOS `sandbox-exec`. Default-deny reads outside the tree and
egress to non-allowed hosts. `sandbox.excludedCommands` carves out
- commands that need host auth the sandbox blocks — `gh` (OS keyring);
- the blast radius is held by layers 3 (`gh auth token` / `gh auth
- refresh` denied) and 4 (`gh` writes gated by `ask`).
+ commands that need host auth the sandbox blocks — `gh` (OS keyring
+ and, on macOS, Security.framework TLS verification); the blast
+ radius is held by layers 3 (`gh auth token` / `gh auth refresh`
+ denied) and 4 (`gh` writes gated by `ask`). The exemption applies
+ only to invocations made of `cd …` / `gh …` parts; the shape rule
+ and its failure signature are in
+ [`sandbox-diagnostics.md`](sandbox-diagnostics.md).
3. **Tool permissions** — the host's `permissions.deny` blocks denied
paths/binaries (`Read(~/.ssh/**)`, `Bash(curl *)`, …).
4. **Forced confirmation** — `permissions.ask` on `git push` and,
diff --git a/tools/spec-loop/specs/overview.md b/tools/spec-loop/specs/overview.md
index f4ae047e..54d5ab65 100644
--- a/tools/spec-loop/specs/overview.md
+++ b/tools/spec-loop/specs/overview.md
@@ -48,6 +48,7 @@ Each mode is an independently toggleable set of skills. Maturity mirrors
| Release-management lifecycle (experimental — all 10 skills shipped) | [release-management-lifecycle.md](release-management-lifecycle.md) |
| Privacy-LLM gate + PII redaction | [privacy-llm-gate.md](privacy-llm-gate.md) |
| Agent isolation / layered sandbox | [agent-isolation-sandbox.md](agent-isolation-sandbox.md) |
+| Sandbox diagnostics — catalog, hint hook, doctor, verify | [sandbox-diagnostics.md](sandbox-diagnostics.md) |
| CVE tooling | [cve-tooling.md](cve-tooling.md) |
| Security reporting & dashboards | [security-reporting.md](security-reporting.md) |
| Adoption & setup | [adoption-and-setup.md](adoption-and-setup.md) |
diff --git a/tools/spec-loop/specs/sandbox-diagnostics.md b/tools/spec-loop/specs/sandbox-diagnostics.md
new file mode 100644
index 00000000..4f93db1e
--- /dev/null
+++ b/tools/spec-loop/specs/sandbox-diagnostics.md
@@ -0,0 +1,175 @@
+<!-- SPDX-License-Identifier: Apache-2.0
+ https://www.apache.org/licenses/LICENSE-2.0 -->
+
+---
+title: Sandbox diagnostics — catalog, hint hook, doctor, verify
+status: stable
+kind: feature
+mode: infra
+source: >
+ MISSION.md § Privacy, security and supply-chain integrity ("Layered
+ sandbox by default") and docs/setup/secure-agent-setup.md § Sandbox-
+ error hint hook. Implemented in docs/setup/sandbox-troubleshooting.md,
+ tools/agent-isolation/sandbox-error-hint.sh, and the
+ setup-isolated-setup-doctor / setup-isolated-setup-verify skills.
+acceptance:
+ - Every sandbox-shaped failure the framework knows about has exactly
+ one catalog entry, in the symptom / root cause / fix / notes shape,
+ and the catalog is the single source of truth for the remediation.
+ - The hint hook recognises every catalogued symptom string and points
+ at that entry; it never changes the tool call's outcome and never
+ fires on benign output.
+ - The doctor probes every catalogued failure mode live and read-only;
+ the verify skill checks the static configuration each entry relies
+ on; neither edits settings.
+ - A gh call that ran inside the sandbox is recognised as such, with
+ the invocation-shape rule that decides whether the exclusion
+ applied.
+---
+
+# Sandbox diagnostics — catalog, hint hook, doctor, verify
+
+## What it does
+
+Makes an over-restrictive sandbox self-explaining. A correct sandbox
+denies credentials and unknown hosts; the same denials also break
+legitimate workflows in ways that look like unrelated bugs — an SSH
+agent that "refuses", a Docker daemon that is "not running", a `gh`
+that "cannot verify" a valid certificate. This area turns each of
+those into a catalogued failure mode with three discovery surfaces:
+the catalog entry itself, a just-in-time hint printed next to the
+error, and two skills that probe or verify the setup on demand.
+
+## Where it lives
+
+- `docs/setup/sandbox-troubleshooting.md` — the catalog. One entry per
+ failure mode, each with **Symptom** (the literal error text),
+ **Root cause** (which sandbox layer blocks it and why), **Fix** (a
+ settings widening with per-entry rationale, or — for the `gh` entry —
+ an invocation-shape rule, because there is nothing to widen), and
+ **Notes**. Six entries today: SSH agent / Yubikey, signed commit
+ failing before any touch (`gpg.format=ssh` key unreadable), localhost
+ port bind, Docker / Podman socket, `/tmp` read-only, and `gh` inside
+ the sandbox (TLS `OSStatus -26276` / `HTTP 401`).
+- `tools/agent-isolation/sandbox-error-hint.sh` — a Claude Code
+ `PostToolUse` hook on the `Bash` matcher. Scans the tool's stdout +
+ stderr for the catalogued symptom strings and, on a match, prints
+ one `[sandbox-hint] …` line to stderr naming the catalog anchor,
+ exiting 1 so the line reaches the model and the user. Tests under
+ `tools/agent-isolation/tests/test_sandbox_error_hint.py`.
+- Skill `setup-isolated-setup-doctor` — live, read-only probes, one per
+ catalog entry (`## The 6 probes`), each reporting ✓ / ✗ / ⊘ / ⚠ with
+ the command and its output as evidence, and each mapping ✗ to the
+ matching catalog anchor. The `gh` probe runs `gh` through `sh -c` so
+ the `excludedCommands` exemption cannot apply to the probe itself,
+ which shows what an un-excluded `gh` does on this machine, then
+ checks that `"gh *"` is configured.
+- Skill `setup-isolated-setup-verify` — static checks of the installed
+ configuration (`## The 11 checks`): settings shape, hook wiring, hook
+ scripts, wrapper, pinned versions, status line, denial canaries,
+ project-root grant, the vetted-ops split, the touch overlay and
+ signing key (check 10), and (check 11) the `gh` exclusion. Mirrors the "Via a Claude Code prompt" checklist in
+ `docs/setup/secure-agent-setup.md`, which is the canonical list.
+- `docs/setup/secure-agent-setup.md` § Sandbox-error hint hook — the
+ signature → anchor table, install recipe, and trade-offs.
+
+## Behaviour & contract
+
+- **The catalog is the source of truth.** The hook, the doctor and the
+ verify skill are discoverability layers over it. A new failure mode
+ lands as a catalog entry first; the same change adds a `match … hint=`
+ branch to the hook, a probe to the doctor, and — where the entry
+ relies on a static setting — a check to the verify skill. A catalog
+ entry with no hook branch, or a hook branch with no catalog anchor,
+ is drift.
+- **Symptom strings are literal.** Entries quote the exact error text
+ so a grep into the catalog finds them; the hook matches those same
+ strings with anchored, specific regexes. False-positive hints are
+ noise, so the pattern set errs on the side of missing a variant.
+- **The hook never changes the outcome.** It exits 0 silently on no
+ match, on a non-`Bash` tool, on unparsable JSON, or on any unexpected
+ envelope shape (fail-open), and exits 1 — never 2 — on a match, so
+ the tool call that already ran is not retroactively blocked.
+- **Doctor and verify are read-only.** They report and link; the
+ settings edit is the operator's, made outside the skill. A probe that
+ cannot run because a prerequisite is absent (no `docker` on `PATH`,
+ no `SSH_AUTH_SOCK`, no `gh`) reports ⊘, not ✗.
+- **Every probe runs even after an early failure**, so one report lists
+ every restriction rather than one per re-run.
+- **The `gh` exclusion rule.** `sandbox.excludedCommands: ["gh *"]`
+ exempts a `gh` call only when every part of the Bash invocation is
+ `cd …` or `gh …`. A pipe, a `$(…)` substitution, a loop, or any file
+ redirection (including `> /dev/null`) keeps the whole invocation in
+ the sandbox, where `gh` fails with `x509: OSStatus -26276` on macOS
+ because Go's TLS verification goes through Security.framework and
+ Seatbelt blocks the trustd / keychain mach services. Nothing on the
+ Go side works around that (no fallback roots in the Homebrew build,
+ `SSL_CERT_FILE` ignored on darwin). The catalog entry carries the
+ measured shape table, the `gh tofile` alias that moves a redirection
+ inside `gh` (guarded to `$PWD` and the Claude scratch tree, because
+ the alias runs unsandboxed), and the upstream report
+ anthropics/claude-code#95532 for the redirection regression.
+- **User-scope install.** The hook is meant for `~/.claude/settings.json`
+ so it fires in every project on the host; the framework's own
+ `.claude/settings.json` does not wire it.
+
+## Out of scope
+
+- The sandbox itself — profiles, allowlists, the clean-env wrapper, the
+ action guard — is `agent-isolation-sandbox.md`.
+- Installing or updating the secure setup (`setup-isolated-setup-install`
+ / `-update`); this area only diagnoses what is installed.
+- Applying a settings widening from a skill. Every fix is the
+ operator's edit, and the doctor says so explicitly.
+- Non-Claude harnesses: the doctor and verify route Codex and Gemini to
+ their adapters before the Claude-specific probes; the hook is a
+ Claude Code `PostToolUse` contract.
+
+## Acceptance criteria
+
+1. Each catalog entry has the four sections, a literal symptom, and a
+ matching `match … hint=` branch in the hook whose anchor resolves
+ (lychee checks the docs; the hook's anchors are asserted by its
+ tests).
+2. The hook exits 1 with a `[sandbox-hint]` line for every catalogued
+ signature, exits 0 with no output on benign output, on a non-Bash
+ tool, and on invalid JSON.
+3. The doctor has one probe per catalog entry and the verify skill has
+ a check for every static setting an entry depends on; both are
+ read-only and both link the catalog anchor on ✗.
+4. The `gh` entry, probe and check agree on the invocation-shape rule
+ and cite the upstream issue.
+
+## Validation
+
+```bash
+uv run --project tools/agent-isolation --group dev python -m pytest \
+ tools/agent-isolation/tests/test_sandbox_error_hint.py
+bash -n tools/agent-isolation/sandbox-error-hint.sh
+prek run lychee --files docs/setup/sandbox-troubleshooting.md docs/setup/secure-agent-setup.md
+# every hook anchor must exist as a heading in the catalog
+python3 - <<'PY'
+import re, sys
+doc = open("docs/setup/sandbox-troubleshooting.md").read()
+hook = open("tools/agent-isolation/sandbox-error-hint.sh").read()
+anchors = set(re.findall(r"\$\{doc_path\}#([a-z0-9-]+)", hook))
+toc = set(re.findall(r"\]\(#([a-z0-9-]+)\)", doc))
+missing = anchors - toc
+print("hook anchors missing from catalog:", sorted(missing) or "none")
+sys.exit(1 if missing else 0)
+PY
+PYTHONPATH=tools/skill-evals/src python3 -m skill_evals.runner \
+ tools/skill-evals/evals/setup-isolated-setup-doctor/
+PYTHONPATH=tools/skill-evals/src python3 -m skill_evals.runner \
+ tools/skill-evals/evals/setup-isolated-setup-verify/
+```
+
+## Known gaps
+
+- The invocation-shape rule is measured on macOS / Claude Code 2.1.278
+ and will change when anthropics/claude-code#95532 is fixed; the
+ catalog entry, the probe text and the verify check all need the same
+ edit on that day.
+- Linux / bubblewrap is not measured for the `gh` entry: Go uses its
+ own root store there, so the TLS half may not apply while the keyring
+ half still can.