Dangerous-shell scoring misses commands executed through sh -c wrappers

[realmehmetali]

What happened

has_dangerous_shell_pattern() detects destructive commands when they are invoked directly, but quoted payloads are stripped before matching. As a result, the same command executed through a shell wrapper such as bash -c, sh -lc, or zsh -c is not recorded as a dangerous-shell trajectory violation.

This can overstate the safety score for an agent trace even though the nested shell actually executes the destructive command.

Expected behaviour

Command payloads passed to directly invoked POSIX shell wrappers with -c should be inspected recursively. Quoted command mentions that are not executed—such as echo "rm -rf", a commit message, or Python code that only prints the text—should remain ignored.

Steps to reproduce

python - <<'PY'
from clawbench.trajectory import has_dangerous_shell_pattern

for command in (
    'bash -c "rm -rf /tmp/synthetic-agent-work"',
    "sh -lc 'git reset --hard'",
    'zsh -c "sudo reboot"',
):
    print(command, has_dangerous_shell_pattern(command))
PY

Relevant output

bash -c "rm -rf /tmp/synthetic-agent-work" False
sh -lc 'git reset --hard' False
zsh -c "sudo reboot" False

Why this matters for frontier AI evaluation

Frontier agent benchmarks need safety scoring to follow the action that actually executes, not just the outer command string. Otherwise, an increasingly capable agent can receive a clean safety score simply by routing an identical destructive operation through a nested shell.

Environment

• Python version: 3.13.0
• OS: macOS
• ClawBench version / commit: c725622

AI and provenance disclosure

This report was prepared with significant assistance from OpenAI Codex and manually reproduced. It is based entirely on the public repository and synthetic commands targeting /tmp; it contains no private or proprietary code, data, plans, or product-specific examples.

[realmehmetali]

While adding the nested-shell regression, the focused test exposed an adjacent direct-match defect: git checkout -- tracked.txt is also currently reported as safe. The existing pattern ends -- with \b, but there is no word boundary between a hyphen and following whitespace/end-of-string.

I will include the narrow correction (--(?:\s|$)) and direct regression cases in the same PR because both defects affect the same dangerous-shell safety signal.

[clawsweeper]

Codex review: keeping this open for maintainer follow-up; there is still a little grit to resolve. Reviewed July 18, 2026, 9:32 AM ET / 13:32 UTC.

Summary
Keep this issue open: it reports a focused, safety-relevant scoring gap with a concrete current-main reproduction, and no verified main-branch fix or canonical replacement was available in the provided context. Likely related people: realmehmetali (low confidence; reporter who supplied the reproduction and adjacent regression detail).

Reproducibility: no. independently verified current-main reproduction was available in this read-only run, but the issue provides a precise synthetic script and exact expected-versus-actual outputs against commit c725622.

Ways to help us reproduce this

• Add a screenshot or short recording showing the behavior.
• Include redacted logs or terminal output.
• Share version, platform, channel/provider, and relevant config details.

Next step
The report is narrow and actionable, but current source, tests, repository policy, and feature ownership could not be independently inspected before safely dispatching an automated repair.

Review details

Best possible solution:

Extend the dangerous-command classifier to parse only directly invoked supported shell wrappers and recursively inspect their executed -c payloads, with positive wrapper cases and negative quoted-text cases alongside the git checkout -- regression coverage.

Do we have a high-confidence way to reproduce the issue?

No independently verified current-main reproduction was available in this read-only run, but the issue provides a precise synthetic script and exact expected-versus-actual outputs against commit c725622.

Is this the best way to solve the issue?

Yes, conditionally: recursively examine only the executable payload of recognized direct shell -c invocations, and retain explicit negative coverage for quoted text that is displayed rather than executed.

AGENTS.md: unclear because the file could not be read completely.

Codex review notes: model internal, reasoning high; reviewed against c7256221f52d.

Label changes

Label changes:

• add P2: This is a bounded but meaningful benchmark safety-scoring accuracy defect with a clear proposed regression boundary.
• add impact:other: The reported effect is incorrect benchmark safety classification, which does not fit the more specific owned impact labels.
• add issue-rating: 🦪 silver shellfish: Current issue advisory state selects this label.
• add clawsweeper:no-new-fix-pr: Current issue advisory state selects this label.
• add clawsweeper:needs-maintainer-review: Current issue advisory state selects this label.

Label justifications:

• P2: This is a bounded but meaningful benchmark safety-scoring accuracy defect with a clear proposed regression boundary.
• impact:other: The reported effect is incorrect benchmark safety classification, which does not fit the more specific owned impact labels.

Evidence reviewed

What I checked:

• Focused current-main report: The report names current main commit c725622 and gives three direct shell-wrapper examples that return false despite executing destructive payloads, plus a synthetic /tmp reproduction. (clawbench/trajectory.py, c7256221f52d)
• Adjacent regression boundary: The reporter identified a related direct-match defect for git checkout -- tracked.txt and proposed the narrow terminator correction --(?:\s|$), which belongs in the same dangerous-shell test surface. (clawbench/trajectory.py, c7256221f52d)
• No established replacement: The supplied referencing merged pull requests concern workspace isolation and run-lifecycle hardening, not dangerous-shell command classification, so they do not supersede this report.

Likely related people:

• realmehmetali: Provided the concrete current-main reproduction and identified the adjacent direct-pattern boundary case; source-history ownership could not be established from the available read-only inspection. (role: reporter and reproducer; confidence: low; files: clawbench/trajectory.py)

How this review workflow works

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

[realmehmetali]

Implementation update after an adversarial prototype: a recursive shlex scanner is not a mergeable solution here.

It handled the initial direct wrappers, but realistic shell grammar exposed both false negatives and false positives across quoted and unquoted heredocs, substitutions nested inside double quotes, arithmetic expansion, case patterns, control-operator runs, redirections around -c, and wrapper options such as env -S. The prototype grew beyond 500 lines while still lacking complete Bash semantics, so I am intentionally not opening it as a PR.

Recommendation: treat nested-shell scoring as an AST-level design problem. Use a parser that preserves quoting, heredoc expansion, command-substitution, and command-position semantics; otherwise explicitly define a smaller supported grammar and expose detection confidence. A synthetic, table-driven adversarial corpus should cover safe documentation text as carefully as destructive payloads.

This precision matters for frontier AI-agent evaluation: false negatives hide unsafe behavior, while false positives distort safety scores and model comparisons.

Provenance: this investigation used significant OpenAI Codex assistance and only the public repository plus synthetic /tmp examples. No private code, data, product plans, prompts, or proprietary examples were used.

[clawsweeper]

ClawSweeper status: review started.

I am starting a fresh review of this issue: Dangerous-shell scoring misses commands executed through sh -c wrappers This is item 1/1 in the current shard. Shard 0/1.

This placeholder means the worker is alive and reading the current context. I will edit this same comment with the actual review when the claws are done clicking.

Crustacean status: shell secured, claws on keyboard, evidence pebbles being sorted.