Windows deep scans can fail when subprocess output contains non-CP1252 bytes

[gillangad]

Summary

A deep repository scan on Windows can abort when the scan orchestrator reads subprocess output using the system CP1252 encoding and encounters bytes that CP1252 cannot decode. The scan then fails to finalize the required scan artifacts.

This appears to be a reliability issue in the local scan orchestration path, rather than a finding in the repository being scanned.

Environment

• Codex Security package: @openai/codex-security@0.1.5
• OS: Windows
• Scan scope: full repository
• Scan mode: deep

Reproduction

This was observed during a full-repository deep scan on Windows:

1. Start a deep scan.
2. Allow discovery workers to run.
3. A worker subprocess emits output containing bytes that are not valid under the active CP1252 locale.

I have not yet isolated a minimal fixture that deterministically emits the offending byte sequence, so this report is based on a captured scan failure.

Observed behavior

The Python subprocess reader raised:

UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d
in position 2397887: character maps to <undefined>

A second worker showed the same class of failure with byte 0x81. The orchestrator then recorded follow-on worker failures:

'NoneType' object has no attribute 'splitlines'

The scan aborted before producing the required final artifacts:

• scan-manifest.json
• findings.json
• coverage.json

Expected behavior

The scan should either:

1. Complete despite non-ASCII subprocess output, or
2. Fail with the original underlying error while preserving useful partial artifacts and a clear diagnostic.

The scan should not depend on the Windows system locale for decoding subprocess output.

Possible fix

• Use explicit, locale-independent decoding for subprocess text output where appropriate, such as UTF-8 with a documented error policy.
• Guard against missing stdout/stderr before calling .splitlines().
• Add a Windows regression test containing non-CP1252 subprocess output.
• Preserve partial scan metadata when finalization fails.

Setting PYTHONUTF8=1 in the parent environment appears to be a useful workaround for the local wrapper, but an explicit fix in the orchestration path would be more robust.

I can provide sanitized logs or help reduce this to a minimal reproduction if useful.