103 test scripts are not executable, so 30 documented invocations fail with Permission denied

[OffgridwithJD]

docs/testing.md tells a reader to run a suite by naming it. Thirty of those invocations fail, because 103 of the harness scripts do not carry the executable bit. CI never meets this, so nothing has reported it.

What happens

$ test/run_all_versions.sh /usr/local/pg18a/bin/pg_config
bash: test/run_all_versions.sh: Permission denied

$ test/parquet_export.sh /usr/local/pg18a/bin/pg_config
bash: test/parquet_export.sh: Permission denied

Both are the exact command docs/testing.md gives (lines 156 and 27). test/smoke.sh, which is mode 100755, runs.

Who this affects, and who it does not

A person following the documentation, on a fresh clone. That is the whole of it.

CI and the version matrix are unaffected, and it is worth being explicit about why, because it is the reason this has been invisible:

• .github/workflows/ci.yml:485 and nightly.yml:188 both invoke it as bash test/run_all_versions.sh ....
• run_all_versions.sh runs each suite as bash "$builddir/test/${s}.sh" "$pgc" (lines 712 and 749), so a suite's own mode never matters inside the matrix.

So every green run in the project's history is honest. The gap is between the documentation and a shell.

The measurement

At 0e4884c1, counted from the ref rather than a working tree:

	100755	100644
test/*.sh	153	95
test/*.py	4	8

30 documented invocations name one of those 103 files: 29 in docs/testing.md (lines 13, 20-24, 26-28, 33, 51-55, 57, 58, 60-65, 156, 188, 189, 217, 272, 412) and one in docs/limitations.md:79. Two further mentions — docs/administration.md:422 and CONTEXT.md:166 — are prose references, not commands, and are fine.

Cause: two different situations, not one

I searched the whole history of test/ for mode transitions. There is exactly one 100755 -> 100644 change:

56ae5f8eb  2026-08-16  perf: iterate needed columns in the batch fold, not all natts
   test/run_all_versions.sh   100755 -> 100644

That commit is mine. It touched four files, added one line to SUITES, and took the bit off run_all_versions.sh on the way past — no part of the change had anything to do with it. run_all_versions.sh was added executable on 2026-07-19 in 1d1b31b and stayed that way for four weeks until I broke it. test/native_batch_fold_projection.sh, added in that same commit, was created 100644.

The other 102 were born 100644 and never regressed. Spot-checked at their own add commits: parquet_export.sh (b6a3b30), native_agg.sh (85b6f29), hardening.sh (d2cb212), native_batch_fold_projection.sh (56ae5f8) — all 100644 from the first commit that introduced them.

So this is not one accident. It is one regression plus a long-standing habit of adding suites without the bit, and nothing anywhere notices either.

Why nothing caught it

test/harness_selftest.sh has no mode check — no chmod, no 100755, no -x test anywhere in it. The suite that exists to pin harness invariants does not pin this one, which is why a mode could drift out of a perf commit unremarked and why 102 files could arrive without it.

Suggested fix, in this order

1. chmod +x the 103 files and commit the mode change alone, so the diff is reviewable as exactly that.
2. Add a harness_selftest check: every test/*.sh and test/*.py that docs/testing.md names must be executable. Run it over the tree before committing step 1 and it flags 103, which is the false-positive budget measured rather than assumed; after step 1 it flags none.
3. Decide the rule and write it in CONTEXT.md beside the "register every suite in SUITES" line, since that is where someone adding a suite is already reading.

An alternative to 1 and 2 is to change the 30 documented invocations to bash test/..., matching what CI does. I would rather not: docs/testing.md presents these as commands to type, a reader will type them, and chmod +x is the smaller change to make the documentation true.

Happy to send the PR for any or all three; step 2 is the part that stops it recurring, and it is the part I would not want left out.

---

Found while gating #851, whose two new files (test/parquet_export_stats.sh, test/parquet_stats.py) are both correctly 100755. My own gate script died on run_all_versions.sh before it ran a single suite, which is how I noticed.

[jdatcmd]

Reviewed. Every number checks out. I tried to break the count twice and broke my own instrument both times instead.

your claim	mine, at 0e4884c1
test/*.sh 153 executable / 95 not	153 / 95
test/*.py 4 / 8	4 / 8, so 103
30 documented invocations	30, 29 in docs/testing.md and docs/limitations.md:79
CI invokes it as bash	ci.yml:485, nightly.yml:188, both exact
suites run as bash "$builddir/test/${s}.sh"	lines 712 and 749, exact
one 100755 -> 100644 in the whole history	one, 56ae5f8
harness_selftest has no mode check	zero hits for chmod, 100755, -x, executable

My first pass said 124/154, and my second said 26 documented invocations, not 30. Both were me: the first counted test/ recursively and swept in selftest/ and fixtures/, and the second required the line to begin with test/, which drops PGC_ALLOW_MISSING_BTREE_GIST=1 test/temporal.sh at 188, 189 and 217. Those are commands a reader types and they fail the same way. Your population was the right one and stated plainly enough that I could find my own error against it.

The 56ae5f8 archaeology is the part I would not have found. Naming your own commit as the single regression, and separating it from the 102 that were born that way, is what turns this from "chmod some files" into two different problems with two different fixes.

Three things to fold into the PR

1. test/crlf_listener.py has no shebang, and chmod +x would make it worse.

It is the only one of the twelve .py files without one:

$ head -1 test/crlf_listener.py
import socket, sys

Give that file the executable bit and it advertises test/crlf_listener.py as a command. Run it and execve returns ENOEXEC, the shell retries it under /bin/sh, and the reader gets a syntax error from a Python file rather than a clean Permission denied. A worse failure than the one you are fixing, on the one file where nobody asked for the bit: it is invoked as python3 "$(dirname ...)/crlf_listener.py" from objstore_crlf.sh:12, and no document names it.

Add #!/usr/bin/env python3 in the same commit, or leave it 100644 and let the check exempt it by name. I prefer the shebang, because every one of its eleven siblings has one.

2. Step 2's population is narrower than step 1's, and that gap is this issue's own failure mode.

Step 1 chmods 103 files. Step 2 pins "every test/*.sh and test/*.py that docs/testing.md names", which is about 30 of them. The other ~73 can drift back to 100644 and nothing notices, which is precisely how 102 files came to be born without the bit.

Pin all top-level test/*.sh and test/*.py, with an explicit exemption list rather than a documentation-derived one. Two reasons beyond the obvious. A documentation-derived population makes the check a hostage to prose: someone deletes a line from docs/testing.md and silently narrows the gate. And the failure you are guarding against is a mode drifting out of an unrelated commit, which has nothing to do with whether the file happens to be documented that week.

3. Say why the other 43 stay 100644, or the next person will "finish the job".

There are 29 more in test/selftest/ and 14 in test/fixtures/. They should stay exactly as they are, and the reason is worth one sentence in the PR:

harness_selftest.sh:53   for _sf in "$PGC_TESTDIR"/selftest/*.sh; do
harness_selftest.sh:56       . "$_sf"

They are sourced, not executed. The bit would advertise a way to run them that does not work, which is the same defect as the one being fixed, pointing the other way.

On the alternative you rejected

Agreed, and there is a stronger argument available than the one you gave. Every one of the 103 except crlf_listener.py begins #!/usr/bin/env bash or #!/usr/bin/env python3. A file whose first line declares its interpreter and whose mode forbids using it contradicts itself. chmod +x makes the file agree with what it already says. Rewriting 30 documentation lines to bash test/... leaves 103 files still lying about what they are, and the next reader who ignores the docs and just types the name hits it again.

Also right, and worth keeping in the PR body: running the new check before the chmod and getting exactly 103 is the false-positive budget measured rather than asserted. That is the number that makes the check's silence afterwards mean something.

Scope

Take all three steps in one PR if you like; the three are separable but nothing is gained by splitting them. Step 2 is the one I would refuse to see dropped, for the reason you gave and for the population reason above.

Not folding any of this into #851, which stays about #850. Thanks for filing it separately: my chmod would have been three files wide and would have left both the regression and the habit in place.