Add a documentation consistency test to prevent stale example references

[anupddas]

Describe the issue

Summary

I would like to propose a lightweight automated test that validates consistency between AWS CLI documentation claims and the actual example layout/files in the repository.

At the moment, example-related documentation can drift over time. When that happens, users can end up following instructions that point to files, entrypoints, or example structures that no longer exist or are no longer representative of the repository.

Problem

The repository already has example-validation infrastructure (test_examples.py) and the contribution guide emphasizes that example changes should be real, working, and tested. However, I could not find a small regression test that checks whether documentation claims about examples still match the repository contents.

This makes it easy for documentation to become stale in cases such as:

• a README describing a file that was renamed or removed
• a documented example directory missing an expected entrypoint
• a new example being added without updating the surrounding docs
• a special-case example being introduced without explicitly documenting the exception

Why this matters

This is a developer-experience issue rather than a runtime bug, but it still affects users directly.

A documentation consistency test would:

• prevent broken instructions from shipping
• reduce maintainer time spent on doc drift
• make example additions safer
• catch regressions earlier in CI
• keep the repository’s documented contract aligned with reality

Proposed solution

Add a small automated test that verifies the repository’s documented example expectations.

A practical approach would be:

1. Enumerate the example directories under the documented examples path.
2. Verify required entrypoints or referenced files exist.
3. Fail when documentation and repository structure diverge.
4. Allow explicit exceptions where a folder is intentionally structured differently.

Example implementation sketch

from pathlib import Path

EXAMPLE_ROOT = Path("awscli/examples")

# Keep intentional exceptions explicit instead of letting the docs drift silently.
SYNC_ONLY_EXAMPLES = {
    # Example: "09_async_parity",
}

def test_documented_example_entrypoints_exist():
    for example_dir in EXAMPLE_ROOT.iterdir():
        if not example_dir.is_dir():
            continue

        sync_py = example_dir / "sync.py"
        async_py = example_dir / "async.py"

        assert sync_py.exists(), f"{example_dir.name} is missing sync.py"

        if example_dir.name not in SYNC_ONLY_EXAMPLES:
            assert async_py.exists(), f"{example_dir.name} is missing async.py"

If the repository prefers a stricter contract, the same idea could be extended so the test parses the relevant documentation text and checks that every documented example path is present on disk.

Suggested acceptance criteria

• A failing example/documentation mismatch is detected in CI.
• Intentional exceptions are explicitly documented in code or config.
• The test is lightweight and deterministic.
• The test does not require live AWS calls.
• The implementation follows the repository’s existing testing patterns.

Additional context

I believe this is a high-value, low-risk improvement because it guards the repository against a common class of regressions: documentation drift.

Links

.