acc: mutate-based invariant fuzzing for bundle configs - #5686
Draft
radakam wants to merge 105 commits into
Draft
Conversation
Collaborator
Integration test reportCommit: 78ca5f2
8 interesting tests: 4 SKIP, 3 KNOWN, 1 RECOVERED
|
Contributor
Approval status: pending
|
denik
reviewed
Jun 26, 2026
The fuzzer sources the invariant targets with no INPUT_CONFIG, which under set -u aborted migrate/script after invariant_deploy had already printed INPUT_CONFIG_OK, so every migrate seed was classified as a post-deploy bug. Guarding each dereference put an unenforced obligation on a subtree the fuzzer is supposed to leave unaware of it, and the guard evaluates to the same "not any curated config" the fuzzer wants anyway. Export an empty INPUT_CONFIG from fuzz/script.prepare instead, which is accurate rather than a placeholder: the config is generated, so it matches no curated name. migrate/script goes back to its plain dereference. invariant_cleanup keeps an explicit -n guard: it runs from an EXIT trap where an unbound variable would skip the destroy and leak workspace state, and the previous form relied on configs/-cleanup.sh not existing.
…-checking gen_scalar pins name/display_name by leaf name at any depth, so an array of named objects (job parameters) repeated one value across its elements and was rejected as a duplicate every time. Number the pinned values per occurrence. emit_fuzz_config writes to a redirect, and to_yaml emits non-ASCII literally, so the astral-plane probe in DANGEROUS_STRINGS would raise UnicodeEncodeError on a Windows stdout; reconfigure it to UTF-8. classify() reported the first line of a generator failure, which for any traceback is the useless "Traceback (most recent call last):"; report the last line instead. The tables in gen_fuzz_config annotate a schema that moves under them, with no signal when they stop fitting. check_tables now reports a key that is not a resource type, a PERMISSION_LEVEL outside that resource's own level enum, a GRANT_PRIVILEGE that is not a catalog privilege, a resource whose levels the schema names but the table omits, and a SKIP_PROPERTY_NAMES entry no resource declares. Levels resolving to the generic iam.PermissionLevel union are skipped, since it lists every level of every resource type. It found two: instance_pools has a precise level enum but no entry, so gen_permissions emitted nothing for it, and browse_only is not a property in the bundle schema at all.
Comments across the fuzzer had grown to three and four lines each, often restating the code beside them. Cut to the reason each exists, which drops about 45 lines without losing a why. Four fixes alongside: - MAX_RECURSION's comment promised a guarantee it does not give. It counts object and array nesting only: branch descent reuses the same depth and resolve() follows $ref chains in a loop, so neither is bounded by it. - resource_types and resource_element raised a bare StopIteration if the schema shape changed; object_branch exits naming the failed lookup. - should_skip_property resolved the same node twice. - gen_object's depth-0 scoping and gen's by-name interception of grants/permissions now record why they are safe: only resource elements declare grants or permissions, verified against the schema.
The generator carried three checks for schema markers that cannot reach it. bundle/internal/schema/main.go deletes OUTPUT_ONLY properties and libs/jsonschema/from_type.go skips bundle:"readonly" fields before the schema is emitted, so neither x-databricks-field-behaviors nor readOnly appears in it, and libs/jsonschema.Schema has no readOnly field to emit in the first place. const is never set for the bundle schema either. None of the three occurs in the live or the committed schema. Removing them leaves should_skip_property a name check that no longer needs the property schema. SCALAR_TYPES goes with them: libs/jsonschema.Type is a closed enum of boolean, string, number, object, array and integer, each already handled, so the guard could not fire and its "keep in sync" note was an obligation with no payoff. Alongside: - RESOURCE_REQUIRED_FIELDS is documented as fields absent from the schema's required[], but alerts' display_name and warehouse_id and apps' name are required there. Each element resolves to one concrete branch, so the union was identical. - token() was implemented identically in both modules. - collect_insertions threaded an rtype parameter through its recursion that always equalled the gen.rtype it already reads for should_skip_property. - invariant_cleanup keeps its -n guard but drops the unset default, since the fuzzer sets INPUT_CONFIG; all three reads in the subtree now match. The fuzz selftests reproduce their committed output byte for byte, so none of this changes what the generator emits.
The plan-determinism oracle returned 0 when both plans failed on an unmodeled route, so the seed exited cleanly and run_fuzz.py recorded it as deployed. The TESTSERVER_GAP branch in classify() was therefore unreachable for a gap the CLI hit after the deploy: the marker is only read on a non-zero exit. Fail the seed either way and let classify() decide. It already prefers the gap marker over INPUT_CONFIG_OK for exactly this case, and LOG.plan.failed stays unwritten on a gap, so the oracle does not report it as a bug instead. Without this a window whose seeds all reach an unmodeled route after deploying counts as fully deployed, hiding both the coverage gap and the fact that the invariant never ran.
Nightly and task test-fuzz were finishing the 25-seed ceiling long before FUZZ_TIME_BUDGET, so most of the allotted time went unused. Raise the ceiling to 10000 and let the 900s budget stop the run; keep the committed PR default at 25 for a still-cheap smoke.
Generate spent most seeds on rejects while mutate's schema-aware field injection is what reaches deploy and drift bugs. Keep gen_fuzz_config.py as the library that values injected optionals.
Mutate injects fields at depth 1, so full-resource generation (gen_config, depth-0 required/allowlist tables, non-MUTATE_BASES pins) was unused. Keep only what values injected optionals.
Acceptance python is stdlib-only (no PyYAML), so load/dump stay as a small custom codec next to mutate. Drop the emit dispatcher and move MUTATE_BASES + script entrypoint into mutate_fuzz_config.py; gen_fuzz keeps only schema value generation and table checks.
A destructive mutation can delete a field the schema marks required. `bundle validate` only warns about it, so the config still deploys under terraform while the direct engine refuses it. The seed then landed past INPUT_CONFIG_OK and was filed as a bug, but the difference is one of engine strictness on input the CLI had already called incomplete. Reject those seeds at the fuzz render step so they count as rejected, which is what a config the CLI flagged as incomplete deserves.
State what the harness does; cut the "rather than / instead of / not a …" framing.
Keep the how; cut glosses that duplicate the targets and "rather than / not a" framing.
Rotate seeds by day-of-epoch so PR runs of push.yml do not open gaps, raise test-fuzz's package timeout to cover four budgeted variants, restore set +e around invariant_deploy so a panicking reject is still asserted, and drop the unused prologue.sh that still called emit_fuzz_config.py.
Additive mutate now picks from a hand-curated INJECT table of deploy-proven optionals (including the fields behind past drift findings) instead of walking bundle schema. Drop gen_fuzz_config and the schema dump, and add app and experiment to MUTATE_BASES so those types stay in reach under mutate-only.
Keep the non-obvious reasons; drop restatements and contrastive framing across the harness, Taskfile, and nightly job.
Also diversify the selftest sample seeds and gate nightlies through test-result with a failure summary for triage.
State what the harness does and why; cut "rather than / not a / no X" asides.
Drop the custom YAML dumper for json.dumps (valid YAML 1.2) and check every INJECT field is a settable schema input, which also caught a nonexistent experiments.description entry.
Keep inject and destroy on separate seeds, harden the loader contract check, and keep LOG.repro reachable after a red nightly.
Treat validate type-mismatch warnings as seed rejections so configs like group_name: [] do not deploy and fail mid-migrate. Point the nightly triage upload at the same TMPDIR root the summary step uses.
Drop the hand-rolled YAML loader for committed JSON fixtures, reject schema warnings via validate --strict, and run each seed through seed.sh instead of export -f.
Sibling layout had to copy curated [[Server]] stubs and source helpers by hand. As invariant/fuzz/, parent prepare and stubs come from the directory chain; the leaf only clears INPUT_CONFIG, forces Cloud=false, and adds TESTSERVER_GAP catch-alls.
…inheritance" This reverts commit e1246fb.
The fuzzer kept its own JSON snapshot of each invariant config because the acceptance Python helpers are stdlib-only and cannot parse YAML. That made the curated YAML and its snapshot drift apart silently whenever one was edited. Parse the invariant YAML directly via a new hidden `bundle debug yaml-to-json`, so there is one copy of every base and it is parsed the way the bundle parses it. Output is byte-identical to the snapshots this removes. Also drop the unrelated comment rewording in invariant/script.prepare, leaving only the INVARIANT_DIR indirection and the panic scan on a rejected deploy.
bundle/debug's help output is a golden, and the other tooling-only subcommands (terraform, render-template-schema, list-targets) are hidden so they stay out of it. Hide this one too rather than adding a test helper to the listing.
Fall through to destructive mutate when INJECT has nothing to add, fail windows with zero deploys unless every seed is a gap, and drop incomplete INJECT entries that only produced rejects.
gotestsum places custom flags before the package list, so -keeptmp was rejected before any fuzz seed ran.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bugs found
CLI
Test server
Upstream