feat(vector-index): OpenSearch copy-on-dump into immutable generation indices (#343) - #346
Merged
Conversation
…d generation (#343) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…; fix formatting - Add _generation_index and _read_only attribute initialization in __init__ - Fix extra blank lines in base_backend.py and opensearch.py per ruff format Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…n, read-only (#343) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…erations (#343) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ation; one generation per dump dir (#343) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…-trial replacement (#343) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…aveats (#343) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…generation (#343) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…est dump removal (#343) - dump() no longer strands a generation index in the cluster when _copy_index, put_settings, or the alias swap fails after indices.create() succeeds; the alias swap is now best-effort (never fails the dump), and any other failure after create() best-effort deletes the partial generation and re-raises. - HPO's log_module_optimization no longer deletes the previous best module dump when the new best trial's dump was swallowed by Dumper(raise_errors=False) and produced nothing on disk; it now keeps the previous dump and logs a warning instead. - load() and delete_dumped_generation() now validate remote_manifest.json via a shared _read_manifest() helper, raising a specific RuntimeError instead of an opaque KeyError on a malformed manifest. - concepts.rst: qualify the steady-state cluster footprint claim (holds during optimization; a later Pipeline.dump() re-dump adds a generation per node), and note the {base}-best-* naming reservation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
_one_hot_docs's bare `tuple[np.ndarray, ...]` return annotation tripped mypy's --disallow-any-generics under `mypy src/autointent tests` in CI (missed locally since our gate only ran mypy on src). Use `npt.NDArray[np.float32]` (imported under TYPE_CHECKING, matching the repo's existing style in base_backend.py / vector_index.py) instead. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
voorhs
added a commit
that referenced
this pull request
Aug 13, 2026
Bump version in pyproject.toml and docs/source/conf.py, and add a CHANGELOG section covering the OpenSearch vector-index correctness work (#345, #346), the inference_config.yaml tuple-serialization fix (#344), and the metadata-driven require(extra) dependency guard (#339). Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #343.
Invariant restored
A dumped pipeline serves the data its module was fitted on at dump time. Previously
OpenSearchBackend.dump()wrote only a reference to the live index, which every subsequent HPO trial rewrites — so a loaded pipeline served whatever the last trial wrote (reproduced in the issue's investigation comment).Design (as proposed in #343)
dump()copies the live index server-side (_reindex, task-polled) into a write-blocked generation index{base}-best-{uuid}, stampsmappings._meta.dump_id, and records it inremote_manifest.json. Data never leaves the cluster.load()binds read-only to the verified generation; it raises loudly if the generation is missing, was recreated (dump_idmismatch), or the manifest is malformed. Manifest-less (pre-existing) dumps keep the old reference semantics.remove_module_dump()(new, public) deletes a dump tree together with the cluster indices its manifests reference; the HPO best-trial replacement now uses it instead of a barermtree, so steady-state cluster footprint during optimization is the live scratch index plus one generation per node type.Additions beyond the issue text (from the investigation)
Pipeline.dump()after optimization would otherwise snapshot the last trial's live contents).{base}-besttracks the latest generation (operator convenience;load()deliberately ignores it, and a failed alias swap never fails the dump).Tests
Backend-level integration tests against an OpenSearch testcontainer: the #343 regression (dump survives later rewrites and deletion of the live index), loud load failures (missing / recreated / malformed manifest), write-block enforcement (server- and client-side), generation lifecycle (ownership-checked deletion, same-path replacement, generation-sourced re-dump, re-dump of a loaded read-only instance, failed-dump cleanup, alias-conflict tolerance), plus
OptimizationInfowiring tests proving best-trial replacement goes through the remote-aware helper and spares the previous dump when the new one is missing.Docs: new "Vector Index Lifecycle" section in
concepts.rst(portability trade-off, cleanup story, read-only serving pattern,-best-naming caveat,n_jobs = 1caveat for OpenSearch).🤖 Generated with Claude Code