branch-41: [chore](lance) update the thridparty about lance-c version - #66786
Merged
yiguolei merged 1 commit intoAug 15, 2026
Conversation
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
yiguolei
approved these changes
Aug 15, 2026
FANNG1
added a commit
to FANNG1/doris
that referenced
this pull request
Aug 17, 2026
Issue Number: Part of apache#66495 The vector_search() path is algorithm-agnostic - the BE forwards metric, nprobes, refine_factor and ef to lance-c and lets Lance choose the index - but the only algorithm it has ever been exercised against is IVF_PQ (apache#66779). Nothing in the repository could say whether Doris reads an IVF_FLAT, IVF_SQ or graph index at all, so five of the six entries on the issue checklist rested on an argument about the code path rather than on a test. Extend the committed fixture with one table per remaining algorithm and add the suites that query them: - lance_build_preinstalled_catalog.py gains five VECTOR_TABLES entries - vs_ivf_flat_f32, vs_ivf_sq_f32, vs_ivf_hnsw_flat_f32, vs_ivf_hnsw_sq_f32 and vs_ivf_hnsw_pq_f32 - each carrying one index of its own algorithm over the same 1024-row, two-fragment, 16-dim Float32 data the IVF_PQ table already uses, so a table is exactly one cell of the algorithm matrix. The self-check verifies every index the same way it verified IVF_PQ: exactly one index of the expected type and name, covering every fragment, ANNSubIndex and ANNIvfPartition in the indexed plan, KNNVectorDistance and no ANN node in the flat plan, and the row-256 nprobes=1 discriminator that fails if a query silently falls back to a flat scan. - Only IVF_FLAT gets its indexed-equals-flat equality asserted, since it stores the original vectors and a full-partition probe is an exhaustive scan by another name. IVF_SQ, IVF_HNSW_* and IVF_PQ quantize or traverse a graph, so their agreement with the flat search is recorded, never asserted, and their queries use refine_factor. - The graph indexes need a second discriminator, because nprobes alone cannot show that ef reached the index. Measured on this data, a query at row 512 loses a true neighbour at ef=5 that ef=50 finds - but only on IVF_HNSW_SQ; on 1024 collinear vectors the FLAT and PQ graphs still return the exact rows at ef=5. The generator therefore asserts the ef discriminator on the one table that can carry it, records it for the others, and the suite queries that same table. - BOUNDARY_TOP_K makes the generator probe the boundary at the k the suites actually use. It checked k=10 while the suites query k=9, so a retrained index could have passed the generator and failed the suites. test_lance_vector_search_ivf_flat asserts the IVF_FLAT guarantee against Doris at both ends of the dataset; test_lance_vector_search_index_types covers the other four with per-table nprobes=1 discriminators, the ef pair, and the "ef must be greater than or equal to k" error a too-narrow refined graph query must keep producing. The writer pin stays at pylance 4.0.1, now with the evidence that matters after apache#66786 moved branch-4.1 to lance-c v0.1.6: pylance 7.0.0 - the Lance generation the BE reads through v0.1.6 - reproduces the generator self-check line for line on all six tables, and all 35 queries these three suites issue return identical rows and distances under both. The goldens therefore depend on the frozen fixture bytes, not on which Lance version reads them. Claude-Session: https://claude.ai/code/session_01BQAjAdyfr5Rr94WA4WE45S
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
cherry-pick from master #66698
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)