Test index intersection membership against a Set (#4932) - #4940
Open
batrived wants to merge 1 commit into
Open
Conversation
SubqueryIterator streams the results of the first index and keeps the elements which the other indexes also returned. Membership was tested with List.contains inside the filter, so each streamed element scanned the other list from the start, which costs O(n*m). The list is deliberately unbounded. StandardJanusGraphTx passes Query.NO_LIMIT to processIntersectingRetrievals so that the intersection is complete and no result is missed, which makes subLimit Integer.MAX_VALUE and materialises the whole matching set of every other index. Build the set once outside the filter, which makes the intersection O(n + m). Signed-off-by: Balmukund Trivedi <btrivedipublic@gmail.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Improves performance of index-intersection filtering in SubqueryIterator by avoiding repeated List.contains scans, and adds a focused unit test suite to pin intersection semantics.
Changes:
- Replace per-element
List.containsmembership checks with a precomputedHashSetlookup. - Add
SubqueryIteratorTestto verify intersection behavior (null/empty cases, limits, duplicates, and ordering).
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| janusgraph-core/src/main/java/org/janusgraph/graphdb/util/SubqueryIterator.java | Precomputes a Set for intersection membership to reduce intersection cost from O(n·m) to O(n+m). |
| janusgraph-core/src/test/java/org/janusgraph/graphdb/util/SubqueryIteratorTest.java | Adds unit tests covering semantics that could be affected by switching from list membership to set membership. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| when(subQuery.getProfiler()).thenReturn(QueryProfiler.NO_OP); | ||
|
|
||
| final IndexSerializer indexSerializer = mock(IndexSerializer.class); | ||
| when(indexSerializer.query(any(), any(), any())).thenReturn(firstIndexResults.stream()); |
Comment on lines
+78
to
+83
| //Membership is tested once for every element the first index returns, and otherResults is deliberately | ||
| //unbounded: StandardJanusGraphTx passes NO_LIMIT to processIntersectingRetrievals so that the intersection is | ||
| //complete. Scanning the list for each element would make the intersection cost O(n*m) | ||
| final Set<Object> otherResultSet = otherResults == null ? null : new HashSet<>(otherResults); | ||
| elementIterator = stream | ||
| .filter(e -> otherResults == null || otherResults.contains(e)) | ||
| .filter(e -> otherResultSet == null || otherResultSet.contains(e)) |
| //Membership is tested once for every element the first index returns, and otherResults is deliberately | ||
| //unbounded: StandardJanusGraphTx passes NO_LIMIT to processIntersectingRetrievals so that the intersection is | ||
| //complete. Scanning the list for each element would make the intersection cost O(n*m) | ||
| final Set<Object> otherResultSet = otherResults == null ? null : new HashSet<>(otherResults); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #4932.
SubqueryIteratorstreams the results of the first index and keeps the elements the other indexes also returned. Membership was tested withList.containsinside the filter, so every streamed element scanned the other list from the start, giving O(n·m).This matters more than a typical
contains-on-a-list nit because the list is deliberately unbounded.StandardJanusGraphTxpassesQuery.NO_LIMITtoprocessIntersectingRetrievals, with a comment explaining that this prevents incomplete intersections and therefore missed results. That makessubLimitInteger.MAX_VALUE, so the whole matching set of every other index is materialised, and the linear scan runs against all of it.Building the set once outside the lambda makes the intersection O(n + m). This is the fix suggested in the issue.
Testing
The change is a performance fix with no intended behaviour change, so
SubqueryIteratorTestpins the behaviour that the conversion must preserve rather than trying to measure time: only intersected ids survive;null(the single-index case) is not treated as an empty intersection; an empty intersection yields nothing; the limit still truncates; a repeated id in the other results is harmless; and the streamed order is preserved regardless of the order the other indexes reported.That last group is worth having because a
Listand aHashSetdiffer in more than lookup cost — the null case and the ordering are exactly what a careless conversion would break.One note on the trade-off
The set is a second structure over ids that are already materialised, so peak memory for that data roughly doubles while the iterator is open. It stays the same order as the list that already exists, and it replaces a quadratic scan, so I think it is clearly worth it.
If you would rather not hold both, the alternative is to have
processIntersectingRetrievalsreturn aLinkedHashSetinstead of anArrayList— it already builds aHashSetinternally for each intersection step, and it has exactly one caller, this one. I did not do that here because it would change the meaning of theresults.size() < limitloop condition when a sub-result contains duplicates, which deserves its own change rather than riding along with this one.For all changes:
master)?For code changes: