Skip to content

Add blob range caching for DataFusion metrics scans#6389

Open
alexanderbianchi wants to merge 2 commits intoquickwit-oss:bianchi/parquet-footer-cachefrom
alexanderbianchi:bianchi/parquet-byte-range-cache
Open

Add blob range caching for DataFusion metrics scans#6389
alexanderbianchi wants to merge 2 commits intoquickwit-oss:bianchi/parquet-footer-cachefrom
alexanderbianchi:bianchi/parquet-byte-range-cache

Conversation

@alexanderbianchi
Copy link
Copy Markdown
Collaborator

Summary

Adds Parquet IO caching for DataFusion metrics scans.

This PR includes the footer metadata cache work from #6388 and the follow-up DataFusion-local parquet byte-range cache:

  • installs DataFusion's CachedParquetFileReaderFactory for parquet footer metadata reuse;
  • adds a DataFusion object-store cache path backed by Quickwit's in-memory MemorySizedCache;
  • scopes cache keys by storage URI so equivalent object paths in different buckets/authorities do not collide;
  • keeps DataFusion cache sizing behind the datafusion serve setup with a 4 GiB default and QW_DATAFUSION_PARQUET_RANGE_CACHE_CAPACITY override;
  • documents the path to a future Foyer-backed block/range cache in docs/internals/parquet-byte-range-cache-option2.md.

Validation

  • cargo check -p quickwit-serve --features datafusion
  • cargo test -p quickwit-datafusion
  • cargo test -p quickwit-storage --features testsuite
  • cargo check -p quickwit-lambda-server
  • git diff --check

@alexanderbianchi alexanderbianchi force-pushed the bianchi/parquet-byte-range-cache branch 2 times, most recently from 3b85d88 to e5cacdd Compare May 9, 2026 20:15
@alexanderbianchi alexanderbianchi changed the base branch from main to bianchi/parquet-footer-cache May 9, 2026 20:15
@alexanderbianchi alexanderbianchi force-pushed the bianchi/parquet-byte-range-cache branch from e5cacdd to 1d681e5 Compare May 9, 2026 20:30
@alexanderbianchi alexanderbianchi force-pushed the bianchi/parquet-footer-cache branch from 957da25 to 04b4cd3 Compare May 9, 2026 20:30
@alexanderbianchi alexanderbianchi marked this pull request as ready for review May 9, 2026 20:35
@alexanderbianchi alexanderbianchi requested review from a team as code owners May 9, 2026 20:35
@alexanderbianchi
Copy link
Copy Markdown
Collaborator Author

Temporarily closing/reopening to trigger CI on the updated stacked branch.

@alexanderbianchi alexanderbianchi changed the title Add Parquet caching for DataFusion metrics scans Add blob range caching for DataFusion metrics scans May 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant