Skip to content

Add DataFusion parquet range cache#3

Closed
alexanderbianchi wants to merge 1 commit intobianchi/parquet-footer-cachefrom
bianchi/parquet-byte-range-cache
Closed

Add DataFusion parquet range cache#3
alexanderbianchi wants to merge 1 commit intobianchi/parquet-footer-cachefrom
bianchi/parquet-byte-range-cache

Conversation

@alexanderbianchi
Copy link
Copy Markdown
Owner

Summary

Adds a DataFusion-local parquet byte-range cache on top of the footer metadata cache work in quickwit-oss#6388.

The cache is installed through the Quickwit DataFusion object-store bridge and uses Quickwit's existing in-memory MemorySizedCache machinery. Cache keys are scoped by storage URI to avoid collisions across buckets or authorities.

Also adds an internal note describing the path to a follow-up Foyer-backed block/range cache and when that extra complexity is likely justified.

Validation

  • cargo check -p quickwit-serve --features datafusion
  • cargo test -p quickwit-datafusion
  • cargo test -p quickwit-storage --features testsuite
  • cargo check -p quickwit-lambda-server
  • git diff --check

@alexanderbianchi
Copy link
Copy Markdown
Owner Author

Closing this fork-only stacked PR. Re-opening upstream against quickwit-oss/quickwit with both commits included.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant