add pagination & remote search flow for remote storage

[Light2Dark]

📝 Summary

Fixes #9662. This PR introduces a "Load more" button, that when clicked fetches more entries under the remote storage panel.

It also surfaces this button when a prefix search produces no entries, eg. folder/filename. A bug with Google Drive's file id has been fixed too.

Screen.Recording.2026-05-28.at.10.24.22.AM.mov

📋 Pre-Review Checklist

• For large changes, or changes that affect the public API: this change was discussed or approved through an issue, on Discord, or the community discussions (Please provide a link if applicable).
• Any AI generated code has been reviewed line-by-line by the human PR author, who stands by it.
• Video or media evidence is provided for any visual changes (optional).

✅ Merge Checklist

• I have read the contributor guidelines.
• Documentation has been updated where applicable, including docstrings for API changes.
• Tests have been added for the changes made.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
marimo-docs	Ready	Preview, Comment	Jun 9, 2026 7:26am

[cubic-dev-ai]

2 issues found across 16 files

Architecture diagram

sequenceDiagram
    participant UI as React UI (StorageInspector)
    participant Hook as useStorageEntries Hook
    participant State as Jotai Store (state.ts)
    participant API as Server API (commands.py)
    participant Backend as StorageBackend (storage.py)
    participant Obstore as Obstore (S3/GCS)
    participant Fsspec as FsspecFilesystem (local/SFTP)

    Note over UI,Fsspec: Storage List with Pagination

    UI->>Hook: Expand directory or section
    Hook->>State: Check cache (entriesByPath)
    alt Cache miss
        State-->>Hook: No cached entries
        Hook->>API: NEW: listEntries(namespace, prefix, limit, pageToken=null)
        API->>Backend: list_entries(prefix, limit, page_token)
        
        alt Obstore backend
            Backend->>Obstore: list_with_delimiter(prefix)
            Obstore-->>Backend: Raw entries (may be truncated at 1000)
            Backend->>Backend: _paginate_entries(offset=0, limit)
            Backend-->>API: StorageListResult(entries, nextPageToken, mayHaveMore)
        else Fsspec backend
            Backend->>Fsspec: ls(prefix, detail=True)
            Fsspec-->>Backend: File list
            Backend->>Backend: _paginate_entries(offset=0, limit)
            Backend-->>API: StorageListResult(entries, nextPageToken)
        end
        
        API-->>Hook: NEW: StorageEntriesNotification(entries, nextPageToken, mayHaveMore)
        Hook->>State: NEW: setEntries({entries, nextPageToken, mayHaveMore})
        State-->>Hook: entries, hasMore, mayHaveMore
        Hook-->>UI: Render entries + "Load more" button
    else Cache hit
        State-->>Hook: Cached entries + pagination metadata
        Hook-->>UI: Render entries
        alt hasMore (nextPageToken != null)
            UI->>Hook: Show "Load more" button
        else mayHaveMore (no token, but may exist)
            UI->>Hook: Show "May exist more" indicator
        end
    end

    Note over UI,Hook: Load More Click

    alt hasMore is true
        UI->>Hook: loadMore()
        Hook->>Hook: setIsLoadingMore(true)
        Hook->>API: NEW: listEntries(namespace, prefix, limit, pageToken=nextPageToken)
        API->>Backend: list_entries(prefix, limit, page_token)
        
        alt Obstore backend
            Backend->>Obstore: list_with_delimiter(prefix)
            Obstore-->>Backend: Raw entries
            Backend->>Backend: _paginate_entries(offset=parseInt(pageToken), limit)
            Backend-->>API: StorageListResult(entries, nextPageToken, mayHaveMore)
        else Fsspec backend
            Backend->>Fsspec: ls(prefix, detail=True)
            Fsspec-->>Backend: File list
            Backend->>Backend: _paginate_entries(offset=parseInt(pageToken), limit)
            Backend-->>API: StorageListResult(entries, nextPageToken)
        end
        
        API-->>Hook: NEW: StorageEntriesNotification(entries, nextPageToken, mayHaveMore)
        Hook->>State: NEW: setEntries({entries, append:true, nextPageToken, mayHaveMore})
        State-->>Hook: Appended entries + updated metadata
        Hook-->>UI: Render appended entries
        alt Has more pages
            UI->>Hook: Keep "Load more" button
        else No more pages but mayHaveMore
            UI->>Hook: Show "Maybe more" indicator
        end
    end

    Note over Hook,State: Error Handling

    alt Load more fails
        Hook->>Hook: catch error -> setLoadMoreError
        Hook-->>UI: Show error message with retry
        UI->>Hook: loadMore() retry
    end

Loading

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

2 issues found across 6 files (changes from recent commits).

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[Copilot]

Pull request overview

This PR addresses remote storage listings being effectively truncated (e.g., delimiter-based object store listings commonly cap results), by introducing a pagination model plus a “remote search continuation” flow so users can browse/search beyond the first page of results (Fixes #9662).

Changes:

• Add page-based listing support to the runtime storage command/notification flow (page_token request; next_page_token + may_have_more response metadata).
• Implement offset-based pagination and provider-truncation detection in external storage backends (obstore/fsspec) via a StorageListResult return type.
• Update tests (backend + runtime + frontend) to cover pagination metadata, “load more”, and remote-search continuation behavior.

Reviewed changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
marimo/_data/_external_storage/models.py	Introduces StorageListResult and updates the storage backend interface to return paginated results.
marimo/_data/_external_storage/storage.py	Implements offset-based pagination helpers and backend-specific listing behavior, including may_have_more for provider truncation.
marimo/_runtime/commands.py	Extends StorageListEntriesCommand with page_token.
marimo/_runtime/callbacks/external_storage.py	Threads pagination parameters through to backends and emits pagination metadata in StorageEntriesNotification.
marimo/_messaging/notification.py	Adds next_page_token and may_have_more fields to StorageEntriesNotification.
marimo/_server/models/models.py	Ensures server request models forward page_token into runtime commands.
packages/openapi/api.yaml	Updates OpenAPI schemas for new pagination request/notification fields.
packages/openapi/src/api.ts	Updates generated TS API types to match the OpenAPI pagination schema changes.
frontend/src/core/storage/types.ts	Adds pagination metadata types and updates storageUrl signature.
frontend/src/core/storage/state.ts	Stores per-path pagination metadata and implements “load more” fetching/append behavior.
frontend/src/components/storage/storage-inspector.tsx	Adds UI/logic for “Load more”, “may have more” messaging, and remote-search continuation flow.
tests/_data/_external_storage/test_storage_models.py	Adds/updates backend tests for StorageListResult pagination tokens and provider-boundary signaling.
tests/_runtime/test_runtime_external_storage.py	Adds runtime-level tests for paging requests and notifications carrying pagination metadata.
frontend/src/core/storage/tests/types.test.ts	Updates tests for the new storageUrl signature/behavior.
frontend/src/core/storage/tests/state.test.ts	Updates storage state tests to validate pagination metadata storage and cache behavior.
frontend/src/core/storage/tests/useStorageEntries.test.tsx	Adds coverage for loadMore behavior, append semantics, and pagination metadata handling.
frontend/src/components/storage/tests/storage-inspector.test.ts	Adds coverage for remote-search prefix derivation and filtering behavior.

[Light2Dark]

The PR is quite lengthy, sorry about that. Might be worth just testing if the UX is fine.

The following code snippets connects to your own Google Drive. Or to a public S3 bucket.

from gdrive_fsspec import GoogleDriveFileSystem

fs = GoogleDriveFileSystem(use_listings_cache=False)

import os

for key in [
    "AWS_ACCESS_KEY_ID",
    "AWS_SECRET_ACCESS_KEY",
    "AWS_SESSION_TOKEN",
    "AWS_PROFILE",
]:
    os.environ.pop(key, None)
os.environ["AWS_SKIP_SIGNATURE"] = "true"
from obstore.store import S3Store

store = S3Store(
    "noaa-goes16",
    region="us-east-1",
    skip_signature=True,
)