add table_resolved and schemas_resolved fields

[Light2Dark]

📝 Summary

These fields are to indicate whether a table's schemas has been loaded vs truly empty.
That allows the frontend to distinguish whether to hide empty vs lazy (which we shouldn't hide).

This change makes sense for future work, since it's important to distinguish between truly empty vs not yet loaded. Another option was considered: list[DataTable] | None, where None indicates not yet loaded. However, this is semantically not clear even if a simpler code change.

This is in support of this PR #9708

📋 Pre-Review Checklist

• For large changes, or changes that affect the public API: this change was discussed or approved through an issue, on Discord, or the community discussions (Please provide a link if applicable).
• Any AI generated code has been reviewed line-by-line by the human PR author, who stands by it.
• Video or media evidence is provided for any visual changes (optional).

✅ Merge Checklist

• I have read the contributor guidelines.
• Documentation has been updated where applicable, including docstrings for API changes.
• Tests have been added for the changes made.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
marimo-docs	Ready	Preview, Comment	May 29, 2026 5:11am

[cubic-dev-ai]

2 issues found across 15 files

Architecture diagram

sequenceDiagram
    participant FE as Frontend (React)
    participant API as API Route / Kernel
    participant Engine as SQL Engine (adbc/ibis/clickhouse/...)
    participant DB as Database (DuckDB/Snowflake/Redshift/...)
    participant Store as Connection State Store

    Note over FE,DB: DATABASE EXPLORER - Schema/Table Discovery Flow

    FE->>API: GET /datasources (initial load)
    API->>Engine: get_databases(include_schemas, include_tables)
    Engine->>DB: Fetch catalogs/databases
    DB-->>Engine: Catalog list

    alt include_schemas=True
        Engine->>DB: Fetch schemas for each catalog
        DB-->>Engine: Schema list
        Engine->>Engine: Build Schema objects with schemas_resolved=True
        alt include_tables=True
            Engine->>DB: Fetch tables for each schema
            DB-->>Engine: Table list
            Engine->>Engine: Build DataTable objects with tables_resolved=True
        else include_tables=False (deferred)
            Engine->>Engine: Build Schema with tables=[], tables_resolved=False
        end
    else include_schemas=False (deferred)
        Engine->>Engine: Build Database with schemas=[], schemas_resolved=False
    end

    Engine-->>API: List[Database] with schemas_resolved & tables_resolved flags
    API->>Store: Update connection state with resolved flags
    Store-->>API: Confirmed
    API-->>FE: Serialized Database list

    Note over FE,Store: CLIENT-SIDE FILTERING (hideEmptyDatasources)

    FE->>FE: filterEmptyDatabases(databases)
    alt Database.schemas_resolved != false AND schemas empty
        FE->>FE: Hide database (truly empty catalog)
    else Database.schemas_resolved == false
        FE->>FE: Keep database visible (deferred discovery)
    end

    alt Schema.tables_resolved != false AND tables empty
        FE->>FE: Hide schema (truly empty schema)
    else Schema.tables_resolved == false
        FE->>FE: Keep schema visible (deferred table loading)
    end

    Note over FE: LAZY TABLE LOADING

    FE->>FE: User expands schema node
    FE->>API: GET /datasources/schema-tables (lazy fetch)
    API->>Engine: get_schemas(include_tables=True)
    Engine->>DB: Fetch tables for this schema
    DB-->>Engine: Table list
    Engine-->>API: Updated Schema with tables_resolved=True
    API->>Store: Update schema with resolved tables
    Store-->>API: Confirmed
    API-->>FE: DataTable list with resolved flag
    FE->>FE: Re-filter using updated tables_resolved state

Loading

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[Copilot]

Pull request overview

This PR introduces two backward-compatible “resolved” flags (schemas_resolved on Database, tables_resolved on Schema) so clients can distinguish “truly empty” schema/table lists from “not yet enumerated / deferred discovery” lists, enabling the frontend to avoid hiding lazy-loaded items when “hide empty” is enabled.

Changes:

• Extend backend Database/Schema models with schemas_resolved / tables_resolved (defaulting to True) and propagate these flags through SQL engine introspection paths.
• Update connection update utilities to mark schemas/tables as resolved after a lazy fetch updates the in-memory connection model.
• Update frontend datasource filtering and connection-state update logic to respect/set the new resolved flags; update OpenAPI artifacts and SQL engine tests accordingly.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
marimo/_data/models.py	Adds tables_resolved / schemas_resolved fields (default True) to core data models.
marimo/_sql/connection_utils.py	Marks schema/table lists as resolved when updating a connection after lazy fetch.
marimo/_sql/engines/sqlalchemy.py	Populates schemas_resolved and tables_resolved during introspection.
marimo/_sql/engines/redshift.py	Populates schemas_resolved and tables_resolved during introspection.
marimo/_sql/engines/ibis.py	Threads schemas_resolved / tables_resolved through Ibis introspection.
marimo/_sql/engines/adbc.py	Threads resolved flags through ADBC metadata enumeration.
marimo/_sql/engines/clickhouse.py	Sets tables_resolved depending on whether tables were introspected for a database.
marimo/_sql/engines/pyiceberg.py	Sets tables_resolved based on whether table discovery ran.
frontend/src/components/datasources/datasources.tsx	Updates “hide empty” filtering to treat unresolved empties as “unknown” (keep visible).
frontend/src/core/datasets/data-source-connections.ts	Ensures lazy-loaded schema/table updates set schemas_resolved / tables_resolved to true.
packages/openapi/api.yaml	Updates OpenAPI schema/docs for the new optional flags (default true).
packages/openapi/src/api.ts	Updates generated TS types/docs for the new optional flags (default true).
tests/_sql/test_sqlalchemy.py	Updates expectations to cover the new resolved flags.
tests/_sql/test_ibis.py	Adds coverage for empty-vs-deferred semantics using the new flags.

[cubic-dev-ai]

1 issue found across 15 files

Architecture diagram

sequenceDiagram
    participant UI as Frontend UI
    participant Filter as filterEmptyDatabases()
    participant Model as Data Model (Schema/Database)
    participant Engine as SQL Engine Layer
    participant DB as Database

    Note over UI,DB: Data Source Tree Rendering

    UI->>Engine: getDatabases(include_schemas, include_tables)
    
    alt include_schemas=True, include_tables=True
        Engine->>DB: Full metadata query
        DB-->>Engine: Raw schemas & tables
        Engine->>Engine: Build Database with schemas_resolved=True
        Engine->>Engine: Build Schema with tables_resolved=True
    else include_schemas=True, include_tables=False
        Engine->>DB: Schema names only
        DB-->>Engine: Schema list
        Engine->>Engine: Build Database with schemas_resolved=True
        Engine->>Engine: Build Schema with tables=[], tables_resolved=False
    else include_schemas=False
        Engine-->>UI: Database with schemas=[], schemas_resolved=False
    end

    Engine-->>UI: Database list with resolved flags

    UI->>Filter: filterEmptyDatabases(databases)

    loop For each database
        alt schemas_resolved == false
            Filter-->>UI: Keep database (deferred)
        else schemas_resolved == true AND schemas.length == 0
            Filter-->>UI: Hide database (confirmed empty)
        else schemas.length > 0
            loop For each schema
                alt tables_resolved == false
                    Filter-->>Filter: Keep schema (deferred)
                else tables_resolved == true AND tables.length == 0
                    Filter-->>Filter: Hide schema (confirmed empty)
                else tables.length > 0
                    Filter-->>Filter: Keep schema (has tables)
                end
            end
            alt All schemas hidden
                Filter-->>UI: Hide database
            else Some schemas kept
                Filter-->>UI: Database with visible schemas
            end
        end
    end

    Note over UI: User expands a deferred node
    
    alt User clicks deferred schema
        UI->>Engine: getTablesForSchema(schema)
        Engine->>DB: Query tables
        DB-->>Engine: Table list
        Engine->>Engine: Update schema.tables, set tables_resolved=True
        Engine-->>UI: Updated schema
        UI->>UI: Re-render with table names
    else User clicks deferred database
        UI->>Engine: getSchemasForDatabase(database)
        Engine->>DB: Query schemas
        DB-->>Engine: Schema list
        Engine->>Engine: Update database.schemas, set schemas_resolved=True
        Engine-->>UI: Updated database
        UI->>UI: Re-render with schema list
    end

    Note over UI,Filter: Backward Compatibility
    Note over Model: Old payloads missing resolved flags<br/>default to True (resolved)

Loading

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>