Skip to content

Add Result return types to CatalogProvider / SchemaProvider fallible methods #21778

@shyjsarah

Description

@shyjsarah

Is your feature request related to a problem or challenge?

Several methods on CatalogProvider and SchemaProvider perform potentially fallible operations but have no way to report errors:

Trait Method Current signature
CatalogProvider schema_names() -> Vec<String>
CatalogProvider schema() -> Option<Arc<dyn SchemaProvider>>
SchemaProvider table_names() -> Vec<String>
SchemaProvider table_exist() -> bool

For in-memory catalogs this is fine, but remote catalog implementations (e.g. paimon-rust, delta-rs Unity Catalog) need to make network
calls in these methods. When a request fails (403, timeout, connection refused, etc.), the implementation has no choice but to silently return an empty vec / None / false,
which is indistinguishable from "no schemas exist" or "table not found".

This makes debugging very difficult — a network blip looks identical to an empty catalog.

Describe the solution you'd like

Change the return types to Result<...>:

// CatalogProvider
fn schema_names(&self) -> Result<Vec<String>>;
fn schema(&self, name: &str) -> Result<Option<Arc<dyn SchemaProvider>>>;

// SchemaProvider
fn table_names(&self) -> Result<Vec<String>>;
fn table_exist(&self, name: &str) -> Result<bool>;

This is a breaking API change. All existing implementations would need to wrap their return values in Ok(...). The migration is mechanical but touches every catalog implementation.

Describe alternatives you've considered

  1. Logging (current workaround): Add log::error! before returning the fallback value. This surfaces the error for debugging but callers still can't react to it.
  2. async versions of these methods: Would also solve the problem but is a much larger change and has been intentionally avoided for planning performance reasons (as documented in CatalogProvider's doc comments).
  3. Keep as-is: Accept that remote catalogs must silently swallow errors in these methods.

Additional context

The other methods on these traits (table(), register_schema(), deregister_schema(), register_table(), deregister_table()) already return Result, so this would make the API consistent.

Affected files (estimated): ~21 source files + FFI boundary in datafusion-ffi.

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request
No fields configured for Feature.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions