Is your feature request related to a problem or challenge?
Catalog::create_table eagerly creates and registers an empty table. Callers that need Spark-style staged creation cannot use transaction actions to write data before the table becomes visible, then atomically publish the table and its initial snapshot.
The existing Transaction type also assumes that a table already exists: it reloads a current base table and commits updates against its metadata. Table creation has no existing base and must instead fail atomically if another writer creates the table first.
Describe the solution you would like
Add an opt-in transactional catalog capability for staged table creation.
- Add create_table_transaction to return a transaction backed by catalog-staged table metadata.
- Expose the staged Table so normal transaction actions can write against its schema, partition spec, location, and FileIO.
- Keep the table invisible until the final transaction commit.
- Commit initial metadata updates with an assert-create requirement.
- Do not retry a failed create, because a conflicting table must not be replaced or rebased.
- Implement the capability for MemoryCatalog first, with additional catalogs handled separately.
The existing eager Catalog::create_table operation should remain unchanged. This staged-create capability can also serve as the foundation for later CREATE OR REPLACE support.
Willingness to contribute
I can contribute to this feature independently.
Implementation: #3039
Is your feature request related to a problem or challenge?
Catalog::create_table eagerly creates and registers an empty table. Callers that need Spark-style staged creation cannot use transaction actions to write data before the table becomes visible, then atomically publish the table and its initial snapshot.
The existing Transaction type also assumes that a table already exists: it reloads a current base table and commits updates against its metadata. Table creation has no existing base and must instead fail atomically if another writer creates the table first.
Describe the solution you would like
Add an opt-in transactional catalog capability for staged table creation.
The existing eager Catalog::create_table operation should remain unchanged. This staged-create capability can also serve as the foundation for later CREATE OR REPLACE support.
Willingness to contribute
I can contribute to this feature independently.
Implementation: #3039