feat(rest): support rest catalog for databases, tables and snapshots

[SteNicholas]

Purpose

Linked issue: #92

Add a REST catalog implementation, selected via the metastore=rest option, that talks the Paimon REST catalog open API.

New components

• RestHttpClient — blocking libcurl client with exponential-backoff retries:
  • 429/503 responses are retried for all methods, transport errors only for idempotent methods via an explicit allowlist of transient failures (name-resolution and connect failures, timeouts, TLS failures and a server closing the connection without responding are never retried).
  • A Retry-After response header — delta-seconds or HTTP-date form, parsed locale-independently — takes precedence over the backoff when it yields a positive delay. Every retry sleep is bounded by a per-sleep cap and an overall retry budget; a Retry-After beyond either bound stops retrying rather than sleeping less than the server requested.
  • Redirects keep the method and body of POST/DELETE requests (CURLOPT_POSTREDIR) and are restricted to http(s) targets.
  • Header names must be valid HTTP tokens and header values must not contain CR/LF/NUL; a violating header fails the request before anything is sent.
  • Easy handles are pooled and reset between requests, so a request reuses the connection opened by the previous one instead of paying a TCP and TLS handshake every time; TLS below 1.2 is refused and requests carry a default paimon-cpp user agent that a header.User-Agent option overrides.
  • Per-request logging carries the request id (debug on success, warn on a retry or a final failure).
• RestApi — HTTP + JSON layer:
  • /v1/config option merging (overrides > client options > defaults), header. options sent as request headers, and paged listing.
  • Error mapping to Status (404 → NotExist, 409 → Exist, 400 → Invalid, 501 → NotImplemented) that prefers the code of the parsed error body over the http status.
  • Sensitive server messages are redacted, and response bodies are never echoed into error messages (they may carry credentials), so a body that is not an error object is reported as unparsable, distinct from an error object carrying no message.
  • Errors carry the request id (x-request-id, falling back to any request-id header) and attach a RestErrorDetail with the mapped code, so callers can distinguish e.g. authentication failures programmatically.
• RestCatalog — database/table create/drop/rename/list/get, snapshot listing, table-default. option defaults on table creation, and system/branch table checks:
  • The server table response is converted into a TableSchema with the computed highestFieldId: ids at or above SpecialFieldIds::SYSTEM_FIELD_ID_START are excluded, a duplicated field id at any nesting level is rejected, and a missing or wrong-typed partitionKeys/primaryKeys/options member fails instead of silently defaulting.
  • Branch identifiers are passed through to the server, which resolves the branch and returns the branch's own schema (the default branch main maps case-insensitively to the bare table); snapshots of a branch are listed under the branch object name.
  • The sys database serves the local global system tables like FileSystemCatalog.
• Bear token authentication provider (token.provider=bear, the protocol's historical spelling of "bearer").

Shared code

• The pieces overlapping with the object-store file systems are unified under common/utils: the generic HTTP client moved there from common/fs and is shared by the S3 file system and the REST catalog, UrlUtils carries both URL-encoding flavors (form-urlencoded for the REST api, RFC 3986 for S3) with the S3 client reusing it, and libcurl detection in CMake is a single find_package(CURL) gated on PAIMON_ENABLE_S3 OR PAIMON_ENABLE_REST.
• CatalogUtils holds the system-database and system/branch table checks shared by FileSystemCatalog and RestCatalog, with the check order and messages aligned with the Java CatalogUtils (Cannot 'createTable' for system table ...). FileSystemCatalog now also rejects branch identifiers on create/drop/rename — a branch identifier resolves to the main table directory, so dropping t$branch_b would otherwise delete the whole table.
• The new SensitiveConfigUtils centralizes credential redaction with the Java SensitiveConfigUtils key markers and masking rule (matched on the key lower-cased with separators removed; a key naming a true secret is masked as a whole, while an identifier-like key such as dlf.access-key-id keeps at most a four-character tail of a long enough value). sys.catalog_options masks every row through it so credentials such as the bearer token never surface to users who can query that table, and a server error message carrying a secret marker is redacted as a whole.
• RestUtil::ExtractRequestId and RestAuthParameter::Create likewise hold the request-id lookup and the query-parameter encoding in one place, so the log line and the error message report the same id and no call site can sign an unencoded parameter.

Build

libcurl is used from the system rather than bundled, so the new CMake option PAIMON_ENABLE_REST defaults to OFF, consistent with the other optional components that need something outside the bundled third-party toolchain (jindo, lance, lumina, lucene, tantivy). CI enables it explicitly in ci/scripts/build_paimon.sh and installs the libcurl development package, so the REST code and its tests are still built and run on every job. With the option off, Catalog::Create reports that metastore=rest requires a build with PAIMON_ENABLE_REST=ON.

Follow-ups (not in this PR): DLF signing, data-token file IO, paged listing parameters (maxResults/name patterns), and the remaining endpoints (alter/commit/views/partitions/tags/functions).

Tests

New rest_test binary (62 cases, registered under the unittest label so it runs in CI), all against an in-process mock HTTP/REST server:

• RestHttpClientTest: uri normalization, query encoding, a pooled handle carrying no method or body over from the previous request, retry classification (429/503 retried, 404 and non-allowlisted transport errors such as a refused connection or a redirect loop not, truncated responses retried only for idempotent methods), Retry-After precedence in both forms plus the per-sleep and overall budget bounds, retry exhaustion, redirect following with POST keeping its method and body, rejection of invalid header names/values, and transport errors omitting the url and query.
• RestUtilTest / ResourcePathsTest / RestMessagesTest, each next to its implementation: prefix extraction and the request-id lookup including the gateway-header fallback, resource path building with url-encoded segments, JSON round trips of all request/response messages, and config merge with null-value filtering (a key present in defaults, client options and overrides at once, plus the null-override and null-default paths).
• RestCatalogTest: client-side option validation (missing uri/token, unsupported token provider), end-to-end catalog operations including pagination, ignore_if_exists / ignore_if_not_exists paths, client/server headers and the json content type, table-default. defaults, system tables, branch tables resolved by the server, broken-schema rejection (missing and wrong-typed partitionKeys/primaryKeys/options members), a non-empty partition-key round trip, server errors surfacing as errors instead of "does not exist", and snapshot listing with sorting.
• RestApiErrorTest: http-status-to-Status mapping including the request-id fallback, redaction and the RestErrorDetail code.

The shared pieces are covered next to their implementations:

• UrlUtilsTest: both encoding flavors, including that a literal % is encoded even when / is preserved, which is what the S3 client relies on.
• SensitiveConfigUtilsTest: key classification, whole-value and tail-preserving masking, and free-form text redaction.
• HttpClientUtilTest in the common utils suite.
• TableSchemaTest.TestComputeHighestFieldId: the highest-field-id computation.
• SnapshotTest.TestToSnapshotInfo: the snapshot-to-info conversion.
• FileSystemCatalogTest.TestBranchIdentifierRejectedForTableOperations: the branch rejection.
• The sys.catalog_options integration test: credential-carrying options (token, access-key secret, account key, credential, SAS) are masked in every emitted row, and an access-key id keeps only its four-character tail.

API and Format

• New public header include/paimon/catalog_options.h with a PAIMON_EXPORTed CatalogOptions struct holding the catalog-level option keys METASTORE, URI, TOKEN, TOKEN_PROVIDER and the TABLE_DEFAULT_OPTION_PREFIX (table-default.) key prefix; table-level keys stay in Options (defs.h), mirroring the CatalogOptions/CoreOptions separation of the Java implementation.
• RestCatalog implements the Catalog interface including GetOptions(), which returns the client options merged with the server-side /v1/config.
• Catalog::Create now dispatches on the metastore option (filesystem remains the default; unknown values are rejected). For metastore=rest, root_path is not a filesystem path but the warehouse (instance) name registered on the REST server, documented on the API.
• No storage format changes. The wire protocol is the Paimon REST catalog open API.

Documentation

• docs/source/user_guide/catalog.rst now documents both metastores: the existing filesystem metastore section and a new REST Catalog section covering the PAIMON_ENABLE_REST build requirement, the root_path semantics (warehouse/instance name), the CatalogOptions keys (metastore, uri, token.provider, token, table-default.<key>) and a configuration example, replacing the note that claimed REST catalog support is future work. It also states which parts of the REST catalog are not implemented yet (altering a database or a table, views, functions, partitions, tags, branch management, consumers and the dlf token provider), so the guide is not read as promising them.
• docs/source/building.rst lists -DPAIMON_ENABLE_REST=ON under the optional components, including its libcurl requirement.

Generative AI tooling

Generated-by: Claude Code (claude-fable-5, claude-opus-5)

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Note

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Adds a new REST-backed catalog implementation selectable via metastore=rest, including HTTP client + auth + REST API layers plus unit tests and build/CI wiring.

Changes:

• Introduces HttpClient, RestApi, RestCatalog, auth provider, message models, and URL/path utilities for the REST catalog protocol.
• Adds an in-process mock REST server and a new REST unit test binary with coverage across retry logic, message round-trips, and catalog operations.
• Extends catalog factory/options and updates CMake/CI to optionally build/link REST support (libcurl) and run REST tests.

Reviewed changes

Copilot reviewed 31 out of 31 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
src/paimon/rest/rest_util.h	Declares REST helper utilities (URL encode/decode, JSON helpers, prefix extraction).
src/paimon/rest/rest_util.cpp	Implements URL encoding/decoding, prefix extraction, and rapidjson serialization helpers.
src/paimon/rest/rest_messages_test.cpp	Adds unit tests for REST utilities, paths, and JSON message round-trips.
src/paimon/rest/rest_messages.h	Adds REST protocol request/response models (JSON-serializable).
src/paimon/rest/rest_messages.cpp	Implements JSON serialization/deserialization and config merge behavior.
src/paimon/rest/rest_catalog_test.cpp	End-to-end tests for REST catalog behavior against a mock server.
src/paimon/rest/rest_catalog.h	Declares RestCatalog implementing database/table operations + snapshot listing.
src/paimon/rest/rest_catalog.cpp	Implements REST catalog behavior and schema conversion/highestFieldId logic.
src/paimon/rest/rest_auth.h	Declares REST auth provider interface and bearer-token provider.
src/paimon/rest/rest_auth.cpp	Implements bearer-token header injection and provider selection via options.
src/paimon/rest/rest_api.h	Declares REST API layer (HTTP+JSON) and error mapping to Status.
src/paimon/rest/rest_api.cpp	Implements config fetching/merging, request execution, pagination, and error mapping.
src/paimon/rest/resource_paths.h	Declares URL path builder for REST endpoints with optional prefix.
src/paimon/rest/resource_paths.cpp	Implements REST endpoint path construction with URL-encoded segments.
src/paimon/rest/mock_rest_server.h	Declares a minimal blocking HTTP server for REST unit tests.
src/paimon/rest/mock_rest_server.cpp	Implements the mock HTTP server (parsing, dispatch, responses).
src/paimon/rest/http_client_test.cpp	Adds unit tests for URI normalization, query encoding, and retry behavior.
src/paimon/rest/http_client.h	Declares blocking libcurl client with retry/backoff behavior.
src/paimon/rest/http_client.cpp	Implements libcurl execution, retry classification, backoff, and header parsing.
src/paimon/core/snapshot.h	Adds SnapshotInfo include and Snapshot::ToSnapshotInfo() declaration.
src/paimon/core/snapshot.cpp	Implements Snapshot::ToSnapshotInfo() commit-kind mapping and field transfer.
src/paimon/core/catalog/file_system_catalog.cpp	Refactors snapshot listing to reuse Snapshot::ToSnapshotInfo().
src/paimon/core/catalog/catalog.cpp	Adds metastore dispatch for filesystem vs rest (guarded by PAIMON_ENABLE_REST).
src/paimon/common/defs.cpp	Defines new option keys: metastore, uri, token, token.provider.
src/paimon/CMakeLists.txt	Adds REST sources/linking behind PAIMON_ENABLE_REST and registers REST tests.
include/paimon/defs.h	Exposes new public options for selecting/configuring the REST catalog.
ci/scripts/build_paimon.sh	Forces -DPAIMON_ENABLE_REST=ON in CI build script.
CMakeLists.txt	Adds PAIMON_ENABLE_REST option, finds libcurl, and defines compile macro.
.github/workflows/gcc8_test.yaml	Installs libcurl dev package for GCC8 CI.
.github/workflows/build_and_test.yaml	Installs libcurl dev package for general CI builds/tests.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[zjw1111]

Thanks for putting this together — the alignment with the Java implementation is impressively thorough (config merge precedence, retry classification, isSuccessful status set, paging termination, audit option keys all match).

One small consistency point on snapshot listing, left inline.