What is the problem the feature request solves?
Motivation
Comet today accepts only s3:// and s3a:// URIs, which effectively couples native execution to AWS S3 and the Hadoop s3a connector. In practice, most production Spark deployments live on S3-compliant
storage that speaks the S3 protocol against a non-AWS endpoint. Users on these systems either lose Comet acceleration entirely or have to rewrite their table locations to fit the s3a mold.
Common S3-compliant systems Spark users route through today:
- Google Cloud Storage via its S3-compatible interoperability endpoint (
storage.googleapis.com).
- Cloudflare R2, Backblaze B2, Wasabi, DigitalOcean Spaces, Linode Object Storage.
- MinIO (self-hosted and as the storage layer for many Kubernetes / on-prem data platforms).
- Ceph RADOS Gateway, SeaweedFS, OpenStack Swift (with the S3 middleware).
- IBM Cloud Object Storage, Oracle Cloud Object Storage (S3 compatibility API).
- Dell ECS, NetApp StorageGRID, Pure FlashBlade and other enterprise on-prem object stores.
- Internal / vendor-specific S3-compliant blob services that many organizations expose through a custom
blob:// (or similarly aliased) Hadoop FileSystem.
These systems share a common shape: S3 protocol, custom endpoint, usually path-style addressing, frequently no meaningful region, and often a vendor-specific config namespace instead of fs.s3a.*. The
current gate treats any of these as "unsupported filesystem" and falls back to Spark, which defeats the purpose of Comet on precisely the deployments that would benefit most.
Goal
Allow Comet to natively read (and Iceberg-scan) tables backed by any S3-compliant object store, regardless of the URI scheme the user's Hadoop FileSystem exposes or which vendor namespace they configure
credentials under. Behavior on AWS S3 and the standard s3a connector must remain unchanged.
Out of scope
- Native writes to S3-compliant stores (there is no S3 write path today).
- Non-S3 protocols (GCS native HMAC / OAuth, Azure Blob / ADLS Gen2). Those already have their own paths and are not affected.
Compatibility
Additive. Users on AWS S3 or s3a see no change. Users on S3-compliant storage go from "falls back to Spark" to "runs natively in Comet".
Describe the potential solution
No response
Additional context
No response
What is the problem the feature request solves?
Motivation
Comet today accepts only
s3://ands3a://URIs, which effectively couples native execution to AWS S3 and the Hadoops3aconnector. In practice, most production Spark deployments live on S3-compliantstorage that speaks the S3 protocol against a non-AWS endpoint. Users on these systems either lose Comet acceleration entirely or have to rewrite their table locations to fit the
s3amold.Common S3-compliant systems Spark users route through today:
storage.googleapis.com).blob://(or similarly aliased) HadoopFileSystem.These systems share a common shape: S3 protocol, custom endpoint, usually path-style addressing, frequently no meaningful region, and often a vendor-specific config namespace instead of
fs.s3a.*. Thecurrent gate treats any of these as "unsupported filesystem" and falls back to Spark, which defeats the purpose of Comet on precisely the deployments that would benefit most.
Goal
Allow Comet to natively read (and Iceberg-scan) tables backed by any S3-compliant object store, regardless of the URI scheme the user's Hadoop
FileSystemexposes or which vendor namespace they configurecredentials under. Behavior on AWS S3 and the standard
s3aconnector must remain unchanged.Out of scope
Compatibility
Additive. Users on AWS S3 or
s3asee no change. Users on S3-compliant storage go from "falls back to Spark" to "runs natively in Comet".Describe the potential solution
No response
Additional context
No response