feat(io): make FileIO serializable

[blackmwk]

Which issue does this PR close?

• Closes Make FileIO serializable. #3088.

What changes are included in this PR?

• Add explicit FileIO::serialize_all and FileIO::deserialize_all byte APIs without implementing serde's public Serialize or Deserialize traits for FileIO.
• Serialize the storage factory and configuration while excluding the process-local lazy storage cache.
• Rebuild the storage cache lazily after deserialization.
• Document that the version-unstable payload includes every storage property, including credentials, and must be protected in transit and at rest.
• Document typetag registration and matching OpenDAL backend-feature requirements on the receiving process.
• Reject serialization when an OpenDAL S3 factory contains a process-local custom credential loader.

Are these changes tested?

• Kept core memory and local-filesystem roundtrip tests as unit tests with I/O after deserialization.
• Added backend integration coverage with I/O after deserialization for OpenDAL memory, filesystem, S3, GCS, HuggingFace, and resolving storage.
• Added unit tests confirming that direct S3 and resolving factories reject custom credential-loader serialization.
• cargo test -p iceberg io::file_io::tests --lib
• cargo test -p iceberg-storage-opendal --all-features --lib custom_credential_loader
• cargo test -p iceberg-storage-opendal --all-features --test file_io_memory_test --test file_io_fs_test
• RUSTDOCFLAGS='-D warnings' cargo doc -p iceberg-storage-opendal --no-default-features --no-deps
• cargo check -p iceberg-storage-opendal --no-default-features --all-targets
• cargo clippy --all-targets --all-features --workspace -- -D warnings
• cargo test --doc --all-features --workspace
• cargo machete
• cargo fmt --all -- --check
• cargo public-api -p iceberg --all-features -ss | diff - crates/iceberg/public-api.txt

AI Disclosure

Codex was used to help implement the change, add the regression tests and documentation, and run verification. The resulting code, documentation, and test behavior were reviewed before submission.

[laskoviymishka]

Nice, the mechanical wiring here is clean. The #[serde(skip)] on the cache with the OnceLock rebuilding lazily is exactly right, and the roundtrip does what it says.

The one thing I'd want to settle before this merges is the credential story. Serializing a FileIO emits the whole StorageConfig.props map verbatim, so an S3-configured one writes its secret access key and session token into the output in plaintext. The stated use case is shipping these across process boundaries, which is exactly where a plaintext credential blob is most dangerous — and because these serde impls are now public API, the format is hard to change later. I'd like the serialized form to be safe by default before we commit to it.

Stepping back a little: did we consider serializing just the StorageConfig plus a factory discriminant and reconstructing through FileIOBuilder, rather than deriving on FileIO directly? That's the pattern the REST catalog already uses to rebuild FileIO from properties, and it's how Java/PyIceberg/iceberg-go all handle it — config in, live object rebuilt. It keeps runtime state out of the wire format and gives us a natural spot to redact. Not a hard blocker, but worth weighing before this becomes API.

A few smaller things I left inline: the custom credential loader gets silently dropped on roundtrip, the new serde behavior (typetag registry, ephemeral storage) needs docs, and the test only exercises MemoryStorageFactory so it doesn't really prove roundtrip fidelity.

Once the credential handling's settled, happy to take another pass and approve.

[CTTY]

LGTM!

[blackmwk]

cc @laskoviymishka Comments address, PTAL

[laskoviymishka]

This is really close now — thanks for the thorough revision.

The one thing still holding me is credential safe-by-default, and it's narrower than last round. I flagged the lock-in risk before; the unstable-format disclaimer plus dropping the exact-shape test pins settles that, so I'm no longer worried the format is hard to change. What's left is that the loader now hard-errors on serialize while the secret strings — s3.secret-access-key, s3.session-token, GCS service-account JSON, hf.token — still pass through in plaintext (and test_memory_file_io_serialization_roundtrip pins that). The asymmetry reads backwards: we fail fast on the opaque loader handle and silently emit the values most dangerous to leak. I'd really like by-default redaction of the well-known credential keys before we merge, with full-fidelity passthrough as an explicit opt-in.

Everything else I asked for last round is in:

• loader no longer silently dropped — it hard-fails serialization now, with fail-fast tests for both OpenDalStorageFactory::S3 and OpenDalResolvingStorageFactory (stronger than I asked for)
• serialization docs on FileIO, OpenDalStorageFactory, and OpenDalResolvingStorageFactory, including the ephemeral-storage and credential-sensitivity notes
• real-I/O roundtrip tests across memory / fs / s3 / gcs / hf / resolving, replacing the old shape-pinning tests
• the unstable-format disclaimer

I left a few smaller notes inline — a serde feature gate I'd like while it's cheap, a couple of doc/CI catches, and one dead match arm — but none of those block. Sort the credential default and I'm happy to approve.

[blackmwk]

hi, @laskoviymishka thanks for review, I addressed all comments, PTAL.

What's left is that the loader now hard-errors on serialize while the secret strings — s3.secret-access-key, s3.session-token, GCS service-account JSON, hf.token — still pass through in plaintext (and test_memory_file_io_serialization_roundtrip pins that). The asymmetry reads backwards: we fail fast on the opaque loader handle and silently emit the values most dangerous to leak. I'd really like by-default redaction of the well-known credential keys before we merge, with full-fidelity passthrough as an explicit opt-in.

While I agree that we should redact sensitive credentials, I don't think maintaining a predefine set of well know keys is the right direction. I think #3129 is the right direction to go, e.g. replacing raw hash map with a config struct. For now I've removed the derived easy to use ser/de traits, and ask user to use explict serialize_all methods to call it. WYDT?

[blackmwk]

Thanks @laskoviymishka and @CTTY for review.