Skip to content

Add JSON single-value serialization for timestamp_ns and timestamptz_ns - #3855

Open
takayoshi-makabe wants to merge 1 commit into
apache:mainfrom
takayoshi-makabe:fix-nano-timestamp-json-conversion
Open

Add JSON single-value serialization for timestamp_ns and timestamptz_ns#3855
takayoshi-makabe wants to merge 1 commit into
apache:mainfrom
takayoshi-makabe:fix-nano-timestamp-json-conversion

Conversation

@takayoshi-makabe

@takayoshi-makabe takayoshi-makabe commented Aug 26, 2026

Copy link
Copy Markdown

Rationale for this change

TimestampNanoType and TimestamptzNanoType are registered for byte conversion but not for to_json / from_json, so the JSON single-value serialization path raises for them. A nanosecond timestamp field with a default cannot even be constructed:

from pyiceberg.types import NestedField, TimestampNanoType

NestedField(1, "ts", TimestampNanoType(), required=False,
            initial_default=1510871468123456789)
TypeError: Cannot deserialize bytes, type timestamp_ns not supported: 1510871468123456789

NestedField runs from_json on initial-default / write-default during validation, so this fails on construction as well as on model_dump_json() and model_validate_json().

Implementation notes

Per Appendix D both types serialize as ISO-8601 strings with nanosecond precision, matching Java's SingleValueParser:

timestamp_ns    "2017-11-16T22:31:08.123456789"
timestamptz_ns  "2017-11-16T22:31:08.123456789+00:00"

from_json returns an int (epoch nanoseconds) rather than a datetime, because Python's datetime only holds microseconds and would silently drop the last three digits. from_bytes already returns an int for these two types. to_human_timestamp_ns / to_human_timestamptz_ns are new — no existing helper formats at nanosecond precision.

Two things left out of scope: the TypeError above says "Cannot deserialize bytes" even from to_json, a pre-existing copy of from_bytes's message that #3856 fixes separately. And this is independent of #3853, which refactors the *_to_nanos parsers while this adds the reverse direction plus the JSON dispatch — they touch different functions.

Are these changes tested?

Yes.

  • tests/test_conversions.py: both types added to test_json_single_serialization (int and datetime inputs) and test_json_serialize_roundtrip.
  • tests/utils/test_datetime.py: the new formatters, covering the epoch, zero-padded sub-second digits, and a pre-epoch value — each asserted to round-trip back through timestamp_to_nanos / timestamptz_to_nanos.
  • tests/test_types.py: end-to-end NestedField round-trip for initial-default / write-default, which is the path in the report above.

Are there any user-facing changes?

Yes. Nanosecond timestamp fields with an initial-default or write-default now serialize and deserialize instead of raising TypeError.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant