You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "takayoshi-makabe (via GitHub)" <gi...@apache.org> on 2026/08/26 01:51:29 UTC
[PR] Add JSON single-value serialization for timestamp_ns and timestamptz_ns [iceberg-python]
takayoshi-makabe opened a new pull request, #3855:
URL: https://github.com/apache/iceberg-python/pull/3855
<!--
Thanks for opening a pull request!
-->
<!-- In the case this PR will resolve an issue, please replace ${GITHUB_ISSUE_ID} below with the actual Github issue id. -->
<!-- Closes #${GITHUB_ISSUE_ID} -->
# Rationale for this change
`TimestampNanoType` and `TimestamptzNanoType` are registered for byte conversion but not for `to_json` / `from_json`, so the JSON single-value serialization path raises for them. A nanosecond timestamp field with a default cannot even be constructed:
```python
from pyiceberg.types import NestedField, TimestampNanoType
NestedField(1, "ts", TimestampNanoType(), required=False,
initial_default=1510871468123456789)
```
```shell
TypeError: Cannot deserialize bytes, type timestamp_ns not supported: 1510871468123456789
```
`NestedField` runs `from_json` on `initial-default` / `write-default` during validation, so this fails on construction as well as on `model_dump_json()` and `model_validate_json()`.
## Implementation notes
Per [Appendix D](https://iceberg.apache.org/spec/#json-single-value-serialization) both types serialize as ISO-8601 strings with nanosecond precision, matching Java's `SingleValueParser`:
```
timestamp_ns "2017-11-16T22:31:08.123456789"
timestamptz_ns "2017-11-16T22:31:08.123456789+00:00"
```
`from_json` returns an `int` (epoch nanoseconds) rather than a `datetime`, because Python's `datetime` only holds microseconds and would silently drop the last three digits. `from_bytes` already returns an `int` for these two types. `to_human_timestamp_ns` / `to_human_timestamptz_ns` are new — no existing helper formats at nanosecond precision.
Two things left out of scope: the `TypeError` above says "Cannot deserialize bytes" even from `to_json`, which is a pre-existing copy/paste from `from_json` and unrelated to this fix. And this is independent of #3853, which refactors the `*_to_nanos` parsers while this adds the reverse direction plus the JSON dispatch — they touch different functions.
## Are these changes tested?
Yes.
- `tests/test_conversions.py`: both types added to `test_json_single_serialization` (int and `datetime` inputs) and `test_json_serialize_roundtrip`.
- `tests/utils/test_datetime.py`: the new formatters, covering the epoch, zero-padded sub-second digits, and a pre-epoch value — each asserted to round-trip back through `timestamp_to_nanos` / `timestamptz_to_nanos`.
- `tests/test_types.py`: end-to-end `NestedField` round-trip for `initial-default` / `write-default`, which is the path in the report above.
## Are there any user-facing changes?
Yes. Nanosecond timestamp fields with an `initial-default` or `write-default` now serialize and deserialize instead of raising `TypeError`.
<!-- In the case of user-facing changes, please add the changelog label. -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Add JSON single-value serialization for timestamp_ns and timestamptz_ns [iceberg-python]
Posted by "github-actions[bot] (via GitHub) via issues" <is...@iceberg.apache.org>.
github-actions[bot] closed pull request #3855: Add JSON single-value serialization for timestamp_ns and timestamptz_ns
URL: https://github.com/apache/iceberg-python/pull/3855
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Add JSON single-value serialization for timestamp_ns and timestamptz_ns [iceberg-python]
Posted by "github-actions[bot] (via GitHub) via issues" <is...@iceberg.apache.org>.
github-actions[bot] commented on PR #3855:
URL: https://github.com/apache/iceberg-python/pull/3855#issuecomment-6536367921
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that's incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Add JSON single-value serialization for timestamp_ns and timestamptz_ns [iceberg-python]
Posted by "github-actions[bot] (via GitHub) via issues" <is...@iceberg.apache.org>.
github-actions[bot] commented on PR #3855:
URL: https://github.com/apache/iceberg-python/pull/3855#issuecomment-6714974135
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Add JSON single-value serialization for timestamp_ns and timestamptz_ns [iceberg-python]
Posted by "takayoshi-makabe (via GitHub)" <gi...@apache.org>.
takayoshi-makabe commented on PR #3855:
URL: https://github.com/apache/iceberg-python/pull/3855#issuecomment-5912017270
@ebyhr @kevinjqliu Friendly ping — this is a small fix and CI is green. Would you mind taking a look when you have time? Happy to address any feedback.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Add JSON single-value serialization for timestamp_ns and timestamptz_ns [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3855:
URL: https://github.com/apache/iceberg-python/pull/3855#issuecomment-5841529275
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that's incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org