You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "FredKhayat (via GitHub)" <gi...@apache.org> on 2025/08/04 09:15:16 UTC
[I] Improve timestamp support [iceberg-python]
FredKhayat opened a new issue, #2270:
URL: https://github.com/apache/iceberg-python/issues/2270
### Apache Iceberg version
0.9.1 (latest release)
### Please describe the bug 🐞
Currently, the `pyiceberg` library does not support nanosecond timestamps, or timestamp with timezones.
Now is a great time to support that since the Java Iceberg library is [actively implementing those features](https://github.com/apache/iceberg/pull/13562).
### How to reproduce
```python
import pyarrow as pa
from datetime import datetime
# Create a pyarrow table with a nanosecond timestamp column
table = pa.Table.from_arrays(
[pa.array([datetime.now()], type=pa.timestamp("ns"))], names=["timestamps"]
)
catalog.create_table("ns.table1", schema=table.schema) # Fails since nanosecond timestamps are not supported
```
```python
# Create a pyarrow table with a timestampz column
table = pa.Table.from_arrays(
[pa.array([datetime.now()], type=pa.timestamp("ms", tz='America/New_York'))], names=["timestamps"]
)
catalog.create_table("ns.table2", schema=table.schema) # Fails since timezones are not supported
```
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Improve timestamp support [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko closed issue #2270: Improve timestamp support
URL: https://github.com/apache/iceberg-python/issues/2270
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Improve timestamp support [iceberg-python]
Posted by "rambleraptor (via GitHub)" <gi...@apache.org>.
rambleraptor commented on issue #2270:
URL: https://github.com/apache/iceberg-python/issues/2270#issuecomment-3161026562
I can take this on! I'll have a PR out shortly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Improve timestamp support [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #2270:
URL: https://github.com/apache/iceberg-python/issues/2270#issuecomment-3152157275
Hey @FredKhayat Thanks for raising this. Are you interested in adding these features?
For the timestamp nanoseconds: We need to extend [_ConvertToIceberg](https://github.com/apache/iceberg-python/blob/14ee8da8ff73864b271396853aa2ea5e3e3a04b3/pyiceberg/io/pyarrow.py#L1218-L1220) to also pass in the table version, that can be passed in explicitly through the `format-version` table property, or take the `DEFAULT_FORMAT_VERSION` constant. For V3 tables, we can directly convert into nanoseconds, and we can skip the downscaling to microseconds.
For the timezones, I'm not sure why this is not allowed. While everyone should avoid using timezones at all to keep everyone sane. In the case you want to use it, we can convert from a timestamp that has a timezone, we should return `TimestamptzType` instead:
https://github.com/apache/iceberg-python/blob/14ee8da8ff73864b271396853aa2ea5e3e3a04b3/pyiceberg/types.py#L751-L762
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Improve timestamp support [iceberg-python]
Posted by "bepec (via GitHub)" <gi...@apache.org>.
bepec commented on issue #2270:
URL: https://github.com/apache/iceberg-python/issues/2270#issuecomment-3474658473
Hi @kevinjqliu - it seems all related changes are included in 0.10.0 release, but this snippet still doesn't work?
```
# Create a pyarrow table with a timestampz column
table = pa.Table.from_arrays(
[pa.array([datetime.now()], type=pa.timestamp("ms", tz='America/New_York'))], names=["timestamps"]
)
catalog.create_table("ns.table2", schema=table.schema, properties={"format-version": "3"})
```
Sorry for bothering after issue is resolved, I just ran across with impression that non-UTC timezones should be supported.
But then checked Iceberg specification and it only supports timestamps with tz=UTC - so this example isn't something that should work, is that understanding correct?
https://py.iceberg.apache.org/reference/pyiceberg/types/#pyiceberg.types.TimestamptzType
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Improve timestamp support [iceberg-python]
Posted by "rambleraptor (via GitHub)" <gi...@apache.org>.
rambleraptor commented on issue #2270:
URL: https://github.com/apache/iceberg-python/issues/2270#issuecomment-3169411479
@kevinjqliu great find. just sent over a PR that should fix this!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Improve timestamp support [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #2270:
URL: https://github.com/apache/iceberg-python/issues/2270#issuecomment-3481615151
hey @bepec
> But then checked Iceberg specification and it only supports timestamps with tz=UTC - so this example isn't something that should work, is that understanding correct?
Iceberg indeed stores the value of timestamp with timezone in UTC. This is documented in https://iceberg.apache.org/spec/#primitive-types
The engine is expected to convert the timestamp to UTC before storing in Iceberg
And no, that example should not work. We have an unit test against using timezone other than UTC
https://github.com/apache/iceberg-python/blob/8878b2c260bf62ad16ef6f8e8d3a6ae93103eff7/tests/io/test_pyarrow_visitor.py#L222-L225
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Improve timestamp support [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu closed issue #2270: Improve timestamp support
URL: https://github.com/apache/iceberg-python/issues/2270
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Improve timestamp support [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #2270:
URL: https://github.com/apache/iceberg-python/issues/2270#issuecomment-3169017210
hey @rambleraptor i dont think we've fully addressed this issue yet.
im still able to reproduce the issue on `main`
```
import pyarrow as pa
from datetime import datetime
from pyiceberg.catalog import load_catalog
catalog = load_catalog("default", type="in-memory")
# Create a pyarrow table with a nanosecond timestamp column
table = pa.Table.from_arrays(
[pa.array([datetime.now()], type=pa.timestamp("ns"))], names=["timestamps"]
)
catalog.create_table("ns.table1", schema=table.schema, properties={"format-version": "3"})
# Create a pyarrow table with a timestampz column
table = pa.Table.from_arrays(
[pa.array([datetime.now()], type=pa.timestamp("ms", tz='America/New_York'))], names=["timestamps"]
)
catalog.create_table("ns.table2", schema=table.schema, properties={"format-version": "3"})
```
I believe this is due to a few usage of [`_convert_schema_if_needed` ](https://github.com/search?q=repo%3Aapache%2Ficeberg-python%20_convert_schema_if_needed&type=code) not using the new `format_version` parameter.
This issue above is caused specifically by the one in sql.py
https://github.com/apache/iceberg-python/blob/24b12ddd8fdab4a62650786a2c3cdd56a53f8719/pyiceberg/catalog/sql.py#L203
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org