You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "Samreay (via GitHub)" <gi...@apache.org> on 2024/12/11 04:04:37 UTC
[I] PyIceberg appending data creates snapshots incompatible with Athena/Spark [iceberg-python]
Samreay opened a new issue, #1424:
URL: https://github.com/apache/iceberg-python/issues/1424
### Apache Iceberg version
0.8.0
### Please describe the bug 🐞
We append data to our iceberg table using the `Table.overwrite` function, and this is saving out snapshots which have IDs that cannot be parsed by athena's `OPTIMIZE` command, or Sparks:
```
java.lang.IllegalArgumentException: Cannot parse to a long value: snapshot-id: 9223372036854775808
at org.apache.iceberg.relocated.com.google.common.base.Preconditions.checkArgument(Preconditions.java:446)
at org.apache.iceberg.util.JsonUtil.getLong(JsonUtil.java:139)
at org.apache.iceberg.SnapshotParser.fromJson(SnapshotParser.java:116)
at org.apache.iceberg.TableMetadataParser.fromJson(TableMetadataParser.java:478)
```
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [X] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] PyIceberg appending data creates snapshots incompatible with Athena/Spark [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #1424:
URL: https://github.com/apache/iceberg-python/issues/1424#issuecomment-2545885155
The above replicates the logic of `_generate_snapshot_id`
https://github.com/apache/iceberg-python/blob/b981780d313f7fa6fb911381962fe00017073cfe/pyiceberg/table/metadata.py#L322-L333
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] PyIceberg appending data creates snapshots incompatible with Athena/Spark [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu closed issue #1424: PyIceberg appending data creates snapshots incompatible with Athena/Spark
URL: https://github.com/apache/iceberg-python/issues/1424
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] PyIceberg appending data creates snapshots incompatible with Athena/Spark [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #1424:
URL: https://github.com/apache/iceberg-python/issues/1424#issuecomment-2549099056
I think the snapshot id is generated on the client side. So its possible only if glue is also committing the table.
If you can share the metadata json, that would be helpful!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] PyIceberg appending data creates snapshots incompatible with Athena/Spark [iceberg-python]
Posted by "Samreay (via GitHub)" <gi...@apache.org>.
Samreay commented on issue #1424:
URL: https://github.com/apache/iceberg-python/issues/1424#issuecomment-2547264185
I'll see if I can track down the snapshot metadata. I'm also not sure how this would happen, but we've been exclusively using pyiceberg to create, remove, and append data to our iceberg tables. Granted, the creation is using the glue catalog, so I suppose there's potential for Amazon to be muddying the waters here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] PyIceberg appending data creates snapshots incompatible with Athena/Spark [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on issue #1424:
URL: https://github.com/apache/iceberg-python/issues/1424#issuecomment-2544900271
I'm also not seeing how this could happen, to test this, I also ran this script:
```
Python 3.10.14 (main, Mar 19 2024, 21:46:16) [Clang 15.0.0 (clang-1500.3.9.4)]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.26.0 -- An enhanced Interactive Python. Type '?' for help.
In [1]: import uuid
...:
...: while True:
...: rnd_uuid = uuid.uuid4()
...: snapshot_id = int.from_bytes(
...: bytes(lhs ^ rhs for lhs, rhs in zip(rnd_uuid.bytes[0:8], rnd_uuid.bytes[8:16])), byteorder="little", signed=True
...: )
...: snapshot_id = snapshot_id if snapshot_id >= 0 else snapshot_id * -1
...: if snapshot_id > 9223372036854775807:
...: print("Boom")
...:
```
I think we should add some checks to ensure that the `snapshot_id` is in the range of `[0, 9223372036854775807]` or raise a `ValueError` instead. Would you be interested in contributing that @Samreay ? I think this should happen in the `SnapshotProducer`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] PyIceberg appending data creates snapshots incompatible with Athena/Spark [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #1424:
URL: https://github.com/apache/iceberg-python/issues/1424#issuecomment-2541805668
hi @Samreay thanks for reporting this issue! Very odd that its 1+MAX_VALUE.
I took a look at the write path and didn't see anything that stood put that would cause this issue.
Could you post the metadata json file associated with this snapshot-id so we can debug further?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org