You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "yshcz (via GitHub)" <gi...@apache.org> on 2025/12/31 09:09:44 UTC
[PR] Spec: Add implementation note for determining manifest list format version [iceberg]
yshcz opened a new pull request, #14949:
URL: https://github.com/apache/iceberg/pull/14949
Closes https://github.com/apache/iceberg/issues/14926.
While implementing the spec, I looked into how other implementations detect the format version of manifest list files and found some notable differences in approaches:
| Implementation | Description |
|----------------|-------------|
| Java | Uses Avro schema resolution [without version-specific read path](https://github.com/apache/iceberg/blob/apache-iceberg-1.10.1/core/src/main/java/org/apache/iceberg/ManifestLists.java#L33-L48) - always uses [the same schema](https://github.com/apache/iceberg/blob/apache-iceberg-1.10.1/api/src/main/java/org/apache/iceberg/ManifestFile.java#L100-L117) that includes all fields up to V3+ |
| Python | Similar to Java - uses [schema resolution](https://github.com/apache/iceberg-python/blob/pyiceberg-0.10.0/pyiceberg/avro/resolver.py#L236-L253) with a [fixed read schema](https://github.com/apache/iceberg-python/blob/pyiceberg-0.10.0/pyiceberg/manifest.py#L897) |
| Rust | Uses current version of table metadata and [branches on it](https://github.com/apache/iceberg-rust/blob/v0.7.0/crates/iceberg/src/spec/manifest_list.rs#L60-L72), though a TODO comment notes this [may not be needed](https://github.com/apache/iceberg-rust/blob/v0.7.0/crates/iceberg/src/spec/snapshot.rs#L195-L196) |
| Go | Reads `format-version` from [Avro metadata](https://github.com/apache/iceberg-go/blob/v0.4.0/manifest.go#L809-L812) and errors if missing, which is not strictly following the spec since this field is not required for manifest lists |
This PR adds an implementation note to Appendix F to document the recommended approach and guide future implementations. It explains how Avro schema resolution enables implementations to use a single read schema without version detection, how to determine the exact format version when needed (by examining the writer schema), and why `format-version` in Avro metadata cannot be reliably used for manifest lists.
If manifest lists are removed in v4, this note would only apply to v1 to v3. I'm not sure how to frame this, any guidance from reviewers would be appreciated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Add implementation note for determining manifest list format version [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #14949: Spec: Add implementation note for determining manifest list format version
URL: https://github.com/apache/iceberg/pull/14949
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Add implementation note for determining manifest list format version [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #14949:
URL: https://github.com/apache/iceberg/pull/14949#issuecomment-3826679059
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spec: Add implementation note for determining manifest list format version [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #14949:
URL: https://github.com/apache/iceberg/pull/14949#issuecomment-3863123125
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org