You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "yshcz (via GitHub)" <gi...@apache.org> on 2025/12/24 04:32:14 UTC
[I] Spec: Add Avro metadata requirements for manifest list files [iceberg]
yshcz opened a new issue, #14926:
URL: https://github.com/apache/iceberg/issues/14926
### Feature Request / Improvement
The current spec defines Avro file metadata requirements for manifest files in a clear table:
| v1 | v2 | Key | Value |
|------------|------------|---------------------|------------------------------------------------------------------------------|
| _required_ | _required_ | `schema` | JSON representation of the table schema at the time the manifest was written |
| _optional_ | _required_ | `schema-id` | ID of the schema used to write the manifest as a string |
| _required_ | _required_ | `partition-spec` | JSON representation of the partition spec used to write the manifest |
| _optional_ | _required_ | `partition-spec-id` | ID of the partition spec used to write the manifest as a string |
| _optional_ | _required_ | `format-version` | Table format version number of the manifest as a string |
| | _required_ | `content` | Type of content files tracked by the manifest: "data" or "deletes" |
But manifest **list** files have no equivalent specification for their Avro metadata, despite the Java implementation writing metadata such as `format-version`, `snapshot-id`, `parent-snapshot-id`, and `sequence-number` to manifest list files since 2020.
For manifests: #913 added `format-version` to code (2020-04), and #1499 added the spec (2020-10).
For manifest lists: #907 added `format-version` to code (2020-04), but there are no corresponding spec changes.
As a result, implementations have no standard way to detect the format version from a manifest list file alone. They are forced to either infer the version based on the presence of certain fields, or simply trust the table metadata version. The latter is unreliable in upgrade scenarios where a v2 table may contain v1 snapshots, introducing unnecessary complexity.
The following table might be a reasonable addition, though I'm not entirely certain about the requirements:
| v1 | v2 | v3 | Key | Value |
|------------|------------|------------|-----------------------|------------------------------------------------------|
| _required_ | _required_ | _required_ | `snapshot-id` | The snapshot ID for this manifest list as a string |
| _required_ | _required_ | _required_ | `parent-snapshot-id` | The parent snapshot ID as a string |
| | _required_ | _required_ | `sequence-number` | The sequence number of the snapshot as a string |
| | | _required_ | `first-row-id` | The first row ID for row lineage as a string |
| _optional_ | _required_ | _required_ | `format-version` | Table format version number as a string |
### Query engine
None
### Willingness to contribute
- [ ] I can contribute this improvement/feature independently
- [x] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Spec: Add implementation note for determining manifest list format version [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #14926:
URL: https://github.com/apache/iceberg/issues/14926#issuecomment-4975453487
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Spec: Add implementation note for determining manifest list format version [iceberg]
Posted by "yshcz (via GitHub)" <gi...@apache.org>.
yshcz commented on issue #14926:
URL: https://github.com/apache/iceberg/issues/14926#issuecomment-3701800500
Thanks for the feedback. I've updated the issue title and opened a PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Spec: Add Avro metadata requirements for manifest list files [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on issue #14926:
URL: https://github.com/apache/iceberg/issues/14926#issuecomment-3690364309
We can't add in a requirement for avro metadata for any released version since that would be breaking for all existing manifest lists. We could have it be optional but I'm not sure thats really worth it.
I'm interested in why checking the schema isn't sufficient for the version check? I know it's a bit more complicated than reading a single value but is there a reason why it's not working in your use case? We consider adding this a as a requirement in v4 but we are also considering just dropping manifest_lists entiresly in V4 so it probably doesn't matter
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Spec: Add Avro metadata requirements for manifest list files [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on issue #14926:
URL: https://github.com/apache/iceberg/issues/14926#issuecomment-3697154268
We could definitely add in an implementation note on how to determine format version, I don't think there is any problem with noting things we can already do that we want to codify as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Spec: Add Avro metadata requirements for manifest list files [iceberg]
Posted by "yshcz (via GitHub)" <gi...@apache.org>.
yshcz commented on issue #14926:
URL: https://github.com/apache/iceberg/issues/14926#issuecomment-3690537366
I was just exploring all the possible edge cases while implementing Iceberg and noticed this part of the spec felt inconsistent. Maybe we should at least specify the concrete format version detection mechanism for manifest list somewhere?
I'm fine with the avro schema based detection, but it's not documented anywhere, making it an implicit workaround rather than a standard method. The spec even includes a concrete algorithm for snapshot retention policy. Having the same level of specification for manifest lists would improve overall consistency. Even if v4 drops it, older tables will benefit from a clearer spec
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Spec: Add implementation note for determining manifest list format version [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #14926: Spec: Add implementation note for determining manifest list format version
URL: https://github.com/apache/iceberg/issues/14926
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org