You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "hcrosse (via GitHub)" <gi...@apache.org> on 2026/04/13 21:42:07 UTC
[I] ManifestFile cannot read manifest lists written by Iceberg Java before 1.4 [iceberg-go]
hcrosse opened a new issue, #889:
URL: https://github.com/apache/iceberg-go/issues/889
### Problem
The Iceberg Java implementation used \`added_data_files_count\`,
\`existing_data_files_count\`, and \`deleted_data_files_count\` as field names for
manifest list entries (IDs 504/505/506) from its initial release until
apache/iceberg#5338 fixed the names to match the spec in the 1.4 release
(merged 2023-10-02). Any table written before that fix, or by an engine still
on an older Java version (Athena, some Trino deployments, etc.), has the
legacy names embedded in the writer schema of every manifest list OCF file.
### Symptom
When iceberg-go reads such a manifest list, the three count fields decode to
zero regardless of their actual values. Downstream this causes \`HasAddedFiles()\`
and \`HasExistingFiles()\` to return \`false\`, so fast-append operations silently
drop all inherited manifests and \`Table.Scan()\` returns no data.
### Root cause
hamba/avro's OCF decoder resolves writer-schema field names against struct \`avro:\`
tags. When the writer schema says \`added_data_files_count\` and the struct tag says
\`avro:"added_files_count"\`, there is no match and the struct field stays zero.
### Fix
Add parallel struct fields with the legacy tags and coalesce in the accessors:
\`\`\`go
AddedFilesCount int32 \`avro:"added_files_count"\`
AddedDataFilesCount int32 \`avro:"added_data_files_count"\` // pre-1.4 Java name
\`\`\`
\`AddedDataFiles()\` returns \`AddedFilesCount\` when non-zero, falling back to
\`AddedDataFilesCount\`. Same pattern for existing/deleted. Read-path only,
no write-path or schema-registration changes needed.
### Relation to existing issue
apache/iceberg-go#869 fixes the downstream data-loss (manifest inheritance filter
in fast-append). This issue fixes the upstream root cause: zero counts when
reading pre-1.4 manifest lists in the first place.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] ManifestFile cannot read manifest lists written by Iceberg Java before 1.4 [iceberg-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade closed issue #889: ManifestFile cannot read manifest lists written by Iceberg Java before 1.4
URL: https://github.com/apache/iceberg-go/issues/889
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] ManifestFile cannot read manifest lists written by Iceberg Java before 1.4 [iceberg-go]
Posted by "tanmayrauth (via GitHub)" <gi...@apache.org>.
tanmayrauth commented on issue #889:
URL: https://github.com/apache/iceberg-go/issues/889#issuecomment-4270104630
@hcrosse the fix is now merged to main.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] ManifestFile cannot read manifest lists written by Iceberg Java before 1.4 [iceberg-go]
Posted by "tanmayrauth (via GitHub)" <gi...@apache.org>.
tanmayrauth commented on issue #889:
URL: https://github.com/apache/iceberg-go/issues/889#issuecomment-4240134179
Created [PR](https://github.com/apache/iceberg-go/pull/890) for this issue. @hcrosse @laskoviymishka can you please review it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org