You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "manuzhang (via GitHub)" <gi...@apache.org> on 2026/10/13 09:05:52 UTC
[I] Core: RewriteManifests.addManifest mishandles v1-format manifests on v2+ tables [iceberg]
manuzhang opened a new issue, #18482:
URL: https://github.com/apache/iceberg/issues/18482
### Apache Iceberg version
main (development). Reproduced at db43b88a6c; the same code paths are in 1.11.0.
### Query engine
None (core API).
### Please describe the bug 🐞
`RewriteManifests.addManifest` does not check the format version of the manifest it is given. On a v2+ table, a manifest written in v1 format (no `sequence_number` / `file_sequence_number` columns) is mishandled on both paths.
**Reused manifest** (`snapshotId()` is null, snapshot ID inheritance enabled): the commit succeeds and the v1 manifest is added to the manifest list as-is, with the new commit's sequence number. On read, `BaseInheritableMetadata` only fills in sequence numbers for `ADDED` entries or when the manifest's sequence number is 0, so the `EXISTING` entries come back with null data and file sequence numbers. Scan planning then throws a `NullPointerException`, because `DeleteFileIndex.forEntry` passes `entry.dataSequenceNumber()` to `forDataFile(long, DataFile)`.
**Copied manifest** (`snapshotId()` is -1): `ManifestFiles.copyRewriteManifest` reads the entries with empty inheritable metadata, so the sequence numbers stay null and the v2+ writer fails inside `addManifest` with:
```
IllegalStateException: Found unassigned sequence number for an entry from snapshot: <id>
```
The spec says that "when reading v1 manifests with no sequence number column, sequence numbers for all files must default to 0", and that existing entries written to a new manifest must carry non-null sequence numbers.
#### Reproduction
In `TestRewriteManifests` with format version 2 or later:
```java
table.newFastAppend().appendFile(FILE_A).appendFile(FILE_B).commit();
Snapshot first = table.currentSnapshot();
ManifestFile original = Iterables.getOnlyElement(first.allManifests(table.io()));
OutputFile out = table.ops().io().newOutputFile(temp.resolve("v1-manifest.avro").toString());
ManifestWriter<DataFile> writer = ManifestFiles.write(1, table.spec(), out, null);
try {
writer.existing(FILE_A, first.snapshotId(), first.sequenceNumber(), first.sequenceNumber());
writer.existing(FILE_B, first.snapshotId(), first.sequenceNumber(), first.sequenceNumber());
} finally {
writer.close();
}
table.rewriteManifests().deleteManifest(original).addManifest(writer.toManifestFile()).commit();
// the committed entries read back with null data and file sequence numbers
Lists.newArrayList(table.newScan().planFiles()); // NullPointerException
```
Writing the manifest with snapshot ID `-1L` instead of `null` takes the copy path, and `addManifest` throws the `IllegalStateException` above. Both behaviors reproduce on format versions 2, 3 and 4.
#### Expected behavior
`addManifest` should handle v1-format manifests on v2+ tables consistently: either reject them with a clear error, or copy them and write the spec default of 0 as explicit sequence numbers. It should not commit a manifest that breaks scan planning.
This came up while reviewing the same operation in iceberg-cpp (apache/iceberg-cpp#774), which copies v1-format manifests and writes sequence number 0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Core: RewriteManifests.addManifest mishandles v1-format manifests on v2+ tables [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on issue #18482:
URL: https://github.com/apache/iceberg/issues/18482#issuecomment-6142541156
@adityaksolves Feel free to open a PR with your own proposal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Core: RewriteManifests.addManifest mishandles v1-format manifests on v2+ tables [iceberg]
Posted by "adityaksolves (via GitHub)" <gi...@apache.org>.
adityaksolves commented on issue #18482:
URL: https://github.com/apache/iceberg/issues/18482#issuecomment-6142444570
Hi @manuzhang, I would like to take this up. Could you or someone please assign this issue to me?
Regarding the expected behavior, which approach is preferred for the Java codebase: rejecting v1-format manifests on v2+ tables with a clear error, or copying them and writing the spec default of 0 (similar to iceberg-cpp)? I can start implementing the preferred approach.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Core: RewriteManifests.addManifest mishandles v1-format manifests on v2+ tables [iceberg]
Posted by "adityaksolves (via GitHub)" <gi...@apache.org>.
adityaksolves commented on issue #18482:
URL: https://github.com/apache/iceberg/issues/18482#issuecomment-6150617320
Thanks @manuzhang. I've reproduced both behaviors locally on formats v2, v3 and v4, and I have a fix ready. Will raise a PR shortly.
My proposal is to reject v1-format manifests in `RewriteManifests.addManifest` on v2+ tables with a clear error (`Cannot add v1 manifest to a v2 table: <path>`), rather than copying them with a default sequence number of 0. Reasons:
- It fails fast at the API call instead of committing a manifest that breaks scan planning later.
- A single check at the top of `addManifest` covers both the reused-manifest and copied-manifest paths.
- Silently rewriting a v1 manifest into v2+ is a larger behavior change; callers who really want that can rewrite the manifest with a v2+ writer first.
The check reads the `format-version` key from the Avro manifest header (absent means v1). Non-Avro manifests are skipped since v1 manifests are always Avro.
I'm adding a test in `TestRewriteManifests` that covers the reproduction from the issue. Happy to switch to the "copy with sequence number 0" approach if you or other reviewers prefer that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org