You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@fluss.apache.org by "dstolis (via GitHub)" <gi...@apache.org> on 2026/10/15 15:43:13 UTC
[PR] [lake] Add versioned v2 format for Iceberg split and tiering state [fluss]
dstolis opened a new pull request, #4660:
URL: https://github.com/apache/fluss/pull/4660
### Purpose
Linked issue: close #4591
The Iceberg split, write result and committable are Java-serialized today, so they can only be read by the Iceberg version that wrote them. This adds a Fluss-owned v2 format that can be read across Iceberg versions, as the basis for savepoint migration between Iceberg plugins.
### Brief change log
- `IcebergSplitSerializer`, `IcebergWriteResultSerializer` and `IcebergCommittableSerializer` now write v2 and still read v1.
- v2 is a JSON document with Fluss-defined fields, written by `IcebergSplitJsonSerde`, `IcebergWriteResultJsonSerde` and `IcebergCommittableJsonSerde`. Iceberg objects (scan task, data/delete files, schema, partition spec) are nested as the JSON of Iceberg's own parsers, which the REST catalog also uses and which is stable across Iceberg versions.
- `IcebergWriteResult` and `IcebergCommittable` now carry the table's partition specs, which are needed to parse the files back. The writer and committer fill them from the table.
- The format is documented in the Javadoc of the serdes and `IcebergJsonSerdeUtils`.
### Tests
- v1 and v2 golden files for the split, write result and committable. The committable v1 file was generated with the class as it was before this change.
- Each serializer test checks that all versions deserialize, and that the current output matches its golden file byte for byte.
- `IcebergSplitSerializerTest`, `IcebergWriteResultSerializerTest`, `IcebergCommittableSerializerTest`
- `fluss-lake-iceberg-common` unit and IT cases pass, including the union read and tiering ITs.
### API and Format
Changes the state format of the Iceberg split, write result and committable (v1 → v2). v1 state is still readable. A write result or committable restored from v1 has no partition specs and cannot be serialized again. They are not kept in checkpoint state, so this only affects in-flight records.
Golden files are excluded from the RAT check in the root `pom.xml`, since they cannot carry a license header.
### Documentation
No.
Generated-by: Claude Code (Claude Opus 5.5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [lake] Add versioned v2 format for Iceberg split and tiering state [fluss]
Posted by "polyzos (via GitHub)" <gi...@apache.org>.
polyzos commented on PR #4660:
URL: https://github.com/apache/fluss/pull/4660#issuecomment-6208743062
LGTM 👍 Just one note so its clear:
```
I tested it, and that only holds in one direction. A savepoint written with Iceberg 1.10 restores fine on 1.12. A savepoint written with 1.12 fails to restore on 1.10, because Iceberg 1.12 changed how it writes data files to JSON
```
Let's add this to the ticket description to remember to document this later, so it's clear to the users, just in case - although - although i expect this to be redundant since users will only look to upgrade
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [lake] Add versioned v2 format for Iceberg split and tiering state [fluss]
Posted by "polyzos (via GitHub)" <gi...@apache.org>.
polyzos merged PR #4660:
URL: https://github.com/apache/fluss/pull/4660
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org