You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@fluss.apache.org by "polyzos (via GitHub)" <gi...@apache.org> on 2026/10/05 07:37:11 UTC
[I] [Iceberg] Bump Iceberg Version to 1.12 [fluss]
polyzos opened a new issue, #4547:
URL: https://github.com/apache/fluss/issues/4547
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
### Description
Iceberg 1.12 was just released I'm Fluss can upgrade to it.
Upgrading to the latest iceberg, will allow **Fluss users to use Iceberg with Flink 2.2 and Flink 2.3** because the current Iceberg 1.10 only supports up to Flink 2.1, which Fluss doesn't.
This upgrade should also set the stage for Fluss DVs with Iceberg.
https://cwiki.apache.org/confluence/spaces/FLUSS/pages/406621432/WIP+FIP-31+Introduce+Deletion+Vectors+to+accelerate+Primary+Key+Table+Union+Read+over+Iceberg
However **Iceberg 1.12 requires Java 17.**
The upgrade should preserve Fluss's existing Java requirements outside the new Iceberg plugin. Every JVM loading that plugin, including Fluss servers and Flink jobs, must use Java 17+.
### Scope
- Update the dependency and replace the removed `GenericAppenderFactory` API.
- Use a JDK 17 toolchain to build and test the new plugin while keeping the main build and release process on JDK 11.
- Provide a migration path for union-read savepoints. The current state embeds Java-serialized Iceberg objects that are incompatible across these versions; the proposed replacement is a versioned JSON format.
- Close catalogs after use, verify plugin classloading, align dependencies, and update packaging and license notices.
- Document supported Java/Flink versions, matching Iceberg jars, and upgrade steps.
### Decisions Needed
Agree on these in the community discussion before finalizing the implementation:
1. Keep a Java 11-compatible Iceberg 1.10 plugin during a transition, or replace it immediately? If retained, choose the default plugin and support window.
2. Require savepoints for plugin switches, with a migration release before switching existing state to 1.12?
3. Support REST server-side scan planning in this upgrade, or reject it with a clear error until supported?
### Acceptance Criteria
- The new plugin builds and runs on Java 17 without raising Java requirements for other Fluss users.
- Unit and integration tests pass for the documented engine versions and packaged plugin.
- Savepoint migration is tested with partially consumed data; rollback is tested if both plugins are retained.
- REST catalog and S3 integration tests pass, and catalogs close without leaking resources.
- Release packaging, notices, and compatibility documentation are updated.
Iceberg v3 features and deletion vectors are separate follow-up work.
### Willingness to contribute
- [x] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Iceberg] Bump Iceberg Version to 1.12 [fluss]
Posted by "MehulBatra (via GitHub)" <gi...@apache.org>.
MehulBatra commented on issue #4547:
URL: https://github.com/apache/fluss/issues/4547#issuecomment-6121496964
Thanks for raising this. I agree that the Flink 2.2/2.3 upgrade is worth pursuing.
One clarification on scope, since FIP-31 is referenced here. The Iceberg DV work does not depend on this upgrade, and I think we can decouple the two so neither blocks the other.
Iceberg 1.10.1 already provides the complete format v3 deletion vector toolkit, including `BaseDVFileWriter`, `DVFileWriter`, `PuffinWriter`, `RoaringPositionBitmap`, `BitmapPositionDeleteIndex`, `PositionDeleteIndex`, and `DeleteGranularity`.
The FIP-31 prototype already compiles and runs against 1.10.1. Tiering produces append-only format v3 output with zero equality delete files, and primary-key union reads over tiered data return exactly-once results for inserts, updates, and deletes. No APIs from 1.11 or 1.12 are required, and `RoaringBitmap` is already a declared dependency. The prototype is still being hardened, but there is no version-related blocker for the DV work.
This also aligns with the issue's acceptance criteria, which state that Iceberg v3 features and deletion vectors are separate follow-up work. I read the "sets the stage for Fluss DVs" statement as motivation rather than a prerequisite. We can make this explicit in the FIP to avoid ambiguity.
A couple of observations worth considering:
1. `iceberg-flink-${flink.major.version}` is currently test-scoped in `fluss-lake-iceberg` on main. The runtime tiering path depends only on `iceberg-api`, `iceberg-common`, `iceberg-core`, `iceberg-data`, `iceberg-parquet`, and `iceberg-orc`, none of which are Flink-coupled. The Flink 2.2/2.3 constraint may therefore be a test-matrix limitation rather than a runtime requirement. It would be good to confirm this, as it could change the cost-benefit trade-off.
2. `GenericAppenderFactory` is used in only one place on main, `tiering/writer/TaskWriterFactory.java`, so that part of the migration should be relatively small.
On the proposed decisions, my preference would be:
- Keep the Java 11-compatible 1.10 plugin during the transition rather than replacing it immediately. Requiring Java 17 in every JVM that loads the plugin, including Fluss servers and Flink jobs, is a significant operational change. A dual-plugin transition would give users time to adopt at their own pace.
- Keep the 1.10 plugin as the default until the Java 17 migration is settled.
- Move to versioned JSON for Java-serialized Iceberg objects in union-read state. This is a good improvement on its own merits, independent of the version upgrade.
Happy to help validate the DV path against a 1.12 plugin once one is available. I would just prefer not to make FIP-31 wait on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Iceberg] Bump Iceberg Version to 1.12 [fluss]
Posted by "MehulBatra (via GitHub)" <gi...@apache.org>.
MehulBatra commented on issue #4547:
URL: https://github.com/apache/fluss/issues/4547#issuecomment-6121484429
Thanks for raising this. I agree that the Flink 2.2/2.3 upgrade is worth pursuing.
One clarification on scope regarding FIP-31: I think we can decouple the Iceberg DV work from this upgrade so that neither blocks the other.
Iceberg 1.10.1 already provides the APIs needed for format v3 deletion vectors. The FIP-31 prototype is already running against 1.10.1, with format v3 output for tiered append-only tables and correct results for inserts, updates, and deletes in primary-key union reads. The prototype is still being hardened, but there is no version-related blocker today.
This also aligns with the issue's acceptance criteria, which treat Iceberg v3 features and deletion vectors as separate follow-up work. We can clarify this in the FIP to avoid any confusion.
A couple of observations worth considering:
1. `iceberg-flink-${flink.major.version}` is test-scoped in `fluss-lake-iceberg` on main. The runtime tiering path does not appear to depend on Flink-specific APIs, so the Flink 2.2/2.3 constraint may be primarily a test compatibility issue. It would be good to confirm this before deciding on the urgency of the upgrade.
2. `GenericAppenderFactory` is used in only one place on main, `TaskWriterFactory.java`, so that part of the migration should be relatively small.
My preference on the proposed decisions is:
- Keep the Java 11-compatible 1.10 plugin during the transition. Requiring Java 17 for every JVM loading the plugin, including Fluss servers and Flink jobs, is a significant operational change. A dual-plugin transition would give users more flexibility.
- Keep the 1.10 plugin as the default until the Java 17 migration is settled.
- Move to versioned JSON for Iceberg objects in union-read state. This is a worthwhile improvement independent of the Iceberg version upgrade.
Happy to help validate the DV implementation against a 1.12 plugin when one is available. I would just prefer to keep FIP-31 moving independently rather than make it wait for this upgrade.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org