[PR] [lake] Add Iceberg 1.12 plugin [fluss]

["dstolis (via GitHub)" <gi...@apache.org>]

dstolis opened a new pull request, #4671:
URL: https://github.com/apache/fluss/pull/4671

   ### Purpose
   
   Linked issue: close #4592
   
   Implements the opt-in Iceberg 1.12 lake plugin (`fluss-lake-iceberg-1.12`) on top of the module split from #4646 and the v2 state format from #4660.
   
   ### Brief change log
   
   - **Writers:** Iceberg 1.12 removed `GenericAppenderFactory`, and in 1.10 `GenericFileWriterFactory` is package-private, so the writer code can't be shared. The 1.12 module has its own `TaskWriterFactory`, `GenericRecordAppendOnlyWriter` and `GenericRecordDeltaWriter` using `GenericFileWriterFactory`. They replace the classes of the same name in `fluss-lake-iceberg-common`, which the shaded jar drops.
   - **`FormatModelRegistry` classloader:** the registry registers its format models through the thread context classloader. The 1.12 `IcebergLakeStoragePlugin` initializes it with the plugin classloader.
   - **Java < 17:** the same plugin entry calls `Iceberg112RuntimeCheck` before loading any Iceberg class.
   - **Old split state:** the 1.12 `IcebergSplitSerializer` rejects v1 (Java-serialized) splits with migration steps. v2 is read as before.
   - **Format v3 for primary-key tables:** `IcebergLakeCatalog` rejects `format-version` ≥ 3 for primary-key tables on create and alter, because they write position delete files, which v3 replaces with deletion vectors. This is in the shared module because Iceberg 1.10 rejects these commits too.
   - **Cleanup failures:** `IcebergLakeCatalog.close()` and `IcebergLakeTieringFactory.validateTable()` now throw catalog close failures instead of only logging them.
   - **Relocations and NOTICE:** the shaded jar uses the 13 relocations of `iceberg-flink-runtime` 1.12.0. The module has its own NOTICE listing the 40 bundled artifacts.
   - **Release:** the module is now deployed, and the binary distribution ships it in `opt/iceberg-1.12/`. Both plugins register the `iceberg` format, so only one can be in `plugins/`. The Iceberg docs explain how to switch.
   
   ### Tests
   
   - `fluss-lake-iceberg-1.12`, on JDK 17:
     - `TaskWriterFactoryTest`: log and primary-key tables.
     - `IcebergLakeStoragePluginTest`: Java version check, and format models registered with the plugin classloader, using an isolated classloader with a control case.
     - `IcebergSplitSerializerTest`: v1 rejected.
   - `IcebergLakeCatalogTest`: format v3 rejected for primary-key tables on create and alter, still allowed for log tables, and catalog close failure surfaced.
   - `fluss-lake-iceberg-common` unit and IT cases pass. `fluss-dist` builds with the expected `opt/` and `plugins/` layout.
   
   ### API and Format
   
   No API or storage format change. Creating or altering a primary-key table with Iceberg `format-version` 3 or higher now fails.
   
   ### Documentation
   
   Adds a section to the Iceberg docs on using the Iceberg 1.12 plugin.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [lake] Add Iceberg 1.12 plugin [fluss]

["dstolis (via GitHub)" <gi...@apache.org>]

dstolis commented on PR #4671:
URL: https://github.com/apache/fluss/pull/4671#issuecomment-6214874438

   @polyzos For “surface cleanup failures”, `IcebergLakeCatalog.close()` and `validateTable()` now throw when closing the catalog fails, instead of only logging a warning. Is that enough, or did you mean something else?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [lake] Add Iceberg 1.12 plugin [fluss]

["polyzos (via GitHub)" <gi...@apache.org>]

polyzos merged PR #4671:
URL: https://github.com/apache/fluss/pull/4671


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org