You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@fluss.apache.org by "litiliu (via GitHub)" <gi...@apache.org> on 2026/10/19 02:46:37 UTC

[I] [spark] Support custom Iceberg lake table paths [fluss]

litiliu opened a new issue, #4712:
URL: https://github.com/apache/fluss/issues/4712

   ### Search before asking
   
   - [x] I searched the issues and found no existing issue specifically tracking Spark support for custom Iceberg lake table paths.
   
   ### Motivation
   
   PR #4253 adds Iceberg support for `table.datalake.database-name` and `table.datalake.table-name`, allowing a Fluss table to map to a physical Iceberg table with a different database and/or table name.
   
   Spark lake reads still reject a mapping whose resolved lake table path differs from the logical Fluss table path. `FlussLakeUtils.createLakeSource` contains this guard:
   
   ```text
   Custom lake table path is not supported for Spark lake reads yet.
   ```
   
   This prevents Spark from reading tiered data through Fluss when the Iceberg table uses a custom path, including union reads that need the lake snapshot.
   
   Issue #3832 tracks the corresponding Paimon feature. This issue tracks the Iceberg work separately.
   
   ### Solution
   
   - Resolve the physical Iceberg database and table name from the Fluss table's lake mapping options when creating the Spark lake source.
   - Support mapped Iceberg table paths in Fluss/Iceberg union reads and lake-side scans.
   - Ensure predicate pushdown uses the correct physical Iceberg table.
   - Cover database-only, table-only, and combined database/table mappings, along with the default same-name behavior.
   - Cover both scans that read a lake snapshot and union reads that combine the lake snapshot with the Fluss log. Preserve the existing Fluss-only fallback when there is no lake snapshot.
   
   The mapping identifies the physical Iceberg table; it should not rename the logical Fluss table or change its table ID and tiering offsets.
   
   ### References
   
   - Iceberg custom-path support: #4253
   - Paimon Spark custom-path support: #3832
   - Review requesting the Spark follow-up reference: https://github.com/apache/fluss/pull/4253#pullrequestreview-5565748129
   - Current guard: `fluss-spark/fluss-spark-common/src/main/scala/org/apache/fluss/spark/read/lake/FlussLakeUtils.scala`
   
   ### Willingness to contribute
   
   - [ ] I'm willing to submit a PR!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [spark] Support custom Iceberg lake table paths [fluss]

Posted by "litiliu (via GitHub) via issues" <is...@fluss.apache.org>.
litiliu commented on issue #4712:
URL: https://github.com/apache/fluss/issues/4712#issuecomment-6735085094

   close via https://github.com/apache/fluss/pull/4253


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [spark] Support custom Iceberg lake table paths [fluss]

Posted by "litiliu (via GitHub) via issues" <is...@fluss.apache.org>.
litiliu closed issue #4712: [spark] Support custom Iceberg lake table paths
URL: https://github.com/apache/fluss/issues/4712


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org