[I] Iceberg tables with multiple bucket keys can be created but cannot be written [fluss]

["yangshangqing95 (via GitHub)" <gi...@apache.org>]

yangshangqing95 opened a new issue, #4794:
URL: https://github.com/apache/fluss/issues/4794

   ### Search before asking
   
   - [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
   
   
   ### Motivation
   
   Fluss allows creating tables configured with the Iceberg lake format and multiple bucket keys when lake storage is disabled. However, writing to these tables fails because `IcebergKeyEncoder` only supports a single bucket key.
   
   To reproduce, create a table using Flink SQL:
   
   ```sql
   CREATE TABLE iceberg_multi_bucket (
       a INT,
       b INT,
       payload STRING
   ) WITH (
       'bucket.num' = '4',
       'bucket.key' = 'a,b',
       'table.datalake.format' = 'iceberg',
       'table.datalake.enabled' = 'false'
   );
   ```
   
   Then insert a record:
   
   ```sql
   INSERT INTO iceberg_multi_bucket VALUES (1, 2, 'test');
   ```
   
   **Actual behavior:** Table creation succeeds, but writing fails because `IcebergKeyEncoder` requires exactly one bucket key.
   
   ```
   [ERROR] Could not execute SQL statement. Reason:
   java.lang.IllegalArgumentException: Key fields must have exactly one field for iceberg format, but got: [a, b]
   ```
   
   **Expected behavior:** Fluss should reject unsupported Iceberg bucket key configurations during table creation, even when lake storage is disabled.
   
   ### Solution
   
   Validate the effective bucket keys during table creation and reject Iceberg formatted tables with more than one bucket key.
   
   ### Anything else?
   
   _No response_
   
   ### Willingness to contribute
   
   - [x] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Iceberg tables with multiple bucket keys can be created but cannot be written [fluss]

["polyzos (via GitHub)" <gi...@apache.org>]

polyzos commented on issue #4794:
URL: https://github.com/apache/fluss/issues/4794#issuecomment-6333938756

   @yangshangqing95 for my own curiosity, there is no way to write to iceberg, a table with multiple bucket_keys? you only need to have one key? i.e there is no workaround to this?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Iceberg tables with multiple bucket keys can be created but cannot be written [fluss]

["yangshangqing95 (via GitHub)" <gi...@apache.org>]

yangshangqing95 commented on issue #4794:
URL: https://github.com/apache/fluss/issues/4794#issuecomment-6344349722

   Hi @polyzos, Iceberg itself is not limited to a single bucket partition field. The restriction comes from the current Fluss–Iceberg integration, and it's mainly a design choice to keep bucket assignments consistent between Fluss and Iceberg.
   
   The key idea is that **we want each Fluss bucket to map directly to the corresponding Iceberg bucket**.
   
   Two things:
   
   1. **Enabling lake tiering without redistributing existing data.** Fluss uses Iceberg's bucketing algorithm from the beginning, even when lake storage is disabled. This allows us to enable Iceberg tiering later while preserving the existing bucket assignments.
   
   2. **Efficient bucket-level reads across Fluss and Iceberg.** For example, when reading historical data for Fluss bucket 1, we can locate the corresponding Iceberg bucket directly instead of scanning all Iceberg partitions. This is particularly useful for union reads and historical subscriptions.
   
   The limitation comes from how the bucket is calculated.
   
   Fluss supports combining multiple fields into a single composite bucket key, producing one bucket ID out of N buckets. However, Iceberg's currently supported bucket transform operates on a single field.
   
   Although Iceberg allows multiple bucket transforms, `bucket(4, a)` and `bucket(4, b)` would produce two independent partition values, potentially resulting in 16 combinations, rather than one of the 4 Fluss buckets. Therefore, they cannot directly represent Fluss's composite bucketing.
   
   In the current code, `IcebergKeyEncoder` only supports one bucket key, and `IcebergPartitionSpecUtils` creates a single Iceberg bucket transform. This ensures consistent bucket assignments but means **we cannot use multiple fields together for Iceberg-compatible bucketing.**
   
   For true multi column bucketing, we would need a compatible multi-argument bucket transform supported by Iceberg, together with corresponding changes to Fluss's encoding and partition mapping. Another possible approach would be to explicitly materialize the composite bucket ID in Iceberg, but that would require a different partitioning design and additional compatibility considerations.
   
   So this PR doesn't introduce a new restriction, it simply makes Fluss reject configurations that the current implementation cannot handle, instead of allowing table creation and failing when writing data.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org