[GitHub] [iceberg] kbendick commented on pull request #4228: Flink: The Data Skew Problem on FlinkSink

[GitBox <gi...@apache.org>]

kbendick commented on pull request #4228:
URL: https://github.com/apache/iceberg/pull/4228#issuecomment-1053942934


   > There is a recent Slack thread for the same issue where hash distribution leads to skewed shuffling (for bucketing partitions and probably other partition spec too): https://apache-iceberg.slack.com/archives/C025PH0G1D4/p1645676203340179
   > 
   > I don't necessarily agree with the solution provided in this PR. We can provide some general solution in FlinkSink.
   > 
   > For bucketing partitions, we can implement a custom partitioner to shuffle data by the bucketing value to the downstream tasks/channels: `bucket(key) % downstream operator parallelism`.
   > 
   > I don't know if it makes sense to add `bucketing` enum value to `DistributionMode` or we can make this config only for FlinkSink.
   > 
   > cc @openinx @rdblue @kbendick
   
   I agree @stevenzwu, that I don't necessarily think this is the appropriate solution to the concern. Please do join the Iceberg Slack @zhengchar - as Steven mentioned this has come up very recently there.
   
   I wouldn't add something so specific like `bucketing` to `DistributionMode` (at least as proposed here), I would probably start by making a config to the `FlinkSink` and then trying to solve the problem somewhat more generally with respect to partitioning and required distribution - possibly making that opt-in / opt-out via the flink sink config in case it's not an issue for users. But having a `DistributionMode` of `bucketing` seems a little too situationally specific given the preconditions outlined here (only bucketing partition is used, etc). I agree with the idea of a custom partitioner to preshuffle the data by some (potentially configurable) factor of downstream parallelism (be that tasks or channels).
   
   I'd love to continue the discussion on Slack (or on the dev list, but Slack is great for casual async discussion and the Iceberg Slack is very active).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org