[GitHub] [iceberg] openinx commented on pull request #4228: Flink: The Data Skew Problem on FlinkSink

[GitBox <gi...@apache.org>]

openinx commented on pull request #4228:
URL: https://github.com/apache/iceberg/pull/4228#issuecomment-1054912982


   Thanks @zhengchar for bringing this interesting issue here, and thanks @stevenzwu  and @kbendick for providing the slack context. 
   
   In essence,  the current `PartitionKeySelector` will generate a `String` **key** which is composited by a `bucket` integer value of a narrow range ( usually `[0~bucketNum)` ).  And then in the `keyBy` operator , it will use the **key** to calculate  the `String#hashCode % parallelism` again to choose the preferred downstream channel. 
   
   The value range of the first hash  is becoming narrow, resulting in a large number of row conflicts in the same bucket for the second hash.
   
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org