[I] [Bug] spark fixed bucket write causes task data process unevenly for pk table [paimon]

["askwang (via GitHub)" <gi...@apache.org>]

askwang opened a new issue, #3651:
URL: https://github.com/apache/paimon/issues/3651

   ### Search before asking
   
   - [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
   
   
   ### Paimon version
   
   0.8
   
   ### Compute Engine
   
   spark
   
   ### Minimal reproduce step
   
   When we use fixed bucket to write data for pk table, we set 1000 buckets, and want to process data with 1000 partitions evenly. But we found some tasks process no data and some tasks process many data. This will cause the overall running time of the task to be longer.
   We found the root cause is `repartitionByExpression` will perform hash calculation again based on the bucket column. That will result in uneven distribution of partitions.
   our some configs:
   ```sql
   primary-key = uuid
   bucket = 1000
   spark.sql.shuffle.partitions=1000
   ```
   ![image](https://github.com/apache/paimon/assets/135721692/8e35fa86-ade6-4817-ad0b-d5699d5b2055)
   
   
   ### What doesn't meet your expectations?
   
   partitions data distribution.
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [ ] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Bug] spark fixed bucket write causes task data process unevenly for pk table [paimon]

["PengleiShi (via GitHub)" <gi...@apache.org>]

PengleiShi commented on issue #3651:
URL: https://github.com/apache/paimon/issues/3651#issuecomment-3883387155

   @Zouxxyy Do u know whether paimon's MurmurHashUtils has same behavior with spark's Murmur3Hash? if it is, we can just repartitionBy bucket_key_col instead of bucket_col in HASH_FIXED bucket mode, now it is repartitionBy bucket_col  in paimon-spark, which may cause data skew.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org