You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/12/29 12:59:10 UTC

[GitHub] [spark] cloud-fan commented on pull request #34785: [SPARK-37523][SQL] Support optimize skewed partitions in Distribution and Ordering if numPartitions is not specified

cloud-fan commented on pull request #34785:
URL: https://github.com/apache/spark/pull/34785#issuecomment-1002582437


   Can you update the PR description and fix the code conflicts? Seems we don't need to change `RepartitionByExpression` anymore.
   
   IIUC, the new proposal of this PR is: For distribution required by the v2 sink, we use Rebalance operator instead of `RepartitionByExpression`. This looks reasonable, as it's for performance and best-effort partitioning is good enough. AFAFIK @wangyum is working on something similar for file sources.
   
   We should update the DS v2 API doc though. To mention that the required distribution is just a best effort.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org