You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/05/19 07:05:56 UTC

[GitHub] [spark] cloud-fan commented on pull request #28123: [SPARK-31350][SQL] Coalesce bucketed tables for join if applicable

cloud-fan commented on pull request #28123:
URL: https://github.com/apache/spark/pull/28123#issuecomment-630627303


   The general idea LGTM. Two high-level questions ：
   1. It's not possible to have an accurate cost model to guide this optimization. Do we have a heuristic? like coalescing 10000 buckets to 2 is very likely to cause regression as parallelism is reduced too much.
   2. Are we going to support joins with more than 2 tables? e.g. "100 buckets table" join "50 buckets table" join "10 buckets table".


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org