You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/08/14 03:53:20 UTC

[GitHub] [spark] Karl-WangSK commented on pull request #29360: [SPARK-32542][SQL] Add an optimizer rule to split an Expand into multiple Expands for aggregates

Karl-WangSK commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-673868646


   yes.The shuffle output is the same, because the size of the data is the same. 
   As you can see the benchmark:
   cube 7 fields k1, k2, k3, k4, k5, k6, k7(128x projections)  and cube 6 fields k1, k2, k3, k4, k5, k6(64x projections) with  grouping off
   data size is double ,but the time ,one is 2.4min ,the another one is 8.7min, not just double time .It will be affected by data size Especially when the memory is limited.
   The original data I created is about 20M, executor memory is 1g. when it expands to 64x or  128x. It will have big impact on shuffle performance.
   
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org