[jira] [Commented] (SPARK-33486) Collapse Partial and Final Aggregation into Complete Aggregation mode

["Apache Spark (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-33486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17235290#comment-17235290 ] 

Apache Spark commented on SPARK-33486:
--------------------------------------

User 'prakharjain09' has created a pull request for this issue:
https://github.com/apache/spark/pull/30426

> Collapse Partial and Final Aggregation into Complete Aggregation mode
> ---------------------------------------------------------------------
>
>                 Key: SPARK-33486
>                 URL: https://issues.apache.org/jira/browse/SPARK-33486
>             Project: Spark
>          Issue Type: Task
>          Components: SQL
>    Affects Versions: 2.3.4, 2.4.7, 3.0.0, 3.0.1
>            Reporter: Prakhar Jain
>            Priority: Major
>
> We should merge the Partial and Final Aggregation into one if there is no exchange between them.
>  
> Example: select col1, max(col2) from t1 join t2 on col1 group by col1
> In this case, after the SortMergeJoin, Spark will do PartialAggregation and then FinalAggregation. So it will create HashTables two times which is not required. If there is lot of data after Join with many distinct col1's then there is a possibility of Spill also in HashAggregateExec. So Spill will also happen twice which can be avoided. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org