You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Joe McDonnell (Jira)" <ji...@apache.org> on 2026/06/09 20:49:00 UTC
[jira] [Comment Edited] (IMPALA-15089) Multiple aggregation tuple size is overestimated
[ https://issues.apache.org/jira/browse/IMPALA-15089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18087753#comment-18087753 ]
Joe McDonnell edited comment on IMPALA-15089 at 6/9/26 8:48 PM:
----------------------------------------------------------------
Code for calculating row size in descriptors.cc (used for KrpcDataStreamSender):
[https://github.com/apache/impala/blob/498d24d982affcc92ebaedf700be2c3156d8c1bc/be/src/runtime/descriptors.cc#L506-L512]
Code for calculating row size in PlanNode: [https://github.com/apache/impala/blob/498d24d982affcc92ebaedf700be2c3156d8c1bc/fe/src/main/java/org/apache/impala/planner/PlanNode.java#L742-L756]
was (Author: joemcdonnell):
Code for calculating row size in PlanNode: https://github.com/apache/impala/blob/498d24d982affcc92ebaedf700be2c3156d8c1bc/fe/src/main/java/org/apache/impala/planner/PlanNode.java#L742-L756
> Multiple aggregation tuple size is overestimated
> ------------------------------------------------
>
> Key: IMPALA-15089
> URL: https://issues.apache.org/jira/browse/IMPALA-15089
> Project: IMPALA
> Issue Type: Improvement
> Components: Frontend
> Affects Versions: Impala 5.0.0
> Reporter: Joe McDonnell
> Priority: Major
>
> For a query with multiple aggregations (e.g. TPC-DS Q67), the streaming aggregation is returning multiple tuples, but at most one will be set. However, the row size calculations simply add up all the individual tuple sizes and will overestimate the row size. This interacts with the exchange sender to limit the number of rows included in each message.
> For example, on TPC-DS Q67, we have this streaming aggregation node:
> {noformat}
> 07:AGGREGATE [STREAMING]
> | Class 0
> | output: sum(coalesce(ss_sales_price * CAST(ss_quantity AS DECIMAL(10,0)), CAST(0 AS DECIMAL(18,2))))
> | group by: i_category, i_class, i_brand, i_product_name, d_year, d_qoy, d_moy, s_store_id
> | Class 1
> | output: sum(coalesce(ss_sales_price * CAST(ss_quantity AS DECIMAL(10,0)), CAST(0 AS DECIMAL(18,2))))
> | group by: i_category, i_class, i_brand, i_product_name, d_year, d_qoy, d_moy, NULL
> | Class 2
> | output: sum(coalesce(ss_sales_price * CAST(ss_quantity AS DECIMAL(10,0)), CAST(0 AS DECIMAL(18,2))))
> | group by: i_category, i_class, i_brand, i_product_name, d_year, d_qoy, NULL, NULL
> | Class 3
> | output: sum(coalesce(ss_sales_price * CAST(ss_quantity AS DECIMAL(10,0)), CAST(0 AS DECIMAL(18,2))))
> | group by: i_category, i_class, i_brand, i_product_name, d_year, NULL, NULL, NULL
> | Class 4
> | output: sum(coalesce(ss_sales_price * CAST(ss_quantity AS DECIMAL(10,0)), CAST(0 AS DECIMAL(18,2))))
> | group by: i_category, i_class, i_brand, i_product_name, NULL, NULL, NULL, NULL
> | Class 5
> | output: sum(coalesce(ss_sales_price * CAST(ss_quantity AS DECIMAL(10,0)), CAST(0 AS DECIMAL(18,2))))
> | group by: i_category, i_class, i_brand, NULL, NULL, NULL, NULL, NULL
> | Class 6
> | output: sum(coalesce(ss_sales_price * CAST(ss_quantity AS DECIMAL(10,0)), CAST(0 AS DECIMAL(18,2))))
> | group by: i_category, i_class, NULL, NULL, NULL, NULL, NULL, NULL
> | Class 7
> | output: sum(coalesce(ss_sales_price * CAST(ss_quantity AS DECIMAL(10,0)), CAST(0 AS DECIMAL(18,2))))
> | group by: i_category, NULL, NULL, NULL, NULL, NULL, NULL, NULL
> | Class 8
> | output: sum(coalesce(ss_sales_price * CAST(ss_quantity AS DECIMAL(10,0)), CAST(0 AS DECIMAL(18,2))))
> | group by: NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL
> | mem-estimate=3.14GB mem-reservation=211.00MB thread-reservation=0
> | tuple-ids=4N,5N,6N,7N,8N,9N,10N,11N,12N row-size=1.07KB cardinality=11.63M
> | in pipelines: 00(GETNEXT){noformat}
> The row size of 1.07KB is overestimate what actually gets sent through the subsequent exchange. More accurately estimating the row size for the exchange could help it size the messages appropriately.
> However, AggregationNode also uses the row size to estimate the size of its hash tables. For that, it does need to know about all the tuples, because it will have a hash table for each tuple.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org