You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Joe McDonnell (Jira)" <ji...@apache.org> on 2026/06/09 22:06:00 UTC
[jira] [Updated] (IMPALA-15089) Multiple aggregation row size is overestimated
[ https://issues.apache.org/jira/browse/IMPALA-15089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joe McDonnell updated IMPALA-15089:
-----------------------------------
Summary: Multiple aggregation row size is overestimated (was: Multiple aggregation tuple size is overestimated)
> Multiple aggregation row size is overestimated
> ----------------------------------------------
>
> Key: IMPALA-15089
> URL: https://issues.apache.org/jira/browse/IMPALA-15089
> Project: IMPALA
> Issue Type: Improvement
> Components: Frontend
> Affects Versions: Impala 5.0.0
> Reporter: Joe McDonnell
> Priority: Major
>
> For a query with multiple aggregations (e.g. TPC-DS Q67), the streaming aggregation is returning multiple tuples, but at most one will be set. However, the row size calculations simply add up all the individual tuple sizes and will overestimate the row size. This interacts with the exchange sender to limit the number of rows included in each message.
> For example, on TPC-DS Q67, we have this streaming aggregation node:
> {noformat}
> 07:AGGREGATE [STREAMING]
> | Class 0
> | output: sum(coalesce(ss_sales_price * CAST(ss_quantity AS DECIMAL(10,0)), CAST(0 AS DECIMAL(18,2))))
> | group by: i_category, i_class, i_brand, i_product_name, d_year, d_qoy, d_moy, s_store_id
> | Class 1
> | output: sum(coalesce(ss_sales_price * CAST(ss_quantity AS DECIMAL(10,0)), CAST(0 AS DECIMAL(18,2))))
> | group by: i_category, i_class, i_brand, i_product_name, d_year, d_qoy, d_moy, NULL
> | Class 2
> | output: sum(coalesce(ss_sales_price * CAST(ss_quantity AS DECIMAL(10,0)), CAST(0 AS DECIMAL(18,2))))
> | group by: i_category, i_class, i_brand, i_product_name, d_year, d_qoy, NULL, NULL
> | Class 3
> | output: sum(coalesce(ss_sales_price * CAST(ss_quantity AS DECIMAL(10,0)), CAST(0 AS DECIMAL(18,2))))
> | group by: i_category, i_class, i_brand, i_product_name, d_year, NULL, NULL, NULL
> | Class 4
> | output: sum(coalesce(ss_sales_price * CAST(ss_quantity AS DECIMAL(10,0)), CAST(0 AS DECIMAL(18,2))))
> | group by: i_category, i_class, i_brand, i_product_name, NULL, NULL, NULL, NULL
> | Class 5
> | output: sum(coalesce(ss_sales_price * CAST(ss_quantity AS DECIMAL(10,0)), CAST(0 AS DECIMAL(18,2))))
> | group by: i_category, i_class, i_brand, NULL, NULL, NULL, NULL, NULL
> | Class 6
> | output: sum(coalesce(ss_sales_price * CAST(ss_quantity AS DECIMAL(10,0)), CAST(0 AS DECIMAL(18,2))))
> | group by: i_category, i_class, NULL, NULL, NULL, NULL, NULL, NULL
> | Class 7
> | output: sum(coalesce(ss_sales_price * CAST(ss_quantity AS DECIMAL(10,0)), CAST(0 AS DECIMAL(18,2))))
> | group by: i_category, NULL, NULL, NULL, NULL, NULL, NULL, NULL
> | Class 8
> | output: sum(coalesce(ss_sales_price * CAST(ss_quantity AS DECIMAL(10,0)), CAST(0 AS DECIMAL(18,2))))
> | group by: NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL
> | mem-estimate=3.14GB mem-reservation=211.00MB thread-reservation=0
> | tuple-ids=4N,5N,6N,7N,8N,9N,10N,11N,12N row-size=1.07KB cardinality=11.63M
> | in pipelines: 00(GETNEXT){noformat}
> The row size of 1.07KB is overestimate what actually gets sent through the subsequent exchange. More accurately estimating the row size for the exchange could help it size the messages appropriately.
> However, AggregationNode also uses the row size to estimate the size of its hash tables. For that, it does need to know about all the tuples, because it will have a hash table for each tuple.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org