[jira] [Created] (IMPALA-15089) Multiple aggregation tuple size is overestimated

["Joe McDonnell (Jira)" <ji...@apache.org>]

Joe McDonnell created IMPALA-15089:
--------------------------------------

             Summary: Multiple aggregation tuple size is overestimated
                 Key: IMPALA-15089
                 URL: https://issues.apache.org/jira/browse/IMPALA-15089
             Project: IMPALA
          Issue Type: Improvement
          Components: Frontend
    Affects Versions: Impala 5.0.0
            Reporter: Joe McDonnell


For a query with multiple aggregations (e.g. TPC-DS Q67), the streaming aggregation is returning multiple tuples, but at most one will be set. However, the row size calculations simply add up all the individual tuple sizes and will overestimate the row size. This interacts with the exchange sender to limit the number of rows included in each message.

For example, on TPC-DS Q67, we have this streaming aggregation node:
{noformat}
07:AGGREGATE [STREAMING]
|  Class 0
|    output: sum(coalesce(ss_sales_price * CAST(ss_quantity AS DECIMAL(10,0)), CAST(0 AS DECIMAL(18,2))))
|    group by: i_category, i_class, i_brand, i_product_name, d_year, d_qoy, d_moy, s_store_id
|  Class 1
|    output: sum(coalesce(ss_sales_price * CAST(ss_quantity AS DECIMAL(10,0)), CAST(0 AS DECIMAL(18,2))))
|    group by: i_category, i_class, i_brand, i_product_name, d_year, d_qoy, d_moy, NULL
|  Class 2
|    output: sum(coalesce(ss_sales_price * CAST(ss_quantity AS DECIMAL(10,0)), CAST(0 AS DECIMAL(18,2))))
|    group by: i_category, i_class, i_brand, i_product_name, d_year, d_qoy, NULL, NULL
|  Class 3
|    output: sum(coalesce(ss_sales_price * CAST(ss_quantity AS DECIMAL(10,0)), CAST(0 AS DECIMAL(18,2))))
|    group by: i_category, i_class, i_brand, i_product_name, d_year, NULL, NULL, NULL
|  Class 4
|    output: sum(coalesce(ss_sales_price * CAST(ss_quantity AS DECIMAL(10,0)), CAST(0 AS DECIMAL(18,2))))
|    group by: i_category, i_class, i_brand, i_product_name, NULL, NULL, NULL, NULL
|  Class 5
|    output: sum(coalesce(ss_sales_price * CAST(ss_quantity AS DECIMAL(10,0)), CAST(0 AS DECIMAL(18,2))))
|    group by: i_category, i_class, i_brand, NULL, NULL, NULL, NULL, NULL
|  Class 6
|    output: sum(coalesce(ss_sales_price * CAST(ss_quantity AS DECIMAL(10,0)), CAST(0 AS DECIMAL(18,2))))
|    group by: i_category, i_class, NULL, NULL, NULL, NULL, NULL, NULL
|  Class 7
|    output: sum(coalesce(ss_sales_price * CAST(ss_quantity AS DECIMAL(10,0)), CAST(0 AS DECIMAL(18,2))))
|    group by: i_category, NULL, NULL, NULL, NULL, NULL, NULL, NULL
|  Class 8
|    output: sum(coalesce(ss_sales_price * CAST(ss_quantity AS DECIMAL(10,0)), CAST(0 AS DECIMAL(18,2))))
|    group by: NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL
|  mem-estimate=3.14GB mem-reservation=211.00MB thread-reservation=0
|  tuple-ids=4N,5N,6N,7N,8N,9N,10N,11N,12N row-size=1.07KB cardinality=11.63M
|  in pipelines: 00(GETNEXT){noformat}
The row size of 1.07KB is overestimate what actually gets sent through the subsequent exchange. More accurately estimating the row size for the exchange could help it size the messages appropriately.

However, AggregationNode also uses the row size to estimate the size of its hash tables. For that, it does need to know about all the tuples, because it will have a hash table for each tuple.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org