You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "gaoxiaoqing (Jira)" <ji...@apache.org> on 2022/09/05 12:32:00 UTC
[jira] [Commented] (IMPALA-10891) Avoid hash exchanges in more situations
[ https://issues.apache.org/jira/browse/IMPALA-10891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17600374#comment-17600374 ]
gaoxiaoqing commented on IMPALA-10891:
--------------------------------------
[~liuyao] [~stigahuang]
The definition of high cardinality: cardinality/total > FLAG_HIGH_CARDINALITY_RATIO. Cardinality can get from statistics.
If high cardinality field exists on group by statement, Other fields can be removed. I'll try to implement this task. Do you have some advice?
> Avoid hash exchanges in more situations
> ---------------------------------------
>
> Key: IMPALA-10891
> URL: https://issues.apache.org/jira/browse/IMPALA-10891
> Project: IMPALA
> Issue Type: Improvement
> Components: Frontend
> Affects Versions: Impala 4.0.0
> Reporter: liuyao
> Assignee: gaoxiaoqing
> Priority: Major
>
> if the partition keys of parent fragment and child fragment have an intersecion, and the intersection keys have a high cardinality, We can think parent fragment and child fragment to have compatible partitions
>
> The cardinality of user_id is 10000000.
>
> This case has an unnecessary hash exchange:
>
> {code:java}
> //[localhost.localdomain:21000] rawdata> Explain select count(*) from event_ros_p7 a join /* +shuffle */ event_ros_p1 b on a.user_id = b.user_id and a.month_id = b.month_id group by a.user_id, b.week_id;
> Query: Explain select count(*) from event_ros_p7 a join /* +shuffle */ event_ros_p1 b on a.user_id = b.user_id and a.month_id = b.month_id group by a.user_id, b.week_id
> +------------------------------------------------------------------------------------+
> | Explain String |
> +------------------------------------------------------------------------------------+
> | Max Per-Host Resource Reservation: Memory=81.94MB Threads=6 |
> | Per-Host Resource Estimates: Memory=324MB |
> | WARNING: The following tables are missing relevant table and/or column statistics. |
> | rawdata.event_ros_p1 |
> | |
> | PLAN-ROOT SINK |
> | | |
> | 08:EXCHANGE [UNPARTITIONED] |
> | | |
> | 07:AGGREGATE [FINALIZE] |
> | | output: count:merge(*) |
> | | group by: a.user_id, b.week_id |
> | | row-size=20B cardinality=60.18M |
> | | |
> | 06:EXCHANGE [HASH(a.user_id,b.week_id)] |
> | | |
> | 03:AGGREGATE [STREAMING] |
> | | output: count(*) |
> | | group by: a.user_id, b.week_id |
> | | row-size=20B cardinality=60.18M |
> | | |
> | 02:HASH JOIN [INNER JOIN, PARTITIONED] |
> | | hash predicates: a.month_id = b.month_id, a.user_id = b.user_id |
> | | runtime filters: RF000 <- b.month_id, RF001 <- b.user_id |
> | | row-size=28B cardinality=60.18M |
> | | |
> | |--05:EXCHANGE [HASH(b.month_id,b.user_id)] |
> | | | |
> | | 01:SCAN HDFS [rawdata.event_ros_p1 b] |
> | | partitions=0/0 files=0 size=0B |
> | | row-size=16B cardinality=0 |
> | | |
> | 04:EXCHANGE [HASH(a.month_id,a.user_id)] |
> | | |
> | 00:SCAN HDFS [rawdata.event_ros_p7 a] |
> | partitions=20/22 files=1346 size=10.48GB |
> | runtime filters: RF000 -> a.month_id, RF001 -> a.user_id |
> | row-size=12B cardinality=60.18M |
> +------------------------------------------------------------------------------------+
> {code}
>
>
> This execution plan works as expected, It avoids redundant shuffle and Pre agg:
> {code:java}
> //+------------------------------------------------------------------------------------+
> | Explain String |
> +------------------------------------------------------------------------------------+
> | Max Per-Host Resource Reservation: Memory=47.94MB Threads=5 |
> | Per-Host Resource Estimates: Memory=186MB |
> | WARNING: The following tables are missing relevant table and/or column statistics. |
> | rawdata.event_ros_p1 |
> | |
> | PLAN-ROOT SINK |
> | | |
> | 06:EXCHANGE [UNPARTITIONED] |
> | | |
> | 03:AGGREGATE [FINALIZE] |
> | | output: count(*) |
> | | group by: a.user_id, b.week_id |
> | | row-size=20B cardinality=60.18M |
> | | |
> | 02:HASH JOIN [INNER JOIN, PARTITIONED] |
> | | hash predicates: a.month_id = b.month_id, a.user_id = b.user_id |
> | | runtime filters: RF000 <- b.month_id, RF001 <- b.user_id |
> | | row-size=28B cardinality=60.18M |
> | | |
> | |--05:EXCHANGE [HASH(b.user_id)] |
> | | | |
> | | 01:SCAN HDFS [rawdata.event_ros_p1 b] |
> | | partitions=0/0 files=0 size=0B |
> | | row-size=16B cardinality=0 |
> | | |
> | 04:EXCHANGE [HASH(a.user_id)] |
> | | |
> | 00:SCAN HDFS [rawdata.event_ros_p7 a] |
> | partitions=20/22 files=1346 size=10.48GB |
> | runtime filters: RF000 -> a.month_id, RF001 -> a.user_id |
> | row-size=12B cardinality=60.18M |
> +------------------------------------------------------------------------------------+
> {code}
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org