You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@impala.apache.org by "Tim Armstrong (Jira)" <ji...@apache.org> on 2020/06/11 18:08:00 UTC
[jira] [Resolved] (IMPALA-9814) Analytic planner can
under-parallelise with mt_dop
[ https://issues.apache.org/jira/browse/IMPALA-9814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tim Armstrong resolved IMPALA-9814.
-----------------------------------
Fix Version/s: Impala 4.0
Resolution: Fixed
> Analytic planner can under-parallelise with mt_dop
> --------------------------------------------------
>
> Key: IMPALA-9814
> URL: https://issues.apache.org/jira/browse/IMPALA-9814
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
> Reporter: Tim Armstrong
> Assignee: Tim Armstrong
> Priority: Major
> Labels: multithreading
> Fix For: Impala 4.0
>
>
> This is a TODO-MT in AnalyticPlanner.java. Here is an example of a query that is under-parallelised
> {noformat}
>
> set mt_dop=3;
> explain select count(*) over (partition by ss_addr_sk, ss_store_sk), count(*) over (partition by ss_sold_date_sk, ss_st
> ore_sk)
> from tpcds_parquet.store_sales;
>
> +---------------------------------------------------------------------------+
> | Explain String |
> +---------------------------------------------------------------------------+
> | Max Per-Host Resource Reservation: Memory=61.50MB Threads=7 |
> | Per-Host Resource Estimates: Memory=148MB |
> | |
> | PLAN-ROOT SINK |
> | | |
> | 06:EXCHANGE [UNPARTITIONED] |
> | | |
> | 04:ANALYTIC |
> | | functions: count(*) |
> | | partition by: ss_sold_date_sk, ss_store_sk |
> | | row-size=28B cardinality=2.88M |
> | | |
> | 03:SORT |
> | | order by: ss_sold_date_sk ASC NULLS FIRST, ss_store_sk ASC NULLS FIRST |
> | | row-size=20B cardinality=2.88M |
> | | |
> | 02:ANALYTIC |
> | | functions: count(*) |
> | | partition by: ss_addr_sk, ss_store_sk |
> | | row-size=20B cardinality=2.88M |
> | | |
> | 01:SORT |
> | | order by: ss_addr_sk ASC NULLS FIRST, ss_store_sk ASC NULLS FIRST |
> | | row-size=12B cardinality=2.88M |
> | | |
> | 05:EXCHANGE [HASH(ss_store_sk)] |
> | | |
> | 00:SCAN HDFS [tpcds_parquet.store_sales] |
> | HDFS partitions=1824/1824 files=1824 size=196.96MB |
> | row-size=12B cardinality=2.88M |
> +---------------------------------------------------------------------------+
>
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)