You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2023/05/03 22:53:00 UTC
[jira] [Commented] (IMPALA-12106) Union fragment without scan node can be overparallelized by backend scheduler by 1
[ https://issues.apache.org/jira/browse/IMPALA-12106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17719074#comment-17719074 ]
ASF subversion and git services commented on IMPALA-12106:
----------------------------------------------------------
Commit 69da2ff86ec0914419e3b2d2ce755d1bb73c46aa in impala's branch refs/heads/master from Riza Suminto
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=69da2ff86 ]
IMPALA-12106: Fix overparallelization of Union fragment by 1
IMPALA-10973 has a bug where a union fragment without a scan node can be
over-parallelized by the backend scheduler by 1. It is reproducible by
running TPC-DS Q11 with MT_DOP=1. This patch additionally checks that
such a fragment does not have an input fragment before randomizing the
host assignment.
Testing:
Add TPC-DS Q11 to test_mt_dop.py::TestMtDopScheduling::test_scheduling
and verify the number of fragment instances scheduled in the
ExecSummary.
Change-Id: Ic69e7c8c0cadb4b07ee398aff362fbc6513eb08d
Reviewed-on: http://gerrit.cloudera.org:8080/19816
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
> Union fragment without scan node can be overparallelized by backend scheduler by 1
> ----------------------------------------------------------------------------------
>
> Key: IMPALA-12106
> URL: https://issues.apache.org/jira/browse/IMPALA-12106
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 4.2.0
> Reporter: Riza Suminto
> Assignee: Riza Suminto
> Priority: Major
> Attachments: profile_f04aed6e613865a1_acb29fb100000000.txt
>
>
> IMPALA-10973 has a bug where union fragment without scan node can be overparallelized by backend scheduler by 1. This can be reproduced by running TPC-DS Q11 with MT_DOP=1.
> Planner will plan 2 instances of F11.
> {code:java}
> | F11:PLAN FRAGMENT [RANDOM] hosts=2 instances=2
> | Per-Instance Resources: mem-estimate=27.53MB mem-reservation=17.00MB thread-reservation=1
> | 14:UNION
> | | mem-estimate=0B mem-reservation=0B thread-reservation=0
> | | tuple-ids=28 row-size=44B cardinality=14.80K
> | | in pipelines: 41(GETNEXT)
> | |
> | 41:AGGREGATE [FINALIZE]
> | | output: sum:merge(ws_ext_list_price - ws_ext_discount_amt)
> | | group by: c_customer_id, c_first_name, c_last_name, c_preferred_cust_flag, c_birth_country, c_login, c_email_address, d_year
> | | having: sum(ws_ext_list_price - ws_ext_discount_amt) > CAST(0 AS DECIMAL(3,0))
> | | mem-estimate=17.00MB mem-reservation=17.00MB spill-buffer=1.00MB thread-reservation=0
> | | tuple-ids=27 row-size=169B cardinality=14.80K
> | | in pipelines: 41(GETNEXT), 16(OPEN)
> | |
> | 40:EXCHANGE [HASH(c_customer_id,c_first_name,c_last_name,c_preferred_cust_flag,c_birth_country,c_login,c_email_address,d_year)]
> | | mem-estimate=10.34MB mem-reservation=0B thread-reservation=0
> | | tuple-ids=27 row-size=169B cardinality=148.00K
> | | in pipelines: 16(GETNEXT) {code}
> But backend scheduler will schedule 1 extra instance of F11.
> {code:java}
> | F11:EXCHANGE SENDER 3 3 130.372us 157.038us 52.86 KB 192.00 KB
> | 14:UNION 3 3 113.073us 196.543us 1.33K 14.80K 8.00 KB 0
> | 41:AGGREGATE 3 3 1.437ms 1.787ms 1.33K 14.80K 17.11 MB 17.00 MB FINALIZE
> | 40:EXCHANGE 3 3 82.561us 116.482us 1.33K 148.00K 104.00 KB 10.34 MB HASH(c_customer_id,c_first_name,c_last_name,c_preferred_cust_flag,c_birth_country,c_login,c_email_address,d_year) {code}
> This is because backend scheduler mistakenly think that this fragment is free to get assigned randomly because it does not have scan node and its num input fragment is less than num backend.
> [https://github.com/apache/impala/blob/112bab64b77d6ed966b1c67bd503ed632da6f208/be/src/scheduling/scheduler.cc#L441]
> This branch should additionally check if instances_per_host.empty().
> Attached is the full profile.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org