You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2023/03/23 16:21:00 UTC
[jira] [Commented] (IMPALA-12005) Explain executor group set selection criteria in query profile
[ https://issues.apache.org/jira/browse/IMPALA-12005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17704235#comment-17704235 ]
ASF subversion and git services commented on IMPALA-12005:
----------------------------------------------------------
Commit 5c1693d03d67df49236ed393ac722732986727dc in impala's branch refs/heads/master from Riza Suminto
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=5c1693d03 ]
IMPALA-12005: Describe executor group set selection in query profile
This patch adds new profile counters under the Frontend profile node to
describe executor group set selection during query planning. It modifies
FrontendProfile.java to allow one level of TRuntimeProfileNode nesting
under the Frontend profile node. This makes it possible to group profile
counters specific to each executor group set in consideration.
"fragment-costs" hint is renamed to "segment-costs". A new
"cpu-comparison-result" hint is added after "segment-costs" to help
navigate how cpu sizing decision is made.
This patch also adds some function overloading in runtime-profile.cc to
hide TotalTime and InactiveTotalTime that is meaningless for anything
under the Frontend profile node. Additional context also added into
AnalysisException threw when none of the executor group sets fits the
query requirement.
This is how the Frontend profile node looks like after running
TestExecutorGroups::test_query_cpu_count_divisor_fraction
Frontend:
Referenced Tables: tpcds_parquet.store_sales
- CpuCountDivisor: 0.20
- ExecutorGroupsConsidered: 3 (3)
Executor group 1 (root.tiny):
Verdict: not enough cpu cores
- CpuAsk: 15 (15)
- CpuMax: 2 (2)
- EffectiveParallelism: 3 (3)
- MemoryAsk: 36.83 MB (38617088)
- MemoryMax: 64.00 MB (67108864)
Executor group 2 (root.small):
Verdict: not enough cpu cores
- CpuAsk: 25 (25)
- CpuMax: 16 (16)
- EffectiveParallelism: 5 (5)
- MemoryAsk: 36.83 MB (38624004)
- MemoryMax: 64.00 MB (67108864)
Executor group 3 (root.large):
Verdict: Match
- CpuAsk: 35 (35)
- CpuMax: 192 (192)
- EffectiveParallelism: 7 (7)
- MemoryAsk: 36.84 MB (38633570)
- MemoryMax: 8388608.00 GB (9007199254740992)
Testing:
- Pass core tests
Change-Id: I6c0ac7f5216d631e4439fe97702e21e06d2eda8a
Reviewed-on: http://gerrit.cloudera.org:8080/19628
Reviewed-by: Riza Suminto <ri...@cloudera.com>
Tested-by: Riza Suminto <ri...@cloudera.com>
> Explain executor group set selection criteria in query profile
> --------------------------------------------------------------
>
> Key: IMPALA-12005
> URL: https://issues.apache.org/jira/browse/IMPALA-12005
> Project: IMPALA
> Issue Type: Improvement
> Components: Frontend
> Affects Versions: Impala 4.2.0
> Reporter: Riza Suminto
> Assignee: Riza Suminto
> Priority: Critical
> Fix For: Impala 4.3.0
>
>
> [IMPALA-10992|http://issues.cloudera.org/browse/IMPALA-10992] and [IMPALA-11604|http://issues.cloudera.org/browse/IMPALA-11604] adds memory and cpu constraint criteria for executor group set selection. We should explain this selection criteria in query profile. Things that we should consider adding to query profile:
> * The effective degree of parallelism (EDoP) value that was used to calculate fitness for a given group. Possibly some additional details about how we got to that EDoP like the per fragment values.
> * Per host memory estimate for that group.
> * Reason for not choosing a given group (if any) - was it based on memory or CPU, what thresholds were we comparing to, etc.
> In the event of none executor group set is fit, the returned AnalysisException also need to provide more context. Right now, only these are returned:
> {code:java}
> AnalysisException: The query does not fit any executor group sets. {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org