You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2019/12/04 17:33:00 UTC
[jira] [Commented] (IMPALA-9151) Number of executors during
planning needs to account for suspended executor groupa
[ https://issues.apache.org/jira/browse/IMPALA-9151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16988031#comment-16988031 ]
ASF subversion and git services commented on IMPALA-9151:
---------------------------------------------------------
Commit 69a9ac102dbb193f727e9e0e991e226ceb5f47c9 in impala's branch refs/heads/master from Lars Volker
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=69a9ac1 ]
IMPALA-9151: Maintain cluster size in ExecutorMembershipSnapshot
This change improves the cluster membership snapshot we maintain in the
frontend in cases where all executors have been shut down or none have
started yet.
Prior to this change when configuring Impala with executor groups, the
planner might see a ExecutorMembershipSnapshot that has no executors in
it. This could happen if the first executor group had not started up
yet, or if all executor groups had been shutdown. If this happened, the
planner would make sub-optimal decisions, e.g. decide on a broadcast
join vs a partitioned hash join.
With this change if no executors have been registered so far, the
planner will use the expected number of executors which can be set using
the -num_expected_executors flag and is 20 by default. After executors
come online, the planner will use the size of the largest healthy
executor group, and it will hold on to the group's size even if it shuts
down or becomes unhealthy. This allows the planner to work on the
assumption that a healthy executor group of the same size will
eventually come online to execute the query.
Change-Id: Ib6b05326c82fb3ca625c015cfcdc38f891f5d4f9
Reviewed-on: http://gerrit.cloudera.org:8080/14756
Reviewed-by: Lars Volker <lv...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
> Number of executors during planning needs to account for suspended executor groupa
> ----------------------------------------------------------------------------------
>
> Key: IMPALA-9151
> URL: https://issues.apache.org/jira/browse/IMPALA-9151
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
> Reporter: Lars Volker
> Assignee: Lars Volker
> Priority: Major
> Fix For: Impala 3.4.0
>
>
> When configuring Impala with executor groups, the planner might see a {{ExecutorMembershipSnapshot}} that has no executors in it. This can happen if the first executor group has not started up yet, or if all executor groups have been shutdown. If this happens, the planner will make sub-optimal decisions, e.g. decide on a broadcast join vs a PHJ. In the former case, we should have a configurable fallback cluster size to use during planning. In the latter case, we should hang on to the last executor group size that we had observed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org