[jira] [Commented] (IMPALA-8685) Evaluate default configuration of NUM_REMOTE_EXECUTOR_CANDIDATES

["ASF subversion and git services (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IMPALA-8685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907685#comment-16907685 ] 

ASF subversion and git services commented on IMPALA-8685:
---------------------------------------------------------

Commit 33475a3e2c482681f1bab307055b75b0643ecdee in impala's branch refs/heads/master from Joe McDonnell
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=33475a3 ]

IMPALA-8685,IMPALA-8677: Use consistent scheduling for small clusters

In the original change for consistent scheduling, if a cluster has
fewer nodes than the number of remote executor candidates, then
the scheduler falls back to using the old SelectRemoteExecutor().
SelectRemoteExecutor() considers all backends and picks the backend
with the least assigned bytes; to break ties, it uses randomness.
This means that clusters with fewer backends than
num_remote_executor_candidates do not have consistent placement.

For the file handle cache (the original user of consistent
placement), this is not a major problem. However, for data caching,
it can result in slower warm up of the data cache and greater
duplication of the same data across different nodes.

This changes the algorithm to use consistent placement even for
small clusters (num nodes <= num_remote_executor_candidates).
To make this more predictable, it increases the maximum number
of iterations.

This also changes GetRemoteExecutorCandidates() to return the
candidates in the order that they were selected. While still
using a set for detecting duplicate backends, the vector of
distinct backends is constructed directly rather than by
iterating over the set.

Testing:
 - Modify the scheduler-test backend test to verify that small
   clusters use consistent scheduling.

Change-Id: Icfdb2cc53d7206e316ea8a1cc28ad443f246f741
Reviewed-on: http://gerrit.cloudera.org:8080/14026
Reviewed-by: Lars Volker <lv...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>


> Evaluate default configuration of NUM_REMOTE_EXECUTOR_CANDIDATES
> ----------------------------------------------------------------
>
>                 Key: IMPALA-8685
>                 URL: https://issues.apache.org/jira/browse/IMPALA-8685
>             Project: IMPALA
>          Issue Type: Improvement
>          Components: Backend
>            Reporter: Michael Ho
>            Assignee: Joe McDonnell
>            Priority: Critical
>
> The query option {{NUM_REMOTE_EXECUTOR_CANDIDATES}} is set to 3 by default. This means that there are potentially 3 different executors which can process a remote scan range. Over time, the data of a given remote scan range will be spread across these 3 executors. My understanding of why this is not set to 1 is to avoid hot spots in pathological cases. On the other hand, this may mean that we may not maximize the utilization of the file handle cache and data cache. Also, for small clusters (e.g. a 3 node cluster), the default value may render deterministic remote scan range scheduling ineffective. We may want to re-evaluate the default value of {{NUM_REMOTE_EXECUTOR_CANDIDATES}}. One possible idea is to set it to min(3, half of cluster size) so it works okay with small cluster, which may be rather common for demo purposes. However, it doesn't address the problem of cache effectiveness in larger clusters as the footprint of the cache is still amplified by {{NUM_REMOTE_EXECUTOR_CANDIDATES}}. There may also be other criteria for evaluating the default value.
> cc'ing [~joemcdonnell], [~tlipcon] and [~drorke]



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org