[jira] [Commented] (IMPALA-7486) Admit less memory on dedicated coordinator for admission control purposes

["ASF subversion and git services (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IMPALA-7486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16898331#comment-16898331 ] 

ASF subversion and git services commented on IMPALA-7486:
---------------------------------------------------------

Commit a0b0fd4fce7676255dda52445cb972424f0b7c26 in impala's branch refs/heads/master from Bikramjeet Vig
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=a0b0fd4 ]

IMPALA-7486: Add specialized estimation scheme for dedicated coordinators

This patch computes two memory estimates in the frontend:
an estimate for any host that is an executor (including
a combined coordinator and executor) and an estimate
for a dedicated coordinator. This is computed regardless
of whether it is a dedicated coordinator or not.

Admission control then, in the case when the coordinator
is dedicated, uses the coordinator memory estimate for
the coordinator node and the executor memory estimate
for all other nodes.

Other highlights:
 - if MEM_LIMIT query option is set, it is applied to all backends,
   both executors and coordinators.
 - the min_query_mem_limit pool config is not enforced on the
   dedicated coordinator estimates unless MEM_LIMIT query option is set.
 - the lower cap on estimates and the admission checks based on the
   min mem limit required for reservation are applied separately on
   coordinator's and executors' mem requirements.
 - Added a hidden startup option 'use_dedicated_coordinator_estimates'
   which if set to false, reverts to previous estimation behavior.

Testing:
- Added unit test for admission/rejection in dedicated
  coordinator clusters.
- Added end to end tests.

Change-Id: I2b94e7293b91dec8a18491079c34923eadd94b21
Reviewed-on: http://gerrit.cloudera.org:8080/13740
Reviewed-by: Bikramjeet Vig <bi...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>


> Admit less memory on dedicated coordinator for admission control purposes
> -------------------------------------------------------------------------
>
>                 Key: IMPALA-7486
>                 URL: https://issues.apache.org/jira/browse/IMPALA-7486
>             Project: IMPALA
>          Issue Type: Improvement
>          Components: Backend
>            Reporter: Tim Armstrong
>            Assignee: Bikramjeet Vig
>            Priority: Major
>              Labels: resource-management, scalability
>
> Following on from IMPALA-7349, we should consider handling dedicated coordinators specially rather than admitting a uniform amount of memory on all backends.
> The specific scenario I'm interested in targeting is the case where we a coordinator that is executing many "lightweight" coordinator fragments, e.g. just an ExchangeNode and PlanRootSink, plus maybe other lightweight operators like UnionNode that don't use much memory or CPU. With the current behaviour it's possible for a coordinator to reach capacity from the point-of-view of admission control when at runtime it is actually very lightly loaded.
> This is particularly true if coordinators and executors have different process mem limits. This will be somewhat common since they're often deployed on different hardware or the coordinator will have more memory dedicated to its embedded JVM for the catalog cache.
> More generally we could admit different amounts per backend depending on how many fragments are running, but I think this incremental step would address the most important cases and be a little easier to understand.
> We may want to defer this work until we've implemented distributed runtime filter aggregation, which will significantly reduce coordinator memory pressure, and until we've improved distributed overadmission (since the coordinator behaviour may help throttle overadmission ).



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org