[jira] [Resolved] (IMPALA-7349) Automatically choose mem_limit based on estimate, clamped to range

["Bikramjeet Vig (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/IMPALA-7349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Bikramjeet Vig resolved IMPALA-7349.
------------------------------------
       Resolution: Fixed
    Fix Version/s: Impala 3.1.0

https://github.com/apache/impala/commit/fc91e706b4f3b45cdda28d977f652cee3f050e7b

IMPALA-7349: Add Admission control support for automatically setting

per host memory limit for a query

With this patch the per host memory limit of a query is automatically
set using the mem_limit set in the query options and the mem_estimate
calculated by the planner based on the following pseudo code:

if mem_limit is set in query options:
  use that and if 'clamp-mem-limit-query-option' is true:
    enforce the min/max query mem limits defined in the pool config.
else:
  mem_limit = max(mem_estiamte,
    min_mem_limit_required_to_accomodate_largest_initial_reservation)
  finally, enforce min/max query mem limits defined in the pool
  config on this value.

This calculated mem limit will also be used for admission accounting
and consequently for admission control. Moreover, three new pool
configuration options have been added to enable this behaviour:

"min-query-mem-limit" & "max-query-mem-limit" => help
clamp the per host memory limit for a query. If both these limits
are not configured, then the estimates from planning are not used
as a memory limit and only used for making admission decisions.
Moreover the estimates will no longer have a lower bound based
on the largest initial reservation.

"clamp-mem-limit-query-option" => if false, the mem_limit defined in
the query options is used directly and the max/min query mem limits
are not enforced on it.

Testing:
Added e2e test cases.
Added frontend tests for changes to RequestPoolService.
Successfully passed exhaustive tests.

Change-Id: Ifec00141651982f5975803c2165b7d7a10ebeaa6
Reviewed-on: http://gerrit.cloudera.org:8080/11157
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>

> Automatically choose mem_limit based on estimate, clamped to range
> ------------------------------------------------------------------
>
>                 Key: IMPALA-7349
>                 URL: https://issues.apache.org/jira/browse/IMPALA-7349
>             Project: IMPALA
>          Issue Type: Improvement
>          Components: Backend
>            Reporter: Tim Armstrong
>            Assignee: Bikramjeet Vig
>            Priority: Major
>              Labels: admission-control, resource-management
>             Fix For: Impala 3.1.0
>
>
> We should add admission control support for intelligently choosing how much memory to give a query, based on the memory estimate. Currently we require that you set a single mem_limit per pool. Instead it would be better if we allowed configuring min/max guardrails within which admission control chose an amount of memory that will allow good performance.
> Initially, I think mem_limit will be the same for all backends. Eventually it could make sense to have different limits per backend depending on the fragments.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org