[jira] [Commented] (IMPALA-8791) Handle the case where there is no fragment scheduled on the coordinator for a query

["ASF subversion and git services (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IMPALA-8791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16905625#comment-16905625 ] 

ASF subversion and git services commented on IMPALA-8791:
---------------------------------------------------------

Commit 760169edcbca438c5964380a604b6c271c6bd1a3 in impala's branch refs/heads/master from Bikramjeet Vig
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=760169e ]

IMPALA-8791: Handle the case where there is no fragment scheduled on
the coordinator

This patch fixes a bug where if an insert or CTAS query has no
fragments scheduled on the coordinator and a mem limit is to be
enforced on the query (either through query option or automatically
through estimates) then the same limit is also applied to the
coordinator backend even though it does not execute anything.

Highlights:
- coord_backend_mem_to_admit_/mem_limit will always refer to the memory
to admit/limit for the coordinator regardless of which fragments are
scheduled on it.

- There will always be a BackendExecParams added for the coordinator
because coordinator always spawns a QueryState object with a mem_tracker
for tracking runtime filter mem and the result set cache. For the case
where this BackendExecParams is empty (no instances scheduled) it would
ensure that some minimal amount of memory is accounted for by the
admission controller and the right mem limit is applied to the
QueryState spawned by the coordinator

- added changes to Coordinator and Coordinator::BackendState classes
to handle an empty BackendExecParams object

Testing:
The following cases need to be tested where the kind of fragments
schduled on the coordinator backend are:
1. Coordinator fragment + other exec fragments
2. Coordinator fragment only
3. other exec fragments only (eg. insert into values OR insert
   into select 1)
4. No fragments, but coordinator still creates a QueryState

Case 1 is covered by tests working with non-dedicated coordinators.
Rest are covered by test_mem_limit_dedicated_coordinator in
test_admission_controller.py

Change-Id: If5631fa1490d9612ffac3c4c4715348de47d6df2
Reviewed-on: http://gerrit.cloudera.org:8080/13992
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>


> Handle the case where there is no fragment scheduled on the coordinator for a query
> -----------------------------------------------------------------------------------
>
>                 Key: IMPALA-8791
>                 URL: https://issues.apache.org/jira/browse/IMPALA-8791
>             Project: IMPALA
>          Issue Type: Bug
>    Affects Versions: Product Backlog
>            Reporter: Bikramjeet Vig
>            Assignee: Bikramjeet Vig
>            Priority: Major
>
> For insert statements executed on a dedicated coord, the fragments get schduled only on executors but a query state object still gets started up on the coord host with the coord_mem_limit. we end up with a situation where the mem admitted is zero for the coord but the mem_reserved is non-zero which would affect other admission decisions.
> There is also a case where there is no coordinator fragment but the execution fragment gets scheduled on the coord (eg. insert into <tbl> values....) for this case, the mem admitted is per_backend_mem_limit_ but the mem limit applied to the coord query state is coord_backend_mem_limit_ which again causes a inconsistency.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org