You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (JIRA)" <ji...@apache.org> on 2018/08/16 21:35:00 UTC
[jira] [Commented] (IMPALA-7096) Ensure no memory limit exceeded
regressions from IMPALA-4835 because of non-reserved memory
[ https://issues.apache.org/jira/browse/IMPALA-7096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16583080#comment-16583080 ]
ASF subversion and git services commented on IMPALA-7096:
---------------------------------------------------------
Commit 7ccf7369085aa49a8fc0daf6f91d97b8a3135682 in impala's branch refs/heads/master from [~tarmstrong@cloudera.com]
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=7ccf736 ]
IMPALA-7096: restore scanner thread memory heuristics
This restores some of the heuristics removed in IMPALA-4835 that can
help scans from hitting OOM conditions. The heuristics are implemented
at the query level rather than in each scan node in isolation.
Introduce a ScannerMemLimiter class that belongs to the QueryState that
tracks the amount of memory estimated to be consumed for all scanner
threads running for the query on the current backend.
Also check soft memory limits to see if scanner threads should be
started or the current scanner thread should stop.
The long-term plan is to switch to the MT scan node implementations.
When that happens this code can be removed. In the meantime this
code is imperfect but will help avoid OOM in many scenarios.
Testing:
Added regression tests for HDFS and Kudu where we previously could
run out of memory with a low mem_limit.
Manual testing:
* Ran query tests with --thread_creation_fault_injection=true for a
bit, confirmed no crashes.
* ran single-node stress test for Kudu and Parquet for 10-20 min each.
Change-Id: Ib9907fa8c4d2b0b85f67f4f160899c1c258ad82b
Reviewed-on: http://gerrit.cloudera.org:8080/11103
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
> Ensure no memory limit exceeded regressions from IMPALA-4835 because of non-reserved memory
> -------------------------------------------------------------------------------------------
>
> Key: IMPALA-7096
> URL: https://issues.apache.org/jira/browse/IMPALA-7096
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 2.13.0, Impala 3.1.0
> Reporter: Tim Armstrong
> Assignee: Tim Armstrong
> Priority: Blocker
> Labels: resource-management
> Fix For: Impala 3.1.0
>
> Attachments: ScanConsumingMostMemory.txt
>
>
> IMPALA-7078 showed some cases where non-buffer memory could accumulate in the row batch queue and cause memory consumption problems.
> The decision for whether to spin up a scanner thread in IMPALA-4835 implicitly assumes that buffer memory is the bulk of memory consumed by a scan, but there may be cases where that is not true and the previous heuristic would be more conservative about starting a scanner thread.
> We should investigate this further and figure out how to avoid it if there's an issue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org