You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (JIRA)" <ji...@apache.org> on 2018/06/07 07:01:01 UTC
[jira] [Commented] (IMPALA-7096) Confirm that IMPALA-4835 does not
increase chance of OOM for scans of wide tables
[ https://issues.apache.org/jira/browse/IMPALA-7096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16504321#comment-16504321 ]
ASF subversion and git services commented on IMPALA-7096:
---------------------------------------------------------
Commit 68aeb71b4b849bddb2d55e77ec90d12c6a44dd6d in impala's branch refs/heads/master from [~tarmstrong@cloudera.com]
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=68aeb71 ]
IMPALA-7078: Part 2: reduce queue size based on num_scanner_threads
This gives a tighter bound on memory consumption when running with
a lower num_scanner_threads value. With IMPALA-7096 we'll revisit
the approach to reliably avoid OOM.
Cap the maximum row batch queue size at 5 * max_num_scanner_threads_
so that we guarantee lower amounts of memory in the row batch queue
when num_scanner_threads is set, rather than just achieving it
statistically because of the producer running slower relative to
consumer. It does not reduce the default significantly on typical
server configurations that would have 24+ cores except under high
concurrency or low memory environments where the number of scanner
threads is limited. We should evaluate reducing the default further
or otherwise better controlling memory consumption in a follow-up,
based on experiments.
Testing:
Tested along with Part 1.
Change-Id: I8c0bd14e6bcd9fc1655e344a5307ea0eb4600e8b
Reviewed-on: http://gerrit.cloudera.org:8080/10573
Reviewed-by: Tim Armstrong <ta...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
> Confirm that IMPALA-4835 does not increase chance of OOM for scans of wide tables
> ---------------------------------------------------------------------------------
>
> Key: IMPALA-7096
> URL: https://issues.apache.org/jira/browse/IMPALA-7096
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 2.13.0, Impala 3.1.0
> Reporter: Tim Armstrong
> Assignee: Tim Armstrong
> Priority: Blocker
> Labels: resource-management
>
> IMPALA-7078 showed some cases where non-buffer memory could accumulate in the row batch queue and cause memory consumption problems.
> The decision for whether to spin up a scanner thread in IMPALA-4835 implicitly assumes that buffer memory is the bulk of memory consumed by a scan, but there may be cases where that is not true and the previous heuristic would be more conservative about starting a scanner thread.
> We should investigate this further and figure out how to avoid it if there's an issue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org