You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (JIRA)" <ji...@apache.org> on 2019/01/09 07:01:00 UTC
[jira] [Commented] (IMPALA-7446) Queries can spill earlier than
necessary because of accumulation of free buffers and clean pages
[ https://issues.apache.org/jira/browse/IMPALA-7446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16737902#comment-16737902 ]
ASF subversion and git services commented on IMPALA-7446:
---------------------------------------------------------
Commit ae65ff831966eb99417e15738c1aa96013a66f39 in impala's branch refs/heads/master from Tim Armstrong
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=ae65ff8 ]
IMPALA-7446: enable buffer pool GC when near process mem limit
GC is performed when:
* The amount of memory allocated from the system for the buffer pool
exceeds the reservation (i.e. free buffers and clean pages are not
offset by unused reservation).
* The soft or hard process memory limit would otherwise cause an
allocation to fail.
Testing:
Looped the old version of the semi_joins_exhaustive test, which
reliably reproduced the issue. I confirmed that the buffer pool GC was
running and that it preventing the query failures.
Added a backend test that reproed the issue. A large chunk of the code
change is to add infrastructure to use TCMalloc memory metrics
for the process memory tracker in backend tests.
Ran exhaustive tests.
Change-Id: I81e8e29f1ba319f1b499032f9518d32c511b4b21
Reviewed-on: http://gerrit.cloudera.org:8080/12133
Reviewed-by: Bikramjeet Vig <bi...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
> Queries can spill earlier than necessary because of accumulation of free buffers and clean pages
> ------------------------------------------------------------------------------------------------
>
> Key: IMPALA-7446
> URL: https://issues.apache.org/jira/browse/IMPALA-7446
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 2.10.0, Impala 2.11.0, Impala 3.0, Impala 2.12.0
> Reporter: Tim Armstrong
> Assignee: Tim Armstrong
> Priority: Critical
> Labels: resource-management
> Fix For: Impala 3.2.0
>
>
> See IMPALA-7442 for an example where the query started to spill even when memory could have been made available by freeing buffers or evicting clean pages. Usually this would just result in spilling earlier than necessary, but in the case of IMPALA-7442 it lead to a query failure.
> My original intent was that BufferPool::ReleaseMemory() should be called in situations like this, but that was not done.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org