You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2021/04/12 10:36:00 UTC
[jira] [Commented] (IMPALA-10583) Result spooling hang with
unbounded spooling mem limit.
[ https://issues.apache.org/jira/browse/IMPALA-10583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17319344#comment-17319344 ]
ASF subversion and git services commented on IMPALA-10583:
----------------------------------------------------------
Commit e01b6312593197749a17dd6895f976c35eac5769 in impala's branch refs/heads/master from Riza Suminto
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=e01b631 ]
IMPALA-10583: Fix bug on unbounded result spooling memory config.
Two bugs happening on result spooling when we set two of its query
options to 0 (unbounded).
The first bug happens if max_spilled_result_spooling_mem =
0 (unbounded). max_unpinned_bytes_ in SpillableRowBatchQueue will be set
to 0, SpillableRowBatchQueue::IsFull() then will always return true, and
the query hang. This patch fix it by setting max_unpinned_bytes_ to
INT64_MAX if max_spilled_result_spooling_mem = 0.
The second bug happens if we set max_result_spooling_mem =
0 (unbounded). PlanRootSink.java will peg maxMemReservationBytes to
always equal to minMemReservationBytes. This patch fixes this by
reverting to the default max_result_spooling_mem (100MB).
Testing:
- Add test_unbounded_result_spooling_mem.
- Pass core tests.
Change-Id: If8f5e3668281bba8813f8082f45b4faa7721530e
Reviewed-on: http://gerrit.cloudera.org:8080/17187
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
> Result spooling hang with unbounded spooling mem limit.
> -------------------------------------------------------
>
> Key: IMPALA-10583
> URL: https://issues.apache.org/jira/browse/IMPALA-10583
> Project: IMPALA
> Issue Type: Bug
> Components: Backend, Frontend
> Affects Versions: Impala 4.0
> Reporter: Riza Suminto
> Assignee: Riza Suminto
> Priority: Major
>
> Looking at BufferedPlanRootSink and SpillableRowBatchQueue, it seems the code fail to anticipate if max_spilled_result_spooling_mem = 0 (unbounded). max_unpinned_bytes_ will be set to 0 and SpillableRowBatchQueue::IsFull() will always be true. We should change max_unpinned_bytes_ to INT64_MAX if max_spilled_result_spooling_mem = 0.
> Another bug happen when setting max_result_spooling_mem = 0 (unbounded). When this happen, PlanRootSink.java will peg maxMemReservationBytes to always equal to minMemReservationBytes. I think we should override it to the default max_result_spooling_mem (100MB) or set it equal to mem_limit of the query pool.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org