You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Tim Armstrong (JIRA)" <ji...@apache.org> on 2018/11/02 23:54:00 UTC
[jira] [Resolved] (IMPALA-4816) There is no way to determine if a
filter was dropped due to OOM based on profile
[ https://issues.apache.org/jira/browse/IMPALA-4816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tim Armstrong resolved IMPALA-4816.
-----------------------------------
Resolution: Fixed
Assignee: Bikramjeet Vig (was: Henry Robinson)
Fix Version/s: Impala 2.12.0
IMPALA-5519 avoids this entirely by reserving the memory.
> There is no way to determine if a filter was dropped due to OOM based on profile
> --------------------------------------------------------------------------------
>
> Key: IMPALA-4816
> URL: https://issues.apache.org/jira/browse/IMPALA-4816
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 2.6.0, Impala 2.7.0, Impala 2.8.0, Impala 2.9.0
> Reporter: Tim Armstrong
> Assignee: Bikramjeet Vig
> Priority: Major
> Labels: supportability
> Fix For: Impala 2.12.0
>
>
> I just spent some time debugging a test failure that was the result of a bloom filter not being sent due to out of memory. There's no obvious way to tell if this happened by looking at the profile - it looks like the filter arrived and was just ineffective. It looks like the filter can be dropped in RuntimeFilterBank::PublishGlobalFilter() and PhjBuilder::AllocateRuntimeFilters()
> {code}
> Filter 0 (128.00 MB):
> - Files processed: 8 (8)
> - Files rejected: 0 (0)
> - Files total: 8 (8)
> - RowGroups processed: 8 (8)
> - RowGroups rejected: 0 (0)
> - RowGroups total: 8 (8)
> - Rows processed: 2.46K (2460)
> - Rows rejected: 0 (0)
> - Rows total: 2.46K (2460)
> - Splits processed: 8 (8)
> - Splits rejected: 0 (0)
> - Splits total: 8 (8)
> {code}
> I ran into it with the follow query on my buffer pool branch. You can probably replicate it on master by tweaking mem_limit.
> {code}
> [localhost:21000] > use functional_parquet; SET RUNTIME_FILTER_MODE=GLOBAL; SET RUNTIME_FILTER_WAIT_TIME_MS=30000; SET RUNTIME_FILTER_MIN_SIZE=128MB; SET RUNTIME_FILTER_MAX_SIZE=500MB; SET MEM_LIMIT=140MB; select STRAIGHT_JOIN * from alltypes a join [BROADCAST] alltypes b on a.month = b.id and b.int_col = -3;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org