You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@impala.apache.org by "Tim Armstrong (JIRA)" <ji...@apache.org> on 2017/11/16 15:40:00 UTC
[jira] [Resolved] (IMPALA-6188) test_top_n_reclaim is flaky
[ https://issues.apache.org/jira/browse/IMPALA-6188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tim Armstrong resolved IMPALA-6188.
-----------------------------------
Resolution: Fixed
Fix Version/s: Impala 2.11.0
IMPALA-6188: make test_top_n_reclaim less flaky
Testing:
Previously I needed ~20 iterations to get the test to fail on my local
machine. After these changes I haven't been able to reproduce the
failure
Change-Id: I2bea7b0f770dec362a6df075da4e340402bd1d5d
Reviewed-on: http://gerrit.cloudera.org:8080/8562
Reviewed-by: Jim Apple <jb...@apache.org>
Tested-by: Impala Public Jenkins
---
> test_top_n_reclaim is flaky
> ---------------------------
>
> Key: IMPALA-6188
> URL: https://issues.apache.org/jira/browse/IMPALA-6188
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
> Affects Versions: Impala 2.11.0
> Reporter: Tim Armstrong
> Assignee: Tim Armstrong
> Priority: Critical
> Labels: flaky
> Fix For: Impala 2.11.0
>
>
> [~jbapple] reported a test failure here: https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/607/
> {noformat}
> 03:16:32 TestTopNReclaimQuery.test_top_n_reclaim[exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: text/none]
> 03:16:32 [gw10] linux2 -- Python 2.7.12 /home/ubuntu/Impala/bin/../infra/python/env/bin/python
> 03:16:32 query_test/test_queries.py:246: in test_top_n_reclaim
> 03:16:32 result = self.execute_query(self.QUERY, exec_options)
> 03:16:32 common/impala_test_suite.py:512: in wrapper
> 03:16:32 return function(*args, **kwargs)
> 03:16:32 common/impala_test_suite.py:537: in execute_query
> 03:16:32 return self.__execute_query(self.client, query, query_options)
> 03:16:32 common/impala_test_suite.py:604: in __execute_query
> 03:16:32 return impalad_client.execute(query, user=user)
> 03:16:32 common/impala_connection.py:160: in execute
> 03:16:32 return self.__beeswax_client.execute(sql_stmt, user=user)
> 03:16:32 beeswax/impala_beeswax.py:173: in execute
> 03:16:32 handle = self.__execute_query(query_string.strip(), user=user)
> 03:16:32 beeswax/impala_beeswax.py:341: in __execute_query
> 03:16:32 self.wait_for_completion(handle)
> 03:16:32 beeswax/impala_beeswax.py:361: in wait_for_completion
> 03:16:32 raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
> 03:16:32 E ImpalaBeeswaxException: ImpalaBeeswaxException:
> 03:16:32 E Query aborted:Memory limit exceeded
> 03:16:32 ---------------------------- Captured stderr setup -----------------------------
> 03:16:32 -- connecting to: localhost:21000
> 03:16:32 ----------------------------- Captured stderr call -----------------------------
> 03:16:32 SET batch_size=0;
> 03:16:32 SET num_nodes=0;
> 03:16:32 SET disable_codegen_rows_threshold=0;
> 03:16:32 SET disable_codegen=False;
> 03:16:32 SET abort_on_error=1;
> 03:16:32 SET mem_limit=50m;
> 03:16:32 SET exec_single_node_rows_threshold=0;
> 03:16:32 -- executing against localhost:21000
> 03:16:32 select * from tpch.lineitem order by l_orderkey desc limit 10;;
> {noformat}
> I was able to reproduce something similar locally by running in a loop:
> {noformat}
> E ImpalaBeeswaxException: ImpalaBeeswaxException:
> E Query aborted:Memory limit exceeded: Failed to allocate memory in TopNNode::ReclaimTuplePool.
> E SORT_NODE (id=1) could not allocate 190.00 B without exceeding limit.
> E Error occurred on backend tarmstrong-box:22000 by fragment c641289b5b0652a4:9ce6652000000001
> E Memory left in process limit: 8.15 GB
> E Memory left in query limit: -2.95 MB
> E Query(c641289b5b0652a4:9ce6652000000000): memory limit exceeded. Limit=50.00 MB Reservation=0 ReservationLimit=0 OtherMemory=52.95 MB Total=52.95 MB Peak=52.95 MB
> E Fragment c641289b5b0652a4:9ce6652000000000: Reservation=0 OtherMemory=8.30 KB Total=8.30 KB Peak=232.50 KB
> E EXCHANGE_NODE (id=2): Total=0 Peak=0
> E DataStreamRecvr: Total=0 Peak=0
> E PLAN_ROOT_SINK: Total=0 Peak=0
> E CodeGen: Total=305.00 B Peak=224.50 KB
> E Fragment c641289b5b0652a4:9ce6652000000001: Reservation=0 OtherMemory=52.94 MB Total=52.94 MB Peak=52.94 MB
> E SORT_NODE (id=1): Total=702.00 KB Peak=706.00 KB
> E HDFS_SCAN_NODE (id=0): Total=52.23 MB Peak=52.23 MB
> E DataStreamSender (dst_id=2): Total=688.00 B Peak=688.00 B
> E CodeGen: Total=23.94 KB Peak=1.64 MB
> E
> E Memory limit exceeded: Failed to allocate memory in TopNNode::ReclaimTuplePool.
> E SORT_NODE (id=1) could not allocate 190.00 B without exceeding limit.
> E Error occurred on backend tarmstrong-box:22000 by fragment c641289b5b0652a4:9ce6652000000001
> E Memory left in process limit: 8.15 GB
> E Memory left in query limit: -2.95 MB
> E Query(c641289b5b0652a4:9ce6652000000000): memory limit exceeded. Limit=50.00 MB Reservation=0 ReservationLimit=0 OtherMemory=52.95 MB Total=52.95 MB Peak=52.95 MB
> E Fragment c641289b5b0652a4:9ce6652000000000: Reservation=0 OtherMemory=8.30 KB Total=8.30 KB Peak=232.50 KB
> E EXCHANGE_NODE (id=2): Total=0 Peak=0
> E DataStreamRecvr: Total=0 Peak=0
> E PLAN_ROOT_SINK: Total=0 Peak=0
> E CodeGen: Total=305.00 B Peak=224.50 KB
> E Fragment c641289b5b0652a4:9ce6652000000001: Reservation=0 OtherMemory=52.94 MB Total=52.94 MB Peak=52.94 MB
> E SORT_NODE (id=1): Total=702.00 KB Peak=706.00 KB
> E HDFS_SCAN_NODE (id=0): Total=52.23 MB Peak=52.23 MB
> E DataStreamSender (dst_id=2): Total=688.00 B Peak=688.00 B
> E CodeGen: Total=23.94 KB Peak=1.64 MB
> {noformat}
> My current hypothesis is that it's spinning up an extra scanner thread in the failure case.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)