You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@impala.apache.org by "Tim Armstrong (JIRA)" <ji...@apache.org> on 2017/11/23 00:18:00 UTC
[jira] [Resolved] (IMPALA-2834) TestNestedTypes.test_tpch fails
with memory limit exceeded
[ https://issues.apache.org/jira/browse/IMPALA-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tim Armstrong resolved IMPALA-2834.
-----------------------------------
Resolution: Cannot Reproduce
> TestNestedTypes.test_tpch fails with memory limit exceeded
> ----------------------------------------------------------
>
> Key: IMPALA-2834
> URL: https://issues.apache.org/jira/browse/IMPALA-2834
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 2.5.0
> Reporter: Tim Armstrong
> Priority: Minor
> Labels: resource-management
>
> http://sandbox.jenkins.cloudera.com/job/impala-master-cdh5-trunk/1631/testReport/junit/query_test.test_nested_types/TestNestedTypes/test_tpch_exec_option____disable_codegen___False___abort_on_error___1___exec_single_node_rows_threshold___0___batch_size___0___num_nodes___0____table_format__parquet_none_/
> The scan node was using too much memory. I think this is probably just non-determinism in scan memory usage and the heuristics around it, i.e. IMPALA-2831 , but we should keep an eye on this test to see if there is a possible actual regression, or if we need to fix the flaky test.
> {code}
> Error Message
> query_test/test_nested_types.py:64: in test_tpch self.run_test_case('QueryTest/nested-types-tpch', vector) common/impala_test_suite.py:216: in run_test_case result = self.__execute_query(target_impalad_client, query, user=user) common/impala_test_suite.py:406: in __execute_query return impalad_client.execute(query, user=user) common/impala_connection.py:158: in execute return self.__beeswax_client.execute(sql_stmt, user=user) beeswax/impala_beeswax.py:162: in execute handle = self.__execute_query(query_string.strip(), user=user) beeswax/impala_beeswax.py:328: in __execute_query self.wait_for_completion(handle) beeswax/impala_beeswax.py:348: in wait_for_completion raise ImpalaBeeswaxException("Query aborted:" + error_log, None) E ImpalaBeeswaxException: ImpalaBeeswaxException: E Query aborted: E Memory limit exceeded E Failed to allocate buffer for collection 'tpch_nested_parquet.customer.c_orders.item.o_lineitems'. E E E E Memory Limit Exceeded E Query(6b4a1c66eac9b1e7:fc0c1c45d0a518a1) Limit: Limit=1.00 GB Consumption=1020.76 MB E Fragment 6b4a1c66eac9b1e7:fc0c1c45d0a518a2: Consumption=8.00 KB E EXCHANGE_NODE (id=10): Consumption=0 E DataStreamRecvr: Consumption=0 E Block Manager: Limit=819.20 MB Consumption=0 E Fragment 6b4a1c66eac9b1e7:fc0c1c45d0a518a3: Consumption=1020.75 MB E SORT_NODE (id=9): Consumption=28.00 KB E SUBPLAN_NODE (id=1): Consumption=14.89 MB E HDFS_SCAN_NODE (id=0): Consumption=1005.78 MB E NESTED_LOOP_JOIN_NODE (id=8): Consumption=24.00 KB E SUBPLAN_NODE (id=4): Consumption=8.00 KB E UNNEST_NODE (id=3): Consumption=0 E NESTED_LOOP_JOIN_NODE (id=7): Consumption=16.00 KB E UNNEST_NODE (id=6): Consumption=0 E SINGULAR_ROW_SRC_NODE (id=5): Consumption=0 E SINGULAR_ROW_SRC_NODE (id=2): Consumption=0 E DataStreamSender: Consumption=320.00 B
> Stacktrace
> query_test/test_nested_types.py:64: in test_tpch
> self.run_test_case('QueryTest/nested-types-tpch', vector)
> common/impala_test_suite.py:216: in run_test_case
> result = self.__execute_query(target_impalad_client, query, user=user)
> common/impala_test_suite.py:406: in __execute_query
> return impalad_client.execute(query, user=user)
> common/impala_connection.py:158: in execute
> return self.__beeswax_client.execute(sql_stmt, user=user)
> beeswax/impala_beeswax.py:162: in execute
> handle = self.__execute_query(query_string.strip(), user=user)
> beeswax/impala_beeswax.py:328: in __execute_query
> self.wait_for_completion(handle)
> beeswax/impala_beeswax.py:348: in wait_for_completion
> raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
> E ImpalaBeeswaxException: ImpalaBeeswaxException:
> E Query aborted:
> E Memory limit exceeded
> E Failed to allocate buffer for collection 'tpch_nested_parquet.customer.c_orders.item.o_lineitems'.
> E
> E
> E
> E Memory Limit Exceeded
> E Query(6b4a1c66eac9b1e7:fc0c1c45d0a518a1) Limit: Limit=1.00 GB Consumption=1020.76 MB
> E Fragment 6b4a1c66eac9b1e7:fc0c1c45d0a518a2: Consumption=8.00 KB
> E EXCHANGE_NODE (id=10): Consumption=0
> E DataStreamRecvr: Consumption=0
> E Block Manager: Limit=819.20 MB Consumption=0
> E Fragment 6b4a1c66eac9b1e7:fc0c1c45d0a518a3: Consumption=1020.75 MB
> E SORT_NODE (id=9): Consumption=28.00 KB
> E SUBPLAN_NODE (id=1): Consumption=14.89 MB
> E HDFS_SCAN_NODE (id=0): Consumption=1005.78 MB
> E NESTED_LOOP_JOIN_NODE (id=8): Consumption=24.00 KB
> E SUBPLAN_NODE (id=4): Consumption=8.00 KB
> E UNNEST_NODE (id=3): Consumption=0
> E NESTED_LOOP_JOIN_NODE (id=7): Consumption=16.00 KB
> E UNNEST_NODE (id=6): Consumption=0
> E SINGULAR_ROW_SRC_NODE (id=5): Consumption=0
> E SINGULAR_ROW_SRC_NODE (id=2): Consumption=0
> E DataStreamSender: Consumption=320.00 B
> Standard Error
> -- executing against localhost:21000
> use functional_parquet;
> SET disable_codegen=False;
> SET abort_on_error=1;
> SET exec_single_node_rows_threshold=0;
> SET batch_size=0;
> SET num_nodes=0;
> -- executing against localhost:21000
> use tpch_nested_parquet;
> -- executing against localhost:21000
> select count(*)
> from tpch_nested_parquet.customer c, c.c_orders o, o.o_lineitems i
> where c_custkey in (1, 2) and o_orderkey in (4808192, 1374019);
> -- executing against localhost:21000
> select count(*)
> from tpch_nested_parquet.customer c, c.c_orders o, o.o_lineitems i;
> -- executing against localhost:21000
> select count(l_linenumber)
> from tpch_nested_parquet.customer.c_orders.o_lineitems i;
> -- executing against localhost:21000
> select count(*)
> from tpch_nested_parquet.customer.c_orders.o_lineitems i;
> -- executing against localhost:21000
> select count(l_linenumber) from tpch_parquet.lineitem;
> -- executing against localhost:21000
> select c_custkey, count(*)
> from tpch_parquet.customer join tpch_parquet.orders on c_custkey = o_custkey
> where c_custkey < 10
> group by c_custkey
> order by c_custkey;
> -- executing against localhost:21000
> select c_custkey from tpch_nested_parquet.customer c left anti join c.c_orders
> where c_custkey < 10 order by c_custkey;
> -- executing against localhost:21000
> select c_custkey, c_name, o_orderkey, o_orderpriority
> from tpch_nested_parquet.customer c, c.c_orders o
> where c_custkey < 100
> order by o_orderpriority, o_orderkey
> limit 10;
> -- executing against localhost:21000
> SELECT t2.s_name
> FROM tpch_nested_parquet.customer t1
> INNER JOIN tpch_nested_parquet.supplier t2
> ON t2.s_phone = t1.c_phone AND t2.s_acctbal = t1.c_acctbal,
> t2.s_partsupps t3
> WHERE t2.s_suppkey > t2.s_nationkey;
> -- executing against localhost:21000
> SELECT STRAIGHT_JOIN r_regionkey, p_retailprice
> FROM (SELECT p1.p_partkey, p2.p_retailprice, p2.p_mfgr
> FROM part p1 INNER JOIN part p2 ON p1.p_partkey = p2.p_partkey
> WHERE p1.p_partkey < 100 AND p2.p_partkey < 100) t2
> RIGHT JOIN (SELECT r_name, r_regionkey
> FROM region r, r.r_nations
> WHERE r_name = 'AFRICA') t1
> ON t1.r_regionkey = t2.p_partkey;
> -- executing against localhost:21000
> SELECT COALESCE(t2.p_size, t4.pos, o_orderkey) AS int_col, COUNT(*)
> FROM customer t1
> LEFT JOIN part t2 ON t1.c_custkey = t2.p_partkey,
> t1.c_orders t3,
> t3.item.o_lineitems t4
> WHERE t1.c_custkey < 100 AND t2.p_partkey < 100
> GROUP BY 1
> HAVING int_col <= 5
> ORDER BY 1;
> -- executing against localhost:21000
> SELECT o_orderkey, r FROM
> (SELECT o2.o_orderkey, COUNT(o1.pos) OVER (ORDER BY o2.o_orderkey DESC) r
> FROM tpch_nested_parquet.customer c
> JOIN c.c_orders o1
> JOIN c.c_orders o2 ON (o1.pos = o2.pos)) v
> ORDER BY o_orderkey LIMIT 1;
> -- executing against localhost:21000
> set mem_limit=4m;
> -- executing against localhost:21000
> select max(cnt) from customer c,
> (select count(l_returnflag) cnt from c.c_orders.o_lineitems) v;
> -- executing against localhost:21000
> set num_scanner_threads=1;
> -- executing against localhost:21000
> set mem_limit=1g;
> -- executing against localhost:21000
> select *
> from tpch_nested_parquet.customer c, c.c_orders o, o.o_lineitems
> order by l_partkey desc, l_suppkey desc, l_linenumber desc, c_custkey
> limit 5;
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)