You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@impala.apache.org by "Bikramjeet Vig (JIRA)" <ji...@apache.org> on 2018/02/20 18:46:00 UTC
[jira] [Resolved] (IMPALA-6526) Regression:
query_test.test_spilling.TestSpillingDebugActionDimensions.test_spilling
[ https://issues.apache.org/jira/browse/IMPALA-6526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bikramjeet Vig resolved IMPALA-6526.
------------------------------------
Resolution: Fixed
Fix Version/s: Impala 2.12.0
Impala 3.0
[https://github.com/apache/impala/commit/61d941fad36f3c07cd9ba4daad7ffbac69507a02]
IMPALA-6526: Fix spilling test for running on local FS
One of the spilling test was failing because its minimum bufferpool
mem requirement was more when ran on local FS as compared to when
it is run on HDFS.
The fix is to increase the bufferpool limit to a value just above
the min limit so that it still forces spill to disk on both filesystems.
Testing:
Ran core tests with local FS as target file system. Made sure the
failing test passed.
Change-Id: I50648d7936007a26891cf64d6343c47d9d646596
Reviewed-on: http://gerrit.cloudera.org:8080/9354
Reviewed-by: Tim Armstrong <ta...@cloudera.com>
Tested-by: Impala Public Jenkins
> Regression: query_test.test_spilling.TestSpillingDebugActionDimensions.test_spilling
> ------------------------------------------------------------------------------------
>
> Key: IMPALA-6526
> URL: https://issues.apache.org/jira/browse/IMPALA-6526
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 3.0
> Reporter: Zach Amsden
> Assignee: Bikramjeet Vig
> Priority: Blocker
> Labels: broken-build
> Fix For: Impala 3.0, Impala 2.12.0
>
>
> exec_option:
> { 'debug_action': '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@1.0', 'default_spillable_buffer_size': '256k' }
> table_format:
> parquet/none
>
> {noformat}
> Regression query_test.test_spilling.TestSpillingDebugActionDimensions.test_spilling[exec_option: {'debug_action': '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@1.0', 'default_spillable_buffer_size': '256k'} | table_format: parquet/none] (from pytest) Failing for the past 1 build (Since #276 ) Took 27 sec. add description Error Message query_test/test_spilling.py:49: in test_spilling self.run_test_case('QueryTest/spilling', vector) common/impala_test_suite.py:397: in run_test_case result = self.__execute_query(target_impalad_client, query, user=user) common/impala_test_suite.py:612: in __execute_query return impalad_client.execute(query, user=user) common/impala_connection.py:160: in execute return self.__beeswax_client.execute(sql_stmt, user=user) beeswax/impala_beeswax.py:173: in execute handle = self.__execute_query(query_string.strip(), user=user) beeswax/impala_beeswax.py:339: in __execute_query handle = self.execute_query_async(query_string, user=user) beeswax/impala_beeswax.py:335: in execute_query_async return self.__do_rpc(lambda: self.imp_service.query(query,)) beeswax/impala_beeswax.py:460: in __do_rpc raise ImpalaBeeswaxException(self.__build_error_message(b), b) E ImpalaBeeswaxException: ImpalaBeeswaxException: E INNER EXCEPTION: <class 'beeswaxd.ttypes.BeeswaxException'> E MESSAGE: Rejected query from pool default-pool: minimum memory reservation is greater than memory available to the query for buffer reservations. Increase the buffer_pool_limit to 24.31 MB. See the query profile for more information about the per-node memory requirements. Stacktrace query_test/test_spilling.py:49: in test_spilling self.run_test_case('QueryTest/spilling', vector) common/impala_test_suite.py:397: in run_test_case result = self.__execute_query(target_impalad_client, query, user=user) common/impala_test_suite.py:612: in __execute_query return impalad_client.execute(query, user=user) common/impala_connection.py:160: in execute return self.__beeswax_client.execute(sql_stmt, user=user) beeswax/impala_beeswax.py:173: in execute handle = self.__execute_query(query_string.strip(), user=user) beeswax/impala_beeswax.py:339: in __execute_query handle = self.execute_query_async(query_string, user=user) beeswax/impala_beeswax.py:335: in execute_query_async return self.__do_rpc(lambda: self.imp_service.query(query,)) beeswax/impala_beeswax.py:460: in __do_rpc raise ImpalaBeeswaxException(self.__build_error_message(b), b) E ImpalaBeeswaxException: ImpalaBeeswaxException: E INNER EXCEPTION: <class 'beeswaxd.ttypes.BeeswaxException'> E MESSAGE: Rejected query from pool default-pool: minimum memory reservation is greater than memory available to the query for buffer reservations. Increase the buffer_pool_limit to 24.31 MB. See the query profile for more information about the per-node memory requirements. Standard Error -- executing against localhost:21000 use tpch_parquet; SET debug_action=-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@1.0; SET default_spillable_buffer_size=256k; -- executing against localhost:21000 set buffer_pool_limit=15m; -- executing against localhost:21000 select count(l1.l_tax) from lineitem l1, lineitem l2, lineitem l3 where l1.l_tax < 0.01 and l2.l_tax < 0.04 and l1.l_orderkey = l2.l_orderkey and l1.l_orderkey = l3.l_orderkey and l1.l_comment = l3.l_comment and l1.l_shipdate = l3.l_shipdate; -- executing against localhost:21000 SET BUFFER_POOL_LIMIT=""; -- executing against localhost:21000 set buffer_pool_limit=2m; -- executing against localhost:21000 select max(t1.total_count), max(t1.l_shipinstruct), max(t1.l_comment) from (select l_shipinstruct, l_comment, count(*) over () total_count from lineitem) t1; -- executing against localhost:21000 SET BUFFER_POOL_LIMIT=""; -- executing against localhost:21000 set buffer_pool_limit=20m; -- executing against localhost:21000 select a.int_col, count(*) from functional.alltypessmall a, functional.alltypessmall b, functional.alltypessmall c where a.id = b.id and b.id = c.id group by a.int_col; -- executing against localhost:21000 SET BUFFER_POOL_LIMIT=""; MainThread: Query Name: TPCH-Q22 -- executing against localhost:21000 set buffer_pool_limit=23m; -- executing against localhost:21000 select s_name, count(*) as numwait from supplier, lineitem l1 join [BROADCAST] orders, nation where s_suppkey = l1.l_suppkey and o_orderkey = l1.l_orderkey and o_orderstatus = 'F' and l1.l_receiptdate > l1.l_commitdate and exists ( select * from lineitem l2 where l2.l_orderkey = l1.l_orderkey and l2.l_suppkey <> l1.l_suppkey ) and not exists ( select * from lineitem l3 where l3.l_orderkey = l1.l_orderkey and l3.l_suppkey <> l1.l_suppkey and l3.l_receiptdate > l3.l_commitdate ) and s_nationkey = n_nationkey and n_name = 'SAUDI ARABIA' group by s_name order by numwait desc, s_name limit 100; -- executing against localhost:21000 SET BUFFER_POOL_LIMIT="";
> {noformat}
> I suspect this is a regression in the core-local tests caused by https://issues.apache.org/jira/browse/IMPALA-5519
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)