You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Yida Wu (Jira)" <ji...@apache.org> on 2021/03/03 15:24:00 UTC
[jira] [Created] (IMPALA-10559) TestScratchLimit seems flaky
Yida Wu created IMPALA-10559:
--------------------------------
Summary: TestScratchLimit seems flaky
Key: IMPALA-10559
URL: https://issues.apache.org/jira/browse/IMPALA-10559
Project: IMPALA
Issue Type: Bug
Components: Backend
Affects Versions: Impala 4.0
Reporter: Yida Wu
The TestScratchLimit testcase runs fine alone. But if runs TestScratchDir first, then runs TestScratchLimit, it will always fail as follow shows unless restarts the services (e.g. using "./buildall.sh -noclean -notests -format -start_minicluster -start_impala_cluster").
{code:java}
query_test/test_scratch_limit.py::TestScratchLimit::test_with_low_scratch_limit[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: text/none]
query_test/test_scratch_limit.py::TestScratchLimit::test_without_specifying_scratch_limit[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: text/none]
query_test/test_scratch_limit.py::TestScratchLimit::test_with_high_scratch_limit[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: text/none]
query_test/test_scratch_limit.py::TestScratchLimit::test_with_zero_scratch_limit[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: text/none]
query_test/test_scratch_limit.py::TestScratchLimit::test_with_unlimited_scratch_limit[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: text/none]
query_test/test_scratch_limit.py::TestScratchLimit::test_with_zero_scratch_limit_no_memory_limit[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: text/none]
[gw1] PASSED query_test/test_scratch_limit.py::TestScratchLimit::test_with_zero_scratch_limit[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: text/none]
[gw5] FAILED query_test/test_scratch_limit.py::TestScratchLimit::test_with_low_scratch_limit[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: text/none]
[gw4] FAILED query_test/test_scratch_limit.py::TestScratchLimit::test_with_high_scratch_limit[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: text/none]
[gw2] FAILED query_test/test_scratch_limit.py::TestScratchLimit::test_without_specifying_scratch_limit[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: text/none]
[gw3] FAILED query_test/test_scratch_limit.py::TestScratchLimit::test_with_unlimited_scratch_limit[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: text/none]
[gw0] PASSED query_test/test_scratch_limit.py::TestScratchLimit::test_with_zero_scratch_limit_no_memory_limit[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: text/none]
------------------------------------------- generated xml file: /home/yida/Impala/logs/ee_tests/results/TEST-impala-parallel.xml --------------------------------------------
========================================================================== short test summary info ==========================================================================
FAIL query_test/test_scratch_limit.py::TestScratchLimit::()::test_with_low_scratch_limit[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: text/none]
FAIL query_test/test_scratch_limit.py::TestScratchLimit::()::test_with_high_scratch_limit[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: text/none]
FAIL query_test/test_scratch_limit.py::TestScratchLimit::()::test_without_specifying_scratch_limit[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: text/none]
FAIL query_test/test_scratch_limit.py::TestScratchLimit::()::test_with_unlimited_scratch_limit[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: text/none]
================================================================================= FAILURES ==================================================================================
TestScratchLimit.test_with_low_scratch_limit[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: text/none]
[gw5] linux2 -- Python 2.7.16 /home/yida/Impala/bin/../infra/python/env-gcc7.5.0/bin/python
query_test/test_scratch_limit.py:90: in test_with_low_scratch_limit
assert expected_error % scratch_limit_in_bytes in str(e)
E assert ('Scratch space limit of %s bytes exceeded' % 25165824) in 'ImpalaBeeswaxException:\n Query aborted:Could not create files in any configured scratch directories (--scratch_dirs=...ous errors that may have prevented creating or writing scratch files. The following directories were at capacity: \n\n'
E + where 'ImpalaBeeswaxException:\n Query aborted:Could not create files in any configured scratch directories (--scratch_dirs=...ous errors that may have prevented creating or writing scratch files. The following directories were at capacity: \n\n' = str(ImpalaBeeswaxException())
--------------------------------------------------------------------------- Captured stderr setup ---------------------------------------------------------------------------
SET client_identifier=query_test/test_scratch_limit.py::TestScratchLimit::()::test_with_low_scratch_limit[protocol:beeswax|exec_option:{'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':5000;'disable_codegen':False;'abort_on_error':1;'exec_single_node_rows_threshol;
-- connecting to: localhost:21000
-- connecting to localhost:21050 with impyla
-- 2021-03-03 07:12:08,115 INFO MainThread: Closing active operation
-- connecting to localhost:28000 with impyla
-- 2021-03-03 07:12:08,140 INFO MainThread: Closing active operation
--------------------------------------------------------------------------- Captured stderr call ----------------------------------------------------------------------------
SET client_identifier=query_test/test_scratch_limit.py::TestScratchLimit::()::test_with_low_scratch_limit[protocol:beeswax|exec_option:{'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':5000;'disable_codegen':False;'abort_on_error':1;'exec_single_node_rows_threshol;
SET scratch_limit=24m;
SET batch_size=0;
SET num_nodes=0;
SET disable_codegen_rows_threshold=5000;
SET disable_codegen=False;
SET abort_on_error=1;
SET exec_single_node_rows_threshold=0;
SET buffer_pool_limit=32m;
-- executing against localhost:21000
select o_orderdate, o_custkey, o_comment
from tpch.orders
order by o_orderdate
;
-- 2021-03-03 07:12:08,329 INFO MainThread: Started query f44b4a50bec6b10d:e45a053d00000000
TestScratchLimit.test_with_high_scratch_limit[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: text/none]
[gw4] linux2 -- Python 2.7.16 /home/yida/Impala/bin/../infra/python/env-gcc7.5.0/bin/python
query_test/test_scratch_limit.py:74: in test_with_high_scratch_limit
self.execute_query_expect_success(self.client, self.spilling_sort_query, exec_option)
common/impala_test_suite.py:809: in wrapper
return function(*args, **kwargs)
common/impala_test_suite.py:817: in execute_query_expect_success
result = cls.__execute_query(impalad_client, query, query_options, user)
common/impala_test_suite.py:918: in __execute_query
return impalad_client.execute(query, user=user)
common/impala_connection.py:205: in execute
return self.__beeswax_client.execute(sql_stmt, user=user)
beeswax/impala_beeswax.py:187: in execute
handle = self.__execute_query(query_string.strip(), user=user)
beeswax/impala_beeswax.py:365: in __execute_query
self.wait_for_finished(handle)
beeswax/impala_beeswax.py:386: in wait_for_finished
raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
E ImpalaBeeswaxException: ImpalaBeeswaxException:
E Query aborted:Could not create files in any configured scratch directories (--scratch_dirs=) on backend 'yida-OptiPlex-7060:27001'. 0 of scratch is currently in use by this Impala Daemon (0 by this query). See logs for previous errors that may have prevented creating or writing scratch files. The following directories were at capacity:
--------------------------------------------------------------------------- Captured stderr setup ---------------------------------------------------------------------------
SET client_identifier=query_test/test_scratch_limit.py::TestScratchLimit::()::test_with_high_scratch_limit[protocol:beeswax|exec_option:{'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':5000;'disable_codegen':False;'abort_on_error':1;'exec_single_node_rows_thresho;
-- connecting to: localhost:21000
-- connecting to localhost:21050 with impyla
-- 2021-03-03 07:12:08,114 INFO MainThread: Closing active operation
-- connecting to localhost:28000 with impyla
-- 2021-03-03 07:12:08,136 INFO MainThread: Closing active operation
--------------------------------------------------------------------------- Captured stderr call ----------------------------------------------------------------------------
SET client_identifier=query_test/test_scratch_limit.py::TestScratchLimit::()::test_with_high_scratch_limit[protocol:beeswax|exec_option:{'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':5000;'disable_codegen':False;'abort_on_error':1;'exec_single_node_rows_thresho;
SET scratch_limit=500m;
SET batch_size=0;
SET num_nodes=0;
SET disable_codegen_rows_threshold=5000;
SET disable_codegen=False;
SET abort_on_error=1;
SET exec_single_node_rows_threshold=0;
SET buffer_pool_limit=32m;
-- executing against localhost:21000
select o_orderdate, o_custkey, o_comment
from tpch.orders
order by o_orderdate
;
-- 2021-03-03 07:12:08,329 INFO MainThread: Started query 814fccc9c1242331:83738b0800000000
TestScratchLimit.test_without_specifying_scratch_limit[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: text/none]
[gw2] linux2 -- Python 2.7.16 /home/yida/Impala/bin/../infra/python/env-gcc7.5.0/bin/python
query_test/test_scratch_limit.py:118: in test_without_specifying_scratch_limit
self.execute_query_expect_success(self.client, self.spilling_sort_query, exec_option)
common/impala_test_suite.py:809: in wrapper
return function(*args, **kwargs)
common/impala_test_suite.py:817: in execute_query_expect_success
result = cls.__execute_query(impalad_client, query, query_options, user)
common/impala_test_suite.py:918: in __execute_query
return impalad_client.execute(query, user=user)
common/impala_connection.py:205: in execute
return self.__beeswax_client.execute(sql_stmt, user=user)
beeswax/impala_beeswax.py:187: in execute
handle = self.__execute_query(query_string.strip(), user=user)
beeswax/impala_beeswax.py:365: in __execute_query
self.wait_for_finished(handle)
beeswax/impala_beeswax.py:386: in wait_for_finished
raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
E ImpalaBeeswaxException: ImpalaBeeswaxException:
E Query aborted:Could not create files in any configured scratch directories (--scratch_dirs=) on backend 'yida-OptiPlex-7060:27002'. 0 of scratch is currently in use by this Impala Daemon (0 by this query). See logs for previous errors that may have prevented creating or writing scratch files. The following directories were at capacity:
--------------------------------------------------------------------------- Captured stderr setup ---------------------------------------------------------------------------
SET client_identifier=query_test/test_scratch_limit.py::TestScratchLimit::()::test_without_specifying_scratch_limit[protocol:beeswax|exec_option:{'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':5000;'disable_codegen':False;'abort_on_error':1;'exec_single_node_row;
-- connecting to: localhost:21000
-- connecting to localhost:21050 with impyla
-- 2021-03-03 07:12:08,114 INFO MainThread: Closing active operation
-- connecting to localhost:28000 with impyla
-- 2021-03-03 07:12:08,137 INFO MainThread: Closing active operation
--------------------------------------------------------------------------- Captured stderr call ----------------------------------------------------------------------------
SET client_identifier=query_test/test_scratch_limit.py::TestScratchLimit::()::test_without_specifying_scratch_limit[protocol:beeswax|exec_option:{'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':5000;'disable_codegen':False;'abort_on_error':1;'exec_single_node_row;
SET batch_size=0;
SET num_nodes=0;
SET disable_codegen_rows_threshold=5000;
SET disable_codegen=False;
SET abort_on_error=1;
SET exec_single_node_rows_threshold=0;
SET buffer_pool_limit=32m;
-- executing against localhost:21000
select o_orderdate, o_custkey, o_comment
from tpch.orders
order by o_orderdate
;
-- 2021-03-03 07:12:08,329 INFO MainThread: Started query 944af93b81ee62cf:ab33f08800000000
TestScratchLimit.test_with_unlimited_scratch_limit[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: text/none]
[gw3] linux2 -- Python 2.7.16 /home/yida/Impala/bin/../infra/python/env-gcc7.5.0/bin/python
query_test/test_scratch_limit.py:110: in test_with_unlimited_scratch_limit
self.execute_query_expect_success(self.client, self.spilling_sort_query, exec_option)
common/impala_test_suite.py:809: in wrapper
return function(*args, **kwargs)
common/impala_test_suite.py:817: in execute_query_expect_success
result = cls.__execute_query(impalad_client, query, query_options, user)
common/impala_test_suite.py:918: in __execute_query
return impalad_client.execute(query, user=user)
common/impala_connection.py:205: in execute
return self.__beeswax_client.execute(sql_stmt, user=user)
beeswax/impala_beeswax.py:187: in execute
handle = self.__execute_query(query_string.strip(), user=user)
beeswax/impala_beeswax.py:365: in __execute_query
self.wait_for_finished(handle)
beeswax/impala_beeswax.py:386: in wait_for_finished
raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
E ImpalaBeeswaxException: ImpalaBeeswaxException:
E Query aborted:Could not create files in any configured scratch directories (--scratch_dirs=) on backend 'yida-OptiPlex-7060:27001'. 0 of scratch is currently in use by this Impala Daemon (0 by this query). See logs for previous errors that may have prevented creating or writing scratch files. The following directories were at capacity:
--------------------------------------------------------------------------- Captured stderr setup ---------------------------------------------------------------------------
SET client_identifier=query_test/test_scratch_limit.py::TestScratchLimit::()::test_with_unlimited_scratch_limit[protocol:beeswax|exec_option:{'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':5000;'disable_codegen':False;'abort_on_error':1;'exec_single_node_rows_th;
-- connecting to: localhost:21000
-- connecting to localhost:21050 with impyla
-- 2021-03-03 07:12:08,114 INFO MainThread: Closing active operation
-- connecting to localhost:28000 with impyla
-- 2021-03-03 07:12:08,146 INFO MainThread: Closing active operation
--------------------------------------------------------------------------- Captured stderr call ----------------------------------------------------------------------------
SET client_identifier=query_test/test_scratch_limit.py::TestScratchLimit::()::test_with_unlimited_scratch_limit[protocol:beeswax|exec_option:{'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':5000;'disable_codegen':False;'abort_on_error':1;'exec_single_node_rows_th;
SET scratch_limit=-1;
SET batch_size=0;
SET num_nodes=0;
SET disable_codegen_rows_threshold=5000;
SET disable_codegen=False;
SET abort_on_error=1;
SET exec_single_node_rows_threshold=0;
SET buffer_pool_limit=32m;
-- executing against localhost:21000
select o_orderdate, o_custkey, o_comment
from tpch.orders
order by o_orderdate
;
-- 2021-03-03 07:12:08,329 INFO MainThread: Started query 5c492cfd62287b03:f9c3370b00000000
==================================================================== 4 failed, 2 passed in 14.11 seconds ====================================================================
Running TestExecutor with args: ['--ignore', 'query_test', '--ignore', 'catalog_service', '--ignore', 'hs2', '--ignore', 'experiments', '--ignore', 'data_errors', '--ignore', 'performance', '--ignore', 'beeswax', '--ignore', 'authorization', '--ignore', 'metadata', '--ignore', 'shell', '--ignore', 'benchmark', '--ignore', 'infra', '--ignore', 'statestore', '--ignore', 'failure', '--ignore', 'webserver', '--ignore', 'comparison', '--ignore', 'stress', '--ignore', 'util', '--ignore', 'custom_cluster', '--ignore', 'common', '--ignore', 'observability', '--ignore', 'unittests', '--resultlog', '/home/yida/Impala/logs/ee_tests/results/TEST-impala-verify-metrics.log', '--junitxml', '/home/yida/Impala/logs/ee_tests/results/TEST-impala-verify-metrics.xml', 'verifiers/test_verify_metrics.py']
rootLoggerLevel = INFO
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org