You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Wenzhe Zhou (Jira)" <ji...@apache.org> on 2021/02/21 06:01:00 UTC
[jira] [Commented] (IMPALA-10534)
TestScratchDir.test_scratch_dirs_remote_spill_with_options seems flaky
[ https://issues.apache.org/jira/browse/IMPALA-10534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17287852#comment-17287852 ]
Wenzhe Zhou commented on IMPALA-10534:
--------------------------------------
Standard Error
-- 2021-02-20 20:02:57,279 INFO MainThread: Starting cluster with command: /data/jenkins/workspace/impala-cdpd-master-staging-exhaustive-release/repos/Impala/bin/start-impala-cluster.py '--state_store_args=--statestore_update_frequency_ms=50 --statestore_priority_update_frequency_ms=50 --statestore_heartbeat_frequency_ms=50' --cluster_size=1 --num_coordinators=3 --log_dir=/data/jenkins/workspace/impala-cdpd-master-staging-exhaustive-release/repos/Impala/logs/custom_cluster_tests --log_level=1 --impalad_args=--enable_orc_scanner=true '--impalad_args=-logbuflevel=-1 -scratch_dirs=/tmp/tmpB_5A33:16MB:0,hdfs://localhost' --impalad_args=--allow_multiple_scratch_dirs_per_device=true --impalad_args=--allow_spill_to_hdfs=true --impalad_args=--remote_tmp_file_size=8M --impalad_args=--remote_tmp_file_block_size=1m --impalad_args=--default_query_options=
20:02:57 MainThread: Found 0 impalad/0 statestored/0 catalogd process(es)
20:02:57 MainThread: Starting State Store logging to /data/jenkins/workspace/impala-cdpd-master-staging-exhaustive-release/repos/Impala/logs/custom_cluster_tests/statestored.INFO
20:02:57 MainThread: Starting Catalog Service logging to /data/jenkins/workspace/impala-cdpd-master-staging-exhaustive-release/repos/Impala/logs/custom_cluster_tests/catalogd.INFO
20:02:57 MainThread: Starting Impala Daemon logging to /data/jenkins/workspace/impala-cdpd-master-staging-exhaustive-release/repos/Impala/logs/custom_cluster_tests/impalad.INFO
20:03:00 MainThread: Found 1 impalad/1 statestored/1 catalogd process(es)
20:03:00 MainThread: Found 1 impalad/1 statestored/1 catalogd process(es)
20:03:00 MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-00a7.vpc.cloudera.com:25000
20:03:01 MainThread: 'backends'
20:03:01 MainThread: Waiting for num_known_live_backends=1. Current value: None
20:03:02 MainThread: Found 1 impalad/1 statestored/1 catalogd process(es)
20:03:02 MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-00a7.vpc.cloudera.com:25000
20:03:02 MainThread: Waiting for num_known_live_backends=1. Current value: 0
20:03:03 MainThread: Found 1 impalad/1 statestored/1 catalogd process(es)
20:03:03 MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-00a7.vpc.cloudera.com:25000
20:03:03 MainThread: num_known_live_backends has reached value: 1
20:03:03 MainThread: Impala Cluster Running with 1 nodes (1 coordinators, 1 executors).
-- 2021-02-20 20:03:03,642 DEBUG MainThread: Found 1 impalad/1 statestored/1 catalogd process(es)
-- 2021-02-20 20:03:03,642 INFO MainThread: Getting metric: statestore.live-backends from impala-ec2-centos74-m5-4xlarge-ondemand-00a7.vpc.cloudera.com:25010
-- 2021-02-20 20:03:03,643 INFO MainThread: Starting new HTTP connection (1): impala-ec2-centos74-m5-4xlarge-ondemand-00a7.vpc.cloudera.com
-- 2021-02-20 20:03:03,645 INFO MainThread: Metric 'statestore.live-backends' has reached desired value: 2
-- 2021-02-20 20:03:03,645 DEBUG MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-00a7.vpc.cloudera.com:25000
-- 2021-02-20 20:03:03,645 INFO MainThread: Starting new HTTP connection (1): impala-ec2-centos74-m5-4xlarge-ondemand-00a7.vpc.cloudera.com
-- 2021-02-20 20:03:03,646 INFO MainThread: num_known_live_backends has reached value: 1
SET client_identifier=custom_cluster/test_scratch_disk.py::TestScratchDir::()::test_scratch_dirs_remote_spill_with_options[protocol:beeswax|exec_option:{'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':0;'disable_codegen':False;'abort_on_error':1;'exec_single_node;
-- connecting to: impala-ec2-centos74-m5-4xlarge-ondemand-00a7.vpc.cloudera.com:21000
SET client_identifier=custom_cluster/test_scratch_disk.py::TestScratchDir::()::test_scratch_dirs_remote_spill_with_options[protocol:beeswax|exec_option:{'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':0;'disable_codegen':False;'abort_on_error':1;'exec_single_node;
-- executing against impala-ec2-centos74-m5-4xlarge-ondemand-00a7.vpc.cloudera.com:21000
use functional;
-- 2021-02-20 20:03:03,793 INFO MainThread: Started query 724b34552dd4c36d:e633011400000000
SET client_identifier=custom_cluster/test_scratch_disk.py::TestScratchDir::()::test_scratch_dirs_remote_spill_with_options[protocol:beeswax|exec_option:{'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':0;'disable_codegen':False;'abort_on_error':1;'exec_single_node;
SET batch_size=0;
SET num_nodes=0;
SET disable_codegen_rows_threshold=0;
SET disable_codegen=False;
SET abort_on_error=1;
SET exec_single_node_rows_threshold=0;
SET buffer_pool_limit=45m;
-- executing async: impala-ec2-centos74-m5-4xlarge-ondemand-00a7.vpc.cloudera.com:21000
select o_orderdate, o_custkey, o_comment
from tpch.orders
order by o_orderdate
;
-- 2021-02-20 20:03:04,514 INFO MainThread: Started query 8d43a5b279269987:477d5cdb00000000
-- 2021-02-20 20:03:04,514 INFO MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-ec2-centos74-m5-4xlarge-ondemand-00a7.vpc.cloudera.com:25000
-- 2021-02-20 20:03:04,516 INFO MainThread: Starting new HTTP connection (1): impala-ec2-centos74-m5-4xlarge-ondemand-00a7.vpc.cloudera.com
-- 2021-02-20 20:03:04,525 INFO MainThread: Waiting for metric value 'impala-server.num-fragments-in-flight'=2. Current value: 0
-- 2021-02-20 20:03:04,525 INFO MainThread: Sleeping 1s before next retry.
-- 2021-02-20 20:03:05,526 INFO MainThread: Getting metric: impala-server.num-fragments-in-flight from impala-ec2-centos74-m5-4xlarge-ondemand-00a7.vpc.cloudera.com:25000
-- 2021-02-20 20:03:05,527 INFO MainThread: Starting new HTTP connection (1): impala-ec2-centos74-m5-4xlarge-ondemand-00a7.vpc.cloudera.com
-- 2021-02-20 20:03:05,530 INFO MainThread: Metric 'impala-server.num-fragments-in-flight' has reached desired value: 2
-- 2021-02-20 20:03:05,530 INFO MainThread: Starting new HTTP connection (1): localhost
-- fetching results from: <tests.common.impala_connection.OperationHandle object at 0x7fa721b87e90>
> TestScratchDir.test_scratch_dirs_remote_spill_with_options seems flaky
> ----------------------------------------------------------------------
>
> Key: IMPALA-10534
> URL: https://issues.apache.org/jira/browse/IMPALA-10534
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 4.0
> Reporter: Wenzhe Zhou
> Assignee: Yida Wu
> Priority: Major
> Labels: broken-build
>
> TestScratchDir.test_scratch_dirs_remote_spill_with_options failed in exhaustive release build.
> Error Message
> ImpalaBeeswaxException: ImpalaBeeswaxException: INNER EXCEPTION: <class 'beeswaxd.ttypes.BeeswaxException'> MESSAGE: Cancelled in TmpFileBufferPool because: Timeout waiting for a local buffer in 60 seconds Cancelled in TmpFileBufferPool because: Timeout waiting for a local buffer in 60 seconds
> Stacktrace
> custom_cluster/test_scratch_disk.py:384: in test_scratch_dirs_remote_spill_with_options
> results = client.fetch(self.spill_query, handle)
> common/impala_connection.py:254: in fetch
> sql_stmt, operation_handle.get_handle(), max_rows)
> beeswax/impala_beeswax.py:452: in fetch_results
> exec_result = self.__fetch_results(query_handle, max_rows)
> beeswax/impala_beeswax.py:463: in __fetch_results
> results = self.__do_rpc(lambda: self.imp_service.fetch(handle, False, fetch_rows))
> beeswax/impala_beeswax.py:520: in __do_rpc
> raise ImpalaBeeswaxException(self.__build_error_message(b), b)
> E ImpalaBeeswaxException: ImpalaBeeswaxException:
> E INNER EXCEPTION: <class 'beeswaxd.ttypes.BeeswaxException'>
> E MESSAGE: Cancelled in TmpFileBufferPool because: Timeout waiting for a local buffer in 60 seconds
> E
> E Cancelled in TmpFileBufferPool because: Timeout waiting for a local buffer in 60 seconds
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org