You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Quanlong Huang (Jira)" <ji...@apache.org> on 2021/08/30 01:55:00 UTC

[jira] [Created] (IMPALA-10895) TestQueryRetries.test_retrying_query_cancel is flaky

Quanlong Huang created IMPALA-10895:
---------------------------------------

             Summary: TestQueryRetries.test_retrying_query_cancel is flaky
                 Key: IMPALA-10895
                 URL: https://issues.apache.org/jira/browse/IMPALA-10895
             Project: IMPALA
          Issue Type: Bug
            Reporter: Quanlong Huang
            Assignee: Quanlong Huang


Saw this failed in an ASAN build:
{code}
custom_cluster.test_query_retries.TestQueryRetries.test_retrying_query_cancel
{code}
Stacktrace
{code}
custom_cluster/test_query_retries.py:742: in test_retrying_query_cancel
    assert retry_status.group(1) == 'RETRYING'
E   assert 'RETRIED' == 'RETRYING'
E     - RETRIED
E     + RETRYING
{code}
Standard Error
{code}
-- 2021-08-29 08:29:29,112 INFO     MainThread: Starting cluster with command: /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/bin/start-impala-cluster.py '--state_store_args=--statestore_update_frequency_ms=50     --statestore_priority_update_frequency_ms=50     --statestore_heartbeat_frequency_ms=50' --cluster_size=3 --num_coordinators=3 --log_dir=/data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/logs/custom_cluster_tests --log_level=1 '--impalad_args=--debug_actions=RETRY_DELAY_CHECKING_ORIGINAL_DRIVER:SLEEP@1000 ' '--state_store_args=--statestore_heartbeat_frequency_ms=60000 ' --impalad_args=--default_query_options=
08:29:29 MainThread: Found 0 impalad/0 statestored/0 catalogd process(es)
08:29:29 MainThread: Starting State Store logging to /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/logs/custom_cluster_tests/statestored.INFO
08:29:29 MainThread: Starting Catalog Service logging to /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/logs/custom_cluster_tests/catalogd.INFO
08:29:29 MainThread: Starting Impala Daemon logging to /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/logs/custom_cluster_tests/impalad.INFO
08:29:29 MainThread: Starting Impala Daemon logging to /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/logs/custom_cluster_tests/impalad_node1.INFO
08:29:29 MainThread: Starting Impala Daemon logging to /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/logs/custom_cluster_tests/impalad_node2.INFO
08:29:32 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
08:29:32 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
08:29:32 MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-061e.vpc.cloudera.com:25000
08:29:32 MainThread: Debug webpage not yet available: HTTPConnectionPool(host='impala-ec2-centos74-m5-4xlarge-ondemand-061e.vpc.cloudera.com', port=25000): Max retries exceeded with url: /backends?json (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f10c1d5d150>: Failed to establish a new connection: [Errno 111] Connection refused',))
08:29:34 MainThread: Debug webpage did not become available in expected time.
08:29:34 MainThread: Waiting for num_known_live_backends=3. Current value: None
08:29:35 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
08:29:35 MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-061e.vpc.cloudera.com:25000
08:29:35 MainThread: Waiting for num_known_live_backends=3. Current value: 0
08:29:36 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
08:29:36 MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-061e.vpc.cloudera.com:25000
08:29:36 MainThread: num_known_live_backends has reached value: 3
08:29:37 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
08:29:37 MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-061e.vpc.cloudera.com:25001
08:29:37 MainThread: num_known_live_backends has reached value: 3
08:29:38 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
08:29:38 MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-061e.vpc.cloudera.com:25002
08:29:38 MainThread: num_known_live_backends has reached value: 3
08:29:38 MainThread: Impala Cluster Running with 3 nodes (3 coordinators, 3 executors).
-- 2021-08-29 08:29:38,626 DEBUG    MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
-- 2021-08-29 08:29:38,626 INFO     MainThread: Getting metric: statestore.live-backends from impala-ec2-centos74-m5-4xlarge-ondemand-061e.vpc.cloudera.com:25010
-- 2021-08-29 08:29:38,629 INFO     MainThread: Metric 'statestore.live-backends' has reached desired value: 4
-- 2021-08-29 08:29:38,629 DEBUG    MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-061e.vpc.cloudera.com:25000
-- 2021-08-29 08:29:38,631 INFO     MainThread: num_known_live_backends has reached value: 3
-- 2021-08-29 08:29:38,631 DEBUG    MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-061e.vpc.cloudera.com:25001
-- 2021-08-29 08:29:38,633 INFO     MainThread: num_known_live_backends has reached value: 3
-- 2021-08-29 08:29:38,633 DEBUG    MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-061e.vpc.cloudera.com:25002
-- 2021-08-29 08:29:38,635 INFO     MainThread: num_known_live_backends has reached value: 3
SET client_identifier=custom_cluster/test_query_retries.py::TestQueryRetries::()::test_retrying_query_cancel;
-- connecting to: localhost:21000
-- connecting to localhost:21050 with impyla
-- 2021-08-29 08:29:38,801 INFO     MainThread: Closing active operation
-- connecting to localhost:28000 with impyla
-- 2021-08-29 08:29:38,822 INFO     MainThread: Closing active operation
-- 2021-08-29 08:29:38,853 INFO     MainThread: Killing <ImpaladProcess PID: 6501 (/data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/latest/service/impalad -disconnected_session_timeout 21600 -kudu_client_rpc_timeout_ms 60000 -kudu_master_hosts localhost -mem_limit=12884901888 -logbufsecs=5 -v=1 -max_log_files=0 -log_filename=impalad_node1 -log_dir=/data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/logs/custom_cluster_tests -beeswax_port=21001 -hs2_port=21051 -hs2_http_port=28001 -krpc_port=27001 -state_store_subscriber_port=23001 -webserver_port=25001 --debug_actions=RETRY_DELAY_CHECKING_ORIGINAL_DRIVER:SLEEP@1000 --default_query_options=)> with signal 9
SET client_identifier=custom_cluster/test_query_retries.py::TestQueryRetries::()::test_retrying_query_cancel;
SET retry_failed_queries=true;
-- executing async: localhost:21000

select count(*) from tpch_parquet.lineitem;

-- 2021-08-29 08:29:39,647 INFO     MainThread: Started query de46b6f56f935fa5:f4c701b200000000
-- canceling operation: <tests.common.impala_connection.OperationHandle object at 0x7fb2a8026250>
{code}
CC [~xqhe]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org