You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Bikramjeet Vig (JIRA)" <ji...@apache.org> on 2019/06/10 23:02:00 UTC
[jira] [Resolved] (IMPALA-8570)
test_restart_statestore_query_resilience AssertionError: Query expected to
fail
[ https://issues.apache.org/jira/browse/IMPALA-8570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bikramjeet Vig resolved IMPALA-8570.
------------------------------------
Resolution: Fixed
Fix Version/s: Impala 3.3.0
> test_restart_statestore_query_resilience AssertionError: Query expected to fail
> -------------------------------------------------------------------------------
>
> Key: IMPALA-8570
> URL: https://issues.apache.org/jira/browse/IMPALA-8570
> Project: IMPALA
> Issue Type: Bug
> Affects Versions: Impala 3.3.0
> Reporter: Thomas Tauber-Marshall
> Assignee: Bikramjeet Vig
> Priority: Blocker
> Fix For: Impala 3.3.0
>
>
> Seen in an erasure coding build
> {noformat}
> Error Message
> AssertionError: Query expected to fail assert False
> Stacktrace
> custom_cluster/test_restart_services.py:129: in test_restart_statestore_query_resilience
> assert False, "Query expected to fail"
> E AssertionError: Query expected to fail
> E assert False
> Standard Output
> Redirecting stdout to /data/jenkins/workspace/impala-asf-master-core-erasure-coding/repos/Impala/logs/custom_cluster_tests/statestored.INFO
> Redirecting stdout to /data/jenkins/workspace/impala-asf-master-core-erasure-coding/repos/Impala/logs/custom_cluster_tests/statestored.INFO
> Standard Error
> -- 2019-05-21 20:09:58,348 INFO MainThread: Starting cluster with command: /data/jenkins/workspace/impala-asf-master-core-erasure-coding/repos/Impala/bin/start-impala-cluster.py --cluster_size=3 --num_coordinators=3 --log_dir=/data/jenkins/workspace/impala-asf-master-core-erasure-coding/repos/Impala/logs/custom_cluster_tests --log_level=1 '--impalad_args=--statestore_subscriber_timeout_seconds=2 --failed_backends_query_cancellation_grace_period_ms=5000 ' '--state_store_args=--statestore_update_frequency_ms=50 --statestore_priority_update_frequency_ms=50 --statestore_heartbeat_frequency_ms=50 ' '--catalogd_args=--statestore_subscriber_timeout_seconds=2 ' --impalad_args=--default_query_options=allow_erasure_coded_files=true
> 20:09:58 MainThread: Starting State Store logging to /data/jenkins/workspace/impala-asf-master-core-erasure-coding/repos/Impala/logs/custom_cluster_tests/statestored.INFO
> 20:09:58 MainThread: Starting Catalog Service logging to /data/jenkins/workspace/impala-asf-master-core-erasure-coding/repos/Impala/logs/custom_cluster_tests/catalogd.INFO
> 20:09:58 MainThread: Starting Impala Daemon logging to /data/jenkins/workspace/impala-asf-master-core-erasure-coding/repos/Impala/logs/custom_cluster_tests/impalad.INFO
> 20:09:58 MainThread: Starting Impala Daemon logging to /data/jenkins/workspace/impala-asf-master-core-erasure-coding/repos/Impala/logs/custom_cluster_tests/impalad_node1.INFO
> 20:09:58 MainThread: Starting Impala Daemon logging to /data/jenkins/workspace/impala-asf-master-core-erasure-coding/repos/Impala/logs/custom_cluster_tests/impalad_node2.INFO
> 20:10:01 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
> 20:10:01 MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-1173.vpc.cloudera.com:25000
> 20:10:01 MainThread: Waiting for num_known_live_backends=3. Current value: 0
> 20:10:02 MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-1173.vpc.cloudera.com:25000
> 20:10:02 MainThread: Waiting for num_known_live_backends=3. Current value: 0
> 20:10:03 MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-1173.vpc.cloudera.com:25000
> 20:10:03 MainThread: num_known_live_backends has reached value: 3
> 20:10:04 MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-1173.vpc.cloudera.com:25001
> 20:10:04 MainThread: num_known_live_backends has reached value: 3
> 20:10:05 MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-1173.vpc.cloudera.com:25002
> 20:10:05 MainThread: num_known_live_backends has reached value: 3
> 20:10:05 MainThread: Impala Cluster Running with 3 nodes (3 coordinators, 3 executors).
> -- 2019-05-21 20:10:05,585 INFO MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
> -- 2019-05-21 20:10:05,585 INFO MainThread: Getting metric: statestore.live-backends from impala-ec2-centos74-m5-4xlarge-ondemand-1173.vpc.cloudera.com:25010
> -- 2019-05-21 20:10:05,587 INFO MainThread: Starting new HTTP connection (1): impala-ec2-centos74-m5-4xlarge-ondemand-1173.vpc.cloudera.com
> -- 2019-05-21 20:10:05,588 INFO MainThread: Metric 'statestore.live-backends' has reached desired value: 4
> -- 2019-05-21 20:10:05,588 INFO MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-1173.vpc.cloudera.com:25000
> -- 2019-05-21 20:10:05,589 INFO MainThread: Starting new HTTP connection (1): impala-ec2-centos74-m5-4xlarge-ondemand-1173.vpc.cloudera.com
> -- 2019-05-21 20:10:05,590 INFO MainThread: num_known_live_backends has reached value: 3
> -- 2019-05-21 20:10:05,590 INFO MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-1173.vpc.cloudera.com:25001
> -- 2019-05-21 20:10:05,591 INFO MainThread: Starting new HTTP connection (1): impala-ec2-centos74-m5-4xlarge-ondemand-1173.vpc.cloudera.com
> -- 2019-05-21 20:10:05,592 INFO MainThread: num_known_live_backends has reached value: 3
> -- 2019-05-21 20:10:05,592 INFO MainThread: Getting num_known_live_backends from impala-ec2-centos74-m5-4xlarge-ondemand-1173.vpc.cloudera.com:25002
> -- 2019-05-21 20:10:05,592 INFO MainThread: Starting new HTTP connection (1): impala-ec2-centos74-m5-4xlarge-ondemand-1173.vpc.cloudera.com
> -- 2019-05-21 20:10:05,593 INFO MainThread: num_known_live_backends has reached value: 3
> SET client_identifier=custom_cluster/test_restart_services.py::TestRestart::()::test_restart_statestore_query_resilience;
> -- connecting to: localhost:21000
> -- connecting to localhost:21050 with impyla
> -- 2019-05-21 20:10:05,749 INFO MainThread: Closing active operation
> SET client_identifier=custom_cluster/test_restart_services.py::TestRestart::()::test_restart_statestore_query_resilience;
> -- connecting to: impala-ec2-centos74-m5-4xlarge-ondemand-1173.vpc.cloudera.com:21000
> -- executing async: impala-ec2-centos74-m5-4xlarge-ondemand-1173.vpc.cloudera.com:21000
> select distinct * from tpch_parquet.lineitem where l_orderkey > sleep(1000);
> -- 2019-05-21 20:10:07,491 INFO MainThread: Started query a745b14e6a951582:ef6b8c9600000000
> -- getting state for operation: <tests.common.impala_connection.OperationHandle object at 0x5001ed0>
> -- getting state for operation: <tests.common.impala_connection.OperationHandle object at 0x5001ed0>
> -- getting state for operation: <tests.common.impala_connection.OperationHandle object at 0x5001ed0>
> -- 2019-05-21 20:10:08,518 INFO MainThread: Killing <StateStoreProcess PID: 23688 (/data/jenkins/workspace/impala-asf-master-core-erasure-coding/repos/Impala/be/build/latest/service/statestored -logbufsecs=5 -v=1 -max_log_files=0 -log_filename=statestored -log_dir=/data/jenkins/workspace/impala-asf-master-core-erasure-coding/repos/Impala/logs/custom_cluster_tests --statestore_update_frequency_ms=50 --statestore_priority_update_frequency_ms=50 --statestore_heartbeat_frequency_ms=50)> with signal 9
> -- 2019-05-21 20:10:08,541 INFO MainThread: No PID found for process cmdline: /data/jenkins/workspace/impala-asf-master-core-erasure-coding/repos/Impala/be/build/latest/service/statestored -logbufsecs=5 -v=1 -max_log_files=0 -log_filename=statestored -log_dir=/data/jenkins/workspace/impala-asf-master-core-erasure-coding/repos/Impala/logs/custom_cluster_tests --statestore_update_frequency_ms=50 --statestore_priority_update_frequency_ms=50 --statestore_heartbeat_frequency_ms=50. Process is dead?
> -- 2019-05-21 20:10:08,541 INFO MainThread: Waiting for exit: /data/jenkins/workspace/impala-asf-master-core-erasure-coding/repos/Impala/be/build/latest/service/statestored -logbufsecs=5 -v=1 -max_log_files=0 -log_filename=statestored -log_dir=/data/jenkins/workspace/impala-asf-master-core-erasure-coding/repos/Impala/logs/custom_cluster_tests --statestore_update_frequency_ms=50 --statestore_priority_update_frequency_ms=50 --statestore_heartbeat_frequency_ms=50 (PID: None)
> -- 2019-05-21 20:10:08,552 INFO MainThread: Starting statestored with arguments
> -- 2019-05-21 20:10:08,560 INFO MainThread: Getting metric: statestore.live-backends from impala-ec2-centos74-m5-4xlarge-ondemand-1173.vpc.cloudera.com:25010
> -- 2019-05-21 20:10:08,561 INFO MainThread: Starting new HTTP connection (1): impala-ec2-centos74-m5-4xlarge-ondemand-1173.vpc.cloudera.com
> -- 2019-05-21 20:10:08,561 INFO MainThread: Debug webpage not yet available: ('Connection aborted.', error(111, 'Connection refused'))
> Turning perftools heap leak checking off
> Redirecting stderr to /data/jenkins/workspace/impala-asf-master-core-erasure-coding/repos/Impala/logs/custom_cluster_tests/statestored.ERROR
> -- 2019-05-21 20:10:09,564 INFO MainThread: Starting new HTTP connection (1): impala-ec2-centos74-m5-4xlarge-ondemand-1173.vpc.cloudera.com
> -- 2019-05-21 20:10:09,565 INFO MainThread: Waiting for metric value 'statestore.live-backends'=4. Current value: 0
> -- 2019-05-21 20:10:09,565 INFO MainThread: Sleeping 1s before next retry.
> -- 2019-05-21 20:10:10,566 INFO MainThread: Getting metric: statestore.live-backends from impala-ec2-centos74-m5-4xlarge-ondemand-1173.vpc.cloudera.com:25010
> -- 2019-05-21 20:10:10,567 INFO MainThread: Starting new HTTP connection (1): impala-ec2-centos74-m5-4xlarge-ondemand-1173.vpc.cloudera.com
> -- 2019-05-21 20:10:10,568 INFO MainThread: Waiting for metric value 'statestore.live-backends'=4. Current value: 0
> -- 2019-05-21 20:10:10,568 INFO MainThread: Sleeping 1s before next retry.
> -- 2019-05-21 20:10:11,570 INFO MainThread: Getting metric: statestore.live-backends from impala-ec2-centos74-m5-4xlarge-ondemand-1173.vpc.cloudera.com:25010
> -- 2019-05-21 20:10:11,571 INFO MainThread: Starting new HTTP connection (1): impala-ec2-centos74-m5-4xlarge-ondemand-1173.vpc.cloudera.com
> -- 2019-05-21 20:10:11,572 INFO MainThread: Waiting for metric value 'statestore.live-backends'=4. Current value: 1
> -- 2019-05-21 20:10:11,572 INFO MainThread: Sleeping 1s before next retry.
> -- 2019-05-21 20:10:12,573 INFO MainThread: Getting metric: statestore.live-backends from impala-ec2-centos74-m5-4xlarge-ondemand-1173.vpc.cloudera.com:25010
> -- 2019-05-21 20:10:12,574 INFO MainThread: Starting new HTTP connection (1): impala-ec2-centos74-m5-4xlarge-ondemand-1173.vpc.cloudera.com
> -- 2019-05-21 20:10:12,575 INFO MainThread: Metric 'statestore.live-backends' has reached desired value: 4
> -- getting state for operation: <tests.common.impala_connection.OperationHandle object at 0x5001ed0>
> -- 2019-05-21 20:10:18,607 INFO MainThread: Killing <StateStoreProcess PID: 24937 (/data/jenkins/workspace/impala-asf-master-core-erasure-coding/repos/Impala/be/build/latest/service/statestored -logbufsecs=5 -v=1 -max_log_files=0 -log_filename=statestored -log_dir=/data/jenkins/workspace/impala-asf-master-core-erasure-coding/repos/Impala/logs/custom_cluster_tests --statestore_update_frequency_ms=50 --statestore_priority_update_frequency_ms=50 --statestore_heartbeat_frequency_ms=50)> with signal 9
> -- 2019-05-21 20:10:18,641 INFO MainThread: Killing <ImpaladProcess PID: 23778 (/data/jenkins/workspace/impala-asf-master-core-erasure-coding/repos/Impala/be/build/latest/service/impalad -kudu_client_rpc_timeout_ms 0 -kudu_master_hosts localhost -mem_limit=12884901888 -logbufsecs=5 -v=1 -max_log_files=0 -log_filename=impalad_node1 -log_dir=/data/jenkins/workspace/impala-asf-master-core-erasure-coding/repos/Impala/logs/custom_cluster_tests -beeswax_port=21001 -hs2_port=21051 -be_port=22001 -krpc_port=27001 -state_store_subscriber_port=23001 -webserver_port=25001 --statestore_subscriber_timeout_seconds=2 --failed_backends_query_cancellation_grace_period_ms=5000 --default_query_options=allow_erasure_coded_files=true)> with signal 9
> -- 2019-05-21 20:10:18,651 INFO MainThread: Starting statestored with arguments
> -- waiting for query to reach FINISHED state: <tests.common.impala_connection.OperationHandle object at 0x5001ed0>
> Turning perftools heap leak checking off
> Redirecting stderr to /data/jenkins/workspace/impala-asf-master-core-erasure-coding/repos/Impala/logs/custom_cluster_tests/statestored.ERROR
> -- closing connection to: impala-ec2-centos74-m5-4xlarge-ondemand-1173.vpc.cloudera.com:21000
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org