You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Thomas Tauber-Marshall (JIRA)" <ji...@apache.org> on 2018/06/01 17:20:00 UTC
[jira] [Commented] (IMPALA-6642) start-impala-cluster.py failing in
some customer cluster tests
[ https://issues.apache.org/jira/browse/IMPALA-6642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16498280#comment-16498280 ]
Thomas Tauber-Marshall commented on IMPALA-6642:
------------------------------------------------
[~tarasbob] picked you at random. Got some time to look into this? If not, let me know. Seems its been happening in builds more frequently this week, so it would be good to get it fixed asap. Thanks
> start-impala-cluster.py failing in some customer cluster tests
> --------------------------------------------------------------
>
> Key: IMPALA-6642
> URL: https://issues.apache.org/jira/browse/IMPALA-6642
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
> Affects Versions: Impala 2.12.0
> Reporter: Thomas Tauber-Marshall
> Priority: Blocker
> Labels: broken-build
>
> Seen in recent builds, both on the 2.x branch. Not quite the same test/error, but they seem similar enough to probably be related:
> {noformat}
> 17:01:18 _________ ERROR at setup of TestAdmissionController.test_require_user __________
> 17:01:18 common/custom_cluster_test_suite.py:109: in setup_method
> 17:01:18 self._start_impala_cluster(cluster_args)
> 17:01:18 common/custom_cluster_test_suite.py:144: in _start_impala_cluster
> 17:01:18 check_call(cmd + options, close_fds=True)
> 17:01:18 /usr/lib64/python2.6/subprocess.py:505: in check_call
> 17:01:18 raise CalledProcessError(retcode, cmd)
> 17:01:18 E CalledProcessError: Command '['/data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/bin/start-impala-cluster.py', '--cluster_size=3', '--num_coordinators=3', '--log_dir=/data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/logs/custom_cluster_tests', '--log_level=1', '--impalad_args="-vmodule admission-controller=3 -fair_scheduler_allocation_path /data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/fe/src/test/resources/fair-scheduler-test2.xml -llama_site_path /data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/fe/src/test/resources/llama-site-test2.xml -disable_admission_control=false -require_username" ', '--state_store_args="-statestore_heartbeat_frequency_ms=100 -statestore_priority_update_frequency_ms=100" ']' returned non-zero exit status 1
> 17:01:18 ---------------------------- Captured stdout setup -----------------------------
> 17:01:18 Starting State Store logging to /data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/logs/custom_cluster_tests/statestored.INFO
> 17:01:18 Starting Catalog Service logging to /data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/logs/custom_cluster_tests/catalogd.INFO
> 17:01:18 Starting Impala Daemon logging to /data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/logs/custom_cluster_tests/impalad.INFO
> 17:01:18 Starting Impala Daemon logging to /data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/logs/custom_cluster_tests/impalad_node1.INFO
> 17:01:18 Starting Impala Daemon logging to /data/jenkins/workspace/impala-asf-2.x-core-asan/repos/Impala/logs/custom_cluster_tests/impalad_node2.INFO
> 17:01:18 Error starting cluster: Expected 3 impalad(s), only 2 found
> 17:01:18
> 17:01:18 ---------------------------- Captured stderr setup -----------------------------
> 17:01:18 MainThread: Found 2 impalad/1 statestored/1 catalogd process(es)
> 17:01:18 MainThread: Found 2 impalad/1 statestored/1 catalogd process(es)
> {noformat}
> {noformat}
> 16:42:41 _______ ERROR at setup of TestAuthorization.test_access_runtime_profile ________
> 16:42:41 common/custom_cluster_test_suite.py:109: in setup_method
> 16:42:41 self._start_impala_cluster(cluster_args)
> 16:42:41 common/custom_cluster_test_suite.py:144: in _start_impala_cluster
> 16:42:41 check_call(cmd + options, close_fds=True)
> 16:42:41 /usr/lib64/python2.6/subprocess.py:505: in check_call
> 16:42:41 raise CalledProcessError(retcode, cmd)
> 16:42:41 E CalledProcessError: Command '['/data/jenkins/workspace/impala-asf-2.x-exhaustive/repos/Impala/bin/start-impala-cluster.py', '--cluster_size=3', '--num_coordinators=3', '--log_dir=/data/jenkins/workspace/impala-asf-2.x-exhaustive/repos/Impala/logs/custom_cluster_tests', '--log_level=1', '--impalad_args="--server_name=server1 --authorization_policy_file=/test-warehouse/authz-policy.ini --authorized_proxy_user_config=hue=jenkins" ']' returned non-zero exit status 1
> 16:42:41 ---------------------------- Captured stdout setup -----------------------------
> 16:42:41 Starting State Store logging to /data/jenkins/workspace/impala-asf-2.x-exhaustive/repos/Impala/logs/custom_cluster_tests/statestored.INFO
> 16:42:41 Starting Catalog Service logging to /data/jenkins/workspace/impala-asf-2.x-exhaustive/repos/Impala/logs/custom_cluster_tests/catalogd.INFO
> 16:42:41 Starting Impala Daemon logging to /data/jenkins/workspace/impala-asf-2.x-exhaustive/repos/Impala/logs/custom_cluster_tests/impalad.INFO
> 16:42:41 Starting Impala Daemon logging to /data/jenkins/workspace/impala-asf-2.x-exhaustive/repos/Impala/logs/custom_cluster_tests/impalad_node1.INFO
> 16:42:41 Starting Impala Daemon logging to /data/jenkins/workspace/impala-asf-2.x-exhaustive/repos/Impala/logs/custom_cluster_tests/impalad_node2.INFO
> 16:42:41 Error starting cluster: num_known_live_backends did not reach expected value in time
> 16:42:41 ---------------------------- Captured stderr setup -----------------------------
> 16:42:41 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
> 16:42:41 MainThread: Getting num_known_live_backends from ec2-m2-4xlarge-centos-6-4-0a73.vpc.cloudera.com:25000
> 16:42:41 MainThread: Debug webpage not yet available.
> 16:42:41 MainThread: Debug webpage not yet available.
> 16:42:41 MainThread: Waiting for num_known_live_backends=3. Current value: 0
> 16:42:41 MainThread: Getting num_known_live_backends from ec2-m2-4xlarge-centos-6-4-0a73.vpc.cloudera.com:25000
> 16:42:41 MainThread: Waiting for num_known_live_backends=3. Current value: 0
> 16:42:41 MainThread: Getting num_known_live_backends from ec2-m2-4xlarge-centos-6-4-0a73.vpc.cloudera.com:25000
> 16:42:41 MainThread: Waiting for num_known_live_backends=3. Current value: 1
> 16:42:41 MainThread: Getting num_known_live_backends from ec2-m2-4xlarge-centos-6-4-0a73.vpc.cloudera.com:25000
> 16:42:41 MainThread: Waiting for num_known_live_backends=3. Current value: 2
> 16:42:41 MainThread: Getting num_known_live_backends from ec2-m2-4xlarge-centos-6-4-0a73.vpc.cloudera.com:25000
> 16:42:41 MainThread: Waiting for num_known_live_backends=3. Current value: 2
> 16:42:41 MainThread: Getting num_known_live_backends from ec2-m2-4xlarge-centos-6-4-0a73.vpc.cloudera.com:25000
> 16:42:41 MainThread: Waiting for num_known_live_backends=3. Current value: 2
> 16:42:41 MainThread: Getting num_known_live_backends from ec2-m2-4xlarge-centos-6-4-0a73.vpc.cloudera.com:25000
> 16:42:41 MainThread: Waiting for num_known_live_backends=3. Current value: 2
> ...
> 16:42:41 MainThread: Getting num_known_live_backends from ec2-m2-4xlarge-centos-6-4-0a73.vpc.cloudera.com:25000
> 16:42:41 MainThread: Waiting for num_known_live_backends=3. Current value: 2
> 16:42:41 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org