You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Michael Ho (JIRA)" <ji...@apache.org> on 2019/04/24 21:02:00 UTC
[jira] [Comment Edited] (IMPALA-8174) test_catalog_restart failed
with exception "Detected catalog service ID change"
[ https://issues.apache.org/jira/browse/IMPALA-8174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16825500#comment-16825500 ]
Michael Ho edited comment on IMPALA-8174 at 4/24/19 9:01 PM:
-------------------------------------------------------------
Just happened again:
{noformat}
Stacktrace
authorization/test_authorization.py:437: in test_catalog_restart
self.role_cleanup(unique_role)
authorization/test_authorization.py:443: in role_cleanup
self.client.execute("drop role %s" % role_name)
common/impala_connection.py:180: in execute
return self.__beeswax_client.execute(sql_stmt, user=user)
beeswax/impala_beeswax.py:183: in execute
handle = self.__execute_query(query_string.strip(), user=user)
beeswax/impala_beeswax.py:358: in __execute_query
handle = self.execute_query_async(query_string, user=user)
beeswax/impala_beeswax.py:352: in execute_query_async
handle = self.__do_rpc(lambda: self.imp_service.query(query,))
beeswax/impala_beeswax.py:512: in __do_rpc
raise ImpalaBeeswaxException(self.__build_error_message(b), b)
E ImpalaBeeswaxException: ImpalaBeeswaxException:
E INNER EXCEPTION: <class 'beeswaxd.ttypes.BeeswaxException'>
E MESSAGE: CatalogException: Detected catalog service ID change. Aborting updateCatalog()
Standard Error
-- 2019-04-24 10:53:40,692 INFO MainThread: Starting cluster with command: /data/jenkins/workspace/impala-asf-master-exhaustive-centos6/repos/Impala/bin/start-impala-cluster.py --cluster_size=3 --num_coordinators=3 --log_dir=/data/jenkins/workspace/impala-asf-master-exhaustive-centos6/repos/Impala/logs/custom_cluster_tests/test_catalog_restart_voZb_Y --log_level=1 '--impalad_args=--server_name=server1 --sentry_config=/data/jenkins/workspace/impala-asf-master-exhaustive-centos6/repos/Impala/fe/src/test/resources/sentry-site.xml ' '--state_store_args=--statestore_update_frequency_ms=50 --statestore_priority_update_frequency_ms=50 --statestore_heartbeat_frequency_ms=50 ' '--catalogd_args=--sentry_config=/data/jenkins/workspace/impala-asf-master-exhaustive-centos6/repos/Impala/fe/src/test/resources/sentry-site.xml ' --impalad_args=--default_query_options=
10:53:42 MainThread: Starting State Store logging to /data/jenkins/workspace/impala-asf-master-exhaustive-centos6/repos/Impala/logs/custom_cluster_tests/test_catalog_restart_voZb_Y/statestored.INFO
10:53:43 MainThread: Starting Catalog Service logging to /data/jenkins/workspace/impala-asf-master-exhaustive-centos6/repos/Impala/logs/custom_cluster_tests/test_catalog_restart_voZb_Y/catalogd.INFO
10:53:46 MainThread: Starting Impala Daemon logging to /data/jenkins/workspace/impala-asf-master-exhaustive-centos6/repos/Impala/logs/custom_cluster_tests/test_catalog_restart_voZb_Y/impalad.INFO
10:53:46 MainThread: Starting Impala Daemon logging to /data/jenkins/workspace/impala-asf-master-exhaustive-centos6/repos/Impala/logs/custom_cluster_tests/test_catalog_restart_voZb_Y/impalad_node1.INFO
10:53:46 MainThread: Starting Impala Daemon logging to /data/jenkins/workspace/impala-asf-master-exhaustive-centos6/repos/Impala/logs/custom_cluster_tests/test_catalog_restart_voZb_Y/impalad_node2.INFO
10:53:49 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
10:53:49 MainThread: Getting num_known_live_backends from shared-centos64-ec2-m2-4xlarge-ondemand-068c.vpc.cloudera.com:25000
10:53:49 MainThread: Debug webpage not yet available: ('Connection aborted.', error(111, 'Connection refused'))
10:53:51 MainThread: Debug webpage not yet available: ('Connection aborted.', error(111, 'Connection refused'))
10:53:53 MainThread: Debug webpage not yet available: ('Connection aborted.', error(111, 'Connection refused'))
10:53:55 MainThread: Waiting for num_known_live_backends=3. Current value: 0
10:53:56 MainThread: Getting num_known_live_backends from shared-centos64-ec2-m2-4xlarge-ondemand-068c.vpc.cloudera.com:25000
10:53:56 MainThread: Waiting for num_known_live_backends=3. Current value: 0
10:53:57 MainThread: Getting num_known_live_backends from shared-centos64-ec2-m2-4xlarge-ondemand-068c.vpc.cloudera.com:25000
10:53:57 MainThread: num_known_live_backends has reached value: 3
10:53:58 MainThread: Getting num_known_live_backends from shared-centos64-ec2-m2-4xlarge-ondemand-068c.vpc.cloudera.com:25001
10:53:58 MainThread: num_known_live_backends has reached value: 3
10:53:58 MainThread: Getting num_known_live_backends from shared-centos64-ec2-m2-4xlarge-ondemand-068c.vpc.cloudera.com:25002
10:53:58 MainThread: num_known_live_backends has reached value: 3
10:53:59 MainThread: Impala Cluster Running with 3 nodes (3 coordinators, 3 executors).
-- 2019-04-24 10:53:59,670 INFO MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
-- 2019-04-24 10:53:59,671 INFO MainThread: Getting metric: statestore.live-backends from shared-centos64-ec2-m2-4xlarge-ondemand-068c.vpc.cloudera.com:25010
-- 2019-04-24 10:53:59,678 INFO MainThread: Starting new HTTP connection (1): shared-centos64-ec2-m2-4xlarge-ondemand-068c.vpc.cloudera.com
-- 2019-04-24 10:53:59,686 INFO MainThread: Metric 'statestore.live-backends' has reached desired value: 4
-- 2019-04-24 10:53:59,686 INFO MainThread: Getting num_known_live_backends from shared-centos64-ec2-m2-4xlarge-ondemand-068c.vpc.cloudera.com:25000
-- 2019-04-24 10:53:59,687 INFO MainThread: Starting new HTTP connection (1): shared-centos64-ec2-m2-4xlarge-ondemand-068c.vpc.cloudera.com
-- 2019-04-24 10:53:59,692 INFO MainThread: num_known_live_backends has reached value: 3
-- 2019-04-24 10:53:59,692 INFO MainThread: Getting num_known_live_backends from shared-centos64-ec2-m2-4xlarge-ondemand-068c.vpc.cloudera.com:25001
-- 2019-04-24 10:53:59,694 INFO MainThread: Starting new HTTP connection (1): shared-centos64-ec2-m2-4xlarge-ondemand-068c.vpc.cloudera.com
-- 2019-04-24 10:53:59,698 INFO MainThread: num_known_live_backends has reached value: 3
-- 2019-04-24 10:53:59,698 INFO MainThread: Getting num_known_live_backends from shared-centos64-ec2-m2-4xlarge-ondemand-068c.vpc.cloudera.com:25002
-- 2019-04-24 10:53:59,699 INFO MainThread: Starting new HTTP connection (1): shared-centos64-ec2-m2-4xlarge-ondemand-068c.vpc.cloudera.com
-- 2019-04-24 10:53:59,704 INFO MainThread: num_known_live_backends has reached value: 3
SET client_identifier=authorization/test_authorization.py::TestAuthorization::()::test_catalog_restart;
-- connecting to: localhost:21000
-- connecting to localhost:21050 with impyla
Conn <impala.hiveserver2.HiveServer2Connection object at 0x7fa441ca3750>
-- 2019-04-24 10:53:59,987 INFO MainThread: Closing active operation
-- executing against localhost:21000
show roles;
-- 2019-04-24 10:54:00,099 INFO MainThread: Started query be480e462c34ca8d:7f87d97700000000
-- executing against localhost:21000
create role test_catalog_restart_268745a6_role_foo;
-- 2019-04-24 10:54:00,273 INFO MainThread: Started query 424b4a8776c151bc:3a85269e00000000
-- executing against localhost:21000
create role test_catalog_restart_268745a6_role_bar;
-- 2019-04-24 10:54:00,314 INFO MainThread: Started query 2c4206ed8920b68e:c7ec9dc100000000
-- executing against localhost:21000
create role test_catalog_restart_268745a6_role_baz;
-- 2019-04-24 10:54:00,351 INFO MainThread: Started query f3432158841cbde7:39dc568f00000000
-- executing against localhost:21000
grant all on database functional to role test_catalog_restart_268745a6_role_foo;
-- 2019-04-24 10:54:00,453 INFO MainThread: Started query 854f9f2d7ff7126c:48a6456300000000
-- executing against localhost:21000
grant all on database functional_kudu to role test_catalog_restart_268745a6_role_bar;
-- 2019-04-24 10:54:00,496 INFO MainThread: Started query 594a381555ea1208:24c36a9700000000
-- executing against localhost:21000
grant all on database functional_avro to role test_catalog_restart_268745a6_role_baz;
-- 2019-04-24 10:54:00,531 INFO MainThread: Started query dd477f3e7de516fe:2a8c839f00000000
-- executing against localhost:21000
show grant role test_catalog_restart_268745a6_role_foo;
-- 2019-04-24 10:54:00,617 INFO MainThread: Started query 5044f2e5c89903d6:11eb95e900000000
-- executing against localhost:21000
show grant role test_catalog_restart_268745a6_role_bar;
-- 2019-04-24 10:54:00,659 INFO MainThread: Started query a74e7f8005ab5ff0:c001bef500000000
-- executing against localhost:21000
show grant role test_catalog_restart_268745a6_role_baz;
-- 2019-04-24 10:54:00,698 INFO MainThread: Started query 1043993ad381db85:e1f4fe8e00000000
-- 2019-04-24 10:54:00,703 INFO MainThread: Starting cluster with command: /data/jenkins/workspace/impala-asf-master-exhaustive-centos6/repos/Impala/bin/start-impala-cluster.py --cluster_size=3 --num_coordinators=3 --log_dir=/data/jenkins/workspace/impala-asf-master-exhaustive-centos6/repos/Impala/logs/custom_cluster_tests --log_level=1 --catalogd_args=--sentry_config=/data/jenkins/workspace/impala-asf-master-exhaustive-centos6/repos/Impala/fe/src/test/resources/sentry-site.xml --restart_catalogd_only --impalad_args=--default_query_options=
10:54:02 MainThread: Starting Catalog Service logging to /data/jenkins/workspace/impala-asf-master-exhaustive-centos6/repos/Impala/logs/custom_cluster_tests/catalogd.INFO
10:54:08 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
10:54:08 MainThread: Getting num_known_live_backends from shared-centos64-ec2-m2-4xlarge-ondemand-068c.vpc.cloudera.com:25000
10:54:08 MainThread: num_known_live_backends has reached value: 3
10:54:08 MainThread: Getting num_known_live_backends from shared-centos64-ec2-m2-4xlarge-ondemand-068c.vpc.cloudera.com:25001
10:54:08 MainThread: num_known_live_backends has reached value: 3
10:54:09 MainThread: Getting num_known_live_backends from shared-centos64-ec2-m2-4xlarge-ondemand-068c.vpc.cloudera.com:25002
10:54:09 MainThread: num_known_live_backends has reached value: 3
10:54:09 MainThread: Impala Cluster Running with 3 nodes (3 coordinators, 3 executors).
-- 2019-04-24 10:54:10,034 INFO MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
-- 2019-04-24 10:54:10,035 INFO MainThread: Getting metric: statestore.live-backends from shared-centos64-ec2-m2-4xlarge-ondemand-068c.vpc.cloudera.com:25010
-- 2019-04-24 10:54:10,042 INFO MainThread: Starting new HTTP connection (1): shared-centos64-ec2-m2-4xlarge-ondemand-068c.vpc.cloudera.com
-- 2019-04-24 10:54:10,050 INFO MainThread: Metric 'statestore.live-backends' has reached desired value: 4
-- 2019-04-24 10:54:10,051 INFO MainThread: Getting num_known_live_backends from shared-centos64-ec2-m2-4xlarge-ondemand-068c.vpc.cloudera.com:25000
-- 2019-04-24 10:54:10,052 INFO MainThread: Starting new HTTP connection (1): shared-centos64-ec2-m2-4xlarge-ondemand-068c.vpc.cloudera.com
-- 2019-04-24 10:54:10,057 INFO MainThread: num_known_live_backends has reached value: 3
-- 2019-04-24 10:54:10,057 INFO MainThread: Getting num_known_live_backends from shared-centos64-ec2-m2-4xlarge-ondemand-068c.vpc.cloudera.com:25001
-- 2019-04-24 10:54:10,059 INFO MainThread: Starting new HTTP connection (1): shared-centos64-ec2-m2-4xlarge-ondemand-068c.vpc.cloudera.com
-- 2019-04-24 10:54:10,062 INFO MainThread: num_known_live_backends has reached value: 3
-- 2019-04-24 10:54:10,063 INFO MainThread: Getting num_known_live_backends from shared-centos64-ec2-m2-4xlarge-ondemand-068c.vpc.cloudera.com:25002
-- 2019-04-24 10:54:10,064 INFO MainThread: Starting new HTTP connection (1): shared-centos64-ec2-m2-4xlarge-ondemand-068c.vpc.cloudera.com
-- 2019-04-24 10:54:10,068 INFO MainThread: num_known_live_backends has reached value: 3
-- executing against localhost:21000
show grant role test_catalog_restart_268745a6_role_foo;
-- 2019-04-24 10:54:10,117 INFO MainThread: Started query 6647c72f4b7ef1c7:a71a6dd600000000
-- executing against localhost:21000
show grant role test_catalog_restart_268745a6_role_bar;
-- 2019-04-24 10:54:10,154 INFO MainThread: Started query ce4807be3308a553:4f87ee9400000000
-- executing against localhost:21000
show grant role test_catalog_restart_268745a6_role_baz;
-- 2019-04-24 10:54:10,186 INFO MainThread: Started query d74becd643adfa5b:1b709c9f00000000
-- executing against localhost:21000
show roles;
-- 2019-04-24 10:54:10,217 INFO MainThread: Started query 854e80786fc64166:fbfd09db00000000
-- executing against localhost:21000
drop role test_catalog_restart_268745a6_role_bar;
{noformat}
was (Author: kwho):
Just happened again.
> test_catalog_restart failed with exception "Detected catalog service ID change"
> -------------------------------------------------------------------------------
>
> Key: IMPALA-8174
> URL: https://issues.apache.org/jira/browse/IMPALA-8174
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Affects Versions: Impala 3.2.0
> Reporter: Michael Ho
> Assignee: Fredy Wijaya
> Priority: Blocker
> Labels: broken-build
>
> test_catalog_restart failed with exception "Detected catalog service ID change". Didn't dig too much into the details.
> cc'ing [~fredyw],[~bharathv],[~paul.rogers]
> {noformat}
> Error Message
> ImpalaBeeswaxException: ImpalaBeeswaxException: INNER EXCEPTION: <class 'beeswaxd.ttypes.BeeswaxException'> MESSAGE: CatalogException: Detected catalog service ID change. Aborting updateCatalog()
> Stacktrace
> authorization/test_authorization.py:432: in test_catalog_restart
> self.role_cleanup(unique_role)
> authorization/test_authorization.py:438: in role_cleanup
> self.client.execute("drop role %s" % role_name)
> common/impala_connection.py:174: in execute
> return self.__beeswax_client.execute(sql_stmt, user=user)
> beeswax/impala_beeswax.py:183: in execute
> handle = self.__execute_query(query_string.strip(), user=user)
> beeswax/impala_beeswax.py:358: in __execute_query
> handle = self.execute_query_async(query_string, user=user)
> beeswax/impala_beeswax.py:352: in execute_query_async
> handle = self.__do_rpc(lambda: self.imp_service.query(query,))
> beeswax/impala_beeswax.py:512: in __do_rpc
> raise ImpalaBeeswaxException(self.__build_error_message(b), b)
> E ImpalaBeeswaxException: ImpalaBeeswaxException:
> E INNER EXCEPTION: <class 'beeswaxd.ttypes.BeeswaxException'>
> E MESSAGE: CatalogException: Detected catalog service ID change. Aborting updateCatalog()
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org