You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Quanlong Huang (Jira)" <ji...@apache.org> on 2021/03/08 08:14:00 UTC
[jira] [Created] (IMPALA-10562)
TestGracefulShutdown::test_shutdown_idle failes in GCE instance
Quanlong Huang created IMPALA-10562:
---------------------------------------
Summary: TestGracefulShutdown::test_shutdown_idle failes in GCE instance
Key: IMPALA-10562
URL: https://issues.apache.org/jira/browse/IMPALA-10562
Project: IMPALA
Issue Type: Test
Reporter: Quanlong Huang
Google Cloud will add a line in /etc/hosts for resolving the hostname of the instance. E.g.
{code:java}
[systest@quanlong-compile impala]$ cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
x.x.x.x quanlong-compile.gce.cloudera.com quanlong-compile # Added by Google
x.x.x.x metadata.google.internal # Added by Google
127.0.0.1 quanlong-compile quanlong-compile.gce.cloudera.com
{code}
(The IPs are manually masked)
This causes connection refused failure in TestGracefulShutdown::test_shutdown_idle
{code:java}
tests/custom_cluster/test_restart_services.py:295: in test_shutdown_idle
query_options={})
tests/common/impala_test_suite.py:814: in wrapper
return function(*args, **kwargs)
tests/common/impala_test_suite.py:822: in execute_query_expect_success
result = cls.__execute_query(impalad_client, query, query_options, user)
tests/common/impala_test_suite.py:923: in __execute_query
return impalad_client.execute(query, user=user)
tests/common/impala_connection.py:205: in execute
return self.__beeswax_client.execute(sql_stmt, user=user)
tests/beeswax/impala_beeswax.py:187: in execute
handle = self.__execute_query(query_string.strip(), user=user)
tests/beeswax/impala_beeswax.py:363: in __execute_query
handle = self.execute_query_async(query_string, user=user)
tests/beeswax/impala_beeswax.py:357: in execute_query_async
handle = self.__do_rpc(lambda: self.imp_service.query(query,))
tests/beeswax/impala_beeswax.py:520: in __do_rpc
raise ImpalaBeeswaxException(self.__build_error_message(b), b)
E ImpalaBeeswaxException: ImpalaBeeswaxException:
E INNER EXCEPTION: <class 'beeswaxd.ttypes.BeeswaxException'>
E MESSAGE: Rpc to 127.0.0.1:27001 failed with error 'RemoteShutdown() RPC failed: Network error: Client connection negotiation failed: client connection to 127.0.0.1:27001: connect: Connection refused (error 111)'
{code}
The failed command is "{{:shutdown('localhost:27001')}}".
Checked with the impalad, it's listening on "quanlong-compile.gce.cloudera.com:27001":
{code:java}
[systest@quanlong-compile impala]$ ps aux | grep 28771
systest 28771 11.0 3.5 22579424 2165200 pts/1 Sl 23:28 0:15 /home/systest/impala/be/build/latest/service/impalad -disconnected_session_timeout 21600 -kudu_client_rpc_timeout_ms 0 -kudu_master_hosts localhost -mem_limit=12884
901888 -logbufsecs=5 -v=1 -max_log_files=10 -log_filename=impalad_node1 -log_dir=/tmp/ -beeswax_port=21001 -hs2_port=21051 -hs2_http_port=28001 -krpc_port=27001 -state_store_subscriber_port=23001 -webserver_port=25001 --shutdown_
grace_period_s=1 --hostname=quanlong-compile.gce.cloudera.com --default_query_options=
systest 30727 0.0 0.0 112828 1032 pts/1 S+ 23:31 0:00 grep --color=auto 28771
[systest@quanlong-compile impala]$ lsof -p 28771 | grep 27001
impalad 28771 systest 341u IPv4 577338 0t0 TCP quanlong-compile.gce.cloudera.com:27001 (LISTEN)
{code}
The shutdown command works with hostname:
{code:java}
[localhost.localdomain:21050] default> :shutdown('localhost:27001');
Query: :shutdown('localhost:27001')
Query submitted at: 2021-03-07 23:39:25 (Coordinator: http://quanlong-compile.gce.cloudera.com:25000)
ERROR: Rpc to 127.0.0.1:27001 failed with error 'RemoteShutdown() RPC failed: Network error: Client connection negotiation failed: client connection to 127.0.0.1:27001: connect: Connection refused (error 111)'
[localhost.localdomain:21050] default> :shutdown('quanlong-compile.gce.cloudera.com:27001');
Query: :shutdown('quanlong-compile.gce.cloudera.com:27001')
Query submitted at: 2021-03-07 23:41:35 (Coordinator: http://quanlong-compile.gce.cloudera.com:25000)
Query progress can be monitored at: http://quanlong-compile.gce.cloudera.com:25000/query_plan?query_id=81475574dca0db03:6e2f767d00000000
+-------------------------------------------------------------------------------------------------------------------------------------------+
| summary |
+-------------------------------------------------------------------------------------------------------------------------------------------+
| shutdown grace period left: 1s000ms, deadline left: 1h, queries registered on coordinator: 0, queries executing: 0, fragment instances: 0 |
+-------------------------------------------------------------------------------------------------------------------------------------------+
Fetched 1 row(s) in 0.01s
{code}
Removing the line "x.x.x.x quanlong-compile.gce.cloudera.com quanlong-compile # Added by Google" can pass the test. Maybe we should add the logics in bin/bootstrap_system.sh when running in GCE instance.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org