You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Quanlong Huang (Jira)" <ji...@apache.org> on 2021/02/28 13:14:00 UTC
[jira] [Commented] (IMPALA-10120) Beeline hangs when connecting to
coordinators
[ https://issues.apache.org/jira/browse/IMPALA-10120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17292401#comment-17292401 ]
Quanlong Huang commented on IMPALA-10120:
-----------------------------------------
Look back into this, the issue still exists. The invalid query option doesn't cause the hanging. In the jstack of beeline client, it's waiting for response of GetInfo():
{code:java}
"main" #1 prio=5 os_prio=0 tid=0x00007f5c9c034000 nid=0x3bec runnable [0x00007f5ca29bd000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
- locked <0x00000000feb239c8> (a java.io.BufferedInputStream)
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
at org.apache.thrift.protocol.TBinaryProtocol.readByte(TBinaryProtocol.java:285)
at org.apache.thrift.protocol.TBinaryProtocol.readFieldBegin(TBinaryProtocol.java:243)
at org.apache.hive.service.rpc.thrift.TCLIService$GetInfo_result$GetInfo_resultStandardScheme.read(TCLIService.java:5261)
at org.apache.hive.service.rpc.thrift.TCLIService$GetInfo_result$GetInfo_resultStandardScheme.read(TCLIService.java:5254)
at org.apache.hive.service.rpc.thrift.TCLIService$GetInfo_result.read(TCLIService.java:5201)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86)
at org.apache.hive.service.rpc.thrift.TCLIService$Client.recv_GetInfo(TCLIService.java:222)
at org.apache.hive.service.rpc.thrift.TCLIService$Client.GetInfo(TCLIService.java:209) <------------ A GetInfo Request
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hive.jdbc.HiveConnection$SynchronizedHandler.invoke(HiveConnection.java:1727)
at com.sun.proxy.$Proxy21.GetInfo(Unknown Source)
at org.apache.hive.jdbc.HiveDatabaseMetaData.getServerInfo(HiveDatabaseMetaData.java:1232)
at org.apache.hive.jdbc.HiveDatabaseMetaData.getSQLKeywords(HiveDatabaseMetaData.java:632)
at org.apache.hive.beeline.SQLCompleter.getSQLCompleters(SQLCompleter.java:56)
at org.apache.hive.beeline.DatabaseConnection.setCompletions(DatabaseConnection.java:99)
at org.apache.hive.beeline.BeeLine.setCompletions(BeeLine.java:2411)
at org.apache.hive.beeline.Commands.connect(Commands.java:1693)
at org.apache.hive.beeline.Commands.connect(Commands.java:1574)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hive.beeline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:56)
at org.apache.hive.beeline.BeeLine.execCommandWithPrefix(BeeLine.java:1456)
at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1495)
at org.apache.hive.beeline.BeeLine.connectUsingArgs(BeeLine.java:917)
at org.apache.hive.beeline.BeeLine.initArgs(BeeLine.java:803)
at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1108)
at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1082)
at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:546)
at org.apache.hive.beeline.BeeLine.main(BeeLine.java:528)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
{code}
However, doing a minidump on impalad and resolving it doesn't find any hanging threads doing the GetInfo request. Impalad may already return it. But Beeline client fails to parse it correctly. The thrift version of Beeline we used is 0.9.3-1. It's 0.9.3-p8 in Impala. Not sure if the difference in patch versions cause the issue.
> Beeline hangs when connecting to coordinators
> ---------------------------------------------
>
> Key: IMPALA-10120
> URL: https://issues.apache.org/jira/browse/IMPALA-10120
> Project: IMPALA
> Issue Type: Bug
> Components: Clients
> Reporter: Quanlong Huang
> Priority: Critical
>
> Beeline is always hanging when connecting to a coordinator:
> {code:java}
> $ beeline -u "jdbc:hive2://localhost:21050/default;auth=noSasl"
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in [jar:file:/home/quanlong/workspace/Impala/toolchain/cdp_components-4493826/apache-hive-3.1.3000.7.2.1.0-287-bin/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in [jar:file:/home/quanlong/workspace/Impala/toolchain/cdp_components-4493826/hadoop-3.1.1.7.2.1.0-287/share/hadoop/common/lib/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console. Set system property 'log4j2.debug' to show Log4j2 internal initialization logging.
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in [jar:file:/home/quanlong/workspace/Impala/toolchain/cdp_components-4493826/apache-hive-3.1.3000.7.2.1.0-287-bin/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in [jar:file:/home/quanlong/workspace/Impala/toolchain/cdp_components-4493826/hadoop-3.1.1.7.2.1.0-287/share/hadoop/common/lib/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
> SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
> Connecting to jdbc:hive2://localhost:21050/default;auth=noSasl
> Connected to: Impala (version 4.0.0-SNAPSHOT)
> Driver: Hive JDBC (version 3.1.3000.7.2.1.0-287)
> Transaction isolation: TRANSACTION_REPEATABLE_READ
> {code}
> Looking into the impalad log file, there is a wrong option set:
> {code:java}
> I0901 15:41:14.577576 25325 TAcceptQueueServer.cpp:340] New connection to server hiveserver2-frontend from client <Host: ::ffff:127.0.0.1 Port: 51776>
> I0901 15:41:14.577911 25597 impala-hs2-server.cc:300] Opening session: 204a3f33cc8e28ea:d6a915ab96b26aa7 request username:
> I0901 15:41:14.577970 25597 status.cc:129] Invalid query option: set:hiveconf:hive.server2.thrift.resultset.default.fetch.size
> @ 0x1cdba3d impala::Status::Status()
> @ 0x24c673f impala::SetQueryOption()
> @ 0x250c1d1 impala::ImpalaServer::OpenSession()
> @ 0x2b0dc45 apache::hive::service::cli::thrift::TCLIServiceProcessor::process_OpenSession()
> @ 0x2b0d993 apache::hive::service::cli::thrift::TCLIServiceProcessor::dispatchCall()
> @ 0x2acd15a impala::ImpalaHiveServer2ServiceProcessor::dispatchCall()
> @ 0x1c8a483 apache::thrift::TDispatchProcessor::process()
> @ 0x218ab4a apache::thrift::server::TAcceptQueueServer::Task::run()
> @ 0x218004a impala::ThriftThread::RunRunnable()
> @ 0x2181686 boost::_mfi::mf2<>::operator()()
> @ 0x218151a boost::_bi::list3<>::operator()<>()
> @ 0x2181260 boost::_bi::bind_t<>::operator()()
> @ 0x2181172 boost::detail::function::void_function_obj_invoker0<>::invoke()
> @ 0x20fba57 boost::function0<>::operator()()
> @ 0x26cb779 impala::Thread::SuperviseThread()
> @ 0x26d3716 boost::_bi::list5<>::operator()<>()
> @ 0x26d363a boost::_bi::bind_t<>::operator()()
> @ 0x26d35fb boost::detail::thread_data<>::run()
> @ 0x3eb7ae1 thread_proxy
> @ 0x7fc9443456b9 start_thread
> @ 0x7fc940e334dc clone
> I0901 15:41:14.739985 25597 impala-hs2-server.cc:405] Opened session: 204a3f33cc8e28ea:d6a915ab96b26aa7 effective username:
> I0901 15:41:14.781677 25597 impala-hs2-server.cc:426] GetInfo(): request=TGetInfoReq {
> 01: sessionHandle (struct) = TSessionHandle {
> 01: sessionId (struct) = THandleIdentifier {
> 01: guid (string) = "\xea(\x8e\xcc3?J \xa7j\xb2\x96\xab\x15\xa9\xd6",
> 02: secret (string) = "\xf9\x11\xac\xf5wTB\xe8\xacN\r\x8e\xe3MJ\xa9",
> },
> },
> 02: infoType (i32) = 17,
> }
> I0901 15:41:14.785768 25597 impala-hs2-server.cc:426] GetInfo(): request=TGetInfoReq {
> 01: sessionHandle (struct) = TSessionHandle {
> 01: sessionId (struct) = THandleIdentifier {
> 01: guid (string) = "\xea(\x8e\xcc3?J \xa7j\xb2\x96\xab\x15\xa9\xd6",
> 02: secret (string) = "\xf9\x11\xac\xf5wTB\xe8\xacN\r\x8e\xe3MJ\xa9",
> },
> },
> 02: infoType (i32) = 18,
> }
> I0901 15:41:14.788753 25597 impala-hs2-server.cc:426] GetInfo(): request=TGetInfoReq {
> 01: sessionHandle (struct) = TSessionHandle {
> 01: sessionId (struct) = THandleIdentifier {
> 01: guid (string) = "\xea(\x8e\xcc3?J \xa7j\xb2\x96\xab\x15\xa9\xd6",
> 02: secret (string) = "\xf9\x11\xac\xf5wTB\xe8\xacN\r\x8e\xe3MJ\xa9",
> },
> },
> 02: infoType (i32) = 10006,
> }
> {code}
> Maybe the hanging is due to this.
> [~stakiar], do you know the background of the fetch.size setting of hive?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org