[jira] [Created] (THRIFT-5435) Hive Server 2 crashes when Thrift library encounters particular security error

["Matt McCline (Jira)" <ji...@apache.org>]

Matt McCline created THRIFT-5435:
------------------------------------

             Summary: Hive Server 2 crashes when Thrift library encounters particular security error
                 Key: THRIFT-5435
                 URL: https://issues.apache.org/jira/browse/THRIFT-5435
             Project: Thrift
          Issue Type: Improvement
            Reporter: Matt McCline


These portion of a blog called "NO DATA OR NO SASL DATA IN THE STREAM” ERROR IN HIVESERVER2 LOG" by Eric Liu. NOTE: The web link causes the Apache bot to remove it. You will need to Google it yourself.

It addresses HIve Server 2 crashes our customers have seen and filed customer service complaints on. The Apache Thrift library raises a fatal RuntimeException when In my opinion it should raise a non-fatal Exception instead.

Portion of the blog:

-----------------------------------------------------------------------------------------------------

"I have seen lots of users complain about seeing lots of “No data or no sasl data in the stream” errors in the HiveServer2 server log, yet they have not noticed any performance impact nor query failure for Hive. So I think it would be good to write a blog about the possible reason behind this to clarify and remove the concerns that users have. The following shows the full error message and stacktrace taken from HiveServer2 log:
 
ERROR org.apache.thrift.server.TThreadPoolServer: [HiveServer2-Handler-Pool: Thread-533556]: Error occurred during processing of message.
java.lang.RuntimeException: org.apache.thrift.transport.TSaslTransportException: No data or no sasl data in the stream
at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:765)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:762)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1687)
at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:762)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:268)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.thrift.transport.TSaslTransportException: No data or no sasl data in the stream
The likely cause is below:
 # You have kerberos enabled
 # You have multiple HiveServer2 hosts
 # You have Load Balancer enabled in front of all HS2 servers that have such errors

If you have above setup, the error message you saw in HiveServer2 is harmless and can be safely ignored. This just indicated that SASL negotiation failed for one particular Hive client, which in this case would be the Load Balancer who pings regularly to those HiveServer2’s to check for connectivity. Those pings from LB were trying with PLAIN TCP connection, hence those messages.

-----------------------------------------------------------------------------------------------------

This is similar exception information from Hive Server 2 that crashed on our customer's machine:

ExceptionClassName
java.lang.RuntimeException
ExceptionStackTrace
java.lang.RuntimeException: org.apache.thrift.transport.TSaslTransportException: No data or no sasl data in the stream

              at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)

              at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:694)

              at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:691)

              at java.security.AccessController.doPrivileged(Native Method)

              at javax.security.auth.Subject.doAs(Subject.java:360)

              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1710)

              at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:691)

              at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269)

              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

              at java.lang.Thread.run(Thread.java:748)

Caused by: org.apache.thrift.transport.TSaslTransportException: No data or no sasl data in the stream

              at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:326)

              at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)

              at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)

              ... 10 more

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)