[jira] [Assigned] (HIVE-20789) HiveServer2 should have Timeouts against clients that never close sockets

["Szehon Ho (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HIVE-20789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Szehon Ho reassigned HIVE-20789:
--------------------------------

    Assignee: Szehon Ho

> HiveServer2 should have Timeouts against clients that never close sockets
> -------------------------------------------------------------------------
>
>                 Key: HIVE-20789
>                 URL: https://issues.apache.org/jira/browse/HIVE-20789
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Szehon Ho
>            Assignee: Szehon Ho
>            Priority: Major
>
> We have had a scenario that health checks sending 0 bytes to HiveServer2 sockets would DDOS the HiveServer2, if for some reason they hang or otherwise don't send TCP FIN, then all HiveServer2 thrift thread-pool threads will block reading the socket.
> This is the stack (we are running an older version of Hive here)
> {noformat}
> "HiveServer2-Handler-Pool: Thread-2512239" - Thread t@2512239
> java.lang.Thread.State: RUNNABLE
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
> at java.net.SocketInputStream.read(SocketInputStream.java:171)
> at java.net.SocketInputStream.read(SocketInputStream.java:141)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> - locked <23781b74> (a java.io.BufferedInputStream)
> at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
> at org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:346)
> at org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:423)
> at org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:405)
> at org.apache.thrift.transport.TSaslServerTransport.read(TSaslServerTransport.java:41)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
> at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
> at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
> at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
> at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:746)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748){noformat}
> Eventually HiveServer2 has no more free threads left.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)