[jira] [Created] (KNOX-2003) HTTP connections between Knox gateway and Service backend in CLOSE_WAIT, not getting closed and hang around seemingly forever

["Rajat Goel (Jira)" <ji...@apache.org>]

Rajat Goel created KNOX-2003:
--------------------------------

             Summary: HTTP connections between Knox gateway and Service backend in CLOSE_WAIT, not getting closed and hang around seemingly forever
                 Key: KNOX-2003
                 URL: https://issues.apache.org/jira/browse/KNOX-2003
             Project: Apache Knox
          Issue Type: Bug
          Components: Server
    Affects Versions: 1.0.0
         Environment: HDP 3.1.0
            Reporter: Rajat Goel
         Attachments: Screenshot 2019-08-27 at 10.49.05 PM.png, gateway.out_close_wait_thread_dump

With UI sending multiple HTTP requests for different UI pages, TCP connections (between Knox and Service backend) corresponding to some of these requests move to CLOSE_WAIT state few seconds after the request is complete. These connections are stuck in this state indefinitely until Knox server is restarted. ‘lsof’ output for Knox server process shows that Knox is still holding ‘fd’ for these connections. I took raw packet dump using Tshark/Wireshark and following are the observations:
 * UI sends GET request for Knox which is proxied correctly to backend.
 * Backend sends 200 OK response with data to Knox which is proxied to Frontend.
 * After few seconds, backend server sends a TCP [FIN] [ACK] packet to Knox for connection closure, possible due to idle timeout expiring at backend service.
 * Knox server send a TCP [ACK] => Connection moves to CLOSE_WAIT state and thereafter no communication.
 * In normal scenario for connections which are closed properly, Knox also sends a [FIN] {ACK] TCP packet to backend and then backend responds with [ACK] and connection is closed properly. 

 

The issue here is Knox is not sending [FIN][ACK] to backend. This looks to be an issue with Jetty server and might possibly related to [https://github.com/eclipse/jetty.project/issues/2169].

 

Attaching Wireshark UI screenshot and Thread dump of Knox gateway server.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)