You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tomcat.apache.org by bu...@apache.org on 2017/12/19 17:44:34 UTC

[Bug 61918] New: Nio2Endpoint may miscount connectionLimitLatch while closing socket

https://bz.apache.org/bugzilla/show_bug.cgi?id=61918

            Bug ID: 61918
           Summary: Nio2Endpoint may miscount connectionLimitLatch while
                    closing socket
           Product: Tomcat 8
           Version: 8.5.16
          Hardware: PC
                OS: All
            Status: NEW
          Severity: normal
          Priority: P2
         Component: Connectors
          Assignee: dev@tomcat.apache.org
          Reporter: ryan.fong@workday.com
  Target Milestone: ----

CentOS 6.6
Tomcat 8.5.16
Connector protocol: org.apache.coyote.http11.Http11Nio2Protocol

We have noticed that the connectionCount jmx value for the nio2 connector
trends upwards on our servers during the week and never drops down to 1. 

When we used netstat to trace these connections, we found the expected handful
of connections active but not the value shown by jmx. We also enabled tomcat
debug logging to see if any exceptions occurred during socket operations but
found nothing of interest.

We believe that our data center's network configuration is causing sockets to
be closed independently of the Tomcat Coyote handling but do not know what
specifically is triggering it. We also found a suspicious line in Nio2Endpoint
connector which could explain this false floor that keeps rising. 

I can artificially re-create this situation by forcefully closing the socket in
a Spring MVC controller handling. Every time this endpoint is called, we
prevent Tomcat from decrementing the connection count upon close. See
Nio2Endpoint.closeSocket, line 358, only decrements the connection limit latch
when the socket is open. 

Please see the attachment for:
* MyController.java - artificial recreation using Spring mvc and test framework
(easily placed in a servlet if you also include org.springframework:spring-test
module from maven)
* jmx.png - screenshot of jmx statistic we are monitoring
* connectionCounts.png - dashboard view of our server pool connection count for
a week
* Nio2Endpoint.java.patch - proposed patch for this situation

-- 
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org


[Bug 61918] Nio2Endpoint may miscount connectionLimitLatch while closing socket that is already closed

Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=61918

Remy Maucherat <re...@apache.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |FIXED
             Status|NEW                         |RESOLVED

--- Comment #2 from Remy Maucherat <re...@apache.org> ---
I'm not convinced closing sockets like this is a really good idea. The patch
may reintroduce more serious problems, but adding a closed flag seems possible.
This will be in 9.0.3 and 8.5.25.

-- 
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org


[Bug 61918] Nio2Endpoint may miscount connectionLimitLatch while closing socket

Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=61918

--- Comment #1 from Ryan Fong <ry...@workday.com> ---
Created attachment 35619
  --> https://bz.apache.org/bugzilla/attachment.cgi?id=35619&action=edit
data and patch

-- 
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org


[Bug 61918] Nio2Endpoint may miscount connectionLimitLatch while closing socket that is already closed

Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=61918

Ryan Fong <ry...@workday.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|Nio2Endpoint may miscount   |Nio2Endpoint may miscount
                   |connectionLimitLatch while  |connectionLimitLatch while
                   |closing socket              |closing socket that is
                   |                            |already closed

-- 
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org