[jira] [Resolved] (HBASE-24155) When running the tests, a tremendous number of connections are put into TIME_WAIT.

["Mark Robert Miller (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HBASE-24155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mark Robert Miller resolved HBASE-24155.
----------------------------------------
    Resolution: Information Provided

Man it took me a long time to finally see a lot of what was going on here.

Mostly just seems to be hdfs short circuit read socket polling management that you can just call me not a fan of and with defaults that you can call me an anti fan of. Couple that with some hbase fail and fast retry stuff, especially in like snapshooting or splitting stuff, and well, the number of potential sockets (many without a proper tcp lifecycle) are just one part of the resulting fun.

> When running the tests, a tremendous number of connections are put into TIME_WAIT.
> ----------------------------------------------------------------------------------
>
>                 Key: HBASE-24155
>                 URL: https://issues.apache.org/jira/browse/HBASE-24155
>             Project: HBase
>          Issue Type: Test
>          Components: test
>            Reporter: Mark Robert Miller
>            Priority: Major
>
> When you run the test suite and monitor the number of connections in TIME_WAIT, it appears that a very large number of connections do not end up with a proper connection close lifecycle or perhaps proper reuse.
> Given connections can stay in TIME_WAIT from 1-4 minutes depending on OS/Env, running the tests faster or with more tests in parallel increases the TIME_WAIT connection buildup. Some tests spin up a very, very large number of connections and if the wrong ones run at the same time, this can also greatly increase the number of connections put into TIME_WAIT. This can have a dramatic affect on performance (as it can take longer to create a new connection) or flat out fail or timeout.
> In my experience, a much, much smaller number of connections in a test suite would end up in TIME_WAIT when connection handling is all correct.
> Notes to come in comments below.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)