You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@couchdb.apache.org by "MohammadrezaNasrabadi (via GitHub)" <gi...@apache.org> on 2024/08/22 10:58:08 UTC

[I] Too Many Pending Connections [couchdb]

MohammadrezaNasrabadi opened a new issue, #5196:
URL: https://github.com/apache/couchdb/issues/5196

   Title: Too Many Pending Connections
   
   On a Couchdb cluster with three nodes, the number of connections in
   CLOSE-WAIT state to one of the nodes gradually increases.  This is
   what I get right now (Couchdb has been running for about four days):
   ```
   $ ss -nt | grep CLOSE-WAIT | wc -l
   4514
   $ ss -nt | grep CLOSE-WAIT | head
   CLOSE-WAIT 1      0       192.168.40.2:35938    192.168.40.2:5984
   CLOSE-WAIT 1      0       192.168.40.2:44694    192.168.40.2:5984
   CLOSE-WAIT 1      0       192.168.40.2:60596    192.168.40.2:5984
   CLOSE-WAIT 1      0       192.168.40.2:34240    192.168.40.2:5984
   CLOSE-WAIT 1      0       192.168.40.2:40818    192.168.40.2:5984
   CLOSE-WAIT 1      0       192.168.40.2:43376    192.168.40.2:5984
   CLOSE-WAIT 1      0       192.168.40.2:56616    192.168.40.2:5984
   CLOSE-WAIT 1      0       192.168.40.2:47966    192.168.40.2:5984
   CLOSE-WAIT 1      0       192.168.40.2:48366    192.168.40.2:5984
   CLOSE-WAIT 1      0       192.168.40.2:59302    192.168.40.2:5984
   ```
   Maybe Couchdb does not close connections sometimes?  We use Couchdb 3.3.3.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Too Many Pending Connections [couchdb]

Posted by "MohammadrezaNasrabadi (via GitHub)" <gi...@apache.org>.
MohammadrezaNasrabadi commented on issue #5196:
URL: https://github.com/apache/couchdb/issues/5196#issuecomment-2466663169

   We performed experiments to find the main reason for this problem.
   The problem (connections in CLOSE-WAIT state) happens even without
   a CouchDB cluster and with a local loopback connection.  It seems
   that CouchDB does not close some of its connections after the
   client closes it; therefore, the connections remain in CLOSE-WAIT
   state.
   
   We connect to CouchDB from an Elixir application using the hackney
   library.  After a few weeks of running, we have thousands of CouchDB
   connections in CLOSE-WAIT state that disappear only by restarting
   CouchDB.  We have not yet found which of the connections made by
   hackney remain in CLOSE-WAIT state when hackney closes it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Too Many Pending Connections [couchdb]

Posted by "MohammadrezaNasrabadi (via GitHub)" <gi...@apache.org>.
MohammadrezaNasrabadi commented on issue #5196:
URL: https://github.com/apache/couchdb/issues/5196#issuecomment-2311786564

   We don't apply any configuration about the firewall to have limitation or control over the connections.
   
   Do you have any benchmark results based on the number of shards, replicas, amount of database requests and the available resources? It will be useful to compare our CouchDB infrastructure status with the ideal resource requirements.
   
   I will share the detailed information of our infrastructure if needed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Too Many Pending Connections [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on issue #5196:
URL: https://github.com/apache/couchdb/issues/5196#issuecomment-2314124648

   It's hard to compare architectures, it depends on the requirements. For instance some of our larger clusters have 30+ nodes and our load balancer handles 100k concurrent open connections. Even there, on the load balancer usually there almost no CLOSE_WAIT connections. At most I could see a short temporary increase in CLOSE_WAIT up to 3k out of 100k established connections, then they get cleared back to a few hundred only.
   
   It's good that your resource usage is low it seems. Since you have other nodes to compare to, and others don't seem to show the same behavior try to see what's different between them (configuration, hardware resources). Maybe check your logs for any errors on some vs others?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Too Many Pending Connections [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on issue #5196:
URL: https://github.com/apache/couchdb/issues/5196#issuecomment-2305363699

   From what I remember from a while back, close-wait means the remote client closed the connection, then, the our (server) kernel found out about it, and is now waiting on the application code (CouchDB) to notice as well and close it.
   
   So, perhaps maybe that one node is overwhelmed, sometimes is blocked? (maybe a firewall?), or not getting enough CPU time to run?. Check your resource usage (cpu, memory, open file handles) and the logs to see if there are exceptions or errors on that one node. At least you have a few other nodes to compare to where this doesn't happen.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Too Many Pending Connections [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on issue #5196:
URL: https://github.com/apache/couchdb/issues/5196#issuecomment-2743944351

   We have put fix to close connections proactively in CouchDB  in https://github.com/apache/couchdb/pull/4736. That fix should be in the 3.4.x series. Thanks for your report @MohammadrezaNasrabadi.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Too Many Pending Connections [couchdb]

Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva closed issue #5196: Too Many Pending Connections
URL: https://github.com/apache/couchdb/issues/5196


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org