You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@couchdb.apache.org by "MohammadrezaNasrabadi (via GitHub)" <gi...@apache.org> on 2024/08/22 10:58:08 UTC
[I] Too Many Pending Connections [couchdb]
MohammadrezaNasrabadi opened a new issue, #5196:
URL: https://github.com/apache/couchdb/issues/5196
Title: Too Many Pending Connections
On a Couchdb cluster with three nodes, the number of connections in
CLOSE-WAIT state to one of the nodes gradually increases. This is
what I get right now (Couchdb has been running for about four days):
```
$ ss -nt | grep CLOSE-WAIT | wc -l
4514
$ ss -nt | grep CLOSE-WAIT | head
CLOSE-WAIT 1 0 192.168.40.2:35938 192.168.40.2:5984
CLOSE-WAIT 1 0 192.168.40.2:44694 192.168.40.2:5984
CLOSE-WAIT 1 0 192.168.40.2:60596 192.168.40.2:5984
CLOSE-WAIT 1 0 192.168.40.2:34240 192.168.40.2:5984
CLOSE-WAIT 1 0 192.168.40.2:40818 192.168.40.2:5984
CLOSE-WAIT 1 0 192.168.40.2:43376 192.168.40.2:5984
CLOSE-WAIT 1 0 192.168.40.2:56616 192.168.40.2:5984
CLOSE-WAIT 1 0 192.168.40.2:47966 192.168.40.2:5984
CLOSE-WAIT 1 0 192.168.40.2:48366 192.168.40.2:5984
CLOSE-WAIT 1 0 192.168.40.2:59302 192.168.40.2:5984
```
Maybe Couchdb does not close connections sometimes? We use Couchdb 3.3.3.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Too Many Pending Connections [couchdb]
Posted by "MohammadrezaNasrabadi (via GitHub)" <gi...@apache.org>.
MohammadrezaNasrabadi commented on issue #5196:
URL: https://github.com/apache/couchdb/issues/5196#issuecomment-2466663169
We performed experiments to find the main reason for this problem.
The problem (connections in CLOSE-WAIT state) happens even without
a CouchDB cluster and with a local loopback connection. It seems
that CouchDB does not close some of its connections after the
client closes it; therefore, the connections remain in CLOSE-WAIT
state.
We connect to CouchDB from an Elixir application using the hackney
library. After a few weeks of running, we have thousands of CouchDB
connections in CLOSE-WAIT state that disappear only by restarting
CouchDB. We have not yet found which of the connections made by
hackney remain in CLOSE-WAIT state when hackney closes it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Too Many Pending Connections [couchdb]
Posted by "MohammadrezaNasrabadi (via GitHub)" <gi...@apache.org>.
MohammadrezaNasrabadi commented on issue #5196:
URL: https://github.com/apache/couchdb/issues/5196#issuecomment-2311786564
We don't apply any configuration about the firewall to have limitation or control over the connections.
Do you have any benchmark results based on the number of shards, replicas, amount of database requests and the available resources? It will be useful to compare our CouchDB infrastructure status with the ideal resource requirements.
I will share the detailed information of our infrastructure if needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Too Many Pending Connections [couchdb]
Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on issue #5196:
URL: https://github.com/apache/couchdb/issues/5196#issuecomment-2314124648
It's hard to compare architectures, it depends on the requirements. For instance some of our larger clusters have 30+ nodes and our load balancer handles 100k concurrent open connections. Even there, on the load balancer usually there almost no CLOSE_WAIT connections. At most I could see a short temporary increase in CLOSE_WAIT up to 3k out of 100k established connections, then they get cleared back to a few hundred only.
It's good that your resource usage is low it seems. Since you have other nodes to compare to, and others don't seem to show the same behavior try to see what's different between them (configuration, hardware resources). Maybe check your logs for any errors on some vs others?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Too Many Pending Connections [couchdb]
Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on issue #5196:
URL: https://github.com/apache/couchdb/issues/5196#issuecomment-2305363699
From what I remember from a while back, close-wait means the remote client closed the connection, then, the our (server) kernel found out about it, and is now waiting on the application code (CouchDB) to notice as well and close it.
So, perhaps maybe that one node is overwhelmed, sometimes is blocked? (maybe a firewall?), or not getting enough CPU time to run?. Check your resource usage (cpu, memory, open file handles) and the logs to see if there are exceptions or errors on that one node. At least you have a few other nodes to compare to where this doesn't happen.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Too Many Pending Connections [couchdb]
Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on issue #5196:
URL: https://github.com/apache/couchdb/issues/5196#issuecomment-2743944351
We have put fix to close connections proactively in CouchDB in https://github.com/apache/couchdb/pull/4736. That fix should be in the 3.4.x series. Thanks for your report @MohammadrezaNasrabadi.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Too Many Pending Connections [couchdb]
Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva closed issue #5196: Too Many Pending Connections
URL: https://github.com/apache/couchdb/issues/5196
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org