You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@couchdb.apache.org by nickva <gi...@git.apache.org> on 2016/03/04 06:03:05 UTC
[GitHub] couchdb-couch-replicator pull request: Adjust minimum number of ht...
GitHub user nickva opened a pull request:
https://github.com/apache/couchdb-couch-replicator/pull/29
Adjust minimum number of http connections to 2
Replication changes feed and main replicator process could
end up waiting on the http connection to be available, and also
waiting on each other in a gen_server call. So set minimum
number of http connections to 2 to avoid deadlock.
JIRA: COUCHDB-2959
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cloudant/couchdb-couch-replicator couchdb-2959
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/couchdb-couch-replicator/pull/29.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #29
----
commit a6921daa1c9d06497d9714ae72072fa13e71d344
Author: Nick Vatamaniuc <va...@gmail.com>
Date: 2016-03-04T04:38:20Z
Adjust minimum number of http connections to 2
Replication changes feed and main replicator process could
end up waiting on the http connection to be available, and also
waiting on each other in a gen_server call. So set minimum
number of http connections to 2 to avoid deadlock.
JIRA: COUCHDB-2959
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] couchdb-couch-replicator pull request: Adjust minimum number of ht...
Posted by kxepal <gi...@git.apache.org>.
Github user kxepal commented on the pull request:
https://github.com/apache/couchdb-couch-replicator/pull/29#issuecomment-192385737
@nickva
Yes, docs are wrong on this. At those time I didn't now exactly how things works. Now I'm not sure if it was good or not (:
Making source only bump makes a bit more sense.
@davisp
I was suspect this, but didn't get into the sources yet. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] couchdb-couch-replicator pull request: Adjust minimum number of ht...
Posted by kxepal <gi...@git.apache.org>.
Github user kxepal commented on the pull request:
https://github.com/apache/couchdb-couch-replicator/pull/29#issuecomment-192126471
That's a bad idea to fool user on their configuration. If I set `http_connections = 1` I know what I'm doing and I'm expect exact one connection. For instance, this is useful for debugging: turn off any concurrency to see how it goes.
So, either we need to forbid set httpd_connections to 1 or less (I feel that's quite strange) or figure out why 1 connection is not enough since it was fine for 1.x. Why replication needs at least two of them?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] couchdb-couch-replicator pull request: Adjust minimum number of ht...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/couchdb-couch-replicator/pull/29
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] couchdb-couch-replicator pull request: Adjust minimum number of ht...
Posted by davisp <gi...@git.apache.org>.
Github user davisp commented on the pull request:
https://github.com/apache/couchdb-couch-replicator/pull/29#issuecomment-192382324
The underlying cause here is that the replicator now uses the http connection pool for the changes feed on the source database. This was done to handle cases where there are many long lived replications. The changes feed reconnecting was causing significant CPU usage due to having to renegotiate TLS.
However, using a worker from the pool means that if the pool only has one worker, then everything trying to read from the database (get_pending_changes *or* reading documents) will have to wait until the changes feed to exit (ie, its dead locked). Alternatively, previously we *did* use more HTTP connections that requested, it was just that one of them wasn't part of the pool. So this isn't a change in behavior.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] couchdb-couch-replicator pull request: Adjust minimum number of ht...
Posted by nickva <gi...@git.apache.org>.
Github user nickva commented on the pull request:
https://github.com/apache/couchdb-couch-replicator/pull/29#issuecomment-192353044
@kxepal
`http_connections` is a bit misleading as it configures 3 separate things -- httpc connection pool for source, for target and also constrains the number of replication workers. Docs say configuration means "Maximum number of HTTP connections per replication" but user ends up with two of them -- one for target and one for source, even if they set it to 1
> So, either we need to forbid set http_connections to 1 ...
Because this configuration is overloaded, 1 is still valid for target http connection pool and to constrain the number of replication workers. Here we only adjust the size of the http connection pool for an http source. This is also logged so if they care user can find out why they have an extra connection.
> why 1 connection is not enough since it was fine for 1.x. Why replication needs at least two of them?
I don't see `get_pending_changes` in 1.6.x branch that might explain why:
https://github.com/apache/couchdb-couch-replicator/blob/master/src/couch_replicator.erl#L308
vs
https://github.com/apache/couchdb/blob/1.6.x/src/couch_replicator/src/couch_replicator.erl#L280
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] couchdb-couch-replicator pull request: Adjust minimum number of ht...
Posted by davisp <gi...@git.apache.org>.
Github user davisp commented on the pull request:
https://github.com/apache/couchdb-couch-replicator/pull/29#issuecomment-192500158
Also +1 that i forgot.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---