You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@couchdb.apache.org by GitBox <gi...@apache.org> on 2021/06/01 21:26:33 UTC
[GitHub] [couchdb] byrneml opened a new issue #3594: Replication extremely slow with millions of changes pending
byrneml opened a new issue #3594:
URL: https://github.com/apache/couchdb/issues/3594
[NOTE]: # ( ^^ Provide a general summary of the issue in the title above. ^^ )
## Description
Continuous replication of https://skimdb.npmjs.com/registry gets extremely slow over time - it worked fine for up to 400k documents, then became extremely slow. The entire dataset contains 1.6M documents.
**Here is the active task**
```
[
{
"node": "couchdb@localhost",
"pid": "<0.1300.89>",
"changes_pending": 13442245,
"checkpoint_interval": 30000,
"checkpointed_source_seq": 611203,
"continuous": true,
"database": "shards/80000000-ffffffff/_replicator.1622573219",
"doc_id": "ad71dbf4189be812bfb41c30d6000da6",
"doc_write_failures": 1,
"docs_read": 533605,
"docs_written": 533604,
"missing_revisions_found": 533605,
"replication_id": "a5df5f8f23b0d94f28a1ed02d4a2f103+continuous",
"revisions_checked": 540275,
"source": "https://skimdb.npmjs.com/registry/",
"source_seq": 612872,
"started_on": 1622581539,
"target": "http://localhost:5984/npm_mirror/",
"through_seq": 612872,
"type": "replication",
"updated_on": 1622581842,
"user": "admin"
},
{
"node": "couchdb@localhost",
"pid": "<0.22157.91>",
"changes_done": 6107,
"database": "shards/00000000-7fffffff/npm_mirror.1622573154",
"phase": "document_copy",
"progress": 2,
"started_on": 1622581839,
"total_changes": 268814,
"type": "database_compaction",
"updated_on": 1622581841
}
]
```
## Steps to Reproduce
Using the Couchbase GUI:
1. Navigate to replication
2. Set the remote source as https://skimdb.npmjs.com/registry
3. Set the remote as a new local database
## Expected Behaviour
For the replication to run more quickly than it does (3+ hours to get through ~30% of the replication).
## Your Environment
We're running 1 central instance locally for now.
```
{
"couchdb": "Welcome",
"version": "3.1.1",
"git_sha": "ce596c65d",
"uuid": "5e70eea19ba38ca970bd9a8020677dbf",
"features": [
"access-ready",
"partitioned",
"pluggable-storage-engines",
"reshard",
"scheduler"
],
"vendor": {
"name": "The Apache Software Foundation"
}
}
```
[TIP]: # ( You can paste the output of curl http://YOUR-COUCHDB:5984/ here. )
* CouchDB version used:
* Browser name and version:
* Operating system and version:
## Additional Context
We have no attachments, just JSON documents.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] byrneml commented on issue #3594: Replication extremely slow with millions of changes pending
Posted by GitBox <gi...@apache.org>.
byrneml commented on issue #3594:
URL: https://github.com/apache/couchdb/issues/3594#issuecomment-852459407
Issue resolved when I set retries_per_request from 20 to 5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [couchdb] byrneml closed issue #3594: Replication extremely slow with millions of changes pending
Posted by GitBox <gi...@apache.org>.
byrneml closed issue #3594:
URL: https://github.com/apache/couchdb/issues/3594
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org