[GitHub] [couchdb] tadeuszuherek opened a new issue #2154: Continuous synchronisation is hanging despite pending changes.

[GitBox <gi...@apache.org>]

tadeuszuherek opened a new issue #2154: Continuous synchronisation is hanging despite pending changes.
URL: https://github.com/apache/couchdb/issues/2154
 
 
   ## Description
   
   During the continuous synchronisation we're observing that synchronisation of actual documents gets stucked. Even if number of pending changes grows, actual documents are not transferred. It can be hanged for couple of minutes or several (if not dozens) hours until data is synchronised for a given database.
   
   Here is an example of such active task:
   
   ```
   curl localinstance.url:5984/_active_tasks | jq .
   
   [...]
   {
       "node": "nonode@nohost",
       "pid": "<0.29234.1394>",
       "changes_pending": 1355,
       "checkpoint_interval": 30000,
       "checkpointed_source_seq": "3165235-g1AAAAI7eJyV0EsOgjAQBuBGlIdGN8a1HqFA0bKSmygjNIQgXeher-AJ9BLuTPQmeg1XWFrCQmMCm5nkz-TLzGQIITPRIjTMec6jOMh5wnf7TMSdEMG0KIo00QDp-ngrMoOyDaNAvuf_CDATFZY10jtJxF_gue3HTZGgRFY1Yl4lEtluTCluiqxL5FAjg4lEXOI4lHkNkbwrKjqKJpyzgvqGhDDF4DCnFXRR0F1Bo7eEgLIwxmEr6KGgZ7WRpTaCiGCgraCXgqofWTcJeeD5BH5OSz9XAqs7",
       "continuous": true,
       "database": "shards/20000000-3fffffff/_replicator.1559734367",
       "doc_id": "events_central_to_remote",
       "doc_write_failures": 0,
       "docs_read": 9319,
       "docs_written": 9319,
       "missing_revisions_found": 9319,
       "replication_id": "d37b2ffbfd3c618125e5d8a5d44a6999+continuous",
       "revisions_checked": 3011221,
       "source": "http://centralinstance.url:80/cdb/events/",
       "source_seq": "3165235-g1AAAAI7eJyV0EsOgjAQBuBGlIdGN8a1HqFA0bKSmygjNIQgXeher-AJ9BLuTPQmeg1XWFrCQmMCm5nkz-TLzGQIITPRIjTMec6jOMh5wnf7TMSdEMG0KIo00QDp-ngrMoOyDaNAvuf_CDATFZY10jtJxF_gue3HTZGgRFY1Yl4lEtluTCluiqxL5FAjg4lEXOI4lHkNkbwrKjqKJpyzgvqGhDDF4DCnFXRR0F1Bo7eEgLIwxmEr6KGgZ7WRpTaCiGCgraCXgqofWTcJeeD5BH5OSz9XAqs7",
       "started_on": 1567018767,
       "target": "http://localinstance.url:5984/events/",
       "through_seq": "3165235-g1AAAAI7eJyV0EsOgjAQBuBGlIdGN8a1HqFA0bKSmygjNIQgXeher-AJ9BLuTPQmeg1XWFrCQmMCm5nkz-TLzGQIITPRIjTMec6jOMh5wnf7TMSdEMG0KIo00QDp-ngrMoOyDaNAvuf_CDATFZY10jtJxF_gue3HTZGgRFY1Yl4lEtluTCluiqxL5FAjg4lEXOI4lHkNkbwrKjqKJpyzgvqGhDDF4DCnFXRR0F1Bo7eEgLIwxmEr6KGgZ7WRpTaCiGCgraCXgqofWTcJeeD5BH5OSz9XAqs7",
       "type": "replication",
       "updated_on": 1567085097,
       "user": null
     },
   [...]
   ```
   some active tasks returns null (but in such cases we know that there are no changes) like:
   `"changes_pending": null,`
   There are no errors or warnings in CouchDB logs.
   
   ## Steps to Reproduce
   
   Not sure if it can be reproduced as it does not occur on test or staging instances (central instance + 2-3 remote ones).
   
   ## Expected Behaviour
   
   Replicator should synchronise data when there are pending changes and should not hang.
   
   ## Your Environment
   
   We're running one central instance and 14 remote instances. Each of remote instances is running continous replication for 13 databases (which creates a 26 replicators per instance).
   
   CouchDB is run as a Docker container based on apache/couchdb:2.3.1 image.
   
   Here is an output of http://centralinstance:5984:
   
   `{"couchdb":"Welcome","version":"2.3.1","git_sha":"c298091a4","uuid":"dab47a850777bb5df191177447125088","features":["pluggable-storage-engines","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}`
   
   Remote instances uses satellite connection to access internet - sometimes it can be slow and sometimes instances are offline. However case described above is for situations when internet is available and there are plenty of server resources (RAM, cpu, IO, net etc.).
   
   We're using this configuration file (on every instance):
   
   ```
   [httpd]
   server_options = [{buffer, 16384}]
   
   [replicator]
   retries_per_request = 20
   worker_processes = 10
   worker_batch_size = 200
   ```
   
   * CouchDB Version used: 2.3.1
   * Operating System and version: RHEL 7
   
   ## Additional context
   
   We have no attachments in our databases, only JSON documents.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services