[GitHub] evsukov89 opened a new issue #1151: {db}/_changes?filter=_selector lags towards almost the end of the response

[GitBox <gi...@apache.org>]

evsukov89 opened a new issue #1151: {db}/_changes?filter=_selector lags towards almost the end of the response
URL: https://github.com/apache/couchdb/issues/1151
 
 
   While benchmarking the performance of mango selectors I stumbled upon an interesting behaviour: towards almost the end of the response, where there's 3 - 5 - 10 more changes expected to be received the response pauses from 0.5 sec to more, depending on the doc_count in the db, then finishes as normal.
   
   ## Expected Behavior
   No pause/lag in the _changes feed response
   
   ## Current Behavior
   Consistent one-time pause/lag towards the end of the response. The bigger the db ??the bigger the pause. 
   <!--- If describing a bug, tell us what happens instead of the expected behavior -->
   <!--- If suggesting a change/improvement, explain the difference from current behavior -->
   
   ## Possible Solution
   Cannot suggest much at the moment, still looking into the code.
   
   ## Steps to Reproduce (for bugs)
   <!--- Provide a link to a live example, or an unambiguous set of steps to -->
   <!--- reproduce this bug. Include code to reproduce, if relevant -->
   1. run the latest version from the official Docker container `docker run -p 5984:5984 -e COUCHDB_USER=admin -e COUCHDB_PASSWORD=password --name couchdb apache/couchdb`
   2. configure as a single node
   3. generate a db using the following script https://gist.github.com/evsukov89/0101dd1fcea728ae6c5807eef70a7820
   4. query the _changes feed `curl -s -u admin:password -H "Content-Type: application/json" -X POST 'http://127.0.0.1:5984/db-1000/_changes?filter=_selector' --data '{"selector":{"owner":"global"}}'`
   5. observe the lag towards the end of the response, see example video https://www.dropbox.com/s/jc99cnyytznbmt6/couchdb2_filter_selector_lag.mov?dl=0
   
   Change the `NUMGLOBAL` variable in the script to create more/less documents and experiment with different database sizes.
   
   ## Context
   <!--- How has this issue affected you? What are you trying to accomplish? -->
   <!--- Providing context helps us come up with a solution that is most useful in the real world -->
   I'm investigating mango selectors for replication as an alternative to view _change which appears not to be functional in 2.x (see #831). In my use-case, I operate over a multi-tenant environment (think one couch database ? one application instance/project/customer/etc.), a subset of documents, typically around 10% is being replicated by mobile clients. Having a performant way to get a filtered _changes feed is a must.
   
   While it's not really a bug nor a blocker in my case, the delay present 100% during my tests and is proportional to the number of docs in the db; I would simply like to understand the root cause and potential performance implications when having 100s of DBs on the same cluster with 10s of thousands of mobile clients replicating over with typical db size of 50-80k docs.
   
   ## Your Environment
   <!--- Include as many relevant details about the environment you experienced the bug in -->
   * Version used: 
   ```
   Docker version 17.12.0-ce, build c97c6d6
   ```
   ```
   {"couchdb":"Welcome","version":"2.1.1","features":["scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
   ```
   * Browser Name and version: 
   ```
   curl 7.54.0 (x86_64-apple-darwin17.0) libcurl/7.54.0 LibreSSL/2.0.20 zlib/1.2.11 nghttp2/1.24.0 
   Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp 
   Features: AsynchDNS IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz HTTP2 UnixSockets HTTPS-proxy
   ```
   * Operating System and version (desktop or mobile): 
   ```
   ProductName:	Mac OS X
   ProductVersion:	10.13.3
   BuildVersion:	17D47
   ```
   * Link to your project: n/a
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services