[GitHub] [couchdb] dsshane opened a new issue #2329: Add option to enforce fetching data from local shards, instead of from shards on on remote nodes(if data present on local node)

[GitBox <gi...@apache.org>]

dsshane opened a new issue #2329: Add option to enforce fetching data from local shards, instead of from shards on on remote nodes(if data present on local node)
URL: https://github.com/apache/couchdb/issues/2329
 
 
   Add option to enforce fetching data from local shards, instead of from shards/copies on on the remote nodes(if data present on local node), something like GET **/mydb/doc1?preferlocal=true**.
   
   ## Summary
   
   In the current software, it seems CouchDB will always try to distribute disk i/o across nodes during query.
   
   For example: in a two nodes cluster(MACHINE1.SOMECOMPANY.COM and MACHINE2.SOMECOMPANY.COM), database mydb was created with q=8 and n=2, as following. Each node contains a copy of all the data. **Documents in the example database contain large binary data(i.e, as attachments.**)
   ````
   GET /_membership
   {
       "all_nodes": [
           "couchdb@MACHINE1.SOMECOMPANY.COM",
           "couchdb@MACHINE2.SOMECOMPANY.COM"
       ],
       "cluster_nodes": [
           "couchdb@MACHINE1.SOMECOMPANY.COM",
           "couchdb@MACHINE2.SOMECOMPANY.COM"
       ]
   }
   
   
   GET /mydb
   {
       "db_name": "mydb",
       "update_seq": "167626-g1AAAAGbeJzLYWBg4MhgTmEQT84vTc5ISXIIcAn2MTLRcw0O8tRz9vfNASpgSmRIkv___39WEgND4B2CqpMUgGSSPUzDbEwNZmgaHEAa4mEazAhrSABpqIdqCHImqCGPBUgyNAApoJ75YE0TiNS0AKJpP9hpxGo6ANF0n8gAgGh6ANEE8ZN8FgBSO3e3",
       "sizes": {
           "file": 31189227296,
           "external": 30639090294,
           "active": 30491445284
       },
       "purge_seq": 0,
       "other": {
           "data_size": 30639090294
       },
       "doc_del_count": 0,
       "doc_count": 166294,
       "disk_size": 31189227296,
       "disk_format_version": 6,
       "data_size": 30491445284,
       "compact_running": false,
       "cluster": {
           "q": 8,
           "n": 2,
           "w": 2,
           "r": 2
       },
       "instance_start_time": "0"
   }
   
   GET /mydb/_shards
   {
       "shards": {
           "00000000-1fffffff": [
               "couchdb@MACHINE1.SOMECOMPANY.COM",
               "couchdb@MACHINE2.SOMECOMPANY.COM"
           ],
           "20000000-3fffffff": [
               "couchdb@MACHINE1.SOMECOMPANY.COM",
               "couchdb@MACHINE2.SOMECOMPANY.COM"
           ],
           "40000000-5fffffff": [
               "couchdb@MACHINE1.SOMECOMPANY.COM",
               "couchdb@MACHINE2.SOMECOMPANY.COM"
           ],
           "60000000-7fffffff": [
               "couchdb@MACHINE1.SOMECOMPANY.COM",
               "couchdb@MACHINE2.SOMECOMPANY.COM"
           ],
           "80000000-9fffffff": [
               "couchdb@MACHINE1.SOMECOMPANY.COM",
               "couchdb@MACHINE2.SOMECOMPANY.COM"
           ],
           "a0000000-bfffffff": [
               "couchdb@MACHINE1.SOMECOMPANY.COM",
               "couchdb@MACHINE2.SOMECOMPANY.COM"
           ],
           "c0000000-dfffffff": [
               "couchdb@MACHINE1.SOMECOMPANY.COM",
               "couchdb@MACHINE2.SOMECOMPANY.COM"
           ],
           "e0000000-ffffffff": [
               "couchdb@MACHINE1.SOMECOMPANY.COM",
               "couchdb@MACHINE2.SOMECOMPANY.COM"
           ]
       }
   }
   ````
   When querying/fetching data from mydb, it seems CouchDB may load some data from local shards, and some from shards on the remote machine. Due to the large document size( thus the heavy network traffic to to transfer data from the remote note the the coordinate node), the query speed is much slower than the standalone deployment. 
   
   **Query is 2+ times faster with the standalone deployment for the same testing data.**
   
   Adding ?r=1 did not help.
   
   ## Possible Solution
   
   It will be very helpful to add an option in the query request, something like **GET /mydb/id0?preferlocal=true**, to enforce fetching data from shards on local node if it presents locally; otherwise, fetching from remote shards.
   
   ## Additional context
   
   Software Tested: CouchDB 2.2
   OS: Linux.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services