[jira] [Commented] (COUCHDB-2791) Allow for direct parallel access to shards via _changes

["Paul Joseph Davis (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/COUCHDB-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15373845#comment-15373845 ] 

Paul Joseph Davis commented on COUCHDB-2791:
--------------------------------------------

I'm taking another look at this and contemplating a few different approaches. Originally when we were kicking this idea around the idea was purely motivated by trying to make the changes feed faster by allowing clients to stream from individual shards. However [~kxepal] makes a good point that it probably makes better sense to turn this into a more generic feature allowing access to individual shards.

One thing I wanted to put out explicitly is that the idea for this is that it would be available to users over the clustered 5984 port if they want to do fancy advanced stuff client side. Ie, this isn't something for the 5986 port (and will try and avoid using 5986 things since we're looking to get rid of that anyway).

Also, as I think about this I think it'd be bad to allow write/modification APIs across this new shard specific interface as that seems like it'd be a super easy way to mess up a clustered database by getting docs in the wrong shard and/or getting shards desynchronized with other settings. So for the time being at least I'm going to limit this to read-only APIs which will basically be fetching shard db info, individual docs, all docs, views, and changes off the top of my head. Beyond that I think I can make this happen as a change to chttpd plus some additional support code to fabric for the new local operations. The end result API I'm looking at will be something like this:

http://hostname:5984/dbname/_shard/00000000-ffffffff/$rest

Where $rest is any supported API call that will match the same operations in the cluster case.

To implement this i'm planning on adding a new field to the #httpd record that selects the fabric module to use. By default this will be set to fabric which is the current default. I'll then add a fabric_local (or something, if anyone wants to suggest a better name) that will support just the set of things we want to export over this interface. This will then be fairly similar to fabric_rpc internally but without going through RPC/rexi calls and the like. Once that's done then we should hopefully be good to go for making everything work all magically.

That seem sane to everyone?

> Allow for direct parallel access to shards via _changes
> -------------------------------------------------------
>
>                 Key: COUCHDB-2791
>                 URL: https://issues.apache.org/jira/browse/COUCHDB-2791
>             Project: CouchDB
>          Issue Type: New Feature
>          Components: Database Core
>            Reporter: Tony Sun
>            Assignee: Tony Sun
>
> For performance gains, we introduce a new _changes feed option parallel that returns a list of urls that the user can use to directly access individual shards. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)