Re: Shard level querying in CouchDB Proposal

[Mike Rhodes <mr...@linux.vnet.ibm.com>]

> On 8 Feb 2018, at 17:22, Geoffrey Cox <re...@gmail.com> wrote:
> 
> Thanks for the clarification, Mike. I figured that having multiple shard
> keys per doc would be a big change, but I was hoping I was wrong ;). I
> still think your proposed solution will add a lot of value to CouchDB.
> Unfortunately, it isn't going to be part of the silver bullet that makes it
> feasible to step away from db-per-user in many cases as in the end you need
> shards/partitions that correspond with your user's queries or else it will
> require visiting each shard when a query is issued, something that wouldn't
> be very scalable with a very large database with lots of nodes.
> 
> In my mind, either the database has to "replicate" the data to the right
> shards/partitions or you have to do this manually.

I think there's a misunderstanding somewhere here. In terms of query, it should totally allow moving away from db-per-user -- db-per-user is essentially the same as a shard key being the user name (with the caveat that the direct equivalent is that number of shards per user database is 1, which places limitations on the amount of data per user to the amount 1 shard can hold).

Mike.

Re: Shard level querying in CouchDB Proposal

[Geoffrey Cox <re...@gmail.com>]

I don’t think I have misunderstood the design. IMO, one of the most
difficult pieces of dealing with db-per-user is that you have to replicate
data between DBs. In the most simple example, if two users have access to
the same doc then that doc needs to be in both user’s DBs. The shard key
proposal is great, but it wouldn’t replace the need for this “replication”
as you’d still need say 2 docs in the same DB, each doc with a shard key
that corresponds to the user. In the real world, the use case gets even
more complicated so you may need even more types of “replications”

The shard key proposal should go a long way with certain designs, but I
don’t think it will help much with this “replication” complexity present
with db-per-user as these “replications” would still be needed in a
shard-key design.
On Mon, Apr 16, 2018 at 2:50 AM Mike Rhodes <mr...@linux.vnet.ibm.com>
wrote:

>
> > On 8 Feb 2018, at 17:22, Geoffrey Cox <re...@gmail.com> wrote:
> >
> > Thanks for the clarification, Mike. I figured that having multiple shard
> > keys per doc would be a big change, but I was hoping I was wrong ;). I
> > still think your proposed solution will add a lot of value to CouchDB.
> > Unfortunately, it isn't going to be part of the silver bullet that makes
> it
> > feasible to step away from db-per-user in many cases as in the end you
> need
> > shards/partitions that correspond with your user's queries or else it
> will
> > require visiting each shard when a query is issued, something that
> wouldn't
> > be very scalable with a very large database with lots of nodes.
> >
> > In my mind, either the database has to "replicate" the data to the right
> > shards/partitions or you have to do this manually.
>
> I think there's a misunderstanding somewhere here. In terms of query, it
> should totally allow moving away from db-per-user -- db-per-user is
> essentially the same as a shard key being the user name (with the caveat
> that the direct equivalent is that number of shards per user database is 1,
> which places limitations on the amount of data per user to the amount 1
> shard can hold).
>
> Mike.
>