You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Markus Doppelbauer <do...@gmx.net.INVALID> on 2024/06/12 16:59:52 UTC
View performance
Hello,
Is the CouchDB-view a "global" or "local" index?
For example, if a cluster has 100 nodes, would the query ask
for a single node - or 100 nodes?
/.../_view/posts?startkey="foobar"&endkey="foobaz"
Best wishes
Marcus
Re: View performance
Posted by Markus Doppelbauer <do...@gmx.net.INVALID>.
Hi Nick,
Thank you for your clarification.
Best wishesMarcus
Am Mittwoch, dem 12.06.2024 um 16:47 -0400 schrieb Nick Vatamaniuc:
> On Wed, Jun 12, 2024 at 3:54 PM Markus Doppelbauer<
> doppelbauer@gmx.net.invalid> wrote:
> > Dear Nick,
> > This means:If Q+N are large enough to distribute the data to
> > allnodes, e.g. 120 nodes Q=40, N=3Then the view-query:
> > startkey=foobar&endkey=foobazhas to ask all 120 nodes?
>
> In the initial startup phase it could query all 120 nodes, then it
> willpick 40 shard workers and will stream from those only.
> You do have some control where each of the 3 copies of shards end up:
> https://docs.couchdb.org/en/stable/cluster/databases.html#placing-a-database-on-specific-nodes
>
>
> > If I use a partion-key, is it possible to distribute the partioned-
> > viewamong multiple nodes?The docs say it should stay under 10GB.
>
> A partition cannot be split across multiple shard ranges. A single
> shardrange can contain multiple partitions. So, for a given partition
> key, itwould pick a particular shard range from Q, and then it would
> pick one ofthe N copies (usually 3) to stream from.
> Cheers,
> > Markus
> >
> >
> > Am Mittwoch, dem 12.06.2024 um 13:54 -0400 schrieb Nick Vatamaniuc:
> > > Another feature related to efficient view querying
> > > arepartitioneddatabases:
> > > https://docs.couchdb.org/en/stable/partitioned-dbs/index.html.It's
> > > abit of a niche, as you'd need to have a good partition key,
> > > butasidefrom that, it can speed up your queries as responses
> > > would becomingfrom a single shard only instead of Q shards.
> > >
> > > On Wed, Jun 12, 2024 at 1:30 PM Markus Doppelbauer<
> > > doppelbauer@gmx.net.invalid> wrote:
> > > > Hi Nick,Thank you very much for your reply.This is exactly what
> > > > weare lookingfor.There are so many DBs that store the
> > > > secondaryindexlocally(Cassandra, Aerospike, SyllaDB, ...)Thanks
> > > > again forthe answerMarcus
> > > > Am Mittwoch, dem 12.06.2024 um 13:23 -0400 schrieb Nick
> > > > Vatamaniuc:
> > > > > Hi Marcus,The node handling the request only queries the
> > > > > nodeswith shardcopies ofthat database. In a 100 node cluster
> > > > > theshards for thatparticulardatabase might be present on only
> > > > > 6nodes, depending on theQ and Nsharding factors, so it will
> > > > > query6 out 100 nodes. Forinstance, for N=3and Q=2 sharding
> > > > > factors, itwill first send N*Q=6requests, and wait untilit
> > > > > gets at least oneresponse for each of theQ=2 shard ranges.
> > > > > Thishappens veryquickly. Then, for the duration ofthe
> > > > > response, it willonlystream responses from those Q=2
> > > > > workers.So, to summarize fora Q=2database, it will be a
> > > > > streaming responsefrom 2 workers. ForQ=4, from 4workers,
> > > > > etc...Cheers,-NickOn Wed, Jun 12, 2024 at 1:00 PM Markus
> > > > > Doppelbauer<do...@gmx.net.invalid> wrote:
> > > > > > Hello,Is the CouchDB-view a "global" or "local"
> > > > > > index?Forexample, if acluster has 100 nodes, would the
> > > > > > query askfor asingle node -
> > > > > > or100nodes?/.../_view/posts?startkey="foobar"&endkey="fooba
> > > > > > z"BestwishesMarcus
Re: View performance
Posted by Jan Lehnardt <ja...@apache.org>.
Patches welcome :)
—
Professional Support for Apache CouchDB:
https://neighbourhood.ie/couchdb-support/
24/7 Observation for your CouchDB Instances:
https://opservatory.app
SQL Queries for CouchDB:
https://neighbourhood.ie/products-and-services/structured-query-server
> On 14. Jun 2024, at 13:59, Markus Doppelbauer <do...@gmx.net.INVALID> wrote:
>
> Perhaps it wouldn't be too hard to make CouchDB scalable:
>
> a) When creating a view: add an option to a "key-range map":
> Maybe via callback(): "byte-range -> shard"
> Something like: from "aa" to "bb" -> node-0; and so on
>
> b) During "indexing": Store the emitted view-index on the shard
> according to the "range-map"
> Similar to a partition key, but scalable
>
> c) During "query": e.g.: startkey=fooAAA & endkey=fooBBB & limit=10
> Map-reduce to query "fooAAA" from shard-8 and "fooBBB" from shard-9
>
> CouchDB scales out to 10000+ nodes.
> The query only asks 1, maybe 2 shards
>
>
>
> Am Mittwoch, dem 12.06.2024 um 16:47 -0400 schrieb Nick Vatamaniuc:
>> On Wed, Jun 12, 2024 at 3:54 PM Markus Doppelbauer
>> <
>> doppelbauer@gmx.net.invalid
>>> wrote:
>>
>>> Dear Nick,
>>>
>>> This means:
>>> If Q+N are large enough to distribute the data to all
>>> nodes, e.g. 120 nodes Q=40, N=3
>>> Then the view-query: startkey=foobar&endkey=foobaz
>>> has to ask all 120 nodes?
>>>
>>
>> In the initial startup phase it could query all 120 nodes, then it
>> will
>> pick 40 shard workers and will stream from those only.
>>
>> You do have some control where each of the 3 copies of shards end up:
>> https://docs.couchdb.org/en/stable/cluster/databases.html#placing-a-database-on-specific-nodes
>>
>>
>>
>>> If I use a partion-key, is it possible to distribute the partioned-
>>> view
>>> among multiple nodes?
>>> The docs say it should stay under 10GB.
>>>
>>
>> A partition cannot be split across multiple shard ranges. A single
>> shard
>> range can contain multiple partitions. So, for a given partition key,
>> it
>> would pick a particular shard range from Q, and then it would pick
>> one of
>> the N copies (usually 3) to stream from.
>>
>> Cheers,
>>> Markus
>>>
>>>
>>>
>>> Am Mittwoch, dem 12.06.2024 um 13:54 -0400 schrieb Nick Vatamaniuc:
>>>> Another feature related to efficient view querying are
>>>> partitioneddatabases:
>>>> https://docs.couchdb.org/en/stable/partitioned-dbs/index.html.It's
>>>> a
>>>> bit of a niche, as you'd need to have a good partition key,
>>>> butaside
>>>> from that, it can speed up your queries as responses would be
>>>> comingfrom a single shard only instead of Q shards.
>>>>
>>>>
>>>> On Wed, Jun 12, 2024 at 1:30 PM Markus Doppelbauer<
>>>> doppelbauer@gmx.net.invalid
>>>>> wrote:
>>>>> Hi Nick,Thank you very much for your reply.This is exactly what
>>>>> we
>>>>> are lookingfor.There are so many DBs that store the secondary
>>>>> indexlocally(Cassandra, Aerospike, SyllaDB, ...)Thanks again
>>>>> for
>>>>> the answerMarcus
>>>>>
>>>>> Am Mittwoch, dem 12.06.2024 um 13:23 -0400 schrieb Nick
>>>>> Vatamaniuc:
>>>>>> Hi Marcus,The node handling the request only queries the
>>>>>> nodes
>>>>>> with shardcopies ofthat database. In a 100 node cluster the
>>>>>> shards for thatparticulardatabase might be present on only 6
>>>>>> nodes, depending on theQ and Nsharding factors, so it will
>>>>>> query
>>>>>> 6 out 100 nodes. Forinstance, for N=3and Q=2 sharding
>>>>>> factors, it
>>>>>> will first send N*Q=6requests, and wait untilit gets at least
>>>>>> one
>>>>>> response for each of theQ=2 shard ranges. Thishappens very
>>>>>> quickly. Then, for the duration ofthe response, it will
>>>>>> onlystream responses from those Q=2 workers.So, to summarize
>>>>>> for
>>>>>> a Q=2database, it will be a streaming responsefrom 2 workers.
>>>>>> For
>>>>>> Q=4, from 4workers, etc...Cheers,-Nick
>>>>>> On Wed, Jun 12, 2024 at 1:00 PM Markus Doppelbauer<
>>>>>> doppelbauer@gmx.net.invalid
>>>>>>> wrote:
>>>>>>> Hello,Is the CouchDB-view a "global" or "local" index?For
>>>>>>> example, if acluster has 100 nodes, would the query askfor
>>>>>>> a
>>>>>>> single node - or100
>>>>>>> nodes?/.../_view/posts?startkey="foobar"&endkey="foobaz"Bes
>>>>>>> t
>>>>>>> wishesMarcus
>
Re: View performance
Posted by Markus Doppelbauer <do...@gmx.net.INVALID>.
Perhaps it wouldn't be too hard to make CouchDB scalable:
a) When creating a view: add an option to a "key-range map":
Maybe via callback(): "byte-range -> shard"
Something like: from "aa" to "bb" -> node-0; and so on
b) During "indexing": Store the emitted view-index on the shard
according to the "range-map"
Similar to a partition key, but scalable
c) During "query": e.g.: startkey=fooAAA & endkey=fooBBB & limit=10
Map-reduce to query "fooAAA" from shard-8 and "fooBBB" from shard-9
CouchDB scales out to 10000+ nodes.
The query only asks 1, maybe 2 shards
Am Mittwoch, dem 12.06.2024 um 16:47 -0400 schrieb Nick Vatamaniuc:
> On Wed, Jun 12, 2024 at 3:54 PM Markus Doppelbauer
> <
> doppelbauer@gmx.net.invalid
> > wrote:
>
> > Dear Nick,
> >
> > This means:
> > If Q+N are large enough to distribute the data to all
> > nodes, e.g. 120 nodes Q=40, N=3
> > Then the view-query: startkey=foobar&endkey=foobaz
> > has to ask all 120 nodes?
> >
>
> In the initial startup phase it could query all 120 nodes, then it
> will
> pick 40 shard workers and will stream from those only.
>
> You do have some control where each of the 3 copies of shards end up:
> https://docs.couchdb.org/en/stable/cluster/databases.html#placing-a-database-on-specific-nodes
>
>
>
> > If I use a partion-key, is it possible to distribute the partioned-
> > view
> > among multiple nodes?
> > The docs say it should stay under 10GB.
> >
>
> A partition cannot be split across multiple shard ranges. A single
> shard
> range can contain multiple partitions. So, for a given partition key,
> it
> would pick a particular shard range from Q, and then it would pick
> one of
> the N copies (usually 3) to stream from.
>
> Cheers,
> > Markus
> >
> >
> >
> > Am Mittwoch, dem 12.06.2024 um 13:54 -0400 schrieb Nick Vatamaniuc:
> > > Another feature related to efficient view querying are
> > > partitioneddatabases:
> > > https://docs.couchdb.org/en/stable/partitioned-dbs/index.html.It's
> > > a
> > > bit of a niche, as you'd need to have a good partition key,
> > > butaside
> > > from that, it can speed up your queries as responses would be
> > > comingfrom a single shard only instead of Q shards.
> > >
> > >
> > > On Wed, Jun 12, 2024 at 1:30 PM Markus Doppelbauer<
> > > doppelbauer@gmx.net.invalid
> > > > wrote:
> > > > Hi Nick,Thank you very much for your reply.This is exactly what
> > > > we
> > > > are lookingfor.There are so many DBs that store the secondary
> > > > indexlocally(Cassandra, Aerospike, SyllaDB, ...)Thanks again
> > > > for
> > > > the answerMarcus
> > > >
> > > > Am Mittwoch, dem 12.06.2024 um 13:23 -0400 schrieb Nick
> > > > Vatamaniuc:
> > > > > Hi Marcus,The node handling the request only queries the
> > > > > nodes
> > > > > with shardcopies ofthat database. In a 100 node cluster the
> > > > > shards for thatparticulardatabase might be present on only 6
> > > > > nodes, depending on theQ and Nsharding factors, so it will
> > > > > query
> > > > > 6 out 100 nodes. Forinstance, for N=3and Q=2 sharding
> > > > > factors, it
> > > > > will first send N*Q=6requests, and wait untilit gets at least
> > > > > one
> > > > > response for each of theQ=2 shard ranges. Thishappens very
> > > > > quickly. Then, for the duration ofthe response, it will
> > > > > onlystream responses from those Q=2 workers.So, to summarize
> > > > > for
> > > > > a Q=2database, it will be a streaming responsefrom 2 workers.
> > > > > For
> > > > > Q=4, from 4workers, etc...Cheers,-Nick
> > > > > On Wed, Jun 12, 2024 at 1:00 PM Markus Doppelbauer<
> > > > > doppelbauer@gmx.net.invalid
> > > > > > wrote:
> > > > > > Hello,Is the CouchDB-view a "global" or "local" index?For
> > > > > > example, if acluster has 100 nodes, would the query askfor
> > > > > > a
> > > > > > single node - or100
> > > > > > nodes?/.../_view/posts?startkey="foobar"&endkey="foobaz"Bes
> > > > > > t
> > > > > > wishesMarcus
Re: View performance
Posted by Nick Vatamaniuc <va...@gmail.com>.
On Wed, Jun 12, 2024 at 3:54 PM Markus Doppelbauer
<do...@gmx.net.invalid> wrote:
> Dear Nick,
>
> This means:
> If Q+N are large enough to distribute the data to all
> nodes, e.g. 120 nodes Q=40, N=3
> Then the view-query: startkey=foobar&endkey=foobaz
> has to ask all 120 nodes?
>
In the initial startup phase it could query all 120 nodes, then it will
pick 40 shard workers and will stream from those only.
You do have some control where each of the 3 copies of shards end up:
https://docs.couchdb.org/en/stable/cluster/databases.html#placing-a-database-on-specific-nodes
> If I use a partion-key, is it possible to distribute the partioned-view
> among multiple nodes?
> The docs say it should stay under 10GB.
>
A partition cannot be split across multiple shard ranges. A single shard
range can contain multiple partitions. So, for a given partition key, it
would pick a particular shard range from Q, and then it would pick one of
the N copies (usually 3) to stream from.
Cheers,
> Markus
>
>
>
> Am Mittwoch, dem 12.06.2024 um 13:54 -0400 schrieb Nick Vatamaniuc:
> > Another feature related to efficient view querying are
> > partitioneddatabases:
> > https://docs.couchdb.org/en/stable/partitioned-dbs/index.html.It's a
> > bit of a niche, as you'd need to have a good partition key, butaside
> > from that, it can speed up your queries as responses would be
> > comingfrom a single shard only instead of Q shards.
> >
> >
> > On Wed, Jun 12, 2024 at 1:30 PM Markus Doppelbauer<
> > doppelbauer@gmx.net.invalid> wrote:
> > > Hi Nick,Thank you very much for your reply.This is exactly what we
> > > are lookingfor.There are so many DBs that store the secondary
> > > indexlocally(Cassandra, Aerospike, SyllaDB, ...)Thanks again for
> > > the answerMarcus
> > >
> > > Am Mittwoch, dem 12.06.2024 um 13:23 -0400 schrieb Nick Vatamaniuc:
> > > > Hi Marcus,The node handling the request only queries the nodes
> > > > with shardcopies ofthat database. In a 100 node cluster the
> > > > shards for thatparticulardatabase might be present on only 6
> > > > nodes, depending on theQ and Nsharding factors, so it will query
> > > > 6 out 100 nodes. Forinstance, for N=3and Q=2 sharding factors, it
> > > > will first send N*Q=6requests, and wait untilit gets at least one
> > > > response for each of theQ=2 shard ranges. Thishappens very
> > > > quickly. Then, for the duration ofthe response, it will
> > > > onlystream responses from those Q=2 workers.So, to summarize for
> > > > a Q=2database, it will be a streaming responsefrom 2 workers. For
> > > > Q=4, from 4workers, etc...Cheers,-Nick
> > > > On Wed, Jun 12, 2024 at 1:00 PM Markus Doppelbauer<
> > > > doppelbauer@gmx.net.invalid> wrote:
> > > > > Hello,Is the CouchDB-view a "global" or "local" index?For
> > > > > example, if acluster has 100 nodes, would the query askfor a
> > > > > single node - or100
> > > > > nodes?/.../_view/posts?startkey="foobar"&endkey="foobaz"Best
> > > > > wishesMarcus
>
Re: View performance
Posted by Markus Doppelbauer <do...@gmx.net.INVALID>.
Dear Nick,
This means:
If Q+N are large enough to distribute the data to all
nodes, e.g. 120 nodes Q=40, N=3
Then the view-query: startkey=foobar&endkey=foobaz
has to ask all 120 nodes?
If I use a partion-key, is it possible to distribute the partioned-view
among multiple nodes?
The docs say it should stay under 10GB.
Cheers,
Markus
Am Mittwoch, dem 12.06.2024 um 13:54 -0400 schrieb Nick Vatamaniuc:
> Another feature related to efficient view querying are
> partitioneddatabases:
> https://docs.couchdb.org/en/stable/partitioned-dbs/index.html.It's a
> bit of a niche, as you'd need to have a good partition key, butaside
> from that, it can speed up your queries as responses would be
> comingfrom a single shard only instead of Q shards.
>
>
> On Wed, Jun 12, 2024 at 1:30 PM Markus Doppelbauer<
> doppelbauer@gmx.net.invalid> wrote:
> > Hi Nick,Thank you very much for your reply.This is exactly what we
> > are lookingfor.There are so many DBs that store the secondary
> > indexlocally(Cassandra, Aerospike, SyllaDB, ...)Thanks again for
> > the answerMarcus
> >
> > Am Mittwoch, dem 12.06.2024 um 13:23 -0400 schrieb Nick Vatamaniuc:
> > > Hi Marcus,The node handling the request only queries the nodes
> > > with shardcopies ofthat database. In a 100 node cluster the
> > > shards for thatparticulardatabase might be present on only 6
> > > nodes, depending on theQ and Nsharding factors, so it will query
> > > 6 out 100 nodes. Forinstance, for N=3and Q=2 sharding factors, it
> > > will first send N*Q=6requests, and wait untilit gets at least one
> > > response for each of theQ=2 shard ranges. Thishappens very
> > > quickly. Then, for the duration ofthe response, it will
> > > onlystream responses from those Q=2 workers.So, to summarize for
> > > a Q=2database, it will be a streaming responsefrom 2 workers. For
> > > Q=4, from 4workers, etc...Cheers,-Nick
> > > On Wed, Jun 12, 2024 at 1:00 PM Markus Doppelbauer<
> > > doppelbauer@gmx.net.invalid> wrote:
> > > > Hello,Is the CouchDB-view a "global" or "local" index?For
> > > > example, if acluster has 100 nodes, would the query askfor a
> > > > single node - or100
> > > > nodes?/.../_view/posts?startkey="foobar"&endkey="foobaz"Best
> > > > wishesMarcus
Re: View performance
Posted by Markus Doppelbauer <do...@gmx.net.INVALID>.
It is a pity that a fast geo-replicated database does not exist.
Plague or cholera:
1) Distributed sorted heap-tables with local-machine indexing:
Couchbase, MongoDB, ...
2) Distributed hash-tables with local-machine indexing:
Cassandra, ScyllaDB, Riak, Aerospike, ...
Cons: Not scalable: all nodes need to be queried
3) Distribued sorted heap with global consistency:
Spanner, CockroachDB, FoundationDB, TiKV, etcd, Yugabyte
Cons: Scalability, the more DCs the more write amplification (easily
500ms)
4) Fast AP-databases:
Google-Bigtable, DynamoDB, HBase, Hypertable (dead+flaws)
Cons: Proprietary - Hbase-API sucks if you're not JVM
Regards,
Marcus
Am Donnerstag, dem 13.06.2024 um 13:40 +0100 schrieb Robert Newson:
> Hi,
> Right. a database is divided into Q shards, each of which covers part
> of a numeric range from 0 to 2^32-1, the doc id is run through a
> hashing function that outputs a number in that range, and the
> document is then stored inside the shard whose range contains that
> number.
> as this effectively randomises which doc goes in which shard when we
> need to perform a query across the database we need to ask each shard
> range to perform the query and then combine the results.
> This is good in some ways (the shards can be stored across multiple
> machines and thus benefit from more cores/disks/memory/etc) it's bad
> in others, the number of workers that need to respond goes up as Q
> goes up.
> "Partition" is just our word (and arguably a misleading word) for
> allowing users to control which shard range their documents end up
> in. Only part of the doc id is passed through the hashing
> function(doc id "foo:bar" would be partition foo, and we'd hash only
> "foo"). Thus the user can ensure that a group of documents lands in
> the same shard range. The benefit is as Nick described, when querying
> _within_ a given partition we know that only one shard range is in
> play, so we do not consult shards for other ranges (we already know
> they will have no results).
> So, to your specific list of questions;
> - yes- yes, but the reason to keep partitions small is because of the
> fixed assignment to a given shard range. the default scheme would
> load up each shard range fairly evenly. it's about disk space
> management and predictable performance.- when querying a global view
> we need Q responses (and will issue Q*N requests, but cancel Q*(N-1)
> of those when the first response from each range has been received,
> though they may have already done some, or all, of their work, before
> that happens). for a partitioned view we need 1 response (but will
> issue N requests).- quorum applies to document operations (read,
> write, delete), but not view queries (R is inherently 1)
> B.
> > On 13 Jun 2024, at 09:01, Matthieu Rakotojaona <
> > matthieu.rakotojaona-rainimangavelo@imt-atlantique.fr> wrote:
> > Hello, I'm still a bit confused with shards versus partitions, is
> > this correct ?
> > - A database is composed of Q shards, shards splice the database
> > "automatically"
> > - A shard is composed of 1 or more partitions, partitions splice
> > the database according to the user's choice. A partition should
> > thus be made so that it is not huge, in fact must be under (total
> > size/Q)
> > - When querying a view, the whole database needs to be processed,
> > that means all Q shards
> > - When querying a view only on documents inside a partition, since
> > one partition is on 1 shard, only one shard is queried
> > - Whatever the number of shards, quorum still needs to be verified,
> > ie for Q=2 and N=3 I still need to wait for (3+1)/2 = 2 replies for
> > each shard
> > On 12/06/2024 19:54, Nick Vatamaniuc wrote:
> > > Another feature related to efficient view querying are
> > > partitioneddatabases:
> > > https://docs.couchdb.org/en/stable/partitioned-dbs/index.html.It's
> > > a bit of a niche, as you'd need to have a good partition key,
> > > butaside from that, it can speed up your queries as responses
> > > would be comingfrom a single shard only instead of Q shards.
> > >
> > >
> > > On Wed, Jun 12, 2024 at 1:30 PM Markus Doppelbauer<
> > > doppelbauer@gmx.net.invalid> wrote:
> > > > Hi Nick,Thank you very much for your reply.This is exactly what
> > > > we are lookingfor.There are so many DBs that store the
> > > > secondary indexlocally(Cassandra, Aerospike, SyllaDB,
> > > > ...)Thanks again for the answerMarcus
> > > >
> > > > Am Mittwoch, dem 12.06.2024 um 13:23 -0400 schrieb Nick
> > > > Vatamaniuc:
> > > > > Hi Marcus,The node handling the request only queries the
> > > > > nodes with shardcopies ofthat database. In a 100 node cluster
> > > > > the shards for thatparticulardatabase might be present on
> > > > > only 6 nodes, depending on theQ and Nsharding factors, so it
> > > > > will query 6 out 100 nodes. Forinstance, for N=3and Q=2
> > > > > sharding factors, it will first send N*Q=6requests, and wait
> > > > > untilit gets at least one response for each of theQ=2 shard
> > > > > ranges. Thishappens very quickly. Then, for the duration
> > > > > ofthe response, it will onlystream responses from those Q=2
> > > > > workers.So, to summarize for a Q=2database, it will be a
> > > > > streaming responsefrom 2 workers. For Q=4, from 4workers,
> > > > > etc...Cheers,-Nick
> > > > > On Wed, Jun 12, 2024 at 1:00 PM Markus Doppelbauer<
> > > > > doppelbauer@gmx.net.invalid> wrote:
> > > > > > Hello,Is the CouchDB-view a "global" or "local" index?For
> > > > > > example, if acluster has 100 nodes, would the query askfor
> > > > > > a single node - or100
> > > > > > nodes?/.../_view/posts?startkey="foobar"&endkey="foobaz"Bes
> > > > > > t wishesMarcus
>
>
Re: View performance
Posted by Markus Doppelbauer <do...@gmx.net.INVALID>.
It is a pity that a fast geo-replicated database does not exist.Plague
or cholera:
1) Distributed sorted heap-tables with local-machine
indexing: Couchbase, MongoDB, ...2) Distributed hash-tables with
local-machine indexing: Cassandra, ScyllaDB, Riak, Aerospike,
...Cons: Not scalable: all nodes need to be queried
3) Distribued sorted heap with global consistency: Spanner,
CockroachDB, FoundationDB, TiKV, etcd, YugabyteCons: Scalability, the
more DCs the more write amplification (easily 500ms)
4) Fast AP-databases: Google-Bigtable, DynamoDB, HBase, Hypertable
(dead+flaws)Cons: Proprietary - Hbase-API sucks if you're not JVM
Regards,Marcus
Am Donnerstag, dem 13.06.2024 um 13:40 +0100 schrieb Robert Newson:
> Hi,
> Right. a database is divided into Q shards, each of which covers part
> of a numeric range from 0 to 2^32-1, the doc id is run through a
> hashing function that outputs a number in that range, and the
> document is then stored inside the shard whose range contains that
> number.
> as this effectively randomises which doc goes in which shard when we
> need to perform a query across the database we need to ask each shard
> range to perform the query and then combine the results.
> This is good in some ways (the shards can be stored across multiple
> machines and thus benefit from more cores/disks/memory/etc) it's bad
> in others, the number of workers that need to respond goes up as Q
> goes up.
> "Partition" is just our word (and arguably a misleading word) for
> allowing users to control which shard range their documents end up
> in. Only part of the doc id is passed through the hashing
> function(doc id "foo:bar" would be partition foo, and we'd hash only
> "foo"). Thus the user can ensure that a group of documents lands in
> the same shard range. The benefit is as Nick described, when querying
> _within_ a given partition we know that only one shard range is in
> play, so we do not consult shards for other ranges (we already know
> they will have no results).
> So, to your specific list of questions;
> - yes- yes, but the reason to keep partitions small is because of the
> fixed assignment to a given shard range. the default scheme would
> load up each shard range fairly evenly. it's about disk space
> management and predictable performance.- when querying a global view
> we need Q responses (and will issue Q*N requests, but cancel Q*(N-1)
> of those when the first response from each range has been received,
> though they may have already done some, or all, of their work, before
> that happens). for a partitioned view we need 1 response (but will
> issue N requests).- quorum applies to document operations (read,
> write, delete), but not view queries (R is inherently 1)
> B.
> > On 13 Jun 2024, at 09:01, Matthieu Rakotojaona <
> > matthieu.rakotojaona-rainimangavelo@imt-atlantique.fr> wrote:
> > Hello, I'm still a bit confused with shards versus partitions, is
> > this correct ?
> > - A database is composed of Q shards, shards splice the database
> > "automatically"
> > - A shard is composed of 1 or more partitions, partitions splice
> > the database according to the user's choice. A partition should
> > thus be made so that it is not huge, in fact must be under (total
> > size/Q)
> > - When querying a view, the whole database needs to be processed,
> > that means all Q shards
> > - When querying a view only on documents inside a partition, since
> > one partition is on 1 shard, only one shard is queried
> > - Whatever the number of shards, quorum still needs to be verified,
> > ie for Q=2 and N=3 I still need to wait for (3+1)/2 = 2 replies for
> > each shard
> > On 12/06/2024 19:54, Nick Vatamaniuc wrote:
> > > Another feature related to efficient view querying are
> > > partitioneddatabases:
> > > https://docs.couchdb.org/en/stable/partitioned-dbs/index.html.It's
> > > a bit of a niche, as you'd need to have a good partition key,
> > > butaside from that, it can speed up your queries as responses
> > > would be comingfrom a single shard only instead of Q shards.
> > >
> > >
> > > On Wed, Jun 12, 2024 at 1:30 PM Markus Doppelbauer<
> > > doppelbauer@gmx.net.invalid> wrote:
> > > > Hi Nick,Thank you very much for your reply.This is exactly what
> > > > we are lookingfor.There are so many DBs that store the
> > > > secondary indexlocally(Cassandra, Aerospike, SyllaDB,
> > > > ...)Thanks again for the answerMarcus
> > > >
> > > > Am Mittwoch, dem 12.06.2024 um 13:23 -0400 schrieb Nick
> > > > Vatamaniuc:
> > > > > Hi Marcus,The node handling the request only queries the
> > > > > nodes with shardcopies ofthat database. In a 100 node cluster
> > > > > the shards for thatparticulardatabase might be present on
> > > > > only 6 nodes, depending on theQ and Nsharding factors, so it
> > > > > will query 6 out 100 nodes. Forinstance, for N=3and Q=2
> > > > > sharding factors, it will first send N*Q=6requests, and wait
> > > > > untilit gets at least one response for each of theQ=2 shard
> > > > > ranges. Thishappens very quickly. Then, for the duration
> > > > > ofthe response, it will onlystream responses from those Q=2
> > > > > workers.So, to summarize for a Q=2database, it will be a
> > > > > streaming responsefrom 2 workers. For Q=4, from 4workers,
> > > > > etc...Cheers,-Nick
> > > > > On Wed, Jun 12, 2024 at 1:00 PM Markus Doppelbauer<
> > > > > doppelbauer@gmx.net.invalid> wrote:
> > > > > > Hello,Is the CouchDB-view a "global" or "local" index?For
> > > > > > example, if acluster has 100 nodes, would the query askfor
> > > > > > a single node - or100
> > > > > > nodes?/.../_view/posts?startkey="foobar"&endkey="foobaz"Bes
> > > > > > t wishesMarcus
Re: View performance
Posted by Matthieu Rakotojaona <ma...@imt-atlantique.fr>.
Hey,
Thanks for the reply, it is clearer now! Only one point remains
"mysterious" to me:
> - quorum applies to document operations (read, write, delete), but
not view queries (R is inherently 1)
This means that queries might be inconsistent. View queries aren't
necessarily different from reads, so I don't get why it's ok in this case
On 13/06/2024 14:40, Robert Newson wrote:
> Hi,
>
> Right. a database is divided into Q shards, each of which covers part of a numeric range from 0 to 2^32-1, the doc id is run through a hashing function that outputs a number in that range, and the document is then stored inside the shard whose range contains that number.
>
> as this effectively randomises which doc goes in which shard when we need to perform a query across the database we need to ask each shard range to perform the query and then combine the results.
>
> This is good in some ways (the shards can be stored across multiple machines and thus benefit from more cores/disks/memory/etc) it's bad in others, the number of workers that need to respond goes up as Q goes up.
>
> "Partition" is just our word (and arguably a misleading word) for allowing users to control which shard range their documents end up in. Only part of the doc id is passed through the hashing function(doc id "foo:bar" would be partition foo, and we'd hash only "foo"). Thus the user can ensure that a group of documents lands in the same shard range. The benefit is as Nick described, when querying _within_ a given partition we know that only one shard range is in play, so we do not consult shards for other ranges (we already know they will have no results).
>
> So, to your specific list of questions;
>
> - yes
> - yes, but the reason to keep partitions small is because of the fixed assignment to a given shard range. the default scheme would load up each shard range fairly evenly. it's about disk space management and predictable performance.
> - when querying a global view we need Q responses (and will issue Q*N requests, but cancel Q*(N-1) of those when the first response from each range has been received, though they may have already done some, or all, of their work, before that happens). for a partitioned view we need 1 response (but will issue N requests).
> - quorum applies to document operations (read, write, delete), but not view queries (R is inherently 1)
>
> B.
>
>> On 13 Jun 2024, at 09:01, Matthieu Rakotojaona <ma...@imt-atlantique.fr> wrote:
>>
>> Hello, I'm still a bit confused with shards versus partitions, is this correct ?
>>
>> - A database is composed of Q shards, shards splice the database "automatically"
>>
>> - A shard is composed of 1 or more partitions, partitions splice the database according to the user's choice. A partition should thus be made so that it is not huge, in fact must be under (total size/Q)
>>
>> - When querying a view, the whole database needs to be processed, that means all Q shards
>>
>> - When querying a view only on documents inside a partition, since one partition is on 1 shard, only one shard is queried
>>
>> - Whatever the number of shards, quorum still needs to be verified, ie for Q=2 and N=3 I still need to wait for (3+1)/2 = 2 replies for each shard
>>
>> On 12/06/2024 19:54, Nick Vatamaniuc wrote:
>>> Another feature related to efficient view querying are partitioned
>>> databases: https://docs.couchdb.org/en/stable/partitioned-dbs/index.html.
>>> It's a bit of a niche, as you'd need to have a good partition key, but
>>> aside from that, it can speed up your queries as responses would be coming
>>> from a single shard only instead of Q shards.
>>>
>>>
>>>
>>> On Wed, Jun 12, 2024 at 1:30 PM Markus Doppelbauer
>>> <do...@gmx.net.invalid> wrote:
>>>
>>>> Hi Nick,
>>>> Thank you very much for your reply.This is exactly what we are looking
>>>> for.There are so many DBs that store the secondary indexlocally
>>>> (Cassandra, Aerospike, SyllaDB, ...)
>>>> Thanks again for the answerMarcus
>>>>
>>>>
>>>> Am Mittwoch, dem 12.06.2024 um 13:23 -0400 schrieb Nick Vatamaniuc:
>>>>> Hi Marcus,
>>>>> The node handling the request only queries the nodes with shard
>>>>> copies ofthat database. In a 100 node cluster the shards for that
>>>>> particulardatabase might be present on only 6 nodes, depending on the
>>>>> Q and Nsharding factors, so it will query 6 out 100 nodes. For
>>>>> instance, for N=3and Q=2 sharding factors, it will first send N*Q=6
>>>>> requests, and wait untilit gets at least one response for each of the
>>>>> Q=2 shard ranges. Thishappens very quickly. Then, for the duration of
>>>>> the response, it will onlystream responses from those Q=2 workers.
>>>>> So, to summarize for a Q=2database, it will be a streaming response
>>>>> from 2 workers. For Q=4, from 4workers, etc...
>>>>> Cheers,-Nick
>>>>>
>>>>> On Wed, Jun 12, 2024 at 1:00 PM Markus Doppelbauer<
>>>>> doppelbauer@gmx.net.invalid> wrote:
>>>>>> Hello,
>>>>>> Is the CouchDB-view a "global" or "local" index?For example, if a
>>>>>> cluster has 100 nodes, would the query askfor a single node - or
>>>>>> 100 nodes?
>>>>>> /.../_view/posts?startkey="foobar"&endkey="foobaz"
>>>>>> Best wishesMarcus
>>>>>>
Re: View performance
Posted by Robert Newson <rn...@apache.org>.
Hi,
Right. a database is divided into Q shards, each of which covers part of a numeric range from 0 to 2^32-1, the doc id is run through a hashing function that outputs a number in that range, and the document is then stored inside the shard whose range contains that number.
as this effectively randomises which doc goes in which shard when we need to perform a query across the database we need to ask each shard range to perform the query and then combine the results.
This is good in some ways (the shards can be stored across multiple machines and thus benefit from more cores/disks/memory/etc) it's bad in others, the number of workers that need to respond goes up as Q goes up.
"Partition" is just our word (and arguably a misleading word) for allowing users to control which shard range their documents end up in. Only part of the doc id is passed through the hashing function(doc id "foo:bar" would be partition foo, and we'd hash only "foo"). Thus the user can ensure that a group of documents lands in the same shard range. The benefit is as Nick described, when querying _within_ a given partition we know that only one shard range is in play, so we do not consult shards for other ranges (we already know they will have no results).
So, to your specific list of questions;
- yes
- yes, but the reason to keep partitions small is because of the fixed assignment to a given shard range. the default scheme would load up each shard range fairly evenly. it's about disk space management and predictable performance.
- when querying a global view we need Q responses (and will issue Q*N requests, but cancel Q*(N-1) of those when the first response from each range has been received, though they may have already done some, or all, of their work, before that happens). for a partitioned view we need 1 response (but will issue N requests).
- quorum applies to document operations (read, write, delete), but not view queries (R is inherently 1)
B.
> On 13 Jun 2024, at 09:01, Matthieu Rakotojaona <ma...@imt-atlantique.fr> wrote:
>
> Hello, I'm still a bit confused with shards versus partitions, is this correct ?
>
> - A database is composed of Q shards, shards splice the database "automatically"
>
> - A shard is composed of 1 or more partitions, partitions splice the database according to the user's choice. A partition should thus be made so that it is not huge, in fact must be under (total size/Q)
>
> - When querying a view, the whole database needs to be processed, that means all Q shards
>
> - When querying a view only on documents inside a partition, since one partition is on 1 shard, only one shard is queried
>
> - Whatever the number of shards, quorum still needs to be verified, ie for Q=2 and N=3 I still need to wait for (3+1)/2 = 2 replies for each shard
>
> On 12/06/2024 19:54, Nick Vatamaniuc wrote:
>> Another feature related to efficient view querying are partitioned
>> databases: https://docs.couchdb.org/en/stable/partitioned-dbs/index.html.
>> It's a bit of a niche, as you'd need to have a good partition key, but
>> aside from that, it can speed up your queries as responses would be coming
>> from a single shard only instead of Q shards.
>>
>>
>>
>> On Wed, Jun 12, 2024 at 1:30 PM Markus Doppelbauer
>> <do...@gmx.net.invalid> wrote:
>>
>>> Hi Nick,
>>> Thank you very much for your reply.This is exactly what we are looking
>>> for.There are so many DBs that store the secondary indexlocally
>>> (Cassandra, Aerospike, SyllaDB, ...)
>>> Thanks again for the answerMarcus
>>>
>>>
>>> Am Mittwoch, dem 12.06.2024 um 13:23 -0400 schrieb Nick Vatamaniuc:
>>>> Hi Marcus,
>>>> The node handling the request only queries the nodes with shard
>>>> copies ofthat database. In a 100 node cluster the shards for that
>>>> particulardatabase might be present on only 6 nodes, depending on the
>>>> Q and Nsharding factors, so it will query 6 out 100 nodes. For
>>>> instance, for N=3and Q=2 sharding factors, it will first send N*Q=6
>>>> requests, and wait untilit gets at least one response for each of the
>>>> Q=2 shard ranges. Thishappens very quickly. Then, for the duration of
>>>> the response, it will onlystream responses from those Q=2 workers.
>>>> So, to summarize for a Q=2database, it will be a streaming response
>>>> from 2 workers. For Q=4, from 4workers, etc...
>>>> Cheers,-Nick
>>>>
>>>> On Wed, Jun 12, 2024 at 1:00 PM Markus Doppelbauer<
>>>> doppelbauer@gmx.net.invalid> wrote:
>>>>> Hello,
>>>>> Is the CouchDB-view a "global" or "local" index?For example, if a
>>>>> cluster has 100 nodes, would the query askfor a single node - or
>>>>> 100 nodes?
>>>>> /.../_view/posts?startkey="foobar"&endkey="foobaz"
>>>>> Best wishesMarcus
>>>>>
Re: View performance
Posted by Matthieu Rakotojaona <ma...@imt-atlantique.fr>.
Hello, I'm still a bit confused with shards versus partitions, is this
correct ?
- A database is composed of Q shards, shards splice the database
"automatically"
- A shard is composed of 1 or more partitions, partitions splice the
database according to the user's choice. A partition should thus be made
so that it is not huge, in fact must be under (total size/Q)
- When querying a view, the whole database needs to be processed, that
means all Q shards
- When querying a view only on documents inside a partition, since one
partition is on 1 shard, only one shard is queried
- Whatever the number of shards, quorum still needs to be verified, ie
for Q=2 and N=3 I still need to wait for (3+1)/2 = 2 replies for each shard
On 12/06/2024 19:54, Nick Vatamaniuc wrote:
> Another feature related to efficient view querying are partitioned
> databases: https://docs.couchdb.org/en/stable/partitioned-dbs/index.html.
> It's a bit of a niche, as you'd need to have a good partition key, but
> aside from that, it can speed up your queries as responses would be coming
> from a single shard only instead of Q shards.
>
>
>
> On Wed, Jun 12, 2024 at 1:30 PM Markus Doppelbauer
> <do...@gmx.net.invalid> wrote:
>
>> Hi Nick,
>> Thank you very much for your reply.This is exactly what we are looking
>> for.There are so many DBs that store the secondary indexlocally
>> (Cassandra, Aerospike, SyllaDB, ...)
>> Thanks again for the answerMarcus
>>
>>
>> Am Mittwoch, dem 12.06.2024 um 13:23 -0400 schrieb Nick Vatamaniuc:
>>> Hi Marcus,
>>> The node handling the request only queries the nodes with shard
>>> copies ofthat database. In a 100 node cluster the shards for that
>>> particulardatabase might be present on only 6 nodes, depending on the
>>> Q and Nsharding factors, so it will query 6 out 100 nodes. For
>>> instance, for N=3and Q=2 sharding factors, it will first send N*Q=6
>>> requests, and wait untilit gets at least one response for each of the
>>> Q=2 shard ranges. Thishappens very quickly. Then, for the duration of
>>> the response, it will onlystream responses from those Q=2 workers.
>>> So, to summarize for a Q=2database, it will be a streaming response
>>> from 2 workers. For Q=4, from 4workers, etc...
>>> Cheers,-Nick
>>>
>>> On Wed, Jun 12, 2024 at 1:00 PM Markus Doppelbauer<
>>> doppelbauer@gmx.net.invalid> wrote:
>>>> Hello,
>>>> Is the CouchDB-view a "global" or "local" index?For example, if a
>>>> cluster has 100 nodes, would the query askfor a single node - or
>>>> 100 nodes?
>>>> /.../_view/posts?startkey="foobar"&endkey="foobaz"
>>>> Best wishesMarcus
>>>>
Re: View performance
Posted by Nick Vatamaniuc <va...@gmail.com>.
Good question, Marcus! Yeah, that's possible and that's how you would scale
and get better performance. So db1 might have Q=2 (that's the default) and
with N=3, it might pick shards on nodes 2, 15, 18, 41, 54, 93. So two
ranges and 3 copies for each range. But you can also db2 with Q=16 and N=3
and then Q*N=48 shards would be spread amongst other nodes randomly.
You can read more about it at
https://docs.couchdb.org/en/stable/cluster/theory.html
Cheers,
-Nick
On Wed, Jun 12, 2024 at 2:45 PM Markus Doppelbauer
<do...@gmx.net.invalid> wrote:
> Dear Nick,
>
> Is it possible to set the number of shards (Q) lower than the number of
> nodes?
> Wouldn't that result in unused nodes?
>
> Best wishes!
> Marcus
>
>
> Am Mittwoch, dem 12.06.2024 um 13:54 -0400 schrieb Nick Vatamaniuc:
> > Another feature related to efficient view querying are
> > partitioneddatabases:
> > https://docs.couchdb.org/en/stable/partitioned-dbs/index.html.It's a
> > bit of a niche, as you'd need to have a good partition key, butaside
> > from that, it can speed up your queries as responses would be
> > comingfrom a single shard only instead of Q shards.
> >
> >
> > On Wed, Jun 12, 2024 at 1:30 PM Markus Doppelbauer<
> > doppelbauer@gmx.net.invalid> wrote:
> > > Hi Nick,Thank you very much for your reply.This is exactly what we
> > > are lookingfor.There are so many DBs that store the secondary
> > > indexlocally(Cassandra, Aerospike, SyllaDB, ...)Thanks again for
> > > the answerMarcus
> > >
> > > Am Mittwoch, dem 12.06.2024 um 13:23 -0400 schrieb Nick Vatamaniuc:
> > > > Hi Marcus,The node handling the request only queries the nodes
> > > > with shardcopies ofthat database. In a 100 node cluster the
> > > > shards for thatparticulardatabase might be present on only 6
> > > > nodes, depending on theQ and Nsharding factors, so it will query
> > > > 6 out 100 nodes. Forinstance, for N=3and Q=2 sharding factors, it
> > > > will first send N*Q=6requests, and wait untilit gets at least one
> > > > response for each of theQ=2 shard ranges. Thishappens very
> > > > quickly. Then, for the duration ofthe response, it will
> > > > onlystream responses from those Q=2 workers.So, to summarize for
> > > > a Q=2database, it will be a streaming responsefrom 2 workers. For
> > > > Q=4, from 4workers, etc...Cheers,-Nick
> > > > On Wed, Jun 12, 2024 at 1:00 PM Markus Doppelbauer<
> > > > doppelbauer@gmx.net.invalid> wrote:
> > > > > Hello,Is the CouchDB-view a "global" or "local" index?For
> > > > > example, if acluster has 100 nodes, would the query askfor a
> > > > > single node - or100
> > > > > nodes?/.../_view/posts?startkey="foobar"&endkey="foobaz"Best
> > > > > wishesMarcus
>
Re: View performance
Posted by Markus Doppelbauer <do...@gmx.net.INVALID>.
Dear Nick,
Is it possible to set the number of shards (Q) lower than the number of
nodes?
Wouldn't that result in unused nodes?
Best wishes!
Marcus
Am Mittwoch, dem 12.06.2024 um 13:54 -0400 schrieb Nick Vatamaniuc:
> Another feature related to efficient view querying are
> partitioneddatabases:
> https://docs.couchdb.org/en/stable/partitioned-dbs/index.html.It's a
> bit of a niche, as you'd need to have a good partition key, butaside
> from that, it can speed up your queries as responses would be
> comingfrom a single shard only instead of Q shards.
>
>
> On Wed, Jun 12, 2024 at 1:30 PM Markus Doppelbauer<
> doppelbauer@gmx.net.invalid> wrote:
> > Hi Nick,Thank you very much for your reply.This is exactly what we
> > are lookingfor.There are so many DBs that store the secondary
> > indexlocally(Cassandra, Aerospike, SyllaDB, ...)Thanks again for
> > the answerMarcus
> >
> > Am Mittwoch, dem 12.06.2024 um 13:23 -0400 schrieb Nick Vatamaniuc:
> > > Hi Marcus,The node handling the request only queries the nodes
> > > with shardcopies ofthat database. In a 100 node cluster the
> > > shards for thatparticulardatabase might be present on only 6
> > > nodes, depending on theQ and Nsharding factors, so it will query
> > > 6 out 100 nodes. Forinstance, for N=3and Q=2 sharding factors, it
> > > will first send N*Q=6requests, and wait untilit gets at least one
> > > response for each of theQ=2 shard ranges. Thishappens very
> > > quickly. Then, for the duration ofthe response, it will
> > > onlystream responses from those Q=2 workers.So, to summarize for
> > > a Q=2database, it will be a streaming responsefrom 2 workers. For
> > > Q=4, from 4workers, etc...Cheers,-Nick
> > > On Wed, Jun 12, 2024 at 1:00 PM Markus Doppelbauer<
> > > doppelbauer@gmx.net.invalid> wrote:
> > > > Hello,Is the CouchDB-view a "global" or "local" index?For
> > > > example, if acluster has 100 nodes, would the query askfor a
> > > > single node - or100
> > > > nodes?/.../_view/posts?startkey="foobar"&endkey="foobaz"Best
> > > > wishesMarcus
Re: View performance
Posted by Nick Vatamaniuc <va...@gmail.com>.
Another feature related to efficient view querying are partitioned
databases: https://docs.couchdb.org/en/stable/partitioned-dbs/index.html.
It's a bit of a niche, as you'd need to have a good partition key, but
aside from that, it can speed up your queries as responses would be coming
from a single shard only instead of Q shards.
On Wed, Jun 12, 2024 at 1:30 PM Markus Doppelbauer
<do...@gmx.net.invalid> wrote:
> Hi Nick,
> Thank you very much for your reply.This is exactly what we are looking
> for.There are so many DBs that store the secondary indexlocally
> (Cassandra, Aerospike, SyllaDB, ...)
> Thanks again for the answerMarcus
>
>
> Am Mittwoch, dem 12.06.2024 um 13:23 -0400 schrieb Nick Vatamaniuc:
> > Hi Marcus,
> > The node handling the request only queries the nodes with shard
> > copies ofthat database. In a 100 node cluster the shards for that
> > particulardatabase might be present on only 6 nodes, depending on the
> > Q and Nsharding factors, so it will query 6 out 100 nodes. For
> > instance, for N=3and Q=2 sharding factors, it will first send N*Q=6
> > requests, and wait untilit gets at least one response for each of the
> > Q=2 shard ranges. Thishappens very quickly. Then, for the duration of
> > the response, it will onlystream responses from those Q=2 workers.
> > So, to summarize for a Q=2database, it will be a streaming response
> > from 2 workers. For Q=4, from 4workers, etc...
> > Cheers,-Nick
> >
> > On Wed, Jun 12, 2024 at 1:00 PM Markus Doppelbauer<
> > doppelbauer@gmx.net.invalid> wrote:
> > > Hello,
> > > Is the CouchDB-view a "global" or "local" index?For example, if a
> > > cluster has 100 nodes, would the query askfor a single node - or
> > > 100 nodes?
> > > /.../_view/posts?startkey="foobar"&endkey="foobaz"
> > > Best wishesMarcus
> > >
>
Re: View performance
Posted by Markus Doppelbauer <do...@gmx.net.INVALID>.
Hi Nick,
Thank you very much for your reply.This is exactly what we are looking
for.There are so many DBs that store the secondary indexlocally
(Cassandra, Aerospike, SyllaDB, ...)
Thanks again for the answerMarcus
Am Mittwoch, dem 12.06.2024 um 13:23 -0400 schrieb Nick Vatamaniuc:
> Hi Marcus,
> The node handling the request only queries the nodes with shard
> copies ofthat database. In a 100 node cluster the shards for that
> particulardatabase might be present on only 6 nodes, depending on the
> Q and Nsharding factors, so it will query 6 out 100 nodes. For
> instance, for N=3and Q=2 sharding factors, it will first send N*Q=6
> requests, and wait untilit gets at least one response for each of the
> Q=2 shard ranges. Thishappens very quickly. Then, for the duration of
> the response, it will onlystream responses from those Q=2 workers.
> So, to summarize for a Q=2database, it will be a streaming response
> from 2 workers. For Q=4, from 4workers, etc...
> Cheers,-Nick
>
> On Wed, Jun 12, 2024 at 1:00 PM Markus Doppelbauer<
> doppelbauer@gmx.net.invalid> wrote:
> > Hello,
> > Is the CouchDB-view a "global" or "local" index?For example, if a
> > cluster has 100 nodes, would the query askfor a single node - or
> > 100 nodes?
> > /.../_view/posts?startkey="foobar"&endkey="foobaz"
> > Best wishesMarcus
> >
Re: View performance
Posted by Nick Vatamaniuc <va...@gmail.com>.
Hi Marcus,
The node handling the request only queries the nodes with shard copies of
that database. In a 100 node cluster the shards for that particular
database might be present on only 6 nodes, depending on the Q and N
sharding factors, so it will query 6 out 100 nodes. For instance, for N=3
and Q=2 sharding factors, it will first send N*Q=6 requests, and wait until
it gets at least one response for each of the Q=2 shard ranges. This
happens very quickly. Then, for the duration of the response, it will only
stream responses from those Q=2 workers. So, to summarize for a Q=2
database, it will be a streaming response from 2 workers. For Q=4, from 4
workers, etc...
Cheers,
-Nick
On Wed, Jun 12, 2024 at 1:00 PM Markus Doppelbauer
<do...@gmx.net.invalid> wrote:
> Hello,
>
> Is the CouchDB-view a "global" or "local" index?
> For example, if a cluster has 100 nodes, would the query ask
> for a single node - or 100 nodes?
>
> /.../_view/posts?startkey="foobar"&endkey="foobaz"
>
> Best wishes
> Marcus
>
>