You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Fabio Forno <fa...@gmail.com> on 2010/02/22 12:00:23 UTC

key entropy and view performance

Hi,
during view generation I've noticed that view re-indexing speed seems
very related to the entropy of keys. For example I've map/reduce view
which counts the a set of resources with a score and a type. The score
has a very high entropy (few tens of resources for each score), while
the resource type is limited (>100k per type). If I emit keys as
[type, score] map/reduce computation seems much slower than [score,
type]. Is it normal or a just a case? Isn't there the risk of
degeneration if the numbers of resources grows too much (eg. millions
per resource type)?
Since I need to do range queries according the score, but only on
selected types of resources, I think that the [type, score] key is
preferable, however this degradation in speed concerns me. Any hint?
thanks in advance.

-- 
Fabio Forno,
Bluendo srl http://www.bluendo.com
jabber id: ff@jabber.bluendo.com

Re: key entropy and view performance

Posted by Fabio Forno <fa...@gmail.com>.
On Mon, Feb 22, 2010 at 1:24 PM, Sebastian Cohnen
<se...@googlemail.com> wrote:
> Hi Fabio,
>
> this sounds "normal" to me, since couch views use btrees and (high) entropy hurts btree's update/insert performance.

Therefore I guess that a valid alternative is to emit only scores and
make specialized views for any resource type.

-- 
Fabio Forno,
Ooros srl
jabber id: ff@jabber.bluendo.com

Re: key entropy and view performance

Posted by Sebastian Cohnen <se...@googlemail.com>.
Hi Fabio,

this sounds "normal" to me, since couch views use btrees and (high) entropy hurts btree's update/insert performance.

greets
tisba

On 22.02.2010, at 12:00, Fabio Forno wrote:

> Hi,
> during view generation I've noticed that view re-indexing speed seems
> very related to the entropy of keys. For example I've map/reduce view
> which counts the a set of resources with a score and a type. The score
> has a very high entropy (few tens of resources for each score), while
> the resource type is limited (>100k per type). If I emit keys as
> [type, score] map/reduce computation seems much slower than [score,
> type]. Is it normal or a just a case? Isn't there the risk of
> degeneration if the numbers of resources grows too much (eg. millions
> per resource type)?
> Since I need to do range queries according the score, but only on
> selected types of resources, I think that the [type, score] key is
> preferable, however this degradation in speed concerns me. Any hint?
> thanks in advance.
> 
> -- 
> Fabio Forno,
> Bluendo srl http://www.bluendo.com
> jabber id: ff@jabber.bluendo.com