You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by paul jobs <we...@gmail.com> on 2008/12/21 05:04:36 UTC
slow views
map:
function(doc) { emit(doc.zodiac, 1) }
reduce:
function(keys, values) { return sum(values)}
Key
Value *null*149877*"aquarius"*98747*"aries"*103072*"cancer"*104318*
"capricorn"*97605*"gemini"*103708*"leo"*107982*"libra"*104955*"pisces"*
101741*"sagittarius"*92975
total number of records ~2 million
this query takes quite some time to run
is there any way to make this query run faster
thanks
Re: slow views
Posted by paul jobs <we...@gmail.com>.
The queries that involve running through the entire dband counting star
signs or sex takes time
yes. i think paul i got the point you are trying to make, by preparing a
permanent view in the design document is all the info couchdb needs, there
is no other stuff to make it faster
On Sat, Dec 20, 2008 at 8:26 PM, Paul Davis <pa...@gmail.com>wrote:
> I have no idea what this means.
>
> On Sat, Dec 20, 2008 at 11:24 PM, paul jobs <we...@gmail.com> wrote:
> > Key
> > Value *null*38822*""*263251*"female"*615820*"male"*538011
> > Same applies for this view too, these are the number of users in our app
> and
> > their sex
> > map: function(doc) { emit(doc.sex, 1) }
> >
> > reduce: function(keys, values) { return sum(values)}
> >
> > On 12/20/08, paul jobs <we...@gmail.com> wrote:
> >>
> >> it is a permanent viewbut every repeat request takes time as new users
> are
> >> constantly added to the db
> >>
> >> On Sat, Dec 20, 2008 at 8:16 PM, Paul Davis <
> paul.joseph.davis@gmail.com>wrote:
> >>
> >>> If it's a temp view, no.
> >>>
> >>> If it's a permanent view, no. But *repeated* requests should be nearly
> >>> instantaneous.
> >>>
> >>> That is all.
> >>>
> >>> On Sat, Dec 20, 2008 at 11:04 PM, paul jobs <we...@gmail.com>
> wrote:
> >>> > map:
> >>> > function(doc) { emit(doc.zodiac, 1) }
> >>> > reduce:
> >>> > function(keys, values) { return sum(values)}
> >>> >
> >>> >
> >>> > Key
> >>> > Value *null*149877*"aquarius"*98747*"aries"*103072*"cancer"*104318*
> >>> >
> "capricorn"*97605*"gemini"*103708*"leo"*107982*"libra"*104955*"pisces"*
> >>> > 101741*"sagittarius"*92975
> >>> >
> >>> > total number of records ~2 million
> >>> >
> >>> > this query takes quite some time to run
> >>> > is there any way to make this query run faster
> >>> >
> >>> > thanks
> >>> >
> >>>
> >>
> >>
> >
>
Re: slow views
Posted by Paul Davis <pa...@gmail.com>.
I have no idea what this means.
On Sat, Dec 20, 2008 at 11:24 PM, paul jobs <we...@gmail.com> wrote:
> Key
> Value *null*38822*""*263251*"female"*615820*"male"*538011
> Same applies for this view too, these are the number of users in our app and
> their sex
> map: function(doc) { emit(doc.sex, 1) }
>
> reduce: function(keys, values) { return sum(values)}
>
> On 12/20/08, paul jobs <we...@gmail.com> wrote:
>>
>> it is a permanent viewbut every repeat request takes time as new users are
>> constantly added to the db
>>
>> On Sat, Dec 20, 2008 at 8:16 PM, Paul Davis <pa...@gmail.com>wrote:
>>
>>> If it's a temp view, no.
>>>
>>> If it's a permanent view, no. But *repeated* requests should be nearly
>>> instantaneous.
>>>
>>> That is all.
>>>
>>> On Sat, Dec 20, 2008 at 11:04 PM, paul jobs <we...@gmail.com> wrote:
>>> > map:
>>> > function(doc) { emit(doc.zodiac, 1) }
>>> > reduce:
>>> > function(keys, values) { return sum(values)}
>>> >
>>> >
>>> > Key
>>> > Value *null*149877*"aquarius"*98747*"aries"*103072*"cancer"*104318*
>>> > "capricorn"*97605*"gemini"*103708*"leo"*107982*"libra"*104955*"pisces"*
>>> > 101741*"sagittarius"*92975
>>> >
>>> > total number of records ~2 million
>>> >
>>> > this query takes quite some time to run
>>> > is there any way to make this query run faster
>>> >
>>> > thanks
>>> >
>>>
>>
>>
>
Re: slow views
Posted by paul jobs <we...@gmail.com>.
Key
Value *null*38822*""*263251*"female"*615820*"male"*538011
Same applies for this view too, these are the number of users in our app and
their sex
map: function(doc) { emit(doc.sex, 1) }
reduce: function(keys, values) { return sum(values)}
On 12/20/08, paul jobs <we...@gmail.com> wrote:
>
> it is a permanent viewbut every repeat request takes time as new users are
> constantly added to the db
>
> On Sat, Dec 20, 2008 at 8:16 PM, Paul Davis <pa...@gmail.com>wrote:
>
>> If it's a temp view, no.
>>
>> If it's a permanent view, no. But *repeated* requests should be nearly
>> instantaneous.
>>
>> That is all.
>>
>> On Sat, Dec 20, 2008 at 11:04 PM, paul jobs <we...@gmail.com> wrote:
>> > map:
>> > function(doc) { emit(doc.zodiac, 1) }
>> > reduce:
>> > function(keys, values) { return sum(values)}
>> >
>> >
>> > Key
>> > Value *null*149877*"aquarius"*98747*"aries"*103072*"cancer"*104318*
>> > "capricorn"*97605*"gemini"*103708*"leo"*107982*"libra"*104955*"pisces"*
>> > 101741*"sagittarius"*92975
>> >
>> > total number of records ~2 million
>> >
>> > this query takes quite some time to run
>> > is there any way to make this query run faster
>> >
>> > thanks
>> >
>>
>
>
Re: slow views
Posted by Chris Anderson <jc...@gmail.com>.
On Sat, Dec 20, 2008 at 8:50 PM, Antony Blakey <an...@gmail.com> wrote:
>
> On 21/12/2008, at 3:15 PM, Paul Davis wrote:
>
>> On Sat, Dec 20, 2008 at 11:32 PM, Antony Blakey <an...@gmail.com>
>> wrote:
>>>
>>> On 21/12/2008, at 2:54 PM, Paul Davis wrote:
>>>
>>>> View update times should be linear in terms of *new* records.
>>>
>>> Well, with a factor to deal with the reduction, which isn't O(n).
>>>
>>
>> Kinda. I'm pretty sure they're still O(N), just not as pre calculated
>> as one might expect.
>
> I would have thought + a tiny O(log M) where M is determined by the grouping
> size of the partial reductions.
I think there's at least a linear cost associated with the number of
groups in the final reduction. eg. a group=true result requires N as
many as there are unique keys. If your range encompasses 5 groups,
then you'd require 5 final reductions to be run. Requesting the same
range with group=false, I'd expect a substantial speedup.
If it's less than a linear relation to the number of groups, I'd be happy.
--
Chris Anderson
http://jchris.mfdz.com
Re: slow views
Posted by Antony Blakey <an...@gmail.com>.
On 21/12/2008, at 10:48 PM, Paul Davis wrote:
> On Sun, Dec 21, 2008 at 1:20 AM, Antony Blakey <antony.blakey@gmail.com
> > wrote:
>>
>> On 21/12/2008, at 3:25 PM, Paul Davis wrote:
>>
>>> Hmm. I don't think so. Technically we could change the
>>> implementation
>>> to feed reduce functions a single map-row at a time which would be
>>> O(N). Might end up violating some (incorrect) expectations of reduce
>>> input, but I'm pretty certain it shows that it's O(N).
>>
>> But you have to combine it with previously reduced input, which means
>> fetching a number of previous results, with a cost that isn't clearly
>> related to N. Isn't the number of reductions needed to be combined
>> is going
>> to be related to key distribution?
>>
>
> Good point. I just woke up so the brain is working kinda slow, but are
> we describing best case and worst case scenarios too? If each key had
> exactly one row and it was group=true, then it'd be O(N). If it was
> group=false and every key was identical it'd be log(N). And then
> normal use cases are somewhere in the middle?
Where the original N was new rows, versus this N being total rows?
e.g. average case is O(new rows) + O(log old rows)
My complexity maths is weak.
Antony Blakey
-------------
CTO, Linkuistics Pty Ltd
Ph: 0438 840 787
In anything at all, perfection is finally attained not when there is
no longer anything to add, but when there is no longer anything to
take away.
-- Antoine de Saint-Exupery
Re: slow views
Posted by Paul Davis <pa...@gmail.com>.
On Sun, Dec 21, 2008 at 1:20 AM, Antony Blakey <an...@gmail.com> wrote:
>
> On 21/12/2008, at 3:25 PM, Paul Davis wrote:
>
>> Hmm. I don't think so. Technically we could change the implementation
>> to feed reduce functions a single map-row at a time which would be
>> O(N). Might end up violating some (incorrect) expectations of reduce
>> input, but I'm pretty certain it shows that it's O(N).
>
> But you have to combine it with previously reduced input, which means
> fetching a number of previous results, with a cost that isn't clearly
> related to N. Isn't the number of reductions needed to be combined is going
> to be related to key distribution?
>
Good point. I just woke up so the brain is working kinda slow, but are
we describing best case and worst case scenarios too? If each key had
exactly one row and it was group=true, then it'd be O(N). If it was
group=false and every key was identical it'd be log(N). And then
normal use cases are somewhere in the middle?
> Antony Blakey
> -------------
> CTO, Linkuistics Pty Ltd
> Ph: 0438 840 787
>
> It is no measure of health to be well adjusted to a profoundly sick society.
> -- Jiddu Krishnamurti
>
>
>
Re: slow views
Posted by Antony Blakey <an...@gmail.com>.
On 21/12/2008, at 3:25 PM, Paul Davis wrote:
> Hmm. I don't think so. Technically we could change the implementation
> to feed reduce functions a single map-row at a time which would be
> O(N). Might end up violating some (incorrect) expectations of reduce
> input, but I'm pretty certain it shows that it's O(N).
But you have to combine it with previously reduced input, which means
fetching a number of previous results, with a cost that isn't clearly
related to N. Isn't the number of reductions needed to be combined is
going to be related to key distribution?
Antony Blakey
-------------
CTO, Linkuistics Pty Ltd
Ph: 0438 840 787
It is no measure of health to be well adjusted to a profoundly sick
society.
-- Jiddu Krishnamurti
Re: slow views
Posted by Paul Davis <pa...@gmail.com>.
On Sat, Dec 20, 2008 at 11:50 PM, Antony Blakey <an...@gmail.com> wrote:
>
> On 21/12/2008, at 3:15 PM, Paul Davis wrote:
>
>> On Sat, Dec 20, 2008 at 11:32 PM, Antony Blakey <an...@gmail.com>
>> wrote:
>>>
>>> On 21/12/2008, at 2:54 PM, Paul Davis wrote:
>>>
>>>> View update times should be linear in terms of *new* records.
>>>
>>> Well, with a factor to deal with the reduction, which isn't O(n).
>>>
>>
>> Kinda. I'm pretty sure they're still O(N), just not as pre calculated
>> as one might expect.
>
> I would have thought + a tiny O(log M) where M is determined by the grouping
> size of the partial reductions.
>
Hmm. I don't think so. Technically we could change the implementation
to feed reduce functions a single map-row at a time which would be
O(N). Might end up violating some (incorrect) expectations of reduce
input, but I'm pretty certain it shows that it's O(N).
> Antony Blakey
> --------------------------
> CTO, Linkuistics Pty Ltd
> Ph: 0438 840 787
>
> Don't anthropomorphize computers. They hate that.
>
>
>
Re: slow views
Posted by Antony Blakey <an...@gmail.com>.
On 21/12/2008, at 3:15 PM, Paul Davis wrote:
> On Sat, Dec 20, 2008 at 11:32 PM, Antony Blakey <antony.blakey@gmail.com
> > wrote:
>>
>> On 21/12/2008, at 2:54 PM, Paul Davis wrote:
>>
>>> View update times should be linear in terms of *new* records.
>>
>> Well, with a factor to deal with the reduction, which isn't O(n).
>>
>
> Kinda. I'm pretty sure they're still O(N), just not as pre calculated
> as one might expect.
I would have thought + a tiny O(log M) where M is determined by the
grouping size of the partial reductions.
Antony Blakey
--------------------------
CTO, Linkuistics Pty Ltd
Ph: 0438 840 787
Don't anthropomorphize computers. They hate that.
Re: slow views
Posted by Paul Davis <pa...@gmail.com>.
On Sat, Dec 20, 2008 at 11:32 PM, Antony Blakey <an...@gmail.com> wrote:
>
> On 21/12/2008, at 2:54 PM, Paul Davis wrote:
>
>> View update times should be linear in terms of *new* records.
>
> Well, with a factor to deal with the reduction, which isn't O(n).
>
Kinda. I'm pretty sure they're still O(N), just not as pre calculated
as one might expect.
>>
>>
>> On Sat, Dec 20, 2008 at 11:20 PM, paul jobs <we...@gmail.com> wrote:
>>>
>>> it is a permanent viewbut every repeat request takes time as new users
>>> are
>>> constantly added to the db
>>>
>>> On Sat, Dec 20, 2008 at 8:16 PM, Paul Davis
>>> <pa...@gmail.com>wrote:
>>>
>>>> If it's a temp view, no.
>>>>
>>>> If it's a permanent view, no. But *repeated* requests should be nearly
>>>> instantaneous.
>>>>
>>>> That is all.
>>>>
>>>> On Sat, Dec 20, 2008 at 11:04 PM, paul jobs <we...@gmail.com> wrote:
>>>>>
>>>>> map:
>>>>> function(doc) { emit(doc.zodiac, 1) }
>>>>> reduce:
>>>>> function(keys, values) { return sum(values)}
>>>>>
>>>>>
>>>>> Key
>>>>> Value *null*149877*"aquarius"*98747*"aries"*103072*"cancer"*104318*
>>>>> "capricorn"*97605*"gemini"*103708*"leo"*107982*"libra"*104955*"pisces"*
>>>>> 101741*"sagittarius"*92975
>>>>>
>>>>> total number of records ~2 million
>>>>>
>>>>> this query takes quite some time to run
>>>>> is there any way to make this query run faster
>>>>>
>>>>> thanks
>>>>>
>>>>
>>>
>
> Antony Blakey
> --------------------------
> CTO, Linkuistics Pty Ltd
> Ph: 0438 840 787
>
> Only two things are infinite, the universe and human stupidity, and I'm not
> sure about the former.
> -- Albert Einstein
>
>
Re: slow views
Posted by Antony Blakey <an...@gmail.com>.
On 21/12/2008, at 2:54 PM, Paul Davis wrote:
> View update times should be linear in terms of *new* records.
Well, with a factor to deal with the reduction, which isn't O(n).
>
>
> On Sat, Dec 20, 2008 at 11:20 PM, paul jobs <we...@gmail.com>
> wrote:
>> it is a permanent viewbut every repeat request takes time as new
>> users are
>> constantly added to the db
>>
>> On Sat, Dec 20, 2008 at 8:16 PM, Paul Davis <paul.joseph.davis@gmail.com
>> >wrote:
>>
>>> If it's a temp view, no.
>>>
>>> If it's a permanent view, no. But *repeated* requests should be
>>> nearly
>>> instantaneous.
>>>
>>> That is all.
>>>
>>> On Sat, Dec 20, 2008 at 11:04 PM, paul jobs <we...@gmail.com>
>>> wrote:
>>>> map:
>>>> function(doc) { emit(doc.zodiac, 1) }
>>>> reduce:
>>>> function(keys, values) { return sum(values)}
>>>>
>>>>
>>>> Key
>>>> Value
>>>> *null*149877*"aquarius"*98747*"aries"*103072*"cancer"*104318*
>>>> "capricorn
>>>> "*97605*"gemini"*103708*"leo"*107982*"libra"*104955*"pisces"*
>>>> 101741*"sagittarius"*92975
>>>>
>>>> total number of records ~2 million
>>>>
>>>> this query takes quite some time to run
>>>> is there any way to make this query run faster
>>>>
>>>> thanks
>>>>
>>>
>>
Antony Blakey
--------------------------
CTO, Linkuistics Pty Ltd
Ph: 0438 840 787
Only two things are infinite, the universe and human stupidity, and
I'm not sure about the former.
-- Albert Einstein
Re: slow views
Posted by Paul Davis <pa...@gmail.com>.
View update times should be linear in terms of *new* records.
On Sat, Dec 20, 2008 at 11:20 PM, paul jobs <we...@gmail.com> wrote:
> it is a permanent viewbut every repeat request takes time as new users are
> constantly added to the db
>
> On Sat, Dec 20, 2008 at 8:16 PM, Paul Davis <pa...@gmail.com>wrote:
>
>> If it's a temp view, no.
>>
>> If it's a permanent view, no. But *repeated* requests should be nearly
>> instantaneous.
>>
>> That is all.
>>
>> On Sat, Dec 20, 2008 at 11:04 PM, paul jobs <we...@gmail.com> wrote:
>> > map:
>> > function(doc) { emit(doc.zodiac, 1) }
>> > reduce:
>> > function(keys, values) { return sum(values)}
>> >
>> >
>> > Key
>> > Value *null*149877*"aquarius"*98747*"aries"*103072*"cancer"*104318*
>> > "capricorn"*97605*"gemini"*103708*"leo"*107982*"libra"*104955*"pisces"*
>> > 101741*"sagittarius"*92975
>> >
>> > total number of records ~2 million
>> >
>> > this query takes quite some time to run
>> > is there any way to make this query run faster
>> >
>> > thanks
>> >
>>
>
Re: slow views
Posted by paul jobs <we...@gmail.com>.
it is a permanent viewbut every repeat request takes time as new users are
constantly added to the db
On Sat, Dec 20, 2008 at 8:16 PM, Paul Davis <pa...@gmail.com>wrote:
> If it's a temp view, no.
>
> If it's a permanent view, no. But *repeated* requests should be nearly
> instantaneous.
>
> That is all.
>
> On Sat, Dec 20, 2008 at 11:04 PM, paul jobs <we...@gmail.com> wrote:
> > map:
> > function(doc) { emit(doc.zodiac, 1) }
> > reduce:
> > function(keys, values) { return sum(values)}
> >
> >
> > Key
> > Value *null*149877*"aquarius"*98747*"aries"*103072*"cancer"*104318*
> > "capricorn"*97605*"gemini"*103708*"leo"*107982*"libra"*104955*"pisces"*
> > 101741*"sagittarius"*92975
> >
> > total number of records ~2 million
> >
> > this query takes quite some time to run
> > is there any way to make this query run faster
> >
> > thanks
> >
>
Re: slow views
Posted by Paul Davis <pa...@gmail.com>.
If it's a temp view, no.
If it's a permanent view, no. But *repeated* requests should be nearly
instantaneous.
That is all.
On Sat, Dec 20, 2008 at 11:04 PM, paul jobs <we...@gmail.com> wrote:
> map:
> function(doc) { emit(doc.zodiac, 1) }
> reduce:
> function(keys, values) { return sum(values)}
>
>
> Key
> Value *null*149877*"aquarius"*98747*"aries"*103072*"cancer"*104318*
> "capricorn"*97605*"gemini"*103708*"leo"*107982*"libra"*104955*"pisces"*
> 101741*"sagittarius"*92975
>
> total number of records ~2 million
>
> this query takes quite some time to run
> is there any way to make this query run faster
>
> thanks
>