You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Zachary Zolton <za...@gmail.com> on 2009/08/27 20:06:52 UTC

0.9 to 0.10 breaking changes

Ohai,

I'm prepping my app for the 0.10 release, and on the Breaking Changes
wiki page, I came across this:

Stricter enforcing of reduciness in reduce-functions
Reduce functions are now required to reduce the number of values for a key.

Can someone tell me what this means?

(Also, I'm considering adding some kind of warning in there about
reduce=false on map-only views, since I was bitten by that change...)


Cheers,

Zach

Re: 0.9 to 0.10 breaking changes

Posted by Paul Davis <pa...@gmail.com>.
> This was my first approach and in general it works for the purpose.
> The only problem is when the features are not independent one of each
> other. I guess that in that case the only approach is to download all
> the [k1, features] and handle them in the client (same problem as
> sorting reduce results).

Sounds like it. I don't have any other ideas off the top of my head.
There's always the possibility of writing an external indexer to do
your logic client side if your deployment picture allows for it.

HTH,
Paul Davis

Re: 0.9 to 0.10 breaking changes

Posted by Fabio Forno <fa...@gmail.com>.
On Fri, Aug 28, 2009 at 9:55 PM, Paul Davis<pa...@gmail.com> wrote:
>> in this case I'm reducing the result, but the output is greater the
>> half the size of the input. For what I've understood this would fail,
>> wouldn't it?
>
> Yep. And this is a poster child of the sort of thing this patch was
> trying to catch. The CouchDB-y way to do this would be to emit like
> such:
>
> Doc1
> [k1, id(F1)] -> F1
> [k1, id(F2)] -> F2
> [k1, id(F3)] -> F3
> [k1, id(F4)] -> F4
> [k1, id(F5)] -> F5
>
> Doc2
> [k1, id(F1)] -> F1
> [k1, id(F3)] -> F3
>
> And then call your reduce with a group=true.
>

This was my first approach and in general it works for the purpose.
The only problem is when the features are not independent one of each
other. I guess that in that case the only approach is to download all
the [k1, features] and handle them in the client (same problem as
sorting reduce results).


-- 
Fabio Forno, Ph.D.
Bluendo srl http://www.bluendo.com
jabber id: ff@jabber.bluendo.com

Re: 0.9 to 0.10 breaking changes

Posted by Paul Davis <pa...@gmail.com>.
> in this case I'm reducing the result, but the output is greater the
> half the size of the input. For what I've understood this would fail,
> wouldn't it?

Yep. And this is a poster child of the sort of thing this patch was
trying to catch. The CouchDB-y way to do this would be to emit like
such:

Doc1
[k1, id(F1)] -> F1
[k1, id(F2)] -> F2
[k1, id(F3)] -> F3
[k1, id(F4)] -> F4
[k1, id(F5)] -> F5

Doc2
[k1, id(F1)] -> F1
[k1, id(F3)] -> F3

And then call your reduce with a group=true.

HTH,
Paul Davis

Re: 0.9 to 0.10 breaking changes

Posted by Fabio Forno <fa...@gmail.com>.
On Fri, Aug 28, 2009 at 5:47 PM, Paul Davis<pa...@gmail.com> wrote:
>>
>> Why less then half? Suppose I must merge two docs A,B where I'm
>> counting the occurrences of a set of items ans size(A) >> size(B),
>> size(out) would be less than size(input), but more than half of it
>>
>
> I don't follow. Can you paste some pseudo code?
>

Perhaps I'm thinking of a case where reduce is not supposed to work in
that way.

I've a map producing these intermediate documents

key -> set of features

then the reduce takes the set of features and does the merge.

So let's have after the map:
k1 -> {F1, F2, F3, F4, F5}
k1 -> {F1, F3}

the reduce is called with [{F1, F2}, {F1, F2, F3, F4, F5}], for which I return
{F1', F2, F3', F4, F5} (the features are not simple keys, but are
complex objects which are transformed during reduce)

in this case I'm reducing the result, but the output is greater the
half the size of the input. For what I've understood this would fail,
wouldn't it?

-- 
Fabio Forno, Ph.D.
Bluendo srl http://www.bluendo.com
jabber id: ff@jabber.bluendo.com

Re: 0.9 to 0.10 breaking changes

Posted by Paul Davis <pa...@gmail.com>.
On Fri, Aug 28, 2009 at 11:41 AM, Fabio Forno<fa...@gmail.com> wrote:
> On Fri, Aug 28, 2009 at 4:45 PM, Paul Davis<pa...@gmail.com> wrote:
>> By default, reduce output must be less than half the size of it's
>> input when. Input and output data sizes are measured by the length of
>> their serialized JSON representations. This check is only made when
>> the length of the serialized output is greater than 200 characters.
>>
>
> Why less then half? Suppose I must merge two docs A,B where I'm
> counting the occurrences of a set of items ans size(A) >> size(B),
> size(out) would be less than size(input), but more than half of it
>

I don't follow. Can you paste some pseudo code?

> --
> Fabio Forno, Ph.D.
> Bluendo srl http://www.bluendo.com
> jabber id: ff@jabber.bluendo.com
>

Re: 0.9 to 0.10 breaking changes

Posted by Paul Davis <pa...@gmail.com>.
> Really the requirement should be less than 1/10th or so (decrease at
> roughly log(N) rate) but I put in 1/2 because it's just a rough rubric
> to catch the worst offenders.

Actually, the really real requirement I think should be
1/values.length because there can be weird cases in the btree
reductions. I never tracked them down, but I ran into it when doing
the Top-N-Tags examples.

HTH,
Paul Davis

Re: 0.9 to 0.10 breaking changes

Posted by Chris Anderson <jc...@apache.org>.
On Fri, Aug 28, 2009 at 8:41 AM, Fabio Forno<fa...@gmail.com> wrote:
> On Fri, Aug 28, 2009 at 4:45 PM, Paul Davis<pa...@gmail.com> wrote:
>> By default, reduce output must be less than half the size of it's
>> input when. Input and output data sizes are measured by the length of
>> their serialized JSON representations. This check is only made when
>> the length of the serialized output is greater than 200 characters.
>>
>
> Why less then half? Suppose I must merge two docs A,B where I'm
> counting the occurrences of a set of items ans size(A) >> size(B),
> size(out) would be less than size(input), but more than half of it
>

Really the requirement should be less than 1/10th or so (decrease at
roughly log(N) rate) but I put in 1/2 because it's just a rough rubric
to catch the worst offenders.

> --
> Fabio Forno, Ph.D.
> Bluendo srl http://www.bluendo.com
> jabber id: ff@jabber.bluendo.com
>



-- 
Chris Anderson
http://jchrisa.net
http://couch.io

Re: 0.9 to 0.10 breaking changes

Posted by Fabio Forno <fa...@gmail.com>.
On Fri, Aug 28, 2009 at 4:45 PM, Paul Davis<pa...@gmail.com> wrote:
> By default, reduce output must be less than half the size of it's
> input when. Input and output data sizes are measured by the length of
> their serialized JSON representations. This check is only made when
> the length of the serialized output is greater than 200 characters.
>

Why less then half? Suppose I must merge two docs A,B where I'm
counting the occurrences of a set of items ans size(A) >> size(B),
size(out) would be less than size(input), but more than half of it

-- 
Fabio Forno, Ph.D.
Bluendo srl http://www.bluendo.com
jabber id: ff@jabber.bluendo.com

Re: 0.9 to 0.10 breaking changes

Posted by Paul Davis <pa...@gmail.com>.
By default, reduce output must be less than half the size of it's
input when. Input and output data sizes are measured by the length of
their serialized JSON representations. This check is only made when
the length of the serialized output is greater than 200 characters.

HTH,
Paul Davis

On Fri, Aug 28, 2009 at 10:38 AM, Zachary
Zolton<za...@gmail.com> wrote:
> So, if I've got this right, we're not really checking the "number of
> values for a key", but instead checking whether the size of the
> reduced JSON string representation is smaller than twice the size of
> the its input line of JSON.
>
> LOL there's got to less wordy version of that I could put into the wiki. Hmm...
>
> On Thu, Aug 27, 2009 at 5:26 PM, Chris Anderson<jc...@apache.org> wrote:
>> On Thu, Aug 27, 2009 at 12:13 PM, Paul Davis<pa...@gmail.com> wrote:
>>> On Thu, Aug 27, 2009 at 3:08 PM, Adam Wolff<aw...@gmail.com> wrote:
>>>> How is reduce size computed? Is it strictly number of bytes?
>>>> Thanks,
>>>> A
>>>
>>> The basic algorithm is:
>>>
>>> lenght1 = JSON.stringify(input).length;
>>> length2 = JSON.stringify(output).length
>>> if(enforce_reduce_limit && length2 > 200 && (length2 * 2) > length1)
>>>    throw(reduce_overflow_error);
>>>
>>> Check line 49-55 or so in ./share/server/view.js
>>>
>>
>> There are a million ways this could be made more subtle. Also, as
>> mentioned on another thread, it'd be nice to have the 200 byte
>> threshold configurable, instead of just a boolean case.
>>
>> Chris
>>
>>
>> --
>> Chris Anderson
>> http://jchrisa.net
>> http://couch.io
>>
>

Re: 0.9 to 0.10 breaking changes

Posted by Zachary Zolton <za...@gmail.com>.
So, if I've got this right, we're not really checking the "number of
values for a key", but instead checking whether the size of the
reduced JSON string representation is smaller than twice the size of
the its input line of JSON.

LOL there's got to less wordy version of that I could put into the wiki. Hmm...

On Thu, Aug 27, 2009 at 5:26 PM, Chris Anderson<jc...@apache.org> wrote:
> On Thu, Aug 27, 2009 at 12:13 PM, Paul Davis<pa...@gmail.com> wrote:
>> On Thu, Aug 27, 2009 at 3:08 PM, Adam Wolff<aw...@gmail.com> wrote:
>>> How is reduce size computed? Is it strictly number of bytes?
>>> Thanks,
>>> A
>>
>> The basic algorithm is:
>>
>> lenght1 = JSON.stringify(input).length;
>> length2 = JSON.stringify(output).length
>> if(enforce_reduce_limit && length2 > 200 && (length2 * 2) > length1)
>>    throw(reduce_overflow_error);
>>
>> Check line 49-55 or so in ./share/server/view.js
>>
>
> There are a million ways this could be made more subtle. Also, as
> mentioned on another thread, it'd be nice to have the 200 byte
> threshold configurable, instead of just a boolean case.
>
> Chris
>
>
> --
> Chris Anderson
> http://jchrisa.net
> http://couch.io
>

Re: 0.9 to 0.10 breaking changes

Posted by Chris Anderson <jc...@apache.org>.
On Thu, Aug 27, 2009 at 12:13 PM, Paul Davis<pa...@gmail.com> wrote:
> On Thu, Aug 27, 2009 at 3:08 PM, Adam Wolff<aw...@gmail.com> wrote:
>> How is reduce size computed? Is it strictly number of bytes?
>> Thanks,
>> A
>
> The basic algorithm is:
>
> lenght1 = JSON.stringify(input).length;
> length2 = JSON.stringify(output).length
> if(enforce_reduce_limit && length2 > 200 && (length2 * 2) > length1)
>    throw(reduce_overflow_error);
>
> Check line 49-55 or so in ./share/server/view.js
>

There are a million ways this could be made more subtle. Also, as
mentioned on another thread, it'd be nice to have the 200 byte
threshold configurable, instead of just a boolean case.

Chris


-- 
Chris Anderson
http://jchrisa.net
http://couch.io

Re: 0.9 to 0.10 breaking changes

Posted by Paul Davis <pa...@gmail.com>.
On Thu, Aug 27, 2009 at 3:08 PM, Adam Wolff<aw...@gmail.com> wrote:
> How is reduce size computed? Is it strictly number of bytes?
> Thanks,
> A

The basic algorithm is:

lenght1 = JSON.stringify(input).length;
length2 = JSON.stringify(output).length
if(enforce_reduce_limit && length2 > 200 && (length2 * 2) > length1)
    throw(reduce_overflow_error);

Check line 49-55 or so in ./share/server/view.js

HTH,
Paul Davis

Re: 0.9 to 0.10 breaking changes

Posted by Adam Wolff <aw...@gmail.com>.
How is reduce size computed? Is it strictly number of bytes?
Thanks,
A

On Thu, Aug 27, 2009 at 11:11 AM, Paul Davis <pa...@gmail.com>wrote:

> On Thu, Aug 27, 2009 at 2:06 PM, Zachary Zolton<za...@gmail.com>
> wrote:
> > Ohai,
> >
> > I'm prepping my app for the 0.10 release, and on the Breaking Changes
> > wiki page, I came across this:
> >
> > Stricter enforcing of reduciness in reduce-functions
> > Reduce functions are now required to reduce the number of values for a
> key.
> >
> > Can someone tell me what this means?
>
> By default, reduces are checked for exploding data. Basically, if the
> input to the reduce doesn't shrink then it starts throwing errors.
> This was to combat the once common, "I don't understand why it works
> with 10 documents, but not 100,000" threads.
>
> >
> > (Also, I'm considering adding some kind of warning in there about
> > reduce=false on map-only views, since I was bitten by that change...)
> >
>
> Feel free to add anything as you see fit.
>
> >
> > Cheers,
> >
> > Zach
> >
>
> HTH,
> Paul Davis
>

Re: 0.9 to 0.10 breaking changes

Posted by Paul Davis <pa...@gmail.com>.
On Thu, Aug 27, 2009 at 2:06 PM, Zachary Zolton<za...@gmail.com> wrote:
> Ohai,
>
> I'm prepping my app for the 0.10 release, and on the Breaking Changes
> wiki page, I came across this:
>
> Stricter enforcing of reduciness in reduce-functions
> Reduce functions are now required to reduce the number of values for a key.
>
> Can someone tell me what this means?

By default, reduces are checked for exploding data. Basically, if the
input to the reduce doesn't shrink then it starts throwing errors.
This was to combat the once common, "I don't understand why it works
with 10 documents, but not 100,000" threads.

>
> (Also, I'm considering adding some kind of warning in there about
> reduce=false on map-only views, since I was bitten by that change...)
>

Feel free to add anything as you see fit.

>
> Cheers,
>
> Zach
>

HTH,
Paul Davis