You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@couchdb.apache.org by Paul Carey <pa...@gmail.com> on 2009/08/16 17:49:14 UTC
Re: reduce_limit error
On Tue, May 5, 2009 at 8:50 PM, Brian Candler<B....@pobox.com> wrote:
> On Mon, May 04, 2009 at 03:08:38PM -0700, Chris Anderson wrote:
>> I'm checking in a patch that should cut down on the number of mailing
>> list questions asking why a particular reduce function is hella slow.
>> Essentially the patch throws an error if the reduce function return
>> value is not at least half the size of the values array that was
>> passed in. (The check is skipped if the size is below a fixed amount,
>> 200 bytes for now).
>
> I think that 200 byte limit is too low, as I have now had to turn off the
> reduce_limit on my server for this:
I'm using a reduce function to sort data in so that clients can query
for the most recent piece of data. For example
function most_recent_reading-map(doc) {
if(doc.type === "TemperatureReading") {
emit(doc.station_id, doc);
}
}
function most_recent_reading-reduce(keys, values) {
var sorted = values.sort(function (a,b) {
return b.created_at.localeCompare(a.created_at)
});
return sorted[0];
}
The main reason I might to do this is to simplify client logic, but
another valid reason is to prevent sending and processing
unnecessarily large chunks of JSON.
This kind of reduce function may fall foul of the
reduce_overflow_error, but only if the document is greater than 200
bytes. So, I'm echoing the opinion that 200 bytes is too low. I also
believe that throwing an exception is a bit draconian as it could
result in an unjustified failure in production. I think a warning
would be more appropriate.
Paul
Re: reduce_limit error
Posted by Chris Anderson <jc...@apache.org>.
On Tue, Aug 18, 2009 at 2:34 AM, Brian Candler<B....@pobox.com> wrote:
> On Sun, Aug 16, 2009 at 05:55:56PM +0100, Robert Newson wrote:
>> You can turn it off (caveat emptor);
>>
>> ; Changing reduce_limit to false will disable reduce_limit.
>> ; If you think you're hitting reduce_limit with a "good" reduce function,
>> ; please let us know on the mailing list so we can fine tune the heuristic.
>> [query_server_config]
>> reduce_limit = true
>
> I think that it should be a limit in bytes, not a true/false. I have also
> had cases where I generate a limited-sized reduce value which is a bit
> larger than 200 bytes.
>
Oh I see that now - returning the first item in the list makes more sense.
As far as a byte limit for the threshold, that would be entirely
possible and not that hard to patch. I'd love to see a patch along
these lines. I like the suggestion to make the threshold configurable
instead of just an on/off switch.
Chris
--
Chris Anderson
http://jchrisa.net
http://couch.io
Re: reduce_limit error
Posted by Brian Candler <B....@pobox.com>.
On Sun, Aug 16, 2009 at 05:55:56PM +0100, Robert Newson wrote:
> You can turn it off (caveat emptor);
>
> ; Changing reduce_limit to false will disable reduce_limit.
> ; If you think you're hitting reduce_limit with a "good" reduce function,
> ; please let us know on the mailing list so we can fine tune the heuristic.
> [query_server_config]
> reduce_limit = true
I think that it should be a limit in bytes, not a true/false. I have also
had cases where I generate a limited-sized reduce value which is a bit
larger than 200 bytes.
Re: reduce_limit error
Posted by Robert Newson <ro...@gmail.com>.
You can turn it off (caveat emptor);
; Changing reduce_limit to false will disable reduce_limit.
; If you think you're hitting reduce_limit with a "good" reduce function,
; please let us know on the mailing list so we can fine tune the heuristic.
[query_server_config]
reduce_limit = true
B.
On Sun, Aug 16, 2009 at 4:49 PM, Paul Carey<pa...@gmail.com> wrote:
> On Tue, May 5, 2009 at 8:50 PM, Brian Candler<B....@pobox.com> wrote:
>> On Mon, May 04, 2009 at 03:08:38PM -0700, Chris Anderson wrote:
>>> I'm checking in a patch that should cut down on the number of mailing
>>> list questions asking why a particular reduce function is hella slow.
>>> Essentially the patch throws an error if the reduce function return
>>> value is not at least half the size of the values array that was
>>> passed in. (The check is skipped if the size is below a fixed amount,
>>> 200 bytes for now).
>>
>> I think that 200 byte limit is too low, as I have now had to turn off the
>> reduce_limit on my server for this:
>
> I'm using a reduce function to sort data in so that clients can query
> for the most recent piece of data. For example
>
> function most_recent_reading-map(doc) {
> if(doc.type === "TemperatureReading") {
> emit(doc.station_id, doc);
> }
> }
>
> function most_recent_reading-reduce(keys, values) {
> var sorted = values.sort(function (a,b) {
> return b.created_at.localeCompare(a.created_at)
> });
> return sorted[0];
> }
>
> The main reason I might to do this is to simplify client logic, but
> another valid reason is to prevent sending and processing
> unnecessarily large chunks of JSON.
>
> This kind of reduce function may fall foul of the
> reduce_overflow_error, but only if the document is greater than 200
> bytes. So, I'm echoing the opinion that 200 bytes is too low. I also
> believe that throwing an exception is a bit draconian as it could
> result in an unjustified failure in production. I think a warning
> would be more appropriate.
>
> Paul
>
Re: reduce_limit error
Posted by Brian Candler <B....@pobox.com>.
On Sun, Aug 16, 2009 at 11:18:32AM -0700, Chris Anderson wrote:
> you should never accumulate a list in a reduce function...
He's not. He's just returning a single document, which is the one with the
largest value of created_at.
IMO that's a pretty reasonable thing to do, because it's not unbounded
expansion: the largest reduce value will be the largest single document in
the database. Note that we explicitly say it's OK to emit an entire document
as a value in a k/v index, i.e. emit(null, doc), and his reduce function is
just picking one of those 'values'.
Regards,
Brian.
Re: reduce_limit error
Posted by Paul Carey <pa...@gmail.com>.
>> function most_recent_reading-map(doc) {
>> if(doc.type === "TemperatureReading") {
>> emit(doc.station_id, doc);
>> }
>> }
>>
>> function most_recent_reading-reduce(keys, values) {
>> var sorted = values.sort(function (a,b) {
>> return b.created_at.localeCompare(a.created_at)
>> });
>> return sorted[0];
>> }
>>
>
> you should never accumulate a list in a reduce function...
>
> if you want to create a compressed final JSON output, the thing to do
> would be to run a list function on a group reduce query, and have it
> make the final aggregate. that way you don't end up with an infinitely
> long overflowing list in your reduce values.
Given that the reduce function returns a single value, I don't
understand why you consider it to be accumulating a list. I see it as
being roughly equivalent to returning a very large scalar.
As the argument against having the output of a reduce function grow
too fast is based on degraded performance with a large dataset, I ran
a test case with 1 million docs. The query returns in about 0.03s,
which is significantly faster than a group_level based query against a
dataset of a similar size.
Paul
Re: reduce_limit error
Posted by Chris Anderson <jc...@apache.org>.
On Sun, Aug 16, 2009 at 8:49 AM, Paul Carey<pa...@gmail.com> wrote:
> On Tue, May 5, 2009 at 8:50 PM, Brian Candler<B....@pobox.com> wrote:
>> On Mon, May 04, 2009 at 03:08:38PM -0700, Chris Anderson wrote:
>>> I'm checking in a patch that should cut down on the number of mailing
>>> list questions asking why a particular reduce function is hella slow.
>>> Essentially the patch throws an error if the reduce function return
>>> value is not at least half the size of the values array that was
>>> passed in. (The check is skipped if the size is below a fixed amount,
>>> 200 bytes for now).
>>
>> I think that 200 byte limit is too low, as I have now had to turn off the
>> reduce_limit on my server for this:
>
> I'm using a reduce function to sort data in so that clients can query
> for the most recent piece of data. For example
>
> function most_recent_reading-map(doc) {
> if(doc.type === "TemperatureReading") {
> emit(doc.station_id, doc);
> }
> }
>
> function most_recent_reading-reduce(keys, values) {
> var sorted = values.sort(function (a,b) {
> return b.created_at.localeCompare(a.created_at)
> });
> return sorted[0];
> }
>
you should never accumulate a list in a reduce function...
if you want to create a compressed final JSON output, the thing to do
would be to run a list function on a group reduce query, and have it
make the final aggregate. that way you don't end up with an infinitely
long overflowing list in your reduce values.
> The main reason I might to do this is to simplify client logic, but
> another valid reason is to prevent sending and processing
> unnecessarily large chunks of JSON.
>
> This kind of reduce function may fall foul of the
> reduce_overflow_error, but only if the document is greater than 200
> bytes. So, I'm echoing the opinion that 200 bytes is too low. I also
> believe that throwing an exception is a bit draconian as it could
> result in an unjustified failure in production. I think a warning
> would be more appropriate.
>
> Paul
>
--
Chris Anderson
http://jchrisa.net
http://couch.io