Reduces out of order?

[Charles Lehner <ce...@gmail.com>]

Hi,

I have a reduce function that is sensitive to row order. My understanding is that reduces are supposed to get their rows sorted by key. But this one is getting them out of order; in fact, the order changes inconsistently between replications.

I'm seeing this on both 1.0.2 and trunk. Is this the expected behavior?

Thanks,
Charles

Re: Reduces out of order?

[Dave Cottlehuber <da...@muse.net.nz>]

On 18 June 2011 14:07, Sean Copenhaver <se...@gmail.com> wrote:
> Hmmm... I don't think it's a good idea to have an order sensitive reduce function. Also I think you are thinking about the actual view index which is ordered by the key you emit from the map function.
>
> I believe couch will reduce each b-tree node worth of the view and store those, then rereduce those and parts of nodes as needed so your reduced queries only do part of the work.
>
> I hope that made some sense.
>
>
>
> On Jun 17, 2011, at 9:17 PM, Charles Lehner <ce...@gmail.com> wrote:
>
>> Hi,
>>
>> I have a reduce function that is sensitive to row order. My understanding is that reduces are supposed to get their rows sorted by key. But this one is getting them out of order; in fact, the order changes inconsistently between replications.
>>
>> I'm seeing this on both 1.0.2 and trunk. Is this the expected behavior?
>>
>> Thanks,
>> Charles
>

Sean's spot on; MR should not be sensitive to row order; that's kinda
the point of being able to divide and conquer arbitrarily is that the
result is identical and independent of the order of processing.

http://wiki.apache.org/couchdb/Introduction_to_CouchDB_views#Restrictions_on_map_and_reduce_functions

I'm not sure what you meant by between replications, but after more
data is added to the couch, impacted b-tree leaf and intermediate
nodes get re-written. As its append-only, this continues right to the
top of the tree. So your "row order" changes each time more data is
added, potentially.

Perhaps a list might be more appropriate for your post-processing
sorting? or client side?

A+
Dave

Re: Reduces out of order?

[Sean Copenhaver <se...@gmail.com>]

Hmmm... I don't think it's a good idea to have an order sensitive reduce function. Also I think you are thinking about the actual view index which is ordered by the key you emit from the map function.

I believe couch will reduce each b-tree node worth of the view and store those, then rereduce those and parts of nodes as needed so your reduced queries only do part of the work.

I hope that made some sense. 



On Jun 17, 2011, at 9:17 PM, Charles Lehner <ce...@gmail.com> wrote:

> Hi,
> 
> I have a reduce function that is sensitive to row order. My understanding is that reduces are supposed to get their rows sorted by key. But this one is getting them out of order; in fact, the order changes inconsistently between replications.
> 
> I'm seeing this on both 1.0.2 and trunk. Is this the expected behavior?
> 
> Thanks,
> Charles