You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by "Kropp, Henning" <hk...@microlution.de> on 2010/05/18 11:52:37 UTC

Multiple map reduce stages

Hi,

as far as I know working with map reduce commonly involves multiple map
and reduce stages. A view in couchdb solely consists of one map and if
necessary one reduce stage!? To have multiple map and reduce stages one
would have to conjunct views in couchdb!? How can I do that? Is it
possible to give the function(doc){..} another parameter? There is the
shows which have the extra parameter req for the http request.
Unfortunately my javascript knowledge of the underlaying Prototype
concept is not very funded which could be helpful here?

Kind regards and many thanks in advanced

Re: Multiple map reduce stages

Posted by Jarrod Roberson <ja...@vertigrated.com>.
On Tue, May 18, 2010 at 5:52 AM, Kropp, Henning <hk...@microlution.de>wrote:

> Hi,
>
> as far as I know working with map reduce commonly involves multiple map
> and reduce stages. A view in couchdb solely consists of one map and if
> necessary one reduce stage!?
>

actually it does a Reduce stage on each bucket of results from the b-tree
and then a re-reduce phase on all the results of the b-tree bucket results.
Your reduce function has to handle the initial reduce and the re-reduce
possibility as well. Personally I wish there were two functions one for
reduce and one for re-reduce, it would get make things clearer and easier to
manage as having to manage reducing the view results AND the reduced results
which may be a completely different data structure can be really confusing
and error prone in the maintenance phase of a project.



-- 
Jarrod Roberson

Re: Multiple map reduce stages

Posted by Robert Newson <ro...@gmail.com>.
If it helps, you can only group from the left side of the array. for
['a', 'b', 'c'], group_level=1 is  ['a'], group_level=2 is  ['a', 'b']
and group_level=3 is ['a', 'b', 'c'].

B.

On Thu, May 20, 2010 at 1:25 PM, Kropp, Henning <hk...@microlution.de> wrote:
> Am 20.05.2010 12:25, schrieb Simon Metson:
>> Hi,
>>
>>> I came up with the following solution. Grouping by value (uri) and time
>>> using the group_level=1 and the start and end key like follow:
>>>
>>> /_temp_view?group=true&group_level=1&startkey=[1270826004.0]&endkey=[{},1270826011.0]
>>>
>>>
>>> and simply counting
>>>
>>> function(doc) { emit([doc.URI,doc.Time], 1 );
>>
>> Shouldn't your start/end key also include the doc's URI? That way
>> you'd only get the URI you are interested in for the date range that
>> you specify. e.g. &startkey=[document1,
>> 1270826004.0]&endkey=[document1,1270826011.0]. If you switch the view
>> around to emit [doc.Time, doc.URI] as the key you can find all URI's
>> in the time range using &startkey=[1270826004.0,
>> 0]&endkey=[1270826011.0, {}].
>> Cheers
>> Simon
>
> That should work, but if I change the order of URI and Time in the key I
> am unable to group the result by URI using the group_level 1. And using
> group_level 2 would result in grouping URIs with the same URI *and*
> Time. Unfortunately there seems to be no way to tell exact what part of
> the key to use for grouping, if I am not mistaken. Therefor I have
> thought about a second map stage.
>
> Thank you!
>

Re: Multiple map reduce stages

Posted by "Kropp, Henning" <hk...@microlution.de>.
Am 20.05.2010 12:25, schrieb Simon Metson:
> Hi,
>
>> I came up with the following solution. Grouping by value (uri) and time
>> using the group_level=1 and the start and end key like follow:
>>
>> /_temp_view?group=true&group_level=1&startkey=[1270826004.0]&endkey=[{},1270826011.0]
>>
>>
>> and simply counting
>>
>> function(doc) { emit([doc.URI,doc.Time], 1 );
>
> Shouldn't your start/end key also include the doc's URI? That way
> you'd only get the URI you are interested in for the date range that
> you specify. e.g. &startkey=[document1,
> 1270826004.0]&endkey=[document1,1270826011.0]. If you switch the view
> around to emit [doc.Time, doc.URI] as the key you can find all URI's
> in the time range using &startkey=[1270826004.0,
> 0]&endkey=[1270826011.0, {}].
> Cheers
> Simon

That should work, but if I change the order of URI and Time in the key I
am unable to group the result by URI using the group_level 1. And using
group_level 2 would result in grouping URIs with the same URI *and*
Time. Unfortunately there seems to be no way to tell exact what part of
the key to use for grouping, if I am not mistaken. Therefor I have
thought about a second map stage.

Thank you!

Re: Multiple map reduce stages

Posted by Simon Metson <si...@googlemail.com>.
Hi,

> I came up with the following solution. Grouping by value (uri) and  
> time
> using the group_level=1 and the start and end key like follow:
>
> /_temp_view? 
> group=true&group_level=1&startkey=[1270826004.0]&endkey=[{}, 
> 1270826011.0]
>
> and simply counting
>
> function(doc) { emit([doc.URI,doc.Time], 1 );

Shouldn't your start/end key also include the doc's URI? That way  
you'd only get the URI you are interested in for the date range that  
you specify. e.g. &startkey=[document1,  
1270826004.0]&endkey=[document1,1270826011.0]. If you switch the view  
around to emit [doc.Time, doc.URI] as the key you can find all URI's  
in the time range using &startkey=[1270826004.0,  
0]&endkey=[1270826011.0, {}].
Cheers
Simon

Re: Multiple map reduce stages

Posted by "Kropp, Henning" <hk...@microlution.de>.
Am 20.05.2010 10:48, schrieb Kropp, Henning:
> Am 18.05.2010 20:16, schrieb J Chris Anderson:
>   
>> On May 18, 2010, at 2:52 AM, Kropp, Henning wrote:
>>
>>   
>>     
>>> Hi,
>>>
>>> as far as I know working with map reduce commonly involves multiple map
>>> and reduce stages. A view in couchdb solely consists of one map and if
>>> necessary one reduce stage!? To have multiple map and reduce stages one
>>> would have to conjunct views in couchdb!? How can I do that? Is it
>>> possible to give the function(doc){..} another parameter? There is the
>>> shows which have the extra parameter req for the http request.
>>> Unfortunately my javascript knowledge of the underlaying Prototype
>>> concept is not very funded which could be helpful here?
>>>
>>> Kind regards and many thanks in advanced
>>>     
>>>       
>> CouchDB Map Reduce is a realtime incremental model, so it is quite different from the Hadoop-style batch model. Of course you can still chain map reduce by copying the rows from a view query to a new db, and writing another view on the new db.
>>
>> Chris
>>     
> That is interesting to know. Hive adopts the batch model but obviously
> serves a different purpose.
>
> I was asking because of an actual problem I am having, maybe one can
> help. The problem I am having is that I would like to group documents by
> a value, but only those documents in a certain time interval. In this
> scenario couchdb is used for logging, which might not be a purpose
> couchdb initially is designed for.
>
> I came up with the following solution. Grouping by value (uri) and time
> using the group_level=1 and the start and end key like follow:
>
> /_temp_view?group=true&group_level=1&startkey=[1270826004.0]&endkey=[{},1270826011.0]
>
> and simply counting
>
> function(doc) { emit([doc.URI,doc.Time], 1 );
>
> Now experienced couchdb users might already see, that this results in
> all documents being grouped no difference of the time set in the start
> and end key. I needed some time to figure out why but finally realized
> the problem even so I can not explain it right and maybe I am totally
> wrong after all.
>
> So I thought I might help first mapping the documents by the time value
> and in a next step mapping and reducing it by the uri value. A different
> approach I came up with could be designing a 3 value for each document
> consisting of a conjunction of time and uri and working with that as the
> key!?
>
> Maybe and hopefully there is even a third approach I am not thinking of.
> I really appreciate the help.
>
> Thanks
>
>   

I forgot about the possibility to make an if statement in the map phase
like

if( start < doc.Time < end){ emit ... }

But I simply dont know and asked early on this list how to pass such a
parameter (end & start) to a permanent view / design document using
couchdb.js.

regards 


Re: Multiple map reduce stages

Posted by "Kropp, Henning" <hk...@microlution.de>.
Am 18.05.2010 20:16, schrieb J Chris Anderson:
> On May 18, 2010, at 2:52 AM, Kropp, Henning wrote:
>
>   
>> Hi,
>>
>> as far as I know working with map reduce commonly involves multiple map
>> and reduce stages. A view in couchdb solely consists of one map and if
>> necessary one reduce stage!? To have multiple map and reduce stages one
>> would have to conjunct views in couchdb!? How can I do that? Is it
>> possible to give the function(doc){..} another parameter? There is the
>> shows which have the extra parameter req for the http request.
>> Unfortunately my javascript knowledge of the underlaying Prototype
>> concept is not very funded which could be helpful here?
>>
>> Kind regards and many thanks in advanced
>>     
>
> CouchDB Map Reduce is a realtime incremental model, so it is quite different from the Hadoop-style batch model. Of course you can still chain map reduce by copying the rows from a view query to a new db, and writing another view on the new db.
>
> Chris

That is interesting to know. Hive adopts the batch model but obviously
serves a different purpose.

I was asking because of an actual problem I am having, maybe one can
help. The problem I am having is that I would like to group documents by
a value, but only those documents in a certain time interval. In this
scenario couchdb is used for logging, which might not be a purpose
couchdb initially is designed for.

I came up with the following solution. Grouping by value (uri) and time
using the group_level=1 and the start and end key like follow:

/_temp_view?group=true&group_level=1&startkey=[1270826004.0]&endkey=[{},1270826011.0]

and simply counting

function(doc) { emit([doc.URI,doc.Time], 1 );

Now experienced couchdb users might already see, that this results in
all documents being grouped no difference of the time set in the start
and end key. I needed some time to figure out why but finally realized
the problem even so I can not explain it right and maybe I am totally
wrong after all.

So I thought I might help first mapping the documents by the time value
and in a next step mapping and reducing it by the uri value. A different
approach I came up with could be designing a 3 value for each document
consisting of a conjunction of time and uri and working with that as the
key!?

Maybe and hopefully there is even a third approach I am not thinking of.
I really appreciate the help.

Thanks

 

 

Re: Multiple map reduce stages

Posted by J Chris Anderson <jc...@gmail.com>.
On May 18, 2010, at 2:52 AM, Kropp, Henning wrote:

> Hi,
> 
> as far as I know working with map reduce commonly involves multiple map
> and reduce stages. A view in couchdb solely consists of one map and if
> necessary one reduce stage!? To have multiple map and reduce stages one
> would have to conjunct views in couchdb!? How can I do that? Is it
> possible to give the function(doc){..} another parameter? There is the
> shows which have the extra parameter req for the http request.
> Unfortunately my javascript knowledge of the underlaying Prototype
> concept is not very funded which could be helpful here?
> 
> Kind regards and many thanks in advanced


CouchDB Map Reduce is a realtime incremental model, so it is quite different from the Hadoop-style batch model. Of course you can still chain map reduce by copying the rows from a view query to a new db, and writing another view on the new db.

Chris