You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Dominic Tarr <do...@gmail.com> on 2011/12/14 05:46:21 UTC
iterative map reduce
hi,
I need a iterative map reduce, prefurably, in couchdb.
I want to be able to do a map reduce to generate unique items, then another
map reduce to generate stats about those unique items.
from what I know about couchdb at the moment, it seems like id have to do
the first step at the document creation step, which is just ugly.
has anyone else dealt with this problem?
Re: iterative map reduce
Posted by Alon Keren <al...@gmail.com>.
Dave, if you need chaining, I really wouldn't mind anyone testing
CouchIncarnate and reporting bugs :)
On 14 December 2011 11:16, Dave Cottlehuber <da...@muse.net.nz> wrote:
> On 14 December 2011 10:05, Robert Newson <ro...@gmail.com> wrote:
> > Chained map-reduce is not available in the open source BigCouch
> > project, only on our hosted service at cloudant.com. Sorry!
> >
> > B.
>
> A cryin' shame!!!
>
> > On 14 December 2011 09:01, Alon Keren <al...@gmail.com> wrote:
> >> I've been working on an open-source tool with a similar approach
> chaining
> >> map-reduce:
> >> https://github.com/afters/Couch-Incarnate
> >>
> >> The need to pay the bills withheld me from putting more time into it
> >> lately, so I can't say it's production quality.
> >> Hopefully, this will change soon.
> >>
> >> Dominic - the method I currently use in production is to poll the
> changes
> >> on my DB every few seconds and update a 'summary' DB accordingly. It's
> not
> >> pretty, but it works. I hope to replace it with Couch-Incarnate asap.
> >>
> >> Alon
> >>
> >> On 14 December 2011 10:50, Dave Cottlehuber <da...@muse.net.nz> wrote:
> >>
> >>> On 14 December 2011 05:46, Dominic Tarr <do...@gmail.com>
> wrote:
> >>> > hi,
> >>> >
> >>> > I need a iterative map reduce, prefurably, in couchdb.
> >>> >
> >>> > I want to be able to do a map reduce to generate unique items, then
> >>> another
> >>> > map reduce to generate stats about those unique items.
> >>> >
> >>> > from what I know about couchdb at the moment, it seems like id have
> to do
> >>> > the first step at the document creation step, which is just ugly.
> >>> >https://mail.google.com/mail/?shva=1#inbox
> >>> > has anyone else dealt with this problem?
> >>>
> >>> Hi Dom,
> >>>
> >>> Cloudant do this in BigCouch which they released as open source a
> >>> while back
> http://support.cloudant.com/kb/views/chained-mapreduce-views
> >>> there are a few other posts on chained MR around but I don't know of a
> >>> better Couch-style implementation. Their basic approach is to copy the
> >>> view output into a new db which you'd then re-MR as required. Rinse
> >>> and repeat until complete.
> >>>
> >>> Depending on your data it might be possible to implement their
> >>> view->copy to db -> new view approach yourself in native couch. It
> >>> would be interesting to do this as a couch extension using that nifty
> >>> plugin interface that hasn't been developed yet!
> >>>
> >>> A+
> >>> Dave
> >>>
>
Re: iterative map reduce
Posted by Dave Cottlehuber <da...@muse.net.nz>.
On 14 December 2011 10:05, Robert Newson <ro...@gmail.com> wrote:
> Chained map-reduce is not available in the open source BigCouch
> project, only on our hosted service at cloudant.com. Sorry!
>
> B.
A cryin' shame!!!
> On 14 December 2011 09:01, Alon Keren <al...@gmail.com> wrote:
>> I've been working on an open-source tool with a similar approach chaining
>> map-reduce:
>> https://github.com/afters/Couch-Incarnate
>>
>> The need to pay the bills withheld me from putting more time into it
>> lately, so I can't say it's production quality.
>> Hopefully, this will change soon.
>>
>> Dominic - the method I currently use in production is to poll the changes
>> on my DB every few seconds and update a 'summary' DB accordingly. It's not
>> pretty, but it works. I hope to replace it with Couch-Incarnate asap.
>>
>> Alon
>>
>> On 14 December 2011 10:50, Dave Cottlehuber <da...@muse.net.nz> wrote:
>>
>>> On 14 December 2011 05:46, Dominic Tarr <do...@gmail.com> wrote:
>>> > hi,
>>> >
>>> > I need a iterative map reduce, prefurably, in couchdb.
>>> >
>>> > I want to be able to do a map reduce to generate unique items, then
>>> another
>>> > map reduce to generate stats about those unique items.
>>> >
>>> > from what I know about couchdb at the moment, it seems like id have to do
>>> > the first step at the document creation step, which is just ugly.
>>> >https://mail.google.com/mail/?shva=1#inbox
>>> > has anyone else dealt with this problem?
>>>
>>> Hi Dom,
>>>
>>> Cloudant do this in BigCouch which they released as open source a
>>> while back http://support.cloudant.com/kb/views/chained-mapreduce-views
>>> there are a few other posts on chained MR around but I don't know of a
>>> better Couch-style implementation. Their basic approach is to copy the
>>> view output into a new db which you'd then re-MR as required. Rinse
>>> and repeat until complete.
>>>
>>> Depending on your data it might be possible to implement their
>>> view->copy to db -> new view approach yourself in native couch. It
>>> would be interesting to do this as a couch extension using that nifty
>>> plugin interface that hasn't been developed yet!
>>>
>>> A+
>>> Dave
>>>
Re: iterative map reduce
Posted by Robert Newson <ro...@gmail.com>.
Chained map-reduce is not available in the open source BigCouch
project, only on our hosted service at cloudant.com. Sorry!
B.
On 14 December 2011 09:01, Alon Keren <al...@gmail.com> wrote:
> I've been working on an open-source tool with a similar approach chaining
> map-reduce:
> https://github.com/afters/Couch-Incarnate
>
> The need to pay the bills withheld me from putting more time into it
> lately, so I can't say it's production quality.
> Hopefully, this will change soon.
>
> Dominic - the method I currently use in production is to poll the changes
> on my DB every few seconds and update a 'summary' DB accordingly. It's not
> pretty, but it works. I hope to replace it with Couch-Incarnate asap.
>
> Alon
>
> On 14 December 2011 10:50, Dave Cottlehuber <da...@muse.net.nz> wrote:
>
>> On 14 December 2011 05:46, Dominic Tarr <do...@gmail.com> wrote:
>> > hi,
>> >
>> > I need a iterative map reduce, prefurably, in couchdb.
>> >
>> > I want to be able to do a map reduce to generate unique items, then
>> another
>> > map reduce to generate stats about those unique items.
>> >
>> > from what I know about couchdb at the moment, it seems like id have to do
>> > the first step at the document creation step, which is just ugly.
>> >https://mail.google.com/mail/?shva=1#inbox
>> > has anyone else dealt with this problem?
>>
>> Hi Dom,
>>
>> Cloudant do this in BigCouch which they released as open source a
>> while back http://support.cloudant.com/kb/views/chained-mapreduce-views
>> there are a few other posts on chained MR around but I don't know of a
>> better Couch-style implementation. Their basic approach is to copy the
>> view output into a new db which you'd then re-MR as required. Rinse
>> and repeat until complete.
>>
>> Depending on your data it might be possible to implement their
>> view->copy to db -> new view approach yourself in native couch. It
>> would be interesting to do this as a couch extension using that nifty
>> plugin interface that hasn't been developed yet!
>>
>> A+
>> Dave
>>
Re: iterative map reduce
Posted by Alon Keren <al...@gmail.com>.
I've been working on an open-source tool with a similar approach chaining
map-reduce:
https://github.com/afters/Couch-Incarnate
The need to pay the bills withheld me from putting more time into it
lately, so I can't say it's production quality.
Hopefully, this will change soon.
Dominic - the method I currently use in production is to poll the changes
on my DB every few seconds and update a 'summary' DB accordingly. It's not
pretty, but it works. I hope to replace it with Couch-Incarnate asap.
Alon
On 14 December 2011 10:50, Dave Cottlehuber <da...@muse.net.nz> wrote:
> On 14 December 2011 05:46, Dominic Tarr <do...@gmail.com> wrote:
> > hi,
> >
> > I need a iterative map reduce, prefurably, in couchdb.
> >
> > I want to be able to do a map reduce to generate unique items, then
> another
> > map reduce to generate stats about those unique items.
> >
> > from what I know about couchdb at the moment, it seems like id have to do
> > the first step at the document creation step, which is just ugly.
> >https://mail.google.com/mail/?shva=1#inbox
> > has anyone else dealt with this problem?
>
> Hi Dom,
>
> Cloudant do this in BigCouch which they released as open source a
> while back http://support.cloudant.com/kb/views/chained-mapreduce-views
> there are a few other posts on chained MR around but I don't know of a
> better Couch-style implementation. Their basic approach is to copy the
> view output into a new db which you'd then re-MR as required. Rinse
> and repeat until complete.
>
> Depending on your data it might be possible to implement their
> view->copy to db -> new view approach yourself in native couch. It
> would be interesting to do this as a couch extension using that nifty
> plugin interface that hasn't been developed yet!
>
> A+
> Dave
>
Re: iterative map reduce
Posted by Dave Cottlehuber <da...@muse.net.nz>.
On 14 December 2011 05:46, Dominic Tarr <do...@gmail.com> wrote:
> hi,
>
> I need a iterative map reduce, prefurably, in couchdb.
>
> I want to be able to do a map reduce to generate unique items, then another
> map reduce to generate stats about those unique items.
>
> from what I know about couchdb at the moment, it seems like id have to do
> the first step at the document creation step, which is just ugly.
>https://mail.google.com/mail/?shva=1#inbox
> has anyone else dealt with this problem?
Hi Dom,
Cloudant do this in BigCouch which they released as open source a
while back http://support.cloudant.com/kb/views/chained-mapreduce-views
there are a few other posts on chained MR around but I don't know of a
better Couch-style implementation. Their basic approach is to copy the
view output into a new db which you'd then re-MR as required. Rinse
and repeat until complete.
Depending on your data it might be possible to implement their
view->copy to db -> new view approach yourself in native couch. It
would be interesting to do this as a couch extension using that nifty
plugin interface that hasn't been developed yet!
A+
Dave