You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@couchdb.apache.org by Li Zhengji <zh...@gmail.com> on 2009/11/18 10:17:22 UTC
some thoughts on view
Hi all,
I am planning to use CouchDB now, and coming into some thoughts today,
which may be silly.
1. One map, multiple reduces.
There are possibilities that multiple views share the same map
function. Then, I think these views could be combined, sharing the
same map function. The outcome is obvious: duplicated operations on
the same doc are eliminated.
2. Why "reduce", let's "fold"
The reduce function is a little ugly due to the existance of parameter
"rereduce".
The reduce process is much like the "fold" function in functional
programming languages, except that the function to be reduced is
required to be assocative and commutative (then, the reduce process
could be solved/implemented in map/reduce manner, in parallel, and
incrementally).
It's better to introduce an "Acc" parameter and remove the "rereduce"
flag (like fold), then the reduce function would be simpler:
fun (keys, values, Acc) -> updated Acc.
We just need to attach an "Acc" field for reduce in view doc.
--
Li Zhengji
Re: some thoughts on view
Posted by Chris Anderson <jc...@apache.org>.
On Thu, Nov 19, 2009 at 7:06 PM, Li Zhengji <zh...@gmail.com> wrote:
> Hi Jan,
>
> On Thu, Nov 19, 2009 at 9:50 PM, Jan Lehnardt <ja...@apache.org> wrote:
>>
>> Why is the current implementation not a good idea? It works well :)
>>
>> Cheers
>> Jan
>> --
>
> Because it's not a good idea to implement the same thing several
> times. Bulbs work well, but compact fluorescent lamps are better, :)
>
> It could be a bit terrible when upgrading the map function shared by
> several views. It's even worse when the upgrading is just an
> optimization: if we forget to upgrade some views' map function, the
> total performance downgrades, which will be hard to figure out.
>
> In current implementation, design docs are self-contained. This is the
> advantage. Am I correct?
>
Yes all view definitions in a given document are indexed together to
gain bulk efficiencies. So changing any of them will rebuild all,
anyway.
Thanks for the questions!
> --
> Li Zhengji
>
--
Chris Anderson
http://jchrisa.net
http://couch.io
Re: some thoughts on view
Posted by Li Zhengji <zh...@gmail.com>.
Hi Jan,
On Thu, Nov 19, 2009 at 9:50 PM, Jan Lehnardt <ja...@apache.org> wrote:
>
> Why is the current implementation not a good idea? It works well :)
>
> Cheers
> Jan
> --
Because it's not a good idea to implement the same thing several
times. Bulbs work well, but compact fluorescent lamps are better, :)
It could be a bit terrible when upgrading the map function shared by
several views. It's even worse when the upgrading is just an
optimization: if we forget to upgrade some views' map function, the
total performance downgrades, which will be hard to figure out.
In current implementation, design docs are self-contained. This is the
advantage. Am I correct?
--
Li Zhengji
Re: some thoughts on view
Posted by Jan Lehnardt <ja...@apache.org>.
On 19 Nov 2009, at 02:16, Li Zhengji wrote:
> Hi,
>
> On Thu, Nov 19, 2009 at 12:46 AM, Adam Kocoloski <ko...@apache.org> wrote:
>> Hi,
>>
>>
>> We already do this -- if two views in a design document have byte-identical map functions, the map is only run once, the index is only saved once, and the different reduce functions share that index.
>
> I realized that it might be hard to cache results of reduce, so
> multiple reduce is hard to do.
>
> But for the binary compare of map functions, I don't think it's a
> good idea. Maybe, we can use a special field to say that we are
> sharing the same map function of another view.
Why is the current implementation not a good idea? It works well :)
Cheers
Jan
--
Re: some thoughts on view
Posted by Li Zhengji <zh...@gmail.com>.
Hi,
On Thu, Nov 19, 2009 at 12:46 AM, Adam Kocoloski <ko...@apache.org> wrote:
> Hi,
>
>
> We already do this -- if two views in a design document have byte-identical map functions, the map is only run once, the index is only saved once, and the different reduce functions share that index.
I realized that it might be hard to cache results of reduce, so
multiple reduce is hard to do.
But for the binary compare of map functions, I don't think it's a
good idea. Maybe, we can use a special field to say that we are
sharing the same map function of another view.
>> 2. Why "reduce", let's "fold"
>>
>> fun (keys, values, Acc) -> updated Acc.
>>
>> We just need to attach an "Acc" field for reduce in view doc.
When I am riding home, I found that I am wrong on fun(keys, values,
Acc) which can't do "rereduce". At least, it should be something like
fun(keys, values, [Acc..]).
> I agree, reduce is a bit warty. I'm seeing an awful lot of people write reduce functions that don't properly account for the possibility of rereduce. Their views work fine on small datasets, but they've got this hidden bug that may only manifest itself in production.
I agree. With a better idea, it's harder for people to write bad code.
I believe that there would be a better solution for reduce, just as
the "map" API in the map function is replaced by "emit".
>
> On the face of it, I'm not sure the "fold" approach is a good match for the btree-based index storage that CouchDB uses. But maybe I'm just not thinking about it hard enough.
>
> Thanks for posting these thoughts -- always nice to get a fresh look at things. Best,
>
> Adam
>
--
Li Zhengji
Re: some thoughts on view
Posted by Adam Kocoloski <ko...@apache.org>.
Hi,
On Nov 18, 2009, at 4:17 AM, Li Zhengji wrote:
> Hi all,
>
> I am planning to use CouchDB now, and coming into some thoughts today,
> which may be silly.
>
> 1. One map, multiple reduces.
>
> There are possibilities that multiple views share the same map
> function. Then, I think these views could be combined, sharing the
> same map function. The outcome is obvious: duplicated operations on
> the same doc are eliminated.
We already do this -- if two views in a design document have byte-identical map functions, the map is only run once, the index is only saved once, and the different reduce functions share that index.
> 2. Why "reduce", let's "fold"
>
> The reduce function is a little ugly due to the existance of parameter
> "rereduce".
>
> The reduce process is much like the "fold" function in functional
> programming languages, except that the function to be reduced is
> required to be assocative and commutative (then, the reduce process
> could be solved/implemented in map/reduce manner, in parallel, and
> incrementally).
>
> It's better to introduce an "Acc" parameter and remove the "rereduce"
> flag (like fold), then the reduce function would be simpler:
>
> fun (keys, values, Acc) -> updated Acc.
>
> We just need to attach an "Acc" field for reduce in view doc.
I agree, reduce is a bit warty. I'm seeing an awful lot of people write reduce functions that don't properly account for the possibility of rereduce. Their views work fine on small datasets, but they've got this hidden bug that may only manifest itself in production.
On the face of it, I'm not sure the "fold" approach is a good match for the btree-based index storage that CouchDB uses. But maybe I'm just not thinking about it hard enough.
Thanks for posting these thoughts -- always nice to get a fresh look at things. Best,
Adam