You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Hank Knight <hk...@gmail.com> on 2013/07/28 19:22:56 UTC

CouchDB: Counting Total Occurrences of Unique Items

This returns exactly the results I want however I am concerned with performance.

There will only ever be a few dozen unique kinds of fruit however
there will tens of thousands of occurrences of each fruit.

Can I expect this to function well when there are tens of thousands
and even hundreds of thousands of documents?

For performance reasons, should anything be moved from the reduce
function to the map function or to an additional reduce function?

Here is my map function:
function(doc) {
  if (doc.fruit) emit("fruit",doc.fruit);
}

Here is my reduce function:
function (keys, values, rereduce) {
    var r = {};
    for(var f=0;f<values.length;f++) r[values[f]]?r[values[f]]++:r[values[f]]=1;
    return r;
}

Here is a link:
https://zurv3aqzm.iriscouch.com/_utils/database.html?fruit/_design/zzz3/_view/zzz3

Re: CouchDB: Counting Total Occurrences of Unique Items

Posted by Hank Knight <hk...@gmail.com>.
Thank you for your patience with me.  That works!

On Sun, Jul 28, 2013 at 6:56 PM, Stanley Iriele
<si...@breaktimestudios.com> wrote:
> The problem here is that you seem to not understand how the emit and key
> value map reduce works.
>
> Emit (key,value) couch db sorts and groups based off key.
>
> In short your emit should be: emit( doc.fruit, null) if that's all you care
> about... And your reduce function should be _count
>
> The group param is for grouping the keys.
>
> You can use group_level if there was an array of keys but this should
> answer your question
> On Jul 28, 2013 5:45 PM, "Hank Knight" <hk...@gmail.com> wrote:
>
>> My map function is here:
>> function (doc) {
>>  if (doc.fruit) emit("fruit",doc.fruit);
>> }
>>
>> My reduce function is here:
>> function (keys, values, rereduce){
>>    var r  = {};
>>    for (var f=0;f<values.length;f++)
>> r[values[f]]?r[values[f]]++:r[values[f]]=1;
>>    return r ;
>> }
>>
>> View it here::
>>
>> https://zurv3aqzm.iriscouch.com/_utils/database.html?fruit/_design/zzz3/_view/zzz3#
>>
>> I do not understand how to group the results.
>>

Re: CouchDB: Counting Total Occurrences of Unique Items

Posted by Stanley Iriele <si...@breaktimestudios.com>.
The problem here is that you seem to not understand how the emit and key
value map reduce works.

Emit (key,value) couch db sorts and groups based off key.

In short your emit should be: emit( doc.fruit, null) if that's all you care
about... And your reduce function should be _count

The group param is for grouping the keys.

You can use group_level if there was an array of keys but this should
answer your question
On Jul 28, 2013 5:45 PM, "Hank Knight" <hk...@gmail.com> wrote:

> My map function is here:
> function (doc) {
>  if (doc.fruit) emit("fruit",doc.fruit);
> }
>
> My reduce function is here:
> function (keys, values, rereduce){
>    var r  = {};
>    for (var f=0;f<values.length;f++)
> r[values[f]]?r[values[f]]++:r[values[f]]=1;
>    return r ;
> }
>
> View it here::
>
> https://zurv3aqzm.iriscouch.com/_utils/database.html?fruit/_design/zzz3/_view/zzz3#
>
> I do not understand how to group the results.
>

Re: CouchDB: Counting Total Occurrences of Unique Items

Posted by Hank Knight <hk...@gmail.com>.
My map function is here:
function (doc) {
 if (doc.fruit) emit("fruit",doc.fruit);
}

My reduce function is here:
function (keys, values, rereduce){
   var r  = {};
   for (var f=0;f<values.length;f++) r[values[f]]?r[values[f]]++:r[values[f]]=1;
   return r ;
}

View it here::
https://zurv3aqzm.iriscouch.com/_utils/database.html?fruit/_design/zzz3/_view/zzz3#

I do not understand how to group the results.

Re: CouchDB: Counting Total Occurrences of Unique Items

Posted by Stanley Iriele <si...@gmail.com>.
Could you post your map function... It will make perfect sense that way
what's going on... I'm guessing you're doing something like "fruit",
doc.fruit somewhere... But just paste it here
On Jul 28, 2013 4:33 PM, "Aurélien Bénel" <au...@utt.fr> wrote:

> > "fruit"       : 19
>
> How do you manage to get this?!
>
> Here is the raw MapReduce of my first post:
>
> https://zurv3aqzm.iriscouch.com/fruit/_design/good_way/_view/fruit_count?group=true
>
> Look at my second post for better formatting with a list.
>
>
> Regards,
>
> Aurélien

Re: CouchDB: Counting Total Occurrences of Unique Items

Posted by Aurélien Bénel <au...@utt.fr>.
> "fruit"	: 19

How do you manage to get this?!

Here is the raw MapReduce of my first post:
  https://zurv3aqzm.iriscouch.com/fruit/_design/good_way/_view/fruit_count?group=true

Look at my second post for better formatting with a list.


Regards,

Aurélien 

Re: CouchDB: Counting Total Occurrences of Unique Items

Posted by Aurélien Bénel <au...@utt.fr>.
> I do not like the resulting format.

MapReduce is about optimizing algorithms
not about formatting.

If you really need formatting, create a list:

function(head, req) {
  var result = {}
  while (r = getRow()) {
    result[r.key] = r.value;
  }
  send(JSON.stringify(result));
}

This should work. However, this simple implementation is not memory savvy.
Send row one by one for better memory management.
And if you really need high performance, forget formatting.


Regards,

Aurélien

Re: CouchDB: Counting Total Occurrences of Unique Items

Posted by Robert Newson <rn...@apache.org>.
add ?group=true

On 28 July 2013 19:36, Hank Knight <hk...@gmail.com> wrote:
> Aurélien,
>
> I do not like the resulting format.
>
> Using your code, I get this:
> "fruit" : 19
>
> I want this:
> {gooseberry: 3, cranberry: 1, plumb: 1, grapefruit: 2, lime: 1,
> strawberry: 4, orange: 1, blueberry: 1, watermelon: 2, peach: 2,
> lemon: 1}
>
> So how can I get a count of all types of fruit, broken down by the
> type of fruit?
>
>
> I need to know specifically how many occurrences there are are
> "strawberry" as well each of the other types of fruit.
>
> https://zurv3aqzm.iriscouch.com/_utils/database.html?fruit/_design/zzz3/_view/zzz3
>
>
>
> On Sun, Jul 28, 2013 at 2:48 PM, Aurélien Bénel <au...@utt.fr> wrote:
>> Hi Hank,
>>
>>> For performance reasons, should anything be moved from the reduce function to the map function or to an additional reduce function?
>>
>>
>> Indeed.
>>
>> Here is my tip: always try to change your map so that you can use a builtin reduce function.
>>
>> MAP
>> ===
>>
>> function (o) {
>>   if (o.fruit) {
>>     emit(o.fruit, null)
>>   }
>> }
>>
>> REDUCE
>> ======
>>
>> _count
>>
>>
>> In the case you wouldn't like the resulting format, use a list on it.
>>
>>
>> Regards,
>>
>> Aurélien

Re: CouchDB: Counting Total Occurrences of Unique Items

Posted by Hank Knight <hk...@gmail.com>.
Aurélien,

I do not like the resulting format.

Using your code, I get this:
"fruit"	: 19

I want this:	
{gooseberry: 3, cranberry: 1, plumb: 1, grapefruit: 2, lime: 1,
strawberry: 4, orange: 1, blueberry: 1, watermelon: 2, peach: 2,
lemon: 1}

So how can I get a count of all types of fruit, broken down by the
type of fruit?


I need to know specifically how many occurrences there are are
"strawberry" as well each of the other types of fruit.

https://zurv3aqzm.iriscouch.com/_utils/database.html?fruit/_design/zzz3/_view/zzz3



On Sun, Jul 28, 2013 at 2:48 PM, Aurélien Bénel <au...@utt.fr> wrote:
> Hi Hank,
>
>> For performance reasons, should anything be moved from the reduce function to the map function or to an additional reduce function?
>
>
> Indeed.
>
> Here is my tip: always try to change your map so that you can use a builtin reduce function.
>
> MAP
> ===
>
> function (o) {
>   if (o.fruit) {
>     emit(o.fruit, null)
>   }
> }
>
> REDUCE
> ======
>
> _count
>
>
> In the case you wouldn't like the resulting format, use a list on it.
>
>
> Regards,
>
> Aurélien

Re: CouchDB: Counting Total Occurrences of Unique Items

Posted by Aurélien Bénel <au...@utt.fr>.
Hi Hank,

> For performance reasons, should anything be moved from the reduce function to the map function or to an additional reduce function?


Indeed. 

Here is my tip: always try to change your map so that you can use a builtin reduce function.

MAP
===

function (o) {
  if (o.fruit) {
    emit(o.fruit, null)
  }
}

REDUCE
======

_count


In the case you wouldn't like the resulting format, use a list on it. 


Regards,

Aurélien