You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Rajkumar S <ra...@gmail.com> on 2009/10/21 16:46:55 UTC
Slow Map Reduce
Hello,
I am using couch db 0.9.0 for storing logs from my mail server.
Logs are sent from mail servers to a RabbitMQ queue server. Log
insertions into couchdb is done by a python program, after fetching it
from RabbitMQ and converting to Json, using couchdb module (from
couchdb import *). I have a single document storing entire history of
the email transactions. I also have multiple RabbitMQ clients each
pulling from same queue and updating the same coudhdb. This means I
have to update the same document from different clients several times
during the life time of an email message.
To do this I use the message id of each mail transaction as it's key.
(this appears in every log entry) When a first log entry arrives I
check if a doc with that key is present in db, if not I create a new
doc with that key. When second log arrives I extract the doc, convert
it to a hash table in my program, merge the new log entry with the
hash table and update the doc with the updated hash table's json. If a
conflict occurs, the program retries, fetching the doc and updating it
and storing again till conflict is resolved.
This means for every write there is a corresponding read.
Currently I am running it as a pilot and just have a single server
logging to couchdb. I have about 0.75 GB per day right now, with
GET/PUT happening almost continuously (say 1 - 2 per second).
Previously I had a test server running and I tested couple of map
reduce using that DB (about 5 mb)
Now after logging from a single production machine I am not able to
run a single view so far. I get the following error if I wait long
enough:
Error: case_clause
{{bad_return_value,{os_process_error,"OS process timed out."}},
{gen_server,call,
[<0.436.0>,
{prompt,[<<"rereduce">>,
[<<"function(keys, values)\n{\n return
values;\n}">>],.....
I have changed os_process_timeout to 50000, removed the reduce part
but even after about 6 hours my map is not yet finished. Currently the
db size is 3.6G
The map function I am using is:
function(doc) {
if ("msgtype" in doc){
if (doc.msgtype == "allow"){
if ((doc.event == "action_allowed_ip") || (doc.event ==
"action_allow_new")){
result = {};
ip = doc.parameters.client_address;
result["helo"] = doc.parameters.helo_name;
result["event"] = doc.event;
result["timestamp"] = doc.timestamp;
result["id"] = doc._id;
result["from"] = doc.parameters.sender;
result["to"] = doc.parameters.recipient;
emit (ip,result);
}
}
}
}
Top shows that couchjs is most active process and it shows the
following line right now,
11410 root 20 0 90752 27m 752 R 76 0.7 1235:05 couchjs
My hardware is Intel(R) Core(TM)2 Duo CPU E6750 @ 2.66GHz, 4Gig RAM
and one SATA hard disk. I do not think this is the expected
performance of couchdb, so is there some thing I am doing wrong? Any
tips to enhance the performance to acceptable levels?
thanks and much regards,
raj
Re: Slow Map Reduce
Posted by Seggy Umboh <se...@gmail.com>.
Thanks for the prod! I've added it to the bottom of this page as Map/Reduce
debugging:
http://wiki.apache.org/couchdb/Troubleshooting
On Fri, Oct 23, 2009 at 5:37 PM, Chris Anderson <jc...@apache.org> wrote:
> On Fri, Oct 23, 2009 at 5:00 PM, Seggy Umboh <se...@gmail.com>
> wrote:
> > One thing I have found useful is to fire up js on the command line and
> just
> > assign a sample document to the variable doc, then run the map function
> on
> > that.
>
> Sounds cool, if you put together a blog post on the technique I bet
> people (including me) would read it.
>
> >
> > On Fri, Oct 23, 2009 at 4:18 AM, Rajkumar S <ra...@gmail.com> wrote:
> >
> >> Hi,
> >>
> >> Thanks, I had caught this bug and fixed it in the map, but did not
> >> update the mail. I just wanted the sum. but right now I am not using a
> >> reduce function just a map. Even for that I am not getting a result.
> >>
> >> Any way I am just learning the ropes so might take some time before I
> >> figure out the stuff :)
> >>
> >> raj
> >>
> >> On Fri, Oct 23, 2009 at 6:40 AM, Seggy Umboh <se...@gmail.com>
> >> wrote:
> >> > It looks like you just want to find out how many rows were emitted?
> That
> >> > would just be the typical sum right?
> >> > return sum(values);
> >> >
> >> >
> >> >
> >> > js> var v = 6
> >> > js> 0 + v.length
> >> > NaN
> >> >
> >> >
> >> >
> >> > On Wed, Oct 21, 2009 at 11:45 PM, Rajkumar S <ra...@gmail.com>
> >> wrote:
> >> >
> >> >> On Wed, Oct 21, 2009 at 9:48 PM, Paul Joseph Davis
> >> >> <pa...@gmail.com> wrote:
> >> >> > I suspect "return values;" is the culprit here. That's pretty much
> >> >> guranteed
> >> >> > to cause you problems as the data grows recursing up the tree.
> 0.10.0
> >> >> even
> >> >> > detects this and throws an error to avoid precisely this issue.
> >> >> >
> >> >> > It's been discussed a couple times on the list but my stomach is
> >> >> rumbling.
> >> >>
> >> >> That reduce function used in that log was the following.
> >> >>
> >> >> function(keys, values, rereduce)
> >> >> {
> >> >> length = 0;
> >> >> if (rereduce){
> >> >> for(var v = 0; v < values.length; v++){
> >> >> length = length + v.length;
> >> >> }
> >> >> } else {
> >> >> length = values.length;
> >> >> }
> >> >> return length;
> >> >> }
> >> >>
> >> >> The reduce function should returns a single integer and not cause
> data
> >> >> to grow. Let me try with simpler map with no reduce and will report
> >> >> the results.
> >> >>
> >> >> with regards,
> >> >> raj
> >> >>
> >> >
> >>
> >
>
>
>
> --
> Chris Anderson
> http://jchrisa.net
> http://couch.io
>
Re: Slow Map Reduce
Posted by Chris Anderson <jc...@apache.org>.
On Fri, Oct 23, 2009 at 5:00 PM, Seggy Umboh <se...@gmail.com> wrote:
> One thing I have found useful is to fire up js on the command line and just
> assign a sample document to the variable doc, then run the map function on
> that.
Sounds cool, if you put together a blog post on the technique I bet
people (including me) would read it.
>
> On Fri, Oct 23, 2009 at 4:18 AM, Rajkumar S <ra...@gmail.com> wrote:
>
>> Hi,
>>
>> Thanks, I had caught this bug and fixed it in the map, but did not
>> update the mail. I just wanted the sum. but right now I am not using a
>> reduce function just a map. Even for that I am not getting a result.
>>
>> Any way I am just learning the ropes so might take some time before I
>> figure out the stuff :)
>>
>> raj
>>
>> On Fri, Oct 23, 2009 at 6:40 AM, Seggy Umboh <se...@gmail.com>
>> wrote:
>> > It looks like you just want to find out how many rows were emitted? That
>> > would just be the typical sum right?
>> > return sum(values);
>> >
>> >
>> >
>> > js> var v = 6
>> > js> 0 + v.length
>> > NaN
>> >
>> >
>> >
>> > On Wed, Oct 21, 2009 at 11:45 PM, Rajkumar S <ra...@gmail.com>
>> wrote:
>> >
>> >> On Wed, Oct 21, 2009 at 9:48 PM, Paul Joseph Davis
>> >> <pa...@gmail.com> wrote:
>> >> > I suspect "return values;" is the culprit here. That's pretty much
>> >> guranteed
>> >> > to cause you problems as the data grows recursing up the tree. 0.10.0
>> >> even
>> >> > detects this and throws an error to avoid precisely this issue.
>> >> >
>> >> > It's been discussed a couple times on the list but my stomach is
>> >> rumbling.
>> >>
>> >> That reduce function used in that log was the following.
>> >>
>> >> function(keys, values, rereduce)
>> >> {
>> >> length = 0;
>> >> if (rereduce){
>> >> for(var v = 0; v < values.length; v++){
>> >> length = length + v.length;
>> >> }
>> >> } else {
>> >> length = values.length;
>> >> }
>> >> return length;
>> >> }
>> >>
>> >> The reduce function should returns a single integer and not cause data
>> >> to grow. Let me try with simpler map with no reduce and will report
>> >> the results.
>> >>
>> >> with regards,
>> >> raj
>> >>
>> >
>>
>
--
Chris Anderson
http://jchrisa.net
http://couch.io
Re: Slow Map Reduce
Posted by Seggy Umboh <se...@gmail.com>.
One thing I have found useful is to fire up js on the command line and just
assign a sample document to the variable doc, then run the map function on
that.
On Fri, Oct 23, 2009 at 4:18 AM, Rajkumar S <ra...@gmail.com> wrote:
> Hi,
>
> Thanks, I had caught this bug and fixed it in the map, but did not
> update the mail. I just wanted the sum. but right now I am not using a
> reduce function just a map. Even for that I am not getting a result.
>
> Any way I am just learning the ropes so might take some time before I
> figure out the stuff :)
>
> raj
>
> On Fri, Oct 23, 2009 at 6:40 AM, Seggy Umboh <se...@gmail.com>
> wrote:
> > It looks like you just want to find out how many rows were emitted? That
> > would just be the typical sum right?
> > return sum(values);
> >
> >
> >
> > js> var v = 6
> > js> 0 + v.length
> > NaN
> >
> >
> >
> > On Wed, Oct 21, 2009 at 11:45 PM, Rajkumar S <ra...@gmail.com>
> wrote:
> >
> >> On Wed, Oct 21, 2009 at 9:48 PM, Paul Joseph Davis
> >> <pa...@gmail.com> wrote:
> >> > I suspect "return values;" is the culprit here. That's pretty much
> >> guranteed
> >> > to cause you problems as the data grows recursing up the tree. 0.10.0
> >> even
> >> > detects this and throws an error to avoid precisely this issue.
> >> >
> >> > It's been discussed a couple times on the list but my stomach is
> >> rumbling.
> >>
> >> That reduce function used in that log was the following.
> >>
> >> function(keys, values, rereduce)
> >> {
> >> length = 0;
> >> if (rereduce){
> >> for(var v = 0; v < values.length; v++){
> >> length = length + v.length;
> >> }
> >> } else {
> >> length = values.length;
> >> }
> >> return length;
> >> }
> >>
> >> The reduce function should returns a single integer and not cause data
> >> to grow. Let me try with simpler map with no reduce and will report
> >> the results.
> >>
> >> with regards,
> >> raj
> >>
> >
>
Re: Slow Map Reduce
Posted by Rajkumar S <ra...@gmail.com>.
Hi,
Thanks, I had caught this bug and fixed it in the map, but did not
update the mail. I just wanted the sum. but right now I am not using a
reduce function just a map. Even for that I am not getting a result.
Any way I am just learning the ropes so might take some time before I
figure out the stuff :)
raj
On Fri, Oct 23, 2009 at 6:40 AM, Seggy Umboh <se...@gmail.com> wrote:
> It looks like you just want to find out how many rows were emitted? That
> would just be the typical sum right?
> return sum(values);
>
>
>
> js> var v = 6
> js> 0 + v.length
> NaN
>
>
>
> On Wed, Oct 21, 2009 at 11:45 PM, Rajkumar S <ra...@gmail.com> wrote:
>
>> On Wed, Oct 21, 2009 at 9:48 PM, Paul Joseph Davis
>> <pa...@gmail.com> wrote:
>> > I suspect "return values;" is the culprit here. That's pretty much
>> guranteed
>> > to cause you problems as the data grows recursing up the tree. 0.10.0
>> even
>> > detects this and throws an error to avoid precisely this issue.
>> >
>> > It's been discussed a couple times on the list but my stomach is
>> rumbling.
>>
>> That reduce function used in that log was the following.
>>
>> function(keys, values, rereduce)
>> {
>> length = 0;
>> if (rereduce){
>> for(var v = 0; v < values.length; v++){
>> length = length + v.length;
>> }
>> } else {
>> length = values.length;
>> }
>> return length;
>> }
>>
>> The reduce function should returns a single integer and not cause data
>> to grow. Let me try with simpler map with no reduce and will report
>> the results.
>>
>> with regards,
>> raj
>>
>
Re: Slow Map Reduce
Posted by Seggy Umboh <se...@gmail.com>.
It looks like you just want to find out how many rows were emitted? That
would just be the typical sum right?
return sum(values);
js> var v = 6
js> 0 + v.length
NaN
On Wed, Oct 21, 2009 at 11:45 PM, Rajkumar S <ra...@gmail.com> wrote:
> On Wed, Oct 21, 2009 at 9:48 PM, Paul Joseph Davis
> <pa...@gmail.com> wrote:
> > I suspect "return values;" is the culprit here. That's pretty much
> guranteed
> > to cause you problems as the data grows recursing up the tree. 0.10.0
> even
> > detects this and throws an error to avoid precisely this issue.
> >
> > It's been discussed a couple times on the list but my stomach is
> rumbling.
>
> That reduce function used in that log was the following.
>
> function(keys, values, rereduce)
> {
> length = 0;
> if (rereduce){
> for(var v = 0; v < values.length; v++){
> length = length + v.length;
> }
> } else {
> length = values.length;
> }
> return length;
> }
>
> The reduce function should returns a single integer and not cause data
> to grow. Let me try with simpler map with no reduce and will report
> the results.
>
> with regards,
> raj
>
Re: Slow Map Reduce
Posted by Rajkumar S <ra...@gmail.com>.
On Wed, Oct 21, 2009 at 9:48 PM, Paul Joseph Davis
<pa...@gmail.com> wrote:
> I suspect "return values;" is the culprit here. That's pretty much guranteed
> to cause you problems as the data grows recursing up the tree. 0.10.0 even
> detects this and throws an error to avoid precisely this issue.
>
> It's been discussed a couple times on the list but my stomach is rumbling.
That reduce function used in that log was the following.
function(keys, values, rereduce)
{
length = 0;
if (rereduce){
for(var v = 0; v < values.length; v++){
length = length + v.length;
}
} else {
length = values.length;
}
return length;
}
The reduce function should returns a single integer and not cause data
to grow. Let me try with simpler map with no reduce and will report
the results.
with regards,
raj
Re: Slow Map Reduce
Posted by Paul Joseph Davis <pa...@gmail.com>.
On Oct 21, 2009, at 10:46 AM, Rajkumar S <ra...@gmail.com> wrote:
> Hello,
>
> I am using couch db 0.9.0 for storing logs from my mail server.
>
> Logs are sent from mail servers to a RabbitMQ queue server. Log
> insertions into couchdb is done by a python program, after fetching it
> from RabbitMQ and converting to Json, using couchdb module (from
> couchdb import *). I have a single document storing entire history of
> the email transactions. I also have multiple RabbitMQ clients each
> pulling from same queue and updating the same coudhdb. This means I
> have to update the same document from different clients several times
> during the life time of an email message.
>
> To do this I use the message id of each mail transaction as it's key.
> (this appears in every log entry) When a first log entry arrives I
> check if a doc with that key is present in db, if not I create a new
> doc with that key. When second log arrives I extract the doc, convert
> it to a hash table in my program, merge the new log entry with the
> hash table and update the doc with the updated hash table's json. If a
> conflict occurs, the program retries, fetching the doc and updating it
> and storing again till conflict is resolved.
>
> This means for every write there is a corresponding read.
>
> Currently I am running it as a pilot and just have a single server
> logging to couchdb. I have about 0.75 GB per day right now, with
> GET/PUT happening almost continuously (say 1 - 2 per second).
> Previously I had a test server running and I tested couple of map
> reduce using that DB (about 5 mb)
>
> Now after logging from a single production machine I am not able to
> run a single view so far. I get the following error if I wait long
> enough:
>
> Error: case_clause
>
> {{bad_return_value,{os_process_error,"OS process timed out."}},
> {gen_server,call,
> [<0.436.0>,
> {prompt,[<<"rereduce">>,
> [<<"function(keys, values)\n{\n return
> values;\n}">>],.....
>
I suspect "return values;" is the culprit here. That's pretty much
guranteed to cause you problems as the data grows recursing up the
tree. 0.10.0 even detects this and throws an error to avoid precisely
this issue.
It's been discussed a couple times on the list but my stomach is
rumbling.
> I have changed os_process_timeout to 50000, removed the reduce part
> but even after about 6 hours my map is not yet finished. Currently the
> db size is 3.6G
>
> The map function I am using is:
>
> function(doc) {
> if ("msgtype" in doc){
> if (doc.msgtype == "allow"){
> if ((doc.event == "action_allowed_ip") || (doc.event ==
> "action_allow_new")){
> result = {};
> ip = doc.parameters.client_address;
> result["helo"] = doc.parameters.helo_name;
> result["event"] = doc.event;
> result["timestamp"] = doc.timestamp;
> result["id"] = doc._id;
> result["from"] = doc.parameters.sender;
> result["to"] = doc.parameters.recipient;
> emit (ip,result);
> }
> }
> }
> }
>
> Top shows that couchjs is most active process and it shows the
> following line right now,
> 11410 root 20 0 90752 27m 752 R 76 0.7 1235:05 couchjs
>
> My hardware is Intel(R) Core(TM)2 Duo CPU E6750 @ 2.66GHz, 4Gig RAM
> and one SATA hard disk. I do not think this is the expected
> performance of couchdb, so is there some thing I am doing wrong? Any
> tips to enhance the performance to acceptable levels?
>
> thanks and much regards,
>
> raj
Re: Slow Map Reduce
Posted by Rajkumar S <ra...@gmail.com>.
Hi,
Total number of docs are 341948. growing at the rate of about a couple
per second. I know that first time indexing takes some time but I have
not yet been able to complete even a single map so far. I am now
trying with a simpler map with no reduce.
function(doc) {
if ("msgtype" in doc){
if (doc.msgtype == "allow"){
if ((doc.event == "action_allowed_ip") || (doc.event ==
"action_allow_new")){
emit (ip, 1);
}
}
}
}
On Wed, Oct 21, 2009 at 8:25 PM, Alex P <ap...@kolosy.com> wrote:
> how many docs is that, and have you run the view incrementally? first time
> index builds are painful...
>
> On Wed, Oct 21, 2009 at 9:46 AM, Rajkumar S <ra...@gmail.com> wrote:
>
>> Hello,
>>
>> I am using couch db 0.9.0 for storing logs from my mail server.
>>
>> Logs are sent from mail servers to a RabbitMQ queue server. Log
>> insertions into couchdb is done by a python program, after fetching it
>> from RabbitMQ and converting to Json, using couchdb module (from
>> couchdb import *). I have a single document storing entire history of
>> the email transactions. I also have multiple RabbitMQ clients each
>> pulling from same queue and updating the same coudhdb. This means I
>> have to update the same document from different clients several times
>> during the life time of an email message.
>>
>> To do this I use the message id of each mail transaction as it's key.
>> (this appears in every log entry) When a first log entry arrives I
>> check if a doc with that key is present in db, if not I create a new
>> doc with that key. When second log arrives I extract the doc, convert
>> it to a hash table in my program, merge the new log entry with the
>> hash table and update the doc with the updated hash table's json. If a
>> conflict occurs, the program retries, fetching the doc and updating it
>> and storing again till conflict is resolved.
>>
>> This means for every write there is a corresponding read.
>>
>> Currently I am running it as a pilot and just have a single server
>> logging to couchdb. I have about 0.75 GB per day right now, with
>> GET/PUT happening almost continuously (say 1 - 2 per second).
>> Previously I had a test server running and I tested couple of map
>> reduce using that DB (about 5 mb)
>>
>> Now after logging from a single production machine I am not able to
>> run a single view so far. I get the following error if I wait long
>> enough:
>>
>> Error: case_clause
>>
>> {{bad_return_value,{os_process_error,"OS process timed out."}},
>> {gen_server,call,
>> [<0.436.0>,
>> {prompt,[<<"rereduce">>,
>> [<<"function(keys, values)\n{\n return
>> values;\n}">>],.....
>>
>> I have changed os_process_timeout to 50000, removed the reduce part
>> but even after about 6 hours my map is not yet finished. Currently the
>> db size is 3.6G
>>
>> The map function I am using is:
>>
>> function(doc) {
>> if ("msgtype" in doc){
>> if (doc.msgtype == "allow"){
>> if ((doc.event == "action_allowed_ip") || (doc.event ==
>> "action_allow_new")){
>> result = {};
>> ip = doc.parameters.client_address;
>> result["helo"] = doc.parameters.helo_name;
>> result["event"] = doc.event;
>> result["timestamp"] = doc.timestamp;
>> result["id"] = doc._id;
>> result["from"] = doc.parameters.sender;
>> result["to"] = doc.parameters.recipient;
>> emit (ip,result);
>> }
>> }
>> }
>> }
>>
>> Top shows that couchjs is most active process and it shows the
>> following line right now,
>> 11410 root 20 0 90752 27m 752 R 76 0.7 1235:05 couchjs
>>
>> My hardware is Intel(R) Core(TM)2 Duo CPU E6750 @ 2.66GHz, 4Gig RAM
>> and one SATA hard disk. I do not think this is the expected
>> performance of couchdb, so is there some thing I am doing wrong? Any
>> tips to enhance the performance to acceptable levels?
>>
>> thanks and much regards,
>>
>> raj
>>
>
Re: Slow Map Reduce
Posted by Alex P <ap...@kolosy.com>.
how many docs is that, and have you run the view incrementally? first time
index builds are painful...
On Wed, Oct 21, 2009 at 9:46 AM, Rajkumar S <ra...@gmail.com> wrote:
> Hello,
>
> I am using couch db 0.9.0 for storing logs from my mail server.
>
> Logs are sent from mail servers to a RabbitMQ queue server. Log
> insertions into couchdb is done by a python program, after fetching it
> from RabbitMQ and converting to Json, using couchdb module (from
> couchdb import *). I have a single document storing entire history of
> the email transactions. I also have multiple RabbitMQ clients each
> pulling from same queue and updating the same coudhdb. This means I
> have to update the same document from different clients several times
> during the life time of an email message.
>
> To do this I use the message id of each mail transaction as it's key.
> (this appears in every log entry) When a first log entry arrives I
> check if a doc with that key is present in db, if not I create a new
> doc with that key. When second log arrives I extract the doc, convert
> it to a hash table in my program, merge the new log entry with the
> hash table and update the doc with the updated hash table's json. If a
> conflict occurs, the program retries, fetching the doc and updating it
> and storing again till conflict is resolved.
>
> This means for every write there is a corresponding read.
>
> Currently I am running it as a pilot and just have a single server
> logging to couchdb. I have about 0.75 GB per day right now, with
> GET/PUT happening almost continuously (say 1 - 2 per second).
> Previously I had a test server running and I tested couple of map
> reduce using that DB (about 5 mb)
>
> Now after logging from a single production machine I am not able to
> run a single view so far. I get the following error if I wait long
> enough:
>
> Error: case_clause
>
> {{bad_return_value,{os_process_error,"OS process timed out."}},
> {gen_server,call,
> [<0.436.0>,
> {prompt,[<<"rereduce">>,
> [<<"function(keys, values)\n{\n return
> values;\n}">>],.....
>
> I have changed os_process_timeout to 50000, removed the reduce part
> but even after about 6 hours my map is not yet finished. Currently the
> db size is 3.6G
>
> The map function I am using is:
>
> function(doc) {
> if ("msgtype" in doc){
> if (doc.msgtype == "allow"){
> if ((doc.event == "action_allowed_ip") || (doc.event ==
> "action_allow_new")){
> result = {};
> ip = doc.parameters.client_address;
> result["helo"] = doc.parameters.helo_name;
> result["event"] = doc.event;
> result["timestamp"] = doc.timestamp;
> result["id"] = doc._id;
> result["from"] = doc.parameters.sender;
> result["to"] = doc.parameters.recipient;
> emit (ip,result);
> }
> }
> }
> }
>
> Top shows that couchjs is most active process and it shows the
> following line right now,
> 11410 root 20 0 90752 27m 752 R 76 0.7 1235:05 couchjs
>
> My hardware is Intel(R) Core(TM)2 Duo CPU E6750 @ 2.66GHz, 4Gig RAM
> and one SATA hard disk. I do not think this is the expected
> performance of couchdb, so is there some thing I am doing wrong? Any
> tips to enhance the performance to acceptable levels?
>
> thanks and much regards,
>
> raj
>