You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Niket Patel <ne...@me.com> on 2009/01/29 21:40:57 UTC
CouchDB huge performance difference between Mac OS / Linux
While views are generating first time on 6000 docs size db. ( same
database for both machines )
Mac ( 2Ghz core 2 duo, 2 GB ) take few seconds
Ubuntu Linux ( 2.4 Ghz 1.7 GB ) on Amazon takes forever ( 60 mins
still waiting )
CouchDB version : 0.9.0a738990-incubating
Erlang Version :
- Mac : Erlang (BEAM) emulator version 5.6.5 [source] [smp:2] [async-
threads:0] [kernel-poll:false]
- Linux : Erlang (BEAM) emulator version 5.6.5 [source] [async-threads:
0] [hipe] [kernel-poll:false]
One observation, First "couchjs" and "beam" processes are running when
view generation starts after few second couchjs goes away but
'beam' ( /usr/local/lib/erlang/erts-5.6.5/bin/beam ) keep running
consuming 45% CPU all time
Re: CouchDB huge performance difference between Mac OS / Linux
Posted by Jan Lehnardt <ja...@apache.org>.
On 29 Jan 2009, at 23:45, Niket Patel wrote:
>
> On Jan 30, 2009, at 3:43 AM, Paul Davis wrote:
>
>>
>> Whoa. Turn on debug logging and see if you're having a huge crash of
>> the view process. I'm guessing you are. Send us any tracebacks you
>> get.
>
>
> Following is the start of error message rest of the part contains,
> huge list of actual docs.
>
> [error] [<0.94.0>] ** Generic server <0.94.0> terminating
> ** Last message in was {'EXIT',<0.96.0>,
> {timeout,
> {gen_server,call,
> [<0.101.0>,
> {prompt,
> [<<"rereduce">>,
>
>
> I have used "couchdb -i", Please let me know if you need more details.
Send all output to http://friendpaste.com/
Also, show your map and reduce functions.
Cheers
Jan
--
Re: CouchDB huge performance difference between Mac OS / Linux
Posted by Dirk-Willem van Gulik <Di...@bbc.co.uk>.
Chris Anderson wrote:
> Maybe we can setup an error-coral, where people post unhelpful error
> message to friendpaste with the proper tags, and others can set about
> trying to make the errors more helpful. Does Friendpaste have tags? :)
File a Jira :) Heck - ship the next version with a 'couchdbbug' command :)
Dw
http://www.bbc.co.uk/
This e-mail (and any attachments) is confidential and may contain personal views which are not the views of the BBC unless specifically stated.
If you have received it in error, please delete it from your system.
Do not use, copy or disclose the information in any way nor act in reliance on it and notify the sender immediately.
Please note that the BBC monitors e-mails sent or received.
Further communication will signify your consent to this.
Re: CouchDB huge performance difference between Mac OS / Linux
Posted by Chris Anderson <jc...@apache.org>.
On Fri, Jan 30, 2009 at 5:51 AM, Niket Patel <ne...@me.com> wrote:
> My problem seems to solve for now. with following change in M/R functions,
>
> // from forEach loop ...
> values.forEach(function(r) {
> ....
> });
>
> // to simple for loops ...
> for( var i in values ){
> ....
> }
>
> I don't know what is cost of each approach.
>
curious. I'd expect those to to be the same (functionally) with
perhaps a slightly greater cost for the forEach version, as it has to
set up a scope context for the inner function on each iteration.
> I wish if couchdb handle this things more gracefully with descriptive
> errors.
>
We've got a push going to increase error handling message quality.
It's hard because it usually doesn't happen until one of the core-devs
finds themselves having created an error (which happens less as you
get accustomed to CouchDB). When I see something unhelpful, I tend to
fix it, as I might not have that opportunity again.
Maybe we can setup an error-coral, where people post unhelpful error
message to friendpaste with the proper tags, and others can set about
trying to make the errors more helpful. Does Friendpaste have tags? :)
Chris
--
Chris Anderson
http://jchris.mfdz.com
Re: CouchDB huge performance difference between Mac OS / Linux
Posted by Niket Patel <ne...@me.com>.
On Jan 30, 2009, at 5:37 PM, Jan Lehnardt wrote:
>
> Reduce functions should not grow its output larger
> than log(n) where n is the number of input rows.
>
> Can somebody update the wiki with that info, I thought
> we have it? :)
On wiki, this may not be mentioned but discussed on mailing list
certainly many times.
Training one's procedural / OO brains to M/R/Rereduce with certain
restrictions is not easy.
Well, in my case reduce output is may not be strictly following log(n)
But if you mean per Key , it is reducing to single row from any number
of input rows for that key.
My problem seems to solve for now. with following change in M/R
functions,
// from forEach loop ...
values.forEach(function(r) {
....
});
// to simple for loops ...
for( var i in values ){
....
}
I don't know what is cost of each approach.
I wish if couchdb handle this things more gracefully with descriptive
errors.
Thanks for your answers.
Re: CouchDB huge performance difference between Mac OS / Linux
Posted by Jan Lehnardt <ja...@apache.org>.
I glanced at your reduce:
Reduce functions should not grow its output larger
than log(n) where n is the number of input rows.
Can somebody update the wiki with that info, I thought
we have it? :)
Cheers
Jan
--
On 30 Jan 2009, at 12:55, Niket Patel wrote:
> On Jan 30, 2009, at 2:26 PM, Jan Lehnardt wrote:
>
>> Oh hey, keep it open, please. We are not a private help desk and
>> the more eyes can look at things, the better for you.
>
> I apologize but data owned by someone else and sharing this not
> possible for me as of now.
> If you can't look problem into privately.
> I fully understand... Thanks for any efforts you have made.
>
> Some details that Paul asked:
> Linux spidermonkey : libmozjs-dev 1.8.1.13+n
> Mac spidermonkey : @1.7.0
>
> Sorry for wrong subject line, now it appears entirely different :-)
>
> Thanks again,
> Niket
>
Re: CouchDB huge performance difference between Mac OS / Linux
Posted by Niket Patel <ne...@me.com>.
On Jan 30, 2009, at 2:26 PM, Jan Lehnardt wrote:
> Oh hey, keep it open, please. We are not a private help desk and
> the more eyes can look at things, the better for you.
I apologize but data owned by someone else and sharing this not
possible for me as of now.
If you can't look problem into privately.
I fully understand... Thanks for any efforts you have made.
Some details that Paul asked:
Linux spidermonkey : libmozjs-dev 1.8.1.13+n
Mac spidermonkey : @1.7.0
Sorry for wrong subject line, now it appears entirely different :-)
Thanks again,
Niket
Re: CouchDB huge performance difference between Mac OS / Linux
Posted by Jan Lehnardt <ja...@apache.org>.
On 30 Jan 2009, at 09:15, Niket Patel wrote:
>
> On Jan 30, 2009, at 4:57 AM, Paul Davis wrote:
>
>>
>> As Jan mentioned you'll want to paste your M/R functions. Do you
>> perhaps have different versions of spidermonkey installed on the two
>> machines?
>
> Thanks Paul, I will check on versions.
> And will send you M/R function in private email to you and Jan.
>
Oh hey, keep it open, please. We are not a private help desk and
the more eyes can look at things, the better for you.
Cheers
Jan
--
Re: CouchDB huge performance difference between Mac OS / Linux
Posted by Niket Patel <ne...@me.com>.
On Jan 30, 2009, at 4:57 AM, Paul Davis wrote:
>
> As Jan mentioned you'll want to paste your M/R functions. Do you
> perhaps have different versions of spidermonkey installed on the two
> machines?
Thanks Paul, I will check on versions.
And will send you M/R function in private email to you and Jan.
Re: CouchDB huge performance difference between Mac OS / Linux
Posted by Paul Davis <pa...@gmail.com>.
On Thu, Jan 29, 2009 at 5:45 PM, Niket Patel <ne...@me.com> wrote:
>
> On Jan 30, 2009, at 3:43 AM, Paul Davis wrote:
>
>>
>> Whoa. Turn on debug logging and see if you're having a huge crash of
>> the view process. I'm guessing you are. Send us any tracebacks you
>> get.
>
>
> Following is the start of error message rest of the part contains, huge list
> of actual docs.
>
> [error] [<0.94.0>] ** Generic server <0.94.0> terminating
> ** Last message in was {'EXIT',<0.96.0>,
> {timeout,
> {gen_server,call,
> [<0.101.0>,
> {prompt,
> [<<"rereduce">>,
>
>
> I have used "couchdb -i", Please let me know if you need more details.
>
As Jan mentioned you'll want to paste your M/R functions. Do you
perhaps have different versions of spidermonkey installed on the two
machines?
Re: CouchDB huge performance difference between Mac OS / Linux
Posted by Niket Patel <ne...@me.com>.
On Jan 30, 2009, at 3:43 AM, Paul Davis wrote:
>
> Whoa. Turn on debug logging and see if you're having a huge crash of
> the view process. I'm guessing you are. Send us any tracebacks you
> get.
Following is the start of error message rest of the part contains,
huge list of actual docs.
[error] [<0.94.0>] ** Generic server <0.94.0> terminating
** Last message in was {'EXIT',<0.96.0>,
{timeout,
{gen_server,call,
[<0.101.0>,
{prompt,
[<<"rereduce">>,
I have used "couchdb -i", Please let me know if you need more details.
Re: CouchDB huge performance difference between Mac OS / Linux
Posted by Paul Davis <pa...@gmail.com>.
On Thu, Jan 29, 2009 at 4:52 PM, Niket Patel <ne...@me.com> wrote:
> On Jan 30, 2009, at 3:00 AM, Paul Davis wrote:
>
>> Also discussing on IRC, we're wondering if the view is actually built
>> correctly and something else is going wrong. An easy method to debug
>> this if you get a copy of trunk, Futon has a panel that shows you
>> actively running tasks so you'll be able to keep an eye on what
>> couchdb is actually doing in terms of things like view generation and
>> compaction. (replication is on the agenda)
>
> When view generation starts ( or when I access views )
> 'couchjs' starts and at that time Futon "Status" shows "View Group Indexer"
> running
> 'couchjs' goes away and Futon "Status" shows "No tasks running"
> this things take under 30 seconds
> .db_design/ directory size increase 1.6 MB every-time I access view and when
> couchjs running
>
> After couchjs finish strange things start
> 'beam' process now starts consuming 40-45% CPU and keep running until I
> respawn with "couchdb -k"
> .db_design/ directory size remains same so nothing written to disk
>
> Above issue not happen with Mac OS X
>
Whoa. Turn on debug logging and see if you're having a huge crash of
the view process. I'm guessing you are. Send us any tracebacks you
get.
Re: CouchDB huge performance difference between Mac OS / Linux
Posted by Niket Patel <ne...@me.com>.
On Jan 30, 2009, at 3:00 AM, Paul Davis wrote:
> Also discussing on IRC, we're wondering if the view is actually built
> correctly and something else is going wrong. An easy method to debug
> this if you get a copy of trunk, Futon has a panel that shows you
> actively running tasks so you'll be able to keep an eye on what
> couchdb is actually doing in terms of things like view generation and
> compaction. (replication is on the agenda)
When view generation starts ( or when I access views )
'couchjs' starts and at that time Futon "Status" shows "View Group
Indexer" running
'couchjs' goes away and Futon "Status" shows "No tasks running"
this things take under 30 seconds
.db_design/ directory size increase 1.6 MB every-time I access view
and when couchjs running
After couchjs finish strange things start
'beam' process now starts consuming 40-45% CPU and keep running until
I respawn with "couchdb -k"
.db_design/ directory size remains same so nothing written to disk
Above issue not happen with Mac OS X
Re: CouchDB huge performance difference between Mac OS / Linux
Posted by Paul Davis <pa...@gmail.com>.
Also discussing on IRC, we're wondering if the view is actually built
correctly and something else is going wrong. An easy method to debug
this if you get a copy of trunk, Futon has a panel that shows you
actively running tasks so you'll be able to keep an eye on what
couchdb is actually doing in terms of things like view generation and
compaction. (replication is on the agenda)
HTH,
Paul Davis
On Thu, Jan 29, 2009 at 4:15 PM, Paul Davis <pa...@gmail.com> wrote:
> Without thinking too hard, could this be an effect of fsync in a VM?
> If you're feeling experimentally inclined, you could try and see the
> performance after commenting out the two calls to fsync in write
> header to see what happens.
>
> Also, the idea that fsync is slow in a VM could be outdated. Its just
> one of those fuzzy memories of reading something somewhere as opposed
> to any personal experience.
>
> HTH,
> Paul Davis
>
> On Thu, Jan 29, 2009 at 3:40 PM, Niket Patel <ne...@me.com> wrote:
>> While views are generating first time on 6000 docs size db. ( same database
>> for both machines )
>> Mac ( 2Ghz core 2 duo, 2 GB ) take few seconds
>> Ubuntu Linux ( 2.4 Ghz 1.7 GB ) on Amazon takes forever ( 60 mins still
>> waiting )
>>
>> CouchDB version : 0.9.0a738990-incubating
>> Erlang Version :
>> - Mac : Erlang (BEAM) emulator version 5.6.5 [source] [smp:2]
>> [async-threads:0] [kernel-poll:false]
>> - Linux : Erlang (BEAM) emulator version 5.6.5 [source] [async-threads:0]
>> [hipe] [kernel-poll:false]
>>
>> One observation, First "couchjs" and "beam" processes are running when view
>> generation starts after few second couchjs goes away but 'beam' (
>> /usr/local/lib/erlang/erts-5.6.5/bin/beam ) keep running consuming 45% CPU
>> all time
>>
>
Re: CouchDB huge performance difference between Mac OS / Linux
Posted by Paul Davis <pa...@gmail.com>.
Without thinking too hard, could this be an effect of fsync in a VM?
If you're feeling experimentally inclined, you could try and see the
performance after commenting out the two calls to fsync in write
header to see what happens.
Also, the idea that fsync is slow in a VM could be outdated. Its just
one of those fuzzy memories of reading something somewhere as opposed
to any personal experience.
HTH,
Paul Davis
On Thu, Jan 29, 2009 at 3:40 PM, Niket Patel <ne...@me.com> wrote:
> While views are generating first time on 6000 docs size db. ( same database
> for both machines )
> Mac ( 2Ghz core 2 duo, 2 GB ) take few seconds
> Ubuntu Linux ( 2.4 Ghz 1.7 GB ) on Amazon takes forever ( 60 mins still
> waiting )
>
> CouchDB version : 0.9.0a738990-incubating
> Erlang Version :
> - Mac : Erlang (BEAM) emulator version 5.6.5 [source] [smp:2]
> [async-threads:0] [kernel-poll:false]
> - Linux : Erlang (BEAM) emulator version 5.6.5 [source] [async-threads:0]
> [hipe] [kernel-poll:false]
>
> One observation, First "couchjs" and "beam" processes are running when view
> generation starts after few second couchjs goes away but 'beam' (
> /usr/local/lib/erlang/erts-5.6.5/bin/beam ) keep running consuming 45% CPU
> all time
>