You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@couchdb.apache.org by Filipe David Manana <fd...@apache.org> on 2010/12/12 13:17:24 UTC
worst performance in 1.1.x (compared to 1.0.x)
Hi,
While running a relaximation test to compare read/write performance
between 1.1.x and 1.0.x, I found out that 1.1.x has worst performance
It seems the cause is related to the new Mochiweb version introduced
in commit 4b0948ddb3a428f8a5330e05745b2fbd4ccf9375 -
https://github.com/apache/couchdb/commit/4b0948ddb3a428f8a5330e05745b2fbd4ccf9375
Comparing the performance of this revision with the previous one
(cd214b23e8129868d4a7020ddafd55a16e496652), I get the following
results:
http://graphs.mikeal.couchone.com/#/graph/df0f79455c9c600f66d1ce42ea0125e5
Both read and write performance are worse for
4b0948ddb3a428f8a5330e05745b2fbd4ccf9375.
The cause could be the configuration we pass to Mochiweb in
couch_httpd. The new Mochiweb sets the nodelay option to false by
default and it uses now several acceptor processes (16 by default).
However even with the following small patch I still get about the same
disappointing results:
diff --git a/src/couchdb/couch_httpd.erl b/src/couchdb/couch_httpd.erl
index 23ff7f9..e93c7e7 100644
--- a/src/couchdb/couch_httpd.erl
+++ b/src/couchdb/couch_httpd.erl
@@ -97,7 +97,9 @@ start_link(Name, Options) ->
{ok, Pid} = case mochiweb_http:start(Options ++ [
{loop, Loop},
{name, Name},
- {ip, BindAddress}
+ {ip, BindAddress},
+ {nodelay, true},
+ {acceptor_pool_size, 32}
]) of
{ok, MochiPid} -> {ok, MochiPid};
{error, Reason} ->
(Also tried higher values for acceptor_pool_size, which doesn't help)
The test was run like this:
$ node tests/compare_write_and_read.js --wclients 100 --rclients 200 \
-name1 new_4b0948ddb3a428f8a5330e05745b2fbd4ccf9375 -name2
old_cd214b23e8129868d4a7020ddafd55a16e496652 \
-url1 http://localhost:5984/ -url2 http://localhost:5985/ \
--duration 120
With the following document template (file
relaximation/common/small_doc.json):
http://friendpaste.com/7GKUEg0SZHmOf0g7Dh5IWC
Can anyone confirm these results?
If confirmed, I would say this is a blocker for 1.1.0.
thanks
--
Filipe David Manana,
fdmanana@gmail.com, fdmanana@apache.org
"Reasonable men adapt themselves to the world.
Unreasonable men adapt the world to themselves.
That's why all progress depends on unreasonable men."
Re: worst performance in 1.1.x (compared to 1.0.x)
Posted by Filipe David Manana <fd...@apache.org>.
Adam,
My tests were done with OTP R14B, dual-core Thinkpad (with Linux) and
a 5400 rpms hard disk.
I think the cause for such huge differences between our tests is the
OS. The time I spent with Joel, we always had very different results
for the same tests. He also used a Macbook pro (with OS X). (I've
heard several times that OS X's IO scheduler is worst than Linux's
one).
It's weird that in your 2nd graph, trunk reads are a lot worse
compared 4b0948d-after-R14B01-reads.
Do you also have a Linux machine to test this?
On Mon, Dec 13, 2010 at 3:02 AM, Adam Kocoloski <ko...@apache.org> wrote:
> I tried R14B01, the custom small_doc.json, and lowering schedulers_online to 1, but the writers continue to starve. Trunk (4137a8e) does not starve the writers, almost certainly due to your updater_fd patch. Comparing trunk and the mochiweb upgrade commit I get
>
> http://graphs.mikeal.couchone.com/#/graph/df0f79455c9c600f66d1ce42ea018d7e
>
> I think it's still important that you observe a performance regression with the introduction of that patch, but something else in our respective setups clearly has a much greater effect on the results. Regards,
>
> Adam
>
> On Dec 12, 2010, at 8:42 PM, Adam Kocoloski wrote:
>
>> Hi Filipe, I cannot reproduce those results at all, though I didn't try loading the custom small_doc.json. If I use 200 readers and 100 writers the writers are completely starved and I get e.g.
>>
>> http://graphs.mikeal.couchone.com/#/graph/df0f79455c9c600f66d1ce42ea016e07
>>
>> I need to lower the readers down to ~20 to keep the write throughput reasonable. I'm running R13B04 on a dual-core OS X 10.6 MacBook. I'll try a few more things including the custom doc and R14B01, but do you have suggestions for why these results might be so dramatically different?
>>
>> Adam
>>
>> On Dec 12, 2010, at 7:17 AM, Filipe David Manana wrote:
>>
>>> Hi,
>>>
>>> While running a relaximation test to compare read/write performance
>>> between 1.1.x and 1.0.x, I found out that 1.1.x has worst performance
>>>
>>> It seems the cause is related to the new Mochiweb version introduced
>>> in commit 4b0948ddb3a428f8a5330e05745b2fbd4ccf9375 -
>>> https://github.com/apache/couchdb/commit/4b0948ddb3a428f8a5330e05745b2fbd4ccf9375
>>>
>>> Comparing the performance of this revision with the previous one
>>> (cd214b23e8129868d4a7020ddafd55a16e496652), I get the following
>>> results:
>>>
>>> http://graphs.mikeal.couchone.com/#/graph/df0f79455c9c600f66d1ce42ea0125e5
>>>
>>> Both read and write performance are worse for
>>> 4b0948ddb3a428f8a5330e05745b2fbd4ccf9375.
>>>
>>> The cause could be the configuration we pass to Mochiweb in
>>> couch_httpd. The new Mochiweb sets the nodelay option to false by
>>> default and it uses now several acceptor processes (16 by default).
>>> However even with the following small patch I still get about the same
>>> disappointing results:
>>>
>>> diff --git a/src/couchdb/couch_httpd.erl b/src/couchdb/couch_httpd.erl
>>> index 23ff7f9..e93c7e7 100644
>>> --- a/src/couchdb/couch_httpd.erl
>>> +++ b/src/couchdb/couch_httpd.erl
>>> @@ -97,7 +97,9 @@ start_link(Name, Options) ->
>>> {ok, Pid} = case mochiweb_http:start(Options ++ [
>>> {loop, Loop},
>>> {name, Name},
>>> - {ip, BindAddress}
>>> + {ip, BindAddress},
>>> + {nodelay, true},
>>> + {acceptor_pool_size, 32}
>>> ]) of
>>> {ok, MochiPid} -> {ok, MochiPid};
>>> {error, Reason} ->
>>>
>>>
>>> (Also tried higher values for acceptor_pool_size, which doesn't help)
>>>
>>> The test was run like this:
>>>
>>> $ node tests/compare_write_and_read.js --wclients 100 --rclients 200 \
>>> -name1 new_4b0948ddb3a428f8a5330e05745b2fbd4ccf9375 -name2
>>> old_cd214b23e8129868d4a7020ddafd55a16e496652 \
>>> -url1 http://localhost:5984/ -url2 http://localhost:5985/ \
>>> --duration 120
>>>
>>> With the following document template (file
>>> relaximation/common/small_doc.json):
>>> http://friendpaste.com/7GKUEg0SZHmOf0g7Dh5IWC
>>>
>>> Can anyone confirm these results?
>>> If confirmed, I would say this is a blocker for 1.1.0.
>>>
>>> thanks
>>>
>>>
>>> --
>>> Filipe David Manana,
>>> fdmanana@gmail.com, fdmanana@apache.org
>>>
>>> "Reasonable men adapt themselves to the world.
>>> Unreasonable men adapt the world to themselves.
>>> That's why all progress depends on unreasonable men."
>>
>
>
--
Filipe David Manana,
fdmanana@gmail.com, fdmanana@apache.org
"Reasonable men adapt themselves to the world.
Unreasonable men adapt the world to themselves.
That's why all progress depends on unreasonable men."
Re: worst performance in 1.1.x (compared to 1.0.x)
Posted by Adam Kocoloski <ko...@apache.org>.
I tried R14B01, the custom small_doc.json, and lowering schedulers_online to 1, but the writers continue to starve. Trunk (4137a8e) does not starve the writers, almost certainly due to your updater_fd patch. Comparing trunk and the mochiweb upgrade commit I get
http://graphs.mikeal.couchone.com/#/graph/df0f79455c9c600f66d1ce42ea018d7e
I think it's still important that you observe a performance regression with the introduction of that patch, but something else in our respective setups clearly has a much greater effect on the results. Regards,
Adam
On Dec 12, 2010, at 8:42 PM, Adam Kocoloski wrote:
> Hi Filipe, I cannot reproduce those results at all, though I didn't try loading the custom small_doc.json. If I use 200 readers and 100 writers the writers are completely starved and I get e.g.
>
> http://graphs.mikeal.couchone.com/#/graph/df0f79455c9c600f66d1ce42ea016e07
>
> I need to lower the readers down to ~20 to keep the write throughput reasonable. I'm running R13B04 on a dual-core OS X 10.6 MacBook. I'll try a few more things including the custom doc and R14B01, but do you have suggestions for why these results might be so dramatically different?
>
> Adam
>
> On Dec 12, 2010, at 7:17 AM, Filipe David Manana wrote:
>
>> Hi,
>>
>> While running a relaximation test to compare read/write performance
>> between 1.1.x and 1.0.x, I found out that 1.1.x has worst performance
>>
>> It seems the cause is related to the new Mochiweb version introduced
>> in commit 4b0948ddb3a428f8a5330e05745b2fbd4ccf9375 -
>> https://github.com/apache/couchdb/commit/4b0948ddb3a428f8a5330e05745b2fbd4ccf9375
>>
>> Comparing the performance of this revision with the previous one
>> (cd214b23e8129868d4a7020ddafd55a16e496652), I get the following
>> results:
>>
>> http://graphs.mikeal.couchone.com/#/graph/df0f79455c9c600f66d1ce42ea0125e5
>>
>> Both read and write performance are worse for
>> 4b0948ddb3a428f8a5330e05745b2fbd4ccf9375.
>>
>> The cause could be the configuration we pass to Mochiweb in
>> couch_httpd. The new Mochiweb sets the nodelay option to false by
>> default and it uses now several acceptor processes (16 by default).
>> However even with the following small patch I still get about the same
>> disappointing results:
>>
>> diff --git a/src/couchdb/couch_httpd.erl b/src/couchdb/couch_httpd.erl
>> index 23ff7f9..e93c7e7 100644
>> --- a/src/couchdb/couch_httpd.erl
>> +++ b/src/couchdb/couch_httpd.erl
>> @@ -97,7 +97,9 @@ start_link(Name, Options) ->
>> {ok, Pid} = case mochiweb_http:start(Options ++ [
>> {loop, Loop},
>> {name, Name},
>> - {ip, BindAddress}
>> + {ip, BindAddress},
>> + {nodelay, true},
>> + {acceptor_pool_size, 32}
>> ]) of
>> {ok, MochiPid} -> {ok, MochiPid};
>> {error, Reason} ->
>>
>>
>> (Also tried higher values for acceptor_pool_size, which doesn't help)
>>
>> The test was run like this:
>>
>> $ node tests/compare_write_and_read.js --wclients 100 --rclients 200 \
>> -name1 new_4b0948ddb3a428f8a5330e05745b2fbd4ccf9375 -name2
>> old_cd214b23e8129868d4a7020ddafd55a16e496652 \
>> -url1 http://localhost:5984/ -url2 http://localhost:5985/ \
>> --duration 120
>>
>> With the following document template (file
>> relaximation/common/small_doc.json):
>> http://friendpaste.com/7GKUEg0SZHmOf0g7Dh5IWC
>>
>> Can anyone confirm these results?
>> If confirmed, I would say this is a blocker for 1.1.0.
>>
>> thanks
>>
>>
>> --
>> Filipe David Manana,
>> fdmanana@gmail.com, fdmanana@apache.org
>>
>> "Reasonable men adapt themselves to the world.
>> Unreasonable men adapt the world to themselves.
>> That's why all progress depends on unreasonable men."
>
Re: worst performance in 1.1.x (compared to 1.0.x)
Posted by Adam Kocoloski <ko...@apache.org>.
Hi Filipe, I cannot reproduce those results at all, though I didn't try loading the custom small_doc.json. If I use 200 readers and 100 writers the writers are completely starved and I get e.g.
http://graphs.mikeal.couchone.com/#/graph/df0f79455c9c600f66d1ce42ea016e07
I need to lower the readers down to ~20 to keep the write throughput reasonable. I'm running R13B04 on a dual-core OS X 10.6 MacBook. I'll try a few more things including the custom doc and R14B01, but do you have suggestions for why these results might be so dramatically different?
Adam
On Dec 12, 2010, at 7:17 AM, Filipe David Manana wrote:
> Hi,
>
> While running a relaximation test to compare read/write performance
> between 1.1.x and 1.0.x, I found out that 1.1.x has worst performance
>
> It seems the cause is related to the new Mochiweb version introduced
> in commit 4b0948ddb3a428f8a5330e05745b2fbd4ccf9375 -
> https://github.com/apache/couchdb/commit/4b0948ddb3a428f8a5330e05745b2fbd4ccf9375
>
> Comparing the performance of this revision with the previous one
> (cd214b23e8129868d4a7020ddafd55a16e496652), I get the following
> results:
>
> http://graphs.mikeal.couchone.com/#/graph/df0f79455c9c600f66d1ce42ea0125e5
>
> Both read and write performance are worse for
> 4b0948ddb3a428f8a5330e05745b2fbd4ccf9375.
>
> The cause could be the configuration we pass to Mochiweb in
> couch_httpd. The new Mochiweb sets the nodelay option to false by
> default and it uses now several acceptor processes (16 by default).
> However even with the following small patch I still get about the same
> disappointing results:
>
> diff --git a/src/couchdb/couch_httpd.erl b/src/couchdb/couch_httpd.erl
> index 23ff7f9..e93c7e7 100644
> --- a/src/couchdb/couch_httpd.erl
> +++ b/src/couchdb/couch_httpd.erl
> @@ -97,7 +97,9 @@ start_link(Name, Options) ->
> {ok, Pid} = case mochiweb_http:start(Options ++ [
> {loop, Loop},
> {name, Name},
> - {ip, BindAddress}
> + {ip, BindAddress},
> + {nodelay, true},
> + {acceptor_pool_size, 32}
> ]) of
> {ok, MochiPid} -> {ok, MochiPid};
> {error, Reason} ->
>
>
> (Also tried higher values for acceptor_pool_size, which doesn't help)
>
> The test was run like this:
>
> $ node tests/compare_write_and_read.js --wclients 100 --rclients 200 \
> -name1 new_4b0948ddb3a428f8a5330e05745b2fbd4ccf9375 -name2
> old_cd214b23e8129868d4a7020ddafd55a16e496652 \
> -url1 http://localhost:5984/ -url2 http://localhost:5985/ \
> --duration 120
>
> With the following document template (file
> relaximation/common/small_doc.json):
> http://friendpaste.com/7GKUEg0SZHmOf0g7Dh5IWC
>
> Can anyone confirm these results?
> If confirmed, I would say this is a blocker for 1.1.0.
>
> thanks
>
>
> --
> Filipe David Manana,
> fdmanana@gmail.com, fdmanana@apache.org
>
> "Reasonable men adapt themselves to the world.
> Unreasonable men adapt the world to themselves.
> That's why all progress depends on unreasonable men."