You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Chris Stockton <ch...@gmail.com> on 2010/04/20 21:51:23 UTC
Re: A simple comparative benchmark and look at include_docs
performance.
Hello,
On Tue, Apr 20, 2010 at 10:58 AM, Adam Kocoloski <ko...@apache.org> wrote:
> Hi Chris, for the type of access pattern in your benchmark I generally recommend to use emit(doc.model, doc) and avoid include_docs=true. include_docs introduces an extra lookup back in the DB for every row of your view. If you emit the document into the view index the index will get large but streaming requests such as yours can be accomplished with a minimum of disk IO.
We have tried this approach and it was indeed faster, however we wound
up with what I remember to be over 19G view file. For 300mb sized
database this trade off did not seem reasonable, although disk is
cheaper in many cases, we found the bloat to be unacceptable. Do you
know of a way to limit the size of the view when including the doc?
Additionally may I ask if include_docs = true has potential room for
optimization?
> On the other hand, your sar report shows negligible iowait, so perhaps that's not your immediate problem. It may be the case that you're CPU-limited in the (pure Erlang) JSON encoder, although I would've expected JSON encoding CPU usage to scale with network traffic.
It would surprise me if 13mb of json encoding could cause such spikes
in CPU. I also expected network traffic to scale with our CPU usage.
Have you seen issues in this area before? At first thought I would
think of the encoding stage as being one of the lighter areas in the
request, given the simple nature of json.
> You might try running eprof while you do this test. It's quite heavyweight and will slow your system down. If you start couchdb with the -i flag you can get an Erlang shell and execute
> <snip>
This was good information and I will look into profiling with erlang.
May I ask if any effort is currently being put into performance and
optimization for couchdb? I am also very interested in any reads on
large-scall couchdb deployments, that are not so high-level (I.E.
hardware specs, use cases, etc).
Kind Regards,
-Chris
Re: A simple comparative benchmark and look at include_docs
performance.
Posted by Chris Stockton <ch...@gmail.com>.
Hello,
On Wed, Apr 21, 2010 at 5:57 PM, Adam Kocoloski <ko...@apache.org> wrote:
> Yes. Were you running 0.10 before this build? Do you have a mochiweb-r97 in $PREFIX/lib/couchdb/erlang/lib? Due to a quirk in the VM code path algorithms, r97 is considered newer than the r113 that ships with 0.11, which produces unintended effects such as this.
>
> If you remove all old versions of apps in that directory the problem should go away. If you don't have any old versions, I'm not sure what to tell you. I've used mochijson2 with native code compilation successfully in the past. Best,
Excellent Adam, this fixed my issue. I would have had a lot of
difficulty finding this on my own, much appreciated.
Kind Regards,
-Chris
Re: A simple comparative benchmark and look at include_docs performance.
Posted by Adam Kocoloski <ko...@apache.org>.
On Apr 22, 2010, at 12:55 PM, David Coallier wrote:
>>
>> Yes. Were you running 0.10 before this build? Do you have a mochiweb-r97 in $PREFIX/lib/couchdb/erlang/lib? Due to a quirk in the VM code path algorithms, r97 is considered newer than the r113 that ships with 0.11, which produces unintended effects such as this.
>>
>> If you remove all old versions of apps in that directory the problem should go away. If you don't have any old versions, I'm not sure what to tell you. I've used mochijson2 with native code compilation successfully in the past. Best,
>>
>
> I think we should put this somewhere in the UPGRADE file (Perhaps good
> idea to create one for people that will upgrade from 0.1x to 1.0 or
> even from 0.10 to 0.11?) or in some FAQ section. I've seen this
> question come up at least 3 times in the past week which shows the
> importance of the issue/question :)
>
> --
> David Coallier
Agreed. Or else we should fix the bug in the first place. Or do both (mention in 0.11 UPGRADE, fix for 1.0)
Adam
Re: A simple comparative benchmark and look at include_docs
performance.
Posted by David Coallier <da...@gmail.com>.
>
> Yes. Were you running 0.10 before this build? Do you have a mochiweb-r97 in $PREFIX/lib/couchdb/erlang/lib? Due to a quirk in the VM code path algorithms, r97 is considered newer than the r113 that ships with 0.11, which produces unintended effects such as this.
>
> If you remove all old versions of apps in that directory the problem should go away. If you don't have any old versions, I'm not sure what to tell you. I've used mochijson2 with native code compilation successfully in the past. Best,
>
I think we should put this somewhere in the UPGRADE file (Perhaps good
idea to create one for people that will upgrade from 0.1x to 1.0 or
even from 0.10 to 0.11?) or in some FAQ section. I've seen this
question come up at least 3 times in the past week which shows the
importance of the issue/question :)
--
David Coallier
Re: A simple comparative benchmark and look at include_docs performance.
Posted by Adam Kocoloski <ko...@apache.org>.
On Apr 21, 2010, at 7:29 PM, Chris Stockton wrote:
> Hello,
>
> On Tue, Apr 20, 2010 at 7:36 PM, Adam Kocoloski <ko...@apache.org> wrote:
>> Hmm, so that was a bit of a brain dump :/ I'd say try out the native code compilation for mochijson2 if your Erlang build supports it (you should see a [hipe] in the erl banner). Cheers,
>
> I have built couchdb .11 with +native +inline flags, although we got a
> decent speed increase we unfortunately are seeing very unusual errors,
> some json that is perfectly valid is being rejected. A example is
> below:
>
> [Thu, 22 Apr 2010 05:56:02 GMT] [debug] [<0.1981.0>] 'PUT'
> /db_1/67a6629e56327fb90a07aa79432f7d81 {1,1}
> Headers: [{'Accept',"*/*"},
> {'Authorization',"Basic bmV0d29yazpzYXZldGhlU1FM"},
> {'Content-Length',"198"},
> {'Content-Type',"application/json; charset=ascii"},
> {"Expect","100-continue"},
> {'Host',"g1dlsdbdb001.dev.glbt1.gdg:59845"},
> {'User-Agent',"PECL::HTTP/1.6.5 (PHP/5.3.0)"}]
>
> [Thu, 22 Apr 2010 05:56:02 GMT] [debug] [<0.1981.0>] OAuth Params: []
>
> [Thu, 22 Apr 2010 05:56:02 GMT] [error] [<0.1981.0>] attempted
> upload of invalid JSON
> {"_id":"67a6629e56327fb90a07aa79432f7d81","_rev":"10-0e5902b082406967be785682eebb69a4","data":{"C003651":"asfsafasfdasfd","C012193":"asdfasdfasdfsafdasdfsf"},"type":"model","model":"M4bccfd8c44043"}
>
> [Thu, 22 Apr 2010 05:56:02 GMT] [info] [<0.1981.0>] 172.19.68.122 -
> - 'PUT' /db_1/67a6629e56327fb90a07aa79432f7d81 400
>
> [Thu, 22 Apr 2010 05:56:02 GMT] [debug] [<0.1981.0>] httpd 400 error response:
> {"error":"bad_request","reason":"invalid UTF-8 JSON"}
>
> Any thoughts on this by chance?
Yes. Were you running 0.10 before this build? Do you have a mochiweb-r97 in $PREFIX/lib/couchdb/erlang/lib? Due to a quirk in the VM code path algorithms, r97 is considered newer than the r113 that ships with 0.11, which produces unintended effects such as this.
If you remove all old versions of apps in that directory the problem should go away. If you don't have any old versions, I'm not sure what to tell you. I've used mochijson2 with native code compilation successfully in the past. Best,
Adam
>
> Kind Regards,
>
> -Chris
Re: A simple comparative benchmark and look at include_docs
performance.
Posted by Chris Stockton <ch...@gmail.com>.
Hello,
On Tue, Apr 20, 2010 at 7:36 PM, Adam Kocoloski <ko...@apache.org> wrote:
> Hmm, so that was a bit of a brain dump :/ I'd say try out the native code compilation for mochijson2 if your Erlang build supports it (you should see a [hipe] in the erl banner). Cheers,
I have built couchdb .11 with +native +inline flags, although we got a
decent speed increase we unfortunately are seeing very unusual errors,
some json that is perfectly valid is being rejected. A example is
below:
[Thu, 22 Apr 2010 05:56:02 GMT] [debug] [<0.1981.0>] 'PUT'
/db_1/67a6629e56327fb90a07aa79432f7d81 {1,1}
Headers: [{'Accept',"*/*"},
{'Authorization',"Basic bmV0d29yazpzYXZldGhlU1FM"},
{'Content-Length',"198"},
{'Content-Type',"application/json; charset=ascii"},
{"Expect","100-continue"},
{'Host',"g1dlsdbdb001.dev.glbt1.gdg:59845"},
{'User-Agent',"PECL::HTTP/1.6.5 (PHP/5.3.0)"}]
[Thu, 22 Apr 2010 05:56:02 GMT] [debug] [<0.1981.0>] OAuth Params: []
[Thu, 22 Apr 2010 05:56:02 GMT] [error] [<0.1981.0>] attempted
upload of invalid JSON
{"_id":"67a6629e56327fb90a07aa79432f7d81","_rev":"10-0e5902b082406967be785682eebb69a4","data":{"C003651":"asfsafasfdasfd","C012193":"asdfasdfasdfsafdasdfsf"},"type":"model","model":"M4bccfd8c44043"}
[Thu, 22 Apr 2010 05:56:02 GMT] [info] [<0.1981.0>] 172.19.68.122 -
- 'PUT' /db_1/67a6629e56327fb90a07aa79432f7d81 400
[Thu, 22 Apr 2010 05:56:02 GMT] [debug] [<0.1981.0>] httpd 400 error response:
{"error":"bad_request","reason":"invalid UTF-8 JSON"}
Any thoughts on this by chance?
Kind Regards,
-Chris
Re: A simple comparative benchmark and look at include_docs performance.
Posted by Adam Kocoloski <ko...@apache.org>.
Hi Chris, thanks for this report. So, next steps:
1) What Erlang VM version are you using? Does it have native code capabilities? Pasting the banner from an invocation of `erl` would suffice.
2) Two common compiler optimizations are +native and +inline. The first one causes the compiler to generate native machine code instead of BEAM bytecode for a particular module. You need to have a VM that supports it (hence my question above). The second one inlines functions, as you might have guessed. json_bin_is_safe/2 is recursive, so I'm not sure +inline makes a lot of sense in this case. You can activate these by setting the ERLC_FLAGS environment variable when you build.
json_bin_is_safe is an old optimization; the VM has improved dramatically in the past 2 years, and its handling of binaries has changed in subtle ways. It could be that the assumptions that went into that code are no longer true. I'd have to do some experimentation myself to say for certain.
In the longer term, we've done some work on a fast interface to the excellent Yajl JSON library. JSON processing in C has a different performance profile that may be better suited to Couch's use case. But again, we'll need to measure.
Hmm, so that was a bit of a brain dump :/ I'd say try out the native code compilation for mochijson2 if your Erlang build supports it (you should see a [hipe] in the erl banner). Cheers,
Adam
On Apr 20, 2010, at 7:48 PM, Chris Stockton wrote:
> Hello,
>
> After some profiling we found some pretty big slowdowns as you
> suspected in the json encoding. I am very surprised by this, it seems
> that there is a lot of room for improvement in the json encoding side
> of things. I do not know the inner workings of erlang, but seeing that
> this call: mochijson2:json_bin_is_safe/ is 12% of the work of the
> response, but is a very simple string check perhaps it can be inlined
> into the callers? That would be a significant speedup in most
> languages.
>
> -Chris
>
> ----------------------------------
>
> 24> eprof:total_analyse().
> FUNCTION CALLS TIME
> user_drv:io_request/3 266 19 %
> mochijson2:json_bin_is_safe/1 59095133 12 %
> erlang:port_command/2 2445617 3 %
> gen:wait_resp_mon/3 2449385 3 %
> mochijson2:json_encode_string/2 6111299 3 %
> couch_util:to_hex/1 3114803 3 %
> couch_file:read_raw_iolist/3 2272537 3 %
> mochijson2:'-json_encode_proplist/2-fun-0-'/3 3317537 2 %
> gen_server:loop/6 2281063 2 %
> prim_file:drv_command/4 2272768 2 %
> lists:foldl/3 4435469 2 %
> gen_server:handle_msg/5 2281060 2 %
> gen:do_call/4 2449386 2 %
> couch_btree:find_first_gteq/5 3914715 2 %
> prim_file:drv_get_response/1 2272768 2 %
> prim_file:pread/3 2272542 1 %
> couch_btree:lookup_kpnode/5 1380410 1 %
> couch_file:pread_iolist/2 1049994 1 %
> couch_file:handle_call/3 2272598 1 %
> gen_server:call/3 2272615 1 %
> prim_file:reverse/2 2272542 1 %
> prim_file:drv_command/2 2272767 1 %
> prim_file:translate_response/2 2272768 1 %
> couch_file:pread_term/2 1049994 1 %
> couch_btree:less/3 4088828 1 %
> couch_file:pread_binary/2 1049994 1 %
> couch_util:to_digit/1 5537536 1 %
> couch_db_updater:'-init_db/4-fun-0-'/2 4088460 1 %
> prim_file:transform_ldata/4 2272542 1 %
> mochijson2:json_encode/2 3490228 1 %
> erlang:binary_to_term/1 1049995 1 %
> erlang:monitor/2 2449861 1 %
> couch_btree:lookup/3 862756 1 %
> file:pread/3 2272542 1 %
> gen_server:reply/2 2276495 1 %
> couch_db_updater:less_docid/2 4088764 1 %
> lists:splitwith/3 1380530 1 %
> prim_file:transform_ldata/1 2272542 1 %
> couch_file:remove_block_prefixes/2 2618126 1 %
> mochijson2:json_string_is_safe/1 2934384 1 %
> gen:call/4 2449386 1 %
> erlang:demonitor/1 2449849 1 %
> lists:reverse/2 3485939 1 %
> lists:reverse/1 2249330 0 %
> couch_doc:to_json_obj/2 172539 0 %
> prim_file:drv_command/3 2272767 0 %
> erlang:iolist_to_binary/1 2273617 0 %
> couch_file:calculate_total_read_len/2 2272684 0 %
> couch_view:less_json/2 517614 0 %
> couch_db:open_doc_int/3 345101 0 %
> gen_server:decode_msg/8 2281060 0 %
> couch_btree:get_node/2 877445 0 %
> couch_db:doc_meta_info/3 172549 0 %
> erlang:md5_trap/2 0 0 %
> couch_key_tree:get_full_key_paths/4 517831 0 %
> prim_file:drv_get_response/2 2272768 0 %
> erlang:bump_reductions/1 2272768 0 %
> couch_btree:lookup_kvnode/5 345102 0 %
> couch_btree:'-lookup_kpnode/5-fun-0-'/3 690205 0 %
> erlang:'++'/2 2074206 0 %
> couch_doc:to_doc_info_path/1 172853 0 %
> mochijson2:json_encode_proplist/2 519798 0 %
> couch_httpd_view:send_json_view_row/5 172508 0 %
> dict:on_bucket/3 172906 0 %
> couch_key_tree:map_simple/3 518815 0 %
> gen_event:fetch_msg/5 173052 0 %
> gen_event:server_call/4 172890 0 %
> couch_key_tree:map/2 345706 0 %
> couch_httpd_view:view_row_with_doc/3 172485 0 %
> couch_btree:lookup/2 172552 0 %
> couch_doc:to_json_rev/2 172539 0 %
> dict:get_slot/2 346014 0 %
> prim_inet:send/2 172843 0 %
> lists:map/2 652977 0 %
> couch_httpd_view:'-make_view_fold_fun/6-fun-0 172544 0 %
Re: A simple comparative benchmark and look at include_docs
performance.
Posted by Chris Stockton <ch...@gmail.com>.
Hello,
After some profiling we found some pretty big slowdowns as you
suspected in the json encoding. I am very surprised by this, it seems
that there is a lot of room for improvement in the json encoding side
of things. I do not know the inner workings of erlang, but seeing that
this call: mochijson2:json_bin_is_safe/ is 12% of the work of the
response, but is a very simple string check perhaps it can be inlined
into the callers? That would be a significant speedup in most
languages.
-Chris
----------------------------------
24> eprof:total_analyse().
FUNCTION CALLS TIME
user_drv:io_request/3 266 19 %
mochijson2:json_bin_is_safe/1 59095133 12 %
erlang:port_command/2 2445617 3 %
gen:wait_resp_mon/3 2449385 3 %
mochijson2:json_encode_string/2 6111299 3 %
couch_util:to_hex/1 3114803 3 %
couch_file:read_raw_iolist/3 2272537 3 %
mochijson2:'-json_encode_proplist/2-fun-0-'/3 3317537 2 %
gen_server:loop/6 2281063 2 %
prim_file:drv_command/4 2272768 2 %
lists:foldl/3 4435469 2 %
gen_server:handle_msg/5 2281060 2 %
gen:do_call/4 2449386 2 %
couch_btree:find_first_gteq/5 3914715 2 %
prim_file:drv_get_response/1 2272768 2 %
prim_file:pread/3 2272542 1 %
couch_btree:lookup_kpnode/5 1380410 1 %
couch_file:pread_iolist/2 1049994 1 %
couch_file:handle_call/3 2272598 1 %
gen_server:call/3 2272615 1 %
prim_file:reverse/2 2272542 1 %
prim_file:drv_command/2 2272767 1 %
prim_file:translate_response/2 2272768 1 %
couch_file:pread_term/2 1049994 1 %
couch_btree:less/3 4088828 1 %
couch_file:pread_binary/2 1049994 1 %
couch_util:to_digit/1 5537536 1 %
couch_db_updater:'-init_db/4-fun-0-'/2 4088460 1 %
prim_file:transform_ldata/4 2272542 1 %
mochijson2:json_encode/2 3490228 1 %
erlang:binary_to_term/1 1049995 1 %
erlang:monitor/2 2449861 1 %
couch_btree:lookup/3 862756 1 %
file:pread/3 2272542 1 %
gen_server:reply/2 2276495 1 %
couch_db_updater:less_docid/2 4088764 1 %
lists:splitwith/3 1380530 1 %
prim_file:transform_ldata/1 2272542 1 %
couch_file:remove_block_prefixes/2 2618126 1 %
mochijson2:json_string_is_safe/1 2934384 1 %
gen:call/4 2449386 1 %
erlang:demonitor/1 2449849 1 %
lists:reverse/2 3485939 1 %
lists:reverse/1 2249330 0 %
couch_doc:to_json_obj/2 172539 0 %
prim_file:drv_command/3 2272767 0 %
erlang:iolist_to_binary/1 2273617 0 %
couch_file:calculate_total_read_len/2 2272684 0 %
couch_view:less_json/2 517614 0 %
couch_db:open_doc_int/3 345101 0 %
gen_server:decode_msg/8 2281060 0 %
couch_btree:get_node/2 877445 0 %
couch_db:doc_meta_info/3 172549 0 %
erlang:md5_trap/2 0 0 %
couch_key_tree:get_full_key_paths/4 517831 0 %
prim_file:drv_get_response/2 2272768 0 %
erlang:bump_reductions/1 2272768 0 %
couch_btree:lookup_kvnode/5 345102 0 %
couch_btree:'-lookup_kpnode/5-fun-0-'/3 690205 0 %
erlang:'++'/2 2074206 0 %
couch_doc:to_doc_info_path/1 172853 0 %
mochijson2:json_encode_proplist/2 519798 0 %
couch_httpd_view:send_json_view_row/5 172508 0 %
dict:on_bucket/3 172906 0 %
couch_key_tree:map_simple/3 518815 0 %
gen_event:fetch_msg/5 173052 0 %
gen_event:server_call/4 172890 0 %
couch_key_tree:map/2 345706 0 %
couch_httpd_view:view_row_with_doc/3 172485 0 %
couch_btree:lookup/2 172552 0 %
couch_doc:to_json_rev/2 172539 0 %
dict:get_slot/2 346014 0 %
prim_inet:send/2 172843 0 %
lists:map/2 652977 0 %
couch_httpd_view:'-make_view_fold_fun/6-fun-0 172544 0 %
Re: A simple comparative benchmark and look at include_docs performance.
Posted by Adam Kocoloski <ko...@apache.org>.
On Apr 20, 2010, at 3:51 PM, Chris Stockton wrote:
> Hello,
>
> On Tue, Apr 20, 2010 at 10:58 AM, Adam Kocoloski <ko...@apache.org> wrote:
>> Hi Chris, for the type of access pattern in your benchmark I generally recommend to use emit(doc.model, doc) and avoid include_docs=true. include_docs introduces an extra lookup back in the DB for every row of your view. If you emit the document into the view index the index will get large but streaming requests such as yours can be accomplished with a minimum of disk IO.
>
> We have tried this approach and it was indeed faster, however we wound
> up with what I remember to be over 19G view file. For 300mb sized
> database this trade off did not seem reasonable, although disk is
> cheaper in many cases, we found the bloat to be unacceptable. Do you
> know of a way to limit the size of the view when including the doc?
> Additionally may I ask if include_docs = true has potential room for
> optimization?
You should make sure to compact the view index, it doesn't take too long and can offer some huge space savings (as well as better query time performance). You should expect the view index to be comparable in size to the DB in that case.
include_docs=true uses the same code path as a single-document GET request. I'm not aware of any extreme hot spots there. Of course we're always keeping an eye on performance and looking out for optimizations, especially as CouchDB stabilizes and heads towards 1.0.
>> On the other hand, your sar report shows negligible iowait, so perhaps that's not your immediate problem. It may be the case that you're CPU-limited in the (pure Erlang) JSON encoder, although I would've expected JSON encoding CPU usage to scale with network traffic.
>
> It would surprise me if 13mb of json encoding could cause such spikes
> in CPU. I also expected network traffic to scale with our CPU usage.
> Have you seen issues in this area before? At first thought I would
> think of the encoding stage as being one of the lighter areas in the
> request, given the simple nature of json.
A thought -- do your documents have a very large number of edits? I _have_ seen heavy CPU utilization when dealing with documents containing 100+ revisions. Even after those revisions have been compacted away, the revision tree hangs around and is processed for every single-document (and include_docs) request. I've profiled couch_key_tree as a significant bottleneck in that case.
If you do have a large number of revisions and you don't worry too much about spurious conflicts on replication, you can lower the _revs_limit setting for your DB to trim that history down a bit. The default value is 1000 revisions.
>> You might try running eprof while you do this test. It's quite heavyweight and will slow your system down. If you start couchdb with the -i flag you can get an Erlang shell and execute
>> <snip>
>
> This was good information and I will look into profiling with erlang.
> May I ask if any effort is currently being put into performance and
> optimization for couchdb?
Yes, all the time, particularly as the codebase stabilizes. I've submitted performance-related patches for DB compaction and view key collation in the past week, for instance.
> I am also very interested in any reads on
> large-scall couchdb deployments, that are not so high-level (I.E.
> hardware specs, use cases, etc).
Ah, I'm not aware of so many low-level case studies like that. We should get around to writing up so of our accumulated experience at cloudant.com. Cheers,
Adam
>
> Kind Regards,
>
> -Chris