You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Manjunath Somashekhar <ma...@yahoo.com> on 2009/04/01 19:12:07 UTC
Suggestions on View performance optimization/improvement
hi All,
We have been using couchdb (built out of trunk) for prototyping an idea and would like to thank and congratulate you folks for a simple and usable schema free db.
We plan to store few million documents in couchdb and we would like to create couple of views to fetch the data appropriately. We have inserted a million documents (each containing about 20 fields). We are indexing/creating a view on a particular field of the document. The map function of the view is simple straight forward emit (emit(doc.field, doc)). It takes about 90 mins to build the required B-Tree index the first time. All the subsequent queries are performing extremely well (milli second responses). Can anything be done to reduce the 90 mins taken to build the required B-Tree index the first time?
Environment details:
Couchdb - 0.9.0a757326
Erlang - 5.6.5
Linux kernel - 2.6.24-23-generic #1 SMP Mon Jan 26 00:13:11 UTC 2009 i686 GNU/Linux
Ubuntu distribution
Centrino Dual core, 4GB RAM laptop
Thanks
Manju
Re: Suggestions on View performance optimization/improvement
Posted by Adam Kocoloski <ko...@apache.org>.
On Apr 2, 2009, at 1:51 PM, Luke Tucker wrote:
>
> On Apr 1, 2009, at 1:37 PM, Paul Davis wrote:
>>
>> And then last but not least is adding
>> concurrency to passing documents through the JS process. The last bit
>> is probably easiest, but most reports say that couchjs is sitting
>> around 30% cpu indicating its not really the bottleneck quite yet.
>>
>> HTH,
>> Paul Davis
>
>
>
> Does couchdb supply the couchjs process with work to do while it is
> performing i/o with previous results?
Nope.
> If not, I could also see this cpu utilization partially resulting
> from cpu and i/o being used too serially vs. pipelined -- ie i/o may
> very well be the bottleneck, but is there some room for improvement
> before we're actually hitting it?
In my opinion, yes, there's quite a bit of room. I've been meaning to
code that pipelining up but haven't gotten around to it. Cheers,
Adam
Re: Suggestions on View performance optimization/improvement
Posted by Luke Tucker <lt...@openplans.org>.
On Apr 1, 2009, at 1:37 PM, Paul Davis wrote:
>
> And then last but not least is adding
> concurrency to passing documents through the JS process. The last bit
> is probably easiest, but most reports say that couchjs is sitting
> around 30% cpu indicating its not really the bottleneck quite yet.
>
> HTH,
> Paul Davis
Does couchdb supply the couchjs process with work to do while it is
performing i/o with previous results? If not, I could also see this
cpu utilization partially resulting from cpu and i/o being used too
serially vs. pipelined -- ie i/o may very well be the bottleneck, but
is there some room for improvement before we're actually hitting it?
Wishing I understood erlang better already :)
- Luke
Re: Suggestions on View performance optimization/improvement
Posted by Paul Davis <pa...@gmail.com>.
> Can anything be done to reduce the 90 mins taken to build the required B-Tree index the first time?
Probably :D There has been nearly zero effort in making the view
generation faster. Other than a couple experiments here and there by
various people, no one has even really spent time benchmarking and
measuring the various bits to see what's actually taking time.
At the moment, the big three contenders are disk io, for which there's
little hope to optimize other than maybe some clever thinking on
ordering of writes to the btree. Moving JSON conversion from Erlang to
C has had some work put into it, but who's to say when or if that
would hit Erlang trunk. And then last but not least is adding
concurrency to passing documents through the JS process. The last bit
is probably easiest, but most reports say that couchjs is sitting
around 30% cpu indicating its not really the bottleneck quite yet.
The only thing I can say at the moment is either, wait a bit for the
optimization work, submit a patch that has a demonstrable effect, or
submit reports and scripts to get timings of view generation on a
public reference database.
Also, for JSON in C, the most current code is at
http://github.com/davisp/eep0018 its been used with trunk to some
extent, but there are still some issues with 100% compatibility.
HTH,
Paul Davis
Re: Suggestions on View performance optimization/improvement
Posted by Paul Davis <pa...@gmail.com>.
On Wed, Apr 1, 2009 at 1:51 PM, Michael McDaniel <co...@autosys.us> wrote:
> On Thu, Apr 02, 2009 at 12:31:17AM +0700, Jason Smith wrote:
>> I'd be very interested to know the performance impact of that
>> optimization as well. What is the overhead or bottleneck with large
>> view values? Estimating 100 bytes per key/value pair within each of the
>> million documents, that's 2GB of raw data, which should write to a
>> laptop disk within 2 minutes.
>>
>> I'm wondering whether it matters how large the view values are, since
>> they would seem not to be involved in the view processing very
>> much--only written to disk in the order defined by the keys.
>>
>> Of course, that goes against the common wisdom that the fastest thing to
>> do is emit(key, null); but that could impact the application
>> significantly since you have to query again for the documents. (I'm
>> unsure whether include_docs has a performance penalty either.)
>>
>> I guess what I'm asking is, why does the value side of views impact
>> performance so greatly?
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>
> last I checked, there is an Erlang term() <-> JSON <-> Erlang term()
> conversion for values on the initial view (to/from view server)
>
Not to mention the JSON -> JS_Object* -> JSON inside of couchjs though
that's probably quite a bit quicker :D
> ~M
>
>>
>> kowsik wrote:
>>> I would highly recommend that you do emit(doc.field, null) so that the
>>> key space doesn't get unwieldy and large. Since the id of the document
>>> is part of the map results, you can always fetch it using
>>> include_docs=true.
>>>
>>> K.
>>>
>>> On Wed, Apr 1, 2009 at 10:12 AM, Manjunath Somashekhar
>>> <ma...@yahoo.com> wrote:
>>>> hi All,
>>>>
>>>> We have been using couchdb (built out of trunk) for prototyping an idea and would like to thank and congratulate you folks for a simple and usable schema free db.
>>>>
>>>> We plan to store few million documents in couchdb and we would like to create couple of views to fetch the data appropriately. We have inserted a million documents (each containing about 20 fields). We are indexing/creating a view on a particular field of the document. The map function of the view is simple straight forward emit (emit(doc.field, doc)). It takes about 90 mins to build the required B-Tree index the first time. All the subsequent queries are performing extremely well (milli second responses). Can anything be done to reduce the 90 mins taken to build the required B-Tree index the first time?
>>>>
>>>> Environment details:
>>>> Couchdb - 0.9.0a757326
>>>> Erlang - 5.6.5
>>>> Linux kernel - 2.6.24-23-generic #1 SMP Mon Jan 26 00:13:11 UTC 2009 i686 GNU/Linux
>>>> Ubuntu distribution
>>>> Centrino Dual core, 4GB RAM laptop
>>>>
>>>> Thanks
>>>> Manju
>>>>
>>>>
>>>>
>>>>
>>
>> --
>> Jason Smith
>> Proven Corporation
>> Bangkok, Thailand
>> http://www.proven-corporation.com
>
> --
> Michael McDaniel
> Portland, Oregon, USA
> http://trip.autosys.us
> http://autosys.us
> http://mmcdaniel.com/erlview
>
>
Re: Suggestions on View performance optimization/improvement
Posted by Michael McDaniel <co...@autosys.us>.
On Thu, Apr 02, 2009 at 12:31:17AM +0700, Jason Smith wrote:
> I'd be very interested to know the performance impact of that
> optimization as well. What is the overhead or bottleneck with large
> view values? Estimating 100 bytes per key/value pair within each of the
> million documents, that's 2GB of raw data, which should write to a
> laptop disk within 2 minutes.
>
> I'm wondering whether it matters how large the view values are, since
> they would seem not to be involved in the view processing very
> much--only written to disk in the order defined by the keys.
>
> Of course, that goes against the common wisdom that the fastest thing to
> do is emit(key, null); but that could impact the application
> significantly since you have to query again for the documents. (I'm
> unsure whether include_docs has a performance penalty either.)
>
> I guess what I'm asking is, why does the value side of views impact
> performance so greatly?
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
last I checked, there is an Erlang term() <-> JSON <-> Erlang term()
conversion for values on the initial view (to/from view server)
~M
>
> kowsik wrote:
>> I would highly recommend that you do emit(doc.field, null) so that the
>> key space doesn't get unwieldy and large. Since the id of the document
>> is part of the map results, you can always fetch it using
>> include_docs=true.
>>
>> K.
>>
>> On Wed, Apr 1, 2009 at 10:12 AM, Manjunath Somashekhar
>> <ma...@yahoo.com> wrote:
>>> hi All,
>>>
>>> We have been using couchdb (built out of trunk) for prototyping an idea and would like to thank and congratulate you folks for a simple and usable schema free db.
>>>
>>> We plan to store few million documents in couchdb and we would like to create couple of views to fetch the data appropriately. We have inserted a million documents (each containing about 20 fields). We are indexing/creating a view on a particular field of the document. The map function of the view is simple straight forward emit (emit(doc.field, doc)). It takes about 90 mins to build the required B-Tree index the first time. All the subsequent queries are performing extremely well (milli second responses). Can anything be done to reduce the 90 mins taken to build the required B-Tree index the first time?
>>>
>>> Environment details:
>>> Couchdb - 0.9.0a757326
>>> Erlang - 5.6.5
>>> Linux kernel - 2.6.24-23-generic #1 SMP Mon Jan 26 00:13:11 UTC 2009 i686 GNU/Linux
>>> Ubuntu distribution
>>> Centrino Dual core, 4GB RAM laptop
>>>
>>> Thanks
>>> Manju
>>>
>>>
>>>
>>>
>
> --
> Jason Smith
> Proven Corporation
> Bangkok, Thailand
> http://www.proven-corporation.com
--
Michael McDaniel
Portland, Oregon, USA
http://trip.autosys.us
http://autosys.us
http://mmcdaniel.com/erlview
Re: Suggestions on View performance optimization/improvement
Posted by Jason Smith <jh...@proven-corporation.com>.
Paul Davis wrote:
> That said, the best way to find out would be to measure. I don't think
> I've seen numbers on this yet so anything you can show would be
> definitely be valuable information. Especially if you can demonstrate
> the trade offs between emit(key, doc); vs emit(key, null) &
> include_docs=True
That's an interesting challenge ("task," not "dare"). Feeling satisfied
with my "Getting started on Amazon EC2" page, I've been thinking about
building a system where people could easily set up a cluster for
benchmarking.
If the instructions and configuration were on the wiki, one could set up
his or her own cluster of 5 dual-core systems (e.g. four servers, one
torture tester) which could run for an hour and report the results.
That would set you back US $2.00, or a bit extra for the block devices,
which are rumored to have gigabit throughput to the servers.
(Mmmmmmmm.)
--
Jason Smith
Proven Corporation
Bangkok, Thailand
http://www.proven-corporation.com
Re: Suggestions on View performance optimization/improvement
Posted by Paul Davis <pa...@gmail.com>.
On Wed, Apr 1, 2009 at 1:31 PM, Jason Smith <jh...@proven-corporation.com> wrote:
> I'd be very interested to know the performance impact of that optimization
> as well. What is the overhead or bottleneck with large view values?
> Estimating 100 bytes per key/value pair within each of the million
> documents, that's 2GB of raw data, which should write to a laptop disk
> within 2 minutes.
>
> I'm wondering whether it matters how large the view values are, since they
> would seem not to be involved in the view processing very much--only written
> to disk in the order defined by the keys.
>
> Of course, that goes against the common wisdom that the fastest thing to do
> is emit(key, null); but that could impact the application significantly
> since you have to query again for the documents. (I'm unsure whether
> include_docs has a performance penalty either.)
>
> I guess what I'm asking is, why does the value side of views impact
> performance so greatly?
>
Other than the extra disk I/O, there's also the necessary
term_to_binary calls that add overhead. No idea how much that factors
in though. For include_docs, you're adding an N log(N) cost to reading
from the view but I haven't the slightest how that might translate to
wall clock time. It'd be an interesting thing to measure though. If
you include_docs for M rows with M 1 -> 1M or so and then find the
math to get an approximation for how long it takes to read through the
tree. I'm almost tempted but work is calling.
That said, the best way to find out would be to measure. I don't think
I've seen numbers on this yet so anything you can show would be
definitely be valuable information. Especially if you can demonstrate
the trade offs between emit(key, doc); vs emit(key, null) &
include_docs=True
HTH,
Paul Davis
> kowsik wrote:
>>
>> I would highly recommend that you do emit(doc.field, null) so that the
>> key space doesn't get unwieldy and large. Since the id of the document
>> is part of the map results, you can always fetch it using
>> include_docs=true.
>>
>> K.
>>
>> On Wed, Apr 1, 2009 at 10:12 AM, Manjunath Somashekhar
>> <ma...@yahoo.com> wrote:
>>>
>>> hi All,
>>>
>>> We have been using couchdb (built out of trunk) for prototyping an idea
>>> and would like to thank and congratulate you folks for a simple and usable
>>> schema free db.
>>>
>>> We plan to store few million documents in couchdb and we would like to
>>> create couple of views to fetch the data appropriately. We have inserted a
>>> million documents (each containing about 20 fields). We are
>>> indexing/creating a view on a particular field of the document. The map
>>> function of the view is simple straight forward emit (emit(doc.field, doc)).
>>> It takes about 90 mins to build the required B-Tree index the first time.
>>> All the subsequent queries are performing extremely well (milli second
>>> responses). Can anything be done to reduce the 90 mins taken to build the
>>> required B-Tree index the first time?
>>>
>>> Environment details:
>>> Couchdb - 0.9.0a757326
>>> Erlang - 5.6.5
>>> Linux kernel - 2.6.24-23-generic #1 SMP Mon Jan 26 00:13:11 UTC 2009 i686
>>> GNU/Linux
>>> Ubuntu distribution
>>> Centrino Dual core, 4GB RAM laptop
>>>
>>> Thanks
>>> Manju
>>>
>>>
>>>
>>>
>
> --
> Jason Smith
> Proven Corporation
> Bangkok, Thailand
> http://www.proven-corporation.com
>
Re: Suggestions on View performance optimization/improvement
Posted by Jason Smith <jh...@proven-corporation.com>.
I'd be very interested to know the performance impact of that
optimization as well. What is the overhead or bottleneck with large
view values? Estimating 100 bytes per key/value pair within each of the
million documents, that's 2GB of raw data, which should write to a
laptop disk within 2 minutes.
I'm wondering whether it matters how large the view values are, since
they would seem not to be involved in the view processing very
much--only written to disk in the order defined by the keys.
Of course, that goes against the common wisdom that the fastest thing to
do is emit(key, null); but that could impact the application
significantly since you have to query again for the documents. (I'm
unsure whether include_docs has a performance penalty either.)
I guess what I'm asking is, why does the value side of views impact
performance so greatly?
kowsik wrote:
> I would highly recommend that you do emit(doc.field, null) so that the
> key space doesn't get unwieldy and large. Since the id of the document
> is part of the map results, you can always fetch it using
> include_docs=true.
>
> K.
>
> On Wed, Apr 1, 2009 at 10:12 AM, Manjunath Somashekhar
> <ma...@yahoo.com> wrote:
>> hi All,
>>
>> We have been using couchdb (built out of trunk) for prototyping an idea and would like to thank and congratulate you folks for a simple and usable schema free db.
>>
>> We plan to store few million documents in couchdb and we would like to create couple of views to fetch the data appropriately. We have inserted a million documents (each containing about 20 fields). We are indexing/creating a view on a particular field of the document. The map function of the view is simple straight forward emit (emit(doc.field, doc)). It takes about 90 mins to build the required B-Tree index the first time. All the subsequent queries are performing extremely well (milli second responses). Can anything be done to reduce the 90 mins taken to build the required B-Tree index the first time?
>>
>> Environment details:
>> Couchdb - 0.9.0a757326
>> Erlang - 5.6.5
>> Linux kernel - 2.6.24-23-generic #1 SMP Mon Jan 26 00:13:11 UTC 2009 i686 GNU/Linux
>> Ubuntu distribution
>> Centrino Dual core, 4GB RAM laptop
>>
>> Thanks
>> Manju
>>
>>
>>
>>
--
Jason Smith
Proven Corporation
Bangkok, Thailand
http://www.proven-corporation.com
Re: Suggestions on View performance optimization/improvement
Posted by Paul Davis <pa...@gmail.com>.
Nope, include_docs=true throws an error when hitting a reduce view.
On Wed, Apr 1, 2009 at 1:26 PM, Adam Wolff <aw...@gmail.com> wrote:
> Related question though: can you use include_docs=true with reduce, or
> do you have to make a separate query?
>
> A
>
> On Wed, Apr 1, 2009 at 10:16 AM, kowsik <ko...@gmail.com> wrote:
>> I would highly recommend that you do emit(doc.field, null) so that the
>> key space doesn't get unwieldy and large. Since the id of the document
>> is part of the map results, you can always fetch it using
>> include_docs=true.
>>
>> K.
>>
>> On Wed, Apr 1, 2009 at 10:12 AM, Manjunath Somashekhar
>> <ma...@yahoo.com> wrote:
>>>
>>> hi All,
>>>
>>> We have been using couchdb (built out of trunk) for prototyping an idea and would like to thank and congratulate you folks for a simple and usable schema free db.
>>>
>>> We plan to store few million documents in couchdb and we would like to create couple of views to fetch the data appropriately. We have inserted a million documents (each containing about 20 fields). We are indexing/creating a view on a particular field of the document. The map function of the view is simple straight forward emit (emit(doc.field, doc)). It takes about 90 mins to build the required B-Tree index the first time. All the subsequent queries are performing extremely well (milli second responses). Can anything be done to reduce the 90 mins taken to build the required B-Tree index the first time?
>>>
>>> Environment details:
>>> Couchdb - 0.9.0a757326
>>> Erlang - 5.6.5
>>> Linux kernel - 2.6.24-23-generic #1 SMP Mon Jan 26 00:13:11 UTC 2009 i686 GNU/Linux
>>> Ubuntu distribution
>>> Centrino Dual core, 4GB RAM laptop
>>>
>>> Thanks
>>> Manju
>>>
>>>
>>>
>>>
>>
>
Re: Suggestions on View performance optimization/improvement
Posted by Adam Wolff <aw...@gmail.com>.
Related question though: can you use include_docs=true with reduce, or
do you have to make a separate query?
A
On Wed, Apr 1, 2009 at 10:16 AM, kowsik <ko...@gmail.com> wrote:
> I would highly recommend that you do emit(doc.field, null) so that the
> key space doesn't get unwieldy and large. Since the id of the document
> is part of the map results, you can always fetch it using
> include_docs=true.
>
> K.
>
> On Wed, Apr 1, 2009 at 10:12 AM, Manjunath Somashekhar
> <ma...@yahoo.com> wrote:
>>
>> hi All,
>>
>> We have been using couchdb (built out of trunk) for prototyping an idea and would like to thank and congratulate you folks for a simple and usable schema free db.
>>
>> We plan to store few million documents in couchdb and we would like to create couple of views to fetch the data appropriately. We have inserted a million documents (each containing about 20 fields). We are indexing/creating a view on a particular field of the document. The map function of the view is simple straight forward emit (emit(doc.field, doc)). It takes about 90 mins to build the required B-Tree index the first time. All the subsequent queries are performing extremely well (milli second responses). Can anything be done to reduce the 90 mins taken to build the required B-Tree index the first time?
>>
>> Environment details:
>> Couchdb - 0.9.0a757326
>> Erlang - 5.6.5
>> Linux kernel - 2.6.24-23-generic #1 SMP Mon Jan 26 00:13:11 UTC 2009 i686 GNU/Linux
>> Ubuntu distribution
>> Centrino Dual core, 4GB RAM laptop
>>
>> Thanks
>> Manju
>>
>>
>>
>>
>
Re: Suggestions on View performance optimization/improvement
Posted by kowsik <ko...@gmail.com>.
I would highly recommend that you do emit(doc.field, null) so that the
key space doesn't get unwieldy and large. Since the id of the document
is part of the map results, you can always fetch it using
include_docs=true.
K.
On Wed, Apr 1, 2009 at 10:12 AM, Manjunath Somashekhar
<ma...@yahoo.com> wrote:
>
> hi All,
>
> We have been using couchdb (built out of trunk) for prototyping an idea and would like to thank and congratulate you folks for a simple and usable schema free db.
>
> We plan to store few million documents in couchdb and we would like to create couple of views to fetch the data appropriately. We have inserted a million documents (each containing about 20 fields). We are indexing/creating a view on a particular field of the document. The map function of the view is simple straight forward emit (emit(doc.field, doc)). It takes about 90 mins to build the required B-Tree index the first time. All the subsequent queries are performing extremely well (milli second responses). Can anything be done to reduce the 90 mins taken to build the required B-Tree index the first time?
>
> Environment details:
> Couchdb - 0.9.0a757326
> Erlang - 5.6.5
> Linux kernel - 2.6.24-23-generic #1 SMP Mon Jan 26 00:13:11 UTC 2009 i686 GNU/Linux
> Ubuntu distribution
> Centrino Dual core, 4GB RAM laptop
>
> Thanks
> Manju
>
>
>
>