You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Jonathan Williamson <jo...@netcopy.co.uk> on 2012/03/23 12:06:04 UTC
Why are reads from CouchDB so slow? (1.5MB/s or thereabouts)
Hi everyone,
I posted this question on Stack Overflow and Jason Smith suggested I
try this group instead.
> I have a CouchDB (1.1.1) server running that contains a lot of documents in the 400-600KB size range.
>
> If I time fetching a full document from the database (not from a view, just the raw document) it takes 200-400ms to complete which equates to around 1.5MB/s throughput.
>
> If I write the same data to raw files on disk they load in 10-20ms (around 25-50 MB/s).
>
> I'd expect CouchDB to have some overhead, but an order of magnitude (and some) seems crazy for what is essentially a read!
>
> Can anyone shed some light onto why this might be the case?
>
> Update: As requested below, a timing from curl:
>
> # time curl http://localhost:5984/[dbname]/[documentname]
>
> real 0m0.684s
> user 0m0.004s
> sys 0m0.020s
>
> The fetched document was 642842 bytes. I've tested it on both a standard 1TB harddisk and an EC2 instance (EBS volume) with similar results.
There was some discussion about the post which can be found here:
http://stackoverflow.com/questions/9807200/why-are-reads-from-couchdb-so-slow-1-5mb-s-or-thereabouts
Thanks,
Jon.
Re: Why are reads from CouchDB so slow? (1.5MB/s or thereabouts)
Posted by Robert Newson <rn...@apache.org>.
I've lost count of the number of times I started on 'external attachments'. :)
On 23 March 2012 17:09, Jens Alfke <je...@couchbase.com> wrote:
>
> On Mar 23, 2012, at 9:56 AM, Robert Newson wrote:
>
>> One way to avoid this is to
>> use one file for documents and another for attachments. If TouchDB is
>> effectively doing that, then that explains how you escape the issue.
>
> Yes — TouchDB stores every attachment as a separate file. It’s a very simple content-indexed store, where the files are named according to their SHA-1 digests. The document metadata* contains attachment digests. (This then requires that compaction run a mark-sweep phase where it collects the digests of all available attachments, then deletes attachment files that don’t appear in that list.)
>
> —Jens
>
> * https://github.com/couchbaselabs/TouchDB-iOS/wiki/Object-Design-And-Schema
Re: Why are reads from CouchDB so slow? (1.5MB/s or thereabouts)
Posted by Jens Alfke <je...@couchbase.com>.
On Mar 23, 2012, at 9:56 AM, Robert Newson wrote:
> One way to avoid this is to
> use one file for documents and another for attachments. If TouchDB is
> effectively doing that, then that explains how you escape the issue.
Yes — TouchDB stores every attachment as a separate file. It’s a very simple content-indexed store, where the files are named according to their SHA-1 digests. The document metadata* contains attachment digests. (This then requires that compaction run a mark-sweep phase where it collects the digests of all available attachments, then deletes attachment files that don’t appear in that list.)
—Jens
* https://github.com/couchbaselabs/TouchDB-iOS/wiki/Object-Design-And-Schema
Re: Why are reads from CouchDB so slow? (1.5MB/s or thereabouts)
Posted by Robert Newson <rn...@apache.org>.
The issue in couchdb is that we need to distinguish a database footer
(which contains the positions of the roots of the by_id and by_seq
trees, etc) from one inside an attachment. One way to avoid this is to
use one file for documents and another for attachments. If TouchDB is
effectively doing that, then that explains how you escape the issue.
On 23 March 2012 16:43, Jens Alfke <je...@couchbase.com> wrote:
>
> On Mar 23, 2012, at 6:35 AM, Volker Mische wrote:
>
>> You are right. It won't be raw JSON, but it can be way closer to that
>> (hence less processing needed for sending it out) than it currently is
>> the case.
>
> TouchDB* stores document contents as raw JSON without the special (underscored) properties like _id, _rev, _attachments. When serving a document it re-inserts those properties based on document metadata; but it splices them into the document body data before the closing ‘}’, so it doesn’t have to parse the document JSON.
>
> (There are cases where it has to parse the document, but not on a basic GET.)
>
> —Jens
>
> * https://github.com/couchbaselabs/TouchDB-iOS
Re: Why are reads from CouchDB so slow? (1.5MB/s or thereabouts)
Posted by Jens Alfke <je...@couchbase.com>.
On Mar 23, 2012, at 6:35 AM, Volker Mische wrote:
> You are right. It won't be raw JSON, but it can be way closer to that
> (hence less processing needed for sending it out) than it currently is
> the case.
TouchDB* stores document contents as raw JSON without the special (underscored) properties like _id, _rev, _attachments. When serving a document it re-inserts those properties based on document metadata; but it splices them into the document body data before the closing ‘}’, so it doesn’t have to parse the document JSON.
(There are cases where it has to parse the document, but not on a basic GET.)
—Jens
* https://github.com/couchbaselabs/TouchDB-iOS
Re: Why are reads from CouchDB so slow? (1.5MB/s or thereabouts)
Posted by Robert Newson <rn...@apache.org>.
I would really like to get rid of the make_blocks thing as I had the
idea to make the compactor stitch attachments back to completely
contiguous on disk, but it's hard to find a replacement that works
there. Paul Davis had some ideas that reduce the frequency we need the
magic block bit significantly, so perhaps that's the first step.
b.
On 23 March 2012 13:35, Volker Mische <vo...@gmail.com> wrote:
> Hi Robert,
>
> On 03/23/2012 02:28 PM, Robert Newson wrote:
>> I'd be intrigued to know how we could store the raw JSON on disk given
>> the make_blocks behavior but, yes, the reason couchdb isn't giving
>> sendfile() like performance is the json<>erlang conversion at least,
>> and other things like reading btree nodes to find the data (even if
>> they come from cache), etc.
>
> You are right. It won't be raw JSON, but it can be way closer to that
> (hence less processing needed for sending it out) than it currently is
> the case.
>
> Cheers,
> Volker
Re: Why are reads from CouchDB so slow? (1.5MB/s or thereabouts)
Posted by Volker Mische <vo...@gmail.com>.
Hi Robert,
On 03/23/2012 02:28 PM, Robert Newson wrote:
> I'd be intrigued to know how we could store the raw JSON on disk given
> the make_blocks behavior but, yes, the reason couchdb isn't giving
> sendfile() like performance is the json<>erlang conversion at least,
> and other things like reading btree nodes to find the data (even if
> they come from cache), etc.
You are right. It won't be raw JSON, but it can be way closer to that
(hence less processing needed for sending it out) than it currently is
the case.
Cheers,
Volker
Re: Why are reads from CouchDB so slow? (1.5MB/s or thereabouts)
Posted by Robert Newson <rn...@apache.org>.
I'd be intrigued to know how we could store the raw JSON on disk given
the make_blocks behavior but, yes, the reason couchdb isn't giving
sendfile() like performance is the json<>erlang conversion at least,
and other things like reading btree nodes to find the data (even if
they come from cache), etc.
What I was driving at with 'use a benchmarking tool' was to eliminate
artifacts like the time curl takes to start the connection, etc. All
the tools I listed record the time of the actual request/response.
With ab and nodeload I can get similar figures, though I can also
crank up concurrency and get the same numbers for each request (but
10x the total throughput). Without those kinds of options (number of
users, tcp keep-alive, http keep-alive) it's very hard to discuss and
compare benchmarks. Curl just isn't enough.
B.
On 23 March 2012 13:23, Jonathan Williamson <jo...@netcopy.co.uk> wrote:
> Volker,
>
> Thanks for the input, that all sounds likely! It's not a massive
> problem for us to store a raw copy of our data alongside our Couch
> databases.
>
> I do however think this falls slightly under "unexpected behaviour" as
> intuitively I think a lot of people would expect raw read speeds to be
> pretty fast and not require such heavy CPU usage. That said it's easy
> to work around just a bit of a surprise to come across.
>
> Love CouchDB for all the things it does for us so well - it's a great product!
>
> Jon.
>
> On Fri, Mar 23, 2012 at 1:17 PM, Volker Mische <vo...@gmail.com> wrote:
>> I agree that using a trusted benchmarking tool is the way to go. Tough
>> what Jonathan sees is pretty clear. It's the JSON -> Eterm -> String
>> conversion that CouchDB is currently doing. Filipe proposed a patch that
>> store the raw JSON on disk, to get rid most of this conversion. I don't
>> remember exactly, but I'm pretty sure he provided sensible benchmarking
>> results back then.
>>
>> Hence the point of this thread shouldn't be: go, do it properly. But: we
>> have a clue why it is so slow, we don't store raw JSON, but Eterms on
>> disk, that need to be assembled to a string everytime you request it.
>>
>> Cheers,
>> Volker
Re: Why are reads from CouchDB so slow? (1.5MB/s or thereabouts)
Posted by Jonathan Williamson <jo...@netcopy.co.uk>.
Volker,
Thanks for the input, that all sounds likely! It's not a massive
problem for us to store a raw copy of our data alongside our Couch
databases.
I do however think this falls slightly under "unexpected behaviour" as
intuitively I think a lot of people would expect raw read speeds to be
pretty fast and not require such heavy CPU usage. That said it's easy
to work around just a bit of a surprise to come across.
Love CouchDB for all the things it does for us so well - it's a great product!
Jon.
On Fri, Mar 23, 2012 at 1:17 PM, Volker Mische <vo...@gmail.com> wrote:
> I agree that using a trusted benchmarking tool is the way to go. Tough
> what Jonathan sees is pretty clear. It's the JSON -> Eterm -> String
> conversion that CouchDB is currently doing. Filipe proposed a patch that
> store the raw JSON on disk, to get rid most of this conversion. I don't
> remember exactly, but I'm pretty sure he provided sensible benchmarking
> results back then.
>
> Hence the point of this thread shouldn't be: go, do it properly. But: we
> have a clue why it is so slow, we don't store raw JSON, but Eterms on
> disk, that need to be assembled to a string everytime you request it.
>
> Cheers,
> Volker
Re: Why are reads from CouchDB so slow? (1.5MB/s or thereabouts)
Posted by Volker Mische <vo...@gmail.com>.
I agree that using a trusted benchmarking tool is the way to go. Tough
what Jonathan sees is pretty clear. It's the JSON -> Eterm -> String
conversion that CouchDB is currently doing. Filipe proposed a patch that
store the raw JSON on disk, to get rid most of this conversion. I don't
remember exactly, but I'm pretty sure he provided sensible benchmarking
results back then.
Hence the point of this thread shouldn't be: go, do it properly. But: we
have a clue why it is so slow, we don't store raw JSON, but Eterms on
disk, that need to be assembled to a string everytime you request it.
Cheers,
Volker
On 03/23/2012 01:44 PM, Robert Newson wrote:
> I can confirm that you won't get nginx's sendfile level of
> performance, if that helps at all. However, if we're to investigate a
> performance issue I want to be sure that an issue has been competently
> discovered. The minimum for me would be seeing results from a trusted
> benchmarking tool.
>
> B.
>
> On 23 March 2012 12:41, Jonathan Williamson <jo...@netcopy.co.uk> wrote:
>> I'm aware of that - but it still doesn't explain the disparity. I was
>> using curl for the Nginx tests too...
>>
>> Is the reason you're asking the question because you believe the
>> performance should be much better than I'm seeing?
>>
>> On Fri, Mar 23, 2012 at 12:39 PM, Robert Newson <rn...@apache.org> wrote:
>>> curl is not a benchmarking tool. Each curl command is opening a new
>>> tcp connection, etc. Can you try with httperf, apachebench, tsung,
>>> nodeload?
>>>
>>> B.
>>>
>>> On 23 March 2012 12:36, Jonathan Williamson <jo...@netcopy.co.uk> wrote:
>>>> It was just a script that looped over the keys for 1000 documents,
>>>> fetching them one after another (using curl).
>>>>
>>>> On Fri, Mar 23, 2012 at 12:31 PM, Robert Newson <rn...@apache.org> wrote:
>>>>> Using which http benchmarking tool (and exactly which settings)?
>>>>>
>>>>> On 23 March 2012 12:30, Jonathan Williamson <jo...@netcopy.co.uk> wrote:
>>>>>> My initial CouchDB test was over 1000 GETs (all to roughly similar
>>>>>> documents), taking an average of the total time vs the total amount of
>>>>>> data recieved.
>>>>>>
>>>>>> On Fri, Mar 23, 2012 at 12:26 PM, Robert Newson <rn...@apache.org> wrote:
>>>>>>> Are you really extrapolating mb/s from a single curl command?
>>
>>
>>
>> --
>> Jon
>>
>> Jonathan Williamson
>> Systems Architect
>> :::::::::::::::::::::::::::::::::::::::::::
>> +44 (0) 7843 301 246 mobile
>> +44 (0) 208 242 6364 switch
>>
>> www.netcopy.co.uk
>> ::::::::::::::::::::::::::::::::::::::::::
>>
>> IMPORTANT NOTICE:
>> ========================================================================================
>> Netcopy Limited is a Company registered in England and Wales under
>> Company No. 07308350
>> Registered office: 2 East Common, Harpenden, Herts, AL5 1BJ
>> The contents of this email (and any attachment):
>> (1) are confidential and may be legally privileged - if it is not
>> meant for you, please tell the sender, do not forward or copy the
>> contents and delete it from your system immediately;
>> (2) come from its author and may not necessarily reflect the opinions
>> of Netcopy Ltd.
>> While emails and attachments are virus checked, we cannot accept any
>> liability in respect of any viruses.
>> We may monitor emails sent to Netcopy.
>> Want to know more about Netcopy?
>> Visit our website: www.netcopy.co.uk
Re: Why are reads from CouchDB so slow? (1.5MB/s or thereabouts)
Posted by Jason Smith <jh...@iriscouch.com>.
That is a good question. I would also like to see a performance
breakdown of where couch spends its time handling queries. Some kind
of profiler report.
On Fri, Mar 23, 2012 at 12:47 PM, Jonathan Williamson <jo...@netcopy.co.uk> wrote:
> I'm not claiming a performance issue at all, I was asking why :)
>
> If they are slow for a reason then so be it, we'll find a way to work
> around it. It's clear from the testing I've done what kind of
> throughput we can get for our specific use case.
>
> On Fri, Mar 23, 2012 at 12:44 PM, Robert Newson <rn...@apache.org> wrote:
>> I can confirm that you won't get nginx's sendfile level of
>> performance, if that helps at all. However, if we're to investigate a
>> performance issue I want to be sure that an issue has been competently
>> discovered. The minimum for me would be seeing results from a trusted
>> benchmarking tool.
>>
>> B.
--
Iris Couch
Re: Why are reads from CouchDB so slow? (1.5MB/s or thereabouts)
Posted by Jonathan Williamson <jo...@netcopy.co.uk>.
I'm not claiming a performance issue at all, I was asking why :)
If they are slow for a reason then so be it, we'll find a way to work
around it. It's clear from the testing I've done what kind of
throughput we can get for our specific use case.
On Fri, Mar 23, 2012 at 12:44 PM, Robert Newson <rn...@apache.org> wrote:
> I can confirm that you won't get nginx's sendfile level of
> performance, if that helps at all. However, if we're to investigate a
> performance issue I want to be sure that an issue has been competently
> discovered. The minimum for me would be seeing results from a trusted
> benchmarking tool.
>
> B.
Re: Why are reads from CouchDB so slow? (1.5MB/s or thereabouts)
Posted by Jonathan Williamson <jo...@netcopy.co.uk>.
It's set to true.
As an aside I haven't inserted any new documents into this database
for a few weeks, it's just a development snapshot of a real database.
There is no indexing running as all indexes are up to date.
On Fri, Mar 23, 2012 at 12:45 PM, Jason Smith <jh...@iriscouch.com> wrote:
> Hi, Jonathan. What is the configuration value you have set for
> "delayed_commits"?
>
> Thanks.
Re: Why are reads from CouchDB so slow? (1.5MB/s or thereabouts)
Posted by Jason Smith <jh...@iriscouch.com>.
Hi, Jonathan. What is the configuration value you have set for
"delayed_commits"?
Thanks.
On Fri, Mar 23, 2012 at 12:44 PM, Robert Newson <rn...@apache.org> wrote:
> I can confirm that you won't get nginx's sendfile level of
> performance, if that helps at all. However, if we're to investigate a
> performance issue I want to be sure that an issue has been competently
> discovered. The minimum for me would be seeing results from a trusted
> benchmarking tool.
>
> B.
>
> On 23 March 2012 12:41, Jonathan Williamson <jo...@netcopy.co.uk> wrote:
>> I'm aware of that - but it still doesn't explain the disparity. I was
>> using curl for the Nginx tests too...
>>
>> Is the reason you're asking the question because you believe the
>> performance should be much better than I'm seeing?
>>
>> On Fri, Mar 23, 2012 at 12:39 PM, Robert Newson <rn...@apache.org> wrote:
>>> curl is not a benchmarking tool. Each curl command is opening a new
>>> tcp connection, etc. Can you try with httperf, apachebench, tsung,
>>> nodeload?
>>>
>>> B.
>>>
>>> On 23 March 2012 12:36, Jonathan Williamson <jo...@netcopy.co.uk> wrote:
>>>> It was just a script that looped over the keys for 1000 documents,
>>>> fetching them one after another (using curl).
>>>>
>>>> On Fri, Mar 23, 2012 at 12:31 PM, Robert Newson <rn...@apache.org> wrote:
>>>>> Using which http benchmarking tool (and exactly which settings)?
>>>>>
>>>>> On 23 March 2012 12:30, Jonathan Williamson <jo...@netcopy.co.uk> wrote:
>>>>>> My initial CouchDB test was over 1000 GETs (all to roughly similar
>>>>>> documents), taking an average of the total time vs the total amount of
>>>>>> data recieved.
>>>>>>
>>>>>> On Fri, Mar 23, 2012 at 12:26 PM, Robert Newson <rn...@apache.org> wrote:
>>>>>>> Are you really extrapolating mb/s from a single curl command?
>>
>>
>>
>> --
>> Jon
>>
>> Jonathan Williamson
>> Systems Architect
>> :::::::::::::::::::::::::::::::::::::::::::
>> +44 (0) 7843 301 246 mobile
>> +44 (0) 208 242 6364 switch
>>
>> www.netcopy.co.uk
>> ::::::::::::::::::::::::::::::::::::::::::
>>
>> IMPORTANT NOTICE:
>> ========================================================================================
>> Netcopy Limited is a Company registered in England and Wales under
>> Company No. 07308350
>> Registered office: 2 East Common, Harpenden, Herts, AL5 1BJ
>> The contents of this email (and any attachment):
>> (1) are confidential and may be legally privileged - if it is not
>> meant for you, please tell the sender, do not forward or copy the
>> contents and delete it from your system immediately;
>> (2) come from its author and may not necessarily reflect the opinions
>> of Netcopy Ltd.
>> While emails and attachments are virus checked, we cannot accept any
>> liability in respect of any viruses.
>> We may monitor emails sent to Netcopy.
>> Want to know more about Netcopy?
>> Visit our website: www.netcopy.co.uk
--
Iris Couch
Re: Why are reads from CouchDB so slow? (1.5MB/s or thereabouts)
Posted by Robert Newson <rn...@apache.org>.
I can confirm that you won't get nginx's sendfile level of
performance, if that helps at all. However, if we're to investigate a
performance issue I want to be sure that an issue has been competently
discovered. The minimum for me would be seeing results from a trusted
benchmarking tool.
B.
On 23 March 2012 12:41, Jonathan Williamson <jo...@netcopy.co.uk> wrote:
> I'm aware of that - but it still doesn't explain the disparity. I was
> using curl for the Nginx tests too...
>
> Is the reason you're asking the question because you believe the
> performance should be much better than I'm seeing?
>
> On Fri, Mar 23, 2012 at 12:39 PM, Robert Newson <rn...@apache.org> wrote:
>> curl is not a benchmarking tool. Each curl command is opening a new
>> tcp connection, etc. Can you try with httperf, apachebench, tsung,
>> nodeload?
>>
>> B.
>>
>> On 23 March 2012 12:36, Jonathan Williamson <jo...@netcopy.co.uk> wrote:
>>> It was just a script that looped over the keys for 1000 documents,
>>> fetching them one after another (using curl).
>>>
>>> On Fri, Mar 23, 2012 at 12:31 PM, Robert Newson <rn...@apache.org> wrote:
>>>> Using which http benchmarking tool (and exactly which settings)?
>>>>
>>>> On 23 March 2012 12:30, Jonathan Williamson <jo...@netcopy.co.uk> wrote:
>>>>> My initial CouchDB test was over 1000 GETs (all to roughly similar
>>>>> documents), taking an average of the total time vs the total amount of
>>>>> data recieved.
>>>>>
>>>>> On Fri, Mar 23, 2012 at 12:26 PM, Robert Newson <rn...@apache.org> wrote:
>>>>>> Are you really extrapolating mb/s from a single curl command?
>
>
>
> --
> Jon
>
> Jonathan Williamson
> Systems Architect
> :::::::::::::::::::::::::::::::::::::::::::
> +44 (0) 7843 301 246 mobile
> +44 (0) 208 242 6364 switch
>
> www.netcopy.co.uk
> ::::::::::::::::::::::::::::::::::::::::::
>
> IMPORTANT NOTICE:
> ========================================================================================
> Netcopy Limited is a Company registered in England and Wales under
> Company No. 07308350
> Registered office: 2 East Common, Harpenden, Herts, AL5 1BJ
> The contents of this email (and any attachment):
> (1) are confidential and may be legally privileged - if it is not
> meant for you, please tell the sender, do not forward or copy the
> contents and delete it from your system immediately;
> (2) come from its author and may not necessarily reflect the opinions
> of Netcopy Ltd.
> While emails and attachments are virus checked, we cannot accept any
> liability in respect of any viruses.
> We may monitor emails sent to Netcopy.
> Want to know more about Netcopy?
> Visit our website: www.netcopy.co.uk
Re: Why are reads from CouchDB so slow? (1.5MB/s or thereabouts)
Posted by Jonathan Williamson <jo...@netcopy.co.uk>.
I'm aware of that - but it still doesn't explain the disparity. I was
using curl for the Nginx tests too...
Is the reason you're asking the question because you believe the
performance should be much better than I'm seeing?
On Fri, Mar 23, 2012 at 12:39 PM, Robert Newson <rn...@apache.org> wrote:
> curl is not a benchmarking tool. Each curl command is opening a new
> tcp connection, etc. Can you try with httperf, apachebench, tsung,
> nodeload?
>
> B.
>
> On 23 March 2012 12:36, Jonathan Williamson <jo...@netcopy.co.uk> wrote:
>> It was just a script that looped over the keys for 1000 documents,
>> fetching them one after another (using curl).
>>
>> On Fri, Mar 23, 2012 at 12:31 PM, Robert Newson <rn...@apache.org> wrote:
>>> Using which http benchmarking tool (and exactly which settings)?
>>>
>>> On 23 March 2012 12:30, Jonathan Williamson <jo...@netcopy.co.uk> wrote:
>>>> My initial CouchDB test was over 1000 GETs (all to roughly similar
>>>> documents), taking an average of the total time vs the total amount of
>>>> data recieved.
>>>>
>>>> On Fri, Mar 23, 2012 at 12:26 PM, Robert Newson <rn...@apache.org> wrote:
>>>>> Are you really extrapolating mb/s from a single curl command?
--
Jon
Jonathan Williamson
Systems Architect
:::::::::::::::::::::::::::::::::::::::::::
+44 (0) 7843 301 246 mobile
+44 (0) 208 242 6364 switch
www.netcopy.co.uk
::::::::::::::::::::::::::::::::::::::::::
IMPORTANT NOTICE:
========================================================================================
Netcopy Limited is a Company registered in England and Wales under
Company No. 07308350
Registered office: 2 East Common, Harpenden, Herts, AL5 1BJ
The contents of this email (and any attachment):
(1) are confidential and may be legally privileged - if it is not
meant for you, please tell the sender, do not forward or copy the
contents and delete it from your system immediately;
(2) come from its author and may not necessarily reflect the opinions
of Netcopy Ltd.
While emails and attachments are virus checked, we cannot accept any
liability in respect of any viruses.
We may monitor emails sent to Netcopy.
Want to know more about Netcopy?
Visit our website: www.netcopy.co.uk
Re: Why are reads from CouchDB so slow? (1.5MB/s or thereabouts)
Posted by Miles Fidelman <mf...@meetinghouse.net>.
Jonathan Williamson wrote:
> Also the result of:
>
> time curl http://localhost:5984/> /dev/null
> real 0m0.011s
> user 0m0.000s
> sys 0m0.000s
>
> Suggests the overhead to be around 0.01 seconds.
>
> Whereas to fetch a single document (700k in size):
>
> time curl http://localhost:5984/[dbname]/[documentid]> /dev/null
> real 0m0.296s
> user 0m0.010s
> sys 0m0.010s
>
Since you're measuring the end-to-end time of a client-server
transaction, there are two additional components to the real-time number:
- time consumed while other things are happening
- time that is actually consumed by <> processing your request
Pretty clearly, both those numbers are tiny for the baseline
transaction, but for the CouchDB transaction, how much of what's taking
up elapsed time is actual processing by CouchDB, and how much of the
time is CouchDB waiting for something else to happen?
Is there a way to profile what CouchDB does once it receives that
request? For that matter, what kind of profiling has been done of
CouchDB's processing chain?
Miles Fidelman
--
In theory, there is no difference between theory and practice.
In practice, there is. .... Yogi Berra
Re: Why are reads from CouchDB so slow? (1.5MB/s or thereabouts)
Posted by Jonathan Williamson <jo...@netcopy.co.uk>.
Also the result of:
time curl http://localhost:5984/ > /dev/null
real 0m0.011s
user 0m0.000s
sys 0m0.000s
Suggests the overhead to be around 0.01 seconds.
Whereas to fetch a single document (700k in size):
time curl http://localhost:5984/[dbname]/[documentid] > /dev/null
real 0m0.296s
user 0m0.010s
sys 0m0.010s
(Or roughly 2.4MB/s)
On Fri, Mar 23, 2012 at 12:39 PM, Robert Newson <rn...@apache.org> wrote:
> curl is not a benchmarking tool. Each curl command is opening a new
> tcp connection, etc. Can you try with httperf, apachebench, tsung,
> nodeload?
>
> B.
>
> On 23 March 2012 12:36, Jonathan Williamson <jo...@netcopy.co.uk> wrote:
>> It was just a script that looped over the keys for 1000 documents,
>> fetching them one after another (using curl).
>>
>> On Fri, Mar 23, 2012 at 12:31 PM, Robert Newson <rn...@apache.org> wrote:
>>> Using which http benchmarking tool (and exactly which settings)?
>>>
>>> On 23 March 2012 12:30, Jonathan Williamson <jo...@netcopy.co.uk> wrote:
>>>> My initial CouchDB test was over 1000 GETs (all to roughly similar
>>>> documents), taking an average of the total time vs the total amount of
>>>> data recieved.
>>>>
>>>> On Fri, Mar 23, 2012 at 12:26 PM, Robert Newson <rn...@apache.org> wrote:
>>>>> Are you really extrapolating mb/s from a single curl command?
--
Jon
Jonathan Williamson
Systems Architect
:::::::::::::::::::::::::::::::::::::::::::
+44 (0) 7843 301 246 mobile
+44 (0) 208 242 6364 switch
www.netcopy.co.uk
::::::::::::::::::::::::::::::::::::::::::
IMPORTANT NOTICE:
========================================================================================
Netcopy Limited is a Company registered in England and Wales under
Company No. 07308350
Registered office: 2 East Common, Harpenden, Herts, AL5 1BJ
The contents of this email (and any attachment):
(1) are confidential and may be legally privileged - if it is not
meant for you, please tell the sender, do not forward or copy the
contents and delete it from your system immediately;
(2) come from its author and may not necessarily reflect the opinions
of Netcopy Ltd.
While emails and attachments are virus checked, we cannot accept any
liability in respect of any viruses.
We may monitor emails sent to Netcopy.
Want to know more about Netcopy?
Visit our website: www.netcopy.co.uk
Re: Why are reads from CouchDB so slow? (1.5MB/s or thereabouts)
Posted by Robert Newson <rn...@apache.org>.
curl is not a benchmarking tool. Each curl command is opening a new
tcp connection, etc. Can you try with httperf, apachebench, tsung,
nodeload?
B.
On 23 March 2012 12:36, Jonathan Williamson <jo...@netcopy.co.uk> wrote:
> It was just a script that looped over the keys for 1000 documents,
> fetching them one after another (using curl).
>
> On Fri, Mar 23, 2012 at 12:31 PM, Robert Newson <rn...@apache.org> wrote:
>> Using which http benchmarking tool (and exactly which settings)?
>>
>> On 23 March 2012 12:30, Jonathan Williamson <jo...@netcopy.co.uk> wrote:
>>> My initial CouchDB test was over 1000 GETs (all to roughly similar
>>> documents), taking an average of the total time vs the total amount of
>>> data recieved.
>>>
>>> On Fri, Mar 23, 2012 at 12:26 PM, Robert Newson <rn...@apache.org> wrote:
>>>> Are you really extrapolating mb/s from a single curl command?
Re: Why are reads from CouchDB so slow? (1.5MB/s or thereabouts)
Posted by Jason Smith <jh...@iriscouch.com>.
I would be interested to see the time of the same benchmark in
Javascript for the browser (where it can use its connection pool).
You have a fork/exec + TCP connection overhead for each iteration.
That may or may not be relevant to your particular use case.
On Fri, Mar 23, 2012 at 12:36 PM, Jonathan Williamson <jo...@netcopy.co.uk> wrote:
> It was just a script that looped over the keys for 1000 documents,
> fetching them one after another (using curl).
>
> On Fri, Mar 23, 2012 at 12:31 PM, Robert Newson <rn...@apache.org> wrote:
>> Using which http benchmarking tool (and exactly which settings)?
>>
>> On 23 March 2012 12:30, Jonathan Williamson <jo...@netcopy.co.uk> wrote:
>>> My initial CouchDB test was over 1000 GETs (all to roughly similar
>>> documents), taking an average of the total time vs the total amount of
>>> data recieved.
>>>
>>> On Fri, Mar 23, 2012 at 12:26 PM, Robert Newson <rn...@apache.org> wrote:
>>>> Are you really extrapolating mb/s from a single curl command?
--
Iris Couch
Re: Why are reads from CouchDB so slow? (1.5MB/s or thereabouts)
Posted by Jonathan Williamson <jo...@netcopy.co.uk>.
It was just a script that looped over the keys for 1000 documents,
fetching them one after another (using curl).
On Fri, Mar 23, 2012 at 12:31 PM, Robert Newson <rn...@apache.org> wrote:
> Using which http benchmarking tool (and exactly which settings)?
>
> On 23 March 2012 12:30, Jonathan Williamson <jo...@netcopy.co.uk> wrote:
>> My initial CouchDB test was over 1000 GETs (all to roughly similar
>> documents), taking an average of the total time vs the total amount of
>> data recieved.
>>
>> On Fri, Mar 23, 2012 at 12:26 PM, Robert Newson <rn...@apache.org> wrote:
>>> Are you really extrapolating mb/s from a single curl command?
Re: Why are reads from CouchDB so slow? (1.5MB/s or thereabouts)
Posted by Robert Newson <rn...@apache.org>.
Using which http benchmarking tool (and exactly which settings)?
On 23 March 2012 12:30, Jonathan Williamson <jo...@netcopy.co.uk> wrote:
> My initial CouchDB test was over 1000 GETs (all to roughly similar
> documents), taking an average of the total time vs the total amount of
> data recieved.
>
> On Fri, Mar 23, 2012 at 12:26 PM, Robert Newson <rn...@apache.org> wrote:
>> Are you really extrapolating mb/s from a single curl command?
Re: Why are reads from CouchDB so slow? (1.5MB/s or thereabouts)
Posted by Jonathan Williamson <jo...@netcopy.co.uk>.
My initial CouchDB test was over 1000 GETs (all to roughly similar
documents), taking an average of the total time vs the total amount of
data recieved.
On Fri, Mar 23, 2012 at 12:26 PM, Robert Newson <rn...@apache.org> wrote:
> Are you really extrapolating mb/s from a single curl command?
Re: Why are reads from CouchDB so slow? (1.5MB/s or thereabouts)
Posted by Robert Newson <rn...@apache.org>.
Are you really extrapolating mb/s from a single curl command?
On 23 March 2012 12:15, Jonathan Williamson <jo...@netcopy.co.uk> wrote:
> Jason,
>
> Apologies if I came off too demanding - I don't mean to be! I just
> want to understand why so that we can make a sensible decision as to
> how to move forward with or without CouchDB in the long term.
>
> My initial comparison was actually to reading a file off disk, but I
> thought that unfair so added the overhead of a web server. That's not
> to say I'd expect CouchDB to 1:1 match the performance of Nginx, but
> currently it is 1 to 2 orders of magnitude slower for the task I
> described.
>
> It's interesting to hear that CouchDB compares the document to a
> checksum prior to serving it, do you have any idea what overhead this
> adds? What's the reasoning behind it? (I mean data could be corrupted
> in transmission, or in memory after checksumming, etc).
>
> The main reasons I would expect CouchDB to be fast at this specific
> operation are:
>
> - Prebuilt indexes: I was surprised this didn't allow to CouchDB to
> very quickly identify where to retrieve the document from within its
> datafiles.
> - Internal storage format: Seems to be almost raw JSON in the files
> with a bit of metadata, should allow for (almost) direct streaming to
> client?
>
> It's not that CouchDB is slower than Nginx per se, it's that that's
> massively slower. For example having flushed file caches on my dev box
> Nginx can serve a large static file at 83MB/s (which is 25 times
> faster than CouchDB on the same hardware).
>
> On Fri, Mar 23, 2012 at 11:56 AM, Jason Smith <jh...@iriscouch.com> wrote:
>> CouchDB verifies that the document contents match a checksum which
>> does impose computation and codec overhead, yes.
>>
>> Considering that CouchDB stores multiple sorted indices to the
>> documents in a database which is itself a filesystem file, in a safe
>> append-only format, how would you justify an expectation of static
>> Nginx performance? Surely CouchDB must open the file (right there you
>> have tied Nginx at best) and then seek through its metadata to fetch
>> the doc. Note, my disagreement with you is not fundamental, just of
>> degree. Surely it is fair to give CouchDB some elbow room to work, to
>> pay for its benefits?
>>
>> Back to document comprehension, CouchDB does do that and this is a
>> huge opportunity for improvement. I believe Filipe has indeed proposed
>> something much like you describe: store the utf-8 JSON directly on the
>> disk.
>>
>> I'm excited that this conversation can paint a more clear picture of
>> what we expect from CouchDB, to find a speed at which we could say,
>> "this is slower than Brand X, but it's worth it."
>>
>> On Fri, Mar 23, 2012 at 11:41 AM, Jonathan Williamson <jo...@netcopy.co.uk> wrote:
>>> As I'm requesting the documents in the exact format I submitted them
>>> (with no transformations or extra information) I'd expect something
>>> not far off a static file request from Nginx. As far as I can tell the
>>> .couch files aren't compressed (though that wouldn't cause such slow
>>> performance on an i5 anyway) and appear to contain the original
>>> documents almost "as is".
>>>
>>> The other side effect is that while fetching the documents the CPU
>>> usages rises to 100% which suggests, I guess, that CouchDB is reading,
>>> deserialising, serialising, and then streaming the document. But it
>>> doesn't seem like that should be necessary really?
Re: Why are reads from CouchDB so slow? (1.5MB/s or thereabouts)
Posted by Jonathan Williamson <jo...@netcopy.co.uk>.
Jason,
Apologies if I came off too demanding - I don't mean to be! I just
want to understand why so that we can make a sensible decision as to
how to move forward with or without CouchDB in the long term.
My initial comparison was actually to reading a file off disk, but I
thought that unfair so added the overhead of a web server. That's not
to say I'd expect CouchDB to 1:1 match the performance of Nginx, but
currently it is 1 to 2 orders of magnitude slower for the task I
described.
It's interesting to hear that CouchDB compares the document to a
checksum prior to serving it, do you have any idea what overhead this
adds? What's the reasoning behind it? (I mean data could be corrupted
in transmission, or in memory after checksumming, etc).
The main reasons I would expect CouchDB to be fast at this specific
operation are:
- Prebuilt indexes: I was surprised this didn't allow to CouchDB to
very quickly identify where to retrieve the document from within its
datafiles.
- Internal storage format: Seems to be almost raw JSON in the files
with a bit of metadata, should allow for (almost) direct streaming to
client?
It's not that CouchDB is slower than Nginx per se, it's that that's
massively slower. For example having flushed file caches on my dev box
Nginx can serve a large static file at 83MB/s (which is 25 times
faster than CouchDB on the same hardware).
On Fri, Mar 23, 2012 at 11:56 AM, Jason Smith <jh...@iriscouch.com> wrote:
> CouchDB verifies that the document contents match a checksum which
> does impose computation and codec overhead, yes.
>
> Considering that CouchDB stores multiple sorted indices to the
> documents in a database which is itself a filesystem file, in a safe
> append-only format, how would you justify an expectation of static
> Nginx performance? Surely CouchDB must open the file (right there you
> have tied Nginx at best) and then seek through its metadata to fetch
> the doc. Note, my disagreement with you is not fundamental, just of
> degree. Surely it is fair to give CouchDB some elbow room to work, to
> pay for its benefits?
>
> Back to document comprehension, CouchDB does do that and this is a
> huge opportunity for improvement. I believe Filipe has indeed proposed
> something much like you describe: store the utf-8 JSON directly on the
> disk.
>
> I'm excited that this conversation can paint a more clear picture of
> what we expect from CouchDB, to find a speed at which we could say,
> "this is slower than Brand X, but it's worth it."
>
> On Fri, Mar 23, 2012 at 11:41 AM, Jonathan Williamson <jo...@netcopy.co.uk> wrote:
>> As I'm requesting the documents in the exact format I submitted them
>> (with no transformations or extra information) I'd expect something
>> not far off a static file request from Nginx. As far as I can tell the
>> .couch files aren't compressed (though that wouldn't cause such slow
>> performance on an i5 anyway) and appear to contain the original
>> documents almost "as is".
>>
>> The other side effect is that while fetching the documents the CPU
>> usages rises to 100% which suggests, I guess, that CouchDB is reading,
>> deserialising, serialising, and then streaming the document. But it
>> doesn't seem like that should be necessary really?
Re: Why are reads from CouchDB so slow? (1.5MB/s or thereabouts)
Posted by Jason Smith <jh...@iriscouch.com>.
CouchDB verifies that the document contents match a checksum which
does impose computation and codec overhead, yes.
Considering that CouchDB stores multiple sorted indices to the
documents in a database which is itself a filesystem file, in a safe
append-only format, how would you justify an expectation of static
Nginx performance? Surely CouchDB must open the file (right there you
have tied Nginx at best) and then seek through its metadata to fetch
the doc. Note, my disagreement with you is not fundamental, just of
degree. Surely it is fair to give CouchDB some elbow room to work, to
pay for its benefits?
Back to document comprehension, CouchDB does do that and this is a
huge opportunity for improvement. I believe Filipe has indeed proposed
something much like you describe: store the utf-8 JSON directly on the
disk.
I'm excited that this conversation can paint a more clear picture of
what we expect from CouchDB, to find a speed at which we could say,
"this is slower than Brand X, but it's worth it."
On Fri, Mar 23, 2012 at 11:41 AM, Jonathan Williamson <jo...@netcopy.co.uk> wrote:
> As I'm requesting the documents in the exact format I submitted them
> (with no transformations or extra information) I'd expect something
> not far off a static file request from Nginx. As far as I can tell the
> .couch files aren't compressed (though that wouldn't cause such slow
> performance on an i5 anyway) and appear to contain the original
> documents almost "as is".
>
> The other side effect is that while fetching the documents the CPU
> usages rises to 100% which suggests, I guess, that CouchDB is reading,
> deserialising, serialising, and then streaming the document. But it
> doesn't seem like that should be necessary really?
>
> On Fri, Mar 23, 2012 at 11:16 AM, Jason Smith <jh...@iriscouch.com> wrote:
>> Hi, Jonathan.
>>
>> Can we start by identifying how fast you think **is** reasonable. What
>> speeds were you expecting? And on what grounds do you base that
>> expectation?
>>
>> (For example, perhaps connecting and fetching one row from MySQL?)
--
Iris Couch
Re: Why are reads from CouchDB so slow? (1.5MB/s or thereabouts)
Posted by Randall Leeds <ra...@gmail.com>.
On Fri, Mar 23, 2012 at 04:41, Jonathan Williamson <jo...@netcopy.co.uk> wrote:
> As I'm requesting the documents in the exact format I submitted them
> (with no transformations or extra information) I'd expect something
> not far off a static file request from Nginx. As far as I can tell the
> .couch files aren't compressed (though that wouldn't cause such slow
> performance on an i5 anyway) and appear to contain the original
> documents almost "as is".
>
> The other side effect is that while fetching the documents the CPU
> usages rises to 100% which suggests, I guess, that CouchDB is reading,
> deserialising, serialising, and then streaming the document. But it
> doesn't seem like that should be necessary really?
Couch ensures that your doc is valid JSON and indeed reserializes the
JSON for every read. The reason is that we've opted out of the
byte-level twiddling that would be required to insert metadata inside
the document body when certain query parameters are passed that would
allow us to keep the bulk of the doc pre-serialized.
The 100% cpu overhead is likely the JSON serialization. The 1.2.x
branch uses a brand new serializer, written in C. It would be
interesting if you could try it.
Re: Why are reads from CouchDB so slow? (1.5MB/s or thereabouts)
Posted by Jonathan Williamson <jo...@netcopy.co.uk>.
As I'm requesting the documents in the exact format I submitted them
(with no transformations or extra information) I'd expect something
not far off a static file request from Nginx. As far as I can tell the
.couch files aren't compressed (though that wouldn't cause such slow
performance on an i5 anyway) and appear to contain the original
documents almost "as is".
The other side effect is that while fetching the documents the CPU
usages rises to 100% which suggests, I guess, that CouchDB is reading,
deserialising, serialising, and then streaming the document. But it
doesn't seem like that should be necessary really?
On Fri, Mar 23, 2012 at 11:16 AM, Jason Smith <jh...@iriscouch.com> wrote:
> Hi, Jonathan.
>
> Can we start by identifying how fast you think **is** reasonable. What
> speeds were you expecting? And on what grounds do you base that
> expectation?
>
> (For example, perhaps connecting and fetching one row from MySQL?)
Re: Why are reads from CouchDB so slow? (1.5MB/s or thereabouts)
Posted by Jonathan Williamson <jo...@netcopy.co.uk>.
# ping 127.0.0.1
...100 pings...
100 packets transmitted, 100 received, 0% packet loss, time 99004ms
rtt min/avg/max/mdev = 0.021/0.028/0.144/0.013 ms
On Fri, Mar 23, 2012 at 11:21 AM, CGS <cg...@gmail.com> wrote:
> ...also, it would be nice to know how you inserted the documents and if
> they have attachments (at that size, I suppose they have). Could you also
> provide a result from ping 127.0.0.1 for response time fluctuations?
>
> CGS
>
>
>
>
> On Fri, Mar 23, 2012 at 12:16 PM, Jason Smith <jh...@iriscouch.com> wrote:
>
>> Hi, Jonathan.
>>
>> Can we start by identifying how fast you think **is** reasonable. What
>> speeds were you expecting? And on what grounds do you base that
>> expectation?
>>
>> (For example, perhaps connecting and fetching one row from MySQL?)
>>
>> On Fri, Mar 23, 2012 at 11:06 AM, Jonathan Williamson <jo...@netcopy.co.uk>
>> wrote:
>> > Hi everyone,
>> >
>> > I posted this question on Stack Overflow and Jason Smith suggested I
>> > try this group instead.
>> >
>> >> I have a CouchDB (1.1.1) server running that contains a lot of
>> documents in the 400-600KB size range.
>> >>
>> >> If I time fetching a full document from the database (not from a view,
>> just the raw document) it takes 200-400ms to complete which equates to
>> around 1.5MB/s throughput.
>> >>
>> >> If I write the same data to raw files on disk they load in 10-20ms
>> (around 25-50 MB/s).
>> >>
>> >> I'd expect CouchDB to have some overhead, but an order of magnitude
>> (and some) seems crazy for what is essentially a read!
>> >>
>> >> Can anyone shed some light onto why this might be the case?
>> >>
>> >> Update: As requested below, a timing from curl:
>> >>
>> >> # time curl http://localhost:5984/[dbname]/[documentname]
>> >>
>> >> real 0m0.684s
>> >> user 0m0.004s
>> >> sys 0m0.020s
>> >>
>> >> The fetched document was 642842 bytes. I've tested it on both a
>> standard 1TB harddisk and an EC2 instance (EBS volume) with similar results.
>> >
>> > There was some discussion about the post which can be found here:
>> >
>> http://stackoverflow.com/questions/9807200/why-are-reads-from-couchdb-so-slow-1-5mb-s-or-thereabouts
>> >
>> > Thanks,
>> >
>> > Jon.
>>
>>
>>
>> --
>> Iris Couch
>>
--
Jon
Jonathan Williamson
Systems Architect
:::::::::::::::::::::::::::::::::::::::::::
+44 (0) 7843 301 246 mobile
+44 (0) 208 242 6364 switch
www.netcopy.co.uk
::::::::::::::::::::::::::::::::::::::::::
IMPORTANT NOTICE:
========================================================================================
Netcopy Limited is a Company registered in England and Wales under
Company No. 07308350
Registered office: 2 East Common, Harpenden, Herts, AL5 1BJ
The contents of this email (and any attachment):
(1) are confidential and may be legally privileged - if it is not
meant for you, please tell the sender, do not forward or copy the
contents and delete it from your system immediately;
(2) come from its author and may not necessarily reflect the opinions
of Netcopy Ltd.
While emails and attachments are virus checked, we cannot accept any
liability in respect of any viruses.
We may monitor emails sent to Netcopy.
Want to know more about Netcopy?
Visit our website: www.netcopy.co.uk
Re: Why are reads from CouchDB so slow? (1.5MB/s or thereabouts)
Posted by Jonathan Williamson <jo...@netcopy.co.uk>.
The documents were inserted via the HTTP API.
There are no attachments, just (largish and quite complex) documents.
On Fri, Mar 23, 2012 at 11:21 AM, CGS <cg...@gmail.com> wrote:
> ...also, it would be nice to know how you inserted the documents and if
> they have attachments (at that size, I suppose they have). Could you also
> provide a result from ping 127.0.0.1 for response time fluctuations?
>
> CGS
>
>
>
>
> On Fri, Mar 23, 2012 at 12:16 PM, Jason Smith <jh...@iriscouch.com> wrote:
>
>> Hi, Jonathan.
>>
>> Can we start by identifying how fast you think **is** reasonable. What
>> speeds were you expecting? And on what grounds do you base that
>> expectation?
>>
>> (For example, perhaps connecting and fetching one row from MySQL?)
>>
>> On Fri, Mar 23, 2012 at 11:06 AM, Jonathan Williamson <jo...@netcopy.co.uk>
>> wrote:
>> > Hi everyone,
>> >
>> > I posted this question on Stack Overflow and Jason Smith suggested I
>> > try this group instead.
>> >
>> >> I have a CouchDB (1.1.1) server running that contains a lot of
>> documents in the 400-600KB size range.
>> >>
>> >> If I time fetching a full document from the database (not from a view,
>> just the raw document) it takes 200-400ms to complete which equates to
>> around 1.5MB/s throughput.
>> >>
>> >> If I write the same data to raw files on disk they load in 10-20ms
>> (around 25-50 MB/s).
>> >>
>> >> I'd expect CouchDB to have some overhead, but an order of magnitude
>> (and some) seems crazy for what is essentially a read!
>> >>
>> >> Can anyone shed some light onto why this might be the case?
>> >>
>> >> Update: As requested below, a timing from curl:
>> >>
>> >> # time curl http://localhost:5984/[dbname]/[documentname]
>> >>
>> >> real 0m0.684s
>> >> user 0m0.004s
>> >> sys 0m0.020s
>> >>
>> >> The fetched document was 642842 bytes. I've tested it on both a
>> standard 1TB harddisk and an EC2 instance (EBS volume) with similar results.
>> >
>> > There was some discussion about the post which can be found here:
>> >
>> http://stackoverflow.com/questions/9807200/why-are-reads-from-couchdb-so-slow-1-5mb-s-or-thereabouts
>> >
>> > Thanks,
>> >
>> > Jon.
>>
>>
>>
>> --
>> Iris Couch
>>
Re: Why are reads from CouchDB so slow? (1.5MB/s or thereabouts)
Posted by Jonathan Williamson <jo...@netcopy.co.uk>.
# time curl http://localhost:5984/
{"couchdb":"Welcome","version":"1.1.1"}
real 0m0.012s
user 0m0.000s
sys 0m0.000s
On Fri, Mar 23, 2012 at 11:31 AM, Jason Smith <jh...@iriscouch.com> wrote:
> Useful as well would be the response time for:
>
> curl http://localhost:5984/
>
> To get a baseline for CouchDB's raw HTTP response time, with no i/o.
>
> On Fri, Mar 23, 2012 at 11:21 AM, CGS <cg...@gmail.com> wrote:
>> ...also, it would be nice to know how you inserted the documents and if
>> they have attachments (at that size, I suppose they have). Could you also
>> provide a result from ping 127.0.0.1 for response time fluctuations?
>>
>> CGS
>>
>>
>>
>>
>> On Fri, Mar 23, 2012 at 12:16 PM, Jason Smith <jh...@iriscouch.com> wrote:
>>
>>> Hi, Jonathan.
>>>
>>> Can we start by identifying how fast you think **is** reasonable. What
>>> speeds were you expecting? And on what grounds do you base that
>>> expectation?
>>>
>>> (For example, perhaps connecting and fetching one row from MySQL?)
>>>
>>> On Fri, Mar 23, 2012 at 11:06 AM, Jonathan Williamson <jo...@netcopy.co.uk>
>>> wrote:
>>> > Hi everyone,
>>> >
>>> > I posted this question on Stack Overflow and Jason Smith suggested I
>>> > try this group instead.
>>> >
>>> >> I have a CouchDB (1.1.1) server running that contains a lot of
>>> documents in the 400-600KB size range.
>>> >>
>>> >> If I time fetching a full document from the database (not from a view,
>>> just the raw document) it takes 200-400ms to complete which equates to
>>> around 1.5MB/s throughput.
>>> >>
>>> >> If I write the same data to raw files on disk they load in 10-20ms
>>> (around 25-50 MB/s).
>>> >>
>>> >> I'd expect CouchDB to have some overhead, but an order of magnitude
>>> (and some) seems crazy for what is essentially a read!
>>> >>
>>> >> Can anyone shed some light onto why this might be the case?
>>> >>
>>> >> Update: As requested below, a timing from curl:
>>> >>
>>> >> # time curl http://localhost:5984/[dbname]/[documentname]
>>> >>
>>> >> real 0m0.684s
>>> >> user 0m0.004s
>>> >> sys 0m0.020s
>>> >>
>>> >> The fetched document was 642842 bytes. I've tested it on both a
>>> standard 1TB harddisk and an EC2 instance (EBS volume) with similar results.
>>> >
>>> > There was some discussion about the post which can be found here:
>>> >
>>> http://stackoverflow.com/questions/9807200/why-are-reads-from-couchdb-so-slow-1-5mb-s-or-thereabouts
>>> >
>>> > Thanks,
>>> >
>>> > Jon.
>>>
>>>
>>>
>>> --
>>> Iris Couch
>>>
>
>
>
> --
> Iris Couch
--
Jon
Jonathan Williamson
Systems Architect
:::::::::::::::::::::::::::::::::::::::::::
+44 (0) 7843 301 246 mobile
+44 (0) 208 242 6364 switch
www.netcopy.co.uk
::::::::::::::::::::::::::::::::::::::::::
IMPORTANT NOTICE:
========================================================================================
Netcopy Limited is a Company registered in England and Wales under
Company No. 07308350
Registered office: 2 East Common, Harpenden, Herts, AL5 1BJ
The contents of this email (and any attachment):
(1) are confidential and may be legally privileged - if it is not
meant for you, please tell the sender, do not forward or copy the
contents and delete it from your system immediately;
(2) come from its author and may not necessarily reflect the opinions
of Netcopy Ltd.
While emails and attachments are virus checked, we cannot accept any
liability in respect of any viruses.
We may monitor emails sent to Netcopy.
Want to know more about Netcopy?
Visit our website: www.netcopy.co.uk
Re: Why are reads from CouchDB so slow? (1.5MB/s or thereabouts)
Posted by Jason Smith <jh...@iriscouch.com>.
Useful as well would be the response time for:
curl http://localhost:5984/
To get a baseline for CouchDB's raw HTTP response time, with no i/o.
On Fri, Mar 23, 2012 at 11:21 AM, CGS <cg...@gmail.com> wrote:
> ...also, it would be nice to know how you inserted the documents and if
> they have attachments (at that size, I suppose they have). Could you also
> provide a result from ping 127.0.0.1 for response time fluctuations?
>
> CGS
>
>
>
>
> On Fri, Mar 23, 2012 at 12:16 PM, Jason Smith <jh...@iriscouch.com> wrote:
>
>> Hi, Jonathan.
>>
>> Can we start by identifying how fast you think **is** reasonable. What
>> speeds were you expecting? And on what grounds do you base that
>> expectation?
>>
>> (For example, perhaps connecting and fetching one row from MySQL?)
>>
>> On Fri, Mar 23, 2012 at 11:06 AM, Jonathan Williamson <jo...@netcopy.co.uk>
>> wrote:
>> > Hi everyone,
>> >
>> > I posted this question on Stack Overflow and Jason Smith suggested I
>> > try this group instead.
>> >
>> >> I have a CouchDB (1.1.1) server running that contains a lot of
>> documents in the 400-600KB size range.
>> >>
>> >> If I time fetching a full document from the database (not from a view,
>> just the raw document) it takes 200-400ms to complete which equates to
>> around 1.5MB/s throughput.
>> >>
>> >> If I write the same data to raw files on disk they load in 10-20ms
>> (around 25-50 MB/s).
>> >>
>> >> I'd expect CouchDB to have some overhead, but an order of magnitude
>> (and some) seems crazy for what is essentially a read!
>> >>
>> >> Can anyone shed some light onto why this might be the case?
>> >>
>> >> Update: As requested below, a timing from curl:
>> >>
>> >> # time curl http://localhost:5984/[dbname]/[documentname]
>> >>
>> >> real 0m0.684s
>> >> user 0m0.004s
>> >> sys 0m0.020s
>> >>
>> >> The fetched document was 642842 bytes. I've tested it on both a
>> standard 1TB harddisk and an EC2 instance (EBS volume) with similar results.
>> >
>> > There was some discussion about the post which can be found here:
>> >
>> http://stackoverflow.com/questions/9807200/why-are-reads-from-couchdb-so-slow-1-5mb-s-or-thereabouts
>> >
>> > Thanks,
>> >
>> > Jon.
>>
>>
>>
>> --
>> Iris Couch
>>
--
Iris Couch
Re: Why are reads from CouchDB so slow? (1.5MB/s or thereabouts)
Posted by CGS <cg...@gmail.com>.
...also, it would be nice to know how you inserted the documents and if
they have attachments (at that size, I suppose they have). Could you also
provide a result from ping 127.0.0.1 for response time fluctuations?
CGS
On Fri, Mar 23, 2012 at 12:16 PM, Jason Smith <jh...@iriscouch.com> wrote:
> Hi, Jonathan.
>
> Can we start by identifying how fast you think **is** reasonable. What
> speeds were you expecting? And on what grounds do you base that
> expectation?
>
> (For example, perhaps connecting and fetching one row from MySQL?)
>
> On Fri, Mar 23, 2012 at 11:06 AM, Jonathan Williamson <jo...@netcopy.co.uk>
> wrote:
> > Hi everyone,
> >
> > I posted this question on Stack Overflow and Jason Smith suggested I
> > try this group instead.
> >
> >> I have a CouchDB (1.1.1) server running that contains a lot of
> documents in the 400-600KB size range.
> >>
> >> If I time fetching a full document from the database (not from a view,
> just the raw document) it takes 200-400ms to complete which equates to
> around 1.5MB/s throughput.
> >>
> >> If I write the same data to raw files on disk they load in 10-20ms
> (around 25-50 MB/s).
> >>
> >> I'd expect CouchDB to have some overhead, but an order of magnitude
> (and some) seems crazy for what is essentially a read!
> >>
> >> Can anyone shed some light onto why this might be the case?
> >>
> >> Update: As requested below, a timing from curl:
> >>
> >> # time curl http://localhost:5984/[dbname]/[documentname]
> >>
> >> real 0m0.684s
> >> user 0m0.004s
> >> sys 0m0.020s
> >>
> >> The fetched document was 642842 bytes. I've tested it on both a
> standard 1TB harddisk and an EC2 instance (EBS volume) with similar results.
> >
> > There was some discussion about the post which can be found here:
> >
> http://stackoverflow.com/questions/9807200/why-are-reads-from-couchdb-so-slow-1-5mb-s-or-thereabouts
> >
> > Thanks,
> >
> > Jon.
>
>
>
> --
> Iris Couch
>
Re: Why are reads from CouchDB so slow? (1.5MB/s or thereabouts)
Posted by Jason Smith <jh...@iriscouch.com>.
Hi, Jonathan.
Can we start by identifying how fast you think **is** reasonable. What
speeds were you expecting? And on what grounds do you base that
expectation?
(For example, perhaps connecting and fetching one row from MySQL?)
On Fri, Mar 23, 2012 at 11:06 AM, Jonathan Williamson <jo...@netcopy.co.uk> wrote:
> Hi everyone,
>
> I posted this question on Stack Overflow and Jason Smith suggested I
> try this group instead.
>
>> I have a CouchDB (1.1.1) server running that contains a lot of documents in the 400-600KB size range.
>>
>> If I time fetching a full document from the database (not from a view, just the raw document) it takes 200-400ms to complete which equates to around 1.5MB/s throughput.
>>
>> If I write the same data to raw files on disk they load in 10-20ms (around 25-50 MB/s).
>>
>> I'd expect CouchDB to have some overhead, but an order of magnitude (and some) seems crazy for what is essentially a read!
>>
>> Can anyone shed some light onto why this might be the case?
>>
>> Update: As requested below, a timing from curl:
>>
>> # time curl http://localhost:5984/[dbname]/[documentname]
>>
>> real 0m0.684s
>> user 0m0.004s
>> sys 0m0.020s
>>
>> The fetched document was 642842 bytes. I've tested it on both a standard 1TB harddisk and an EC2 instance (EBS volume) with similar results.
>
> There was some discussion about the post which can be found here:
> http://stackoverflow.com/questions/9807200/why-are-reads-from-couchdb-so-slow-1-5mb-s-or-thereabouts
>
> Thanks,
>
> Jon.
--
Iris Couch