perfomance?

[Vasili Batareykin <jo...@gmail.com>]

Hello!

i'm trying to use couchdb in dumb case: as a file storage with http access)
but get performance problem
simple test:
curl -X PUT http://localhost:5984/users
curl -X PUT http://localhost:5984/users/static -d {}
curl -H "Content-Type: image/jpeg" -X PUT
http://localhost:5984/users/static/1.jpg?rev=4-41378c97921c2b3bc2a76f4c47f4ee86--data-binary
@1.jpg

then trying to get that jpg with ab from apache distro:

76.46 [#/sec] ab -c 10 -n 1000
39.13 [#/sec] ab -n 1000
378912 bytes jpg

701.90 [#/sec] ab -c 10 -n 1000
338.66 [#/sec] ab -n 1000
4737 bytes jpg

platform: ubuntu-karmic x86, couchdb, kernel from repo
CouchDB/0.10.0 (Erlang OTP/R13B)
Quad Xeon 2.66, 1Gb memory

any way to get better #/sec for files >100kb? upgrade to 10.1? upgrade
erlang? or use x64 platform?
or this is not for couchdb?

sorry for bad english.

Re: perfomance?

[Robert Newson <ro...@gmail.com>]

writes are also concurrent in a sense, though only one process is
writing to the .couch file. If I send a 100 mib attachment and you
send one at the same time, then chunks from each of our attachments
are written to disk as they are being sent. We both get a 201 at about
the same time.

B.

On Wed, Mar 24, 2010 at 12:04 PM, Vasili Batareykin <jo...@gmail.com> wrote:
> i think this is right place)
>
> 2010/3/24 Robert Newson <ro...@gmail.com>
>
>> There are numerous ways that attachment retrieval performance can be
>> improved, I'm just explaining why you're seeing the numbers you
>> reported.
>>
>> Turning off chunking would also be a code change, one that I don't
>> expect will make it into the product, since it would degrade
>> concurrent writer behavior.
>>
>> hmhmh as i read docs there is no concurrent writing, only concurent read.
> i'm wrong?
>
>
>
>> My current favorite proposed enhancement is to support external
>> attachments (where the attachment is written to its own, separate
>> file). I haven't started work on a patch for this, yet, but it should
>> give all the advantages of apache2/sendfile while still allowing
>> couchdb replication, etc.
>>
>> this will be nice.
>

Re: perfomance?

[Vasili Batareykin <jo...@gmail.com>]

i think this is right place)

2010/3/24 Robert Newson <ro...@gmail.com>

> There are numerous ways that attachment retrieval performance can be
> improved, I'm just explaining why you're seeing the numbers you
> reported.
>
> Turning off chunking would also be a code change, one that I don't
> expect will make it into the product, since it would degrade
> concurrent writer behavior.
>
> hmhmh as i read docs there is no concurrent writing, only concurent read.
i'm wrong?



> My current favorite proposed enhancement is to support external
> attachments (where the attachment is written to its own, separate
> file). I haven't started work on a patch for this, yet, but it should
> give all the advantages of apache2/sendfile while still allowing
> couchdb replication, etc.
>
> this will be nice.

Re: perfomance?

[Robert Newson <ro...@gmail.com>]

There are numerous ways that attachment retrieval performance can be
improved, I'm just explaining why you're seeing the numbers you
reported.

Turning off chunking would also be a code change, one that I don't
expect will make it into the product, since it would degrade
concurrent writer behavior.

My current favorite proposed enhancement is to support external
attachments (where the attachment is written to its own, separate
file). I haven't started work on a patch for this, yet, but it should
give all the advantages of apache2/sendfile while still allowing
couchdb replication, etc.

Further discussions on improving this belong to @dev, I think.

B.

On Wed, Mar 24, 2010 at 11:26 AM, Vasili Batareykin <jo...@gmail.com> wrote:
> Hmmm, how about turn off chunks for attachments? how about shema-less
> concept?)
> why think for FS? or problematic call smth like read N bytes from X pos in
> db thru systemcall?
> sorry i'm not dba-developer. only user(
>
> 2010/3/24 Robert Newson <ro...@gmail.com>
>
>> It's a code change to increase the chunk size, it's not currently a
>> configuration setting. When I was testing this I increased it to 64k
>> and 128k, it didn't make much difference (it's quite possible I didn't
>> do it correctly, though I did verify that I had larger chunks of
>> attachment data in the file).
>>
>>
>

Re: perfomance?

[Vasili Batareykin <jo...@gmail.com>]

Hmmm, how about turn off chunks for attachments? how about shema-less
concept?)
why think for FS? or problematic call smth like read N bytes from X pos in
db thru systemcall?
sorry i'm not dba-developer. only user(

2010/3/24 Robert Newson <ro...@gmail.com>

> It's a code change to increase the chunk size, it's not currently a
> configuration setting. When I was testing this I increased it to 64k
> and 128k, it didn't make much difference (it's quite possible I didn't
> do it correctly, though I did verify that I had larger chunks of
> attachment data in the file).
>
>

Re: perfomance?

[Robert Newson <ro...@gmail.com>]

It's a code change to increase the chunk size, it's not currently a
configuration setting. When I was testing this I increased it to 64k
and 128k, it didn't make much difference (it's quite possible I didn't
do it correctly, though I did verify that I had larger chunks of
attachment data in the file).

B.

On Wed, Mar 24, 2010 at 11:13 AM, Vasili Batareykin <jo...@gmail.com> wrote:
>> I have measured the difference between serving static files from
>> apache2 vs. attachments from couchdb. It's always faster to do so via
>> apache2, and, on average, couchdb was 2-4 times slower at serving the
>> same data as apache2.
>>
>
> in my case nginx vs couchdb 10x slowdown. on static files.
>
>
>>
>> This doesn't surprise me. Attachments are interleaved in chunks so
>> that concurrent writers do not block each others progress (since only
>> one process can append to a file at a time). So reading a file from
>> couchdb involves seeking to those chunks (and they're small, 4k or
>> less) and then sending them. As Randall points out, apache2 can just
>> call sendfile().
>>
>
> couchdb eats CPU not io. on small (two attachments, 800kb file size on disk)
> db.
> i don't know about real work(#/sec) with 50k+ files and 20Gb size of pics
> (80Gb in db?)
>
> can i change size of this chunk? to 32,64kb or 128? yes i know about
> overhead on disk.
>

Re: perfomance?

[Vasili Batareykin <jo...@gmail.com>]

> I have measured the difference between serving static files from
> apache2 vs. attachments from couchdb. It's always faster to do so via
> apache2, and, on average, couchdb was 2-4 times slower at serving the
> same data as apache2.
>

in my case nginx vs couchdb 10x slowdown. on static files.


>
> This doesn't surprise me. Attachments are interleaved in chunks so
> that concurrent writers do not block each others progress (since only
> one process can append to a file at a time). So reading a file from
> couchdb involves seeking to those chunks (and they're small, 4k or
> less) and then sending them. As Randall points out, apache2 can just
> call sendfile().
>

couchdb eats CPU not io. on small (two attachments, 800kb file size on disk)
db.
i don't know about real work(#/sec) with 50k+ files and 20Gb size of pics
(80Gb in db?)

can i change size of this chunk? to 32,64kb or 128? yes i know about
overhead on disk.

Re: perfomance?

[Robert Newson <ro...@gmail.com>]

I checked the source. CouchDB will honor a Connection: Keep-Alive from
an HTTP 1.0 client (couch_httpd.erl's http_1_0_keep_alive/2).

I have measured the difference between serving static files from
apache2 vs. attachments from couchdb. It's always faster to do so via
apache2, and, on average, couchdb was 2-4 times slower at serving the
same data as apache2.

This doesn't surprise me. Attachments are interleaved in chunks so
that concurrent writers do not block each others progress (since only
one process can append to a file at a time). So reading a file from
couchdb involves seeking to those chunks (and they're small, 4k or
less) and then sending them. As Randall points out, apache2 can just
call sendfile().

B.

On Wed, Mar 24, 2010 at 10:17 AM, Vasili Batareykin <jo...@gmail.com> wrote:
> browser/server/proxy with this feature(pipeline)?
> about http-rfc violation. apache process this request without any problems.
> why couchdb not?
>
> 2010/3/24 Robert Newson <ro...@gmail.com>
>
>> iirc apachebench only speaks http/1.0 but uses a common violation to
>> support keep-alive. This likely confuses CouchDB which speaks
>> http/1.1.
>>
>> keep-alive is also not the same as pipelining. keep-alive just reuses
>> connections, whereas HTTP pipelining sends multiple requests without
>> reading the responses. The responses are read as a batch later; this
>> allows you to largely circumvent the latency of a network roundtrip.
>>
>>
>

Re: perfomance?

[Vasili Batareykin <jo...@gmail.com>]

browser/server/proxy with this feature(pipeline)?
about http-rfc violation. apache process this request without any problems.
why couchdb not?

2010/3/24 Robert Newson <ro...@gmail.com>

> iirc apachebench only speaks http/1.0 but uses a common violation to
> support keep-alive. This likely confuses CouchDB which speaks
> http/1.1.
>
> keep-alive is also not the same as pipelining. keep-alive just reuses
> connections, whereas HTTP pipelining sends multiple requests without
> reading the responses. The responses are read as a batch later; this
> allows you to largely circumvent the latency of a network roundtrip.
>
>

Re: perfomance?

[Robert Newson <ro...@gmail.com>]

iirc apachebench only speaks http/1.0 but uses a common violation to
support keep-alive. This likely confuses CouchDB which speaks
http/1.1.

keep-alive is also not the same as pipelining. keep-alive just reuses
connections, whereas HTTP pipelining sends multiple requests without
reading the responses. The responses are read as a batch later; this
allows you to largely circumvent the latency of a network roundtrip.

B.

P.S My local apachebench certainly uses http/1.0;

Benchmarking 127.0.0.1 (be patient)...INFO: POST header ==
---
GET / HTTP/1.0
Host: 127.0.0.1:5984
User-Agent: ApacheBench/2.3
Accept: */*


---
LOG: header received:
HTTP/1.0 200 OK
Server: CouchDB/0.11.0 (Erlang OTP/R13B)
Date: Wed, 24 Mar 2010 10:11:32 GMT
Content-Type: text/plain;charset=utf-8
Content-Length: 41
Cache-Control: must-revalidate


On Wed, Mar 24, 2010 at 10:01 AM, Vasili Batareykin <jo...@gmail.com> wrote:
> about keepalive:
> working:
> req:
> GET /uri HTTP/1.0
> Connection: Keep-Alive
> Host: somehost
> User-Agent: someagent
> Accept: */*
>
> reply:
> HTTP/1.1 200 OK
> Date: Wed, 24 Mar 2010 09:44:36 GMT
> Server: megaserver
> Last-Modified: Sun, 22 Jul 2007 17:00:00 GMT
> ETag: "4d6436-8cdd-435dd17316400"
> Accept-Ranges: bytes
> Content-Length: 36061
> Keep-Alive: timeout=15, max=100
> Connection: Keep-Alive
> Content-Type: somecontent/type
>
> mandory field in req is Connection: Keep-Alive.
> mandory(imho) fields in reply:
> Content-Length: 36061 (how much read ...)
> Keep-Alive: timeout=15, max=100 (timeouts)
> Connection: Keep-Alive (and this or "close" if not support this feature)
>
> but what we see in couchdb's reply to same req:
>
> HTTP/1.0 200 OK
> Server: CouchDB/0.10.0 (Erlang OTP/R13B)
> Date: Wed, 24 Mar 2010 09:37:27 GMT
> Content-Type: text/plain;charset=utf-8
> Content-Length: 41
> Cache-Control: must-revalidate
>
> Connection timeout n/a
> Connection type n/a
>
> i think RFC better source than my dump)
>
> in my case (replicated+vers file storage) erlang eats all CPU to process
> requests.
> mb there is better way to get files from couchdb? by version or by e-tag?
> or increase some kind buffers or memory pages?
> sorry i don't know internals.
>
> 2010/3/24 Randall Leeds <ra...@gmail.com>
>
>> Yes, I do mean KeepAlive, sorry for the confusion.
>> CouchDB should support it. Can you show a dump of the headers received
>> by Couch somehow? Maybe there is something silly like an issue of case
>> with the headers.
>>
>> CouchDB cannot use sendfile (or some Erlang way to do the same)
>> because it puts bytes in the middle of the attachment on disk at
>> regular intervals. As I understand it, this is so that you can store
>> attachments that contain something like a database header without
>> breaking your database. Otherwise, storing something that looks like
>> the BTree header in an attachment could cause couch to open the
>> database incorrectly after an odd crash situation.
>>
>> This behavior maybe sounds very paranoid and strange, but one thing is
>> for sure: CouchDB is designed to be *robust*. Really, really, _really_
>> hard to lose your data.
>>
>> On Wed, Mar 24, 2010 at 01:01, Vasili Batareykin <jo...@gmail.com>
>> wrote:
>> > pipelining? you mean keepalive? ab hold test if you supply -k option (Use
>> > HTTP KeepAlive feature) seems that couchdb's httpd don't know about this)
>> > yes, throughput(in b/s) is better,but on localhost, if i use same test
>> with
>> > nginx i get around 1000 #/sec on 340k file. (344294.81 [Kbytes/sec]).
>> yes,
>> > nginx use sendfile for this operation. yes fs cache used too. but 70
>> #/sec
>> > with couchdb ...
>> >
>> > 2010/3/24 Randall Leeds <ra...@gmail.com>
>> >
>> >> If you multiple (#/sec) by file size, are actually getting _better_
>> >> throughput with the larger files.
>> >> Do you know if ab command uses HTTP 1.1 pipelining? If not, HTTP
>> >> overhead would explain the extra time.
>> >>
>> >
>>
>

Re: perfomance?

[Vasili Batareykin <jo...@gmail.com>]

about keepalive:
working:
req:
GET /uri HTTP/1.0
Connection: Keep-Alive
Host: somehost
User-Agent: someagent
Accept: */*

reply:
HTTP/1.1 200 OK
Date: Wed, 24 Mar 2010 09:44:36 GMT
Server: megaserver
Last-Modified: Sun, 22 Jul 2007 17:00:00 GMT
ETag: "4d6436-8cdd-435dd17316400"
Accept-Ranges: bytes
Content-Length: 36061
Keep-Alive: timeout=15, max=100
Connection: Keep-Alive
Content-Type: somecontent/type

mandory field in req is Connection: Keep-Alive.
mandory(imho) fields in reply:
Content-Length: 36061 (how much read ...)
Keep-Alive: timeout=15, max=100 (timeouts)
Connection: Keep-Alive (and this or "close" if not support this feature)

but what we see in couchdb's reply to same req:

HTTP/1.0 200 OK
Server: CouchDB/0.10.0 (Erlang OTP/R13B)
Date: Wed, 24 Mar 2010 09:37:27 GMT
Content-Type: text/plain;charset=utf-8
Content-Length: 41
Cache-Control: must-revalidate

Connection timeout n/a
Connection type n/a

i think RFC better source than my dump)

in my case (replicated+vers file storage) erlang eats all CPU to process
requests.
mb there is better way to get files from couchdb? by version or by e-tag?
or increase some kind buffers or memory pages?
sorry i don't know internals.

2010/3/24 Randall Leeds <ra...@gmail.com>

> Yes, I do mean KeepAlive, sorry for the confusion.
> CouchDB should support it. Can you show a dump of the headers received
> by Couch somehow? Maybe there is something silly like an issue of case
> with the headers.
>
> CouchDB cannot use sendfile (or some Erlang way to do the same)
> because it puts bytes in the middle of the attachment on disk at
> regular intervals. As I understand it, this is so that you can store
> attachments that contain something like a database header without
> breaking your database. Otherwise, storing something that looks like
> the BTree header in an attachment could cause couch to open the
> database incorrectly after an odd crash situation.
>
> This behavior maybe sounds very paranoid and strange, but one thing is
> for sure: CouchDB is designed to be *robust*. Really, really, _really_
> hard to lose your data.
>
> On Wed, Mar 24, 2010 at 01:01, Vasili Batareykin <jo...@gmail.com>
> wrote:
> > pipelining? you mean keepalive? ab hold test if you supply -k option (Use
> > HTTP KeepAlive feature) seems that couchdb's httpd don't know about this)
> > yes, throughput(in b/s) is better,but on localhost, if i use same test
> with
> > nginx i get around 1000 #/sec on 340k file. (344294.81 [Kbytes/sec]).
> yes,
> > nginx use sendfile for this operation. yes fs cache used too. but 70
> #/sec
> > with couchdb ...
> >
> > 2010/3/24 Randall Leeds <ra...@gmail.com>
> >
> >> If you multiple (#/sec) by file size, are actually getting _better_
> >> throughput with the larger files.
> >> Do you know if ab command uses HTTP 1.1 pipelining? If not, HTTP
> >> overhead would explain the extra time.
> >>
> >
>

Re: perfomance?

[Randall Leeds <ra...@gmail.com>]

Yes, I do mean KeepAlive, sorry for the confusion.
CouchDB should support it. Can you show a dump of the headers received
by Couch somehow? Maybe there is something silly like an issue of case
with the headers.

CouchDB cannot use sendfile (or some Erlang way to do the same)
because it puts bytes in the middle of the attachment on disk at
regular intervals. As I understand it, this is so that you can store
attachments that contain something like a database header without
breaking your database. Otherwise, storing something that looks like
the BTree header in an attachment could cause couch to open the
database incorrectly after an odd crash situation.

This behavior maybe sounds very paranoid and strange, but one thing is
for sure: CouchDB is designed to be *robust*. Really, really, _really_
hard to lose your data.

On Wed, Mar 24, 2010 at 01:01, Vasili Batareykin <jo...@gmail.com> wrote:
> pipelining? you mean keepalive? ab hold test if you supply -k option (Use
> HTTP KeepAlive feature) seems that couchdb's httpd don't know about this)
> yes, throughput(in b/s) is better,but on localhost, if i use same test with
> nginx i get around 1000 #/sec on 340k file. (344294.81 [Kbytes/sec]). yes,
> nginx use sendfile for this operation. yes fs cache used too. but 70 #/sec
> with couchdb ...
>
> 2010/3/24 Randall Leeds <ra...@gmail.com>
>
>> If you multiple (#/sec) by file size, are actually getting _better_
>> throughput with the larger files.
>> Do you know if ab command uses HTTP 1.1 pipelining? If not, HTTP
>> overhead would explain the extra time.
>>
>

Re: perfomance?

[Vasili Batareykin <jo...@gmail.com>]

pipelining? you mean keepalive? ab hold test if you supply -k option (Use
HTTP KeepAlive feature) seems that couchdb's httpd don't know about this)
yes, throughput(in b/s) is better,but on localhost, if i use same test with
nginx i get around 1000 #/sec on 340k file. (344294.81 [Kbytes/sec]). yes,
nginx use sendfile for this operation. yes fs cache used too. but 70 #/sec
with couchdb ...

2010/3/24 Randall Leeds <ra...@gmail.com>

> If you multiple (#/sec) by file size, are actually getting _better_
> throughput with the larger files.
> Do you know if ab command uses HTTP 1.1 pipelining? If not, HTTP
> overhead would explain the extra time.
>

Re: perfomance?

[Randall Leeds <ra...@gmail.com>]

Sorry,

To be clear

76.74 requests/sec * 378.912 kB/request = 28971.61152 kB/sec
701.9 requests/sec * 4.737 kB/request = 3324.9003 kB/sec

Larger files = better throughput. HTTP, TCP, or maybe B+Tree overhead.


On Wed, Mar 24, 2010 at 00:06, Randall Leeds <ra...@gmail.com> wrote:
> If you multiple (#/sec) by file size, are actually getting _better_
> throughput with the larger files.
> Do you know if ab command uses HTTP 1.1 pipelining? If not, HTTP
> overhead would explain the extra time.
>
> Your English is very clear. Please let me know if mine is not.
>
> Regards,
> Randall
>
> On Tue, Mar 23, 2010 at 22:44, Vasili Batareykin <jo...@gmail.com> wrote:
>> Hello!
>>
>> i'm trying to use couchdb in dumb case: as a file storage with http access)
>> but get performance problem
>> simple test:
>> curl -X PUT http://localhost:5984/users
>> curl -X PUT http://localhost:5984/users/static -d {}
>> curl -H "Content-Type: image/jpeg" -X PUT
>> http://localhost:5984/users/static/1.jpg?rev=4-41378c97921c2b3bc2a76f4c47f4ee86--data-binary
>> @1.jpg
>>
>> then trying to get that jpg with ab from apache distro:
>>
>> 76.46 [#/sec] ab -c 10 -n 1000
>> 39.13 [#/sec] ab -n 1000
>> 378912 bytes jpg
>>
>> 701.90 [#/sec] ab -c 10 -n 1000
>> 338.66 [#/sec] ab -n 1000
>> 4737 bytes jpg
>>
>> platform: ubuntu-karmic x86, couchdb, kernel from repo
>> CouchDB/0.10.0 (Erlang OTP/R13B)
>> Quad Xeon 2.66, 1Gb memory
>>
>> any way to get better #/sec for files >100kb? upgrade to 10.1? upgrade
>> erlang? or use x64 platform?
>> or this is not for couchdb?
>>
>> sorry for bad english.
>>
>

Re: perfomance?

[Randall Leeds <ra...@gmail.com>]

If you multiple (#/sec) by file size, are actually getting _better_
throughput with the larger files.
Do you know if ab command uses HTTP 1.1 pipelining? If not, HTTP
overhead would explain the extra time.

Your English is very clear. Please let me know if mine is not.

Regards,
Randall

On Tue, Mar 23, 2010 at 22:44, Vasili Batareykin <jo...@gmail.com> wrote:
> Hello!
>
> i'm trying to use couchdb in dumb case: as a file storage with http access)
> but get performance problem
> simple test:
> curl -X PUT http://localhost:5984/users
> curl -X PUT http://localhost:5984/users/static -d {}
> curl -H "Content-Type: image/jpeg" -X PUT
> http://localhost:5984/users/static/1.jpg?rev=4-41378c97921c2b3bc2a76f4c47f4ee86--data-binary
> @1.jpg
>
> then trying to get that jpg with ab from apache distro:
>
> 76.46 [#/sec] ab -c 10 -n 1000
> 39.13 [#/sec] ab -n 1000
> 378912 bytes jpg
>
> 701.90 [#/sec] ab -c 10 -n 1000
> 338.66 [#/sec] ab -n 1000
> 4737 bytes jpg
>
> platform: ubuntu-karmic x86, couchdb, kernel from repo
> CouchDB/0.10.0 (Erlang OTP/R13B)
> Quad Xeon 2.66, 1Gb memory
>
> any way to get better #/sec for files >100kb? upgrade to 10.1? upgrade
> erlang? or use x64 platform?
> or this is not for couchdb?
>
> sorry for bad english.
>