High latency (40ms) and low request rate (10 rps) under windows

[Konstantin Cherkasov <co...@imarto.net>]

Hi!

I installed CouchDB 1.1.0 on Windows (Winows 7, 32 bit) and tried to
test performance using ab.exe
(Apache Benchmark http://httpd.apache.org/docs/2.2/programs/ab.html)

ab.exe -k -c 1 -n 1000 -p ab.json -T "application/json" http://localhost:5984/bench

where ab.json contains the simplest document {"value": "test"}.

This utility do 1000 POSTs with no concurency (-c 1) in single
connection (-k for keepalive).

On my laptop (it's NOT low cost netbook) I got 10 requests per second.
So it is 0.1 second per single request.
And CPU and HDD utilization is actually ZERO.

I was just wondering what exactly CouchDB doing all this time
(0.1 second) with single little record?

As the utilization of CPU and HDD is 0%, I believe that they are not the bottleneck.

So where is bottleneck?



P.S.
All test I did with delayed_commits = false.
And I tried socket_options = [{nodelay, true}]

-- 
 Konstantin Cherkasov

Re: High latency (40ms) and low request rate (10 rps) under windows

[Konstantin Cherkasov <co...@imarto.net>]

Hi, Alexander

> As for my experience CouchDB on Windows is slower approximately for
> two times than on Linux. Also, check up TCP no_delay option on client
> and server side - it could make greed speed up of rps rate.

Yep. I believe that apache's ab (as it based on APR) already have TCP_NODELAY.
But to be sure one can turn off keepalives ("-k").
In think keepalives affect latency but not too dramatically.


-- 
 Konstantin Cherkasov

Re: High latency (40ms) and low request rate (10 rps) under windows

[Alexander Shorin <kx...@gmail.com>]

Hi, Konstantin

As for my experience CouchDB on Windows is slower approximately for
two times than on Linux. Also, check up TCP no_delay option on client
and server side - it could make greed speed up of rps rate.

--
,,,^..^,,,



2011/10/25 Konstantin Cherkasov <co...@imarto.net>:
> Hi!
>
> I installed CouchDB 1.1.0 on Windows (Winows 7, 32 bit) and tried to
> test performance using ab.exe
> (Apache Benchmark http://httpd.apache.org/docs/2.2/programs/ab.html)
>
> ab.exe -k -c 1 -n 1000 -p ab.json -T "application/json" http://localhost:5984/bench
>
> where ab.json contains the simplest document {"value": "test"}.
>
> This utility do 1000 POSTs with no concurency (-c 1) in single
> connection (-k for keepalive).
>
> On my laptop (it's NOT low cost netbook) I got 10 requests per second.
> So it is 0.1 second per single request.
> And CPU and HDD utilization is actually ZERO.
>
> I was just wondering what exactly CouchDB doing all this time
> (0.1 second) with single little record?
>
> As the utilization of CPU and HDD is 0%, I believe that they are not the bottleneck.
>
> So where is bottleneck?
>
>
>
> P.S.
> All test I did with delayed_commits = false.
> And I tried socket_options = [{nodelay, true}]
>
> --
>  Konstantin Cherkasov
>
>

Re: High latency (40ms) and low request rate (10 rps) under windows

[Keith Gable <zi...@ignition-project.com>]

I've had similar experience, though never with enterprise (Seagate ES or
equivalent) drives. More writes and thrashing in general seems to make
normal drives die sooner. I have only anecdotal evidence though, so it could
just be coincidental.
On Oct 27, 2011 1:18 PM, "CGS" <cg...@gmail.com> wrote:

> No reference, just experience from my former jobs (comparing hdd's of the
> same type bought in the same time). Maybe the new hdd's are better from that
> point of view, I haven't tested on the new hdd's because such tests take a
> while.
>
>
>
>
> On 10/27/2011 07:47 PM, Mark Hahn wrote:
>
>> Nevertheless, writing directly on the harddisk is not quite the best
>>>
>> practice because you reduce the life of your harddisk to at least half
>>
>> Do you have a reference for this?  Drives are designed to be used 100% of
>> the time.
>>
>>
>

Re: High latency (40ms) and low request rate (10 rps) under windows

[CGS <cg...@gmail.com>]

No reference, just experience from my former jobs (comparing hdd's of 
the same type bought in the same time). Maybe the new hdd's are better 
from that point of view, I haven't tested on the new hdd's because such 
tests take a while.




On 10/27/2011 07:47 PM, Mark Hahn wrote:
>> Nevertheless, writing directly on the harddisk is not quite the best
> practice because you reduce the life of your harddisk to at least half
>
> Do you have a reference for this?  Drives are designed to be used 100% of
> the time.
>

Re: High latency (40ms) and low request rate (10 rps) under windows

[Mark Hahn <ma...@hahnca.com>]

> Nevertheless, writing directly on the harddisk is not quite the best
practice because you reduce the life of your harddisk to at least half

Do you have a reference for this?  Drives are designed to be used 100% of
the time.

Re: High latency (40ms) and low request rate (10 rps) under windows

[CGS <cg...@gmail.com>]

English is not my native language as well, so, it may be that I 
misunderstood your point.

That is the general case with or without your specific intervention. 
Nevertheless, writing directly on the harddisk is not quite the best 
practice because you reduce the life of your harddisk to at least half 
(I've experienced such practices and I've seen some monstrous results). 
In addition, as far as I understood how CouchDB works, writing 
one-by-one documents forces the two headers to be updated every time one 
document arrives, while buffering the documents and flush them on the 
harddisk at once requires one update session for the two headers. So, 
you have to balance your design in between speed and safety of your data.

But, as long as you don't expect power failure (a little bit of optimism 
here, I know) and you take care of not trying to use more resources than 
you have, buffering will lose no data if your application is write-safe 
(e.g., check for integrity of the operation before discarding the data 
from the buffer and use of 2 buffers: one as accumulator and one for 
interfacing with the writing part). I know that's a bit of extra-work 
here and you use more resources, but it provides a good quantification 
of your data transfer toward your harddisk. Using this and the power of 
CouchDB of fast recovering from errors, my impression is that those 10 
docs per second rate drawback can be ignored. But, of course, that's my 
opinion and it is subject of less objectivity.

Well, I know you have lot of experience and I hope you don't take this 
post as a lecture (far from me to lecture anyone as long as I need to 
learn more), but as a more detailed opinion to support my previously 
expressed suggestion. I hope I didn't bore you to death with this post.

Cheers,
CGS




On 10/27/2011 10:09 AM, Konstantin Cherkasov wrote:
>> I do not quite understand why to lose 1 day from your customer data.
> Excuse my English.
> I meant "one day" = "once" in the sense that buffer usually does not assume durability.
> In other words, if you choose to buffer the data then you are ready that
> there is a probability the data can be lost.
>

Re: High latency (40ms) and low request rate (10 rps) under windows

[Jens Alfke <je...@couchbase.com>]

On Oct 27, 2011, at 1:09 AM, Konstantin Cherkasov wrote:

I meant "one day" = "once" in the sense that buffer usually does not assume durability.
In other words, if you choose to buffer the data then you are ready that
there is a probability the data can be lost.

Face it: the data’s going to get buffered somewhere. If you don’t buffer it, then either the database server will, or the OS kernel will, or the disk controller will. It’s definitely fastest to let the controller do the buffering, but since it knows nothing of your file format or even the disk’s volume format, the results of data loss are much worse (i.e. file corruption.) Doing the buffering at a higher level provides better reliability at the expense of performance.

—Jens

Re: High latency (40ms) and low request rate (10 rps) under windows

[Konstantin Cherkasov <co...@imarto.net>]

> I do not quite understand why to lose 1 day from your customer data.
Excuse my English.
I meant "one day" = "once" in the sense that buffer usually does not assume durability.
In other words, if you choose to buffer the data then you are ready that
there is a probability the data can be lost.

-- 
 Konstantin Cherkasov

Re: High latency (40ms) and low request rate (10 rps) under windows

[CGS <cg...@gmail.com>]

Hi,

I do not quite understand why to lose 1 day from your customer data. If 
you mean by that the time to upgrade your software, I suppose you don't 
need one day, but maximum few seconds to switch to the new version 
(e.g., if you use Erlang/OTP, you can just upgrade the software without 
any need to stop your process). I am sure you know why I said few 
seconds, so, I won't enter into details here.

Nevertheless, you know better your project, so, if you consider batch 
mode more useful, it's up to you. My point in the previous post was 
mainly related to

"OMG! My database do 10 requests per second (for ex. Postgres do 1000 
rps on the same hardware with fsync on)
"Forget this, just use BATCH MODE"

which I do not understand its message other than mockery when I was 
trying to give you alternatives. I apologize if I offended you with my 
suggestions. I will choose more carefully from now on to whom I will answer.

Good luck with your projects!
CGS




On 10/27/2011 01:20 AM, Konstantin Cherkasov wrote:
> Hi!
>
>> I am not sure what is your point with this thread because I notice
>> frustration and mockery in your posts, but failing to understand why.
>> Maybe because I am slow, who knows? You joined this and devs list just
>> to prove that CouchDB is slow? If you don't like CouchDB, PostgreSQL is
>> faster and it's all you need (speed for one session in serialized
>> writing), why bother to write here?
> Sorry, but why did you conclude that I do not like CouchDB?
> I've been using CouchDB about a year for different applications, but I'm seriously worried about its performance.
> I'm trying to draw attention to an issue (quite possibly it actually exists).
> And yes, to find out what the problem is I'm trying to compare the behavior of different
> databases on the same hardware.
>
> FYI
> Recently quite tricky bug in couchdb-python was fixed.
> It was in the code responsible for http and led to unnecessary delays in Restful API.
> As a result, the latency decreased by 4-5 times. This bug has been there for several
> years until someone tested and started asking questions.
>
>
> [...]
>
>> It's not so difficult to create a buffer and to
>> send all the documents at once using this kind of operation
> The keyword is BUFFER (which means "one day I will lose part of customer's data, but I do not care").
> Actually you don't need any buffer as an external component.
> Just use "batch=ok"(it's 2 times faster than delayed_commits = true) and CouchDB will buffer for you.
>

Re: High latency (40ms) and low request rate (10 rps) under windows

[Konstantin Cherkasov <co...@imarto.net>]

Hi!

> I am not sure what is your point with this thread because I notice 
> frustration and mockery in your posts, but failing to understand why. 
> Maybe because I am slow, who knows? You joined this and devs list just 
> to prove that CouchDB is slow? If you don't like CouchDB, PostgreSQL is 
> faster and it's all you need (speed for one session in serialized 
> writing), why bother to write here?

Sorry, but why did you conclude that I do not like CouchDB?
I've been using CouchDB about a year for different applications, but I'm seriously worried about its performance.
I'm trying to draw attention to an issue (quite possibly it actually exists).
And yes, to find out what the problem is I'm trying to compare the behavior of different
databases on the same hardware.

FYI
Recently quite tricky bug in couchdb-python was fixed.
It was in the code responsible for http and led to unnecessary delays in Restful API.
As a result, the latency decreased by 4-5 times. This bug has been there for several
years until someone tested and started asking questions.


[...]

> It's not so difficult to create a buffer and to
> send all the documents at once using this kind of operation

The keyword is BUFFER (which means "one day I will lose part of customer's data, but I do not care").
Actually you don't need any buffer as an external component.
Just use "batch=ok"(it's 2 times faster than delayed_commits = true) and CouchDB will buffer for you.

-- 
 Konstantin

Re: High latency (40ms) and low request rate (10 rps) under windows

[CGS <cg...@gmail.com>]

Hi,

I am not sure what is your point with this thread because I notice 
frustration and mockery in your posts, but failing to understand why. 
Maybe because I am slow, who knows? You joined this and devs list just 
to prove that CouchDB is slow? If you don't like CouchDB, PostgreSQL is 
faster and it's all you need (speed for one session in serialized 
writing), why bother to write here?

And about batch mode, you know you exaggerated a bit. You can use bulk 
operation in the real life as well (I do that as it is required by the 
type of my projects). It's not so difficult to create a buffer and to 
send all the documents at once using this kind of operation (if you 
expect so many docs one-by-one from the same session). The result in 
this case becomes even better than in PostgreSQL case (the case you were 
speaking of) if that matters to you. Also, coming back to the batch 
mode, I suppose there more events fitting this profile than the one you 
were speaking about (blogging is more often required by the web industry 
than the rest of the cases). If you refer to messaging for example, 10 
documents (or those 173 you found with delayed_commits true, but I take 
the worst case) are enough for such cases. And now, summing up, blogging 
and messaging are the most popular requests in the web industry today. 
For all the rest of the projects which requires automation, you can use 
the bulk operation I mentioned before (if you want to serialize the 
writing part).

So, I repeat, maybe it's me, but I don't see the point of your 
complaint, especially that nobody forced you to join this community. If 
your boss (if you have any) is forcing you to use this product and you 
don't like it, find ways to improve the speed for what you need. Nobody 
forces you to come here and say "your product sucks because in one 
particular case it doesn't work as I want!" I also found many products 
being incompatible with the requirements of my projects, but I don't 
tell the users community this or that product is bad and useless. If I 
really need that product and I cannot make it run as I need, I just ask 
for help if it is possible to tune in that product for my needs. Maybe 
it's my way of behaving wrong.

Good luck!
CGS






On 10/25/2011 11:19 PM, Konstantin Cherkasov wrote:
> Hi!
>
>> 1. delayed_commits helps you in high concurrency (batch mode) and less
>> in no concurrency.
> You are wrong.
>
> curl -X PUT -H "Content-Type: application/json" http://127.0.0.1:5984/_config/couchdb/delayed_commits -d "\"true\""
> ab -k -c 1 -n 1000 -p ab.json -T "application/json" http://localhost:5984/bench
>
> Requests per second:    173.71 [#/sec] (mean)
>
> curl -X PUT -H "Content-Type: application/json" http://127.0.0.1:5984/_config/couchdb/delayed_commits -d "\"false\""
> ab -k -c 1 -n 1000 -p ab.json -T "application/json" http://localhost:5984/bench
>
> Requests per second:    10.15 [#/sec] (mean)
>
> CPU&  HDD utilization 0.01%.
> It looks like there are some delays in TCP or HTTP.
>
>
>> 2. If one wants to send in the same session lots of documents, that
>> someone uses bulk operation. Otherwise, high concurrency comes in play
>> by the means of batch mode.
> Well, in real life there are many cases when events or data come not in batches but one by one.
>
> In truth, it seems to me that this BATCH MODE MANTRA leads to defects in the code that simply are not investigated.
>
> "OMG! My database do 10 requests per second (for ex. Postgres do 1000 rps on the same hardware with fsync on)
> "Forget this, just use BATCH MODE"
>
>
>

Re: High latency (40ms) and low request rate (10 rps) under windows

[Adam Kocoloski <ko...@apache.org>]

On Oct 25, 2011, at 11:16 PM, Mark Hahn wrote:

>> Isn't couchdb crash-proof due to append-only writing?
> 
> Does anyone know for sure about this?  I have used append-only files for
> many years and have never seen any corruption.  I would assume writes happen
> in sequence.  If the trailer is missing then it would just stop at the
> penultimate trailer.

I've seen multiple cases of data corruption when the fsync barrier prior to writing the "trailer" is omitted.  By default CouchDB issues this barrier for database commits but not for commits of derived view index headers.  Best,

Adam

Re: High latency (40ms) and low request rate (10 rps) under windows

[Mark Hahn <ma...@hahnca.com>]

> Isn't couchdb crash-proof due to append-only writing?

Does anyone know for sure about this?  I have used append-only files for
many years and have never seen any corruption.  I would assume writes happen
in sequence.  If the trailer is missing then it would just stop at the
penultimate trailer.

Re: High latency (40ms) and low request rate (10 rps) under windows

[Jens Alfke <je...@couchbase.com>]

On Oct 25, 2011, at 3:09 PM, Mark Hahn wrote:

>> CouchDB does a full sync, because this is the only way to be proof against
>> disasters like power failures
> 
> Isn't couchdb crash-proof due to append-only writing?  What do you gain
> other than possible loss of latest writes, which you can lose anyway with a
> fsync.

You can get corrupted files even with append-only writing. In the worst case, let’s say everything gets written to disk before the power failure _except_ for one disk block in the middle of the update*. After rebooting, you have what appears to be a valid file (it’s got the magic trailer) except that 4096 (or 512 or whatever) bytes in one of the last updated documents are garbage.

I don’t know the details of how CouchDB finds the trailer in the file, but it would have to be doing something like a checksum of every single write to guard against that; which seems too expensive to me.

Instead, to be safe, what you do is write the payload, wait for a full fsync, then write the trailer only after you know that the entire payload is safely on the platters.

—Jens

* Disk controllers don’t write sectors in the order they receive the write requests. They shove them in the cache, then write them out grouped by tracks to minimize seek time.

Re: High latency (40ms) and low request rate (10 rps) under windows

[Mark Hahn <ma...@hahnca.com>]

> CouchDB does a full sync, because this is the only way to be proof against
disasters like power failures

Isn't couchdb crash-proof due to append-only writing?  What do you gain
other than possible loss of latest writes, which you can lose anyway with a
fsync.

Re: High latency (40ms) and low request rate (10 rps) under windows

[Jens Alfke <je...@couchbase.com>]

On Oct 25, 2011, at 2:19 PM, Konstantin Cherkasov wrote:

"OMG! My database do 10 requests per second (for ex. Postgres do 1000 rps on the same hardware with fsync on)
"Forget this, just use BATCH MODE"

It’s probably not the same fsync, so you’re comparing apples and oranges. There is no way a full fsync (pushing all writes onto the disk platters) can be done 1000x a second; 10 is closer to what I’ve seen.

This is a tricky topic. If you just call fsync, you get a “lazy” sync where the kernel pushes writes to the disk controller but doesn’t wait for them to be written; so they just end up in the controller’s cache until sometime in the future, not actually in persistent storage yet. To get a “full” fsync that includes a flush command to the controller, you have to do some extra work (the API depends on the OS.)

CouchDB does a full sync, because this is the only way to be proof against disasters like power failures, but there is an option to turn it off. (I think there is a writeup on the CouchDB wiki about this.)

The same thing happens with sqlite, by the way, if its full-fsync option is on (it is by default on OS X, not sure of other platforms.) I have seen with my own two eyes the kind of fatal corruption of sqlite databases that you can get if full-fsync isn’t enabled and a power failure or kernel panic occurs in the middle of a commit; it’s not pretty.

—Jens

Re: High latency (40ms) and low request rate (10 rps) under windows

[Konstantin Cherkasov <co...@imarto.net>]

Hi!

> 1. delayed_commits helps you in high concurrency (batch mode) and less 
> in no concurrency.
You are wrong.

curl -X PUT -H "Content-Type: application/json" http://127.0.0.1:5984/_config/couchdb/delayed_commits -d "\"true\""
ab -k -c 1 -n 1000 -p ab.json -T "application/json" http://localhost:5984/bench

Requests per second:    173.71 [#/sec] (mean)

curl -X PUT -H "Content-Type: application/json" http://127.0.0.1:5984/_config/couchdb/delayed_commits -d "\"false\""
ab -k -c 1 -n 1000 -p ab.json -T "application/json" http://localhost:5984/bench

Requests per second:    10.15 [#/sec] (mean)

CPU & HDD utilization 0.01%.
It looks like there are some delays in TCP or HTTP.


> 2. If one wants to send in the same session lots of documents, that 
> someone uses bulk operation. Otherwise, high concurrency comes in play 
> by the means of batch mode.

Well, in real life there are many cases when events or data come not in batches but one by one.

In truth, it seems to me that this BATCH MODE MANTRA leads to defects in the code that simply are not investigated.

"OMG! My database do 10 requests per second (for ex. Postgres do 1000 rps on the same hardware with fsync on)
"Forget this, just use BATCH MODE"



-- 
 Konstantin Cherkasov

Re: High latency (40ms) and low request rate (10 rps) under windows

[CGS <cg...@gmail.com>]

Hi,

Few points here:

1. delayed_commits helps you in high concurrency (batch mode) and less 
in no concurrency.
2. If one wants to send in the same session lots of documents, that 
someone uses bulk operation. Otherwise, high concurrency comes in play 
by the means of batch mode.

For more information look at the "official" benchmarks at:

http://guide.couchdb.org/editions/1/en/performance.html

and you will see your hardware shows quite good qualities (comparing 
with 4-5 documents per second written in the official documentation).

Cheers,
CGS




On 10/25/2011 09:22 PM, Konstantin Cherkasov wrote:
> Hi!
>
> I installed CouchDB 1.1.0 on Windows (Winows 7, 32 bit) and tried to
> test performance using ab.exe
> (Apache Benchmark http://httpd.apache.org/docs/2.2/programs/ab.html)
>
> ab.exe -k -c 1 -n 1000 -p ab.json -T "application/json" http://localhost:5984/bench
>
> where ab.json contains the simplest document {"value": "test"}.
>
> This utility do 1000 POSTs with no concurency (-c 1) in single
> connection (-k for keepalive).
>
> On my laptop (it's NOT low cost netbook) I got 10 requests per second.
> So it is 0.1 second per single request.
> And CPU and HDD utilization is actually ZERO.
>
> I was just wondering what exactly CouchDB doing all this time
> (0.1 second) with single little record?
>
> As the utilization of CPU and HDD is 0%, I believe that they are not the bottleneck.
>
> So where is bottleneck?
>
>
>
> P.S.
> All test I did with delayed_commits = false.
> And I tried socket_options = [{nodelay, true}]
>

Re: High latency (40ms) and low request rate (10 rps) under windows

[Robert Newson <rn...@apache.org>]

My guess would be fsync().

B.

2011/10/25 Konstantin Cherkasov <co...@imarto.net>:
> Hi!
>
> I installed CouchDB 1.1.0 on Windows (Winows 7, 32 bit) and tried to
> test performance using ab.exe
> (Apache Benchmark http://httpd.apache.org/docs/2.2/programs/ab.html)
>
> ab.exe -k -c 1 -n 1000 -p ab.json -T "application/json" http://localhost:5984/bench
>
> where ab.json contains the simplest document {"value": "test"}.
>
> This utility do 1000 POSTs with no concurency (-c 1) in single
> connection (-k for keepalive).
>
> On my laptop (it's NOT low cost netbook) I got 10 requests per second.
> So it is 0.1 second per single request.
> And CPU and HDD utilization is actually ZERO.
>
> I was just wondering what exactly CouchDB doing all this time
> (0.1 second) with single little record?
>
> As the utilization of CPU and HDD is 0%, I believe that they are not the bottleneck.
>
> So where is bottleneck?
>
>
>
> P.S.
> All test I did with delayed_commits = false.
> And I tried socket_options = [{nodelay, true}]
>
> --
>  Konstantin Cherkasov
>
>