You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@couchdb.apache.org by jc...@apache.org on 2009/09/15 19:09:23 UTC
svn commit: r815404 - /couchdb/branches/0.10.x/etc/couchdb/default.ini.tpl.in
Author: jchris
Date: Tue Sep 15 17:09:22 2009
New Revision: 815404
URL: http://svn.apache.org/viewvc?rev=815404&view=rev
Log:
changed default for 0.10 to delayed_commits = true for fast out of box experience
Modified:
couchdb/branches/0.10.x/etc/couchdb/default.ini.tpl.in
Modified: couchdb/branches/0.10.x/etc/couchdb/default.ini.tpl.in
URL: http://svn.apache.org/viewvc/couchdb/branches/0.10.x/etc/couchdb/default.ini.tpl.in?rev=815404&r1=815403&r2=815404&view=diff
==============================================================================
--- couchdb/branches/0.10.x/etc/couchdb/default.ini.tpl.in (original)
+++ couchdb/branches/0.10.x/etc/couchdb/default.ini.tpl.in Tue Sep 15 17:09:22 2009
@@ -10,7 +10,7 @@
max_attachment_chunk_size = 4294967296 ; 4GB
os_process_timeout = 5000 ; 5 seconds. for view and external servers.
max_dbs_open = 100
-delayed_commits = false
+delayed_commits = true
batch_save_size = 1000 ; number of docs at which to save a batch
batch_save_interval = 1000 ; milliseconds after which to save batches
Re: svn commit: r815404 - /couchdb/branches/0.10.x/etc/couchdb/default.ini.tpl.in
Posted by Jan Lehnardt <ja...@apache.org>.
On 16 Sep 2009, at 10:05, Brian Candler wrote:
> On Tue, Sep 15, 2009 at 11:30:37AM -0700, Chris Anderson wrote:
>> The basic tradeoff here is between safety and speed. With a simple
>> benchmark on my Mac laptop (OSX 10.5) delayed_commits gives ~230
>> writes/second while we only get about 5/sec with the slow safe
>> option.
>>
>> I think 5 writes/second is too slow to be useful, so it doesn't
>> matter
>> how safe it is.
>
> And aside from that: why is it only 5 writes per second? I understand
> write -> sync -> write metadata -> sync, but any half-decent drive
> should
> still be able to do 100+ random writes per second.
>
> Would OSX be putting a sleep() in there somewhere??
Mac OS X docs encourage to use fcntl(F_FULLFSYNC) instead of
a simple fsync()*. The fcntl() call forces the drive to flush buffers
while fsync() just flushes the kernel buffer.
The guarantee that data ends up on the disk as promised is higher
using fcntl() on Mac OS X compared to fsync() on Linux.
Cheers
Jan
--
* http://developer.apple.com/mac/library/documentation/Darwin/Reference/ManPages/man2/fsync.2.html
Re: svn commit: r815404 -
/couchdb/branches/0.10.x/etc/couchdb/default.ini.tpl.in
Posted by Brian Candler <B....@pobox.com>.
On Tue, Sep 15, 2009 at 11:30:37AM -0700, Chris Anderson wrote:
> The basic tradeoff here is between safety and speed. With a simple
> benchmark on my Mac laptop (OSX 10.5) delayed_commits gives ~230
> writes/second while we only get about 5/sec with the slow safe option.
>
> I think 5 writes/second is too slow to be useful, so it doesn't matter
> how safe it is.
And aside from that: why is it only 5 writes per second? I understand
write -> sync -> write metadata -> sync, but any half-decent drive should
still be able to do 100+ random writes per second.
Would OSX be putting a sleep() in there somewhere??
Re: svn commit: r815404 - /couchdb/branches/0.10.x/etc/couchdb/default.ini.tpl.in
Posted by Robert Newson <ro...@gmail.com>.
A "durability matrix" that lays out all the possible combinations of
headers and behavior would be a good place to start. Something that
clearly states when data is synced to disk (and what fsync does on
different OS'es) would be an excellent deliverable in its own right. I
don't know all the spots on that grid, but perhaps I should start a
wiki page with what we do know?
B.
On Tue, Sep 15, 2009 at 11:54 PM, Jan Lehnardt <ja...@apache.org> wrote:
>
> On 15 Sep 2009, at 20:30, Chris Anderson wrote:
>
>> On Tue, Sep 15, 2009 at 10:22 AM, Jan Lehnardt <ja...@apache.org> wrote:
>>>
>>> On 15 Sep 2009, at 19:09, jchris@apache.org wrote:
>>>
>>>> Author: jchris
>>>> Date: Tue Sep 15 17:09:22 2009
>>>> New Revision: 815404
>>>>
>>>> URL: http://svn.apache.org/viewvc?rev=815404&view=rev
>>>> Log:
>>>> changed default for 0.10 to delayed_commits = true for fast out of box
>>>> experience
>>>
>>> Chris, please discuss this first on dev@.
>>>
>>
>> Sorry about that - just having a strong gut reaction to the potential
>> of releasing 0.10 that's too slow to use.
>>
>> Good thing is that this commit got everyone talking on IRC.
>>
>> The basic tradeoff here is between safety and speed. With a simple
>> benchmark on my Mac laptop (OSX 10.5) delayed_commits gives ~230
>> writes/second while we only get about 5/sec with the slow safe option.
>>
>> I think 5 writes/second is too slow to be useful, so it doesn't matter
>> how safe it is.
>
> I think we need to define use-cases a bit more clearly and then try to
> determine how much safety we can and want to guarantee. I'd like to
> avoid situations where we send a Created 201 response when that
> doc is still in limbo and CouchDB could do extra work to make sure
> it is on disk.
>
> I especially don't want to enable delayed commits by default for simple
> speed-freak reasons. CouchDB could easily detect long running fsync()s
> and send a log entry about enabling delayed commits. Proper documentation
> & release notes will help too. Finally, there will always be people claiming
> CouchDB has whatever properties (slow, fast, green) they see fit*.
>
> If we want to bill CouchDB as everybody's personal database, we better
> keep that personal data safe :)
>
> Cheers
> Jan
> --
> * See Postgres.
>
>
>
>
>
>
Re: svn commit: r815404 - /couchdb/branches/0.10.x/etc/couchdb/default.ini.tpl.in
Posted by Jan Lehnardt <ja...@apache.org>.
On 15 Sep 2009, at 20:30, Chris Anderson wrote:
> On Tue, Sep 15, 2009 at 10:22 AM, Jan Lehnardt <ja...@apache.org> wrote:
>>
>> On 15 Sep 2009, at 19:09, jchris@apache.org wrote:
>>
>>> Author: jchris
>>> Date: Tue Sep 15 17:09:22 2009
>>> New Revision: 815404
>>>
>>> URL: http://svn.apache.org/viewvc?rev=815404&view=rev
>>> Log:
>>> changed default for 0.10 to delayed_commits = true for fast out of
>>> box
>>> experience
>>
>> Chris, please discuss this first on dev@.
>>
>
> Sorry about that - just having a strong gut reaction to the potential
> of releasing 0.10 that's too slow to use.
>
> Good thing is that this commit got everyone talking on IRC.
>
> The basic tradeoff here is between safety and speed. With a simple
> benchmark on my Mac laptop (OSX 10.5) delayed_commits gives ~230
> writes/second while we only get about 5/sec with the slow safe option.
>
> I think 5 writes/second is too slow to be useful, so it doesn't matter
> how safe it is.
I think we need to define use-cases a bit more clearly and then try to
determine how much safety we can and want to guarantee. I'd like to
avoid situations where we send a Created 201 response when that
doc is still in limbo and CouchDB could do extra work to make sure
it is on disk.
I especially don't want to enable delayed commits by default for simple
speed-freak reasons. CouchDB could easily detect long running fsync()s
and send a log entry about enabling delayed commits. Proper
documentation
& release notes will help too. Finally, there will always be people
claiming
CouchDB has whatever properties (slow, fast, green) they see fit*.
If we want to bill CouchDB as everybody's personal database, we better
keep that personal data safe :)
Cheers
Jan
--
* See Postgres.
Re: svn commit: r815404 - /couchdb/branches/0.10.x/etc/couchdb/default.ini.tpl.in
Posted by Chris Anderson <jc...@apache.org>.
On Tue, Sep 15, 2009 at 10:22 AM, Jan Lehnardt <ja...@apache.org> wrote:
>
> On 15 Sep 2009, at 19:09, jchris@apache.org wrote:
>
>> Author: jchris
>> Date: Tue Sep 15 17:09:22 2009
>> New Revision: 815404
>>
>> URL: http://svn.apache.org/viewvc?rev=815404&view=rev
>> Log:
>> changed default for 0.10 to delayed_commits = true for fast out of box
>> experience
>
> Chris, please discuss this first on dev@.
>
Sorry about that - just having a strong gut reaction to the potential
of releasing 0.10 that's too slow to use.
Good thing is that this commit got everyone talking on IRC.
The basic tradeoff here is between safety and speed. With a simple
benchmark on my Mac laptop (OSX 10.5) delayed_commits gives ~230
writes/second while we only get about 5/sec with the slow safe option.
I think 5 writes/second is too slow to be useful, so it doesn't matter
how safe it is.
The other perspective is that we shouldn't say we've written something
unless its verifiably on disk. Due to the nature of fsync on various
platforms, this turns out to be non-trivial, as linux allows the disk
write buffer to lie about writes, but OSX does some tricks to force a
flush.
Damien on IRC explained his reasoning:
"so here's my thinking. if it's fast, but broken by default on the
server class OS, then why make it slow on less important consumer
grade machines."
What's left to resolve: if we decide to keep fast as the default in
0.10, we should also make it the default on trunk (until the
server-controlled batches are implemented.)
Chris
--
Chris Anderson
http://jchrisa.net
http://couch.io
Re: svn commit: r815404 - /couchdb/branches/0.10.x/etc/couchdb/default.ini.tpl.in
Posted by Jan Lehnardt <ja...@apache.org>.
On 15 Sep 2009, at 19:09, jchris@apache.org wrote:
> Author: jchris
> Date: Tue Sep 15 17:09:22 2009
> New Revision: 815404
>
> URL: http://svn.apache.org/viewvc?rev=815404&view=rev
> Log:
> changed default for 0.10 to delayed_commits = true for fast out of
> box experience
Chris, please discuss this first on dev@.
Cheers
Jan
--