You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@couchdb.apache.org by Adam Kocoloski <ad...@gmail.com> on 2009/02/27 02:25:23 UTC
preventing compaction from ruining the OS block cache
Hi, I've noticed that compacting large DBs pretty much kills any
filesystem caching benefits for CouchDB. I believe the problem is
that the OS (Linux 2.6.21 kernel in my case) is caching blocks from
the .compact file, even though those blocks won't be read again until
compaction has finished. In the meantime, the portion of the cache
dedicated to the old DB file shrinks and performance really suffers.
I think a better mode of operation would be to advise/instruct the OS
not to cache any portion of the .compact file until we're ready to
replace the main DB. On Linux, specifying the POSIX_FADV_DONTNEED
option to posix_fadvise() seems like the way to go:
http://linux.die.net/man/2/posix_fadvise
This link has a little more detail and a usage example:
http://insights.oetiker.ch/linux/fadvise.html
Of course, POSIX_FADV_DONTNEED isn't really available from inside the
Erlang VM. Perhaps the simplest approach would be to have a helper
process that we can spawn which calls that function (or its equivalent
on a non-Linux OS) periodically during compaction? I'm not really
sure, but I wanted to get this out on the list for discussion. Best,
Adam
Re: preventing compaction from ruining the OS block cache
Posted by Jan Lehnardt <ja...@apache.org>.
moved this to https://issues.apache.org/jira/browse/COUCHDB-271
so it doesn't fall between the cracks.
Cheers
Jan
--
On 27 Feb 2009, at 19:32, Damien Katz wrote:
> Oh yeah, one more option that is kind of crazy is to spawn a small
> external child process for file io. It would be a very small simple
> process that opens a file and responds to read/write commands from
> the erlang server. Then we can implement exactly the low level apis
> and caching behavior desired. The cost is extra IPC, but that should
> be small compare the the cost of a blown file cache.
>
> -Damien
>
>
> On Feb 27, 2009, at 1:23 PM, Damien Katz wrote:
>
>> The problem is we don't get access to the low level apis or flags
>> passed in to the OS unless Erlang chooses to expose it. We have
>> similar problems with compaction on windows because we need special
>> flags to give us unix file semantics.
>>
>> To fix this, we'll either need the Erlang VM changed or use our own
>> Erlang file driver interface.
>>
>> -Damien
>>
>>
>> On Feb 26, 2009, at 8:25 PM, Adam Kocoloski wrote:
>>
>>> Hi, I've noticed that compacting large DBs pretty much kills any
>>> filesystem caching benefits for CouchDB. I believe the problem is
>>> that the OS (Linux 2.6.21 kernel in my case) is caching blocks
>>> from the .compact file, even though those blocks won't be read
>>> again until compaction has finished. In the meantime, the portion
>>> of the cache dedicated to the old DB file shrinks and performance
>>> really suffers.
>>>
>>> I think a better mode of operation would be to advise/instruct the
>>> OS not to cache any portion of the .compact file until we're ready
>>> to replace the main DB. On Linux, specifying the
>>> POSIX_FADV_DONTNEED option to posix_fadvise() seems like the way
>>> to go:
>>>
>>> http://linux.die.net/man/2/posix_fadvise
>>>
>>> This link has a little more detail and a usage example:
>>>
>>> http://insights.oetiker.ch/linux/fadvise.html
>>>
>>> Of course, POSIX_FADV_DONTNEED isn't really available from inside
>>> the Erlang VM. Perhaps the simplest approach would be to have a
>>> helper process that we can spawn which calls that function (or its
>>> equivalent on a non-Linux OS) periodically during compaction? I'm
>>> not really sure, but I wanted to get this out on the list for
>>> discussion. Best,
>>>
>>> Adam
>>
>
>
Re: preventing compaction from ruining the OS block cache
Posted by Damien Katz <da...@apache.org>.
Oh yeah, one more option that is kind of crazy is to spawn a small
external child process for file io. It would be a very small simple
process that opens a file and responds to read/write commands from the
erlang server. Then we can implement exactly the low level apis and
caching behavior desired. The cost is extra IPC, but that should be
small compare the the cost of a blown file cache.
-Damien
On Feb 27, 2009, at 1:23 PM, Damien Katz wrote:
> The problem is we don't get access to the low level apis or flags
> passed in to the OS unless Erlang chooses to expose it. We have
> similar problems with compaction on windows because we need special
> flags to give us unix file semantics.
>
> To fix this, we'll either need the Erlang VM changed or use our own
> Erlang file driver interface.
>
> -Damien
>
>
> On Feb 26, 2009, at 8:25 PM, Adam Kocoloski wrote:
>
>> Hi, I've noticed that compacting large DBs pretty much kills any
>> filesystem caching benefits for CouchDB. I believe the problem is
>> that the OS (Linux 2.6.21 kernel in my case) is caching blocks from
>> the .compact file, even though those blocks won't be read again
>> until compaction has finished. In the meantime, the portion of the
>> cache dedicated to the old DB file shrinks and performance really
>> suffers.
>>
>> I think a better mode of operation would be to advise/instruct the
>> OS not to cache any portion of the .compact file until we're ready
>> to replace the main DB. On Linux, specifying the
>> POSIX_FADV_DONTNEED option to posix_fadvise() seems like the way to
>> go:
>>
>> http://linux.die.net/man/2/posix_fadvise
>>
>> This link has a little more detail and a usage example:
>>
>> http://insights.oetiker.ch/linux/fadvise.html
>>
>> Of course, POSIX_FADV_DONTNEED isn't really available from inside
>> the Erlang VM. Perhaps the simplest approach would be to have a
>> helper process that we can spawn which calls that function (or its
>> equivalent on a non-Linux OS) periodically during compaction? I'm
>> not really sure, but I wanted to get this out on the list for
>> discussion. Best,
>>
>> Adam
>
Re: preventing compaction from ruining the OS block cache
Posted by Damien Katz <da...@apache.org>.
The problem is we don't get access to the low level apis or flags
passed in to the OS unless Erlang chooses to expose it. We have
similar problems with compaction on windows because we need special
flags to give us unix file semantics.
To fix this, we'll either need the Erlang VM changed or use our own
Erlang file driver interface.
-Damien
On Feb 26, 2009, at 8:25 PM, Adam Kocoloski wrote:
> Hi, I've noticed that compacting large DBs pretty much kills any
> filesystem caching benefits for CouchDB. I believe the problem is
> that the OS (Linux 2.6.21 kernel in my case) is caching blocks from
> the .compact file, even though those blocks won't be read again
> until compaction has finished. In the meantime, the portion of the
> cache dedicated to the old DB file shrinks and performance really
> suffers.
>
> I think a better mode of operation would be to advise/instruct the
> OS not to cache any portion of the .compact file until we're ready
> to replace the main DB. On Linux, specifying the
> POSIX_FADV_DONTNEED option to posix_fadvise() seems like the way to
> go:
>
> http://linux.die.net/man/2/posix_fadvise
>
> This link has a little more detail and a usage example:
>
> http://insights.oetiker.ch/linux/fadvise.html
>
> Of course, POSIX_FADV_DONTNEED isn't really available from inside
> the Erlang VM. Perhaps the simplest approach would be to have a
> helper process that we can spawn which calls that function (or its
> equivalent on a non-Linux OS) periodically during compaction? I'm
> not really sure, but I wanted to get this out on the list for
> discussion. Best,
>
> Adam