CouchDB compaction [again]...

[kowsik <ko...@gmail.com>]

We are running a cluster of 1.0.2 CouchDB's in production
(http://blitz.io) and the one and only thing that we have to look into
periodically is compaction. Are there plans for automatic compaction
in the roadmap so we can completely relax? With the recent work by
@fdmanana and @damienkatz I realize that the db size doesn't grow as
rapidly now as before, what with the snappy compression and what not.
But still this periodic checking in gets in the way of relaxing. Turns
out 1.0.2 has a bug with compaction and _changes where CouchDB can
just go poof and not leave a trace behind. It's been fixed since then,
but still...

While we <3 CouchDB and would like to check on it every now and then,
we'd rather relax and be assured that it's always there in the
background, mapping and reducing and being happy about it. One less
thing to "monitor" and be paged on.

Does anyone else have a write heavy couch cluster that they are
running in production with multi-master replication and _changes?
Would love to exchange notes on what you are doing to keep the DB size
sane.

Thanks,

K.
---
http://blitz.io
http://twitter.com/pcapr
http://blog.mudynamics.com

Re: CouchDB compaction [again]...

[Randall Leeds <ra...@gmail.com>]

Our approach has been to do continuous compaction. It means I can relax and
there are no performance surprises (sudden drops when it's time to compact).
On Jun 25, 2011 2:50 PM, "Filipe David Manana" <fd...@apache.org> wrote:
> On Sat, Jun 25, 2011 at 6:32 AM, kowsik <ko...@gmail.com> wrote:
>> We are running a cluster of 1.0.2 CouchDB's in production
>> (http://blitz.io) and the one and only thing that we have to look into
>> periodically is compaction. Are there plans for automatic compaction
>> in the roadmap so we can completely relax?
>
> Hi Kowsik,
>
> I've recently made a configurable compaction daemon (database and
> views). It's proposed in:
> https://issues.apache.org/jira/browse/COUCHDB-1153
>
> I still have quite a few enhancements in my head todo, but in general
> it's ok for many cases. It was included in the Couchbase 2.0 Single
> preview CouchDB distribution. So far it had positive feedback, and no
> bugs found yet.
>
> The documentation for it:
>
https://github.com/fdmanana/couchdb/blob/compaction_daemon/etc/couchdb/default.ini.tpl.in#L189
>
> It makes use of recent "data_size" attribute added to trunk:
> https://issues.apache.org/jira/browse/COUCHDB-1132
>
>
> With the recent work by
>> @fdmanana and @damienkatz I realize that the db size doesn't grow as
>> rapidly now as before, what with the snappy compression and what not.
>> But still this periodic checking in gets in the way of relaxing. Turns
>> out 1.0.2 has a bug with compaction and _changes where CouchDB can
>> just go poof and not leave a trace behind. It's been fixed since then,
>> but still...
>>
>> While we <3 CouchDB and would like to check on it every now and then,
>> we'd rather relax and be assured that it's always there in the
>> background, mapping and reducing and being happy about it. One less
>> thing to "monitor" and be paged on.
>>
>> Does anyone else have a write heavy couch cluster that they are
>> running in production with multi-master replication and _changes?
>> Would love to exchange notes on what you are doing to keep the DB size
>> sane.
>>
>> Thanks,
>>
>> K.
>> ---
>> http://blitz.io
>> http://twitter.com/pcapr
>> http://blog.mudynamics.com
>>
>
>
>
> --
> Filipe David Manana,
> fdmanana@gmail.com, fdmanana@apache.org
>
> "Reasonable men adapt themselves to the world.
>  Unreasonable men adapt the world to themselves.
>  That's why all progress depends on unreasonable men."

Re: CouchDB compaction [again]...

[Filipe David Manana <fd...@apache.org>]

On Sat, Jun 25, 2011 at 6:32 AM, kowsik <ko...@gmail.com> wrote:
> We are running a cluster of 1.0.2 CouchDB's in production
> (http://blitz.io) and the one and only thing that we have to look into
> periodically is compaction. Are there plans for automatic compaction
> in the roadmap so we can completely relax?

Hi Kowsik,

I've recently made a configurable compaction daemon (database and
views). It's proposed in:
https://issues.apache.org/jira/browse/COUCHDB-1153

I still have quite a few enhancements in my head todo, but in general
it's ok for many cases. It was included in the Couchbase 2.0 Single
preview CouchDB distribution. So far it had positive feedback, and no
bugs found yet.

The documentation for it:
https://github.com/fdmanana/couchdb/blob/compaction_daemon/etc/couchdb/default.ini.tpl.in#L189

It makes use of recent "data_size" attribute added to trunk:
https://issues.apache.org/jira/browse/COUCHDB-1132


With the recent work by
> @fdmanana and @damienkatz I realize that the db size doesn't grow as
> rapidly now as before, what with the snappy compression and what not.
> But still this periodic checking in gets in the way of relaxing. Turns
> out 1.0.2 has a bug with compaction and _changes where CouchDB can
> just go poof and not leave a trace behind. It's been fixed since then,
> but still...
>
> While we <3 CouchDB and would like to check on it every now and then,
> we'd rather relax and be assured that it's always there in the
> background, mapping and reducing and being happy about it. One less
> thing to "monitor" and be paged on.
>
> Does anyone else have a write heavy couch cluster that they are
> running in production with multi-master replication and _changes?
> Would love to exchange notes on what you are doing to keep the DB size
> sane.
>
> Thanks,
>
> K.
> ---
> http://blitz.io
> http://twitter.com/pcapr
> http://blog.mudynamics.com
>



-- 
Filipe David Manana,
fdmanana@gmail.com, fdmanana@apache.org

"Reasonable men adapt themselves to the world.
 Unreasonable men adapt the world to themselves.
 That's why all progress depends on unreasonable men."