You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Boaz Citrin <bc...@gmail.com> on 2014/04/22 14:06:47 UTC
Compaction of a database with the same number of documents is getting
slower over time
Hello,
Our database contains more or less the same number of documents, however
the documents themselves change frequently.
I would expect that compaction time will be the same, but I see that over
time it takes longer to compact the database.
Why is it so? Any way to overcome this?
Thanks,
Boaz
Re: Compaction of a database with the same number of documents is getting slower over time
Posted by Boaz <bc...@gmail.com>.
Thanks Adam!
> On 22 באפר 2014, at 19:32, Adam Kocoloski <ko...@apache.org> wrote:
>
> You can either run a filtered replication or block them with a validate_doc_update function on the target database. Something like the following would allow you to delete docs on the target but block the replication of tombstones:
>
> function (newDoc, oldDoc, userCtx) {
> // any update to an existing doc is OK
> if(oldDoc) {
> return;
> }
>
> // reject tombstones for docs we don't know about
> if(newDoc[\"_deleted\"]) {
> throw({forbidden : \"We're rejecting tombstones for unknown docs\"})
> }
> }
>
> Regards, Adam
>
>> On Apr 22, 2014, at 10:22 AM, Boaz Citrin <bc...@gmail.com> wrote:
>>
>> You got it right.
>>
>> How can I only replicate non deleted docs?
>> בתאריך 22 באפר' 2014 17:15, "Adam Kocoloski" <ko...@apache.org> כתב:
>>
>>> Sorry, just so we're super clear -- the "doc_count" is roughly constant
>>> but the "doc_del_count" is rising? Compaction time scales with the sum of
>>> those two values. Your options to get it back down under control at the
>>> moment are
>>>
>>> 1) Purge the deleted docs (lots of caveats about replication, potential
>>> for view index resets, etc.)
>>> 2) Rotate over to a new database, either by querying both or by
>>> replicating non-deleted docs to the new one
>>>
>>> Neither one is particularly palatable. CouchDB currently keeps the
>>> tombstones around forever so that replication can always work. Making
>>> changes on that front is a pretty subtle thing but maybe not completely
>>> impossible.
>>>
>>> Also, there's a new compactor in the works that is faster and generates
>>> smaller files.
>>>
>>> Adam
>>>
>>>> On Apr 22, 2014, at 9:46 AM, Boaz Citrin <bc...@gmail.com> wrote:
>>>>
>>>> Yes, the documents change quickly, Many deletions and insertions, but
>>> total
>>>> number don't change that much.
>>>>
>>>>
>>>> On Tue, Apr 22, 2014 at 4:41 PM, Adam Kocoloski <
>>> adam.kocoloski@gmail.com>wrote:
>>>>
>>>>> To first order it shouldn't be the case.
>>>>>
>>>>> Is the number of deleted documents growing with time? That'd have an
>>>>> impact.
>>>>>
>>>>> Adam
>>>>>
>>>>>> On Apr 22, 2014, at 8:06 AM, Boaz Citrin <bc...@gmail.com> wrote:
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> Our database contains more or less the same number of documents,
>>> however
>>>>>> the documents themselves change frequently.
>>>>>> I would expect that compaction time will be the same, but I see that
>>> over
>>>>>> time it takes longer to compact the database.
>>>>>>
>>>>>> Why is it so? Any way to overcome this?
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Boaz
>>>>>
>>>
>>>
>
Re: Compaction of a database with the same number of documents is getting slower over time
Posted by Adam Kocoloski <ko...@apache.org>.
You can either run a filtered replication or block them with a validate_doc_update function on the target database. Something like the following would allow you to delete docs on the target but block the replication of tombstones:
function (newDoc, oldDoc, userCtx) {
// any update to an existing doc is OK
if(oldDoc) {
return;
}
// reject tombstones for docs we don't know about
if(newDoc[\"_deleted\"]) {
throw({forbidden : \"We're rejecting tombstones for unknown docs\"})
}
}
Regards, Adam
On Apr 22, 2014, at 10:22 AM, Boaz Citrin <bc...@gmail.com> wrote:
> You got it right.
>
> How can I only replicate non deleted docs?
> בתאריך 22 באפר' 2014 17:15, "Adam Kocoloski" <ko...@apache.org> כתב:
>
>> Sorry, just so we're super clear -- the "doc_count" is roughly constant
>> but the "doc_del_count" is rising? Compaction time scales with the sum of
>> those two values. Your options to get it back down under control at the
>> moment are
>>
>> 1) Purge the deleted docs (lots of caveats about replication, potential
>> for view index resets, etc.)
>> 2) Rotate over to a new database, either by querying both or by
>> replicating non-deleted docs to the new one
>>
>> Neither one is particularly palatable. CouchDB currently keeps the
>> tombstones around forever so that replication can always work. Making
>> changes on that front is a pretty subtle thing but maybe not completely
>> impossible.
>>
>> Also, there's a new compactor in the works that is faster and generates
>> smaller files.
>>
>> Adam
>>
>> On Apr 22, 2014, at 9:46 AM, Boaz Citrin <bc...@gmail.com> wrote:
>>
>>> Yes, the documents change quickly, Many deletions and insertions, but
>> total
>>> number don't change that much.
>>>
>>>
>>> On Tue, Apr 22, 2014 at 4:41 PM, Adam Kocoloski <
>> adam.kocoloski@gmail.com>wrote:
>>>
>>>> To first order it shouldn't be the case.
>>>>
>>>> Is the number of deleted documents growing with time? That'd have an
>>>> impact.
>>>>
>>>> Adam
>>>>
>>>>> On Apr 22, 2014, at 8:06 AM, Boaz Citrin <bc...@gmail.com> wrote:
>>>>>
>>>>> Hello,
>>>>>
>>>>> Our database contains more or less the same number of documents,
>> however
>>>>> the documents themselves change frequently.
>>>>> I would expect that compaction time will be the same, but I see that
>> over
>>>>> time it takes longer to compact the database.
>>>>>
>>>>> Why is it so? Any way to overcome this?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Boaz
>>>>
>>
>>
Re: Compaction of a database with the same number of documents is
getting slower over time
Posted by Boaz Citrin <bc...@gmail.com>.
You got it right.
How can I only replicate non deleted docs?
בתאריך 22 באפר' 2014 17:15, "Adam Kocoloski" <ko...@apache.org> כתב:
> Sorry, just so we're super clear -- the "doc_count" is roughly constant
> but the "doc_del_count" is rising? Compaction time scales with the sum of
> those two values. Your options to get it back down under control at the
> moment are
>
> 1) Purge the deleted docs (lots of caveats about replication, potential
> for view index resets, etc.)
> 2) Rotate over to a new database, either by querying both or by
> replicating non-deleted docs to the new one
>
> Neither one is particularly palatable. CouchDB currently keeps the
> tombstones around forever so that replication can always work. Making
> changes on that front is a pretty subtle thing but maybe not completely
> impossible.
>
> Also, there's a new compactor in the works that is faster and generates
> smaller files.
>
> Adam
>
> On Apr 22, 2014, at 9:46 AM, Boaz Citrin <bc...@gmail.com> wrote:
>
> > Yes, the documents change quickly, Many deletions and insertions, but
> total
> > number don't change that much.
> >
> >
> > On Tue, Apr 22, 2014 at 4:41 PM, Adam Kocoloski <
> adam.kocoloski@gmail.com>wrote:
> >
> >> To first order it shouldn't be the case.
> >>
> >> Is the number of deleted documents growing with time? That'd have an
> >> impact.
> >>
> >> Adam
> >>
> >>> On Apr 22, 2014, at 8:06 AM, Boaz Citrin <bc...@gmail.com> wrote:
> >>>
> >>> Hello,
> >>>
> >>> Our database contains more or less the same number of documents,
> however
> >>> the documents themselves change frequently.
> >>> I would expect that compaction time will be the same, but I see that
> over
> >>> time it takes longer to compact the database.
> >>>
> >>> Why is it so? Any way to overcome this?
> >>>
> >>> Thanks,
> >>>
> >>> Boaz
> >>
>
>
Re: Compaction of a database with the same number of documents is getting slower over time
Posted by Adam Kocoloski <ko...@apache.org>.
Sorry, just so we're super clear -- the "doc_count" is roughly constant but the "doc_del_count" is rising? Compaction time scales with the sum of those two values. Your options to get it back down under control at the moment are
1) Purge the deleted docs (lots of caveats about replication, potential for view index resets, etc.)
2) Rotate over to a new database, either by querying both or by replicating non-deleted docs to the new one
Neither one is particularly palatable. CouchDB currently keeps the tombstones around forever so that replication can always work. Making changes on that front is a pretty subtle thing but maybe not completely impossible.
Also, there's a new compactor in the works that is faster and generates smaller files.
Adam
On Apr 22, 2014, at 9:46 AM, Boaz Citrin <bc...@gmail.com> wrote:
> Yes, the documents change quickly, Many deletions and insertions, but total
> number don't change that much.
>
>
> On Tue, Apr 22, 2014 at 4:41 PM, Adam Kocoloski <ad...@gmail.com>wrote:
>
>> To first order it shouldn't be the case.
>>
>> Is the number of deleted documents growing with time? That'd have an
>> impact.
>>
>> Adam
>>
>>> On Apr 22, 2014, at 8:06 AM, Boaz Citrin <bc...@gmail.com> wrote:
>>>
>>> Hello,
>>>
>>> Our database contains more or less the same number of documents, however
>>> the documents themselves change frequently.
>>> I would expect that compaction time will be the same, but I see that over
>>> time it takes longer to compact the database.
>>>
>>> Why is it so? Any way to overcome this?
>>>
>>> Thanks,
>>>
>>> Boaz
>>
Re: Compaction of a database with the same number of documents is
getting slower over time
Posted by Boaz Citrin <bc...@gmail.com>.
Yes, the documents change quickly, Many deletions and insertions, but total
number don't change that much.
On Tue, Apr 22, 2014 at 4:41 PM, Adam Kocoloski <ad...@gmail.com>wrote:
> To first order it shouldn't be the case.
>
> Is the number of deleted documents growing with time? That'd have an
> impact.
>
> Adam
>
> > On Apr 22, 2014, at 8:06 AM, Boaz Citrin <bc...@gmail.com> wrote:
> >
> > Hello,
> >
> > Our database contains more or less the same number of documents, however
> > the documents themselves change frequently.
> > I would expect that compaction time will be the same, but I see that over
> > time it takes longer to compact the database.
> >
> > Why is it so? Any way to overcome this?
> >
> > Thanks,
> >
> > Boaz
>
Re: Compaction of a database with the same number of documents is getting slower over time
Posted by Adam Kocoloski <ad...@gmail.com>.
To first order it shouldn't be the case.
Is the number of deleted documents growing with time? That'd have an impact.
Adam
> On Apr 22, 2014, at 8:06 AM, Boaz Citrin <bc...@gmail.com> wrote:
>
> Hello,
>
> Our database contains more or less the same number of documents, however
> the documents themselves change frequently.
> I would expect that compaction time will be the same, but I see that over
> time it takes longer to compact the database.
>
> Why is it so? Any way to overcome this?
>
> Thanks,
>
> Boaz