You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Matt Goodall <ma...@gmail.com> on 2009/01/27 14:30:19 UTC
Compaction documentation vs behaviour
Hi,
Both the wiki, http://wiki.apache.org/couchdb/Compaction, and Futon
warn that compacting removes deleted documents. However, a deleted
document is still represented in _all_docs_by_seq after compaction.
$ curl -X "PUT" -d "{}" "http://localhost:5984/compaction/doc"
{"ok":true,"id":"doc","rev":"143469513"}
$ curl -X "DELETE" "http://localhost:5984/compaction/doc?rev=143469513"
{"ok":true,"id":"doc","rev":"1135956802"}
$ curl -X "POST" "http://localhost:5984/compaction/_compact"
{"ok":true}
$ curl "http://localhost:5984/compaction/_all_docs_by_seq"
{"total_rows":0,"offset":0,"rows":[
{"id":"doc","key":2,"value":{"rev":"1135956802","deleted":true}}
]}
Presumably, the wiki & Futon are incorrect as it makes complete sense
to keep deleted documents around for replication, but I'll wait for a
response to this email before I create a ticket.
- Matt
Re: Compaction documentation vs behaviour
Posted by Jan Lehnardt <ja...@apache.org>.
The doc itself is gone, the deleted stub is still there for
replication conflict resolution.
Cheers
Jan
--
On 27 Jan 2009, at 14:30, Matt Goodall wrote:
> Hi,
>
> Both the wiki, http://wiki.apache.org/couchdb/Compaction, and Futon
> warn that compacting removes deleted documents. However, a deleted
> document is still represented in _all_docs_by_seq after compaction.
>
> $ curl -X "PUT" -d "{}" "http://localhost:5984/compaction/doc"
> {"ok":true,"id":"doc","rev":"143469513"}
> $ curl -X "DELETE" "http://localhost:5984/compaction/doc?
> rev=143469513"
> {"ok":true,"id":"doc","rev":"1135956802"}
> $ curl -X "POST" "http://localhost:5984/compaction/_compact"
> {"ok":true}
> $ curl "http://localhost:5984/compaction/_all_docs_by_seq"
> {"total_rows":0,"offset":0,"rows":[
> {"id":"doc","key":2,"value":{"rev":"1135956802","deleted":true}}
> ]}
>
> Presumably, the wiki & Futon are incorrect as it makes complete sense
> to keep deleted documents around for replication, but I'll wait for a
> response to this email before I create a ticket.
>
> - Matt
>
Re: Compaction documentation vs behaviour
Posted by Chris Anderson <jc...@apache.org>.
On Tue, Jan 27, 2009 at 11:45 AM, Matt Goodall <ma...@gmail.com> wrote:
> 2009/1/27 Chris Anderson <jc...@apache.org>:
>> On Tue, Jan 27, 2009 at 5:30 AM, Matt Goodall <ma...@gmail.com> wrote:
>>> Hi,
>>>
>>> Both the wiki, http://wiki.apache.org/couchdb/Compaction, and Futon
>>> warn that compacting removes deleted documents. However, a deleted
>>> document is still represented in _all_docs_by_seq after compaction.
>>
>> Perhaps the wording should be more clear. Compaction removes old revs
>> of all documents. So in the case of a deleted document, the head rev
>> is the one that marks it as deleted, but before compaction, the
>> original version of the document (in your example, rev 143469513) is
>> kept in the file. After compaction, only the latest rev remains.
>
> Yeah, that was pretty much my understanding of it until I read the
> wiki and noticed Futon's warning. (Although I had to double check to
> make sure!)
>
>>
>> Compaction preserves any conflict-revs as well.
>
> So, if we were to fix the wording, are there any other issues you can
> think of other than:
>
> a) old revisions will be removed which may make it difficult to
> correct conflicts
> b) it can take a very long time on large documents
>
Compaction works by recopying the still-valid nodes from the old
database file to a new database file. So it will take longer for large
dbs. Under heavy write load, it is possible that compaction can never
complete, unless write load is abated for a moment to allow compaction
to finish and swap the new file for the old.
It's essentially a form of on-disk generational garbage collection.
>>
>>>
>>> $ curl -X "PUT" -d "{}" "http://localhost:5984/compaction/doc"
>>> {"ok":true,"id":"doc","rev":"143469513"}
>>> $ curl -X "DELETE" "http://localhost:5984/compaction/doc?rev=143469513"
>>> {"ok":true,"id":"doc","rev":"1135956802"}
>>> $ curl -X "POST" "http://localhost:5984/compaction/_compact"
>>> {"ok":true}
>>> $ curl "http://localhost:5984/compaction/_all_docs_by_seq"
>>> {"total_rows":0,"offset":0,"rows":[
>>> {"id":"doc","key":2,"value":{"rev":"1135956802","deleted":true}}
>>> ]}
>>>
>>> Presumably, the wiki & Futon are incorrect as it makes complete sense
>>> to keep deleted documents around for replication, but I'll wait for a
>>> response to this email before I create a ticket.
>>>
>>> - Matt
>>>
>>
>>
>>
>> --
>> Chris Anderson
>> http://jchris.mfdz.com
>>
>
--
Chris Anderson
http://jchris.mfdz.com
Re: Compaction documentation vs behaviour
Posted by Matt Goodall <ma...@gmail.com>.
2009/1/27 Chris Anderson <jc...@apache.org>:
> On Tue, Jan 27, 2009 at 5:30 AM, Matt Goodall <ma...@gmail.com> wrote:
>> Hi,
>>
>> Both the wiki, http://wiki.apache.org/couchdb/Compaction, and Futon
>> warn that compacting removes deleted documents. However, a deleted
>> document is still represented in _all_docs_by_seq after compaction.
>
> Perhaps the wording should be more clear. Compaction removes old revs
> of all documents. So in the case of a deleted document, the head rev
> is the one that marks it as deleted, but before compaction, the
> original version of the document (in your example, rev 143469513) is
> kept in the file. After compaction, only the latest rev remains.
Yeah, that was pretty much my understanding of it until I read the
wiki and noticed Futon's warning. (Although I had to double check to
make sure!)
>
> Compaction preserves any conflict-revs as well.
So, if we were to fix the wording, are there any other issues you can
think of other than:
a) old revisions will be removed which may make it difficult to
correct conflicts
b) it can take a very long time on large documents
- Matt
>
>>
>> $ curl -X "PUT" -d "{}" "http://localhost:5984/compaction/doc"
>> {"ok":true,"id":"doc","rev":"143469513"}
>> $ curl -X "DELETE" "http://localhost:5984/compaction/doc?rev=143469513"
>> {"ok":true,"id":"doc","rev":"1135956802"}
>> $ curl -X "POST" "http://localhost:5984/compaction/_compact"
>> {"ok":true}
>> $ curl "http://localhost:5984/compaction/_all_docs_by_seq"
>> {"total_rows":0,"offset":0,"rows":[
>> {"id":"doc","key":2,"value":{"rev":"1135956802","deleted":true}}
>> ]}
>>
>> Presumably, the wiki & Futon are incorrect as it makes complete sense
>> to keep deleted documents around for replication, but I'll wait for a
>> response to this email before I create a ticket.
>>
>> - Matt
>>
>
>
>
> --
> Chris Anderson
> http://jchris.mfdz.com
>
Re: Compaction documentation vs behaviour
Posted by Adam Petty <ad...@gmail.com>.
Thanks very much - much better!
On Tue, Jan 27, 2009 at 1:44 PM, Chris Anderson <jc...@apache.org> wrote:
> On Tue, Jan 27, 2009 at 10:39 AM, Adam Petty <ad...@gmail.com> wrote:
> > Hello,
> >
> > Is it possible to set the compaction to leave *n* doc revs? I've looked
> > around breifly - but wasn't sure. We'd like to use the doc revs for
> "undo"
> > or "history" features but still be able to compact the db file.
> >
>
> doc rev is used for optimistic concurrency. if you are interested in a
> versioning system you'll need to provide it at the application layer.
> The simplest way to do this is to store subsequent versions of
> application documents as new couch docs, and include the id of the
> original couch document, as well as a timestamp, in each updated
> version. Then you can do a view by
>
> emit([doc.original_id || doc._id, doc.timestamp], null)
>
> to get a list of all versions of a document in order.
>
>
> --
> Chris Anderson
> http://jchris.mfdz.com
>
Re: Compaction documentation vs behaviour
Posted by Chris Anderson <jc...@apache.org>.
On Tue, Jan 27, 2009 at 10:39 AM, Adam Petty <ad...@gmail.com> wrote:
> Hello,
>
> Is it possible to set the compaction to leave *n* doc revs? I've looked
> around breifly - but wasn't sure. We'd like to use the doc revs for "undo"
> or "history" features but still be able to compact the db file.
>
doc rev is used for optimistic concurrency. if you are interested in a
versioning system you'll need to provide it at the application layer.
The simplest way to do this is to store subsequent versions of
application documents as new couch docs, and include the id of the
original couch document, as well as a timestamp, in each updated
version. Then you can do a view by
emit([doc.original_id || doc._id, doc.timestamp], null)
to get a list of all versions of a document in order.
--
Chris Anderson
http://jchris.mfdz.com
Re: Compaction documentation vs behaviour
Posted by Adam Petty <ad...@gmail.com>.
Hello,
Is it possible to set the compaction to leave *n* doc revs? I've looked
around breifly - but wasn't sure. We'd like to use the doc revs for "undo"
or "history" features but still be able to compact the db file.
--Adam
On Tue, Jan 27, 2009 at 12:53 PM, Chris Anderson <jc...@apache.org> wrote:
> On Tue, Jan 27, 2009 at 5:30 AM, Matt Goodall <ma...@gmail.com>
> wrote:
> > Hi,
> >
> > Both the wiki, http://wiki.apache.org/couchdb/Compaction, and Futon
> > warn that compacting removes deleted documents. However, a deleted
> > document is still represented in _all_docs_by_seq after compaction.
>
> Perhaps the wording should be more clear. Compaction removes old revs
> of all documents. So in the case of a deleted document, the head rev
> is the one that marks it as deleted, but before compaction, the
> original version of the document (in your example, rev 143469513) is
> kept in the file. After compaction, only the latest rev remains.
>
> Compaction preserves any conflict-revs as well.
>
> >
> > $ curl -X "PUT" -d "{}" "http://localhost:5984/compaction/doc"
> > {"ok":true,"id":"doc","rev":"143469513"}
> > $ curl -X "DELETE" "http://localhost:5984/compaction/doc?rev=143469513"
> > {"ok":true,"id":"doc","rev":"1135956802"}
> > $ curl -X "POST" "http://localhost:5984/compaction/_compact"
> > {"ok":true}
> > $ curl "http://localhost:5984/compaction/_all_docs_by_seq"
> > {"total_rows":0,"offset":0,"rows":[
> > {"id":"doc","key":2,"value":{"rev":"1135956802","deleted":true}}
> > ]}
> >
> > Presumably, the wiki & Futon are incorrect as it makes complete sense
> > to keep deleted documents around for replication, but I'll wait for a
> > response to this email before I create a ticket.
> >
> > - Matt
> >
>
>
>
> --
> Chris Anderson
> http://jchris.mfdz.com
>
Re: Compaction documentation vs behaviour
Posted by Chris Anderson <jc...@apache.org>.
On Tue, Jan 27, 2009 at 5:30 AM, Matt Goodall <ma...@gmail.com> wrote:
> Hi,
>
> Both the wiki, http://wiki.apache.org/couchdb/Compaction, and Futon
> warn that compacting removes deleted documents. However, a deleted
> document is still represented in _all_docs_by_seq after compaction.
Perhaps the wording should be more clear. Compaction removes old revs
of all documents. So in the case of a deleted document, the head rev
is the one that marks it as deleted, but before compaction, the
original version of the document (in your example, rev 143469513) is
kept in the file. After compaction, only the latest rev remains.
Compaction preserves any conflict-revs as well.
>
> $ curl -X "PUT" -d "{}" "http://localhost:5984/compaction/doc"
> {"ok":true,"id":"doc","rev":"143469513"}
> $ curl -X "DELETE" "http://localhost:5984/compaction/doc?rev=143469513"
> {"ok":true,"id":"doc","rev":"1135956802"}
> $ curl -X "POST" "http://localhost:5984/compaction/_compact"
> {"ok":true}
> $ curl "http://localhost:5984/compaction/_all_docs_by_seq"
> {"total_rows":0,"offset":0,"rows":[
> {"id":"doc","key":2,"value":{"rev":"1135956802","deleted":true}}
> ]}
>
> Presumably, the wiki & Futon are incorrect as it makes complete sense
> to keep deleted documents around for replication, but I'll wait for a
> response to this email before I create a ticket.
>
> - Matt
>
--
Chris Anderson
http://jchris.mfdz.com