You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@couchdb.apache.org by "Michel Meyers (JIRA)" <ji...@apache.org> on 2012/10/25 09:58:12 UTC
[jira] [Created] (COUCHDB-1576) Allow deletions to expire after set
time
Michel Meyers created COUCHDB-1576:
--------------------------------------
Summary: Allow deletions to expire after set time
Key: COUCHDB-1576
URL: https://issues.apache.org/jira/browse/COUCHDB-1576
Project: CouchDB
Issue Type: Improvement
Components: Database Core
Affects Versions: 1.2
Reporter: Michel Meyers
Currently, CouchDB keeps all deletions forever. On a very active database this can lead to the number of deletions actually exceeding the number of active documents.
Lotus Domino solves this by having a cut-off date after which deletion stubs are purged from a replica. It would be great to have a similar functionality in CouchDB, e.g. have a setting that determines how long deletion stubs are kept and then purges them from the local copy. Whether this setting resides in the DB design, and thus replicates everywhere, or possibly an INI setting for the local CouchDB instance is still to be determined.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1576) Allow deletions to expire after
set time
Posted by "Dave Cottlehuber (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-1576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13483970#comment-13483970 ]
Dave Cottlehuber commented on COUCHDB-1576:
-------------------------------------------
Thanks Michel for reporting this. This is a consequence of using MVCC to enable straightforward replication and conflict handling API.
Are you aware of _revs_limit, a per-DB setting? It's not time-based but during compaction and replication this is taken into account:
http://wiki.apache.org/couchdb/HTTP_database_API#Accessing_Database-specific_options
http://wiki.apache.org/couchdb/Compaction/
Compaction is the way to recover that space, and for very active DBs the typical approach is to use a transactional DB and:
- either rotate them on a regular basis
- or aggregate to a separate metrics DB and re-create a fresh / empty DB regularly.
Is there an actual issue or barrier you're running into that won't be addressed by either compaction, or the transactional DB pattern?
> Allow deletions to expire after set time
> ----------------------------------------
>
> Key: COUCHDB-1576
> URL: https://issues.apache.org/jira/browse/COUCHDB-1576
> Project: CouchDB
> Issue Type: Improvement
> Components: Database Core
> Affects Versions: 1.2
> Reporter: Michel Meyers
>
> Currently, CouchDB keeps all deletions forever. On a very active database this can lead to the number of deletions actually exceeding the number of active documents.
> Lotus Domino solves this by having a cut-off date after which deletion stubs are purged from a replica. It would be great to have a similar functionality in CouchDB, e.g. have a setting that determines how long deletion stubs are kept and then purges them from the local copy. Whether this setting resides in the DB design, and thus replicates everywhere, or possibly an INI setting for the local CouchDB instance is still to be determined.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1576) Allow deletions to expire after
set time
Posted by "Michel Meyers (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-1576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484016#comment-13484016 ]
Michel Meyers commented on COUCHDB-1576:
----------------------------------------
Thanks for the comments. The _revs_limit appears to be a per-document setting (set on the DB) so it doesn't help in getting rid of old deleted documents (the documents themselves having been deleted, they never exceed the _revs_limit and remain in the DB forever).
As for rotating the DBs: That's possible but requires an outage across all replicas (of the entire service) to prevent new deletions from coming in and not being applied to the clean replica. If there was a way to do a filtered replication of only the documents and NEW deletions, this could work live, but I'm not sure that's possible.
It's not a big issue, but the perpetual growth of the database to keep these deletions does make it more difficult when you try to create a new replica in a remote location. A lot of data is being checked and transmitted for deleted stuff that no one cares about any more and could just be pruned instead. (In my case there's about a million deletion records amounting to 500MB of data...)
Compaction would be the way to recover space, except it too keeps the "deletion stubs". (That's my primary issue: The deletions never ever go away on their own.)
> Allow deletions to expire after set time
> ----------------------------------------
>
> Key: COUCHDB-1576
> URL: https://issues.apache.org/jira/browse/COUCHDB-1576
> Project: CouchDB
> Issue Type: Improvement
> Components: Database Core
> Affects Versions: 1.2
> Reporter: Michel Meyers
>
> Currently, CouchDB keeps all deletions forever. On a very active database this can lead to the number of deletions actually exceeding the number of active documents.
> Lotus Domino solves this by having a cut-off date after which deletion stubs are purged from a replica. It would be great to have a similar functionality in CouchDB, e.g. have a setting that determines how long deletion stubs are kept and then purges them from the local copy. Whether this setting resides in the DB design, and thus replicates everywhere, or possibly an INI setting for the local CouchDB instance is still to be determined.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1576) Allow deletions to expire after
set time
Posted by "Robert Newson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-1576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484056#comment-13484056 ]
Robert Newson commented on COUCHDB-1576:
----------------------------------------
+1 from me on the idea, though expiration-based sets my spider sense a-tingling. The _revs_limit feature is an explicit acknowledgment that keeping historical knowledge forever, while theoretically require to fulfil the promise of eventual consistency, is impractical.
Extending acknowledgement of that reality to long-deleted documents is a similar idea. The danger, of course, is that should you purge too soon, your set of replicas will never converge on the same state. So, for this to go ahead, I'd like to hear how we can mitigate that, or make it obvious to users.
One thought is not to make this a local decision, but one of agreement between a set of mutually replicating systems. An API endpoint that takes a list of url's to couchdb databases which produces a list of the union of deleted documents across them all, and purges them all. This might well be onerous but it would be safe, assuming the list was comprehensive. An expiration-based approach is potentially as good as this, as long as the expiration always exceeds the time that the systems come back into synch, but potentially falls short. Since there's no way to guarantee that two systems synchronize within a given timeframe, I'm wary of having it trigger based on the local clock.
/ramble
> Allow deletions to expire after set time
> ----------------------------------------
>
> Key: COUCHDB-1576
> URL: https://issues.apache.org/jira/browse/COUCHDB-1576
> Project: CouchDB
> Issue Type: Improvement
> Components: Database Core
> Affects Versions: 1.2
> Reporter: Michel Meyers
>
> Currently, CouchDB keeps all deletions forever. On a very active database this can lead to the number of deletions actually exceeding the number of active documents.
> Lotus Domino solves this by having a cut-off date after which deletion stubs are purged from a replica. It would be great to have a similar functionality in CouchDB, e.g. have a setting that determines how long deletion stubs are kept and then purges them from the local copy. Whether this setting resides in the DB design, and thus replicates everywhere, or possibly an INI setting for the local CouchDB instance is still to be determined.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira