You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@couchdb.apache.org by "nickva (via GitHub)" <gi...@apache.org> on 2026/03/12 22:13:39 UTC
[PR] Use dreyfus checkpoint for purge_seq [couchdb]
nickva opened a new pull request, #5920:
URL: https://github.com/apache/couchdb/pull/5920
Previously, in the index updater we used the purge_seq value from clouseau. In some cases that can return an older value (0 on a new index) than what is in the purge checkpoint doc created in `maybe_create_local_purge_doc/2`. (In that function we initialize the checkpoint with the db purge sequence and call `clouseau_rpc:set_purge_seq/1` to also set the clouseau purge seq value). An older purge sequence than the current minimum db purge sequence would result in an `invalid_start_purge_seq` being thrown during purged infos folding.
In general, if a client updates a purge checkpoint, then it should not query purged infos with a sequence value below that, since if that is the lowest current purge checkpoint value and compaction runs, it could have removed all the purged infos below that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Use dreyfus checkpoint for purge_seq [couchdb]
Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on PR #5920:
URL: https://github.com/apache/couchdb/pull/5920#issuecomment-4056507452
> ok, the purge checkpoint is updated at the end of the updater process so if clouseau crashes before the commit those purges have not happened, and would not be retried as the next update attempt will start from the purge checkpoint seq.
Yup, that's why I suggested a companion clouseau pr in our DM -- to commit when set_purge seq is set. That has to happen in two place: when we first initialize the purge sequence (I think this part maybe missing in nouveau, but that's a separate PR) and when we update it after purging a new batch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Use dreyfus checkpoint for purge_seq [couchdb]
Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva closed pull request #5920: Use dreyfus checkpoint for purge_seq
URL: https://github.com/apache/couchdb/pull/5920
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Use dreyfus checkpoint for purge_seq [couchdb]
Posted by "rnewson (via GitHub)" <gi...@apache.org>.
rnewson commented on PR #5920:
URL: https://github.com/apache/couchdb/pull/5920#issuecomment-4056303430
ok, the purge checkpoint is updated at the end of the updater process so if clouseau crashes before the commit those purges have not happened, and would not be retried as the next update attempt will start from the purge checkpoint seq.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Use dreyfus checkpoint for purge_seq [couchdb]
Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on PR #5920:
URL: https://github.com/apache/couchdb/pull/5920#issuecomment-4056109467
> Is the checkpoint document updated after the clouseau index has committed the purges up to that point to disk?
In the index update process it is updated the document after it calls set_purge on the clouseau index but the commit is not called yet, that's called a bit later after updating the bulk of the docs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Use dreyfus checkpoint for purge_seq [couchdb]
Posted by "rnewson (via GitHub)" <gi...@apache.org>.
rnewson commented on PR #5920:
URL: https://github.com/apache/couchdb/pull/5920#issuecomment-4055451107
Is the checkpoint document updated after the clouseau index has committed the purges up to that point to disk?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Use dreyfus checkpoint for purge_seq [couchdb]
Posted by "nickva (via GitHub)" <gi...@apache.org>.
nickva commented on PR #5920:
URL: https://github.com/apache/couchdb/pull/5920#issuecomment-4057107651
Companion clouseau PR to commit when setting the purge_seq https://github.com/cloudant-labs/clouseau/pull/143
Highlighting the two places we set the purge sequence:
https://github.com/apache/couchdb/blob/45b0fbc5f45fc589af478711942ead7e580ce0bd/src/dreyfus/src/dreyfus_util.erl#L353-L357
https://github.com/apache/couchdb/blob/45b0fbc5f45fc589af478711942ead7e580ce0bd/src/dreyfus/src/dreyfus_index_updater.erl#L115-L117
In both cases we first set the purge sequence and only then update the checkpoint document. If the index committed after updating the purge_seq we'd ensure it won't regress on restart if we don't manage to commit with a pending (in memory purge_seq in clouseau).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org