You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@couchdb.apache.org by "Randall Leeds (JIRA)" <ji...@apache.org> on 2010/03/17 23:37:27 UTC
[jira] Created: (COUCHDB-704) Replication can lose checkpoints
Replication can lose checkpoints
--------------------------------
Key: COUCHDB-704
URL: https://issues.apache.org/jira/browse/COUCHDB-704
Project: CouchDB
Issue Type: Bug
Components: Replication
Affects Versions: 0.12
Reporter: Randall Leeds
Priority: Minor
When saving replication checkpoints in the _local/<repid> document the new entry is always pushed onto the _original_ "history" list property that existed at the start of the replication. When any number of things causes the checkpoint to be written to only one of the databases the head of the history list gets out of sync. Subsequent attempts to start this replication must start from the latest common replication log entry in the _original_ history, as though this replication never occurred.
A better idea is to push every checkpoint onto the history instead of replacing the head on each save.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-704) Replication can lose checkpoints
Posted by "Adam Kocoloski (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12922958#action_12922958 ]
Adam Kocoloski commented on COUCHDB-704:
----------------------------------------
/me facepalm
Sorry, I should have tracked the timestamps on the patch submissions. I'll apply keep_session_id.patch now.
> Replication can lose checkpoints
> --------------------------------
>
> Key: COUCHDB-704
> URL: https://issues.apache.org/jira/browse/COUCHDB-704
> Project: CouchDB
> Issue Type: Bug
> Components: Replication
> Affects Versions: 0.11.2, 1.0.1
> Reporter: Randall Leeds
> Priority: Minor
> Attachments: keep_session_id.patch, save-all-rep-checkpoints.patch, whitespace.patch
>
> Original Estimate: 0h
> Remaining Estimate: 0h
>
> When saving replication checkpoints in the _local/<repid> document the new entry is always pushed onto the _original_ "history" list property that existed at the start of the replication. When any number of things causes the checkpoint to be written to only one of the databases the head of the history list gets out of sync. Subsequent attempts to start this replication must start from the latest common replication log entry in the _original_ history, as though this replication never occurred.
> A better idea is to push every checkpoint onto the history instead of replacing the head on each save.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-704) Replication can lose checkpoints
Posted by "Randall Leeds (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12867724#action_12867724 ]
Randall Leeds commented on COUCHDB-704:
---------------------------------------
Looking over this once more.
If we only append to the replication log, would it be more accurate to clear the stats after each checkpoint?
The last log entry in the reply to a client request for non-continuous replication won't show the total number of documents replicated, but only the number since the last checkpoint. I don't know the best way to address this.
> Replication can lose checkpoints
> --------------------------------
>
> Key: COUCHDB-704
> URL: https://issues.apache.org/jira/browse/COUCHDB-704
> Project: CouchDB
> Issue Type: Bug
> Components: Replication
> Affects Versions: 0.12
> Reporter: Randall Leeds
> Priority: Minor
> Attachments: save-all-rep-checkpoints.patch, whitespace.patch
>
> Original Estimate: 0h
> Remaining Estimate: 0h
>
> When saving replication checkpoints in the _local/<repid> document the new entry is always pushed onto the _original_ "history" list property that existed at the start of the replication. When any number of things causes the checkpoint to be written to only one of the databases the head of the history list gets out of sync. Subsequent attempts to start this replication must start from the latest common replication log entry in the _original_ history, as though this replication never occurred.
> A better idea is to push every checkpoint onto the history instead of replacing the head on each save.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (COUCHDB-704) Replication can lose checkpoints
Posted by "Randall Leeds (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Randall Leeds updated COUCHDB-704:
----------------------------------
Attachment: whitespace.patch
Just cleaning up some whitespace ugliness I saw creep in here since I last mucked around in couch_rep.erl.
> Replication can lose checkpoints
> --------------------------------
>
> Key: COUCHDB-704
> URL: https://issues.apache.org/jira/browse/COUCHDB-704
> Project: CouchDB
> Issue Type: Bug
> Components: Replication
> Affects Versions: 0.12
> Reporter: Randall Leeds
> Priority: Minor
> Attachments: save-all-rep-checkpoints.patch, whitespace.patch
>
> Original Estimate: 0h
> Remaining Estimate: 0h
>
> When saving replication checkpoints in the _local/<repid> document the new entry is always pushed onto the _original_ "history" list property that existed at the start of the replication. When any number of things causes the checkpoint to be written to only one of the databases the head of the history list gets out of sync. Subsequent attempts to start this replication must start from the latest common replication log entry in the _original_ history, as though this replication never occurred.
> A better idea is to push every checkpoint onto the history instead of replacing the head on each save.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-704) Replication can lose checkpoints
Posted by "Randall Leeds (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12921723#action_12921723 ]
Randall Leeds commented on COUCHDB-704:
---------------------------------------
Filipe,
It's true. This is an edge case, but I have had it happen in production with a database that had crawled to *very* slow writes and pull replication. The checkpoint code updated the source first and the local document was written, but the response was too slow so it was taken as a timeout. When the replicator retried the save it got a conflict. Replication crashed and the target was never written.
I can imagine other, rare instances where this could occur. It's an edge case, but a potentially nasty one.
> Replication can lose checkpoints
> --------------------------------
>
> Key: COUCHDB-704
> URL: https://issues.apache.org/jira/browse/COUCHDB-704
> Project: CouchDB
> Issue Type: Bug
> Components: Replication
> Affects Versions: 0.11.2, 1.0.1
> Reporter: Randall Leeds
> Priority: Minor
> Attachments: keep_session_id.patch, save-all-rep-checkpoints.patch, whitespace.patch
>
> Original Estimate: 0h
> Remaining Estimate: 0h
>
> When saving replication checkpoints in the _local/<repid> document the new entry is always pushed onto the _original_ "history" list property that existed at the start of the replication. When any number of things causes the checkpoint to be written to only one of the databases the head of the history list gets out of sync. Subsequent attempts to start this replication must start from the latest common replication log entry in the _original_ history, as though this replication never occurred.
> A better idea is to push every checkpoint onto the history instead of replacing the head on each save.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-704) Replication can lose checkpoints
Posted by "Filipe Manana (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12921726#action_12921726 ]
Filipe Manana commented on COUCHDB-704:
---------------------------------------
I had once or 2 that same issue with conflict when saving the checkpoint in the source DB (for a pull replication). But I don't remember if there was a retry or not, neither could reproduce the issue.
Let's wait for Adam's point of view.
> Replication can lose checkpoints
> --------------------------------
>
> Key: COUCHDB-704
> URL: https://issues.apache.org/jira/browse/COUCHDB-704
> Project: CouchDB
> Issue Type: Bug
> Components: Replication
> Affects Versions: 0.11.2, 1.0.1
> Reporter: Randall Leeds
> Priority: Minor
> Attachments: keep_session_id.patch, save-all-rep-checkpoints.patch, whitespace.patch
>
> Original Estimate: 0h
> Remaining Estimate: 0h
>
> When saving replication checkpoints in the _local/<repid> document the new entry is always pushed onto the _original_ "history" list property that existed at the start of the replication. When any number of things causes the checkpoint to be written to only one of the databases the head of the history list gets out of sync. Subsequent attempts to start this replication must start from the latest common replication log entry in the _original_ history, as though this replication never occurred.
> A better idea is to push every checkpoint onto the history instead of replacing the head on each save.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (COUCHDB-704) Replication can lose checkpoints
Posted by "Randall Leeds (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Randall Leeds updated COUCHDB-704:
----------------------------------
Affects Version/s: 0.11
0.10.2
(was: 0.12)
> Replication can lose checkpoints
> --------------------------------
>
> Key: COUCHDB-704
> URL: https://issues.apache.org/jira/browse/COUCHDB-704
> Project: CouchDB
> Issue Type: Bug
> Components: Replication
> Affects Versions: 0.10.2, 0.11
> Reporter: Randall Leeds
> Priority: Minor
> Attachments: save-all-rep-checkpoints.patch, whitespace.patch
>
> Original Estimate: 0h
> Remaining Estimate: 0h
>
> When saving replication checkpoints in the _local/<repid> document the new entry is always pushed onto the _original_ "history" list property that existed at the start of the replication. When any number of things causes the checkpoint to be written to only one of the databases the head of the history list gets out of sync. Subsequent attempts to start this replication must start from the latest common replication log entry in the _original_ history, as though this replication never occurred.
> A better idea is to push every checkpoint onto the history instead of replacing the head on each save.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-704) Replication can lose checkpoints
Posted by "Filipe Manana (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12921681#action_12921681 ]
Filipe Manana commented on COUCHDB-704:
---------------------------------------
Randall,
I don't see nothing wrong with the patch, but whenever the replicator can't save a checkpoint, it retries (for remote DBs) and stops.
So I don't see exactly how you can loose 3 months of replication checkpoints.
Nevertheless, I would like to know Adam's point of view as well.
> Replication can lose checkpoints
> --------------------------------
>
> Key: COUCHDB-704
> URL: https://issues.apache.org/jira/browse/COUCHDB-704
> Project: CouchDB
> Issue Type: Bug
> Components: Replication
> Affects Versions: 0.11.2, 1.0.1
> Reporter: Randall Leeds
> Priority: Minor
> Attachments: keep_session_id.patch, save-all-rep-checkpoints.patch, whitespace.patch
>
> Original Estimate: 0h
> Remaining Estimate: 0h
>
> When saving replication checkpoints in the _local/<repid> document the new entry is always pushed onto the _original_ "history" list property that existed at the start of the replication. When any number of things causes the checkpoint to be written to only one of the databases the head of the history list gets out of sync. Subsequent attempts to start this replication must start from the latest common replication log entry in the _original_ history, as though this replication never occurred.
> A better idea is to push every checkpoint onto the history instead of replacing the head on each save.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (COUCHDB-704) Replication can lose checkpoints
Posted by "Adam Kocoloski (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Adam Kocoloski resolved COUCHDB-704.
------------------------------------
Resolution: Fixed
Applied to trunk and 1.0.x branch
> Replication can lose checkpoints
> --------------------------------
>
> Key: COUCHDB-704
> URL: https://issues.apache.org/jira/browse/COUCHDB-704
> Project: CouchDB
> Issue Type: Bug
> Components: Replication
> Affects Versions: 0.11.2, 1.0.1
> Reporter: Randall Leeds
> Priority: Minor
> Fix For: 1.0.2, 1.1
>
> Attachments: keep_session_id.patch, save-all-rep-checkpoints.patch, whitespace.patch
>
> Original Estimate: 0h
> Remaining Estimate: 0h
>
> When saving replication checkpoints in the _local/<repid> document the new entry is always pushed onto the _original_ "history" list property that existed at the start of the replication. When any number of things causes the checkpoint to be written to only one of the databases the head of the history list gets out of sync. Subsequent attempts to start this replication must start from the latest common replication log entry in the _original_ history, as though this replication never occurred.
> A better idea is to push every checkpoint onto the history instead of replacing the head on each save.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (COUCHDB-704) Replication can lose checkpoints
Posted by "Adam Kocoloski (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Adam Kocoloski updated COUCHDB-704:
-----------------------------------
Fix Version/s: 1.1
1.0.2
> Replication can lose checkpoints
> --------------------------------
>
> Key: COUCHDB-704
> URL: https://issues.apache.org/jira/browse/COUCHDB-704
> Project: CouchDB
> Issue Type: Bug
> Components: Replication
> Affects Versions: 0.11.2, 1.0.1
> Reporter: Randall Leeds
> Priority: Minor
> Fix For: 1.0.2, 1.1
>
> Attachments: keep_session_id.patch, save-all-rep-checkpoints.patch, whitespace.patch
>
> Original Estimate: 0h
> Remaining Estimate: 0h
>
> When saving replication checkpoints in the _local/<repid> document the new entry is always pushed onto the _original_ "history" list property that existed at the start of the replication. When any number of things causes the checkpoint to be written to only one of the databases the head of the history list gets out of sync. Subsequent attempts to start this replication must start from the latest common replication log entry in the _original_ history, as though this replication never occurred.
> A better idea is to push every checkpoint onto the history instead of replacing the head on each save.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (COUCHDB-704) Replication can lose checkpoints
Posted by "Randall Leeds (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Randall Leeds updated COUCHDB-704:
----------------------------------
Attachment: save-all-rep-checkpoints.patch
Here's an updated patch that applies to current trunk.
Differences to the previous patch:
I bumped the number of checkpoints saved back up to 50 because who doesn't love a little paranoia?
No longer restart on conflict. If this is a self-replication we want to bail. If it's not, let the user restart it.
Don't keep the stats through a restart. Not sure why I thought that was wise. The incremental stats are stored in each log entry.
Just trying to keep this fresh. If you see any reason this shouldn't be applied please let me know.
> Replication can lose checkpoints
> --------------------------------
>
> Key: COUCHDB-704
> URL: https://issues.apache.org/jira/browse/COUCHDB-704
> Project: CouchDB
> Issue Type: Bug
> Components: Replication
> Affects Versions: 0.12
> Reporter: Randall Leeds
> Priority: Minor
> Attachments: save-all-rep-checkpoints.patch
>
> Original Estimate: 0h
> Remaining Estimate: 0h
>
> When saving replication checkpoints in the _local/<repid> document the new entry is always pushed onto the _original_ "history" list property that existed at the start of the replication. When any number of things causes the checkpoint to be written to only one of the databases the head of the history list gets out of sync. Subsequent attempts to start this replication must start from the latest common replication log entry in the _original_ history, as though this replication never occurred.
> A better idea is to push every checkpoint onto the history instead of replacing the head on each save.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-704) Replication can lose checkpoints
Posted by "Randall Leeds (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12922652#action_12922652 ]
Randall Leeds commented on COUCHDB-704:
---------------------------------------
:)
I uploaded the keep_session_ids patch when I realized it was the smarter alternative. I didn't delete the old ones because I wasn't sure what policy was about keeping those kinds of records around on JIRA.
Please feel free to only commit the session_id patch (and the whitespace if still applicable). Thanks for the review!
> Replication can lose checkpoints
> --------------------------------
>
> Key: COUCHDB-704
> URL: https://issues.apache.org/jira/browse/COUCHDB-704
> Project: CouchDB
> Issue Type: Bug
> Components: Replication
> Affects Versions: 0.11.2, 1.0.1
> Reporter: Randall Leeds
> Priority: Minor
> Attachments: keep_session_id.patch, save-all-rep-checkpoints.patch, whitespace.patch
>
> Original Estimate: 0h
> Remaining Estimate: 0h
>
> When saving replication checkpoints in the _local/<repid> document the new entry is always pushed onto the _original_ "history" list property that existed at the start of the replication. When any number of things causes the checkpoint to be written to only one of the databases the head of the history list gets out of sync. Subsequent attempts to start this replication must start from the latest common replication log entry in the _original_ history, as though this replication never occurred.
> A better idea is to push every checkpoint onto the history instead of replacing the head on each save.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (COUCHDB-704) Replication can lose checkpoints
Posted by "Randall Leeds (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Randall Leeds updated COUCHDB-704:
----------------------------------
Attachment: keep_session_id.patch
Here's my new, preferred solution. It keeps the session_id for a replication in the replicator's state. This way, a single replication session still shows up as only one entry in the history log but if only one log gets written during some checkpoint pass the compare_rep_history will still find an older checkpoint if it exists.
It's not even a problem if the recorded_seq doesn't match. Either one will work since the checkpoint isn't written until a full commit has been performed on both source and target dbs. So as long as the session id matches, starting from whichever recorded_seq should be fine. The important thing is that we find a session_id in the log that matches so we don't lose lots of checkpoint progress.
> Replication can lose checkpoints
> --------------------------------
>
> Key: COUCHDB-704
> URL: https://issues.apache.org/jira/browse/COUCHDB-704
> Project: CouchDB
> Issue Type: Bug
> Components: Replication
> Affects Versions: 0.11.2, 1.0.1
> Reporter: Randall Leeds
> Priority: Minor
> Attachments: keep_session_id.patch, save-all-rep-checkpoints.patch, whitespace.patch
>
> Original Estimate: 0h
> Remaining Estimate: 0h
>
> When saving replication checkpoints in the _local/<repid> document the new entry is always pushed onto the _original_ "history" list property that existed at the start of the replication. When any number of things causes the checkpoint to be written to only one of the databases the head of the history list gets out of sync. Subsequent attempts to start this replication must start from the latest common replication log entry in the _original_ history, as though this replication never occurred.
> A better idea is to push every checkpoint onto the history instead of replacing the head on each save.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (COUCHDB-704) Replication can lose checkpoints
Posted by "Randall Leeds (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Randall Leeds updated COUCHDB-704:
----------------------------------
Affects Version/s: 1.0.1
0.11.2
(was: 0.11)
(was: 0.10.2)
Bumping this bug because I think it can be a little dangerous and sneaky and the fix is easy.
> Replication can lose checkpoints
> --------------------------------
>
> Key: COUCHDB-704
> URL: https://issues.apache.org/jira/browse/COUCHDB-704
> Project: CouchDB
> Issue Type: Bug
> Components: Replication
> Affects Versions: 0.11.2, 1.0.1
> Reporter: Randall Leeds
> Priority: Minor
> Attachments: save-all-rep-checkpoints.patch, whitespace.patch
>
> Original Estimate: 0h
> Remaining Estimate: 0h
>
> When saving replication checkpoints in the _local/<repid> document the new entry is always pushed onto the _original_ "history" list property that existed at the start of the replication. When any number of things causes the checkpoint to be written to only one of the databases the head of the history list gets out of sync. Subsequent attempts to start this replication must start from the latest common replication log entry in the _original_ history, as though this replication never occurred.
> A better idea is to push every checkpoint onto the history instead of replacing the head on each save.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (COUCHDB-704) Replication can lose checkpoints
Posted by "Randall Leeds (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Randall Leeds updated COUCHDB-704:
----------------------------------
Attachment: rep-history-update-per-checkpoint.patch
And the patch.
I lowered the replication history cap to 5. 50 is rubbish in my opinion. Who needs to know the last 50 checkpoints? No one (I hope).
> Replication can lose checkpoints
> --------------------------------
>
> Key: COUCHDB-704
> URL: https://issues.apache.org/jira/browse/COUCHDB-704
> Project: CouchDB
> Issue Type: Bug
> Components: Replication
> Affects Versions: 0.12
> Reporter: Randall Leeds
> Priority: Minor
> Attachments: rep-history-update-per-checkpoint.patch
>
> Original Estimate: 0h
> Remaining Estimate: 0h
>
> When saving replication checkpoints in the _local/<repid> document the new entry is always pushed onto the _original_ "history" list property that existed at the start of the replication. When any number of things causes the checkpoint to be written to only one of the databases the head of the history list gets out of sync. Subsequent attempts to start this replication must start from the latest common replication log entry in the _original_ history, as though this replication never occurred.
> A better idea is to push every checkpoint onto the history instead of replacing the head on each save.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (COUCHDB-704) Replication can lose checkpoints
Posted by "Paul Joseph Davis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Paul Joseph Davis updated COUCHDB-704:
--------------------------------------
Skill Level: Committers Level (Medium to Hard)
> Replication can lose checkpoints
> --------------------------------
>
> Key: COUCHDB-704
> URL: https://issues.apache.org/jira/browse/COUCHDB-704
> Project: CouchDB
> Issue Type: Bug
> Components: Replication
> Affects Versions: 0.11.2, 1.0.1
> Reporter: Randall Leeds
> Priority: Minor
> Attachments: keep_session_id.patch, save-all-rep-checkpoints.patch, whitespace.patch
>
> Original Estimate: 0h
> Remaining Estimate: 0h
>
> When saving replication checkpoints in the _local/<repid> document the new entry is always pushed onto the _original_ "history" list property that existed at the start of the replication. When any number of things causes the checkpoint to be written to only one of the databases the head of the history list gets out of sync. Subsequent attempts to start this replication must start from the latest common replication log entry in the _original_ history, as though this replication never occurred.
> A better idea is to push every checkpoint onto the history instead of replacing the head on each save.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-704) Replication can lose checkpoints
Posted by "Randall Leeds (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12920469#action_12920469 ]
Randall Leeds commented on COUCHDB-704:
---------------------------------------
Can I get someone to take a quick look at this, please?
I don't see any reason not to commit this to 1.0.x and trunk and close it, and very good reason to do so.
3 insertions and 1 deletion. Easy review. Get it while it's hot (well, it's already a month and a half cold)!
Summary:
The bug - the replication log is updated by changing the last entry in place with the contents of each checkpoint. This is fine except when nasty network errors cause the log to be written on only one of the two dbs involved. If this occurs then the last history entry will not have a matching session_id in the other log. Imagine 3 months of replication checkpoints lost because a switch flapped. Ouch.
The change - replication keeps the same session_id across checkpoints. Even if only one log is written, the last entries will still have a matching session_id and we can be sure that the recorded_seq is committed to both. At most one checkpoint is lost.
> Replication can lose checkpoints
> --------------------------------
>
> Key: COUCHDB-704
> URL: https://issues.apache.org/jira/browse/COUCHDB-704
> Project: CouchDB
> Issue Type: Bug
> Components: Replication
> Affects Versions: 0.11.2, 1.0.1
> Reporter: Randall Leeds
> Priority: Minor
> Attachments: keep_session_id.patch, save-all-rep-checkpoints.patch, whitespace.patch
>
> Original Estimate: 0h
> Remaining Estimate: 0h
>
> When saving replication checkpoints in the _local/<repid> document the new entry is always pushed onto the _original_ "history" list property that existed at the start of the replication. When any number of things causes the checkpoint to be written to only one of the databases the head of the history list gets out of sync. Subsequent attempts to start this replication must start from the latest common replication log entry in the _original_ history, as though this replication never occurred.
> A better idea is to push every checkpoint onto the history instead of replacing the head on each save.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (COUCHDB-704) Replication can lose checkpoints
Posted by "Randall Leeds (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Randall Leeds updated COUCHDB-704:
----------------------------------
Attachment: (was: rep-history-update-per-checkpoint.patch)
> Replication can lose checkpoints
> --------------------------------
>
> Key: COUCHDB-704
> URL: https://issues.apache.org/jira/browse/COUCHDB-704
> Project: CouchDB
> Issue Type: Bug
> Components: Replication
> Affects Versions: 0.12
> Reporter: Randall Leeds
> Priority: Minor
> Attachments: save-all-rep-checkpoints.patch
>
> Original Estimate: 0h
> Remaining Estimate: 0h
>
> When saving replication checkpoints in the _local/<repid> document the new entry is always pushed onto the _original_ "history" list property that existed at the start of the replication. When any number of things causes the checkpoint to be written to only one of the databases the head of the history list gets out of sync. Subsequent attempts to start this replication must start from the latest common replication log entry in the _original_ history, as though this replication never occurred.
> A better idea is to push every checkpoint onto the history instead of replacing the head on each save.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-704) Replication can lose checkpoints
Posted by "Adam Kocoloski (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12922557#action_12922557 ]
Adam Kocoloski commented on COUCHDB-704:
----------------------------------------
I think it's a _very_ good idea to reuse the session_id over the lifetime of a replication job. I've been burned by this in the past as well. The only thing I take issue with in this patch is the change to push new history entries for every checkpoint. For a long-running replication that means the checkpoint document will cover the last 5 minutes of the current replication and nothing else. I've occasionally found it useful to scan back through the history to see the timestamps and statistics from older sessions for a particular replication, and I don't think saving multiple entries for a given session_id adds anything to the durability of that session.
What do you think about switching back to using #state.history for the OldHistory when saving a checkpoint?
> Replication can lose checkpoints
> --------------------------------
>
> Key: COUCHDB-704
> URL: https://issues.apache.org/jira/browse/COUCHDB-704
> Project: CouchDB
> Issue Type: Bug
> Components: Replication
> Affects Versions: 0.11.2, 1.0.1
> Reporter: Randall Leeds
> Priority: Minor
> Attachments: keep_session_id.patch, save-all-rep-checkpoints.patch, whitespace.patch
>
> Original Estimate: 0h
> Remaining Estimate: 0h
>
> When saving replication checkpoints in the _local/<repid> document the new entry is always pushed onto the _original_ "history" list property that existed at the start of the replication. When any number of things causes the checkpoint to be written to only one of the databases the head of the history list gets out of sync. Subsequent attempts to start this replication must start from the latest common replication log entry in the _original_ history, as though this replication never occurred.
> A better idea is to push every checkpoint onto the history instead of replacing the head on each save.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.