You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@couchdb.apache.org by "Maya Alipin (JIRA)" <ji...@apache.org> on 2012/06/13 08:32:43 UTC
[jira] [Created] (COUCHDB-1496) Incomplete Replication on Several
Circumstances
Maya Alipin created COUCHDB-1496:
------------------------------------
Summary: Incomplete Replication on Several Circumstances
Key: COUCHDB-1496
URL: https://issues.apache.org/jira/browse/COUCHDB-1496
Project: CouchDB
Issue Type: Bug
Affects Versions: 1.2
Environment: iriscouch
Reporter: Maya Alipin
Priority: Minor
I have found that on several circumstances, CouchDB's continous filtered replication doesn't do what I expect.
1. deleted documents in source database ends up not deleted in destination database
2. I notice on continuous replication, the replication status will remain incomplete until check-pointed sequence equals source sequence. On continuous filtered replication, an update may not pass the filter and cause the replication to become incomplete.
I found the replication remains incomplete even if there is an update that pass the filter. I need to restart the replication to get the update replicated to the destination database.
Test Scenario:
1. Trigger a filtered continuous replication from database X to database Y on _replicator database. Replication progress is 100%
2. Add new document(A1) on database X.
3. Document A1 pass the replication filter, thus A1 is replicated to database Y. Replication progress is 100%
4. Delete document A1 on database X. Replication progress become 99%, document A1 still exists on database Y.
6. Restart the filtered continuous replication from database X to database Y by deleting document on _replicator database and create it again
7. Replication progress stops at 99%.
8. Add new document(A2) on database X.
9. Document A2 pass the replication filter, thus A2 is replicated to database Y. Replication progress become 100%
10. Delete document A2 on database X. Replication progress become 99%, document A2 still exists on database Y.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1496) Incomplete Replication on Several
Circumstances
Posted by "Maya Alipin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-1496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13294247#comment-13294247 ]
Maya Alipin commented on COUCHDB-1496:
--------------------------------------
The replication syntax
curl -H 'Content-Type: application/json' -vX POST http://[username]:[password]@mayajs.iriscouch.com/_replicator -d ' {"source":"http://[username]:[password]@mayajs.iriscouch.com/jst","continuous":true,"target":"http://[username]:[password]@mayajs.iriscouch.com/mdr","filter":"timesheet/masterToProject","query_params":{"pid":"timesheets.project:4"}}'
> Incomplete Replication on Several Circumstances
> -----------------------------------------------
>
> Key: COUCHDB-1496
> URL: https://issues.apache.org/jira/browse/COUCHDB-1496
> Project: CouchDB
> Issue Type: Bug
> Affects Versions: 1.2
> Environment: iriscouch
> Reporter: Maya Alipin
> Priority: Minor
>
> I have found that on several circumstances, CouchDB's continous filtered replication doesn't do what I expect.
> 1. deleted documents in source database ends up not deleted in destination database
> 2. I notice on continuous replication, the replication status will remain incomplete until check-pointed sequence equals source sequence. On continuous filtered replication, an update may not pass the filter and cause the replication to become incomplete.
> I found the replication remains incomplete even if there is an update that pass the filter. I need to restart the replication to get the update replicated to the destination database.
> Test Scenario:
> 1. Trigger a filtered continuous replication from database X to database Y on _replicator database. Replication progress is 100%
> 2. Add new document(A1) on database X.
> 3. Document A1 pass the replication filter, thus A1 is replicated to database Y. Replication progress is 100%
> 4. Delete document A1 on database X. Replication progress become 99%, document A1 still exists on database Y.
> 6. Restart the filtered continuous replication from database X to database Y by deleting document on _replicator database and create it again
> 7. Replication progress stops at 99%.
> 8. Add new document(A2) on database X.
> 9. Document A2 pass the replication filter, thus A2 is replicated to database Y. Replication progress become 100%
> 10. Delete document A2 on database X. Replication progress become 99%, document A2 still exists on database Y.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1496) Incomplete Replication on Several
Circumstances
Posted by "Maya Alipin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-1496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13294242#comment-13294242 ]
Maya Alipin commented on COUCHDB-1496:
--------------------------------------
Hi Dave, Thanks for the reply.
> CouchDB effectively tombstones deleted docs by replacing the content with {"_id": myid", "_deleted": true}
Sorry I don't quite understand the above statement.
I know that CouchDB marks deleted document with _deleted = true. But I don't know that CouchDB also changes the _id of the deleted document.
Here is the sample document:
{
"_id": "fd906375bd77a7fa63b248766d012008",
"_rev": "3-4265ec72422ced8030b19bd441ef6371",
"type": [
"timesheets",
"timesheet"
],
"tasktype": "Testing",
"project": {
"_id": "timesheets.project:4",
"_rev": "1-097a42351376d974002ed8892275eb61",
"type": [
"projects",
"project"
],
"id": 4
},
"workdate": "2012-03-05 08:30:00",
"description": "#4955 Test Comparison, Position Tool and Charts.",
"charge": false,
"workhour": 2,
"employee": {
"_id": "timesheets.employee:maya",
"_rev": "1-52f74183f29b7f2c8a9fff5b919cdc03",
"name": "maya",
"type": [
"timesheets",
"employee"
]
},
"hourlyrate": 10,
"chargedfee": 0
}
And the filter:
function(doc, req)
{
if (doc._id.indexOf("_design") == 0 ){
return true;
}
var pid = req.query.pid;
if(doc.type && doc.type.length == 2) {
if ((doc.type[0] == "timesheets" && doc.type[1] == "setting") || (doc.type[0] == "timesheets" && doc.type[1] == "employee")
|| (doc.type[0] == "timesheets" && doc.type[1] == "tasktype")){
return true;
}
if (doc.type[0] == "projects" && doc.type[1] == "project" && doc._id === pid){
return true;
}
if(doc.type[0] == "timesheets" && doc.type[1] == "timesheet" && doc.project && doc.project._id === pid){
return true;
}
}
return false;
}
Is there any specific commands to include _deleted documents on filtered replication?
> Incomplete Replication on Several Circumstances
> -----------------------------------------------
>
> Key: COUCHDB-1496
> URL: https://issues.apache.org/jira/browse/COUCHDB-1496
> Project: CouchDB
> Issue Type: Bug
> Affects Versions: 1.2
> Environment: iriscouch
> Reporter: Maya Alipin
> Priority: Minor
>
> I have found that on several circumstances, CouchDB's continous filtered replication doesn't do what I expect.
> 1. deleted documents in source database ends up not deleted in destination database
> 2. I notice on continuous replication, the replication status will remain incomplete until check-pointed sequence equals source sequence. On continuous filtered replication, an update may not pass the filter and cause the replication to become incomplete.
> I found the replication remains incomplete even if there is an update that pass the filter. I need to restart the replication to get the update replicated to the destination database.
> Test Scenario:
> 1. Trigger a filtered continuous replication from database X to database Y on _replicator database. Replication progress is 100%
> 2. Add new document(A1) on database X.
> 3. Document A1 pass the replication filter, thus A1 is replicated to database Y. Replication progress is 100%
> 4. Delete document A1 on database X. Replication progress become 99%, document A1 still exists on database Y.
> 6. Restart the filtered continuous replication from database X to database Y by deleting document on _replicator database and create it again
> 7. Replication progress stops at 99%.
> 8. Add new document(A2) on database X.
> 9. Document A2 pass the replication filter, thus A2 is replicated to database Y. Replication progress become 100%
> 10. Delete document A2 on database X. Replication progress become 99%, document A2 still exists on database Y.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1496) Incomplete Replication on Several
Circumstances
Posted by "Dave Cottlehuber (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-1496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13294221#comment-13294221 ]
Dave Cottlehuber commented on COUCHDB-1496:
-------------------------------------------
Thanks for logging this issue Maya. A common issue is that the replication filter does not pass _deleted documents; CouchDB
effectively tombstones deleted docs by replacing the content with {"_id": myid", "_deleted": true} so you need to ensure these
pass through the filter even if the original doc does not.
Can you please attach a sample document, the replication filter you are using, and also the replication syntax you're using?
> Incomplete Replication on Several Circumstances
> -----------------------------------------------
>
> Key: COUCHDB-1496
> URL: https://issues.apache.org/jira/browse/COUCHDB-1496
> Project: CouchDB
> Issue Type: Bug
> Affects Versions: 1.2
> Environment: iriscouch
> Reporter: Maya Alipin
> Priority: Minor
>
> I have found that on several circumstances, CouchDB's continous filtered replication doesn't do what I expect.
> 1. deleted documents in source database ends up not deleted in destination database
> 2. I notice on continuous replication, the replication status will remain incomplete until check-pointed sequence equals source sequence. On continuous filtered replication, an update may not pass the filter and cause the replication to become incomplete.
> I found the replication remains incomplete even if there is an update that pass the filter. I need to restart the replication to get the update replicated to the destination database.
> Test Scenario:
> 1. Trigger a filtered continuous replication from database X to database Y on _replicator database. Replication progress is 100%
> 2. Add new document(A1) on database X.
> 3. Document A1 pass the replication filter, thus A1 is replicated to database Y. Replication progress is 100%
> 4. Delete document A1 on database X. Replication progress become 99%, document A1 still exists on database Y.
> 6. Restart the filtered continuous replication from database X to database Y by deleting document on _replicator database and create it again
> 7. Replication progress stops at 99%.
> 8. Add new document(A2) on database X.
> 9. Document A2 pass the replication filter, thus A2 is replicated to database Y. Replication progress become 100%
> 10. Delete document A2 on database X. Replication progress become 99%, document A2 still exists on database Y.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1496) Incomplete Replication on Several
Circumstances
Posted by "James Howe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-1496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13294275#comment-13294275 ]
James Howe commented on COUCHDB-1496:
-------------------------------------
When the document is deleted, all other properties are removed, so the _deleted revisions are not passing "if (doc.type)".
You need to either not delete documents with DELETE, or add "if (doc._deleted) { return true; }" to the filter.
> Incomplete Replication on Several Circumstances
> -----------------------------------------------
>
> Key: COUCHDB-1496
> URL: https://issues.apache.org/jira/browse/COUCHDB-1496
> Project: CouchDB
> Issue Type: Bug
> Affects Versions: 1.2
> Environment: iriscouch
> Reporter: Maya Alipin
> Priority: Minor
>
> I have found that on several circumstances, CouchDB's continous filtered replication doesn't do what I expect.
> 1. deleted documents in source database ends up not deleted in destination database
> 2. I notice on continuous replication, the replication status will remain incomplete until check-pointed sequence equals source sequence. On continuous filtered replication, an update may not pass the filter and cause the replication to become incomplete.
> I found the replication remains incomplete even if there is an update that pass the filter. I need to restart the replication to get the update replicated to the destination database.
> Test Scenario:
> 1. Trigger a filtered continuous replication from database X to database Y on _replicator database. Replication progress is 100%
> 2. Add new document(A1) on database X.
> 3. Document A1 pass the replication filter, thus A1 is replicated to database Y. Replication progress is 100%
> 4. Delete document A1 on database X. Replication progress become 99%, document A1 still exists on database Y.
> 6. Restart the filtered continuous replication from database X to database Y by deleting document on _replicator database and create it again
> 7. Replication progress stops at 99%.
> 8. Add new document(A2) on database X.
> 9. Document A2 pass the replication filter, thus A2 is replicated to database Y. Replication progress become 100%
> 10. Delete document A2 on database X. Replication progress become 99%, document A2 still exists on database Y.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1496) Incomplete Replication on Several
Circumstances
Posted by "Maya Alipin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-1496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13294384#comment-13294384 ]
Maya Alipin commented on COUCHDB-1496:
--------------------------------------
Hi James,
Thank you for the explanation. That means point #1 is not a bug.
What about the replication on step 6 that stops at 99%?
> Incomplete Replication on Several Circumstances
> -----------------------------------------------
>
> Key: COUCHDB-1496
> URL: https://issues.apache.org/jira/browse/COUCHDB-1496
> Project: CouchDB
> Issue Type: Bug
> Affects Versions: 1.2
> Environment: iriscouch
> Reporter: Maya Alipin
> Priority: Minor
>
> I have found that on several circumstances, CouchDB's continous filtered replication doesn't do what I expect.
> 1. deleted documents in source database ends up not deleted in destination database
> 2. I notice on continuous replication, the replication status will remain incomplete until check-pointed sequence equals source sequence. On continuous filtered replication, an update may not pass the filter and cause the replication to become incomplete.
> I found the replication remains incomplete even if there is an update that pass the filter. I need to restart the replication to get the update replicated to the destination database.
> Test Scenario:
> 1. Trigger a filtered continuous replication from database X to database Y on _replicator database. Replication progress is 100%
> 2. Add new document(A1) on database X.
> 3. Document A1 pass the replication filter, thus A1 is replicated to database Y. Replication progress is 100%
> 4. Delete document A1 on database X. Replication progress become 99%, document A1 still exists on database Y.
> 6. Restart the filtered continuous replication from database X to database Y by deleting document on _replicator database and create it again
> 7. Replication progress stops at 99%.
> 8. Add new document(A2) on database X.
> 9. Document A2 pass the replication filter, thus A2 is replicated to database Y. Replication progress become 100%
> 10. Delete document A2 on database X. Replication progress become 99%, document A2 still exists on database Y.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira