You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Brandon Williams (JIRA)" <ji...@apache.org> on 2010/07/25 01:51:52 UTC
[jira] Created: (CASSANDRA-1316) Read repair does not always work
correctly
Read repair does not always work correctly
------------------------------------------
Key: CASSANDRA-1316
URL: https://issues.apache.org/jira/browse/CASSANDRA-1316
Project: Cassandra
Issue Type: Bug
Affects Versions: 0.6.3
Reporter: Brandon Williams
Fix For: 0.6.4
Attachments: cassandra-1.json, cassandra-2.json, cassandra-3.json
Read repair does not always work. At the least, we allow violation of the CL.ALL contract. To reproduce, create a three node cluster with RF=3, and json2sstable one of the attached json files on each node. This creates a row whose key is 'test' with 9 columns, but only 3 columns are on each machine. If you get_count this row in quick succession at CL.ALL, sometimes you will receive a count of 6, sometimes 9. After the ReadRepairManager has sent the repairs, you will always get 9, which is the desired behavior.
I have another data set obtained in the wild which never fully repairs for some reason, but it's a bit large to attach (600ish columns per machine.) I'm still trying to figure out why RR isn't working on this set, but I always get different results when reading at any CL including ALL, no matter how long I wait or how many reads I do.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1316) Read repair does not always work
correctly
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-1316:
--------------------------------------
Fix Version/s: 0.6.5
(was: 0.6.4)
> Read repair does not always work correctly
> ------------------------------------------
>
> Key: CASSANDRA-1316
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1316
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.4
> Reporter: Brandon Williams
> Fix For: 0.6.5
>
> Attachments: 001_correct_responsecount_in_RRR.txt, cassandra-1.json, cassandra-2.json, cassandra-3.json, RRR-v2.txt
>
>
> Read repair does not always work. At the least, we allow violation of the CL.ALL contract. To reproduce, create a three node cluster with RF=3, and json2sstable one of the attached json files on each node. This creates a row whose key is 'test' with 9 columns, but only 3 columns are on each machine. If you get_count this row in quick succession at CL.ALL, sometimes you will receive a count of 6, sometimes 9. After the ReadRepairManager has sent the repairs, you will always get 9, which is the desired behavior.
> I have another data set obtained in the wild which never fully repairs for some reason, but it's a bit large to attach (600ish columns per machine.) I'm still trying to figure out why RR isn't working on this set, but I always get different results when reading at any CL including ALL, no matter how long I wait or how many reads I do.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1316) Read repair does not always work
correctly
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams updated CASSANDRA-1316:
----------------------------------------
Attachment: (was: cassandra-1.json)
> Read repair does not always work correctly
> ------------------------------------------
>
> Key: CASSANDRA-1316
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1316
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.4
> Reporter: Brandon Williams
> Fix For: 0.6.4
>
> Attachments: cassandra-1.json, cassandra-2.json, cassandra-3.json
>
>
> Read repair does not always work. At the least, we allow violation of the CL.ALL contract. To reproduce, create a three node cluster with RF=3, and json2sstable one of the attached json files on each node. This creates a row whose key is 'test' with 9 columns, but only 3 columns are on each machine. If you get_count this row in quick succession at CL.ALL, sometimes you will receive a count of 6, sometimes 9. After the ReadRepairManager has sent the repairs, you will always get 9, which is the desired behavior.
> I have another data set obtained in the wild which never fully repairs for some reason, but it's a bit large to attach (600ish columns per machine.) I'm still trying to figure out why RR isn't working on this set, but I always get different results when reading at any CL including ALL, no matter how long I wait or how many reads I do.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1316) Read repair does not always work
correctly
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams updated CASSANDRA-1316:
----------------------------------------
Attachment: cassandra-1.json
cassandra-2.json
cassandra-3.json
> Read repair does not always work correctly
> ------------------------------------------
>
> Key: CASSANDRA-1316
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1316
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.6.3
> Reporter: Brandon Williams
> Fix For: 0.6.4
>
> Attachments: cassandra-1.json, cassandra-2.json, cassandra-3.json
>
>
> Read repair does not always work. At the least, we allow violation of the CL.ALL contract. To reproduce, create a three node cluster with RF=3, and json2sstable one of the attached json files on each node. This creates a row whose key is 'test' with 9 columns, but only 3 columns are on each machine. If you get_count this row in quick succession at CL.ALL, sometimes you will receive a count of 6, sometimes 9. After the ReadRepairManager has sent the repairs, you will always get 9, which is the desired behavior.
> I have another data set obtained in the wild which never fully repairs for some reason, but it's a bit large to attach (600ish columns per machine.) I'm still trying to figure out why RR isn't working on this set, but I always get different results when reading at any CL including ALL, no matter how long I wait or how many reads I do.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1316) Read repair does not always work
correctly
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams updated CASSANDRA-1316:
----------------------------------------
Attachment: (was: cassandra-2.json)
> Read repair does not always work correctly
> ------------------------------------------
>
> Key: CASSANDRA-1316
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1316
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.4
> Reporter: Brandon Williams
> Fix For: 0.6.4
>
> Attachments: cassandra-1.json, cassandra-2.json, cassandra-3.json
>
>
> Read repair does not always work. At the least, we allow violation of the CL.ALL contract. To reproduce, create a three node cluster with RF=3, and json2sstable one of the attached json files on each node. This creates a row whose key is 'test' with 9 columns, but only 3 columns are on each machine. If you get_count this row in quick succession at CL.ALL, sometimes you will receive a count of 6, sometimes 9. After the ReadRepairManager has sent the repairs, you will always get 9, which is the desired behavior.
> I have another data set obtained in the wild which never fully repairs for some reason, but it's a bit large to attach (600ish columns per machine.) I'm still trying to figure out why RR isn't working on this set, but I always get different results when reading at any CL including ALL, no matter how long I wait or how many reads I do.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1316) Read repair does not always work
correctly
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892049#action_12892049 ]
Brandon Williams commented on CASSANDRA-1316:
---------------------------------------------
It looks like the key difference between the real data and the toy data that is attached is that the real data has the key in multiple sstables. If left this way, RR never fully works, but if I force a compaction then it succeeds.
> Read repair does not always work correctly
> ------------------------------------------
>
> Key: CASSANDRA-1316
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1316
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.6.3
> Reporter: Brandon Williams
> Fix For: 0.6.4
>
> Attachments: cassandra-1.json, cassandra-2.json, cassandra-3.json
>
>
> Read repair does not always work. At the least, we allow violation of the CL.ALL contract. To reproduce, create a three node cluster with RF=3, and json2sstable one of the attached json files on each node. This creates a row whose key is 'test' with 9 columns, but only 3 columns are on each machine. If you get_count this row in quick succession at CL.ALL, sometimes you will receive a count of 6, sometimes 9. After the ReadRepairManager has sent the repairs, you will always get 9, which is the desired behavior.
> I have another data set obtained in the wild which never fully repairs for some reason, but it's a bit large to attach (600ish columns per machine.) I'm still trying to figure out why RR isn't working on this set, but I always get different results when reading at any CL including ALL, no matter how long I wait or how many reads I do.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1316) Read repair does not always work
correctly
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-1316:
--------------------------------------
Attachment: 1316-RRM.txt
patch that simplifies debugging by removing ReadRepairManager which is mostly 100-odd lines of obfuscation around MessagingService.sendOneWay. (backport from CASSANDRA-1077 which was applied to trunk two months ago)
> Read repair does not always work correctly
> ------------------------------------------
>
> Key: CASSANDRA-1316
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1316
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.4
> Reporter: Brandon Williams
> Fix For: 0.6.5
>
> Attachments: 001_correct_responsecount_in_RRR.txt, 1316-RRM.txt, cassandra-1.json, cassandra-2.json, cassandra-3.json, RRR-v2.txt
>
>
> Read repair does not always work. At the least, we allow violation of the CL.ALL contract. To reproduce, create a three node cluster with RF=3, and json2sstable one of the attached json files on each node. This creates a row whose key is 'test' with 9 columns, but only 3 columns are on each machine. If you get_count this row in quick succession at CL.ALL, sometimes you will receive a count of 6, sometimes 9. After the ReadRepairManager has sent the repairs, you will always get 9, which is the desired behavior.
> I have another data set obtained in the wild which never fully repairs for some reason, but it's a bit large to attach (600ish columns per machine.) I'm still trying to figure out why RR isn't working on this set, but I always get different results when reading at any CL including ALL, no matter how long I wait or how many reads I do.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1316) Read repair does not always work
correctly
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-1316:
--------------------------------------
Affects Version/s: 0.4
(was: 0.6.3)
Component/s: Core
> Read repair does not always work correctly
> ------------------------------------------
>
> Key: CASSANDRA-1316
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1316
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.4
> Reporter: Brandon Williams
> Fix For: 0.6.4
>
> Attachments: cassandra-1.json, cassandra-2.json, cassandra-3.json
>
>
> Read repair does not always work. At the least, we allow violation of the CL.ALL contract. To reproduce, create a three node cluster with RF=3, and json2sstable one of the attached json files on each node. This creates a row whose key is 'test' with 9 columns, but only 3 columns are on each machine. If you get_count this row in quick succession at CL.ALL, sometimes you will receive a count of 6, sometimes 9. After the ReadRepairManager has sent the repairs, you will always get 9, which is the desired behavior.
> I have another data set obtained in the wild which never fully repairs for some reason, but it's a bit large to attach (600ish columns per machine.) I'm still trying to figure out why RR isn't working on this set, but I always get different results when reading at any CL including ALL, no matter how long I wait or how many reads I do.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1316) Read repair does not always work
correctly
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892503#action_12892503 ]
Brandon Williams commented on CASSANDRA-1316:
---------------------------------------------
Committed RRRv2.
> Read repair does not always work correctly
> ------------------------------------------
>
> Key: CASSANDRA-1316
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1316
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.4
> Reporter: Brandon Williams
> Fix For: 0.6.5
>
> Attachments: 001_correct_responsecount_in_RRR.txt, cassandra-1.json, cassandra-2.json, cassandra-3.json, RRR-v2.txt
>
>
> Read repair does not always work. At the least, we allow violation of the CL.ALL contract. To reproduce, create a three node cluster with RF=3, and json2sstable one of the attached json files on each node. This creates a row whose key is 'test' with 9 columns, but only 3 columns are on each machine. If you get_count this row in quick succession at CL.ALL, sometimes you will receive a count of 6, sometimes 9. After the ReadRepairManager has sent the repairs, you will always get 9, which is the desired behavior.
> I have another data set obtained in the wild which never fully repairs for some reason, but it's a bit large to attach (600ish columns per machine.) I'm still trying to figure out why RR isn't working on this set, but I always get different results when reading at any CL including ALL, no matter how long I wait or how many reads I do.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1316) Read repair does not always work
correctly
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-1316:
--------------------------------------
Attachment: RRR-v2.txt
v2 updates QRH arguments to use responseCount as well, even though it's ignored
> Read repair does not always work correctly
> ------------------------------------------
>
> Key: CASSANDRA-1316
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1316
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.4
> Reporter: Brandon Williams
> Fix For: 0.6.4
>
> Attachments: 001_correct_responsecount_in_RRR.txt, cassandra-1.json, cassandra-2.json, cassandra-3.json, RRR-v2.txt
>
>
> Read repair does not always work. At the least, we allow violation of the CL.ALL contract. To reproduce, create a three node cluster with RF=3, and json2sstable one of the attached json files on each node. This creates a row whose key is 'test' with 9 columns, but only 3 columns are on each machine. If you get_count this row in quick succession at CL.ALL, sometimes you will receive a count of 6, sometimes 9. After the ReadRepairManager has sent the repairs, you will always get 9, which is the desired behavior.
> I have another data set obtained in the wild which never fully repairs for some reason, but it's a bit large to attach (600ish columns per machine.) I'm still trying to figure out why RR isn't working on this set, but I always get different results when reading at any CL including ALL, no matter how long I wait or how many reads I do.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1316) Read repair does not always work
correctly
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams updated CASSANDRA-1316:
----------------------------------------
Attachment: cassandra-1.json
cassandra-2.json
cassandra-3.json
> Read repair does not always work correctly
> ------------------------------------------
>
> Key: CASSANDRA-1316
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1316
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.4
> Reporter: Brandon Williams
> Fix For: 0.6.4
>
> Attachments: cassandra-1.json, cassandra-2.json, cassandra-3.json
>
>
> Read repair does not always work. At the least, we allow violation of the CL.ALL contract. To reproduce, create a three node cluster with RF=3, and json2sstable one of the attached json files on each node. This creates a row whose key is 'test' with 9 columns, but only 3 columns are on each machine. If you get_count this row in quick succession at CL.ALL, sometimes you will receive a count of 6, sometimes 9. After the ReadRepairManager has sent the repairs, you will always get 9, which is the desired behavior.
> I have another data set obtained in the wild which never fully repairs for some reason, but it's a bit large to attach (600ish columns per machine.) I'm still trying to figure out why RR isn't working on this set, but I always get different results when reading at any CL including ALL, no matter how long I wait or how many reads I do.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1316) Read repair does not always work
correctly
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892048#action_12892048 ]
Brandon Williams commented on CASSANDRA-1316:
---------------------------------------------
I tried to bisect this issue and went as far back as 0.4.2 without finding a successful version. I will see if the data that never repairs can be scrubbed so I can attach it to this ticket for debugging.
> Read repair does not always work correctly
> ------------------------------------------
>
> Key: CASSANDRA-1316
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1316
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.6.3
> Reporter: Brandon Williams
> Fix For: 0.6.4
>
> Attachments: cassandra-1.json, cassandra-2.json, cassandra-3.json
>
>
> Read repair does not always work. At the least, we allow violation of the CL.ALL contract. To reproduce, create a three node cluster with RF=3, and json2sstable one of the attached json files on each node. This creates a row whose key is 'test' with 9 columns, but only 3 columns are on each machine. If you get_count this row in quick succession at CL.ALL, sometimes you will receive a count of 6, sometimes 9. After the ReadRepairManager has sent the repairs, you will always get 9, which is the desired behavior.
> I have another data set obtained in the wild which never fully repairs for some reason, but it's a bit large to attach (600ish columns per machine.) I'm still trying to figure out why RR isn't working on this set, but I always get different results when reading at any CL including ALL, no matter how long I wait or how many reads I do.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1316) Read repair does not always work
correctly
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams updated CASSANDRA-1316:
----------------------------------------
Attachment: (was: cassandra-3.json)
> Read repair does not always work correctly
> ------------------------------------------
>
> Key: CASSANDRA-1316
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1316
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.4
> Reporter: Brandon Williams
> Fix For: 0.6.4
>
> Attachments: cassandra-1.json, cassandra-2.json, cassandra-3.json
>
>
> Read repair does not always work. At the least, we allow violation of the CL.ALL contract. To reproduce, create a three node cluster with RF=3, and json2sstable one of the attached json files on each node. This creates a row whose key is 'test' with 9 columns, but only 3 columns are on each machine. If you get_count this row in quick succession at CL.ALL, sometimes you will receive a count of 6, sometimes 9. After the ReadRepairManager has sent the repairs, you will always get 9, which is the desired behavior.
> I have another data set obtained in the wild which never fully repairs for some reason, but it's a bit large to attach (600ish columns per machine.) I'm still trying to figure out why RR isn't working on this set, but I always get different results when reading at any CL including ALL, no matter how long I wait or how many reads I do.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1316) Read repair does not always work
correctly
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12892142#action_12892142 ]
Brandon Williams commented on CASSANDRA-1316:
---------------------------------------------
Updated json files illustrate one possible scenario: nodes 1 and 2 have a column, and node 3 has the column tombstoned with the same timestamp. It looks like tombstones aren't taking precedence.
> Read repair does not always work correctly
> ------------------------------------------
>
> Key: CASSANDRA-1316
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1316
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.4
> Reporter: Brandon Williams
> Fix For: 0.6.4
>
> Attachments: cassandra-1.json, cassandra-2.json, cassandra-3.json
>
>
> Read repair does not always work. At the least, we allow violation of the CL.ALL contract. To reproduce, create a three node cluster with RF=3, and json2sstable one of the attached json files on each node. This creates a row whose key is 'test' with 9 columns, but only 3 columns are on each machine. If you get_count this row in quick succession at CL.ALL, sometimes you will receive a count of 6, sometimes 9. After the ReadRepairManager has sent the repairs, you will always get 9, which is the desired behavior.
> I have another data set obtained in the wild which never fully repairs for some reason, but it's a bit large to attach (600ish columns per machine.) I'm still trying to figure out why RR isn't working on this set, but I always get different results when reading at any CL including ALL, no matter how long I wait or how many reads I do.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (CASSANDRA-1316) Read repair does not always work
correctly
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis resolved CASSANDRA-1316.
---------------------------------------
Assignee: Brandon Williams
Fix Version/s: 0.6.4
(was: 0.6.5)
Resolution: Fixed
Brandon's first patch fixing reads at CL.ALL turns out to be the only bug. The rest is obscure-but-valid behavior when expired tombstones haven't been replicated across the cluster (i.e., the tombstones exist on some nodes, but not all). Let me give an example:
say node A has columns x and y, where x is an expired tombstone with timestamp T1, and node B has live column x, at time T2 where T2 < T1.
if you read at ALL you will see x from B and y from A. you will _not_ see x from A -- since it is expired, it is no longer relevant off-node. thus, the ALL read will send a repair of column x to A, since it was "missing."
But next time you read from A the tombstone will supress the newly-written copy of x-from-B still, because its timestamp is higher. So the replicas won't converge.
This is not a bug, because the design explicitly allows that behavior when tombstones expire before being propagated to all nodes; see http://wiki.apache.org/cassandra/DistributedDeletes. The best way to avoid this of course is to run repair frequently enough to ensure that tombstones are propagated within GCGraceSeconds of being written.
But if you do find yourself in this situation, you have two options to get things to converge again:
1) the simplest option is to simply perform a major compaction on each node, which will eliminate all expired tombstones.
2) but if you want to propagate as many of the tombstones as possible first, increase your GCGraceSeconds setting everywhere (requires rolling restart), and perform a full repair as described in http://wiki.apache.org/cassandra/Operations. After the repair is complete you can put GCGraceSeconds back to what it was.
> Read repair does not always work correctly
> ------------------------------------------
>
> Key: CASSANDRA-1316
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1316
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.4
> Reporter: Brandon Williams
> Assignee: Brandon Williams
> Fix For: 0.6.4
>
> Attachments: 001_correct_responsecount_in_RRR.txt, 1316-RRM.txt, cassandra-1.json, cassandra-2.json, cassandra-3.json, RRR-v2.txt
>
>
> Read repair does not always work. At the least, we allow violation of the CL.ALL contract. To reproduce, create a three node cluster with RF=3, and json2sstable one of the attached json files on each node. This creates a row whose key is 'test' with 9 columns, but only 3 columns are on each machine. If you get_count this row in quick succession at CL.ALL, sometimes you will receive a count of 6, sometimes 9. After the ReadRepairManager has sent the repairs, you will always get 9, which is the desired behavior.
> I have another data set obtained in the wild which never fully repairs for some reason, but it's a bit large to attach (600ish columns per machine.) I'm still trying to figure out why RR isn't working on this set, but I always get different results when reading at any CL including ALL, no matter how long I wait or how many reads I do.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1316) Read repair does not always work
correctly
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams updated CASSANDRA-1316:
----------------------------------------
Attachment: 001_correct_responsecount_in_RRR.txt
Patch to solve one problem: use derterminBlockFor to set the correct response count passed to RRR, so CL.ALL works.
> Read repair does not always work correctly
> ------------------------------------------
>
> Key: CASSANDRA-1316
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1316
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.4
> Reporter: Brandon Williams
> Fix For: 0.6.4
>
> Attachments: 001_correct_responsecount_in_RRR.txt, cassandra-1.json, cassandra-2.json, cassandra-3.json
>
>
> Read repair does not always work. At the least, we allow violation of the CL.ALL contract. To reproduce, create a three node cluster with RF=3, and json2sstable one of the attached json files on each node. This creates a row whose key is 'test' with 9 columns, but only 3 columns are on each machine. If you get_count this row in quick succession at CL.ALL, sometimes you will receive a count of 6, sometimes 9. After the ReadRepairManager has sent the repairs, you will always get 9, which is the desired behavior.
> I have another data set obtained in the wild which never fully repairs for some reason, but it's a bit large to attach (600ish columns per machine.) I'm still trying to figure out why RR isn't working on this set, but I always get different results when reading at any CL including ALL, no matter how long I wait or how many reads I do.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.