You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Peter Schuller (Issue Comment Edited) (JIRA)" <ji...@apache.org> on 2011/12/09 23:20:40 UTC
[jira] [Issue Comment Edited] (CASSANDRA-3070) counter repair
[ https://issues.apache.org/jira/browse/CASSANDRA-3070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13166617#comment-13166617 ]
Peter Schuller edited comment on CASSANDRA-3070 at 12/9/11 10:19 PM:
---------------------------------------------------------------------
This may be relevant, quoting myself from IRC:
{code}
21:20:01 < scode> pcmanus: Hey, are you there? 21:20:21 < scode> pcmanus: I am investigating something which might be https://issues.apache.org/jira/browse/CASSANDRA-3070 21:20:37 < scode> pcmanus: And I could use the help of someone with his brain all over counters, and Stu isn't here atm. :) 21:21:16 < scode> pcmanus: https://gist.github.com/8202cb46c8bd00c8391b 21:21:37 < scode> pcmanus: I am investigating why with CL.ALL and CL.QUORUM, I get seemingly random/varying results when I read a counter. 21:21:53 < scode> pcmanus: I have the offending sstables on a three-node test setup and am inserting debug printouts in the code to trace the reconiliation. 21:21:57 < scode> pcmanus: The gist above shows what's happening. 21:22:11 < scode> pcmanus: The latter is the wrong one, and the former is the correct one. 21:22:28 < scode> pcmanus: The interesting bit is that I see shards with the same node_id *AND* clock, but *DIFFERENT* counts. 21:22:53 < scode> pcmanus: My understanding of counters is that there should never (globally across an entire cluster in all sstables) exist two shards for the same node_id+clock but with different counts. 21:22:57 < scode> pcmanus: Is my understanding correct there? 21:25:10 < scode> pcmanus: There is one node out of the three that has the "offending" card (with a count of 2 instead of 1). Like with 3070, we observed this after having expanded a cluster (though I'm not sure how that would cause it, and we don't know if there existed a problem before the expansion). {code}
was (Author: scode):
This may be relevant, quoting myself from IRC:
{quote}
21:20:01 < scode> pcmanus: Hey, are you there? 21:20:21 < scode> pcmanus: I am investigating something which might be https://issues.apache.org/jira/browse/CASSANDRA-3070 21:20:37 < scode> pcmanus: And I could use the help of someone with his brain all over counters, and Stu isn't here atm. :) 21:21:16 < scode> pcmanus: https://gist.github.com/8202cb46c8bd00c8391b 21:21:37 < scode> pcmanus: I am investigating why with CL.ALL and CL.QUORUM, I get seemingly random/varying results when I read a counter. 21:21:53 < scode> pcmanus: I have the offending sstables on a three-node test setup and am inserting debug printouts in the code to trace the reconiliation. 21:21:57 < scode> pcmanus: The gist above shows what's happening. 21:22:11 < scode> pcmanus: The latter is the wrong one, and the former is the correct one. 21:22:28 < scode> pcmanus: The interesting bit is that I see shards with the same node_id *AND* clock, but *DIFFERENT* counts. 21:22:53 < scode> pcmanus: My understanding of counters is that there should never (globally across an entire cluster in all sstables) exist two shards for the same node_id+clock but with different counts. 21:22:57 < scode> pcmanus: Is my understanding correct there? 21:25:10 < scode> pcmanus: There is one node out of the three that has the "offending" card (with a count of 2 instead of 1). Like with 3070, we observed this after having expanded a cluster (though I'm not sure how that would cause it, and we don't know if there existed a problem before the expansion). {quote}
> counter repair
> --------------
>
> Key: CASSANDRA-3070
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3070
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.4
> Reporter: ivan
> Assignee: Sylvain Lebresne
> Attachments: counter_local_quroum_maybeschedulerepairs.txt, counter_local_quroum_maybeschedulerepairs_2.txt, counter_local_quroum_maybeschedulerepairs_3.txt
>
>
> Hi!
> We have some counters out of sync but repair doesn't sync values.
> We tried nodetool repair.
> We use LOCAL_QUORUM for read. A repair row mutation is sent to other nodes while reading a bad row but counters wasn't repaired by mutation.
> Output of two nodes were uploaded. (Some new debug messages were added.)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira