You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Brandon Williams (JIRA)" <ji...@apache.org> on 2011/09/08 05:19:09 UTC
[jira] [Created] (CASSANDRA-3156) assertion error in
RowRepairResolver
assertion error in RowRepairResolver
------------------------------------
Key: CASSANDRA-3156
URL: https://issues.apache.org/jira/browse/CASSANDRA-3156
Project: Cassandra
Issue Type: Bug
Components: Core
Reporter: Brandon Williams
Priority: Blocker
Fix For: 1.0
Only seems to happen on a coordinator who does not have a copy of the data:
DEBUG 03:15:59,866 Processing response on a callback from 3840@/10.179.64.227
DEBUG 03:15:59,866 Preprocessed data response
DEBUG 03:15:59,866 Processing response on a callback from 3841@/10.179.111.137
DEBUG 03:15:59,866 Preprocessed digest response
DEBUG 03:15:59,865 Processing response on a callback from 3837@/10.179.111.137
DEBUG 03:15:59,865 Preprocessed data response
DEBUG 03:15:59,865 Preprocessed data response
DEBUG 03:15:59,867 Preprocessed digest response
DEBUG 03:15:59,867 resolving 2 responses
ERROR 03:15:59,866 Fatal exception in thread Thread[ReadRepairStage:526,5,main]
java.lang.AssertionError
at org.apache.cassandra.service.RowRepairResolver.resolve(RowRepairResolver.java:77)
at org.apache.cassandra.service.AsyncRepairCallback$1.runMayThrow(AsyncRepairCallback.java:54)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
ERROR 03:15:59,866 Fatal exception in thread Thread[ReadRepairStage:525,5,main]
java.lang.AssertionError
at org.apache.cassandra.service.RowRepairResolver.resolve(RowRepairResolver.java:77)
at org.apache.cassandra.service.AsyncRepairCallback$1.runMayThrow(AsyncRepairCallback.java:54)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
ERROR 03:15:59,867 Fatal exception in thread Thread[ReadRepairStage:528,5,main]
java.lang.AssertionError
at org.apache.cassandra.service.RowRepairResolver.resolve(RowRepairResolver.java:77)
at org.apache.cassandra.service.AsyncRepairCallback$1.runMayThrow(AsyncRepairCallback.java:54)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
DEBUG 03:15:59,867 resolving 2 responses
DEBUG 03:15:59,867 resolving 2 responses
DEBUG 03:15:59,867 resolving 2 responses
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3156) assertion error in
RowRepairResolver
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100670#comment-13100670 ]
Hudson commented on CASSANDRA-3156:
-----------------------------------
Integrated in Cassandra #1092 (See [https://builds.apache.org/job/Cassandra/1092/])
reset ReadVerbHandler buffer between uses
patch by jbellis; reviewed by brandonwilliams for CASSANDRA-3156
jbellis : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1166865
Files :
* /cassandra/trunk/src/java/org/apache/cassandra/db/ReadVerbHandler.java
> assertion error in RowRepairResolver
> ------------------------------------
>
> Key: CASSANDRA-3156
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3156
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0
> Reporter: Brandon Williams
> Assignee: Jonathan Ellis
> Priority: Blocker
> Fix For: 1.0
>
> Attachments: 3156.txt
>
>
> Only seems to happen on a coordinator who does not have a copy of the data:
> DEBUG 03:15:59,866 Processing response on a callback from 3840@/10.179.64.227
> DEBUG 03:15:59,866 Preprocessed data response
> DEBUG 03:15:59,866 Processing response on a callback from 3841@/10.179.111.137
> DEBUG 03:15:59,866 Preprocessed digest response
> DEBUG 03:15:59,865 Processing response on a callback from 3837@/10.179.111.137
> DEBUG 03:15:59,865 Preprocessed data response
> DEBUG 03:15:59,865 Preprocessed data response
> DEBUG 03:15:59,867 Preprocessed digest response
> DEBUG 03:15:59,867 resolving 2 responses
> ERROR 03:15:59,866 Fatal exception in thread Thread[ReadRepairStage:526,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.service.RowRepairResolver.resolve(RowRepairResolver.java:77)
> at org.apache.cassandra.service.AsyncRepairCallback$1.runMayThrow(AsyncRepairCallback.java:54)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ERROR 03:15:59,866 Fatal exception in thread Thread[ReadRepairStage:525,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.service.RowRepairResolver.resolve(RowRepairResolver.java:77)
> at org.apache.cassandra.service.AsyncRepairCallback$1.runMayThrow(AsyncRepairCallback.java:54)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ERROR 03:15:59,867 Fatal exception in thread Thread[ReadRepairStage:528,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.service.RowRepairResolver.resolve(RowRepairResolver.java:77)
> at org.apache.cassandra.service.AsyncRepairCallback$1.runMayThrow(AsyncRepairCallback.java:54)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> DEBUG 03:15:59,867 resolving 2 responses
> DEBUG 03:15:59,867 resolving 2 responses
> DEBUG 03:15:59,867 resolving 2 responses
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3156) assertion error in
RowRepairResolver
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-3156:
--------------------------------------
Affects Version/s: 1.0
> assertion error in RowRepairResolver
> ------------------------------------
>
> Key: CASSANDRA-3156
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3156
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0
> Reporter: Brandon Williams
> Priority: Blocker
> Fix For: 1.0
>
>
> Only seems to happen on a coordinator who does not have a copy of the data:
> DEBUG 03:15:59,866 Processing response on a callback from 3840@/10.179.64.227
> DEBUG 03:15:59,866 Preprocessed data response
> DEBUG 03:15:59,866 Processing response on a callback from 3841@/10.179.111.137
> DEBUG 03:15:59,866 Preprocessed digest response
> DEBUG 03:15:59,865 Processing response on a callback from 3837@/10.179.111.137
> DEBUG 03:15:59,865 Preprocessed data response
> DEBUG 03:15:59,865 Preprocessed data response
> DEBUG 03:15:59,867 Preprocessed digest response
> DEBUG 03:15:59,867 resolving 2 responses
> ERROR 03:15:59,866 Fatal exception in thread Thread[ReadRepairStage:526,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.service.RowRepairResolver.resolve(RowRepairResolver.java:77)
> at org.apache.cassandra.service.AsyncRepairCallback$1.runMayThrow(AsyncRepairCallback.java:54)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ERROR 03:15:59,866 Fatal exception in thread Thread[ReadRepairStage:525,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.service.RowRepairResolver.resolve(RowRepairResolver.java:77)
> at org.apache.cassandra.service.AsyncRepairCallback$1.runMayThrow(AsyncRepairCallback.java:54)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ERROR 03:15:59,867 Fatal exception in thread Thread[ReadRepairStage:528,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.service.RowRepairResolver.resolve(RowRepairResolver.java:77)
> at org.apache.cassandra.service.AsyncRepairCallback$1.runMayThrow(AsyncRepairCallback.java:54)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> DEBUG 03:15:59,867 resolving 2 responses
> DEBUG 03:15:59,867 resolving 2 responses
> DEBUG 03:15:59,867 resolving 2 responses
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CASSANDRA-3156) assertion error in
RowRepairResolver
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis resolved CASSANDRA-3156.
---------------------------------------
Resolution: Fixed
Reviewer: brandon.williams
Assignee: Jonathan Ellis
committed
> assertion error in RowRepairResolver
> ------------------------------------
>
> Key: CASSANDRA-3156
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3156
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0
> Reporter: Brandon Williams
> Assignee: Jonathan Ellis
> Priority: Blocker
> Fix For: 1.0
>
> Attachments: 3156.txt
>
>
> Only seems to happen on a coordinator who does not have a copy of the data:
> DEBUG 03:15:59,866 Processing response on a callback from 3840@/10.179.64.227
> DEBUG 03:15:59,866 Preprocessed data response
> DEBUG 03:15:59,866 Processing response on a callback from 3841@/10.179.111.137
> DEBUG 03:15:59,866 Preprocessed digest response
> DEBUG 03:15:59,865 Processing response on a callback from 3837@/10.179.111.137
> DEBUG 03:15:59,865 Preprocessed data response
> DEBUG 03:15:59,865 Preprocessed data response
> DEBUG 03:15:59,867 Preprocessed digest response
> DEBUG 03:15:59,867 resolving 2 responses
> ERROR 03:15:59,866 Fatal exception in thread Thread[ReadRepairStage:526,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.service.RowRepairResolver.resolve(RowRepairResolver.java:77)
> at org.apache.cassandra.service.AsyncRepairCallback$1.runMayThrow(AsyncRepairCallback.java:54)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ERROR 03:15:59,866 Fatal exception in thread Thread[ReadRepairStage:525,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.service.RowRepairResolver.resolve(RowRepairResolver.java:77)
> at org.apache.cassandra.service.AsyncRepairCallback$1.runMayThrow(AsyncRepairCallback.java:54)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ERROR 03:15:59,867 Fatal exception in thread Thread[ReadRepairStage:528,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.service.RowRepairResolver.resolve(RowRepairResolver.java:77)
> at org.apache.cassandra.service.AsyncRepairCallback$1.runMayThrow(AsyncRepairCallback.java:54)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> DEBUG 03:15:59,867 resolving 2 responses
> DEBUG 03:15:59,867 resolving 2 responses
> DEBUG 03:15:59,867 resolving 2 responses
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3156) assertion error in
RowRepairResolver
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-3156:
--------------------------------------
Attachment: 3156.txt
The new threadlocal code was omitting to reset the buffer before each use. Patch attached.
> assertion error in RowRepairResolver
> ------------------------------------
>
> Key: CASSANDRA-3156
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3156
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0
> Reporter: Brandon Williams
> Priority: Blocker
> Fix For: 1.0
>
> Attachments: 3156.txt
>
>
> Only seems to happen on a coordinator who does not have a copy of the data:
> DEBUG 03:15:59,866 Processing response on a callback from 3840@/10.179.64.227
> DEBUG 03:15:59,866 Preprocessed data response
> DEBUG 03:15:59,866 Processing response on a callback from 3841@/10.179.111.137
> DEBUG 03:15:59,866 Preprocessed digest response
> DEBUG 03:15:59,865 Processing response on a callback from 3837@/10.179.111.137
> DEBUG 03:15:59,865 Preprocessed data response
> DEBUG 03:15:59,865 Preprocessed data response
> DEBUG 03:15:59,867 Preprocessed digest response
> DEBUG 03:15:59,867 resolving 2 responses
> ERROR 03:15:59,866 Fatal exception in thread Thread[ReadRepairStage:526,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.service.RowRepairResolver.resolve(RowRepairResolver.java:77)
> at org.apache.cassandra.service.AsyncRepairCallback$1.runMayThrow(AsyncRepairCallback.java:54)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ERROR 03:15:59,866 Fatal exception in thread Thread[ReadRepairStage:525,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.service.RowRepairResolver.resolve(RowRepairResolver.java:77)
> at org.apache.cassandra.service.AsyncRepairCallback$1.runMayThrow(AsyncRepairCallback.java:54)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ERROR 03:15:59,867 Fatal exception in thread Thread[ReadRepairStage:528,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.service.RowRepairResolver.resolve(RowRepairResolver.java:77)
> at org.apache.cassandra.service.AsyncRepairCallback$1.runMayThrow(AsyncRepairCallback.java:54)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> DEBUG 03:15:59,867 resolving 2 responses
> DEBUG 03:15:59,867 resolving 2 responses
> DEBUG 03:15:59,867 resolving 2 responses
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3156) assertion error in
RowRepairResolver
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099745#comment-13099745 ]
Jonathan Ellis commented on CASSANDRA-3156:
-------------------------------------------
So, how RR is supposed to work is like this:
Optimistic phase:
Coordinator sends data read to closest replica, digest to others
If there is a mismatch (optimism fail), we go to the repair phase:
Coordinator sends data reads to all replicas to merge + repair
The failing assert is saying "I got a digest reply, during the repair phase--i.e. we sent a data request but got a digest back."
No idea how this is happening.
> assertion error in RowRepairResolver
> ------------------------------------
>
> Key: CASSANDRA-3156
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3156
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0
> Reporter: Brandon Williams
> Priority: Blocker
> Fix For: 1.0
>
>
> Only seems to happen on a coordinator who does not have a copy of the data:
> DEBUG 03:15:59,866 Processing response on a callback from 3840@/10.179.64.227
> DEBUG 03:15:59,866 Preprocessed data response
> DEBUG 03:15:59,866 Processing response on a callback from 3841@/10.179.111.137
> DEBUG 03:15:59,866 Preprocessed digest response
> DEBUG 03:15:59,865 Processing response on a callback from 3837@/10.179.111.137
> DEBUG 03:15:59,865 Preprocessed data response
> DEBUG 03:15:59,865 Preprocessed data response
> DEBUG 03:15:59,867 Preprocessed digest response
> DEBUG 03:15:59,867 resolving 2 responses
> ERROR 03:15:59,866 Fatal exception in thread Thread[ReadRepairStage:526,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.service.RowRepairResolver.resolve(RowRepairResolver.java:77)
> at org.apache.cassandra.service.AsyncRepairCallback$1.runMayThrow(AsyncRepairCallback.java:54)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ERROR 03:15:59,866 Fatal exception in thread Thread[ReadRepairStage:525,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.service.RowRepairResolver.resolve(RowRepairResolver.java:77)
> at org.apache.cassandra.service.AsyncRepairCallback$1.runMayThrow(AsyncRepairCallback.java:54)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ERROR 03:15:59,867 Fatal exception in thread Thread[ReadRepairStage:528,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.service.RowRepairResolver.resolve(RowRepairResolver.java:77)
> at org.apache.cassandra.service.AsyncRepairCallback$1.runMayThrow(AsyncRepairCallback.java:54)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> DEBUG 03:15:59,867 resolving 2 responses
> DEBUG 03:15:59,867 resolving 2 responses
> DEBUG 03:15:59,867 resolving 2 responses
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3156) assertion error in
RowRepairResolver
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100614#comment-13100614 ]
Brandon Williams commented on CASSANDRA-3156:
---------------------------------------------
+1
> assertion error in RowRepairResolver
> ------------------------------------
>
> Key: CASSANDRA-3156
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3156
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0
> Reporter: Brandon Williams
> Priority: Blocker
> Fix For: 1.0
>
> Attachments: 3156.txt
>
>
> Only seems to happen on a coordinator who does not have a copy of the data:
> DEBUG 03:15:59,866 Processing response on a callback from 3840@/10.179.64.227
> DEBUG 03:15:59,866 Preprocessed data response
> DEBUG 03:15:59,866 Processing response on a callback from 3841@/10.179.111.137
> DEBUG 03:15:59,866 Preprocessed digest response
> DEBUG 03:15:59,865 Processing response on a callback from 3837@/10.179.111.137
> DEBUG 03:15:59,865 Preprocessed data response
> DEBUG 03:15:59,865 Preprocessed data response
> DEBUG 03:15:59,867 Preprocessed digest response
> DEBUG 03:15:59,867 resolving 2 responses
> ERROR 03:15:59,866 Fatal exception in thread Thread[ReadRepairStage:526,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.service.RowRepairResolver.resolve(RowRepairResolver.java:77)
> at org.apache.cassandra.service.AsyncRepairCallback$1.runMayThrow(AsyncRepairCallback.java:54)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ERROR 03:15:59,866 Fatal exception in thread Thread[ReadRepairStage:525,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.service.RowRepairResolver.resolve(RowRepairResolver.java:77)
> at org.apache.cassandra.service.AsyncRepairCallback$1.runMayThrow(AsyncRepairCallback.java:54)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ERROR 03:15:59,867 Fatal exception in thread Thread[ReadRepairStage:528,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.service.RowRepairResolver.resolve(RowRepairResolver.java:77)
> at org.apache.cassandra.service.AsyncRepairCallback$1.runMayThrow(AsyncRepairCallback.java:54)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> DEBUG 03:15:59,867 resolving 2 responses
> DEBUG 03:15:59,867 resolving 2 responses
> DEBUG 03:15:59,867 resolving 2 responses
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3156) assertion error in
RowRepairResolver
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100534#comment-13100534 ]
Brandon Williams commented on CASSANDRA-3156:
---------------------------------------------
Appears to be caused by http://svn.apache.org/viewvc?revision=1151304&view=revision
> assertion error in RowRepairResolver
> ------------------------------------
>
> Key: CASSANDRA-3156
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3156
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0
> Reporter: Brandon Williams
> Priority: Blocker
> Fix For: 1.0
>
>
> Only seems to happen on a coordinator who does not have a copy of the data:
> DEBUG 03:15:59,866 Processing response on a callback from 3840@/10.179.64.227
> DEBUG 03:15:59,866 Preprocessed data response
> DEBUG 03:15:59,866 Processing response on a callback from 3841@/10.179.111.137
> DEBUG 03:15:59,866 Preprocessed digest response
> DEBUG 03:15:59,865 Processing response on a callback from 3837@/10.179.111.137
> DEBUG 03:15:59,865 Preprocessed data response
> DEBUG 03:15:59,865 Preprocessed data response
> DEBUG 03:15:59,867 Preprocessed digest response
> DEBUG 03:15:59,867 resolving 2 responses
> ERROR 03:15:59,866 Fatal exception in thread Thread[ReadRepairStage:526,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.service.RowRepairResolver.resolve(RowRepairResolver.java:77)
> at org.apache.cassandra.service.AsyncRepairCallback$1.runMayThrow(AsyncRepairCallback.java:54)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ERROR 03:15:59,866 Fatal exception in thread Thread[ReadRepairStage:525,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.service.RowRepairResolver.resolve(RowRepairResolver.java:77)
> at org.apache.cassandra.service.AsyncRepairCallback$1.runMayThrow(AsyncRepairCallback.java:54)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ERROR 03:15:59,867 Fatal exception in thread Thread[ReadRepairStage:528,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.service.RowRepairResolver.resolve(RowRepairResolver.java:77)
> at org.apache.cassandra.service.AsyncRepairCallback$1.runMayThrow(AsyncRepairCallback.java:54)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> DEBUG 03:15:59,867 resolving 2 responses
> DEBUG 03:15:59,867 resolving 2 responses
> DEBUG 03:15:59,867 resolving 2 responses
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3156) assertion error in
RowRepairResolver
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099732#comment-13099732 ]
Brandon Williams commented on CASSANDRA-3156:
---------------------------------------------
Also some spurious digest mismatches mixed in, even though I have no reason to suspect there is actually a mismatch in my dev env (3 nodes, rf=2):
DEBUG 03:15:59,823 Digest mismatch:
org.apache.cassandra.service.DigestMismatchException: Mismatch for key DecoratedKey(20580074455139572311737153648595094740, 30363933) (fb3f10b793298382b554737490bc78b5 vs db8d74ec919be7c1a1dda15c85754eb0)
at org.apache.cassandra.service.RowDigestResolver.resolve(RowDigestResolver.java:105)
at org.apache.cassandra.service.RowDigestResolver.resolve(RowDigestResolver.java:30)
at org.apache.cassandra.service.ReadCallback$AsyncRepairRunner.runMayThrow(ReadCallback.java:229)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
> assertion error in RowRepairResolver
> ------------------------------------
>
> Key: CASSANDRA-3156
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3156
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Brandon Williams
> Priority: Blocker
> Fix For: 1.0
>
>
> Only seems to happen on a coordinator who does not have a copy of the data:
> DEBUG 03:15:59,866 Processing response on a callback from 3840@/10.179.64.227
> DEBUG 03:15:59,866 Preprocessed data response
> DEBUG 03:15:59,866 Processing response on a callback from 3841@/10.179.111.137
> DEBUG 03:15:59,866 Preprocessed digest response
> DEBUG 03:15:59,865 Processing response on a callback from 3837@/10.179.111.137
> DEBUG 03:15:59,865 Preprocessed data response
> DEBUG 03:15:59,865 Preprocessed data response
> DEBUG 03:15:59,867 Preprocessed digest response
> DEBUG 03:15:59,867 resolving 2 responses
> ERROR 03:15:59,866 Fatal exception in thread Thread[ReadRepairStage:526,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.service.RowRepairResolver.resolve(RowRepairResolver.java:77)
> at org.apache.cassandra.service.AsyncRepairCallback$1.runMayThrow(AsyncRepairCallback.java:54)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ERROR 03:15:59,866 Fatal exception in thread Thread[ReadRepairStage:525,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.service.RowRepairResolver.resolve(RowRepairResolver.java:77)
> at org.apache.cassandra.service.AsyncRepairCallback$1.runMayThrow(AsyncRepairCallback.java:54)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ERROR 03:15:59,867 Fatal exception in thread Thread[ReadRepairStage:528,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.service.RowRepairResolver.resolve(RowRepairResolver.java:77)
> at org.apache.cassandra.service.AsyncRepairCallback$1.runMayThrow(AsyncRepairCallback.java:54)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> DEBUG 03:15:59,867 resolving 2 responses
> DEBUG 03:15:59,867 resolving 2 responses
> DEBUG 03:15:59,867 resolving 2 responses
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-3156) assertion error in
RowRepairResolver
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099732#comment-13099732 ]
Brandon Williams edited comment on CASSANDRA-3156 at 9/8/11 3:23 AM:
---------------------------------------------------------------------
Also some spurious digest mismatches mixed in, even though I have no reason to suspect there is actually a mismatch in my dev env (3 nodes, rf=2):
DEBUG 03:15:59,823 Digest mismatch:
org.apache.cassandra.service.DigestMismatchException: Mismatch for key DecoratedKey(20580074455139572311737153648595094740, 30363933) (fb3f10b793298382b554737490bc78b5 vs db8d74ec919be7c1a1dda15c85754eb0)
at org.apache.cassandra.service.RowDigestResolver.resolve(RowDigestResolver.java:105)
at org.apache.cassandra.service.RowDigestResolver.resolve(RowDigestResolver.java:30)
at org.apache.cassandra.service.ReadCallback$AsyncRepairRunner.runMayThrow(ReadCallback.java:229)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
But this seems to happen when the coordinator _does_ have a copy of the data.
was (Author: brandon.williams):
Also some spurious digest mismatches mixed in, even though I have no reason to suspect there is actually a mismatch in my dev env (3 nodes, rf=2):
DEBUG 03:15:59,823 Digest mismatch:
org.apache.cassandra.service.DigestMismatchException: Mismatch for key DecoratedKey(20580074455139572311737153648595094740, 30363933) (fb3f10b793298382b554737490bc78b5 vs db8d74ec919be7c1a1dda15c85754eb0)
at org.apache.cassandra.service.RowDigestResolver.resolve(RowDigestResolver.java:105)
at org.apache.cassandra.service.RowDigestResolver.resolve(RowDigestResolver.java:30)
at org.apache.cassandra.service.ReadCallback$AsyncRepairRunner.runMayThrow(ReadCallback.java:229)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
> assertion error in RowRepairResolver
> ------------------------------------
>
> Key: CASSANDRA-3156
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3156
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Brandon Williams
> Priority: Blocker
> Fix For: 1.0
>
>
> Only seems to happen on a coordinator who does not have a copy of the data:
> DEBUG 03:15:59,866 Processing response on a callback from 3840@/10.179.64.227
> DEBUG 03:15:59,866 Preprocessed data response
> DEBUG 03:15:59,866 Processing response on a callback from 3841@/10.179.111.137
> DEBUG 03:15:59,866 Preprocessed digest response
> DEBUG 03:15:59,865 Processing response on a callback from 3837@/10.179.111.137
> DEBUG 03:15:59,865 Preprocessed data response
> DEBUG 03:15:59,865 Preprocessed data response
> DEBUG 03:15:59,867 Preprocessed digest response
> DEBUG 03:15:59,867 resolving 2 responses
> ERROR 03:15:59,866 Fatal exception in thread Thread[ReadRepairStage:526,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.service.RowRepairResolver.resolve(RowRepairResolver.java:77)
> at org.apache.cassandra.service.AsyncRepairCallback$1.runMayThrow(AsyncRepairCallback.java:54)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ERROR 03:15:59,866 Fatal exception in thread Thread[ReadRepairStage:525,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.service.RowRepairResolver.resolve(RowRepairResolver.java:77)
> at org.apache.cassandra.service.AsyncRepairCallback$1.runMayThrow(AsyncRepairCallback.java:54)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ERROR 03:15:59,867 Fatal exception in thread Thread[ReadRepairStage:528,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.service.RowRepairResolver.resolve(RowRepairResolver.java:77)
> at org.apache.cassandra.service.AsyncRepairCallback$1.runMayThrow(AsyncRepairCallback.java:54)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> DEBUG 03:15:59,867 resolving 2 responses
> DEBUG 03:15:59,867 resolving 2 responses
> DEBUG 03:15:59,867 resolving 2 responses
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira