You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Jason Harvey (JIRA)" <ji...@apache.org> on 2011/09/24 08:12:26 UTC
[jira] [Created] (CASSANDRA-3256) AssertionError when repairing a
node
AssertionError when repairing a node
------------------------------------
Key: CASSANDRA-3256
URL: https://issues.apache.org/jira/browse/CASSANDRA-3256
Project: Cassandra
Issue Type: Bug
Affects Versions: 0.8.5
Reporter: Jason Harvey
Priority: Minor
When repairing a node, the following exception was thrown two times:
{code}
ERROR [AntiEntropyStage:2] 2011-09-23 23:00:24,016 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[AntiEntropyStage:2,5,main]
java.lang.AssertionError
at org.apache.cassandra.service.AntiEntropyService.rendezvous(AntiEntropyService.java:170)
at org.apache.cassandra.service.AntiEntropyService.access$100(AntiEntropyService.java:90)
at org.apache.cassandra.service.AntiEntropyService$TreeResponseVerbHandler.doVerb(AntiEntropyService.java:518)
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{code}
No other errors occurred on the node. This assertion appears to be called if an existing Repair session could not be found. Interestingly, the repair did continue to run after this as evidenced by several other AntiEntropyService entires in the log.
8 node ring with an RF of 3, if that matters at all. No other nodes in the ring threw exceptions.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3256) AssertionError when repairing a
node
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sylvain Lebresne updated CASSANDRA-3256:
----------------------------------------
Attachment: 3256.patch
My guess is that this is probably harmless. Basically the node received a merkle tree for a session that it doesn't know about. This means that the said repair session has been interrupted. _A priori_, I see only two things that can cause this:
* The repair thread on the host has been interrupted. But 1) in that case you should have found a exception earlier on saying "Interrupted while waiting for repair: repair will continue in the background." and 2) I don't see what could interrupt that thread.
* The node restarted and only now receives a response for request made before the restart. This is imho the more likely scenario. And if so, the previous repair (the one started before the restart) won't succeed, but there is no more consequence than that.
That being throwing an assertion error is probably such a great idea given that it's a scenario that can happen. Attaching a patch that simply log an hopefully more explicit message.
> AssertionError when repairing a node
> ------------------------------------
>
> Key: CASSANDRA-3256
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3256
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.8.5
> Reporter: Jason Harvey
> Assignee: Sylvain Lebresne
> Priority: Minor
> Labels: repair
> Fix For: 0.8.7
>
> Attachments: 3256.patch
>
>
> When repairing a node, the following exception was thrown two times:
> {code}
> ERROR [AntiEntropyStage:2] 2011-09-23 23:00:24,016 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[AntiEntropyStage:2,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.service.AntiEntropyService.rendezvous(AntiEntropyService.java:170)
> at org.apache.cassandra.service.AntiEntropyService.access$100(AntiEntropyService.java:90)
> at org.apache.cassandra.service.AntiEntropyService$TreeResponseVerbHandler.doVerb(AntiEntropyService.java:518)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> No other errors occurred on the node. From peeking at the code, this assertion appears to simply check if an existing repair session could be found. Interestingly, the repair did continue to run after this as evidenced by several other AntiEntropyService entires in the log.
> 8 node ring with an RF of 3, if that matters at all. No other nodes in the ring threw exceptions.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3256) AssertionError when repairing a
node
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114715#comment-13114715 ]
Hudson commented on CASSANDRA-3256:
-----------------------------------
Integrated in Cassandra-0.8 #341 (See [https://builds.apache.org/job/Cassandra-0.8/341/])
Log a miningfull warning when a node receive a message for a repair session that don't exist anymore
patch by slebresne; reviewed by jbellis for CASSANDRA-3256
slebresne : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1175880
Files :
* /cassandra/branches/cassandra-0.8/CHANGES.txt
* /cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/service/AntiEntropyService.java
> AssertionError when repairing a node
> ------------------------------------
>
> Key: CASSANDRA-3256
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3256
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.8.5
> Reporter: Jason Harvey
> Assignee: Sylvain Lebresne
> Priority: Minor
> Labels: repair
> Fix For: 0.8.7
>
> Attachments: 3256.patch
>
>
> When repairing a node, the following exception was thrown two times:
> {code}
> ERROR [AntiEntropyStage:2] 2011-09-23 23:00:24,016 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[AntiEntropyStage:2,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.service.AntiEntropyService.rendezvous(AntiEntropyService.java:170)
> at org.apache.cassandra.service.AntiEntropyService.access$100(AntiEntropyService.java:90)
> at org.apache.cassandra.service.AntiEntropyService$TreeResponseVerbHandler.doVerb(AntiEntropyService.java:518)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> No other errors occurred on the node. From peeking at the code, this assertion appears to simply check if an existing repair session could be found. Interestingly, the repair did continue to run after this as evidenced by several other AntiEntropyService entires in the log.
> 8 node ring with an RF of 3, if that matters at all. No other nodes in the ring threw exceptions.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3256) AssertionError when repairing a
node
Posted by "Jason Harvey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jason Harvey updated CASSANDRA-3256:
------------------------------------
Description:
When repairing a node, the following exception was thrown two times:
{code}
ERROR [AntiEntropyStage:2] 2011-09-23 23:00:24,016 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[AntiEntropyStage:2,5,main]
java.lang.AssertionError
at org.apache.cassandra.service.AntiEntropyService.rendezvous(AntiEntropyService.java:170)
at org.apache.cassandra.service.AntiEntropyService.access$100(AntiEntropyService.java:90)
at org.apache.cassandra.service.AntiEntropyService$TreeResponseVerbHandler.doVerb(AntiEntropyService.java:518)
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{code}
No other errors occurred on the node. From peeking at the code, this assertion appears to simply check if an existing repair session could be found. Interestingly, the repair did continue to run after this as evidenced by several other AntiEntropyService entires in the log.
8 node ring with an RF of 3, if that matters at all. No other nodes in the ring threw exceptions.
was:
When repairing a node, the following exception was thrown two times:
{code}
ERROR [AntiEntropyStage:2] 2011-09-23 23:00:24,016 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[AntiEntropyStage:2,5,main]
java.lang.AssertionError
at org.apache.cassandra.service.AntiEntropyService.rendezvous(AntiEntropyService.java:170)
at org.apache.cassandra.service.AntiEntropyService.access$100(AntiEntropyService.java:90)
at org.apache.cassandra.service.AntiEntropyService$TreeResponseVerbHandler.doVerb(AntiEntropyService.java:518)
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{code}
No other errors occurred on the node. This assertion appears to be called if an existing Repair session could not be found. Interestingly, the repair did continue to run after this as evidenced by several other AntiEntropyService entires in the log.
8 node ring with an RF of 3, if that matters at all. No other nodes in the ring threw exceptions.
> AssertionError when repairing a node
> ------------------------------------
>
> Key: CASSANDRA-3256
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3256
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.8.5
> Reporter: Jason Harvey
> Priority: Minor
>
> When repairing a node, the following exception was thrown two times:
> {code}
> ERROR [AntiEntropyStage:2] 2011-09-23 23:00:24,016 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[AntiEntropyStage:2,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.service.AntiEntropyService.rendezvous(AntiEntropyService.java:170)
> at org.apache.cassandra.service.AntiEntropyService.access$100(AntiEntropyService.java:90)
> at org.apache.cassandra.service.AntiEntropyService$TreeResponseVerbHandler.doVerb(AntiEntropyService.java:518)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> No other errors occurred on the node. From peeking at the code, this assertion appears to simply check if an existing repair session could be found. Interestingly, the repair did continue to run after this as evidenced by several other AntiEntropyService entires in the log.
> 8 node ring with an RF of 3, if that matters at all. No other nodes in the ring threw exceptions.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3256) AssertionError when repairing a
node
Posted by "Jason Harvey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114119#comment-13114119 ]
Jason Harvey commented on CASSANDRA-3256:
-----------------------------------------
The repair which triggered this assertion error did eventually return with no further errors.
> AssertionError when repairing a node
> ------------------------------------
>
> Key: CASSANDRA-3256
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3256
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.8.5
> Reporter: Jason Harvey
> Assignee: Sylvain Lebresne
> Priority: Minor
>
> When repairing a node, the following exception was thrown two times:
> {code}
> ERROR [AntiEntropyStage:2] 2011-09-23 23:00:24,016 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[AntiEntropyStage:2,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.service.AntiEntropyService.rendezvous(AntiEntropyService.java:170)
> at org.apache.cassandra.service.AntiEntropyService.access$100(AntiEntropyService.java:90)
> at org.apache.cassandra.service.AntiEntropyService$TreeResponseVerbHandler.doVerb(AntiEntropyService.java:518)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> No other errors occurred on the node. From peeking at the code, this assertion appears to simply check if an existing repair session could be found. Interestingly, the repair did continue to run after this as evidenced by several other AntiEntropyService entires in the log.
> 8 node ring with an RF of 3, if that matters at all. No other nodes in the ring threw exceptions.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3256) AssertionError when repairing a
node
Posted by "Jason Harvey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113908#comment-13113908 ]
Jason Harvey commented on CASSANDRA-3256:
-----------------------------------------
I should note, this same node has been repaired several times in the past with no issues.
> AssertionError when repairing a node
> ------------------------------------
>
> Key: CASSANDRA-3256
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3256
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.8.5
> Reporter: Jason Harvey
> Priority: Minor
>
> When repairing a node, the following exception was thrown two times:
> {code}
> ERROR [AntiEntropyStage:2] 2011-09-23 23:00:24,016 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[AntiEntropyStage:2,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.service.AntiEntropyService.rendezvous(AntiEntropyService.java:170)
> at org.apache.cassandra.service.AntiEntropyService.access$100(AntiEntropyService.java:90)
> at org.apache.cassandra.service.AntiEntropyService$TreeResponseVerbHandler.doVerb(AntiEntropyService.java:518)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> No other errors occurred on the node. From peeking at the code, this assertion appears to simply check if an existing repair session could be found. Interestingly, the repair did continue to run after this as evidenced by several other AntiEntropyService entires in the log.
> 8 node ring with an RF of 3, if that matters at all. No other nodes in the ring threw exceptions.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (CASSANDRA-3256) AssertionError when repairing a
node
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis reassigned CASSANDRA-3256:
-----------------------------------------
Assignee: Sylvain Lebresne
> AssertionError when repairing a node
> ------------------------------------
>
> Key: CASSANDRA-3256
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3256
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.8.5
> Reporter: Jason Harvey
> Assignee: Sylvain Lebresne
> Priority: Minor
>
> When repairing a node, the following exception was thrown two times:
> {code}
> ERROR [AntiEntropyStage:2] 2011-09-23 23:00:24,016 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[AntiEntropyStage:2,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.service.AntiEntropyService.rendezvous(AntiEntropyService.java:170)
> at org.apache.cassandra.service.AntiEntropyService.access$100(AntiEntropyService.java:90)
> at org.apache.cassandra.service.AntiEntropyService$TreeResponseVerbHandler.doVerb(AntiEntropyService.java:518)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> No other errors occurred on the node. From peeking at the code, this assertion appears to simply check if an existing repair session could be found. Interestingly, the repair did continue to run after this as evidenced by several other AntiEntropyService entires in the log.
> 8 node ring with an RF of 3, if that matters at all. No other nodes in the ring threw exceptions.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3256) AssertionError when repairing a
node
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114645#comment-13114645 ]
Jonathan Ellis commented on CASSANDRA-3256:
-------------------------------------------
+1
> AssertionError when repairing a node
> ------------------------------------
>
> Key: CASSANDRA-3256
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3256
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.8.5
> Reporter: Jason Harvey
> Assignee: Sylvain Lebresne
> Priority: Minor
> Labels: repair
> Fix For: 0.8.7
>
> Attachments: 3256.patch
>
>
> When repairing a node, the following exception was thrown two times:
> {code}
> ERROR [AntiEntropyStage:2] 2011-09-23 23:00:24,016 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[AntiEntropyStage:2,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.service.AntiEntropyService.rendezvous(AntiEntropyService.java:170)
> at org.apache.cassandra.service.AntiEntropyService.access$100(AntiEntropyService.java:90)
> at org.apache.cassandra.service.AntiEntropyService$TreeResponseVerbHandler.doVerb(AntiEntropyService.java:518)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {code}
> No other errors occurred on the node. From peeking at the code, this assertion appears to simply check if an existing repair session could be found. Interestingly, the repair did continue to run after this as evidenced by several other AntiEntropyService entires in the log.
> 8 node ring with an RF of 3, if that matters at all. No other nodes in the ring threw exceptions.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira