You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Scott Fines (Created) (JIRA)" <ji...@apache.org> on 2011/10/24 18:41:32 UTC
[jira] [Created] (CASSANDRA-3400) ConcurrentModificationException
during nodetool repair
ConcurrentModificationException during nodetool repair
------------------------------------------------------
Key: CASSANDRA-3400
URL: https://issues.apache.org/jira/browse/CASSANDRA-3400
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 0.8.7
Environment: Suse Enterprise linux 11.4
Reporter: Scott Fines
Priority: Minor
When running a nodetool repair, the following exception can be thrown:
ERROR [AntiEntropySessions:12] 2011-10-24 11:17:52,154 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[AntiEntropySessions:12,5,RMI Runtime]
java.lang.RuntimeException: java.util.ConcurrentModificationException
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
at java.util.HashMap$KeyIterator.next(HashMap.java:828)
at org.apache.cassandra.service.AntiEntropyService$RepairSession$RepairJob.sendTreeRequests(AntiEntropyService.java:784)
at org.apache.cassandra.service.AntiEntropyService$RepairSession.runMayThrow(AntiEntropyService.java:680)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
... 6 more
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3400) ConcurrentModificationException
during nodetool repair
Posted by "Sylvain Lebresne (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sylvain Lebresne updated CASSANDRA-3400:
----------------------------------------
Attachment: 3400.patch
There is a race between when we send requests and when start handling their result, i.e if a response comes before all the requests have been sent. It's fairly unlikely to happen imho but given than one request is local, if there is no and barely any data to build the merkle tree from, I suppose it's possible the local response comes quickly enough.
Patch attached to resolve the issue.
> ConcurrentModificationException during nodetool repair
> ------------------------------------------------------
>
> Key: CASSANDRA-3400
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3400
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.7
> Environment: Suse Enterprise linux 11.4
> Reporter: Scott Fines
> Assignee: Sylvain Lebresne
> Priority: Minor
> Attachments: 3400.patch
>
>
> When running a nodetool repair, the following exception can be thrown:
> ERROR [AntiEntropySessions:12] 2011-10-24 11:17:52,154 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[AntiEntropySessions:12,5,RMI Runtime]
> java.lang.RuntimeException: java.util.ConcurrentModificationException
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
> at java.util.HashMap$KeyIterator.next(HashMap.java:828)
> at org.apache.cassandra.service.AntiEntropyService$RepairSession$RepairJob.sendTreeRequests(AntiEntropyService.java:784)
> at org.apache.cassandra.service.AntiEntropyService$RepairSession.runMayThrow(AntiEntropyService.java:680)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> ... 6 more
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3400) ConcurrentModificationException
during nodetool repair
Posted by "Scott Fines (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13134228#comment-13134228 ]
Scott Fines commented on CASSANDRA-3400:
----------------------------------------
A few notes as I discover them:
This error occurs every time the nodetool runs repair--repair effectively does not work on this node now.
> ConcurrentModificationException during nodetool repair
> ------------------------------------------------------
>
> Key: CASSANDRA-3400
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3400
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.7
> Environment: Suse Enterprise linux 11.4
> Reporter: Scott Fines
> Assignee: Sylvain Lebresne
> Priority: Minor
>
> When running a nodetool repair, the following exception can be thrown:
> ERROR [AntiEntropySessions:12] 2011-10-24 11:17:52,154 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[AntiEntropySessions:12,5,RMI Runtime]
> java.lang.RuntimeException: java.util.ConcurrentModificationException
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
> at java.util.HashMap$KeyIterator.next(HashMap.java:828)
> at org.apache.cassandra.service.AntiEntropyService$RepairSession$RepairJob.sendTreeRequests(AntiEntropyService.java:784)
> at org.apache.cassandra.service.AntiEntropyService$RepairSession.runMayThrow(AntiEntropyService.java:680)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> ... 6 more
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3400) ConcurrentModificationException
during nodetool repair
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13135297#comment-13135297 ]
Hudson commented on CASSANDRA-3400:
-----------------------------------
Integrated in Cassandra-0.8 #388 (See [https://builds.apache.org/job/Cassandra-0.8/388/])
Fix race in AntiEntropyService
patch by slebresne; reviewed by jbellis for CASSANDRA-3400
slebresne : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1188740
Files :
* /cassandra/branches/cassandra-0.8/CHANGES.txt
* /cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/service/AntiEntropyService.java
> ConcurrentModificationException during nodetool repair
> ------------------------------------------------------
>
> Key: CASSANDRA-3400
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3400
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.7, 1.0.0
> Environment: Suse Enterprise linux 11.4
> Reporter: Scott Fines
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 0.8.8, 1.0.1
>
> Attachments: 3400.patch
>
>
> When running a nodetool repair, the following exception can be thrown:
> ERROR [AntiEntropySessions:12] 2011-10-24 11:17:52,154 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[AntiEntropySessions:12,5,RMI Runtime]
> java.lang.RuntimeException: java.util.ConcurrentModificationException
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
> at java.util.HashMap$KeyIterator.next(HashMap.java:828)
> at org.apache.cassandra.service.AntiEntropyService$RepairSession$RepairJob.sendTreeRequests(AntiEntropyService.java:784)
> at org.apache.cassandra.service.AntiEntropyService$RepairSession.runMayThrow(AntiEntropyService.java:680)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> ... 6 more
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3400) ConcurrentModificationException
during nodetool repair
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13135071#comment-13135071 ]
Jonathan Ellis commented on CASSANDRA-3400:
-------------------------------------------
+1
nit: we never use thread.interrupt() so custom is to throw AssertionError for InterruptedException
> ConcurrentModificationException during nodetool repair
> ------------------------------------------------------
>
> Key: CASSANDRA-3400
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3400
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.7, 1.0.0
> Environment: Suse Enterprise linux 11.4
> Reporter: Scott Fines
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 0.8.8, 1.0.1
>
> Attachments: 3400.patch
>
>
> When running a nodetool repair, the following exception can be thrown:
> ERROR [AntiEntropySessions:12] 2011-10-24 11:17:52,154 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[AntiEntropySessions:12,5,RMI Runtime]
> java.lang.RuntimeException: java.util.ConcurrentModificationException
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
> at java.util.HashMap$KeyIterator.next(HashMap.java:828)
> at org.apache.cassandra.service.AntiEntropyService$RepairSession$RepairJob.sendTreeRequests(AntiEntropyService.java:784)
> at org.apache.cassandra.service.AntiEntropyService$RepairSession.runMayThrow(AntiEntropyService.java:680)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> ... 6 more
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (CASSANDRA-3400) ConcurrentModificationException
during nodetool repair
Posted by "Sylvain Lebresne (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sylvain Lebresne reassigned CASSANDRA-3400:
-------------------------------------------
Assignee: Sylvain Lebresne
> ConcurrentModificationException during nodetool repair
> ------------------------------------------------------
>
> Key: CASSANDRA-3400
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3400
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.7
> Environment: Suse Enterprise linux 11.4
> Reporter: Scott Fines
> Assignee: Sylvain Lebresne
> Priority: Minor
>
> When running a nodetool repair, the following exception can be thrown:
> ERROR [AntiEntropySessions:12] 2011-10-24 11:17:52,154 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[AntiEntropySessions:12,5,RMI Runtime]
> java.lang.RuntimeException: java.util.ConcurrentModificationException
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
> at java.util.HashMap$KeyIterator.next(HashMap.java:828)
> at org.apache.cassandra.service.AntiEntropyService$RepairSession$RepairJob.sendTreeRequests(AntiEntropyService.java:784)
> at org.apache.cassandra.service.AntiEntropyService$RepairSession.runMayThrow(AntiEntropyService.java:680)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> ... 6 more
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira