You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Terje Marthinussen (JIRA)" <ji...@apache.org> on 2011/06/14 09:14:53 UTC
[jira] [Created] (CASSANDRA-2766) ConcurrentModificationException
during node recovery
ConcurrentModificationException during node recovery
----------------------------------------------------
Key: CASSANDRA-2766
URL: https://issues.apache.org/jira/browse/CASSANDRA-2766
Project: Cassandra
Issue Type: Bug
Components: Core
Reporter: Terje Marthinussen
Testing some node recovery operations.
In this case:
1. Data is being added/updated as it would in production
2. repair is running on other nodes in the cluster
3. we wiped data on this node and started up again, but before repair was actually started on this node (but it had gotten data through the regular data feed) we got this error.
I see no indication in the logs that outgoing streams has been started, but the node have finished one incoming stream before this (I guess from some other node doing repair).
INFO [CompactionExecutor:11] 2011-06-14 14:15:09,078 SSTableReader.java (line 155) Opening /data/cassandra/node1/data/JP/test-g-8
INFO [CompactionExecutor:13] 2011-06-14 14:15:09,079 SSTableReader.java (line 155) Opening /data/cassandra/node1/data/JP/test-g-10
INFO [HintedHandoff:1] 2011-06-14 14:15:26,623 HintedHandOffManager.java (line 302) Started hinted handoff for endpoint /1.10.42.216
INFO [HintedHandoff:1] 2011-06-14 14:15:26,623 HintedHandOffManager.java (line 358) Finished hinted handoff of 0 rows to endpoint /1.10.42.216
INFO [CompactionExecutor:9] 2011-06-14 14:15:29,417 SSTableReader.java (line 155) Opening /data/cassandra/node1/data/JP/Datetest-g-2
ERROR [Thread-84] 2011-06-14 14:15:36,755 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-84,5,main]
java.util.ConcurrentModificationException
at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
at java.util.AbstractList$Itr.next(AbstractList.java:343)
at org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:132)
at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
ERROR [Thread-79] 2011-06-14 14:15:36,755 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-79,5,main]
java.util.ConcurrentModificationException
at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
at java.util.AbstractList$Itr.next(AbstractList.java:343)
at org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:132)
at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
ERROR [Thread-83] 2011-06-14 14:15:36,755 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-83,5,main]
java.util.ConcurrentModificationException
at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
at java.util.AbstractList$Itr.next(AbstractList.java:343)
at org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:132)
at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
ERROR [Thread-85] 2011-06-14 14:15:36,755 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-85,5,main]
java.util.ConcurrentModificationException
at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
at java.util.AbstractList$Itr.next(AbstractList.java:343)
at org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:132)
at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2766) ConcurrentModificationException
during node recovery
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049273#comment-13049273 ]
Hudson commented on CASSANDRA-2766:
-----------------------------------
Integrated in Cassandra-0.8 #170 (See [https://builds.apache.org/job/Cassandra-0.8/170/])
use threadsafe collections for StreamInSession
patch by jbellis; reviewed by slebresne for CASSANDRA-2766
jbellis : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1135611
Files :
* /cassandra/branches/cassandra-0.8/CHANGES.txt
* /cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/streaming/StreamInSession.java
> ConcurrentModificationException during node recovery
> ----------------------------------------------------
>
> Key: CASSANDRA-2766
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2766
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Terje Marthinussen
> Assignee: Jonathan Ellis
> Attachments: 2766-v2.txt, 2766.txt
>
>
> Testing some node recovery operations.
> In this case:
> 1. Data is being added/updated as it would in production
> 2. repair is running on other nodes in the cluster
> 3. we wiped data on this node and started up again, but before repair was actually started on this node (but it had gotten data through the regular data feed) we got this error.
> I see no indication in the logs that outgoing streams has been started, but the node have finished one incoming stream before this (I guess from some other node doing repair).
> INFO [CompactionExecutor:11] 2011-06-14 14:15:09,078 SSTableReader.java (line 155) Opening /data/cassandra/node1/data/JP/test-g-8
> INFO [CompactionExecutor:13] 2011-06-14 14:15:09,079 SSTableReader.java (line 155) Opening /data/cassandra/node1/data/JP/test-g-10
> INFO [HintedHandoff:1] 2011-06-14 14:15:26,623 HintedHandOffManager.java (line 302) Started hinted handoff for endpoint /1.10.42.216
> INFO [HintedHandoff:1] 2011-06-14 14:15:26,623 HintedHandOffManager.java (line 358) Finished hinted handoff of 0 rows to endpoint /1.10.42.216
> INFO [CompactionExecutor:9] 2011-06-14 14:15:29,417 SSTableReader.java (line 155) Opening /data/cassandra/node1/data/JP/Datetest-g-2
> ERROR [Thread-84] 2011-06-14 14:15:36,755 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-84,5,main]
> java.util.ConcurrentModificationException
> at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
> at java.util.AbstractList$Itr.next(AbstractList.java:343)
> at org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:132)
> at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
> at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
> ERROR [Thread-79] 2011-06-14 14:15:36,755 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-79,5,main]
> java.util.ConcurrentModificationException
> at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
> at java.util.AbstractList$Itr.next(AbstractList.java:343)
> at org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:132)
> at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
> at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
> ERROR [Thread-83] 2011-06-14 14:15:36,755 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-83,5,main]
> java.util.ConcurrentModificationException
> at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
> at java.util.AbstractList$Itr.next(AbstractList.java:343)
> at org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:132)
> at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
> at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
> ERROR [Thread-85] 2011-06-14 14:15:36,755 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-85,5,main]
> java.util.ConcurrentModificationException
> at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
> at java.util.AbstractList$Itr.next(AbstractList.java:343)
> at org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:132)
> at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
> at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2766) ConcurrentModificationException
during node recovery
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2766:
--------------------------------------
Attachment: 2766.txt
looks like a straightforward case of "not using threadsafe collections means there is no happens-before guarantee where a naive reading of the code would expect one" (i.e., since buildFutures.add is always called before files.remove, after files.isEmpty is true there should be no more changes to buildFutures.add).
patch attached that changes both of these from Arraylist to CSLS.
> ConcurrentModificationException during node recovery
> ----------------------------------------------------
>
> Key: CASSANDRA-2766
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2766
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Terje Marthinussen
> Assignee: Jonathan Ellis
> Attachments: 2766.txt
>
>
> Testing some node recovery operations.
> In this case:
> 1. Data is being added/updated as it would in production
> 2. repair is running on other nodes in the cluster
> 3. we wiped data on this node and started up again, but before repair was actually started on this node (but it had gotten data through the regular data feed) we got this error.
> I see no indication in the logs that outgoing streams has been started, but the node have finished one incoming stream before this (I guess from some other node doing repair).
> INFO [CompactionExecutor:11] 2011-06-14 14:15:09,078 SSTableReader.java (line 155) Opening /data/cassandra/node1/data/JP/test-g-8
> INFO [CompactionExecutor:13] 2011-06-14 14:15:09,079 SSTableReader.java (line 155) Opening /data/cassandra/node1/data/JP/test-g-10
> INFO [HintedHandoff:1] 2011-06-14 14:15:26,623 HintedHandOffManager.java (line 302) Started hinted handoff for endpoint /1.10.42.216
> INFO [HintedHandoff:1] 2011-06-14 14:15:26,623 HintedHandOffManager.java (line 358) Finished hinted handoff of 0 rows to endpoint /1.10.42.216
> INFO [CompactionExecutor:9] 2011-06-14 14:15:29,417 SSTableReader.java (line 155) Opening /data/cassandra/node1/data/JP/Datetest-g-2
> ERROR [Thread-84] 2011-06-14 14:15:36,755 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-84,5,main]
> java.util.ConcurrentModificationException
> at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
> at java.util.AbstractList$Itr.next(AbstractList.java:343)
> at org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:132)
> at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
> at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
> ERROR [Thread-79] 2011-06-14 14:15:36,755 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-79,5,main]
> java.util.ConcurrentModificationException
> at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
> at java.util.AbstractList$Itr.next(AbstractList.java:343)
> at org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:132)
> at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
> at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
> ERROR [Thread-83] 2011-06-14 14:15:36,755 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-83,5,main]
> java.util.ConcurrentModificationException
> at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
> at java.util.AbstractList$Itr.next(AbstractList.java:343)
> at org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:132)
> at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
> at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
> ERROR [Thread-85] 2011-06-14 14:15:36,755 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-85,5,main]
> java.util.ConcurrentModificationException
> at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
> at java.util.AbstractList$Itr.next(AbstractList.java:343)
> at org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:132)
> at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
> at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2766) ConcurrentModificationException
during node recovery
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049191#comment-13049191 ]
Sylvain Lebresne commented on CASSANDRA-2766:
---------------------------------------------
lgtm +1 on v2
> ConcurrentModificationException during node recovery
> ----------------------------------------------------
>
> Key: CASSANDRA-2766
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2766
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Terje Marthinussen
> Assignee: Jonathan Ellis
> Attachments: 2766-v2.txt, 2766.txt
>
>
> Testing some node recovery operations.
> In this case:
> 1. Data is being added/updated as it would in production
> 2. repair is running on other nodes in the cluster
> 3. we wiped data on this node and started up again, but before repair was actually started on this node (but it had gotten data through the regular data feed) we got this error.
> I see no indication in the logs that outgoing streams has been started, but the node have finished one incoming stream before this (I guess from some other node doing repair).
> INFO [CompactionExecutor:11] 2011-06-14 14:15:09,078 SSTableReader.java (line 155) Opening /data/cassandra/node1/data/JP/test-g-8
> INFO [CompactionExecutor:13] 2011-06-14 14:15:09,079 SSTableReader.java (line 155) Opening /data/cassandra/node1/data/JP/test-g-10
> INFO [HintedHandoff:1] 2011-06-14 14:15:26,623 HintedHandOffManager.java (line 302) Started hinted handoff for endpoint /1.10.42.216
> INFO [HintedHandoff:1] 2011-06-14 14:15:26,623 HintedHandOffManager.java (line 358) Finished hinted handoff of 0 rows to endpoint /1.10.42.216
> INFO [CompactionExecutor:9] 2011-06-14 14:15:29,417 SSTableReader.java (line 155) Opening /data/cassandra/node1/data/JP/Datetest-g-2
> ERROR [Thread-84] 2011-06-14 14:15:36,755 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-84,5,main]
> java.util.ConcurrentModificationException
> at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
> at java.util.AbstractList$Itr.next(AbstractList.java:343)
> at org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:132)
> at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
> at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
> ERROR [Thread-79] 2011-06-14 14:15:36,755 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-79,5,main]
> java.util.ConcurrentModificationException
> at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
> at java.util.AbstractList$Itr.next(AbstractList.java:343)
> at org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:132)
> at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
> at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
> ERROR [Thread-83] 2011-06-14 14:15:36,755 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-83,5,main]
> java.util.ConcurrentModificationException
> at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
> at java.util.AbstractList$Itr.next(AbstractList.java:343)
> at org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:132)
> at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
> at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
> ERROR [Thread-85] 2011-06-14 14:15:36,755 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-85,5,main]
> java.util.ConcurrentModificationException
> at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
> at java.util.AbstractList$Itr.next(AbstractList.java:343)
> at org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:132)
> at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
> at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2766) ConcurrentModificationException
during node recovery
Posted by "Muga Nishizawa (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049170#comment-13049170 ]
Muga Nishizawa commented on CASSANDRA-2766:
-------------------------------------------
Thanks for your quick response. But we've encountered the following exception.
ERROR [Thread-18] 2011-06-14 21:54:04,065 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-18,5,main]
java.lang.ClassCastException: org.apache.cassandra.streaming.PendingFile cannot be cast to java.lang.Comparable
at java.util.concurrent.ConcurrentSkipListMap.comparable(ConcurrentSkipListMap.java:621)
at java.util.concurrent.ConcurrentSkipListMap.doPut(ConcurrentSkipListMap.java:862)
at java.util.concurrent.ConcurrentSkipListMap.putIfAbsent(ConcurrentSkipListMap.java:1893)
at java.util.concurrent.ConcurrentSkipListSet.add(ConcurrentSkipListSet.java:202)
at org.apache.cassandra.streaming.StreamInSession.addFiles(StreamInSession.java:100)
at org.apache.cassandra.streaming.IncomingStreamReader.<init>(IncomingStreamReader.java:49)
at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
> ConcurrentModificationException during node recovery
> ----------------------------------------------------
>
> Key: CASSANDRA-2766
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2766
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Terje Marthinussen
> Assignee: Jonathan Ellis
> Attachments: 2766.txt
>
>
> Testing some node recovery operations.
> In this case:
> 1. Data is being added/updated as it would in production
> 2. repair is running on other nodes in the cluster
> 3. we wiped data on this node and started up again, but before repair was actually started on this node (but it had gotten data through the regular data feed) we got this error.
> I see no indication in the logs that outgoing streams has been started, but the node have finished one incoming stream before this (I guess from some other node doing repair).
> INFO [CompactionExecutor:11] 2011-06-14 14:15:09,078 SSTableReader.java (line 155) Opening /data/cassandra/node1/data/JP/test-g-8
> INFO [CompactionExecutor:13] 2011-06-14 14:15:09,079 SSTableReader.java (line 155) Opening /data/cassandra/node1/data/JP/test-g-10
> INFO [HintedHandoff:1] 2011-06-14 14:15:26,623 HintedHandOffManager.java (line 302) Started hinted handoff for endpoint /1.10.42.216
> INFO [HintedHandoff:1] 2011-06-14 14:15:26,623 HintedHandOffManager.java (line 358) Finished hinted handoff of 0 rows to endpoint /1.10.42.216
> INFO [CompactionExecutor:9] 2011-06-14 14:15:29,417 SSTableReader.java (line 155) Opening /data/cassandra/node1/data/JP/Datetest-g-2
> ERROR [Thread-84] 2011-06-14 14:15:36,755 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-84,5,main]
> java.util.ConcurrentModificationException
> at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
> at java.util.AbstractList$Itr.next(AbstractList.java:343)
> at org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:132)
> at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
> at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
> ERROR [Thread-79] 2011-06-14 14:15:36,755 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-79,5,main]
> java.util.ConcurrentModificationException
> at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
> at java.util.AbstractList$Itr.next(AbstractList.java:343)
> at org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:132)
> at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
> at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
> ERROR [Thread-83] 2011-06-14 14:15:36,755 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-83,5,main]
> java.util.ConcurrentModificationException
> at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
> at java.util.AbstractList$Itr.next(AbstractList.java:343)
> at org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:132)
> at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
> at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
> ERROR [Thread-85] 2011-06-14 14:15:36,755 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-85,5,main]
> java.util.ConcurrentModificationException
> at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
> at java.util.AbstractList$Itr.next(AbstractList.java:343)
> at org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:132)
> at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
> at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CASSANDRA-2766) ConcurrentModificationException
during node recovery
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis resolved CASSANDRA-2766.
---------------------------------------
Resolution: Fixed
Fix Version/s: 0.8.1
Reviewer: slebresne
(committed)
> ConcurrentModificationException during node recovery
> ----------------------------------------------------
>
> Key: CASSANDRA-2766
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2766
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Terje Marthinussen
> Assignee: Jonathan Ellis
> Fix For: 0.8.1
>
> Attachments: 2766-v2.txt, 2766.txt
>
>
> Testing some node recovery operations.
> In this case:
> 1. Data is being added/updated as it would in production
> 2. repair is running on other nodes in the cluster
> 3. we wiped data on this node and started up again, but before repair was actually started on this node (but it had gotten data through the regular data feed) we got this error.
> I see no indication in the logs that outgoing streams has been started, but the node have finished one incoming stream before this (I guess from some other node doing repair).
> INFO [CompactionExecutor:11] 2011-06-14 14:15:09,078 SSTableReader.java (line 155) Opening /data/cassandra/node1/data/JP/test-g-8
> INFO [CompactionExecutor:13] 2011-06-14 14:15:09,079 SSTableReader.java (line 155) Opening /data/cassandra/node1/data/JP/test-g-10
> INFO [HintedHandoff:1] 2011-06-14 14:15:26,623 HintedHandOffManager.java (line 302) Started hinted handoff for endpoint /1.10.42.216
> INFO [HintedHandoff:1] 2011-06-14 14:15:26,623 HintedHandOffManager.java (line 358) Finished hinted handoff of 0 rows to endpoint /1.10.42.216
> INFO [CompactionExecutor:9] 2011-06-14 14:15:29,417 SSTableReader.java (line 155) Opening /data/cassandra/node1/data/JP/Datetest-g-2
> ERROR [Thread-84] 2011-06-14 14:15:36,755 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-84,5,main]
> java.util.ConcurrentModificationException
> at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
> at java.util.AbstractList$Itr.next(AbstractList.java:343)
> at org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:132)
> at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
> at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
> ERROR [Thread-79] 2011-06-14 14:15:36,755 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-79,5,main]
> java.util.ConcurrentModificationException
> at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
> at java.util.AbstractList$Itr.next(AbstractList.java:343)
> at org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:132)
> at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
> at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
> ERROR [Thread-83] 2011-06-14 14:15:36,755 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-83,5,main]
> java.util.ConcurrentModificationException
> at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
> at java.util.AbstractList$Itr.next(AbstractList.java:343)
> at org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:132)
> at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
> at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
> ERROR [Thread-85] 2011-06-14 14:15:36,755 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-85,5,main]
> java.util.ConcurrentModificationException
> at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
> at java.util.AbstractList$Itr.next(AbstractList.java:343)
> at org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:132)
> at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
> at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (CASSANDRA-2766) ConcurrentModificationException
during node recovery
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis reassigned CASSANDRA-2766:
-----------------------------------------
Assignee: Jonathan Ellis
> ConcurrentModificationException during node recovery
> ----------------------------------------------------
>
> Key: CASSANDRA-2766
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2766
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Terje Marthinussen
> Assignee: Jonathan Ellis
>
> Testing some node recovery operations.
> In this case:
> 1. Data is being added/updated as it would in production
> 2. repair is running on other nodes in the cluster
> 3. we wiped data on this node and started up again, but before repair was actually started on this node (but it had gotten data through the regular data feed) we got this error.
> I see no indication in the logs that outgoing streams has been started, but the node have finished one incoming stream before this (I guess from some other node doing repair).
> INFO [CompactionExecutor:11] 2011-06-14 14:15:09,078 SSTableReader.java (line 155) Opening /data/cassandra/node1/data/JP/test-g-8
> INFO [CompactionExecutor:13] 2011-06-14 14:15:09,079 SSTableReader.java (line 155) Opening /data/cassandra/node1/data/JP/test-g-10
> INFO [HintedHandoff:1] 2011-06-14 14:15:26,623 HintedHandOffManager.java (line 302) Started hinted handoff for endpoint /1.10.42.216
> INFO [HintedHandoff:1] 2011-06-14 14:15:26,623 HintedHandOffManager.java (line 358) Finished hinted handoff of 0 rows to endpoint /1.10.42.216
> INFO [CompactionExecutor:9] 2011-06-14 14:15:29,417 SSTableReader.java (line 155) Opening /data/cassandra/node1/data/JP/Datetest-g-2
> ERROR [Thread-84] 2011-06-14 14:15:36,755 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-84,5,main]
> java.util.ConcurrentModificationException
> at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
> at java.util.AbstractList$Itr.next(AbstractList.java:343)
> at org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:132)
> at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
> at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
> ERROR [Thread-79] 2011-06-14 14:15:36,755 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-79,5,main]
> java.util.ConcurrentModificationException
> at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
> at java.util.AbstractList$Itr.next(AbstractList.java:343)
> at org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:132)
> at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
> at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
> ERROR [Thread-83] 2011-06-14 14:15:36,755 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-83,5,main]
> java.util.ConcurrentModificationException
> at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
> at java.util.AbstractList$Itr.next(AbstractList.java:343)
> at org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:132)
> at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
> at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
> ERROR [Thread-85] 2011-06-14 14:15:36,755 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-85,5,main]
> java.util.ConcurrentModificationException
> at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
> at java.util.AbstractList$Itr.next(AbstractList.java:343)
> at org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:132)
> at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
> at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2766) ConcurrentModificationException
during node recovery
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2766:
--------------------------------------
Attachment: 2766-v2.txt
oops. v2 w/ collections that don't require a comparable.
> ConcurrentModificationException during node recovery
> ----------------------------------------------------
>
> Key: CASSANDRA-2766
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2766
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Terje Marthinussen
> Assignee: Jonathan Ellis
> Attachments: 2766-v2.txt, 2766.txt
>
>
> Testing some node recovery operations.
> In this case:
> 1. Data is being added/updated as it would in production
> 2. repair is running on other nodes in the cluster
> 3. we wiped data on this node and started up again, but before repair was actually started on this node (but it had gotten data through the regular data feed) we got this error.
> I see no indication in the logs that outgoing streams has been started, but the node have finished one incoming stream before this (I guess from some other node doing repair).
> INFO [CompactionExecutor:11] 2011-06-14 14:15:09,078 SSTableReader.java (line 155) Opening /data/cassandra/node1/data/JP/test-g-8
> INFO [CompactionExecutor:13] 2011-06-14 14:15:09,079 SSTableReader.java (line 155) Opening /data/cassandra/node1/data/JP/test-g-10
> INFO [HintedHandoff:1] 2011-06-14 14:15:26,623 HintedHandOffManager.java (line 302) Started hinted handoff for endpoint /1.10.42.216
> INFO [HintedHandoff:1] 2011-06-14 14:15:26,623 HintedHandOffManager.java (line 358) Finished hinted handoff of 0 rows to endpoint /1.10.42.216
> INFO [CompactionExecutor:9] 2011-06-14 14:15:29,417 SSTableReader.java (line 155) Opening /data/cassandra/node1/data/JP/Datetest-g-2
> ERROR [Thread-84] 2011-06-14 14:15:36,755 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-84,5,main]
> java.util.ConcurrentModificationException
> at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
> at java.util.AbstractList$Itr.next(AbstractList.java:343)
> at org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:132)
> at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
> at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
> ERROR [Thread-79] 2011-06-14 14:15:36,755 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-79,5,main]
> java.util.ConcurrentModificationException
> at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
> at java.util.AbstractList$Itr.next(AbstractList.java:343)
> at org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:132)
> at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
> at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
> ERROR [Thread-83] 2011-06-14 14:15:36,755 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-83,5,main]
> java.util.ConcurrentModificationException
> at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
> at java.util.AbstractList$Itr.next(AbstractList.java:343)
> at org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:132)
> at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
> at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
> ERROR [Thread-85] 2011-06-14 14:15:36,755 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-85,5,main]
> java.util.ConcurrentModificationException
> at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
> at java.util.AbstractList$Itr.next(AbstractList.java:343)
> at org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:132)
> at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
> at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira