You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Jonathan Ellis (JIRA)" <ji...@apache.org> on 2011/06/14 11:38:47 UTC
[jira] [Updated] (CASSANDRA-2766) ConcurrentModificationException
during node recovery
[ https://issues.apache.org/jira/browse/CASSANDRA-2766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2766:
--------------------------------------
Attachment: 2766.txt
looks like a straightforward case of "not using threadsafe collections means there is no happens-before guarantee where a naive reading of the code would expect one" (i.e., since buildFutures.add is always called before files.remove, after files.isEmpty is true there should be no more changes to buildFutures.add).
patch attached that changes both of these from Arraylist to CSLS.
> ConcurrentModificationException during node recovery
> ----------------------------------------------------
>
> Key: CASSANDRA-2766
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2766
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Terje Marthinussen
> Assignee: Jonathan Ellis
> Attachments: 2766.txt
>
>
> Testing some node recovery operations.
> In this case:
> 1. Data is being added/updated as it would in production
> 2. repair is running on other nodes in the cluster
> 3. we wiped data on this node and started up again, but before repair was actually started on this node (but it had gotten data through the regular data feed) we got this error.
> I see no indication in the logs that outgoing streams has been started, but the node have finished one incoming stream before this (I guess from some other node doing repair).
> INFO [CompactionExecutor:11] 2011-06-14 14:15:09,078 SSTableReader.java (line 155) Opening /data/cassandra/node1/data/JP/test-g-8
> INFO [CompactionExecutor:13] 2011-06-14 14:15:09,079 SSTableReader.java (line 155) Opening /data/cassandra/node1/data/JP/test-g-10
> INFO [HintedHandoff:1] 2011-06-14 14:15:26,623 HintedHandOffManager.java (line 302) Started hinted handoff for endpoint /1.10.42.216
> INFO [HintedHandoff:1] 2011-06-14 14:15:26,623 HintedHandOffManager.java (line 358) Finished hinted handoff of 0 rows to endpoint /1.10.42.216
> INFO [CompactionExecutor:9] 2011-06-14 14:15:29,417 SSTableReader.java (line 155) Opening /data/cassandra/node1/data/JP/Datetest-g-2
> ERROR [Thread-84] 2011-06-14 14:15:36,755 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-84,5,main]
> java.util.ConcurrentModificationException
> at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
> at java.util.AbstractList$Itr.next(AbstractList.java:343)
> at org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:132)
> at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
> at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
> ERROR [Thread-79] 2011-06-14 14:15:36,755 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-79,5,main]
> java.util.ConcurrentModificationException
> at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
> at java.util.AbstractList$Itr.next(AbstractList.java:343)
> at org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:132)
> at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
> at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
> ERROR [Thread-83] 2011-06-14 14:15:36,755 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-83,5,main]
> java.util.ConcurrentModificationException
> at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
> at java.util.AbstractList$Itr.next(AbstractList.java:343)
> at org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:132)
> at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
> at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
> ERROR [Thread-85] 2011-06-14 14:15:36,755 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-85,5,main]
> java.util.ConcurrentModificationException
> at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
> at java.util.AbstractList$Itr.next(AbstractList.java:343)
> at org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:132)
> at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:63)
> at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira