You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Thibaut (Created) (JIRA)" <ji...@apache.org> on 2012/03/08 11:48:57 UTC
[jira] [Created] (CASSANDRA-4019)
java.util.ConcurrentModificationException in Gossiper
java.util.ConcurrentModificationException in Gossiper
-----------------------------------------------------
Key: CASSANDRA-4019
URL: https://issues.apache.org/jira/browse/CASSANDRA-4019
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 0.8.9
Reporter: Thibaut
Priority: Minor
I have never seen this one before. Might be triggered by a race condition under heavy load. This error was triggered on 0.8.9
ERROR [GossipTasks:1] 2012-03-05 04:16:55,263 Gossiper.java (line 162) Gossip error
java.util.ConcurrentModificationException
at java.util.ArrayDeque$DeqIterator.next(ArrayDeque.java:605)
at org.apache.cassandra.utils.AbstractStatsDeque.sum(AbstractStatsDeque.java:37)
at org.apache.cassandra.utils.AbstractStatsDeque.mean(AbstractStatsDeque.java:60)
at org.apache.cassandra.gms.ArrivalWindow.mean(FailureDetector.java:259)
at org.apache.cassandra.gms.ArrivalWindow.phi(FailureDetector.java:282)
at org.apache.cassandra.gms.FailureDetector.interpret(FailureDetector.java:155)
at org.apache.cassandra.gms.Gossiper.doStatusCheck(Gossiper.java:538)
at org.apache.cassandra.gms.Gossiper.access$700(Gossiper.java:57)
at org.apache.cassandra.gms.Gossiper$GossipTask.run(Gossiper.java:157)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
INFO [GossipStage:1] 2012-03-05 04:16:55,263 Gossiper.java (line 737) Node /192.168.3.18 has restarted, now UP again
INFO [GossipStage:1] 2012-03-05 04:16:55,264 Gossiper.java (line 705) InetAddress /192.168.3.18 is now UP
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4019)
java.util.ConcurrentModificationException in Gossiper
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225670#comment-13225670 ]
Jonathan Ellis commented on CASSANDRA-4019:
-------------------------------------------
+1 on both
> java.util.ConcurrentModificationException in Gossiper
> -----------------------------------------------------
>
> Key: CASSANDRA-4019
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4019
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.9
> Reporter: Thibaut
> Assignee: Brandon Williams
> Priority: Minor
> Fix For: 0.8.11, 1.0.9, 1.1.0
>
> Attachments: 4019-trunk.txt, 4019.txt
>
>
> I have never seen this one before. Might be triggered by a race condition under heavy load. This error was triggered on 0.8.9
> ERROR [GossipTasks:1] 2012-03-05 04:16:55,263 Gossiper.java (line 162) Gossip error
> java.util.ConcurrentModificationException
> at java.util.ArrayDeque$DeqIterator.next(ArrayDeque.java:605)
> at org.apache.cassandra.utils.AbstractStatsDeque.sum(AbstractStatsDeque.java:37)
> at org.apache.cassandra.utils.AbstractStatsDeque.mean(AbstractStatsDeque.java:60)
> at org.apache.cassandra.gms.ArrivalWindow.mean(FailureDetector.java:259)
> at org.apache.cassandra.gms.ArrivalWindow.phi(FailureDetector.java:282)
> at org.apache.cassandra.gms.FailureDetector.interpret(FailureDetector.java:155)
> at org.apache.cassandra.gms.Gossiper.doStatusCheck(Gossiper.java:538)
> at org.apache.cassandra.gms.Gossiper.access$700(Gossiper.java:57)
> at org.apache.cassandra.gms.Gossiper$GossipTask.run(Gossiper.java:157)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> INFO [GossipStage:1] 2012-03-05 04:16:55,263 Gossiper.java (line 737) Node /192.168.3.18 has restarted, now UP again
> INFO [GossipStage:1] 2012-03-05 04:16:55,264 Gossiper.java (line 705) InetAddress /192.168.3.18 is now UP
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (CASSANDRA-4019)
java.util.ConcurrentModificationException in Gossiper
Posted by "Brandon Williams (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams reassigned CASSANDRA-4019:
-------------------------------------------
Assignee: Brandon Williams
> java.util.ConcurrentModificationException in Gossiper
> -----------------------------------------------------
>
> Key: CASSANDRA-4019
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4019
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.9
> Reporter: Thibaut
> Assignee: Brandon Williams
> Priority: Minor
> Fix For: 0.8.11
>
> Attachments: 4019.txt
>
>
> I have never seen this one before. Might be triggered by a race condition under heavy load. This error was triggered on 0.8.9
> ERROR [GossipTasks:1] 2012-03-05 04:16:55,263 Gossiper.java (line 162) Gossip error
> java.util.ConcurrentModificationException
> at java.util.ArrayDeque$DeqIterator.next(ArrayDeque.java:605)
> at org.apache.cassandra.utils.AbstractStatsDeque.sum(AbstractStatsDeque.java:37)
> at org.apache.cassandra.utils.AbstractStatsDeque.mean(AbstractStatsDeque.java:60)
> at org.apache.cassandra.gms.ArrivalWindow.mean(FailureDetector.java:259)
> at org.apache.cassandra.gms.ArrivalWindow.phi(FailureDetector.java:282)
> at org.apache.cassandra.gms.FailureDetector.interpret(FailureDetector.java:155)
> at org.apache.cassandra.gms.Gossiper.doStatusCheck(Gossiper.java:538)
> at org.apache.cassandra.gms.Gossiper.access$700(Gossiper.java:57)
> at org.apache.cassandra.gms.Gossiper$GossipTask.run(Gossiper.java:157)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> INFO [GossipStage:1] 2012-03-05 04:16:55,263 Gossiper.java (line 737) Node /192.168.3.18 has restarted, now UP again
> INFO [GossipStage:1] 2012-03-05 04:16:55,264 Gossiper.java (line 705) InetAddress /192.168.3.18 is now UP
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4019)
java.util.ConcurrentModificationException in Gossiper
Posted by "Thibaut (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thibaut updated CASSANDRA-4019:
-------------------------------
Fix Version/s: 0.8.11
> java.util.ConcurrentModificationException in Gossiper
> -----------------------------------------------------
>
> Key: CASSANDRA-4019
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4019
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.9
> Reporter: Thibaut
> Priority: Minor
> Fix For: 0.8.11
>
>
> I have never seen this one before. Might be triggered by a race condition under heavy load. This error was triggered on 0.8.9
> ERROR [GossipTasks:1] 2012-03-05 04:16:55,263 Gossiper.java (line 162) Gossip error
> java.util.ConcurrentModificationException
> at java.util.ArrayDeque$DeqIterator.next(ArrayDeque.java:605)
> at org.apache.cassandra.utils.AbstractStatsDeque.sum(AbstractStatsDeque.java:37)
> at org.apache.cassandra.utils.AbstractStatsDeque.mean(AbstractStatsDeque.java:60)
> at org.apache.cassandra.gms.ArrivalWindow.mean(FailureDetector.java:259)
> at org.apache.cassandra.gms.ArrivalWindow.phi(FailureDetector.java:282)
> at org.apache.cassandra.gms.FailureDetector.interpret(FailureDetector.java:155)
> at org.apache.cassandra.gms.Gossiper.doStatusCheck(Gossiper.java:538)
> at org.apache.cassandra.gms.Gossiper.access$700(Gossiper.java:57)
> at org.apache.cassandra.gms.Gossiper$GossipTask.run(Gossiper.java:157)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> INFO [GossipStage:1] 2012-03-05 04:16:55,263 Gossiper.java (line 737) Node /192.168.3.18 has restarted, now UP again
> INFO [GossipStage:1] 2012-03-05 04:16:55,264 Gossiper.java (line 705) InetAddress /192.168.3.18 is now UP
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4019)
java.util.ConcurrentModificationException in Gossiper
Posted by "Brandon Williams (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225255#comment-13225255 ]
Brandon Williams commented on CASSANDRA-4019:
---------------------------------------------
It seems like the Right Way to solve this is to refactor BoundedStatsDeque to operate like the dsnitch's AdapativeLatencyTracker, which *is* threadsafe, and then have ALT extend BSD. BSD isn't used anywhere else except the FD.
> java.util.ConcurrentModificationException in Gossiper
> -----------------------------------------------------
>
> Key: CASSANDRA-4019
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4019
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.9
> Reporter: Thibaut
> Priority: Minor
> Fix For: 0.8.11
>
>
> I have never seen this one before. Might be triggered by a race condition under heavy load. This error was triggered on 0.8.9
> ERROR [GossipTasks:1] 2012-03-05 04:16:55,263 Gossiper.java (line 162) Gossip error
> java.util.ConcurrentModificationException
> at java.util.ArrayDeque$DeqIterator.next(ArrayDeque.java:605)
> at org.apache.cassandra.utils.AbstractStatsDeque.sum(AbstractStatsDeque.java:37)
> at org.apache.cassandra.utils.AbstractStatsDeque.mean(AbstractStatsDeque.java:60)
> at org.apache.cassandra.gms.ArrivalWindow.mean(FailureDetector.java:259)
> at org.apache.cassandra.gms.ArrivalWindow.phi(FailureDetector.java:282)
> at org.apache.cassandra.gms.FailureDetector.interpret(FailureDetector.java:155)
> at org.apache.cassandra.gms.Gossiper.doStatusCheck(Gossiper.java:538)
> at org.apache.cassandra.gms.Gossiper.access$700(Gossiper.java:57)
> at org.apache.cassandra.gms.Gossiper$GossipTask.run(Gossiper.java:157)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> INFO [GossipStage:1] 2012-03-05 04:16:55,263 Gossiper.java (line 737) Node /192.168.3.18 has restarted, now UP again
> INFO [GossipStage:1] 2012-03-05 04:16:55,264 Gossiper.java (line 705) InetAddress /192.168.3.18 is now UP
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4019)
java.util.ConcurrentModificationException in Gossiper
Posted by "Brandon Williams (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams updated CASSANDRA-4019:
----------------------------------------
Attachment: 4019-trunk.txt
Patch for trunk which removes ASD since all we actually use is sum and mean and consolidates it into BDS. Dsnitch's ALT is also removed since BDS itself contains all the functionality it had before.
> java.util.ConcurrentModificationException in Gossiper
> -----------------------------------------------------
>
> Key: CASSANDRA-4019
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4019
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.9
> Reporter: Thibaut
> Assignee: Brandon Williams
> Priority: Minor
> Fix For: 0.8.11
>
> Attachments: 4019-trunk.txt, 4019.txt
>
>
> I have never seen this one before. Might be triggered by a race condition under heavy load. This error was triggered on 0.8.9
> ERROR [GossipTasks:1] 2012-03-05 04:16:55,263 Gossiper.java (line 162) Gossip error
> java.util.ConcurrentModificationException
> at java.util.ArrayDeque$DeqIterator.next(ArrayDeque.java:605)
> at org.apache.cassandra.utils.AbstractStatsDeque.sum(AbstractStatsDeque.java:37)
> at org.apache.cassandra.utils.AbstractStatsDeque.mean(AbstractStatsDeque.java:60)
> at org.apache.cassandra.gms.ArrivalWindow.mean(FailureDetector.java:259)
> at org.apache.cassandra.gms.ArrivalWindow.phi(FailureDetector.java:282)
> at org.apache.cassandra.gms.FailureDetector.interpret(FailureDetector.java:155)
> at org.apache.cassandra.gms.Gossiper.doStatusCheck(Gossiper.java:538)
> at org.apache.cassandra.gms.Gossiper.access$700(Gossiper.java:57)
> at org.apache.cassandra.gms.Gossiper$GossipTask.run(Gossiper.java:157)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> INFO [GossipStage:1] 2012-03-05 04:16:55,263 Gossiper.java (line 737) Node /192.168.3.18 has restarted, now UP again
> INFO [GossipStage:1] 2012-03-05 04:16:55,264 Gossiper.java (line 705) InetAddress /192.168.3.18 is now UP
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4019)
java.util.ConcurrentModificationException in Gossiper
Posted by "Brandon Williams (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams updated CASSANDRA-4019:
----------------------------------------
Attachment: 4019.txt
For 0.8/1.0, do the simplest thing and switch BDS from ArrayDeque to LinkedBlockingDeque, imitating the ALT behavior (which is well tested.) For trunk I'll do the aforementioned refator.
> java.util.ConcurrentModificationException in Gossiper
> -----------------------------------------------------
>
> Key: CASSANDRA-4019
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4019
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.9
> Reporter: Thibaut
> Priority: Minor
> Fix For: 0.8.11
>
> Attachments: 4019.txt
>
>
> I have never seen this one before. Might be triggered by a race condition under heavy load. This error was triggered on 0.8.9
> ERROR [GossipTasks:1] 2012-03-05 04:16:55,263 Gossiper.java (line 162) Gossip error
> java.util.ConcurrentModificationException
> at java.util.ArrayDeque$DeqIterator.next(ArrayDeque.java:605)
> at org.apache.cassandra.utils.AbstractStatsDeque.sum(AbstractStatsDeque.java:37)
> at org.apache.cassandra.utils.AbstractStatsDeque.mean(AbstractStatsDeque.java:60)
> at org.apache.cassandra.gms.ArrivalWindow.mean(FailureDetector.java:259)
> at org.apache.cassandra.gms.ArrivalWindow.phi(FailureDetector.java:282)
> at org.apache.cassandra.gms.FailureDetector.interpret(FailureDetector.java:155)
> at org.apache.cassandra.gms.Gossiper.doStatusCheck(Gossiper.java:538)
> at org.apache.cassandra.gms.Gossiper.access$700(Gossiper.java:57)
> at org.apache.cassandra.gms.Gossiper$GossipTask.run(Gossiper.java:157)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> INFO [GossipStage:1] 2012-03-05 04:16:55,263 Gossiper.java (line 737) Node /192.168.3.18 has restarted, now UP again
> INFO [GossipStage:1] 2012-03-05 04:16:55,264 Gossiper.java (line 705) InetAddress /192.168.3.18 is now UP
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira