You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Jonathan Ellis (JIRA)" <ji...@apache.org> on 2010/06/04 16:00:57 UTC
[jira] Created: (CASSANDRA-1160) race with insufficiently
constructed Gossiper
race with insufficiently constructed Gossiper
---------------------------------------------
Key: CASSANDRA-1160
URL: https://issues.apache.org/jira/browse/CASSANDRA-1160
Project: Cassandra
Issue Type: Bug
Reporter: Jonathan Ellis
Assignee: Matthew F. Dennis
Priority: Minor
Fix For: 0.6.3
Gossiper.start needs to be integrated into the constructor. Currently you can have threads using the gossiper instance before start finishes (or even starts?), resulting in tracebacks like this:
ERROR [GMFD:1] 2010-06-02 10:45:49,878 CassandraDaemon.java (line 78) Fatal exception in thread Thread[GMFD:1,5,main]
java.lang.AssertionError
at org.apache.cassandra.net.Header.<init>(Header.java:56)
at org.apache.cassandra.net.Header.<init>(Header.java:74)
at org.apache.cassandra.net.Message.<init>(Message.java:58)
at org.apache.cassandra.gms.Gossiper.makeGossipDigestAckMessage(Gossiper.java:294)
at org.apache.cassandra.gms.Gossiper$GossipDigestSynVerbHandler.doVerb(Gossiper.java:935)
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
ERROR [GMFD:2] 2010-06-02 10:45:49,880 CassandraDaemon.java (line 78) Fatal exception in thread Thread[GMFD:2,5,main]
java.lang.AssertionError
at org.apache.cassandra.net.Header.<init>(Header.java:56)
at org.apache.cassandra.net.Header.<init>(Header.java:74)
at org.apache.cassandra.net.Message.<init>(Message.java:58)
at org.apache.cassandra.gms.Gossiper.makeGossipDigestAckMessage(Gossiper.java:294)
at org.apache.cassandra.gms.Gossiper$GossipDigestSynVerbHandler.doVerb(Gossiper.java:935)
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1160) race with insufficiently
constructed Gossiper
Posted by "Matthew F. Dennis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876489#action_12876489 ]
Matthew F. Dennis commented on CASSANDRA-1160:
----------------------------------------------
I should have been more clear about that. The failures in StorageServiceClientTest are because files from DatabaseDescriptor.getAllDataFileLocations() exist (the test asserts they do not). This is because when I moved start into <init> I passed the same generation number that that initServer() uses, but initClient (used by the test) passes a generation number based on currentTimeMillis() which doesn't cause the file paths to exist. It wasn't clear to me there was an easy way to determine what generation to use.
More importantly, this lead me to notice that both StorageService and StorageLoadBalancer are registered with Gossiper before start is called. If start was moved into <init> I was afraid of introducing new race conditions (e.g. Gossiper starting up, doing things that would have resulted in publications and then later receiving the registration for such publications).
I agree that Gossiper is likely not being spun up correctly. I think this applies to other things as well (in general things seem to have many interdependencies on boot. I don't have anything specific to point at, but it feels like there are several race conditions that just happen to usually not be exhibited).
One solution I've used in the past that I like for this in general is that none of the singletons are inited in their own classes, but only by a BootInitializer of sorts that completely controls when the objects are inited/started/created/etc, in particular controlling the order things are done. A single entry point into the system if you will. This seems like some work from the point we're at now and probably a bit unnecessary.
Along those lines I had a similar patch that I didn't submit that took a third argument, List<IEndpointStateChangeSubsciber> initialSubscribers. Each initialSubscriber was registered before the existing code in start was called. initServer built a list of the StorageService and StorageLoadBalance instances and passed that to start to ensure none of the publications were missed by either service. Like the 0001-cassandra-0.6-1160.patch references to .instance were replaced with getInstance() calls which busy waited (via yield) until start completed but this meant that StorageService was managing StorageLoadBalancer registrations and that didn't feel right, hence preRegistrations() (so each could manager their own registrations).
So, basically two issues/questions if Gossiper.start moves to Gossiper.<init>:
1) how does Gossiper.<init> get the correct generation number (server v client)?
2) what about missed publications because Gossiper was started before StorageService and/or StorageLoadBalancer got a chance to register?
> race with insufficiently constructed Gossiper
> ---------------------------------------------
>
> Key: CASSANDRA-1160
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1160
> Project: Cassandra
> Issue Type: Bug
> Reporter: Jonathan Ellis
> Assignee: Matthew F. Dennis
> Priority: Minor
> Fix For: 0.6.3
>
> Attachments: 0001-cassandra-0.6-1160.patch
>
>
> Gossiper.start needs to be integrated into the constructor. Currently you can have threads using the gossiper instance before start finishes (or even starts?), resulting in tracebacks like this:
> ERROR [GMFD:1] 2010-06-02 10:45:49,878 CassandraDaemon.java (line 78) Fatal exception in thread Thread[GMFD:1,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.net.Header.<init>(Header.java:56)
> at org.apache.cassandra.net.Header.<init>(Header.java:74)
> at org.apache.cassandra.net.Message.<init>(Message.java:58)
> at org.apache.cassandra.gms.Gossiper.makeGossipDigestAckMessage(Gossiper.java:294)
> at org.apache.cassandra.gms.Gossiper$GossipDigestSynVerbHandler.doVerb(Gossiper.java:935)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> ERROR [GMFD:2] 2010-06-02 10:45:49,880 CassandraDaemon.java (line 78) Fatal exception in thread Thread[GMFD:2,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.net.Header.<init>(Header.java:56)
> at org.apache.cassandra.net.Header.<init>(Header.java:74)
> at org.apache.cassandra.net.Message.<init>(Message.java:58)
> at org.apache.cassandra.gms.Gossiper.makeGossipDigestAckMessage(Gossiper.java:294)
> at org.apache.cassandra.gms.Gossiper$GossipDigestSynVerbHandler.doVerb(Gossiper.java:935)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1160) race with insufficiently
constructed Gossiper
Posted by "Gary Dusbabek (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877454#action_12877454 ]
Gary Dusbabek commented on CASSANDRA-1160:
------------------------------------------
You're right. synchronized establishes happens-before with respect to object state.
> race with insufficiently constructed Gossiper
> ---------------------------------------------
>
> Key: CASSANDRA-1160
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1160
> Project: Cassandra
> Issue Type: Bug
> Reporter: Jonathan Ellis
> Assignee: Matthew F. Dennis
> Priority: Minor
> Fix For: 0.6.3
>
> Attachments: 0001-cassandra-0.6-1160.patch, 0002-cassandra-0.6-1160.patch, 0003-cassandra-0.6-1160.patch, 0004-cassandra-0.6-1160.patch
>
>
> Gossiper.start needs to be integrated into the constructor. Currently you can have threads using the gossiper instance before start finishes (or even starts?), resulting in tracebacks like this:
> ERROR [GMFD:1] 2010-06-02 10:45:49,878 CassandraDaemon.java (line 78) Fatal exception in thread Thread[GMFD:1,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.net.Header.<init>(Header.java:56)
> at org.apache.cassandra.net.Header.<init>(Header.java:74)
> at org.apache.cassandra.net.Message.<init>(Message.java:58)
> at org.apache.cassandra.gms.Gossiper.makeGossipDigestAckMessage(Gossiper.java:294)
> at org.apache.cassandra.gms.Gossiper$GossipDigestSynVerbHandler.doVerb(Gossiper.java:935)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> ERROR [GMFD:2] 2010-06-02 10:45:49,880 CassandraDaemon.java (line 78) Fatal exception in thread Thread[GMFD:2,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.net.Header.<init>(Header.java:56)
> at org.apache.cassandra.net.Header.<init>(Header.java:74)
> at org.apache.cassandra.net.Message.<init>(Message.java:58)
> at org.apache.cassandra.gms.Gossiper.makeGossipDigestAckMessage(Gossiper.java:294)
> at org.apache.cassandra.gms.Gossiper$GossipDigestSynVerbHandler.doVerb(Gossiper.java:935)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1160) race with insufficiently
constructed Gossiper
Posted by "Matthew F. Dennis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876760#action_12876760 ]
Matthew F. Dennis commented on CASSANDRA-1160:
----------------------------------------------
just FYI, if Gossiper.start() is moved before MessageService.listen() the unit tests show sporadic (~ 1 in 5) errors even though the tests still pass.
{code}
[junit] ------------- Standard Error -----------------
[junit] Exception in thread "ACCEPT-/127.0.0.1" java.lang.RuntimeException: java.nio.channels.ClosedChannelException
[junit] at org.apache.cassandra.net.MessagingService$SocketThread.run(MessagingService.java:488)
[junit] Caused by: java.nio.channels.ClosedChannelException
[junit] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:130)
[junit] at sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
[junit] at org.apache.cassandra.net.MessagingService$SocketThread.run(MessagingService.java:477)
{code}
and
{code}
[junit] Testsuite: org.apache.cassandra.service.AntiEntropyServiceTest
[junit] Tests run: 10, Failures: 0, Errors: 0, Time elapsed: 3.282 sec
[junit]
[junit] ------------- Standard Error -----------------
[junit] ERROR 11:18:22,111 Error in executor futuretask
[junit] java.util.concurrent.ExecutionException: java.lang.NullPointerException
[junit] at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
[junit] at java.util.concurrent.FutureTask.get(FutureTask.java:83)
[junit] at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:86)
[junit] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888)
[junit] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
[junit] at java.lang.Thread.run(Thread.java:619)
[junit] Caused by: java.lang.NullPointerException
[junit] at java.util.AbstractCollection.addAll(AbstractCollection.java:303)
[junit] at org.apache.cassandra.service.AntiEntropyService.getNeighbors(AntiEntropyService.java:149)
[junit] at org.apache.cassandra.service.AntiEntropyService$Validator.call(AntiEntropyService.java:491)
[junit] at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
[junit] at java.util.concurrent.FutureTask.run(FutureTask.java:138)
[junit] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
[junit] ... 2 more
{code}
> race with insufficiently constructed Gossiper
> ---------------------------------------------
>
> Key: CASSANDRA-1160
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1160
> Project: Cassandra
> Issue Type: Bug
> Reporter: Jonathan Ellis
> Assignee: Matthew F. Dennis
> Priority: Minor
> Fix For: 0.6.3
>
> Attachments: 0001-cassandra-0.6-1160.patch, 0002-cassandra-0.6-1160.patch
>
>
> Gossiper.start needs to be integrated into the constructor. Currently you can have threads using the gossiper instance before start finishes (or even starts?), resulting in tracebacks like this:
> ERROR [GMFD:1] 2010-06-02 10:45:49,878 CassandraDaemon.java (line 78) Fatal exception in thread Thread[GMFD:1,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.net.Header.<init>(Header.java:56)
> at org.apache.cassandra.net.Header.<init>(Header.java:74)
> at org.apache.cassandra.net.Message.<init>(Message.java:58)
> at org.apache.cassandra.gms.Gossiper.makeGossipDigestAckMessage(Gossiper.java:294)
> at org.apache.cassandra.gms.Gossiper$GossipDigestSynVerbHandler.doVerb(Gossiper.java:935)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> ERROR [GMFD:2] 2010-06-02 10:45:49,880 CassandraDaemon.java (line 78) Fatal exception in thread Thread[GMFD:2,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.net.Header.<init>(Header.java:56)
> at org.apache.cassandra.net.Header.<init>(Header.java:74)
> at org.apache.cassandra.net.Message.<init>(Message.java:58)
> at org.apache.cassandra.gms.Gossiper.makeGossipDigestAckMessage(Gossiper.java:294)
> at org.apache.cassandra.gms.Gossiper$GossipDigestSynVerbHandler.doVerb(Gossiper.java:935)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1160) race with insufficiently
constructed Gossiper
Posted by "Matthew F. Dennis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matthew F. Dennis updated CASSANDRA-1160:
-----------------------------------------
Attachment: 0003-cassandra-0.6-1160.patch
0003-cassandra-0.6-1160.patch as requested moves inner classes and Gossiper to top level classes and has Gossiper TimerTask busy wait until MessagingService is up
> race with insufficiently constructed Gossiper
> ---------------------------------------------
>
> Key: CASSANDRA-1160
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1160
> Project: Cassandra
> Issue Type: Bug
> Reporter: Jonathan Ellis
> Assignee: Matthew F. Dennis
> Priority: Minor
> Fix For: 0.6.3
>
> Attachments: 0001-cassandra-0.6-1160.patch, 0002-cassandra-0.6-1160.patch, 0003-cassandra-0.6-1160.patch
>
>
> Gossiper.start needs to be integrated into the constructor. Currently you can have threads using the gossiper instance before start finishes (or even starts?), resulting in tracebacks like this:
> ERROR [GMFD:1] 2010-06-02 10:45:49,878 CassandraDaemon.java (line 78) Fatal exception in thread Thread[GMFD:1,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.net.Header.<init>(Header.java:56)
> at org.apache.cassandra.net.Header.<init>(Header.java:74)
> at org.apache.cassandra.net.Message.<init>(Message.java:58)
> at org.apache.cassandra.gms.Gossiper.makeGossipDigestAckMessage(Gossiper.java:294)
> at org.apache.cassandra.gms.Gossiper$GossipDigestSynVerbHandler.doVerb(Gossiper.java:935)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> ERROR [GMFD:2] 2010-06-02 10:45:49,880 CassandraDaemon.java (line 78) Fatal exception in thread Thread[GMFD:2,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.net.Header.<init>(Header.java:56)
> at org.apache.cassandra.net.Header.<init>(Header.java:74)
> at org.apache.cassandra.net.Message.<init>(Message.java:58)
> at org.apache.cassandra.gms.Gossiper.makeGossipDigestAckMessage(Gossiper.java:294)
> at org.apache.cassandra.gms.Gossiper$GossipDigestSynVerbHandler.doVerb(Gossiper.java:935)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1160) race with insufficiently
constructed Gossiper
Posted by "Gary Dusbabek (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877410#action_12877410 ]
Gary Dusbabek commented on CASSANDRA-1160:
------------------------------------------
Last patch is looking good, except you should consider replacing the busy-wait Thread.yield() stuff with a CountDownLatch and a simple MessageService.awaitListen() call or something like that. My reasoning is that Thread.yeild() behaves differently across platforms and can possibly starve threads running at a lower priority.
> race with insufficiently constructed Gossiper
> ---------------------------------------------
>
> Key: CASSANDRA-1160
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1160
> Project: Cassandra
> Issue Type: Bug
> Reporter: Jonathan Ellis
> Assignee: Matthew F. Dennis
> Priority: Minor
> Fix For: 0.6.3
>
> Attachments: 0001-cassandra-0.6-1160.patch, 0002-cassandra-0.6-1160.patch, 0003-cassandra-0.6-1160.patch
>
>
> Gossiper.start needs to be integrated into the constructor. Currently you can have threads using the gossiper instance before start finishes (or even starts?), resulting in tracebacks like this:
> ERROR [GMFD:1] 2010-06-02 10:45:49,878 CassandraDaemon.java (line 78) Fatal exception in thread Thread[GMFD:1,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.net.Header.<init>(Header.java:56)
> at org.apache.cassandra.net.Header.<init>(Header.java:74)
> at org.apache.cassandra.net.Message.<init>(Message.java:58)
> at org.apache.cassandra.gms.Gossiper.makeGossipDigestAckMessage(Gossiper.java:294)
> at org.apache.cassandra.gms.Gossiper$GossipDigestSynVerbHandler.doVerb(Gossiper.java:935)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> ERROR [GMFD:2] 2010-06-02 10:45:49,880 CassandraDaemon.java (line 78) Fatal exception in thread Thread[GMFD:2,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.net.Header.<init>(Header.java:56)
> at org.apache.cassandra.net.Header.<init>(Header.java:74)
> at org.apache.cassandra.net.Message.<init>(Message.java:58)
> at org.apache.cassandra.gms.Gossiper.makeGossipDigestAckMessage(Gossiper.java:294)
> at org.apache.cassandra.gms.Gossiper$GossipDigestSynVerbHandler.doVerb(Gossiper.java:935)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1160) race with insufficiently
constructed Gossiper
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877419#action_12877419 ]
Jonathan Ellis commented on CASSANDRA-1160:
-------------------------------------------
also consider using SimpleCondition.
> race with insufficiently constructed Gossiper
> ---------------------------------------------
>
> Key: CASSANDRA-1160
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1160
> Project: Cassandra
> Issue Type: Bug
> Reporter: Jonathan Ellis
> Assignee: Matthew F. Dennis
> Priority: Minor
> Fix For: 0.6.3
>
> Attachments: 0001-cassandra-0.6-1160.patch, 0002-cassandra-0.6-1160.patch, 0003-cassandra-0.6-1160.patch
>
>
> Gossiper.start needs to be integrated into the constructor. Currently you can have threads using the gossiper instance before start finishes (or even starts?), resulting in tracebacks like this:
> ERROR [GMFD:1] 2010-06-02 10:45:49,878 CassandraDaemon.java (line 78) Fatal exception in thread Thread[GMFD:1,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.net.Header.<init>(Header.java:56)
> at org.apache.cassandra.net.Header.<init>(Header.java:74)
> at org.apache.cassandra.net.Message.<init>(Message.java:58)
> at org.apache.cassandra.gms.Gossiper.makeGossipDigestAckMessage(Gossiper.java:294)
> at org.apache.cassandra.gms.Gossiper$GossipDigestSynVerbHandler.doVerb(Gossiper.java:935)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> ERROR [GMFD:2] 2010-06-02 10:45:49,880 CassandraDaemon.java (line 78) Fatal exception in thread Thread[GMFD:2,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.net.Header.<init>(Header.java:56)
> at org.apache.cassandra.net.Header.<init>(Header.java:74)
> at org.apache.cassandra.net.Message.<init>(Message.java:58)
> at org.apache.cassandra.gms.Gossiper.makeGossipDigestAckMessage(Gossiper.java:294)
> at org.apache.cassandra.gms.Gossiper$GossipDigestSynVerbHandler.doVerb(Gossiper.java:935)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1160) race with insufficiently
constructed Gossiper
Posted by "Matthew F. Dennis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matthew F. Dennis updated CASSANDRA-1160:
-----------------------------------------
Attachment: 0002-cassandra-0.6-1160.patch
0002-cassandra-0.6-1160.patch is against r952679
> race with insufficiently constructed Gossiper
> ---------------------------------------------
>
> Key: CASSANDRA-1160
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1160
> Project: Cassandra
> Issue Type: Bug
> Reporter: Jonathan Ellis
> Assignee: Matthew F. Dennis
> Priority: Minor
> Fix For: 0.6.3
>
> Attachments: 0001-cassandra-0.6-1160.patch, 0002-cassandra-0.6-1160.patch
>
>
> Gossiper.start needs to be integrated into the constructor. Currently you can have threads using the gossiper instance before start finishes (or even starts?), resulting in tracebacks like this:
> ERROR [GMFD:1] 2010-06-02 10:45:49,878 CassandraDaemon.java (line 78) Fatal exception in thread Thread[GMFD:1,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.net.Header.<init>(Header.java:56)
> at org.apache.cassandra.net.Header.<init>(Header.java:74)
> at org.apache.cassandra.net.Message.<init>(Message.java:58)
> at org.apache.cassandra.gms.Gossiper.makeGossipDigestAckMessage(Gossiper.java:294)
> at org.apache.cassandra.gms.Gossiper$GossipDigestSynVerbHandler.doVerb(Gossiper.java:935)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> ERROR [GMFD:2] 2010-06-02 10:45:49,880 CassandraDaemon.java (line 78) Fatal exception in thread Thread[GMFD:2,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.net.Header.<init>(Header.java:56)
> at org.apache.cassandra.net.Header.<init>(Header.java:74)
> at org.apache.cassandra.net.Message.<init>(Message.java:58)
> at org.apache.cassandra.gms.Gossiper.makeGossipDigestAckMessage(Gossiper.java:294)
> at org.apache.cassandra.gms.Gossiper$GossipDigestSynVerbHandler.doVerb(Gossiper.java:935)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1160) race with insufficiently
constructed Gossiper
Posted by "Gary Dusbabek (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877446#action_12877446 ]
Gary Dusbabek commented on CASSANDRA-1160:
------------------------------------------
I see. I figured SimpleCondition was something out of the JDK.
Spurious bikeshed comment: SimpleCondition.set should be volatile, no? otherwise set=true ins't guaranteed to be visible to any thread stuck in await().
> race with insufficiently constructed Gossiper
> ---------------------------------------------
>
> Key: CASSANDRA-1160
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1160
> Project: Cassandra
> Issue Type: Bug
> Reporter: Jonathan Ellis
> Assignee: Matthew F. Dennis
> Priority: Minor
> Fix For: 0.6.3
>
> Attachments: 0001-cassandra-0.6-1160.patch, 0002-cassandra-0.6-1160.patch, 0003-cassandra-0.6-1160.patch
>
>
> Gossiper.start needs to be integrated into the constructor. Currently you can have threads using the gossiper instance before start finishes (or even starts?), resulting in tracebacks like this:
> ERROR [GMFD:1] 2010-06-02 10:45:49,878 CassandraDaemon.java (line 78) Fatal exception in thread Thread[GMFD:1,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.net.Header.<init>(Header.java:56)
> at org.apache.cassandra.net.Header.<init>(Header.java:74)
> at org.apache.cassandra.net.Message.<init>(Message.java:58)
> at org.apache.cassandra.gms.Gossiper.makeGossipDigestAckMessage(Gossiper.java:294)
> at org.apache.cassandra.gms.Gossiper$GossipDigestSynVerbHandler.doVerb(Gossiper.java:935)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> ERROR [GMFD:2] 2010-06-02 10:45:49,880 CassandraDaemon.java (line 78) Fatal exception in thread Thread[GMFD:2,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.net.Header.<init>(Header.java:56)
> at org.apache.cassandra.net.Header.<init>(Header.java:74)
> at org.apache.cassandra.net.Message.<init>(Message.java:58)
> at org.apache.cassandra.gms.Gossiper.makeGossipDigestAckMessage(Gossiper.java:294)
> at org.apache.cassandra.gms.Gossiper$GossipDigestSynVerbHandler.doVerb(Gossiper.java:935)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1160) race with insufficiently
constructed Gossiper
Posted by "Gary Dusbabek (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877431#action_12877431 ]
Gary Dusbabek commented on CASSANDRA-1160:
------------------------------------------
Wouldn't a Condition cause the waiter to block if somehow signalAll() (which in this case would only be called once) was called before await()? I know it's not likely, but there isn't any such problem with a CountDownLatch.
> race with insufficiently constructed Gossiper
> ---------------------------------------------
>
> Key: CASSANDRA-1160
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1160
> Project: Cassandra
> Issue Type: Bug
> Reporter: Jonathan Ellis
> Assignee: Matthew F. Dennis
> Priority: Minor
> Fix For: 0.6.3
>
> Attachments: 0001-cassandra-0.6-1160.patch, 0002-cassandra-0.6-1160.patch, 0003-cassandra-0.6-1160.patch
>
>
> Gossiper.start needs to be integrated into the constructor. Currently you can have threads using the gossiper instance before start finishes (or even starts?), resulting in tracebacks like this:
> ERROR [GMFD:1] 2010-06-02 10:45:49,878 CassandraDaemon.java (line 78) Fatal exception in thread Thread[GMFD:1,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.net.Header.<init>(Header.java:56)
> at org.apache.cassandra.net.Header.<init>(Header.java:74)
> at org.apache.cassandra.net.Message.<init>(Message.java:58)
> at org.apache.cassandra.gms.Gossiper.makeGossipDigestAckMessage(Gossiper.java:294)
> at org.apache.cassandra.gms.Gossiper$GossipDigestSynVerbHandler.doVerb(Gossiper.java:935)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> ERROR [GMFD:2] 2010-06-02 10:45:49,880 CassandraDaemon.java (line 78) Fatal exception in thread Thread[GMFD:2,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.net.Header.<init>(Header.java:56)
> at org.apache.cassandra.net.Header.<init>(Header.java:74)
> at org.apache.cassandra.net.Message.<init>(Message.java:58)
> at org.apache.cassandra.gms.Gossiper.makeGossipDigestAckMessage(Gossiper.java:294)
> at org.apache.cassandra.gms.Gossiper$GossipDigestSynVerbHandler.doVerb(Gossiper.java:935)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1160) race with insufficiently
constructed Gossiper
Posted by "Matthew F. Dennis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matthew F. Dennis updated CASSANDRA-1160:
-----------------------------------------
Attachment: 0004-cassandra-0.6-1160.patch
0004-cassandra-0.6-1160.patch uses a gate to wait until MessagingService is listening
> race with insufficiently constructed Gossiper
> ---------------------------------------------
>
> Key: CASSANDRA-1160
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1160
> Project: Cassandra
> Issue Type: Bug
> Reporter: Jonathan Ellis
> Assignee: Matthew F. Dennis
> Priority: Minor
> Fix For: 0.6.3
>
> Attachments: 0001-cassandra-0.6-1160.patch, 0002-cassandra-0.6-1160.patch, 0003-cassandra-0.6-1160.patch, 0004-cassandra-0.6-1160.patch
>
>
> Gossiper.start needs to be integrated into the constructor. Currently you can have threads using the gossiper instance before start finishes (or even starts?), resulting in tracebacks like this:
> ERROR [GMFD:1] 2010-06-02 10:45:49,878 CassandraDaemon.java (line 78) Fatal exception in thread Thread[GMFD:1,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.net.Header.<init>(Header.java:56)
> at org.apache.cassandra.net.Header.<init>(Header.java:74)
> at org.apache.cassandra.net.Message.<init>(Message.java:58)
> at org.apache.cassandra.gms.Gossiper.makeGossipDigestAckMessage(Gossiper.java:294)
> at org.apache.cassandra.gms.Gossiper$GossipDigestSynVerbHandler.doVerb(Gossiper.java:935)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> ERROR [GMFD:2] 2010-06-02 10:45:49,880 CassandraDaemon.java (line 78) Fatal exception in thread Thread[GMFD:2,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.net.Header.<init>(Header.java:56)
> at org.apache.cassandra.net.Header.<init>(Header.java:74)
> at org.apache.cassandra.net.Message.<init>(Message.java:58)
> at org.apache.cassandra.gms.Gossiper.makeGossipDigestAckMessage(Gossiper.java:294)
> at org.apache.cassandra.gms.Gossiper$GossipDigestSynVerbHandler.doVerb(Gossiper.java:935)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1160) race with insufficiently
constructed Gossiper
Posted by "Gary Dusbabek (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876766#action_12876766 ]
Gary Dusbabek commented on CASSANDRA-1160:
------------------------------------------
Intermittent test failures indicate you have a timing problem.
The fact that the assertion (the original intermittent problem) fails because a static class member is null makes me think we're dealing with static initializers firing in the order we don't suspect.
I'd start by getting the inner classes (the verb handlers) out of Gossiper and into their own classes. Then there is the race between Gossiper and MessageService. You could let the gossiper start before the MessageService starts listening, but have the GossipTimerTask check to make sure the MessageService is listening before it sends out any gossip requests. This still isn't going to protect us in the case of a node getting restarted where other nodes already know about it and and diligently trying to contact what they thought was a dead node.
> race with insufficiently constructed Gossiper
> ---------------------------------------------
>
> Key: CASSANDRA-1160
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1160
> Project: Cassandra
> Issue Type: Bug
> Reporter: Jonathan Ellis
> Assignee: Matthew F. Dennis
> Priority: Minor
> Fix For: 0.6.3
>
> Attachments: 0001-cassandra-0.6-1160.patch, 0002-cassandra-0.6-1160.patch
>
>
> Gossiper.start needs to be integrated into the constructor. Currently you can have threads using the gossiper instance before start finishes (or even starts?), resulting in tracebacks like this:
> ERROR [GMFD:1] 2010-06-02 10:45:49,878 CassandraDaemon.java (line 78) Fatal exception in thread Thread[GMFD:1,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.net.Header.<init>(Header.java:56)
> at org.apache.cassandra.net.Header.<init>(Header.java:74)
> at org.apache.cassandra.net.Message.<init>(Message.java:58)
> at org.apache.cassandra.gms.Gossiper.makeGossipDigestAckMessage(Gossiper.java:294)
> at org.apache.cassandra.gms.Gossiper$GossipDigestSynVerbHandler.doVerb(Gossiper.java:935)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> ERROR [GMFD:2] 2010-06-02 10:45:49,880 CassandraDaemon.java (line 78) Fatal exception in thread Thread[GMFD:2,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.net.Header.<init>(Header.java:56)
> at org.apache.cassandra.net.Header.<init>(Header.java:74)
> at org.apache.cassandra.net.Message.<init>(Message.java:58)
> at org.apache.cassandra.gms.Gossiper.makeGossipDigestAckMessage(Gossiper.java:294)
> at org.apache.cassandra.gms.Gossiper$GossipDigestSynVerbHandler.doVerb(Gossiper.java:935)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1160) race with insufficiently
constructed Gossiper
Posted by "Matthew F. Dennis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matthew F. Dennis updated CASSANDRA-1160:
-----------------------------------------
Attachment: 0001-cassandra-0.6-1160.patch
moving start() to Gossiper.<init> causes failures in testClientOnlyMode and I didn't see a way to readily make client/server mode available to Gossiper.<init>
0001-cassandra-0.6-1160.patch delays creation of the Gossiper instance until start is called. Any thread attempting to obtain a reference to Gossiper before it is started busy waits until it is fully initialized.
I'm not sure I actually like this solution, but thought I would submit for comments/feedback.
If we end up taking this solution, I'll submit a patch for trunk as well.
> race with insufficiently constructed Gossiper
> ---------------------------------------------
>
> Key: CASSANDRA-1160
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1160
> Project: Cassandra
> Issue Type: Bug
> Reporter: Jonathan Ellis
> Assignee: Matthew F. Dennis
> Priority: Minor
> Fix For: 0.6.3
>
> Attachments: 0001-cassandra-0.6-1160.patch
>
>
> Gossiper.start needs to be integrated into the constructor. Currently you can have threads using the gossiper instance before start finishes (or even starts?), resulting in tracebacks like this:
> ERROR [GMFD:1] 2010-06-02 10:45:49,878 CassandraDaemon.java (line 78) Fatal exception in thread Thread[GMFD:1,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.net.Header.<init>(Header.java:56)
> at org.apache.cassandra.net.Header.<init>(Header.java:74)
> at org.apache.cassandra.net.Message.<init>(Message.java:58)
> at org.apache.cassandra.gms.Gossiper.makeGossipDigestAckMessage(Gossiper.java:294)
> at org.apache.cassandra.gms.Gossiper$GossipDigestSynVerbHandler.doVerb(Gossiper.java:935)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> ERROR [GMFD:2] 2010-06-02 10:45:49,880 CassandraDaemon.java (line 78) Fatal exception in thread Thread[GMFD:2,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.net.Header.<init>(Header.java:56)
> at org.apache.cassandra.net.Header.<init>(Header.java:74)
> at org.apache.cassandra.net.Message.<init>(Message.java:58)
> at org.apache.cassandra.gms.Gossiper.makeGossipDigestAckMessage(Gossiper.java:294)
> at org.apache.cassandra.gms.Gossiper$GossipDigestSynVerbHandler.doVerb(Gossiper.java:935)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1160) race with insufficiently
constructed Gossiper
Posted by "Gary Dusbabek (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876459#action_12876459 ]
Gary Dusbabek commented on CASSANDRA-1160:
------------------------------------------
What were the failures in testClientOnlyMode? I'm sure they can be worked around.
I'm not a fan of Gossiper.preRegistrations. I think it is a code smell that we're not spinning up Gossiper the right way. What if we were to make Gossiper.start() blocking until we were sure it was in a healthy state (keep in mind the single-node cluster) and then make sure it was called before MessageService.listen()?
> race with insufficiently constructed Gossiper
> ---------------------------------------------
>
> Key: CASSANDRA-1160
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1160
> Project: Cassandra
> Issue Type: Bug
> Reporter: Jonathan Ellis
> Assignee: Matthew F. Dennis
> Priority: Minor
> Fix For: 0.6.3
>
> Attachments: 0001-cassandra-0.6-1160.patch
>
>
> Gossiper.start needs to be integrated into the constructor. Currently you can have threads using the gossiper instance before start finishes (or even starts?), resulting in tracebacks like this:
> ERROR [GMFD:1] 2010-06-02 10:45:49,878 CassandraDaemon.java (line 78) Fatal exception in thread Thread[GMFD:1,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.net.Header.<init>(Header.java:56)
> at org.apache.cassandra.net.Header.<init>(Header.java:74)
> at org.apache.cassandra.net.Message.<init>(Message.java:58)
> at org.apache.cassandra.gms.Gossiper.makeGossipDigestAckMessage(Gossiper.java:294)
> at org.apache.cassandra.gms.Gossiper$GossipDigestSynVerbHandler.doVerb(Gossiper.java:935)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> ERROR [GMFD:2] 2010-06-02 10:45:49,880 CassandraDaemon.java (line 78) Fatal exception in thread Thread[GMFD:2,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.net.Header.<init>(Header.java:56)
> at org.apache.cassandra.net.Header.<init>(Header.java:74)
> at org.apache.cassandra.net.Message.<init>(Message.java:58)
> at org.apache.cassandra.gms.Gossiper.makeGossipDigestAckMessage(Gossiper.java:294)
> at org.apache.cassandra.gms.Gossiper$GossipDigestSynVerbHandler.doVerb(Gossiper.java:935)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1160) race with insufficiently
constructed Gossiper
Posted by "Gary Dusbabek (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876672#action_12876672 ]
Gary Dusbabek commented on CASSANDRA-1160:
------------------------------------------
I'm taking a closer look. In the mean time can you rebase your patch?
> race with insufficiently constructed Gossiper
> ---------------------------------------------
>
> Key: CASSANDRA-1160
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1160
> Project: Cassandra
> Issue Type: Bug
> Reporter: Jonathan Ellis
> Assignee: Matthew F. Dennis
> Priority: Minor
> Fix For: 0.6.3
>
> Attachments: 0001-cassandra-0.6-1160.patch
>
>
> Gossiper.start needs to be integrated into the constructor. Currently you can have threads using the gossiper instance before start finishes (or even starts?), resulting in tracebacks like this:
> ERROR [GMFD:1] 2010-06-02 10:45:49,878 CassandraDaemon.java (line 78) Fatal exception in thread Thread[GMFD:1,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.net.Header.<init>(Header.java:56)
> at org.apache.cassandra.net.Header.<init>(Header.java:74)
> at org.apache.cassandra.net.Message.<init>(Message.java:58)
> at org.apache.cassandra.gms.Gossiper.makeGossipDigestAckMessage(Gossiper.java:294)
> at org.apache.cassandra.gms.Gossiper$GossipDigestSynVerbHandler.doVerb(Gossiper.java:935)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> ERROR [GMFD:2] 2010-06-02 10:45:49,880 CassandraDaemon.java (line 78) Fatal exception in thread Thread[GMFD:2,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.net.Header.<init>(Header.java:56)
> at org.apache.cassandra.net.Header.<init>(Header.java:74)
> at org.apache.cassandra.net.Message.<init>(Message.java:58)
> at org.apache.cassandra.gms.Gossiper.makeGossipDigestAckMessage(Gossiper.java:294)
> at org.apache.cassandra.gms.Gossiper$GossipDigestSynVerbHandler.doVerb(Gossiper.java:935)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1160) race with insufficiently
constructed Gossiper
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877450#action_12877450 ]
Jonathan Ellis commented on CASSANDRA-1160:
-------------------------------------------
I believe the synchronized keyword takes care of that, but wait/notify are admittedly subtle. I could be wrong.
> race with insufficiently constructed Gossiper
> ---------------------------------------------
>
> Key: CASSANDRA-1160
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1160
> Project: Cassandra
> Issue Type: Bug
> Reporter: Jonathan Ellis
> Assignee: Matthew F. Dennis
> Priority: Minor
> Fix For: 0.6.3
>
> Attachments: 0001-cassandra-0.6-1160.patch, 0002-cassandra-0.6-1160.patch, 0003-cassandra-0.6-1160.patch
>
>
> Gossiper.start needs to be integrated into the constructor. Currently you can have threads using the gossiper instance before start finishes (or even starts?), resulting in tracebacks like this:
> ERROR [GMFD:1] 2010-06-02 10:45:49,878 CassandraDaemon.java (line 78) Fatal exception in thread Thread[GMFD:1,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.net.Header.<init>(Header.java:56)
> at org.apache.cassandra.net.Header.<init>(Header.java:74)
> at org.apache.cassandra.net.Message.<init>(Message.java:58)
> at org.apache.cassandra.gms.Gossiper.makeGossipDigestAckMessage(Gossiper.java:294)
> at org.apache.cassandra.gms.Gossiper$GossipDigestSynVerbHandler.doVerb(Gossiper.java:935)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> ERROR [GMFD:2] 2010-06-02 10:45:49,880 CassandraDaemon.java (line 78) Fatal exception in thread Thread[GMFD:2,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.net.Header.<init>(Header.java:56)
> at org.apache.cassandra.net.Header.<init>(Header.java:74)
> at org.apache.cassandra.net.Message.<init>(Message.java:58)
> at org.apache.cassandra.gms.Gossiper.makeGossipDigestAckMessage(Gossiper.java:294)
> at org.apache.cassandra.gms.Gossiper$GossipDigestSynVerbHandler.doVerb(Gossiper.java:935)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1160) race with insufficiently
constructed Gossiper
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12877439#action_12877439 ]
Jonathan Ellis commented on CASSANDRA-1160:
-------------------------------------------
// [SimpleCondition] fulfils the Condition interface without spurious wakeup problems
// (or lost notify problems either: that is, even if you call await()
// _after_ signal(), it will work as desired.)
> race with insufficiently constructed Gossiper
> ---------------------------------------------
>
> Key: CASSANDRA-1160
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1160
> Project: Cassandra
> Issue Type: Bug
> Reporter: Jonathan Ellis
> Assignee: Matthew F. Dennis
> Priority: Minor
> Fix For: 0.6.3
>
> Attachments: 0001-cassandra-0.6-1160.patch, 0002-cassandra-0.6-1160.patch, 0003-cassandra-0.6-1160.patch
>
>
> Gossiper.start needs to be integrated into the constructor. Currently you can have threads using the gossiper instance before start finishes (or even starts?), resulting in tracebacks like this:
> ERROR [GMFD:1] 2010-06-02 10:45:49,878 CassandraDaemon.java (line 78) Fatal exception in thread Thread[GMFD:1,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.net.Header.<init>(Header.java:56)
> at org.apache.cassandra.net.Header.<init>(Header.java:74)
> at org.apache.cassandra.net.Message.<init>(Message.java:58)
> at org.apache.cassandra.gms.Gossiper.makeGossipDigestAckMessage(Gossiper.java:294)
> at org.apache.cassandra.gms.Gossiper$GossipDigestSynVerbHandler.doVerb(Gossiper.java:935)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> ERROR [GMFD:2] 2010-06-02 10:45:49,880 CassandraDaemon.java (line 78) Fatal exception in thread Thread[GMFD:2,5,main]
> java.lang.AssertionError
> at org.apache.cassandra.net.Header.<init>(Header.java:56)
> at org.apache.cassandra.net.Header.<init>(Header.java:74)
> at org.apache.cassandra.net.Message.<init>(Message.java:58)
> at org.apache.cassandra.gms.Gossiper.makeGossipDigestAckMessage(Gossiper.java:294)
> at org.apache.cassandra.gms.Gossiper$GossipDigestSynVerbHandler.doVerb(Gossiper.java:935)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.