[jira] [Commented] (CASSANDRA-4317) AssertionError in handleStateNormal in a mixed cluster

["Brandon Williams (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-4317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13295104#comment-13295104 ] 

Brandon Williams commented on CASSANDRA-4317:
---------------------------------------------

The problem here is similar to what CASSANDRA-4311 is trying to solve, but with a small twist.

Consider node X, Y, and Z.  Both X and Y are on the newer version v2, but Z is on the older version v1.  X is the seed, and Z is the first to talk to X, and they complete a gossip round and both speak v1 to each other.  Now Y contacts X and they complete a gossip round, where Y learns about Z.  The problem, however, is that Y has never talked directly to Z, so it doesn't actually know Z's version yet but needs to handle events for it.  StorageService then does version checks to determine how to handle the events for Z, but getVersion assumes the current version (v2 in this example) when it doesn't actually know what the version is, and thus ends up processing old-style information as new-style and fails.  This isn't limited to handleStateNormal, but likely any event called from onChange.

Solving this is somewhat tricky.  Reviving CASSANDRA-4101 would do it, but then we have two sources of version information and I'm not sure how clean that will be.  getVersion would have to check for null, and if found check for a gossip version, and if that too is not found assume the *lowest* version.


                
> AssertionError in handleStateNormal in a mixed cluster
> ------------------------------------------------------
>
>                 Key: CASSANDRA-4317
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4317
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 1.2
>            Reporter: Brandon Williams
>             Fix For: 1.2
>
>
> In a 3 node cluster with one seed on trunk, a member on trunk, and another member on a previous version, the following occurs only on the non-seed trunk member:
> {noformat}
> ERROR 16:44:18,708 Exception in thread Thread[GossipStage:1,5,main]
> java.lang.AssertionError
>         at org.apache.cassandra.service.StorageService.handleStateNormal(StorageService.java:1072)
>         at org.apache.cassandra.service.StorageService.onChange(StorageService.java:995)
>         at org.apache.cassandra.service.StorageService.onJoin(StorageService.java:1568)
>         at org.apache.cassandra.gms.Gossiper.handleMajorStateChange(Gossiper.java:819)
>         at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:897)
>         at org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:43)
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:57)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:662)
> {noformat}
> This doesn't repro if a non-trunk member is the seed, however upgrading the seed first should still be valid.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira