You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Yan Wu <yw...@prospricing.com> on 2012/11/05 19:55:11 UTC

How to upgrade a ring (0.8.9 nodes) to 1.1.5 with the minimal downtime?

Hello, 

I have a Cassandra ring with 4 nodes in 0.8.9 and like to upgrade all nodes to 1.1.5.
It would be great that the upgrade has no downtime or minimal downtime of the ring.
After I brought down one of the nodes and upgraded it to 1.1.5, when I tried to bring it up, 
the new 1.1.5 node looks good but the rest of three 0.8.9 nodes started throwing exceptions:
---------------
Fatal exception in thread Thread[GossipStage:2,5,main]
java.lang.UnsupportedOperationException: Not a time-based UUID
at org.apache.cassandra.service.MigrationManager.rectify(MigrationManager.java:92)
        at org.apache.cassandra.service.MigrationManager.onAlive(MigrationManager.java:75)
        at org.apache.cassandra.gms.Gossiper.markAlive(Gossiper.java:707)
        at org.apache.cassandra.gms.Gossiper.handleMajorStateChange(Gossiper.java:750)
        at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:809)
        at org.apache.cassandra.gms.GossipDigestAckVerbHandler.doVerb(GossipDigestAckVerbHandler.java:68)
        at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
----------------
Then later
----------------
ERROR 12:03:20,925 Fatal exception in thread Thread[HintedHandoff:1,1,main]
java.lang.RuntimeException: java.lang.RuntimeException: Could not reach schema agreement with /xx.xx.xx.xx in 60000ms
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.RuntimeException: Could not reach schema agreement with /xx.xx.xx.xx in 60000ms
        at org.apache.cassandra.db.HintedHandOffManager.waitForSchemaAgreement(HintedHandOffManager.java:293)
        at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:304)
        at org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:89)
        at org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:397)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
        ... 3 more
----------------

Any suggestions?   Thanks in advance.

Yan

Re: How to upgrade a ring (0.8.9 nodes) to 1.1.5 with the minimal downtime?

Posted by Bryan Talbot <bt...@aeriagames.com>.
Do a rolling upgrade of the ring to 1.0.12 first and then upgrade to 1.1.x.
 After each rolling upgrade, you should probably do the recommend "nodetool
upgradesstables", etc.  The datastax documentation about upgrading might be
helpful for you: http://www.datastax.com/docs/1.1/install/upgrading

-Bryan


On Mon, Nov 5, 2012 at 10:55 AM, Yan Wu <yw...@prospricing.com> wrote:

> Hello,
>
> I have a Cassandra ring with 4 nodes in 0.8.9 and like to upgrade all
> nodes to 1.1.5.
> It would be great that the upgrade has no downtime or minimal downtime of
> the ring.
> After I brought down one of the nodes and upgraded it to 1.1.5, when I
> tried to bring it up,
> the new 1.1.5 node looks good but the rest of three 0.8.9 nodes started
> throwing exceptions:
> ---------------
> Fatal exception in thread Thread[GossipStage:2,5,main]
> java.lang.UnsupportedOperationException: Not a time-based UUID
> at
> org.apache.cassandra.service.MigrationManager.rectify(MigrationManager.java:92)
>         at
> org.apache.cassandra.service.MigrationManager.onAlive(MigrationManager.java:75)
>         at org.apache.cassandra.gms.Gossiper.markAlive(Gossiper.java:707)
>         at
> org.apache.cassandra.gms.Gossiper.handleMajorStateChange(Gossiper.java:750)
>         at
> org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:809)
>         at
> org.apache.cassandra.gms.GossipDigestAckVerbHandler.doVerb(GossipDigestAckVerbHandler.java:68)
>         at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> ----------------
> Then later
> ----------------
> ERROR 12:03:20,925 Fatal exception in thread Thread[HintedHandoff:1,1,main]
> java.lang.RuntimeException: java.lang.RuntimeException: Could not reach
> schema agreement with /xx.xx.xx.xx in 60000ms
>         at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:662)
> Caused by: java.lang.RuntimeException: Could not reach schema agreement
> with /xx.xx.xx.xx in 60000ms
>         at
> org.apache.cassandra.db.HintedHandOffManager.waitForSchemaAgreement(HintedHandOffManager.java:293)
>         at
> org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:304)
>         at
> org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:89)
>         at
> org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:397)
>         at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>         ... 3 more
> ----------------
>
> Any suggestions?   Thanks in advance.
>
> Yan
>