You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Bryce Godfrey <Br...@azaleos.com> on 2011/12/12 08:02:01 UTC

node stuck "leaving" on 1.0.5

I have a dead node I need to remove from the cluster so that I can rebalance among the existing servers (can't replace it for a while).

I used nodetool removetoken and it's been stuck in the "leaving" state for over a day now.  I've tried a rolling restart, which kicks of some streaming for a while under netstats but now even that lists nothing going on.

I'm stuck on what to do next to get this node to finally leave so I can move the tokens around.

Only error I see in the system log:

ERROR [Thread-209] 2011-12-11 01:40:34,347 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[Thread-209,5,main]
java.lang.AssertionError
        at org.apache.cassandra.db.compaction.LeveledManifest.promote(LeveledManifest.java:178)
        at org.apache.cassandra.db.compaction.LeveledCompactionStrategy.handleNotification(LeveledCompactionStrategy.java:141)
        at org.apache.cassandra.db.DataTracker.notifySSTablesChanged(DataTracker.java:481)
        at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:275)
        at org.apache.cassandra.db.DataTracker.addSSTables(DataTracker.java:237)
        at org.apache.cassandra.db.DataTracker.addStreamedSSTable(DataTracker.java:242)
        at org.apache.cassandra.db.ColumnFamilyStore.addSSTable(ColumnFamilyStore.java:920)
        at org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:141)
        at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:103)
        at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:184)
        at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:81)

RE: node stuck "leaving" on 1.0.5

Posted by Bryce Godfrey <Br...@azaleos.com>.
So I got past the leaving problem once I found the removetoken force command.  Now I'm trying to move tokens and that will never complete either, but as I was watching netstats for streaming to the moving node I noticed it seemed to stop all of a sudden and list no more pending streams.  At the same time on the moving node this is in the system log:

ERROR [Thread-455] 2011-12-13 16:15:51,939 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[Thread-455,5,main]
java.lang.AssertionError
        at org.apache.cassandra.db.compaction.LeveledManifest.promote(LeveledManifest.java:178)
        at org.apache.cassandra.db.compaction.LeveledCompactionStrategy.handleNotification(LeveledCompactionStrategy.java:141)
        at org.apache.cassandra.db.DataTracker.notifySSTablesChanged(DataTracker.java:481)
        at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:275)
        at org.apache.cassandra.db.DataTracker.addSSTables(DataTracker.java:237)
        at org.apache.cassandra.db.DataTracker.addStreamedSSTable(DataTracker.java:242)
        at org.apache.cassandra.db.ColumnFamilyStore.addSSTable(ColumnFamilyStore.java:920)
        at org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:141)
        at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:103)
        at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:184)
        at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:81)

Now I see no streams going on between any nodes, and the node is still listed as "moving" when viewing the ring.

From: Bryce Godfrey [mailto:Bryce.Godfrey@azaleos.com]
Sent: Sunday, December 11, 2011 11:02 PM
To: user@cassandra.apache.org
Subject: node stuck "leaving" on 1.0.5

I have a dead node I need to remove from the cluster so that I can rebalance among the existing servers (can't replace it for a while).

I used nodetool removetoken and it's been stuck in the "leaving" state for over a day now.  I've tried a rolling restart, which kicks of some streaming for a while under netstats but now even that lists nothing going on.

I'm stuck on what to do next to get this node to finally leave so I can move the tokens around.

Only error I see in the system log:

ERROR [Thread-209] 2011-12-11 01:40:34,347 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[Thread-209,5,main]
java.lang.AssertionError
        at org.apache.cassandra.db.compaction.LeveledManifest.promote(LeveledManifest.java:178)
        at org.apache.cassandra.db.compaction.LeveledCompactionStrategy.handleNotification(LeveledCompactionStrategy.java:141)
        at org.apache.cassandra.db.DataTracker.notifySSTablesChanged(DataTracker.java:481)
        at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:275)
        at org.apache.cassandra.db.DataTracker.addSSTables(DataTracker.java:237)
        at org.apache.cassandra.db.DataTracker.addStreamedSSTable(DataTracker.java:242)
        at org.apache.cassandra.db.ColumnFamilyStore.addSSTable(ColumnFamilyStore.java:920)
        at org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:141)
        at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:103)
        at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:184)
        at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:81)