You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cassandra.apache.org by Colin Kuo <co...@gmail.com> on 2013/05/29 09:32:26 UTC
Starting up Cassandra occurred errors after upgrading Cassandra to
1.2.5 from 1.0.12
Hi All,
We followed the upgrade guide(
http://www.datastax.com/docs/1.2/install/upgrading) from Datastax web site
and upgraded Cassadra to 1.2.5, but it occurred errors in system.log when
starting up.
After digging into code level, it looks like Cassandra found the file
length of IndexSummary sstable is zero. Thus Cassandra threw
AssertionError. In fact, the file length of the IndexSummary is about 80
bytes, not zero. It's weird.
Also we observed that only happens on the IndexSummary file of secondary
index. The errors can be reproducible. Below are my upgrade steps.
1. Shutdown all of client applications.
2. Run "nodetool drain" before shutting down the existing Cassandra service.
3. Stop old Cassandra process, then start the new binary process using
migrated cassandra.yaml.
4. Run "nodetool upgradesstables -a" in order to upgrade all of sstable
files become new format.
5. Restart Cassandra process and monitor the logs file for any issues.
At step 5, we found the error messages as below.
Any ideas?
Thank you!
Colin
===================================================================================================================================
INFO [SSTableBatchOpen:2] 2013-05-29 04:38:40,085 SSTableReader.java (line
169) Opening
/var/lib/cassandra/data/ks/user/ks-user.ks_user_personalID-ic-61 (58 bytes)
ERROR [SSTableBatchOpen:1] 2013-05-29 04:38:40,085 CassandraDaemon.java
(line 175) Exception in thread Thread[SSTableBatchOpen:1,5,main]
java.lang.AssertionError
at
org.apache.cassandra.utils.ByteBufferUtil.readBytes(ByteBufferUtil.java:401)
at
org.apache.cassandra.io.sstable.IndexSummary$IndexSummarySerializer.deserialize(IndexSummary.java:124)
at
org.apache.cassandra.io.sstable.SSTableReader.loadSummary(SSTableReader.java:426)
at
org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:360)
at
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:201)
at
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:154)
at
org.apache.cassandra.io.sstable.SSTableReader$1.run(SSTableReader.java:241)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
ERROR [SSTableBatchOpen:2] 2013-05-29 04:38:40,085 CassandraDaemon.java
(line 175) Exception in thread Thread[SSTableBatchOpen:2,5,main]
java.lang.AssertionError
at
org.apache.cassandra.utils.ByteBufferUtil.readBytes(ByteBufferUtil.java:401)
at
org.apache.cassandra.io.sstable.IndexSummary$IndexSummarySerializer.deserialize(IndexSummary.java:124)
at
org.apache.cassandra.io.sstable.SSTableReader.loadSummary(SSTableReader.java:426)
at
org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:360)
at
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:201)
at
org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:154)
at
org.apache.cassandra.io.sstable.SSTableReader$1.run(SSTableReader.java:241)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
===================================================================================================================================
Re: Starting up Cassandra occurred errors after upgrading Cassandra
to 1.2.5 from 1.0.12
Posted by Robert Coli <rc...@eventbrite.com>.
On Wed, May 29, 2013 at 12:32 AM, Colin Kuo <co...@gmail.com> wrote:
> We followed the upgrade
> guide(http://www.datastax.com/docs/1.2/install/upgrading) from Datastax web
> site and upgraded Cassadra to 1.2.5, but it occurred errors in system.log
> when starting up.
In general you should not upgrade more than one major version. It may
work between some sets of major versions, but has almost certainly not
been tested thoroughly. For best results, upgrade only a single major
version per upgrade.
I understand this mail is not responsive to the actual issue that
you're having now that you've done this. I would figure, FWIW, that
IndexSummary SStables are safe to nuke.
=Rob
Re: Starting up Cassandra occurred errors after upgrading Cassandra
to 1.2.5 from 1.0.12
Posted by Robert Coli <rc...@eventbrite.com>.
On Wed, May 29, 2013 at 12:32 AM, Colin Kuo <co...@gmail.com> wrote:
> We followed the upgrade
> guide(http://www.datastax.com/docs/1.2/install/upgrading) from Datastax web
> site and upgraded Cassadra to 1.2.5, but it occurred errors in system.log
> when starting up.
In general you should not upgrade more than one major version. It may
work between some sets of major versions, but has almost certainly not
been tested thoroughly. For best results, upgrade only a single major
version per upgrade.
I understand this mail is not responsive to the actual issue that
you're having now that you've done this. I would figure, FWIW, that
IndexSummary SStables are safe to nuke.
=Rob