You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Timo Nentwig <ti...@toptarif.de> on 2010/12/23 09:34:18 UTC

Adding secondary index: java.lang.ArithmeticException: / by zero

I was about to add a secondary index (which apparently failed) to existing data. When I restarted the node it crashed (!) with:

 INFO 09:21:36,510 Opening /var/lib/cassandra/data/test/tracking.6b6579-tmp-e-1
ERROR 09:21:36,512 Exception encountered during startup.
java.lang.ArithmeticException: / by zero
        at org.apache.cassandra.io.sstable.SSTable.estimateRowsFromIndex(SSTable.java:233)
        at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:284)
        at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:200)
        at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:225)
        at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:448)
        at org.apache.cassandra.db.ColumnFamilyStore.addIndex(ColumnFamilyStore.java:305)
        at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:246)
        at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:448)
        at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:436)
        at org.apache.cassandra.db.Table.initCf(Table.java:360)
        at org.apache.cassandra.db.Table.<init>(Table.java:290)
        at org.apache.cassandra.db.Table.open(Table.java:107)
        at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:138)
        at org.apache.cassandra.thrift.CassandraDaemon.setup(CassandraDaemon.java:55)
        at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:216)
        at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:134)
Exception encountered during startup.
java.lang.ArithmeticException: / by zero
        at org.apache.cassandra.io.sstable.SSTable.estimateRowsFromIndex(SSTable.java:233)
        at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:284)
        at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:200)
        at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:225)
        at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:448)
        at org.apache.cassandra.db.ColumnFamilyStore.addIndex(ColumnFamilyStore.java:305)
        at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:246)
        at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:448)
        at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:436)
        at org.apache.cassandra.db.Table.initCf(Table.java:360)
        at org.apache.cassandra.db.Table.<init>(Table.java:290)
        at org.apache.cassandra.db.Table.open(Table.java:107)
        at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:138)
        at org.apache.cassandra.thrift.CassandraDaemon.setup(CassandraDaemon.java:55)
        at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:216)
        at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:134)

So, I deleted the file which let's cassandra to startup again (and starting all over to build the secondary index).

Since 0.7rc2 was too unstable I'm on SNAPSHOT from Dec 17.

Re: Adding secondary index: java.lang.ArithmeticException: / by zero

Posted by Timo Nentwig <ti...@toptarif.de>.
On Dec 27, 2010, at 14:34, Timo Nentwig wrote:

> On Dec 24, 2010, at 14:33, Timo Nentwig wrote:
>> Any advice what to do with it?
> 
> So, to continue this monologue: I reduced the memtable size for that CF and the by means of the MBeans figured out that the secondary index is a CF as well which presumably also holds up to 3 memtables in memory...that would explain why it runs OOM.

Hm. I was wrong. In order to somehow stop cassandra to continue to build the index I deleted system CF and reimported the schema with modified memtable flush parameters. But apparently the information to build the secondary index is stored in the CF's metadata, so shortly after it starts indexing again and the index shows up in mbeans.

However: data is never flushed and grows endlessly because of (ColumnFamilyStore):

            if (DatabaseDescriptor.getCFMetaData(metadata.cfId) == null)
                return null; // column family was dropped. no point in flushing.

Did I hit some problem here or is the data just messed up (was build with rc1, rc2 and I removed system CF). Sigh...

> Anyway, I created an issue (affect rc2 beause there's yet no rc3 to select in Jira) for the crash at startup: CASSANDRA-1904.

Re: Adding secondary index: java.lang.ArithmeticException: / by zero

Posted by Timo Nentwig <ti...@toptarif.de>.
On Dec 24, 2010, at 14:33, Timo Nentwig wrote:
> Any advice what to do with it?


So, to continue this monologue: I reduced the memtable size for that CF and the by means of the MBeans figured out that the secondary index is a CF as well which presumably also holds up to 3 memtables in memory...that would explain why it runs OOM.

Anyway, I created an issue (affect rc2 beause there's yet no rc3 to select in Jira) for the crash at startup: CASSANDRA-1904.

Re: Adding secondary index: java.lang.ArithmeticException: / by zero

Posted by Timo Nentwig <ti...@toptarif.de>.
On Dec 23, 2010, at 12:34, Timo Nentwig wrote:

> On Dec 23, 2010, at 9:34, Timo Nentwig wrote:
> 
>> I was about to add a secondary index (which apparently failed) to existing data. When I restarted the node it crashed (!) with:
> 
> It crashed because it ran out of heap space (2G). So I increased to 3.5G but after a whlie it's caught in full GC again. The node holds some 50G of data. So, if 3.5G isn't sufficient, how do I determine how much memory it's going to need and more important how can I cancel such indexing if I don't have enough memory available? Or did I hit some bug and should simply wait for rc3?

The good news: with 3.5G it didn't crash, yet. 
The bad news: the node is full GCing for over 1 day now... 

Any advice what to do with it?

Re: Adding secondary index: java.lang.ArithmeticException: / by zero

Posted by Timo Nentwig <ti...@toptarif.de>.
On Dec 23, 2010, at 9:34, Timo Nentwig wrote:

> I was about to add a secondary index (which apparently failed) to existing data. When I restarted the node it crashed (!) with:

It crashed because it ran out of heap space (2G). So I increased to 3.5G but after a whlie it's caught in full GC again. The node holds some 50G of data. So, if 3.5G isn't sufficient, how do I determine how much memory it's going to need and more important how can I cancel such indexing if I don't have enough memory available? Or did I hit some bug and should simply wait for rc3?

> INFO 09:21:36,510 Opening /var/lib/cassandra/data/test/tracking.6b6579-tmp-e-1
> ERROR 09:21:36,512 Exception encountered during startup.
> java.lang.ArithmeticException: / by zero
...
> So, I deleted the file which let's cassandra to startup again (and starting all over to build the secondary index).
> 
> Since 0.7rc2 was too unstable I'm on SNAPSHOT from Dec 17.