You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by "Steinmaurer, Thomas" <th...@dynatrace.com> on 2019/02/05 22:02:39 UTC

Cassandra 2.1.18 - NPE during startup

Hello,

at a particular customer location, we are seeing the following NPE during startup with Cassandra 2.1.18.

INFO  [SSTableBatchOpen:2] 2019-02-03 13:32:56,131 SSTableReader.java:475 - Opening /var/opt/data/cassandra/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-130 (256 bytes)
ERROR [main] 2019-02-03 13:32:56,552 CassandraDaemon.java:583 - Exception encountered during startup
org.apache.cassandra.io.FSReadError: java.lang.NullPointerException
                at org.apache.cassandra.db.ColumnFamilyStore.removeUnfinishedCompactionLeftovers(ColumnFamilyStore.java:672) ~[apache-cassandra-2.1.18.jar:2.1.18]
                at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:310) [apache-cassandra-2.1.18.jar:2.1.18]
                at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:566) [apache-cassandra-2.1.18.jar:2.1.18]
                at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:655) [apache-cassandra-2.1.18.jar:2.1.18]
Caused by: java.lang.NullPointerException: null
                at org.apache.cassandra.db.ColumnFamilyStore.removeUnfinishedCompactionLeftovers(ColumnFamilyStore.java:664) ~[apache-cassandra-2.1.18.jar:2.1.18]
                ... 3 common frames omitted

I found https://issues.apache.org/jira/browse/CASSANDRA-10501, but this should be fixed in 2.1.18.

Is the above log stating that it is caused by a system keyspace related SSTable?

This is a 3 node setup with 2 others running fine. If system table related and as LocalStrategy is used as replication strategy (to my knowledge), perhaps simply copying over data for the schema_keyspaces table from another node might fix it?

Any help appreciated.

Thanks.
Thomas
The contents of this e-mail are intended for the named addressee only. It contains information that may be confidential. Unless you are the named addressee or an authorized designee, you may not copy or use it, or disclose it to anyone else. If you received it in error please notify us immediately and then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a company registered in Linz whose registered office is at 4040 Linz, Austria, Freist?dterstra?e 313

Re: Cassandra 2.1.18 - NPE during startup

Posted by Anthony Grasso <an...@gmail.com>.
Hi Thomas,

The process you suggested to get around the issue should work with the
system.keyspaces table.

Make sure to backup the original *system.keyspaces* table files on the node
that fails to start. Then, copy only the *system.keyspaces *table files
from a working node into the *system/schema_keyspaces-...* folder of the
node that fails to start.

This method will only work for certain system tables as some of the data
stored in the system tables will differ between nodes.

Regards,
Anthony

On Thu, 28 Mar 2019 at 05:54, Steinmaurer, Thomas <
thomas.steinmaurer@dynatrace.com> wrote:

> Hello,
>
>
>
> any ideas regarding below, cause it happened again on a different node.
>
>
>
> Thanks
>
> Thomas
>
>
>
> *From:* Steinmaurer, Thomas <th...@dynatrace.com>
> *Sent:* Dienstag, 05. Februar 2019 23:03
> *To:* user@cassandra.apache.org
> *Subject:* Cassandra 2.1.18 - NPE during startup
>
>
>
> Hello,
>
>
>
> at a particular customer location, we are seeing the following NPE during
> startup with Cassandra 2.1.18.
>
>
>
> INFO  [SSTableBatchOpen:2] 2019-02-03 13:32:56,131 SSTableReader.java:475
> - Opening
> /var/opt/data/cassandra/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-130
> (256 bytes)
>
> ERROR [main] 2019-02-03 13:32:56,552 CassandraDaemon.java:583 - Exception
> encountered during startup
>
> org.apache.cassandra.io.FSReadError: java.lang.NullPointerException
>
>                 at
> org.apache.cassandra.db.ColumnFamilyStore.removeUnfinishedCompactionLeftovers(ColumnFamilyStore.java:672)
> ~[apache-cassandra-2.1.18.jar:2.1.18]
>
>                 at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:310)
> [apache-cassandra-2.1.18.jar:2.1.18]
>
>                 at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:566)
> [apache-cassandra-2.1.18.jar:2.1.18]
>
>                 at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:655)
> [apache-cassandra-2.1.18.jar:2.1.18]
>
> Caused by: java.lang.NullPointerException: null
>
>                 at
> org.apache.cassandra.db.ColumnFamilyStore.removeUnfinishedCompactionLeftovers(ColumnFamilyStore.java:664)
> ~[apache-cassandra-2.1.18.jar:2.1.18]
>
>                 ... 3 common frames omitted
>
>
>
> I found https://issues.apache.org/jira/browse/CASSANDRA-10501
> <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCASSANDRA-10501&data=02%7C01%7Cthomas.steinmaurer%40dynatrace.com%7C796c92ba8c4d4c94937208d68bb69f1d%7C70ebe3a35b30435d9d677716d74ca190%7C1%7C0%7C636850013806426682&sdata=5Fx4An63c5R%2Faa2%2BfFXFCg7%2FDejG1V57st7UNtq9hrA%3D&reserved=0>,
> but this should be fixed in 2.1.18.
>
>
>
> Is the above log stating that it is caused by a system keyspace related
> SSTable?
>
>
>
> This is a 3 node setup with 2 others running fine. If system table related
> and as LocalStrategy is used as replication strategy (to my knowledge),
> perhaps simply copying over data for the schema_keyspaces table from
> another node might fix it?
>
>
>
> Any help appreciated.
>
>
>
> Thanks.
>
> Thomas
>
> The contents of this e-mail are intended for the named addressee only. It
> contains information that may be confidential. Unless you are the named
> addressee or an authorized designee, you may not copy or use it, or
> disclose it to anyone else. If you received it in error please notify us
> immediately and then destroy it. Dynatrace Austria GmbH (registration
> number FN 91482h) is a company registered in Linz whose registered office
> is at 4040 Linz, Austria, Freistädterstraße 313
> The contents of this e-mail are intended for the named addressee only. It
> contains information that may be confidential. Unless you are the named
> addressee or an authorized designee, you may not copy or use it, or
> disclose it to anyone else. If you received it in error please notify us
> immediately and then destroy it. Dynatrace Austria GmbH (registration
> number FN 91482h) is a company registered in Linz whose registered office
> is at 4040 Linz, Austria, Freistädterstraße 313
>

RE: Cassandra 2.1.18 - NPE during startup

Posted by "Steinmaurer, Thomas" <th...@dynatrace.com>.
Hello,

any ideas regarding below, cause it happened again on a different node.

Thanks
Thomas

From: Steinmaurer, Thomas <th...@dynatrace.com>
Sent: Dienstag, 05. Februar 2019 23:03
To: user@cassandra.apache.org
Subject: Cassandra 2.1.18 - NPE during startup

Hello,

at a particular customer location, we are seeing the following NPE during startup with Cassandra 2.1.18.

INFO  [SSTableBatchOpen:2] 2019-02-03 13:32:56,131 SSTableReader.java:475 - Opening /var/opt/data/cassandra/system/schema_keyspaces-b0f2235744583cdb9631c43e59ce3676/system-schema_keyspaces-ka-130 (256 bytes)
ERROR [main] 2019-02-03 13:32:56,552 CassandraDaemon.java:583 - Exception encountered during startup
org.apache.cassandra.io.FSReadError: java.lang.NullPointerException
                at org.apache.cassandra.db.ColumnFamilyStore.removeUnfinishedCompactionLeftovers(ColumnFamilyStore.java:672) ~[apache-cassandra-2.1.18.jar:2.1.18]
                at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:310) [apache-cassandra-2.1.18.jar:2.1.18]
                at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:566) [apache-cassandra-2.1.18.jar:2.1.18]
                at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:655) [apache-cassandra-2.1.18.jar:2.1.18]
Caused by: java.lang.NullPointerException: null
                at org.apache.cassandra.db.ColumnFamilyStore.removeUnfinishedCompactionLeftovers(ColumnFamilyStore.java:664) ~[apache-cassandra-2.1.18.jar:2.1.18]
                ... 3 common frames omitted

I found https://issues.apache.org/jira/browse/CASSANDRA-10501<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCASSANDRA-10501&data=02%7C01%7Cthomas.steinmaurer%40dynatrace.com%7C796c92ba8c4d4c94937208d68bb69f1d%7C70ebe3a35b30435d9d677716d74ca190%7C1%7C0%7C636850013806426682&sdata=5Fx4An63c5R%2Faa2%2BfFXFCg7%2FDejG1V57st7UNtq9hrA%3D&reserved=0>, but this should be fixed in 2.1.18.

Is the above log stating that it is caused by a system keyspace related SSTable?

This is a 3 node setup with 2 others running fine. If system table related and as LocalStrategy is used as replication strategy (to my knowledge), perhaps simply copying over data for the schema_keyspaces table from another node might fix it?

Any help appreciated.

Thanks.
Thomas
The contents of this e-mail are intended for the named addressee only. It contains information that may be confidential. Unless you are the named addressee or an authorized designee, you may not copy or use it, or disclose it to anyone else. If you received it in error please notify us immediately and then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a company registered in Linz whose registered office is at 4040 Linz, Austria, Freistädterstraße 313
The contents of this e-mail are intended for the named addressee only. It contains information that may be confidential. Unless you are the named addressee or an authorized designee, you may not copy or use it, or disclose it to anyone else. If you received it in error please notify us immediately and then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a company registered in Linz whose registered office is at 4040 Linz, Austria, Freistädterstraße 313