NullPointerException when trying to compact under 3.2

[Nimi Wariboko Jr <ni...@channelmeter.com>]

Hi,

I recently upgraded from 2.1.12 to 3.2, and one issue I'm having is I can
no longer read certain rows from a table. A simple SELECT * FROM `table`
times out, only when the bad partition keys are reached. Trying to query
the affected partition keys directly also causes a timeout.

I think the SSTable might be corrupted because nodetool compact (and
repair) fails (although scrub succeeds). With Debug logging, a compact
results in:

cassandra/data/cmpayments/report_payments-f675e45076ce11e5938129463d90c3f0/ma-3778-big-Data.db:level=0,
/var/lib/cassandra/data/cmpayments/report_payments-f675e45076ce11e5938129463d90c3f0/ma-3780-big-Data.db:level=0,
/var/lib/cassandra/data/cmpayments/report_payments-f675e45076ce11e5938129463d90c3f0/ma-3775-big-Data.db:level=0,
/var/lib/cassandra/data/cmpayments/report_payments-f675e45076ce11e5938129463d90c3f0/ma-3776-big-Data.db:level=0,
/var/lib/cassandra/data/cmpayments/report_payments-f675e45076ce11e5938129463d90c3f0/ma-3798-big-Data.db:level=0,
]
WARN  [CompactionExecutor:5] 2016-01-25 15:13:02,734 SSTableReader.java:261
- Reading cardinality from Statistics.db failed for
/var/lib/cassandra/data/cmpayments/report_payments-f675e45076ce11e5938129463d90c3f0/ma-3778-big-Data.db
ERROR [CompactionExecutor:5] 2016-01-25 15:13:02,777
CassandraDaemon.java:195 - Exception in thread
Thread[CompactionExecutor:5,1,main]
java.lang.NullPointerException: null

(the NPE has no stack trace).

On disk, the SSTable "ma-3778-big-Data.db" does not exist. Even if I scrub
or restart, Cassandra seems to always try to compact this non existent
table (the SSTables on Disk are numbered 3851+).

I'm assuming the SSTables are fubared, and I'd like to restore a snapshot,
but the fact this "ghost" sstable causes the compaction to fail, I'm unsure
if restoring a new set of sstables would actually solve the issue.

Nimi

RE: NullPointerException when trying to compact under 3.2

[SE...@homedepot.com]

Can you wipe all the data directories, saved_cache, and commitlog and let the node bootstrap again?


Sean Durity

From: Nimi Wariboko Jr [mailto:nimi@channelmeter.com]
Sent: Monday, January 25, 2016 6:59 PM
To: cassandra-user@apache.org
Subject: NullPointerException when trying to compact under 3.2

Hi,

I recently upgraded from 2.1.12 to 3.2, and one issue I'm having is I can no longer read certain rows from a table. A simple SELECT * FROM `table` times out, only when the bad partition keys are reached. Trying to query the affected partition keys directly also causes a timeout.

I think the SSTable might be corrupted because nodetool compact (and repair) fails (although scrub succeeds). With Debug logging, a compact results in:

cassandra/data/cmpayments/report_payments-f675e45076ce11e5938129463d90c3f0/ma-3778-big-Data.db:level=0, /var/lib/cassandra/data/cmpayments/report_payments-f675e45076ce11e5938129463d90c3f0/ma-3780-big-Data.db:level=0, /var/lib/cassandra/data/cmpayments/report_payments-f675e45076ce11e5938129463d90c3f0/ma-3775-big-Data.db:level=0, /var/lib/cassandra/data/cmpayments/report_payments-f675e45076ce11e5938129463d90c3f0/ma-3776-big-Data.db:level=0, /var/lib/cassandra/data/cmpayments/report_payments-f675e45076ce11e5938129463d90c3f0/ma-3798-big-Data.db:level=0, ]
WARN  [CompactionExecutor:5] 2016-01-25 15:13:02,734 SSTableReader.java:261 - Reading cardinality from Statistics.db failed for /var/lib/cassandra/data/cmpayments/report_payments-f675e45076ce11e5938129463d90c3f0/ma-3778-big-Data.db
ERROR [CompactionExecutor:5] 2016-01-25 15:13:02,777 CassandraDaemon.java:195 - Exception in thread Thread[CompactionExecutor:5,1,main]
java.lang.NullPointerException: null

(the NPE has no stack trace).

On disk, the SSTable "ma-3778-big-Data.db" does not exist. Even if I scrub or restart, Cassandra seems to always try to compact this non existent table (the SSTables on Disk are numbered 3851+).

I'm assuming the SSTables are fubared, and I'd like to restore a snapshot, but the fact this "ghost" sstable causes the compaction to fail, I'm unsure if restoring a new set of sstables would actually solve the issue.

Nimi

________________________________

The information in this Internet Email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this Email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. When addressed to our clients any opinions or advice contained in this Email are subject to the terms and conditions expressed in any applicable governing The Home Depot terms of business or client engagement letter. The Home Depot disclaims all responsibility and liability for the accuracy and content of this attachment and for any damages or losses arising from any inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other items of a destructive nature, which may be contained in this attachment and shall not be liable for direct, indirect, consequential or special damages in connection with this e-mail message or its attachment.