[jira] [Created] (CASSANDRA-13624) upgradesstables crashes with OOM when upgrading sstables with lots of range tombstones

["Robin Mahony (JIRA)" <ji...@apache.org>]

Robin Mahony created CASSANDRA-13624:
----------------------------------------

             Summary: upgradesstables crashes with OOM when upgrading sstables with lots of range tombstones
                 Key: CASSANDRA-13624
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-13624
             Project: Cassandra
          Issue Type: Bug
            Reporter: Robin Mahony


CASSANDRA-7953 can lead to range tombstones not properly being compacted. When trying to upgrade from Cassandra 2.0.X to 2.1.X, running upgradesstables (either via nodetool OR the offline version) does not work, as it crashes with OOM. This essentially means if you are running a version of Cassandra 2.0.X for a period of time where lots of range tombstones have been generated; you will be unable to upgrade to Cassandra 2.1.X following the normal procedures. Hence upgrade from 2.0.X to 2.1.X is essentially broken if you hit CASSANDRA-7953.

I hit this while trying to upgrade to Cassandra 2.1.17.

Offline Version:

# sstableupgrade storagegrid s3_usage_delta 
Found 15 sstables that need upgrading. 
Upgrading SSTableReader(path='/var/local/cassandra/data/0/storagegrid/s3_usage_delta/storagegrid-s3_usage_delta-jb-20230-Data.db') 
ERROR 22:38:24,626 LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@6821300e) to class org.apache.cassandra.io.sstable.SSTableReader$InstanceTidier@244363601:/var/local/cassandra/data/0/storagegrid/s3_usage_delta/storagegrid-s3_usage_delta-ka-79531 was not released before the reference was garbage collected 
ERROR 22:38:24,631 LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@2891457f) to class org.apache.cassandra.io.sstable.SSTableReader$InstanceTidier@1230033823:/var/local/cassandra/data/0/storagegrid/s3_usage_delta/storagegrid-s3_usage_delta-ka-79532 was not released before the reference was garbage collected 
Exception in thread "ScheduledTasks:1" Exception in thread "metrics-meter-tick-thread-2" java.lang.OutOfMemoryError: Java heap space 
at java.lang.Class.getName0(Native Method) 
at java.lang.Class.getName(Class.java:642) 
at java.lang.Throwable.toString(Throwable.java:479) 
at java.lang.Throwable.<init>(Throwable.java:311) 
at java.lang.Exception.<init>(Exception.java:102) 
at java.util.concurrent.ExecutionException.<init>(ExecutionException.java:90) 
at java.util.concurrent.FutureTask.report(FutureTask.java:122) 
at java.util.concurrent.FutureTask.get(FutureTask.java:192) 
at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.extractThrowable(DebuggableThreadPoolExecutor.java:246)
at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.logExceptionsAfterExecute(DebuggableThreadPoolExecutor.java:210)
at org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor.afterExecute(DebuggableScheduledThreadPoolExecutor.java:89)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1150) 
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
at java.lang.Thread.run(Thread.java:745) 
java.lang.OutOfMemoryError: Java heap space 
java.lang.OutOfMemoryError: Java heap space 
ERROR 22:41:21,080 Error in ThreadPoolExecutor 
java.lang.OutOfMemoryError: Java heap space 
ERROR 22:41:21,080 JVM state determined to be unstable. Exiting forcefully due to: 
java.lang.OutOfMemoryError: Java heap space 
root@DC1-SN-10-224-6-066:/var/local/cassandra/data/0 #



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org