You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Omri Bahumi (JIRA)" <ji...@apache.org> on 2014/08/15 10:31:18 UTC
[jira] [Created] (CASSANDRA-7775) Cassandra attempts to flush an
empty memtable into disk and fails
Omri Bahumi created CASSANDRA-7775:
--------------------------------------
Summary: Cassandra attempts to flush an empty memtable into disk and fails
Key: CASSANDRA-7775
URL: https://issues.apache.org/jira/browse/CASSANDRA-7775
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: $ nodetool version
ReleaseVersion: 2.0.6
$ java -version
java version "1.7.0_51"
Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode)
Reporter: Omri Bahumi
I'm not sure what triggers this flush, but when it happens the following appears in our logs:
{code}
INFO [OptionalTasks:1] 2014-08-15 02:24:20,115 ColumnFamilyStore.java (line 785) Enqueuing flush of Memtable-app_recs_best_in_expr_prefix2@1219170646(0/0 serialized/live bytes, 0 ops)
INFO [FlushWriter:34] 2014-08-15 02:24:20,116 Memtable.java (line 331) Writing Memtable-app_recs_best_in_expr_prefix2@1219170646(0/0 serialized/live bytes, 0 ops)
ERROR [FlushWriter:34] 2014-08-15 02:24:20,127 CassandraDaemon.java (line 196) Exception in thread Thread[FlushWriter:34,5,main]
java.lang.RuntimeException: Cannot get comparator 1 in org.apache.cassandra.db.marshal.CompositeType(org.apache.cassandra.db.marshal.UTF8Type). This might due to a mismatch between the schema and the data read
at org.apache.cassandra.db.marshal.CompositeType.getComparator(CompositeType.java:133)
at org.apache.cassandra.db.marshal.CompositeType.getComparator(CompositeType.java:140)
at org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:96)
at org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:35)
at org.apache.cassandra.db.RangeTombstone$Tracker$1.compare(RangeTombstone.java:125)
at org.apache.cassandra.db.RangeTombstone$Tracker$1.compare(RangeTombstone.java:122)
at java.util.TreeMap.compare(TreeMap.java:1188)
at java.util.TreeMap$NavigableSubMap.<init>(TreeMap.java:1264)
at java.util.TreeMap$AscendingSubMap.<init>(TreeMap.java:1699)
at java.util.TreeMap.tailMap(TreeMap.java:905)
at java.util.TreeSet.tailSet(TreeSet.java:350)
at java.util.TreeSet.tailSet(TreeSet.java:383)
at org.apache.cassandra.db.RangeTombstone$Tracker.update(RangeTombstone.java:203)
at org.apache.cassandra.db.ColumnIndex$Builder.add(ColumnIndex.java:192)
at org.apache.cassandra.db.ColumnIndex$Builder.build(ColumnIndex.java:138)
at org.apache.cassandra.io.sstable.SSTableWriter.rawAppend(SSTableWriter.java:202)
at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:187)
at org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:365)
at org.apache.cassandra.db.Memtable$FlushRunnable.runWith(Memtable.java:318)
at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.IndexOutOfBoundsException: index (1) must be less than size (1)
at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:306)
at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:285)
at com.google.common.collect.SingletonImmutableList.get(SingletonImmutableList.java:45)
at org.apache.cassandra.db.marshal.CompositeType.getComparator(CompositeType.java:124)
... 23 more
{code}
After this happens, the MemtablePostFlusher thread pool starts piling up.
When trying to restart the cluster, a similar exception occurs when trying to replay the commit log.
Our way of recovering from this is to delete all commit logs in the faulty node, start it and issue a repair.
--
This message was sent by Atlassian JIRA
(v6.2#6252)