You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Francois Richard (JIRA)" <ji...@apache.org> on 2014/06/10 00:53:02 UTC
[jira] [Created] (CASSANDRA-7373) Commit logs no longer deleting
and MemtablePostFlusher pending growing
Francois Richard created CASSANDRA-7373:
-------------------------------------------
Summary: Commit logs no longer deleting and MemtablePostFlusher pending growing
Key: CASSANDRA-7373
URL: https://issues.apache.org/jira/browse/CASSANDRA-7373
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: RHEL 6.5
Cassandra 1.12.16
Replication factor of 3
Reporter: Francois Richard
We have this issue where once in a while, we get into a situation where the MemtablePostFlusher is not executing and the space used by the commit logs on disks keeps on increasing and increasing.
We can observe the problem by invoking nodetool tpstats:
{code}
Pool Name Active Pending Completed Blocked All time blocked
ReadStage 6 6 46650213 0 0
RequestResponseStage 0 0 130547421 0 0
MutationStage 2 2 116813206 0 0
ReadRepairStage 0 0 2322201 0 0
ReplicateOnWriteStage 0 0 0 0 0
GossipStage 0 0 120780 0 0
AntiEntropyStage 0 0 0 0 0
MigrationStage 0 0 0 0 0
MemoryMeter 0 0 456 0 0
MemtablePostFlusher 1 447 6344 0 0
FlushWriter 0 0 6132 0 62
MiscStage 0 0 0 0 0
PendingRangeCalculator 0 0 6 0 0
commitlog_archiver 0 0 0 0 0
InternalResponseStage 0 0 0 0 0
HintedHandoff 2 2 4 0 0
Message type Dropped
RANGE_SLICE 0
READ_REPAIR 0
BINARY 0
READ 0
MUTATION 0
_TRACE 0
REQUEST_RESPONSE 0
COUNTER_MUTATION 0
{code}
Here is a potential error in the logs that can explain this:
{code}
ERROR [FlushWriter:2693] 2014-06-09 22:05:38,452 CassandraDaemon.java (line 191) Exception in thread Thread[FlushWriter:2693,5,main]
java.lang.NegativeArraySizeException
at org.apache.cassandra.io.util.FastByteArrayOutputStream.expand(FastByteArrayOutputStream.java:104)
at org.apache.cassandra.io.util.FastByteArrayOutputStream.write(FastByteArrayOutputStream.java:220)
at java.io.DataOutputStream.write(DataOutputStream.java:107)
at org.apache.cassandra.io.util.DataOutputBuffer.write(DataOutputBuffer.java:60)
at org.apache.cassandra.utils.ByteBufferUtil.write(ByteBufferUtil.java:328)
at org.apache.cassandra.utils.ByteBufferUtil.writeWithLength(ByteBufferUtil.java:315)
at org.apache.cassandra.db.ColumnSerializer.serialize(ColumnSerializer.java:55)
at org.apache.cassandra.db.ColumnSerializer.serialize(ColumnSerializer.java:30)
at org.apache.cassandra.db.OnDiskAtom$Serializer.serializeForSSTable(OnDiskAtom.java:62)
at org.apache.cassandra.db.ColumnIndex$Builder.add(ColumnIndex.java:181)
at org.apache.cassandra.db.ColumnIndex$Builder.build(ColumnIndex.java:133)
at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:185)
at org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:430)
at org.apache.cassandra.db.Memtable$FlushRunnable.runWith(Memtable.java:385)
at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)