You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Tom van der Woerdt (JIRA)" <ji...@apache.org> on 2017/04/23 13:27:04 UTC
[jira] [Created] (CASSANDRA-13469) Compaction stops; new sstables
aren't recognized
Tom van der Woerdt created CASSANDRA-13469:
----------------------------------------------
Summary: Compaction stops; new sstables aren't recognized
Key: CASSANDRA-13469
URL: https://issues.apache.org/jira/browse/CASSANDRA-13469
Project: Cassandra
Issue Type: Bug
Reporter: Tom van der Woerdt
cfstats shows me this, so I know I'm in trouble :
{code}
SSTable count: 3419
SSTables in each level: [2, 20/10, 154/100, 1034/1000, 0, 0, 0, 0, 0]
{code}
Disk space on these nodes (about five of them) doubled overnight, due to the high write volume not getting compacted. `nodetool compactionstats' shows me zero pending compactions.
After a restart things are "better" :
{noformat}
SSTable count: 3432
SSTables in each level: [1280/4, 21/10, 262/100, 1851/1000, 15, 0, 0, 0, 0]
{noformat}
compactionstats also shows some compactions again.
jstack for one compaction thread (there are 24 of them, all looking the exact same) :
{noformat}
"CompactionExecutor:73017" #203194 daemon prio=1 os_prio=4 tid=0x00007edd1e597c00 nid=0xf7e4 waiting on condition [0x00007efd16782000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000002c1859520> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
at org.apache.cassandra.concurrent.NamedThreadFactory$$Lambda$3/335359181.run(Unknown Source)
at java.lang.Thread.run(Thread.java:745)
{noformat}
Sadly I'm pretty clueless about what happened here. :(
Oh, and tpstats :
{noformat}
Pool Name Active Pending Completed Blocked All time blocked
MutationStage 0 0 1098466484 0 0
ViewMutationStage 0 0 0 0 0
ReadStage 2 0 759723051 0 0
RequestResponseStage 0 0 1773795694 0 0
ReadRepairStage 0 0 37372223 0 0
CounterMutationStage 0 0 0 0 0
MiscStage 0 0 0 0 0
CompactionExecutor 0 0 336794 0 0
MemtableReclaimMemory 0 0 75324 0 0
PendingRangeCalculator 0 0 68 0 0
GossipStage 0 0 1363346 0 0
SecondaryIndexManagement 0 0 0 0 0
HintsDispatcher 1 1 38 0 0
MigrationStage 0 0 274693 0 0
MemtablePostFlush 0 0 248358 0 0
ValidationExecutor 0 0 7882 0 0
Sampler 0 0 0 0 0
MemtableFlushWriter 0 0 75324 0 0
InternalResponseStage 0 0 259871 0 0
AntiEntropyStage 0 0 20469 0 0
CacheCleanupExecutor 0 0 0 0 0
Native-Transport-Requests 0 0 1489435995 0 188985
Message type Dropped
READ 2
RANGE_SLICE 0
_TRACE 0
HINT 0
MUTATION 20
COUNTER_MUTATION 0
BATCH_STORE 0
BATCH_REMOVE 0
REQUEST_RESPONSE 0
PAGED_RANGE 0
READ_REPAIR 4
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)