You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Fred Wulff (JIRA)" <ji...@apache.org> on 2014/12/02 23:06:13 UTC
[jira] [Updated] (CASSANDRA-8409) Node generating a huge number of
tiny sstable_activity flushes
[ https://issues.apache.org/jira/browse/CASSANDRA-8409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Fred Wulff updated CASSANDRA-8409:
----------------------------------
Description:
On one of my nodes, I’m seeing hundreds per second of “INFO 21:28:05 Enqueuing flush of sstable_activity: 0 (0%) on-heap, 33 (0%) off-heap”. tpstats shows a steadily climbing # of pending MemtableFlushWriter/MemtablePostFlush until the node OOMs. When the flushes actually happen the sstable written is invariably 121 bytes. I’m writing pretty aggressively to one of my user tables (sev.mdb_group_pit), but that table's flushing behavior seems reasonable.
tpstats:
{quote}
frew@hostname:~/s_dist/apache-cassandra-2.1.0$ bin/nodetool -h hostname tpstats
Pool Name Active Pending Completed Blocked All time blocked
MutationStage 128 4429 36810 0 0
ReadStage 0 0 1205 0 0
RequestResponseStage 0 0 24910 0 0
ReadRepairStage 0 0 26 0 0
CounterMutationStage 0 0 0 0 0
MiscStage 0 0 0 0 0
HintedHandoff 2 2 9 0 0
GossipStage 0 0 5157 0 0
CacheCleanupExecutor 0 0 0 0 0
InternalResponseStage 0 0 0 0 0
CommitLogArchiver 0 0 0 0 0
CompactionExecutor 4 28 429 0 0
ValidationExecutor 0 0 0 0 0
MigrationStage 0 0 0 0 0
AntiEntropyStage 0 0 0 0 0
PendingRangeCalculator 0 0 11 0 0
MemtableFlushWriter 8 38644 8987 0 0
MemtablePostFlush 1 38940 8735 0 0
MemtableReclaimMemory 0 0 8987 0 0
Message type Dropped
READ 0
RANGE_SLICE 0
_TRACE 0
MUTATION 10457
COUNTER_MUTATION 0
BINARY 0
REQUEST_RESPONSE 0
PAGED_RANGE 0
READ_REPAIR 208
{quote}
I've attached one of the produced sstables.
was:
On one of my nodes, I’m seeing hundreds per second of “INFO 21:28:05 Enqueuing flush of sstable_activity: 0 (0%) on-heap, 33 (0%) off-heap”. tpstats shows a steadily climbing # of pending MemtableFlushWriter/MemtablePostFlush until the node OOMs. When the flushes actually happen the sstable written is invariably 121 bytes. I’m writing pretty aggressively to one of my user tables (sev.mdb_group_pit), but that table's flushing behavior seems reasonable.
tpstats:
{quote}
frew@hostname:~/s_dist/apache-cassandra-2.1.0$ bin/nodetool -h sjd-rn1-10b tpstats
Pool Name Active Pending Completed Blocked All time blocked
MutationStage 128 4429 36810 0 0
ReadStage 0 0 1205 0 0
RequestResponseStage 0 0 24910 0 0
ReadRepairStage 0 0 26 0 0
CounterMutationStage 0 0 0 0 0
MiscStage 0 0 0 0 0
HintedHandoff 2 2 9 0 0
GossipStage 0 0 5157 0 0
CacheCleanupExecutor 0 0 0 0 0
InternalResponseStage 0 0 0 0 0
CommitLogArchiver 0 0 0 0 0
CompactionExecutor 4 28 429 0 0
ValidationExecutor 0 0 0 0 0
MigrationStage 0 0 0 0 0
AntiEntropyStage 0 0 0 0 0
PendingRangeCalculator 0 0 11 0 0
MemtableFlushWriter 8 38644 8987 0 0
MemtablePostFlush 1 38940 8735 0 0
MemtableReclaimMemory 0 0 8987 0 0
Message type Dropped
READ 0
RANGE_SLICE 0
_TRACE 0
MUTATION 10457
COUNTER_MUTATION 0
BINARY 0
REQUEST_RESPONSE 0
PAGED_RANGE 0
READ_REPAIR 208
{quote}
I've attached one of the produced sstables.
> Node generating a huge number of tiny sstable_activity flushes
> --------------------------------------------------------------
>
> Key: CASSANDRA-8409
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8409
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Cassandra 2.1.0, Oracle JDK 1.8.0_25, Ubuntu 12.04
> Reporter: Fred Wulff
> Attachments: system-sstable_activity-ka-67802-Data.db
>
>
> On one of my nodes, I’m seeing hundreds per second of “INFO 21:28:05 Enqueuing flush of sstable_activity: 0 (0%) on-heap, 33 (0%) off-heap”. tpstats shows a steadily climbing # of pending MemtableFlushWriter/MemtablePostFlush until the node OOMs. When the flushes actually happen the sstable written is invariably 121 bytes. I’m writing pretty aggressively to one of my user tables (sev.mdb_group_pit), but that table's flushing behavior seems reasonable.
> tpstats:
> {quote}
> frew@hostname:~/s_dist/apache-cassandra-2.1.0$ bin/nodetool -h hostname tpstats
> Pool Name Active Pending Completed Blocked All time blocked
> MutationStage 128 4429 36810 0 0
> ReadStage 0 0 1205 0 0
> RequestResponseStage 0 0 24910 0 0
> ReadRepairStage 0 0 26 0 0
> CounterMutationStage 0 0 0 0 0
> MiscStage 0 0 0 0 0
> HintedHandoff 2 2 9 0 0
> GossipStage 0 0 5157 0 0
> CacheCleanupExecutor 0 0 0 0 0
> InternalResponseStage 0 0 0 0 0
> CommitLogArchiver 0 0 0 0 0
> CompactionExecutor 4 28 429 0 0
> ValidationExecutor 0 0 0 0 0
> MigrationStage 0 0 0 0 0
> AntiEntropyStage 0 0 0 0 0
> PendingRangeCalculator 0 0 11 0 0
> MemtableFlushWriter 8 38644 8987 0 0
> MemtablePostFlush 1 38940 8735 0 0
> MemtableReclaimMemory 0 0 8987 0 0
> Message type Dropped
> READ 0
> RANGE_SLICE 0
> _TRACE 0
> MUTATION 10457
> COUNTER_MUTATION 0
> BINARY 0
> REQUEST_RESPONSE 0
> PAGED_RANGE 0
> READ_REPAIR 208
> {quote}
> I've attached one of the produced sstables.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)