You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Tom Petracca (JIRA)" <ji...@apache.org> on 2019/04/22 15:00:00 UTC
[jira] [Commented] (CASSANDRA-14585) For compactions to progress
when pending is greater than 10K cassandra must be continuously restarted
and compactions_in_progress purged
[ https://issues.apache.org/jira/browse/CASSANDRA-14585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16823162#comment-16823162 ]
Tom Petracca commented on CASSANDRA-14585:
------------------------------------------
I have many clusters (all on 2.2.13) experiencing a similar, though significantly less pronounced, issue. My workflows are largely read-heavy write-light with all columnfamilies using LCS. When I do see write throughput it is generally fairly bursty and across a number of columnfamilies at once.
During these bursty write phases the number of sstables in the system.compactions_in_progress table grow incredibly quickly (thus pending compactions growing with it) and the cluster experiences mildly increased latencies that recover when the backlog of compactions is addressed / brought to zero; unlike the reporter, my nodes are able to recover on their own.
Does anyone know why we force a flush on every modification to the system.compactions_in_progress table? Is there some correctness reason I'm not seeing? It seems like it would cause a needless amount of sstable creation in that table during periods of heavy ingestion/compaction.
[https://github.com/apache/cassandra/blob/cassandra-2.2.13/src/java/org/apache/cassandra/db/SystemKeyspace.java#L354]
[https://github.com/apache/cassandra/blob/cassandra-2.2.13/src/java/org/apache/cassandra/db/SystemKeyspace.java#L368]
> For compactions to progress when pending is greater than 10K cassandra must be continuously restarted and compactions_in_progress purged
> ----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-14585
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14585
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Compaction
> Environment: Apache Cassandra 2.2.12
> Reporter: Zachary Schneider
> Assignee: Tom Petracca
> Priority: Normal
>
> We are experiencing the following behavior when utilizing DTCS on nodes that join the ring and subsequently have many pending compactions.
> The symptoms are the following:
> nodetool compactionstats reports compactions progressing, however pending count is never significantly reduced.
> When investigating we notice the following:
> compactions_in_progress is never compacted
> {code:java}
> find /mnt/data/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/ | wc -l
> 560730
> {code}
> Eventually we will end up running into to many open files.
> In order for compaction to progress we must:
> * truncate compactions_in_progress
> * restart Cassandra
> This compactions will progress rapidly 2K or so and eventually the process will hang again.
> In the debug log we see constant flushes to compactions_in_progress.
> {code:java}
> DEBUG [CompactionExecutor:10] 2018-07-24 07:39:29,552 ColumnFamilyStore.java:956 - Enqueuing flush of compactions_in_progress: 428 (0%) on-heap, 354 (0%) off-heap INFO [MemtableFlushWriter:9] 2018-07-24 07:39:29,552 Memtable.java:352 - Writing Memtable-compactions_in_progress@1407915569(0.263KiB serialized bytes, 12 ops, 0%/0% of on/off-heap limit) DEBUG [MemtableFlushWriter:9] 2018-07-24 07:39:29,553 Memtable.java:386 - Completed flushing /mnt/data/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/tmp-lb-8140-big-Data.db (0.000KiB) for commitlog position ReplayPosition(segmentId=1532415927699, position=28982298) DEBUG [CompactionExecutor:16] 2018-07-24 07:39:29,595 ColumnFamilyStore.java:956 - Enqueuing flush of compactions_in_progress: 460 (0%) on-heap, 397 (0%) off-heap INFO [MemtableFlushWriter:13] 2018-07-24 07:39:29,595 Memtable.java:352 - Writing Memtable-compactions_in_progress@535072625(0.298KiB serialized bytes, 13 ops, 0%/0% of on/off-heap limit) DEBUG [MemtableFlushWriter:13] 2018-07-24 07:39:29,595 Memtable.java:386 - Completed flushing /mnt/data/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/tmp-lb-8141-big-Data.db (0.000KiB) for commitlog position ReplayPosition(segmentId=1532415927699, position=29708684) DEBUG [CompactionExecutor:15] 2018-07-24 07:39:29,619 ColumnFamilyStore.java:956 - Enqueuing flush of compactions_in_progress: 428 (0%) on-heap, 356 (0%) off-heap INFO [MemtableFlushWriter:12] 2018-07-24 07:39:29,619 Memtable.java:352 - Writing Memtable-compactions_in_progress@319301851(0.265KiB serialized bytes, 12 ops, 0%/0% of on/off-heap limit) DEBUG [MemtableFlushWriter:12] 2018-07-24 07:39:29,620 Memtable.java:386 - Completed flushing /mnt/data/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/tmp-lb-8142-big-Data.db (0.000KiB) for commitlog position ReplayPosition(segmentId=1532415927699, position=30591989) DEBUG [CompactionExecutor:9] 2018-07-24 07:39:29,660 ColumnFamilyStore.java:956 - Enqueuing flush of compactions_in_progress: 460 (0%) on-heap, 396 (0%) off-heap INFO [MemtableFlushWriter:15] 2018-07-24 07:39:29,660 Memtable.java:352 - Writing Memtable-compactions_in_progress@1000370375(0.297KiB serialized bytes, 13 ops, 0%/0% of on/off-heap limit) DEBUG [MemtableFlushWriter:15] 2018-07-24 07:39:29,660 Memtable.java:386 - Completed flushing /mnt/data/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/tmp-lb-8143-big-Data.db (0.000KiB) for commitlog position ReplayPosition(segmentId=1532415927699, position=31558471) DEBUG [CompactionExecutor:14] 2018-07-24 07:39:29,688 ColumnFamilyStore.java:956 - Enqueuing flush of compactions_in_progress: 460 (0%) on-heap, 389 (0%) off-heap INFO [MemtableFlushWriter:10] 2018-07-24 07:39:29,689 Memtable.java:352 - Writing Memtable-compactions_in_progress@204811984(0.290KiB serialized bytes, 13 ops, 0%/0% of on/off-heap limit) DEBUG [MemtableFlushWriter:10] 2018-07-24 07:39:29,689 Memtable.java:386 - Completed flushing /mnt/data/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/tmp-lb-8144-big-Data.db (0.000KiB) for commitlog position ReplayPosition(segmentId=1532415927699, position=32153206)
> {code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org