You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Marcus Eriksson (JIRA)" <ji...@apache.org> on 2019/07/22 09:56:00 UTC
[jira] [Commented] (CASSANDRA-15242) Race condition between
flushing and compaction stalls compaction indefinitely
[ https://issues.apache.org/jira/browse/CASSANDRA-15242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16890045#comment-16890045 ]
Marcus Eriksson commented on CASSANDRA-15242:
---------------------------------------------
Could you post more logs? It is either {{md-292348-big-Data.db}} or {{md-292358-big-Data.db}} causing this, so logs starting with when {{md-292348-big-Data.db}} was created would be really helpful
> Race condition between flushing and compaction stalls compaction indefinitely
> -----------------------------------------------------------------------------
>
> Key: CASSANDRA-15242
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15242
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Compaction/LCS
> Reporter: Tom van der Woerdt
> Priority: Normal
>
> Seen on Cassandra 3.11.4 with OpenJDK 8u212, although I've seen this a few times before, also on 3.11.3. It's a rare issue so I've not bothered with trying to trace it until now.
> {noformat}
> DEBUG [NativePoolCleaner] 2019-07-18 01:12:41,799 ColumnFamilyStore.java:1325 - Flushing largest CFS(Keyspace='keyspacename', ColumnFamily='tablename') to free up room. Used total: 0.10/0.33, live: 0.10/0.33, flushing: 0.00/0.00, this: 0.09/0.19
> DEBUG [NativePoolCleaner] 2019-07-18 01:12:41,800 ColumnFamilyStore.java:935 - Enqueuing flush of tablename: 267.930MiB (9%) on-heap, 575.580MiB (19%) off-heap
> DEBUG [PerDiskMemtableFlushWriter_0:204] 2019-07-18 01:12:42,480 Memtable.java:456 - Writing Memtable-tablename@498336646(520.721MiB serialized bytes, 870200 ops, 9%/19% of on/off-heap limit), flushed range = (min(-9223372036854775808), max(9223372036854775807)]
> INFO [Service Thread] 2019-07-18 01:12:43,616 GCInspector.java:284 - G1 Young Generation GC in 227ms. G1 Eden Space: 14713618432 -> 0; G1 Old Gen: 13240876928 -> 13259198848; G1 Survivor Space: 276824064 -> 268435456;
> INFO [Service Thread] 2019-07-18 01:12:56,251 GCInspector.java:284 - G1 Young Generation GC in 206ms. G1 Eden Space: 14713618432 -> 0; G1 Old Gen: 13259198848 -> 13285123456; G1 Survivor Space: 268435456 -> 285212672;
> DEBUG [PerDiskMemtableFlushWriter_0:204] 2019-07-18 01:12:56,693 Memtable.java:485 - Completed flushing /cassandra-data/data/keyspacename/tablename-2c5325e042e911e8a07e9db72d27cf67/md-292358-big-Data.db (524.023MiB) for commitlog position CommitLogPosition(segmentId=1563386911266, position=32127822)
> DEBUG [MemtableFlushWriter:204] 2019-07-18 01:12:57,620 ColumnFamilyStore.java:1233 - Flushed to [BigTableReader(path='/cassandra-data/data/keyspacename/tablename-2c5325e042e911e8a07e9db72d27cf67/md-292358-big-Data.db')] (1 sstables, 518.714MiB), biggest 518.714MiB, smallest 518.714MiB
> WARN [CompactionExecutor:1617] 2019-07-18 01:12:57,628 LeveledCompactionStrategy.java:144 - Could not acquire references for compacting SSTables [BigTableReader(path='/cassandra-data/data/keyspacename/tablename-2c5325e042e911e8a07e9db72d27cf67/md-292348-big-Data.db'), BigTableReader(path='/cassandra-data/data/keyspacename/tablename-2c5325e042e911e8a07e9db72d27cf67/md-292358-big-Data.db')] which is not a problem per se,unless it happens frequently, in which case it must be reported. Will retry later.
> {noformat}
> This final line then starts repeating about once per minute:
> {noformat}
> WARN [CompactionExecutor:1610] 2019-07-18 01:13:18,898 LeveledCompactionStrategy.java:144 - Could not acquire references for compacting SSTables [BigTableReader(path='/cassandra-data/data/keyspacename/tablename-2c5325e042e911e8a07e9db72d27cf67/md-292348-big-Data.db'), BigTableReader(path='/cassandra-data/data/keyspacename/tablename-2c5325e042e911e8a07e9db72d27cf67/md-292358-big-Data.db')] which is not a problem per se,unless it happens frequently, in which case it must be reported. Will retry later.
> WARN [CompactionExecutor:1611] 2019-07-18 01:14:18,899 LeveledCompactionStrategy.java:144 - Could not acquire references for compacting SSTables [BigTableReader(path='/cassandra-data/data/keyspacename/tablename-2c5325e042e911e8a07e9db72d27cf67/md-292348-big-Data.db'), BigTableReader(path='/cassandra-data/data/keyspacename/tablename-2c5325e042e911e8a07e9db72d27cf67/md-292358-big-Data.db')] which is not a problem per se,unless it happens frequently, in which case it must be reported. Will retry later.
> WARN [CompactionExecutor:1622] 2019-07-18 01:15:18,899 LeveledCompactionStrategy.java:144 - Could not acquire references for compacting SSTables [BigTableReader(path='/cassandra-data/data/keyspacename/tablename-2c5325e042e911e8a07e9db72d27cf67/md-292348-big-Data.db'), BigTableReader(path='/cassandra-data/data/keyspacename/tablename-2c5325e042e911e8a07e9db72d27cf67/md-292358-big-Data.db')] which is not a problem per se,unless it happens frequently, in which case it must be reported. Will retry later.
> WARN [CompactionExecutor:1436] 2019-07-18 01:16:15,073 LeveledCompactionStrategy.java:144 - Could not acquire references for compacting SSTables [BigTableReader(path='/cassandra-data/data/keyspacename/tablename-2c5325e042e911e8a07e9db72d27cf67/md-292348-big-Data.db'), BigTableReader(path='/cassandra-data/data/keyspacename/tablename-2c5325e042e911e8a07e9db72d27cf67/md-292358-big-Data.db')] which is not a problem per se,unless it happens frequently, in which case it must be reported. Will retry later.
> WARN [CompactionExecutor:1618] 2019-07-18 01:16:18,899 LeveledCompactionStrategy.java:144 - Could not acquire references for compacting SSTables [BigTableReader(path='/cassandra-data/data/keyspacename/tablename-2c5325e042e911e8a07e9db72d27cf67/md-292348-big-Data.db'), BigTableReader(path='/cassandra-data/data/keyspacename/tablename-2c5325e042e911e8a07e9db72d27cf67/md-292358-big-Data.db')] which is not a problem per se,unless it happens frequently, in which case it must be reported. Will retry later.
> WARN [CompactionExecutor:1611] 2019-07-18 01:17:18,900 LeveledCompactionStrategy.java:144 - Could not acquire references for compacting SSTables [BigTableReader(path='/cassandra-data/data/keyspacename/tablename-2c5325e042e911e8a07e9db72d27cf67/md-292348-big-Data.db'), BigTableReader(path='/cassandra-data/data/keyspacename/tablename-2c5325e042e911e8a07e9db72d27cf67/md-292358-big-Data.db')] which is not a problem per se,unless it happens frequently, in which case it must be reported. Will retry later.
> WARN [CompactionExecutor:1606] 2019-07-18 01:18:18,900 LeveledCompactionStrategy.java:144 - Could not acquire references for compacting SSTables [BigTableReader(path='/cassandra-data/data/keyspacename/tablename-2c5325e042e911e8a07e9db72d27cf67/md-292348-big-Data.db'), BigTableReader(path='/cassandra-data/data/keyspacename/tablename-2c5325e042e911e8a07e9db72d27cf67/md-292358-big-Data.db')] which is not a problem per se,unless it happens frequently, in which case it must be reported. Will retry later.
> WARN [CompactionExecutor:1630] 2019-07-18 01:19:18,902 LeveledCompactionStrategy.java:144 - Could not acquire references for compacting SSTables [BigTableReader(path='/cassandra-data/data/keyspacename/tablename-2c5325e042e911e8a07e9db72d27cf67/md-292348-big-Data.db'), BigTableReader(path='/cassandra-data/data/keyspacename/tablename-2c5325e042e911e8a07e9db72d27cf67/md-292358-big-Data.db')] which is not a problem per se,unless it happens frequently, in which case it must be reported. Will retry later.
> WARN [CompactionExecutor:1627] 2019-07-18 01:20:18,904 LeveledCompactionStrategy.java:144 - Could not acquire references for compacting SSTables [BigTableReader(path='/cassandra-data/data/keyspacename/tablename-2c5325e042e911e8a07e9db72d27cf67/md-292348-big-Data.db'), BigTableReader(path='/cassandra-data/data/keyspacename/tablename-2c5325e042e911e8a07e9db72d27cf67/md-292358-big-Data.db')] which is not a problem per se,unless it happens frequently, in which case it must be reported. Will retry later.
> WARN [CompactionExecutor:1638] 2019-07-18 01:21:18,904 LeveledCompactionStrategy.java:144 - Could not acquire references for compacting SSTables [BigTableReader(path='/cassandra-data/data/keyspacename/tablename-2c5325e042e911e8a07e9db72d27cf67/md-292348-big-Data.db'), BigTableReader(path='/cassandra-data/data/keyspacename/tablename-2c5325e042e911e8a07e9db72d27cf67/md-292358-big-Data.db')] which is not a problem per se,unless it happens frequently, in which case it must be reported. Will retry later.
> WARN [CompactionExecutor:1631] 2019-07-18 01:22:18,905 LeveledCompactionStrategy.java:144 - Could not acquire references for compacting SSTables [BigTableReader(path='/cassandra-data/data/keyspacename/tablename-2c5325e042e911e8a07e9db72d27cf67/md-292348-big-Data.db'), BigTableReader(path='/cassandra-data/data/keyspacename/tablename-2c5325e042e911e8a07e9db72d27cf67/md-292358-big-Data.db')] which is not a problem per se,unless it happens frequently, in which case it must be reported. Will retry later.
> WARN [CompactionExecutor:1636] 2019-07-18 01:22:58,220 LeveledCompactionStrategy.java:144 - Could not acquire references for compacting SSTables [BigTableReader(path='/cassandra-data/data/keyspacename/tablename-2c5325e042e911e8a07e9db72d27cf67/md-292363-big-Data.db'), BigTableReader(path='/cassandra-data/data/keyspacename/tablename-2c5325e042e911e8a07e9db72d27cf67/md-292348-big-Data.db'), BigTableReader(path='/cassandra-data/data/keyspacename/tablename-2c5325e042e911e8a07e9db72d27cf67/md-292358-big-Data.db'), BigTableReader(path='/cassandra-data/data/keyspacename/tablename-2c5325e042e911e8a07e9db72d27cf67/md-292342-big-Data.db'), BigTableReader(path='/cassandra-data/data/keyspacename/tablename-2c5325e042e911e8a07e9db72d27cf67/md-292344-big-Data.db'), BigTableReader(path='/cassandra-data/data/keyspacename/tablename-2c5325e042e911e8a07e9db72d27cf67/md-292343-big-Data.db'), BigTableReader(path='/cassandra-data/data/keyspacename/tablename-2c5325e042e911e8a07e9db72d27cf67/md-292340-big-Data.db'), BigTableReader(path='/cassandra-data/data/keyspacename/tablename-2c5325e042e911e8a07e9db72d27cf67/md-292338-big-Data.db'), BigTableReader(path='/cassandra-data/data/keyspacename/tablename-2c5325e042e911e8a07e9db72d27cf67/md-292336-big-Data.db'), BigTableReader(path='/cassandra-data/data/keyspacename/tablename-2c5325e042e911e8a07e9db72d27cf67/md-292335-big-Data.db'), BigTableReader(path='/cassandra-data/data/keyspacename/tablename-2c5325e042e911e8a07e9db72d27cf67/md-292337-big-Data.db'), BigTableReader(path='/cassandra-data/data/keyspacename/tablename-2c5325e042e911e8a07e9db72d27cf67/md-292346-big-Data.db'), BigTableReader(path='/cassandra-data/data/keyspacename/tablename-2c5325e042e911e8a07e9db72d27cf67/md-292349-big-Data.db')] which is not a problem per se,unless it happens frequently, in which case it must be reported. Will retry later.
> WARN [CompactionExecutor:1625] 2019-07-18 01:23:18,905 LeveledCompactionStrategy.java:144 - Could not acquire references for compacting SSTables [BigTableReader(path='/cassandra-data/data/keyspacename/tablename-2c5325e042e911e8a07e9db72d27cf67/md-292363-big-Data.db'), BigTableReader(path='/cassandra-data/data/keyspacename/tablename-2c5325e042e911e8a07e9db72d27cf67/md-292348-big-Data.db'), BigTableReader(path='/cassandra-data/data/keyspacename/tablename-2c5325e042e911e8a07e9db72d27cf67/md-292358-big-Data.db'), BigTableReader(path='/cassandra-data/data/keyspacename/tablename-2c5325e042e911e8a07e9db72d27cf67/md-292342-big-Data.db'), BigTableReader(path='/cassandra-data/data/keyspacename/tablename-2c5325e042e911e8a07e9db72d27cf67/md-292344-big-Data.db'), BigTableReader(path='/cassandra-data/data/keyspacename/tablename-2c5325e042e911e8a07e9db72d27cf67/md-292343-big-Data.db'), BigTableReader(path='/cassandra-data/data/keyspacename/tablename-2c5325e042e911e8a07e9db72d27cf67/md-292340-big-Data.db'), BigTableReader(path='/cassandra-data/data/keyspacename/tablename-2c5325e042e911e8a07e9db72d27cf67/md-292338-big-Data.db'), BigTableReader(path='/cassandra-data/data/keyspacename/tablename-2c5325e042e911e8a07e9db72d27cf67/md-292336-big-Data.db'), BigTableReader(path='/cassandra-data/data/keyspacename/tablename-2c5325e042e911e8a07e9db72d27cf67/md-292335-big-Data.db'), BigTableReader(path='/cassandra-data/data/keyspacename/tablename-2c5325e042e911e8a07e9db72d27cf67/md-292337-big-Data.db'), BigTableReader(path='/cassandra-data/data/keyspacename/tablename-2c5325e042e911e8a07e9db72d27cf67/md-292346-big-Data.db'), BigTableReader(path='/cassandra-data/data/keyspacename/tablename-2c5325e042e911e8a07e9db72d27cf67/md-292349-big-Data.db')] which is not a problem per se,unless it happens frequently, in which case it must be reported. Will retry later.{noformat}
> It will keep going like this for days, until restarted, but compaction won't run until then, so sstables pile up.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org