You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "wu taiyin (JIRA)" <ji...@apache.org> on 2019/01/12 06:25:00 UTC
[jira] [Comment Edited] (CASSANDRA-14908) Deadlock occurs when
executing a file selection in levelcompact
[ https://issues.apache.org/jira/browse/CASSANDRA-14908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16704580#comment-16704580 ]
wu taiyin edited comment on CASSANDRA-14908 at 1/12/19 6:24 AM:
----------------------------------------------------------------
Thank you for you reply. and the environment as follows:
CPU：48C memory：385G
disk and data size per disk:
/dev/sda3 754G 9.2G 706G 2% /opt/sds
/dev/sdb 1.5T 223G 1.3T 15% /disk/data1
/dev/sdc 1.5T 223G 1.3T 15% /disk/data2
/dev/sdd 1.5T 213G 1.3T 15% /disk/data3
/dev/sde 1.5T 216G 1.3T 15% /disk/data4
/dev/sdf 1.5T 214G 1.3T 15% /disk/data5
Cassandra.yaml :
data_file_directories:
- /disk/data1/cassandra
- /disk/data2/cassandra
- /disk/data3/cassandra
- /disk/data4/cassandra
- /disk/data5/cassandra
commitlog_directory: /opt/sds/logs/cassandra/commitlog
./nodetool cfstats app99
Keyspace: app99
Read Count: 2789855752
Read Latency: 0.6304968936945955 ms.
Write Count: 1490782843
Write Latency: 0.03916580592616869 ms.
Pending Flushes: 0
Table: dn
SSTable count: 1
SSTables in each level: [1, 0, 0, 0, 0, 0, 0, 0, 0]
Space used (live): 6092
Space used (total): 6092
Space used by snapshots (total): 0
Off heap memory used (total): 127
SSTable Compression Ratio: 0.14210985178727115
Number of keys (estimate): 11
Memtable cell count: 483
Memtable data size: 2352
Memtable off heap memory used: 0
Memtable switch count: 2
Local read count: 10133694
Local read latency: 0.136 ms
Local write count: 270
Local write latency: 0.135 ms
Pending flushes: 0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 96
Bloom filter off heap memory used: 88
Index summary off heap memory used: 31
Compression metadata off heap memory used: 8
Compacted partition minimum bytes: 125
Compacted partition maximum bytes: 1916
Compacted partition mean bytes: 974
Average live cells per slice (last five minutes): 1.0
Maximum live cells per slice (last five minutes): 1.0
Average tombstones per slice (last five minutes): 0.0
Maximum tombstones per slice (last five minutes): 0.0
Table: file
SSTable count: 4295
SSTables in each level: [0, 10, 100, 1016/1000, 3169, 0, 0, 0, 0]
Space used (live): 1166100476265
Space used (total): 1169018887208
Space used by snapshots (total): 0
Off heap memory used (total): 9184778389
SSTable Compression Ratio: 0.38421502411531816
Number of keys (estimate): 8849691024
Memtable cell count: 8238732
Memtable data size: 301883508
Memtable off heap memory used: 0
Memtable switch count: 521
Local read count: 2779722909
Local read latency: 0.633 ms
Local write count: 1490783015
Local write latency: 0.040 ms
Pending flushes: 0
Bloom filter false positives: 5092
Bloom filter false ratio: 0.00000
Bloom filter space used: 6387737864
Bloom filter off heap memory used: 6387703504
Index summary off heap memory used: 2575905581
Compression metadata off heap memory used: 221169304
Compacted partition minimum bytes: 43
Compacted partition maximum bytes: 88148
Compacted partition mean bytes: 215
Average live cells per slice (last five minutes): 0.4247412120152055
Maximum live cells per slice (last five minutes): 1.0
Average tombstones per slice (last five minutes): 0.0
Maximum tombstones per slice (last five minutes): 0.0
was (Author: dockerwu):
Thank you for you reply. and the environment as follows:
CPU：48C memory：385G
disk and data size per disk:
/dev/sda3 754G 9.2G 706G 2% /opt/huawei
/dev/sdb 1.5T 223G 1.3T 15% /disk/data1
/dev/sdc 1.5T 223G 1.3T 15% /disk/data2
/dev/sdd 1.5T 213G 1.3T 15% /disk/data3
/dev/sde 1.5T 216G 1.3T 15% /disk/data4
/dev/sdf 1.5T 214G 1.3T 15% /disk/data5
Cassandra.yaml :
data_file_directories:
- /disk/data1/cassandra
- /disk/data2/cassandra
- /disk/data3/cassandra
- /disk/data4/cassandra
- /disk/data5/cassandra
commitlog_directory: /opt/huawei/logs/cassandra/commitlog
./nodetool cfstats app99
Keyspace: app99
Read Count: 2789855752
Read Latency: 0.6304968936945955 ms.
Write Count: 1490782843
Write Latency: 0.03916580592616869 ms.
Pending Flushes: 0
Table: dn
SSTable count: 1
SSTables in each level: [1, 0, 0, 0, 0, 0, 0, 0, 0]
Space used (live): 6092
Space used (total): 6092
Space used by snapshots (total): 0
Off heap memory used (total): 127
SSTable Compression Ratio: 0.14210985178727115
Number of keys (estimate): 11
Memtable cell count: 483
Memtable data size: 2352
Memtable off heap memory used: 0
Memtable switch count: 2
Local read count: 10133694
Local read latency: 0.136 ms
Local write count: 270
Local write latency: 0.135 ms
Pending flushes: 0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 96
Bloom filter off heap memory used: 88
Index summary off heap memory used: 31
Compression metadata off heap memory used: 8
Compacted partition minimum bytes: 125
Compacted partition maximum bytes: 1916
Compacted partition mean bytes: 974
Average live cells per slice (last five minutes): 1.0
Maximum live cells per slice (last five minutes): 1.0
Average tombstones per slice (last five minutes): 0.0
Maximum tombstones per slice (last five minutes): 0.0
Table: file
SSTable count: 4295
SSTables in each level: [0, 10, 100, 1016/1000, 3169, 0, 0, 0, 0]
Space used (live): 1166100476265
Space used (total): 1169018887208
Space used by snapshots (total): 0
Off heap memory used (total): 9184778389
SSTable Compression Ratio: 0.38421502411531816
Number of keys (estimate): 8849691024
Memtable cell count: 8238732
Memtable data size: 301883508
Memtable off heap memory used: 0
Memtable switch count: 521
Local read count: 2779722909
Local read latency: 0.633 ms
Local write count: 1490783015
Local write latency: 0.040 ms
Pending flushes: 0
Bloom filter false positives: 5092
Bloom filter false ratio: 0.00000
Bloom filter space used: 6387737864
Bloom filter off heap memory used: 6387703504
Index summary off heap memory used: 2575905581
Compression metadata off heap memory used: 221169304
Compacted partition minimum bytes: 43
Compacted partition maximum bytes: 88148
Compacted partition mean bytes: 215
Average live cells per slice (last five minutes): 0.4247412120152055
Maximum live cells per slice (last five minutes): 1.0
Average tombstones per slice (last five minutes): 0.0
Maximum tombstones per slice (last five minutes): 0.0
> Deadlock occurs when executing a file selection in levelcompact
> ---------------------------------------------------------------
>
> Key: CASSANDRA-14908
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14908
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Compaction
> Environment: version : cassandra 2.1.15
> jdk: 8
> os:suse
> Reporter: wu taiyin
> Priority: Major
> Attachments: stack.zip
>
>
> detailed exception stack as follows:
> "CompactionExecutor:33616" #142049 daemon prio=1 os_prio=4 tid=0x00007f73244cc000 nid=0x1919a waiting for monitor entry [0x00007fa94e13b000]
> java.lang.Thread.State: {color:#FF0000}BLOCKED{color} (on object monitor)
> at org.apache.cassandra.db.compaction.WrappingCompactionStrategy.handleNotification(WrappingCompactionStrategy.java:265)
> - waiting to lock <0x00007faa776209b0> (a org.apache.cassandra.db.compaction.WrappingCompactionStrategy)
> at org.apache.cassandra.db.DataTracker.notifySSTablesChanged(DataTracker.java:517)
> at org.apache.cassandra.db.DataTracker.replaceReaders(DataTracker.java:408)
> at org.apache.cassandra.db.DataTracker.replaceWithNewInstances(DataTracker.java:305)
> at org.apache.cassandra.io.sstable.SSTableRewriter.moveStarts(SSTableRewriter.java:337)
> at org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:187)
> at org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:126)
> at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:197)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:73)
> at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
> at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:264)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> "CompactionExecutor:33615" #142048 daemon prio=1 os_prio=4 tid=0x00007f7324451800 nid=0x19199 runnable [0x00007fa7a096f000]
> java.lang.Thread.State: RUNNABLE
> at java.util.HashMap.hash(HashMap.java:338)
> at java.util.HashMap.put(HashMap.java:611)
> at java.util.HashSet.add(HashSet.java:219)
> at java.util.AbstractCollection.addAll(AbstractCollection.java:344)
> at java.util.HashSet.<init>(HashSet.java:119)
> at com.google.common.collect.Sets.newHashSet(Sets.java:218)
> at {color:#FF0000}org.apache.cassandra.db.compaction.LeveledManifest.getCompactionCandidates(LeveledManifest.java:307){color}
> {color:#FF0000} - locked <0x00007faa77620e70> (a org.apache.cassandra.db.compaction.LeveledManifest){color}
> {color:#FF0000} at{color} org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getMaximalTask(LeveledCompactionStrategy.java:101)
> at org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getNextBackgroundTask(LeveledCompactionStrategy.java:90)
> - locked <0x00007faa77620e30> (a org.apache.cassandra.db.compaction.LeveledCompactionStrategy)
> at org.apache.cassandra.db.compaction.WrappingCompactionStrategy.getNextBackgroundTask(WrappingCompactionStrategy.java:84)
> - locked <0x00007faa776209b0> (a org.apache.cassandra.db.compaction.WrappingCompactionStrategy)
> at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:258)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> "MemtableFlushWriter:22715" #142123 daemon prio=5 os_prio=0 tid=0x00007f5a3d4ba000 nid=0x24063 waiting for monitor entry [0x00007fa8337bd000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.cassandra.db.compaction.WrappingCompactionStrategy.handleNotification(WrappingCompactionStrategy.java:265)
> - waiting to lock <0x00007faa776209b0> (a org.apache.cassandra.db.compaction.WrappingCompactionStrategy)
> at org.apache.cassandra.db.DataTracker.notifyAdded(DataTracker.java:531)
> at org.apache.cassandra.db.DataTracker.replaceFlushed(DataTracker.java:179)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.replaceFlushed(AbstractCompactionStrategy.java:234)
> at org.apache.cassandra.db.ColumnFamilyStore.replaceFlushed(ColumnFamilyStore.java:1521)
> at org.apache.cassandra.db.Memtable$FlushRunnable.runMayThrow(Memtable.java:336)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
> at org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1127)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> "MemtableFlushWriter:22711" #142116 daemon prio=5 os_prio=0 tid=0x00007f5a3d338000 nid=0x21ccf waiting for monitor entry [0x00007fa94e084000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.cassandra.db.compaction.WrappingCompactionStrategy.handleNotification(WrappingCompactionStrategy.java:265)
> - waiting to lock <0x00007faa776209b0> (a org.apache.cassandra.db.compaction.WrappingCompactionStrategy)
> at org.apache.cassandra.db.DataTracker.notifyAdded(DataTracker.java:531)
> at org.apache.cassandra.db.DataTracker.replaceFlushed(DataTracker.java:179)
> at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.replaceFlushed(AbstractCompactionStrategy.java:234)
> at org.apache.cassandra.db.ColumnFamilyStore.replaceFlushed(ColumnFamilyStore.java:1521)
> at org.apache.cassandra.db.Memtable$FlushRunnable.runMayThrow(Memtable.java:336)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
> at org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1127)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org