You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Caleb Rackliffe <ca...@steelhouse.com> on 2012/01/08 03:51:57 UTC
Lots and Lots of CompactionReducer Threads
Hi Everybody,
JConsole tells me I've got CompactionReducer threads stacking up, consuming memory, and never going away. Eventually, my Java process fails because it can't allocate any more native threads. Here's my setup…
Cassandra 1.0.5 on CentOS 6.0
4 GB of RAM
50 GB SSD HD
Memtable flush threshold = 128 MB
compaction throughput limit = 16 MB/sec
Multithreaded compaction = true
It may very well be that I'm doing something strange here, but it seems like those compaction threads should go away eventually. I'm hoping the combination of a low Memtable flush threshold, low compaction T/P limit, and heavy write load doesn't mean those threads are hanging around because they're actually not done doing their compaction tasks.
Thanks,
Caleb Rackliffe | Software Developer
M 949.981.0159 | caleb@steelhouse.com
Re: Lots and Lots of CompactionReducer Threads
Posted by aaron morton <aa...@thelastpickle.com>.
Thanks, good work.
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 9/01/2012, at 1:25 PM, Caleb Rackliffe wrote:
> After some searching, I think I may have found something in the code itself, and so I've filed a big report - https://issues.apache.org/jira/browse/CASSANDRA-3711
>
> Caleb Rackliffe | Software Developer
> M 949.981.0159 | caleb@steelhouse.com
>
>
> From: Caleb Rackliffe <ca...@steelhouse.com>
> Reply-To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
> Date: Sun, 8 Jan 2012 17:48:59 -0500
> To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
> Cc: "aaron@thelastpickle.com" <aa...@thelastpickle.com>
> Subject: Re: Lots and Lots of CompactionReducer Threads
>
> With the exception of a few little warnings on start-up about the Memtable live ratio, there is nothing at WARN or above in the logs. Just before the JVM terminates, there are about 10,000 threads in Reducer executor pools that look like this in JConsole …
>
>
> Name: CompactionReducer:1
> State: TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@72938aea
> Total blocked: 0 Total waited: 1
>
> Stack trace:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> java.lang.Thread.run(Thread.java:722)
>
>
> The results from tpstats don't look too interesting…
>
> Pool Name Active Pending Completed Blocked All time blocked
> ReadStage 0 0 3455159 0 0
> RequestResponseStage 0 0 10133276 0 0
> MutationStage 0 0 5898833 0 0
> ReadRepairStage 0 0 2078449 0 0
> ReplicateOnWriteStage 0 0 0 0 0
> GossipStage 0 0 236388 0 0
> AntiEntropyStage 0 0 0 0 0
> MigrationStage 0 0 0 0 0
> MemtablePostFlusher 0 0 231 0 0
> StreamStage 0 0 0 0 0
> FlushWriter 0 0 231 0 0
> MiscStage 0 0 0 0 0
> InternalResponseStage 0 0 0 0 0
> HintedHandoff 0 0 35 0 0
>
> Message type Dropped
> RANGE_SLICE 0
> READ_REPAIR 0
> BINARY 0
> READ 0
> MUTATION 0
> REQUEST_RESPONSE 0
>
> The results from info seem unremarkable as well…
>
> Token : 153127065000000000000000000000000000000
> Gossip active : true
> Load : 5.6 GB
> Generation No : 1325995515
> Uptime (seconds) : 67199
> Heap Memory (MB) : 970.32 / 1968.00
> Data Center : datacenter1
> Rack : rack1
> Exceptions : 0
>
> I'm using LeveledCompactionStrategy with no throttling, and I'm not changing the default on the number of concurrent compactors.
>
> What is interesting to me here is that Cassandra creates an executor for every single compaction in ParallelCompactionIterable. Why couldn't we just create a pool with Runtime.availableProcessors() Threads and be done with it?
>
> Let me know if I left any info out.
>
> Thanks!
>
> Caleb Rackliffe | Software Developer
> M 949.981.0159 | caleb@steelhouse.com
>
>
> From: aaron morton <aa...@thelastpickle.com>
> Reply-To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
> Date: Sun, 8 Jan 2012 16:51:50 -0500
> To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
> Subject: Re: Lots and Lots of CompactionReducer Threads
>
> How many threads ? Any errors in the server logs ?
>
> What does noodtool tpstats and nodetool compactionstats say ?
>
> Did you change compaction_strategy for the CF's ?
>
> By default cassandra will use as many compaction threads as you have cores, see concurrent_compactors in cassandra.yaml
>
> Have you set the JVM heap settings ? What does nodetool info show ?
>
> Hope that helps.
>
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 8/01/2012, at 3:51 PM, Caleb Rackliffe wrote:
>
>> Hi Everybody,
>>
>> JConsole tells me I've got CompactionReducer threads stacking up, consuming memory, and never going away. Eventually, my Java process fails because it can't allocate any more native threads. Here's my setup…
>>
>> Cassandra 1.0.5 on CentOS 6.0
>> 4 GB of RAM
>> 50 GB SSD HD
>> Memtable flush threshold = 128 MB
>> compaction throughput limit = 16 MB/sec
>> Multithreaded compaction = true
>>
>> It may very well be that I'm doing something strange here, but it seems like those compaction threads should go away eventually. I'm hoping the combination of a low Memtable flush threshold, low compaction T/P limit, and heavy write load doesn't mean those threads are hanging around because they're actually not done doing their compaction tasks.
>>
>> Thanks,
>>
>> Caleb Rackliffe | Software Developer
>> M 949.981.0159 | caleb@steelhouse.com
>>
>
Re: Lots and Lots of CompactionReducer Threads
Posted by Caleb Rackliffe <ca...@steelhouse.com>.
After some searching, I think I may have found something in the code itself, and so I've filed a big report - https://issues.apache.org/jira/browse/CASSANDRA-3711
Caleb Rackliffe | Software Developer
M 949.981.0159 | caleb@steelhouse.com
From: Caleb Rackliffe <ca...@steelhouse.com>>
Reply-To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Date: Sun, 8 Jan 2012 17:48:59 -0500
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Cc: "aaron@thelastpickle.com<ma...@thelastpickle.com>" <aa...@thelastpickle.com>>
Subject: Re: Lots and Lots of CompactionReducer Threads
With the exception of a few little warnings on start-up about the Memtable live ratio, there is nothing at WARN or above in the logs. Just before the JVM terminates, there are about 10,000 threads in Reducer executor pools that look like this in JConsole …
Name: CompactionReducer:1
State: TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@72938aea
Total blocked: 0 Total waited: 1
Stack trace:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
java.lang.Thread.run(Thread.java:722)
The results from tpstats don't look too interesting…
Pool Name Active Pending Completed Blocked All time blocked
ReadStage 0 0 3455159 0 0
RequestResponseStage 0 0 10133276 0 0
MutationStage 0 0 5898833 0 0
ReadRepairStage 0 0 2078449 0 0
ReplicateOnWriteStage 0 0 0 0 0
GossipStage 0 0 236388 0 0
AntiEntropyStage 0 0 0 0 0
MigrationStage 0 0 0 0 0
MemtablePostFlusher 0 0 231 0 0
StreamStage 0 0 0 0 0
FlushWriter 0 0 231 0 0
MiscStage 0 0 0 0 0
InternalResponseStage 0 0 0 0 0
HintedHandoff 0 0 35 0 0
Message type Dropped
RANGE_SLICE 0
READ_REPAIR 0
BINARY 0
READ 0
MUTATION 0
REQUEST_RESPONSE 0
The results from info seem unremarkable as well…
Token : 153127065000000000000000000000000000000
Gossip active : true
Load : 5.6 GB
Generation No : 1325995515
Uptime (seconds) : 67199
Heap Memory (MB) : 970.32 / 1968.00
Data Center : datacenter1
Rack : rack1
Exceptions : 0
I'm using LeveledCompactionStrategy with no throttling, and I'm not changing the default on the number of concurrent compactors.
What is interesting to me here is that Cassandra creates an executor for every single compaction in ParallelCompactionIterable. Why couldn't we just create a pool with Runtime.availableProcessors() Threads and be done with it?
Let me know if I left any info out.
Thanks!
Caleb Rackliffe | Software Developer
M 949.981.0159 | caleb@steelhouse.com<ma...@steelhouse.com>
From: aaron morton <aa...@thelastpickle.com>>
Reply-To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Date: Sun, 8 Jan 2012 16:51:50 -0500
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Subject: Re: Lots and Lots of CompactionReducer Threads
How many threads ? Any errors in the server logs ?
What does noodtool tpstats and nodetool compactionstats say ?
Did you change compaction_strategy for the CF's ?
By default cassandra will use as many compaction threads as you have cores, see concurrent_compactors in cassandra.yaml
Have you set the JVM heap settings ? What does nodetool info show ?
Hope that helps.
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 8/01/2012, at 3:51 PM, Caleb Rackliffe wrote:
Hi Everybody,
JConsole tells me I've got CompactionReducer threads stacking up, consuming memory, and never going away. Eventually, my Java process fails because it can't allocate any more native threads. Here's my setup…
Cassandra 1.0.5 on CentOS 6.0
4 GB of RAM
50 GB SSD HD
Memtable flush threshold = 128 MB
compaction throughput limit = 16 MB/sec
Multithreaded compaction = true
It may very well be that I'm doing something strange here, but it seems like those compaction threads should go away eventually. I'm hoping the combination of a low Memtable flush threshold, low compaction T/P limit, and heavy write load doesn't mean those threads are hanging around because they're actually not done doing their compaction tasks.
Thanks,
Caleb Rackliffe | Software Developer
M 949.981.0159 | caleb@steelhouse.com<ma...@steelhouse.com>
Re: Lots and Lots of CompactionReducer Threads
Posted by Caleb Rackliffe <ca...@steelhouse.com>.
With the exception of a few little warnings on start-up about the Memtable live ratio, there is nothing at WARN or above in the logs. Just before the JVM terminates, there are about 10,000 threads in Reducer executor pools that look like this in JConsole …
Name: CompactionReducer:1
State: TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@72938aea
Total blocked: 0 Total waited: 1
Stack trace:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
java.lang.Thread.run(Thread.java:722)
The results from tpstats don't look too interesting…
Pool Name Active Pending Completed Blocked All time blocked
ReadStage 0 0 3455159 0 0
RequestResponseStage 0 0 10133276 0 0
MutationStage 0 0 5898833 0 0
ReadRepairStage 0 0 2078449 0 0
ReplicateOnWriteStage 0 0 0 0 0
GossipStage 0 0 236388 0 0
AntiEntropyStage 0 0 0 0 0
MigrationStage 0 0 0 0 0
MemtablePostFlusher 0 0 231 0 0
StreamStage 0 0 0 0 0
FlushWriter 0 0 231 0 0
MiscStage 0 0 0 0 0
InternalResponseStage 0 0 0 0 0
HintedHandoff 0 0 35 0 0
Message type Dropped
RANGE_SLICE 0
READ_REPAIR 0
BINARY 0
READ 0
MUTATION 0
REQUEST_RESPONSE 0
The results from info seem unremarkable as well…
Token : 153127065000000000000000000000000000000
Gossip active : true
Load : 5.6 GB
Generation No : 1325995515
Uptime (seconds) : 67199
Heap Memory (MB) : 970.32 / 1968.00
Data Center : datacenter1
Rack : rack1
Exceptions : 0
I'm using LeveledCompactionStrategy with no throttling, and I'm not changing the default on the number of concurrent compactors.
What is interesting to me here is that Cassandra creates an executor for every single compaction in ParallelCompactionIterable. Why couldn't we just create a pool with Runtime.availableProcessors() Threads and be done with it?
Let me know if I left any info out.
Thanks!
Caleb Rackliffe | Software Developer
M 949.981.0159 | caleb@steelhouse.com
From: aaron morton <aa...@thelastpickle.com>>
Reply-To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Date: Sun, 8 Jan 2012 16:51:50 -0500
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Subject: Re: Lots and Lots of CompactionReducer Threads
How many threads ? Any errors in the server logs ?
What does noodtool tpstats and nodetool compactionstats say ?
Did you change compaction_strategy for the CF's ?
By default cassandra will use as many compaction threads as you have cores, see concurrent_compactors in cassandra.yaml
Have you set the JVM heap settings ? What does nodetool info show ?
Hope that helps.
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 8/01/2012, at 3:51 PM, Caleb Rackliffe wrote:
Hi Everybody,
JConsole tells me I've got CompactionReducer threads stacking up, consuming memory, and never going away. Eventually, my Java process fails because it can't allocate any more native threads. Here's my setup…
Cassandra 1.0.5 on CentOS 6.0
4 GB of RAM
50 GB SSD HD
Memtable flush threshold = 128 MB
compaction throughput limit = 16 MB/sec
Multithreaded compaction = true
It may very well be that I'm doing something strange here, but it seems like those compaction threads should go away eventually. I'm hoping the combination of a low Memtable flush threshold, low compaction T/P limit, and heavy write load doesn't mean those threads are hanging around because they're actually not done doing their compaction tasks.
Thanks,
Caleb Rackliffe | Software Developer
M 949.981.0159 | caleb@steelhouse.com<ma...@steelhouse.com>
Re: Lots and Lots of CompactionReducer Threads
Posted by aaron morton <aa...@thelastpickle.com>.
How many threads ? Any errors in the server logs ?
What does noodtool tpstats and nodetool compactionstats say ?
Did you change compaction_strategy for the CF's ?
By default cassandra will use as many compaction threads as you have cores, see concurrent_compactors in cassandra.yaml
Have you set the JVM heap settings ? What does nodetool info show ?
Hope that helps.
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 8/01/2012, at 3:51 PM, Caleb Rackliffe wrote:
> Hi Everybody,
>
> JConsole tells me I've got CompactionReducer threads stacking up, consuming memory, and never going away. Eventually, my Java process fails because it can't allocate any more native threads. Here's my setup…
>
> Cassandra 1.0.5 on CentOS 6.0
> 4 GB of RAM
> 50 GB SSD HD
> Memtable flush threshold = 128 MB
> compaction throughput limit = 16 MB/sec
> Multithreaded compaction = true
>
> It may very well be that I'm doing something strange here, but it seems like those compaction threads should go away eventually. I'm hoping the combination of a low Memtable flush threshold, low compaction T/P limit, and heavy write load doesn't mean those threads are hanging around because they're actually not done doing their compaction tasks.
>
> Thanks,
>
> Caleb Rackliffe | Software Developer
> M 949.981.0159 | caleb@steelhouse.com
>