You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Caleb William Rackliffe (Created) (JIRA)" <ji...@apache.org> on 2012/01/09 00:48:39 UTC
[jira] [Created] (CASSANDRA-3711) Unsustainable Thread Accumulation
in ParallelCompactionIterable.Reducer ThreadPoolExecutor
Unsustainable Thread Accumulation in ParallelCompactionIterable.Reducer ThreadPoolExecutor
------------------------------------------------------------------------------------------
Key: CASSANDRA-3711
URL: https://issues.apache.org/jira/browse/CASSANDRA-3711
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.0.5
Environment: Linux version 2.6.32-71.29.1.el6.x86_64 (mockbuild@c6b5.bsys.dev.centos.org) (gcc version 4.4.4 20100726 (Red Hat 4.4.4-13) (GCC) ) #1 SMP Mon Jun 27 19:49:27 BST 2011
Reporter: Caleb William Rackliffe
With multithreaded compaction enabled, it looks like Reducer creates a new thread pool for every compaction. These pools seem to just sit around - i.e. "executor.shutdown()" never gets called and the Threads live forever waiting for tasks that will never come. For instance...
Name: CompactionReducer:1
State: TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@72938aea
Total blocked: 0 Total waited: 1
Stack trace:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
java.lang.Thread.run(Thread.java:722)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3711) Unsustainable Thread Accumulation
in ParallelCompactionIterable.Reducer ThreadPoolExecutor
Posted by "Jonathan Ellis (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-3711:
--------------------------------------
Reviewer: slebresne
Priority: Minor (was: Major)
Affects Version/s: (was: 1.0.5)
1.0.0
Assignee: Jonathan Ellis
Labels: compaction memory_leak threading threads (was: memory_leak threading threads)
> Unsustainable Thread Accumulation in ParallelCompactionIterable.Reducer ThreadPoolExecutor
> ------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-3711
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3711
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: - Linux version 2.6.32-71.29.1.el6.x86_64 (mockbuild@c6b5.bsys.dev.centos.org) (gcc version 4.4.4 20100726 (Red Hat 4.4.4-13) (GCC) ) #1 SMP Mon Jun 27 19:49:27 BST 2011
> - java version "1.7.0_01" / Java(TM) SE Runtime Environment (build 1.7.0_01-b08) / Java HotSpot(TM) 64-Bit Server VM (build 21.1-b02, mixed mode)
> Reporter: Caleb William Rackliffe
> Assignee: Jonathan Ellis
> Priority: Minor
> Labels: compaction, memory_leak, threading, threads
> Fix For: 1.0.7
>
> Attachments: 3711.txt
>
>
> With multithreaded compaction enabled, it looks like Reducer creates a new thread pool for every compaction. These pools seem to just sit around - i.e. "executor.shutdown()" never gets called and the Threads live forever waiting for tasks that will never come. For instance...
> Name: CompactionReducer:1
> State: TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@72938aea
> Total blocked: 0 Total waited: 1
> Stack trace:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> java.lang.Thread.run(Thread.java:722)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3711) Unsustainable Thread Accumulation
in ParallelCompactionIterable.Reducer ThreadPoolExecutor
Posted by "Caleb William Rackliffe (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Caleb William Rackliffe updated CASSANDRA-3711:
-----------------------------------------------
Attachment: 3711.2.txt
I guess I should grant the license, just in case ;)
> Unsustainable Thread Accumulation in ParallelCompactionIterable.Reducer ThreadPoolExecutor
> ------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-3711
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3711
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: - Linux version 2.6.32-71.29.1.el6.x86_64 (mockbuild@c6b5.bsys.dev.centos.org) (gcc version 4.4.4 20100726 (Red Hat 4.4.4-13) (GCC) ) #1 SMP Mon Jun 27 19:49:27 BST 2011
> - java version "1.7.0_01" / Java(TM) SE Runtime Environment (build 1.7.0_01-b08) / Java HotSpot(TM) 64-Bit Server VM (build 21.1-b02, mixed mode)
> Reporter: Caleb William Rackliffe
> Assignee: Jonathan Ellis
> Priority: Minor
> Labels: compaction, memory_leak, threading, threads
> Fix For: 1.0.7
>
> Attachments: 3711.2.txt, 3711.txt
>
>
> With multithreaded compaction enabled, it looks like Reducer creates a new thread pool for every compaction. These pools seem to just sit around - i.e. "executor.shutdown()" never gets called and the Threads live forever waiting for tasks that will never come. For instance...
> Name: CompactionReducer:1
> State: TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@72938aea
> Total blocked: 0 Total waited: 1
> Stack trace:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> java.lang.Thread.run(Thread.java:722)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3711) Unsustainable Thread Accumulation
in ParallelCompactionIterable.Reducer ThreadPoolExecutor
Posted by "Caleb William Rackliffe (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Caleb William Rackliffe updated CASSANDRA-3711:
-----------------------------------------------
Attachment: 3711.2.txt
Would it be useful to just create the executor once and reuse it for all compactions, then get rid of it in a shutdown hook or something similar? I've attached a patch, 3711.2.txt, that does roughly that, if you want to take a look.
> Unsustainable Thread Accumulation in ParallelCompactionIterable.Reducer ThreadPoolExecutor
> ------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-3711
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3711
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: - Linux version 2.6.32-71.29.1.el6.x86_64 (mockbuild@c6b5.bsys.dev.centos.org) (gcc version 4.4.4 20100726 (Red Hat 4.4.4-13) (GCC) ) #1 SMP Mon Jun 27 19:49:27 BST 2011
> - java version "1.7.0_01" / Java(TM) SE Runtime Environment (build 1.7.0_01-b08) / Java HotSpot(TM) 64-Bit Server VM (build 21.1-b02, mixed mode)
> Reporter: Caleb William Rackliffe
> Assignee: Jonathan Ellis
> Priority: Minor
> Labels: compaction, memory_leak, threading, threads
> Fix For: 1.0.7
>
> Attachments: 3711.2.txt, 3711.txt
>
>
> With multithreaded compaction enabled, it looks like Reducer creates a new thread pool for every compaction. These pools seem to just sit around - i.e. "executor.shutdown()" never gets called and the Threads live forever waiting for tasks that will never come. For instance...
> Name: CompactionReducer:1
> State: TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@72938aea
> Total blocked: 0 Total waited: 1
> Stack trace:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> java.lang.Thread.run(Thread.java:722)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3711) Unsustainable Thread Accumulation
in ParallelCompactionIterable.Reducer ThreadPoolExecutor
Posted by "Jonathan Ellis (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-3711:
--------------------------------------
Attachment: 3711.txt
You're absolutely right. Patch attached.
> Unsustainable Thread Accumulation in ParallelCompactionIterable.Reducer ThreadPoolExecutor
> ------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-3711
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3711
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: - Linux version 2.6.32-71.29.1.el6.x86_64 (mockbuild@c6b5.bsys.dev.centos.org) (gcc version 4.4.4 20100726 (Red Hat 4.4.4-13) (GCC) ) #1 SMP Mon Jun 27 19:49:27 BST 2011
> - java version "1.7.0_01" / Java(TM) SE Runtime Environment (build 1.7.0_01-b08) / Java HotSpot(TM) 64-Bit Server VM (build 21.1-b02, mixed mode)
> Reporter: Caleb William Rackliffe
> Labels: compaction, memory_leak, threading, threads
> Fix For: 1.0.7
>
> Attachments: 3711.txt
>
>
> With multithreaded compaction enabled, it looks like Reducer creates a new thread pool for every compaction. These pools seem to just sit around - i.e. "executor.shutdown()" never gets called and the Threads live forever waiting for tasks that will never come. For instance...
> Name: CompactionReducer:1
> State: TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@72938aea
> Total blocked: 0 Total waited: 1
> Stack trace:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> java.lang.Thread.run(Thread.java:722)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3711) Unsustainable Thread Accumulation
in ParallelCompactionIterable.Reducer ThreadPoolExecutor
Posted by "Brandon Williams (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams updated CASSANDRA-3711:
----------------------------------------
Fix Version/s: 1.0.7
> Unsustainable Thread Accumulation in ParallelCompactionIterable.Reducer ThreadPoolExecutor
> ------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-3711
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3711
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.5
> Environment: - Linux version 2.6.32-71.29.1.el6.x86_64 (mockbuild@c6b5.bsys.dev.centos.org) (gcc version 4.4.4 20100726 (Red Hat 4.4.4-13) (GCC) ) #1 SMP Mon Jun 27 19:49:27 BST 2011
> - java version "1.7.0_01" / Java(TM) SE Runtime Environment (build 1.7.0_01-b08) / Java HotSpot(TM) 64-Bit Server VM (build 21.1-b02, mixed mode)
> Reporter: Caleb William Rackliffe
> Labels: memory_leak, threading, threads
> Fix For: 1.0.7
>
>
> With multithreaded compaction enabled, it looks like Reducer creates a new thread pool for every compaction. These pools seem to just sit around - i.e. "executor.shutdown()" never gets called and the Threads live forever waiting for tasks that will never come. For instance...
> Name: CompactionReducer:1
> State: TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@72938aea
> Total blocked: 0 Total waited: 1
> Stack trace:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> java.lang.Thread.run(Thread.java:722)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3711) Unsustainable Thread Accumulation
in ParallelCompactionIterable.Reducer ThreadPoolExecutor
Posted by "Caleb William Rackliffe (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Caleb William Rackliffe updated CASSANDRA-3711:
-----------------------------------------------
Environment:
- Linux version 2.6.32-71.29.1.el6.x86_64 (mockbuild@c6b5.bsys.dev.centos.org) (gcc version 4.4.4 20100726 (Red Hat 4.4.4-13) (GCC) ) #1 SMP Mon Jun 27 19:49:27 BST 2011
- java version "1.7.0_01" / Java(TM) SE Runtime Environment (build 1.7.0_01-b08) / Java HotSpot(TM) 64-Bit Server VM (build 21.1-b02, mixed mode)
was:Linux version 2.6.32-71.29.1.el6.x86_64 (mockbuild@c6b5.bsys.dev.centos.org) (gcc version 4.4.4 20100726 (Red Hat 4.4.4-13) (GCC) ) #1 SMP Mon Jun 27 19:49:27 BST 2011
> Unsustainable Thread Accumulation in ParallelCompactionIterable.Reducer ThreadPoolExecutor
> ------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-3711
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3711
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.5
> Environment: - Linux version 2.6.32-71.29.1.el6.x86_64 (mockbuild@c6b5.bsys.dev.centos.org) (gcc version 4.4.4 20100726 (Red Hat 4.4.4-13) (GCC) ) #1 SMP Mon Jun 27 19:49:27 BST 2011
> - java version "1.7.0_01" / Java(TM) SE Runtime Environment (build 1.7.0_01-b08) / Java HotSpot(TM) 64-Bit Server VM (build 21.1-b02, mixed mode)
> Reporter: Caleb William Rackliffe
> Labels: memory_leak, threading, threads
>
> With multithreaded compaction enabled, it looks like Reducer creates a new thread pool for every compaction. These pools seem to just sit around - i.e. "executor.shutdown()" never gets called and the Threads live forever waiting for tasks that will never come. For instance...
> Name: CompactionReducer:1
> State: TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@72938aea
> Total blocked: 0 Total waited: 1
> Stack trace:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> java.lang.Thread.run(Thread.java:722)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3711) Unsustainable Thread Accumulation
in ParallelCompactionIterable.Reducer ThreadPoolExecutor
Posted by "Caleb William Rackliffe (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Caleb William Rackliffe updated CASSANDRA-3711:
-----------------------------------------------
Attachment: (was: 3711.2.txt)
> Unsustainable Thread Accumulation in ParallelCompactionIterable.Reducer ThreadPoolExecutor
> ------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-3711
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3711
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: - Linux version 2.6.32-71.29.1.el6.x86_64 (mockbuild@c6b5.bsys.dev.centos.org) (gcc version 4.4.4 20100726 (Red Hat 4.4.4-13) (GCC) ) #1 SMP Mon Jun 27 19:49:27 BST 2011
> - java version "1.7.0_01" / Java(TM) SE Runtime Environment (build 1.7.0_01-b08) / Java HotSpot(TM) 64-Bit Server VM (build 21.1-b02, mixed mode)
> Reporter: Caleb William Rackliffe
> Assignee: Jonathan Ellis
> Priority: Minor
> Labels: compaction, memory_leak, threading, threads
> Fix For: 1.0.7
>
> Attachments: 3711.2.txt, 3711.txt
>
>
> With multithreaded compaction enabled, it looks like Reducer creates a new thread pool for every compaction. These pools seem to just sit around - i.e. "executor.shutdown()" never gets called and the Threads live forever waiting for tasks that will never come. For instance...
> Name: CompactionReducer:1
> State: TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@72938aea
> Total blocked: 0 Total waited: 1
> Stack trace:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> java.lang.Thread.run(Thread.java:722)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3711) Unsustainable Thread
Accumulation in ParallelCompactionIterable.Reducer ThreadPoolExecutor
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182659#comment-13182659 ]
Jonathan Ellis commented on CASSANDRA-3711:
-------------------------------------------
That changes the behavior, though, since now you have a single pool of AvailableProcessors across all compactions, rather than one per compaction. Increasing the thread count proportionate to concurrent compactions setting isn't a silver bullet either, since you would still have the possibility of a large compaction "starving" out smaller ones, which was the problem that concurrent compactions were introduced to solve.
> Unsustainable Thread Accumulation in ParallelCompactionIterable.Reducer ThreadPoolExecutor
> ------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-3711
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3711
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: - Linux version 2.6.32-71.29.1.el6.x86_64 (mockbuild@c6b5.bsys.dev.centos.org) (gcc version 4.4.4 20100726 (Red Hat 4.4.4-13) (GCC) ) #1 SMP Mon Jun 27 19:49:27 BST 2011
> - java version "1.7.0_01" / Java(TM) SE Runtime Environment (build 1.7.0_01-b08) / Java HotSpot(TM) 64-Bit Server VM (build 21.1-b02, mixed mode)
> Reporter: Caleb William Rackliffe
> Assignee: Jonathan Ellis
> Priority: Minor
> Labels: compaction, memory_leak, threading, threads
> Fix For: 1.0.7
>
> Attachments: 3711.2.txt, 3711.txt
>
>
> With multithreaded compaction enabled, it looks like Reducer creates a new thread pool for every compaction. These pools seem to just sit around - i.e. "executor.shutdown()" never gets called and the Threads live forever waiting for tasks that will never come. For instance...
> Name: CompactionReducer:1
> State: TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@72938aea
> Total blocked: 0 Total waited: 1
> Stack trace:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> java.lang.Thread.run(Thread.java:722)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3711) Unsustainable Thread
Accumulation in ParallelCompactionIterable.Reducer ThreadPoolExecutor
Posted by "Caleb William Rackliffe (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182689#comment-13182689 ]
Caleb William Rackliffe commented on CASSANDRA-3711:
----------------------------------------------------
Yeah, I think you're right. In retrospect, my idea would only have been useful (maybe) if the compactions were CPU-bound, which I guess is not the case.
I can't wait for this release, because I can stop doing rolling restarts every few days!
> Unsustainable Thread Accumulation in ParallelCompactionIterable.Reducer ThreadPoolExecutor
> ------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-3711
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3711
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: - Linux version 2.6.32-71.29.1.el6.x86_64 (mockbuild@c6b5.bsys.dev.centos.org) (gcc version 4.4.4 20100726 (Red Hat 4.4.4-13) (GCC) ) #1 SMP Mon Jun 27 19:49:27 BST 2011
> - java version "1.7.0_01" / Java(TM) SE Runtime Environment (build 1.7.0_01-b08) / Java HotSpot(TM) 64-Bit Server VM (build 21.1-b02, mixed mode)
> Reporter: Caleb William Rackliffe
> Assignee: Jonathan Ellis
> Priority: Minor
> Labels: compaction, memory_leak, threading, threads
> Fix For: 1.0.7
>
> Attachments: 3711.2.txt, 3711.txt
>
>
> With multithreaded compaction enabled, it looks like Reducer creates a new thread pool for every compaction. These pools seem to just sit around - i.e. "executor.shutdown()" never gets called and the Threads live forever waiting for tasks that will never come. For instance...
> Name: CompactionReducer:1
> State: TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@72938aea
> Total blocked: 0 Total waited: 1
> Stack trace:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> java.lang.Thread.run(Thread.java:722)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira