[jira] [Created] (BROOKLYN-181) OutOfMemoryError: unable to create new native thread (when stopping app)

["Aled Sage (JIRA)" <ji...@apache.org>]

Aled Sage created BROOKLYN-181:
----------------------------------

             Summary: OutOfMemoryError: unable to create new native thread (when stopping app)
                 Key: BROOKLYN-181
                 URL: https://issues.apache.org/jira/browse/BROOKLYN-181
             Project: Brooklyn
          Issue Type: Bug
    Affects Versions: 0.8.0
            Reporter: Aled Sage


Using 0.9.0-SNAPSHOT...

I deployed an app to OpenStack (Bluebox) - it failed due to insufficient security groups etc. I terminated the VMs in Bluebox itself, and then I tried to call the "Stop" effector on the app in Brooklyn. Eventually Brooklyn ran out of threads.

My app consisted of a riak cluster + web cluster.

However, the log is messy because I had a couple of other apps that I also started/stopped during this time (I think they are unrelated though).

There are 2045 threads, of which 2012 are of the form "brooklyn-execmanager-*".

998 of these threads are doing:

{noformat}
"brooklyn-execmanager-vvDaziiT-2847" daemon prio=5 tid=0x00007fd4f08aa800 nid=0xc7d0f waiting for monitor entry [0x0000700063d3f000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at org.apache.brooklyn.camp.brooklyn.spi.dsl.BrooklynDslDeferredSupplier.get(BrooklynDslDeferredSupplier.java:83)
        - waiting to lock <0x00000007c5940bc8> (a org.apache.brooklyn.camp.brooklyn.spi.dsl.methods.DslComponent$AttributeWhenReady)
        at org.apache.brooklyn.util.core.task.ValueResolver$2.call(ValueResolver.java:322)
        at org.apache.brooklyn.util.core.task.DynamicSequentialTask$DstJob.call(DynamicSequentialTask.java:342)
        at org.apache.brooklyn.util.core.task.BasicExecutionManager$SubmissionCallable.call(BasicExecutionManager.java:468)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
{noformat}

Another 999 threads look like:

{noformat}
"brooklyn-execmanager-vvDaziiT-2930" daemon prio=5 tid=0x00007fd4ef16f800 nid=0x5e07 in Object.wait() [0x0000700001703000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at org.apache.brooklyn.util.core.task.DynamicSequentialTask$DstJob$1.call(DynamicSequentialTask.java:275)
        - locked <0x00000007c64bafa8> (a java.lang.Object)
        at org.apache.brooklyn.util.core.task.DynamicSequentialTask$DstJob$1.call(DynamicSequentialTask.java:265)
        at org.apache.brooklyn.util.core.task.BasicExecutionManager$SubmissionCallable.call(BasicExecutionManager.java:468)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
{noformat}


The logging shows the number of threads creeping up, with 30 more threads and 30 more active tasks being added every minute, e.g.:

{noformat}
2015-10-12 23:01:14,022 DEBUG o.a.b.c.m.i.BrooklynGarbageCollector [brooklyn-gc]: brooklyn gc (after) - using 164 MB / 264 MB memory (20.9 kB soft); 2001 threads; storage: {datagrid={size=7, createCount=7}, refsMapSize=0, lis
tsMapSize=0}; tasks: 1981 active, 999 unfinished; 1084 remembered, 286554 total submitted)
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)