You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@accumulo.apache.org by Michael Giordano <MG...@netcentricinc.com> on 2013/04/24 15:48:05 UTC
PermGen leak in JBoss
Under version 1.3.7 we are using the following code to initialize a cloudbase connection during initialization of our web app:
ZooKeeperInstance instance = new ZooKeeperInstance(instanceName, zooKeepers);
connector = instance.getConnector(userId, password.getBytes());
The problem is that under the hood, this call creates several (5) threads that are not cleaned up when the app is undeployed in JBoss. This is occurring without performing any scans or interacting with cloudbase in any other way. After relatively few redeploys of the app, the PermGen Space is OOM.
I can't find any reference in the cloudbase API akin to a close() method for the Connector object. This is a classloader leak effecting any webapp that is accessing cloudbase directly. The result of this leak is not simply orphaned threads, but thousands of classes not gc'd because the classloader itself can't be gc'd. This is what is filling up PermGen.
Has anyone discovered this particular issue? Has anyone discovered a solution?
Thanks,
Mike G.
NetCentric Technology, Inc.
3349 Route 138, Building A
Wall, NJ 07719
Phone: 732-544-0888
RE: PermGen leak in JBoss
Posted by Heath Abelson <HA...@netcentricinc.com>.
Nice, I'll try that. That's 1 out of 3 (I'm ignoring the Watchdogs for now)
I am able to kill the SendThread by blocking port 2181. I know that's not really a viable workaround, I just thought it might be relevant info for tracking down the problem.
From: Keith Turner [mailto:keith@deenlo.com]
Sent: Wednesday, April 24, 2013 12:35 PM
To: user@accumulo.apache.org
Subject: Re: PermGen leak in JBoss
If you execute the following code in the before the app is undeployed and in the correct classloader context, then it will cause the "Thrift Connection Pool Checker" thread to die w/ a NPE being printed.
Field cacheField = ThriftTransportPool.class.getDeclaredField("cache");
cacheField.setAccessible(true);
cacheField.set(ThriftTransportPool.getInstance(), null);
On Wed, Apr 24, 2013 at 12:08 PM, Heath Abelson <HA...@netcentricinc.com>> wrote:
As an additional point, we have a second webapp that also uses 1.3.7 that also leaves the same threads - minus the 2 two watchdog threads
From: Eric Newton [mailto:eric.newton@gmail.com<ma...@gmail.com>]
Sent: Wednesday, April 24, 2013 11:47 AM
To: user@accumulo.apache.org<ma...@accumulo.apache.org>
Subject: Re: PermGen leak in JBoss
One is for the dynamic reloading of the log4j configuration, IIRC.
-Eric
On Wed, Apr 24, 2013 at 11:44 AM, Keith Turner <ke...@deenlo.com>> wrote:
On Wed, Apr 24, 2013 at 11:21 AM, Heath Abelson <HA...@netcentricinc.com>> wrote:
Hi, I am a colleague of Michael's, I can provide more info
The following are the 5 threads that are not being cleaned up on an undeploy. It's hard to get more stack info than this.
Daemon Thread [FileWatchdog] (Running)
Daemon Thread [MSC service thread 1-5-SendThread(cbdbtestbox:2181)] (Running)
Daemon Thread [MSC service thread 1-5-EventThread] (Running)
Daemon Thread [Thrift Connection Pool Checker] (Running)
Daemon Thread [FileWatchdog] (Running)
Using Plumbr, the report states that the SendThread and EventThread above both contain an instance of org.apache.zookeeper.ClientCnxn
So two zookeeper threads. An Accumulo thread (Thrift Connection Pool Checker). What are the FileWatchdog threads? Those could be related to accumulo-start or hdfs.
-Heath
-----Original Message-----
From: Michael Giordano
Sent: Wednesday, April 24, 2013 9:48 AM
To: user@accumulo.apache.org<ma...@accumulo.apache.org>
Subject: PermGen leak in JBoss
Under version 1.3.7 we are using the following code to initialize a cloudbase connection during initialization of our web app:
ZooKeeperInstance instance = new ZooKeeperInstance(instanceName, zooKeepers);
connector = instance.getConnector(userId, password.getBytes());
The problem is that under the hood, this call creates several (5) threads that are not cleaned up when the app is undeployed in JBoss. This is occurring without performing any scans or interacting with cloudbase in any other way. After relatively few redeploys of the app, the PermGen Space is OOM.
I can't find any reference in the cloudbase API akin to a close() method for the Connector object. This is a classloader leak effecting any webapp that is accessing cloudbase directly. The result of this leak is not simply orphaned threads, but thousands of classes not gc'd because the classloader itself can't be gc'd. This is what is filling up PermGen.
Has anyone discovered this particular issue? Has anyone discovered a solution?
Thanks,
Mike G.
NetCentric Technology, Inc.
3349 Route 138, Building A
Wall, NJ 07719
Phone: 732-544-0888<tel:732-544-0888>
Re: PermGen leak in JBoss
Posted by Benson Margulies <bi...@gmail.com>.
would it not be better to make the accumulo connection a JNDI resource
outside the webapp?
On Wed, Apr 24, 2013 at 12:34 PM, Keith Turner <ke...@deenlo.com> wrote:
> If you execute the following code in the before the app is undeployed and
> in the correct classloader context, then it will cause the "Thrift
> Connection Pool Checker" thread to die w/ a NPE being printed.
>
> Field cacheField = ThriftTransportPool.class.getDeclaredField("cache");
> cacheField.setAccessible(true);
> cacheField.set(ThriftTransportPool.getInstance(), null);
>
>
> On Wed, Apr 24, 2013 at 12:08 PM, Heath Abelson <
> HAbelson@netcentricinc.com> wrote:
>
>> As an additional point, we have a second webapp that also uses 1.3.7
>> that also leaves the same threads – minus the 2 two watchdog threads****
>>
>> ** **
>>
>> *From:* Eric Newton [mailto:eric.newton@gmail.com]
>> *Sent:* Wednesday, April 24, 2013 11:47 AM
>> *To:* user@accumulo.apache.org
>> *Subject:* Re: PermGen leak in JBoss****
>>
>> ** **
>>
>> One is for the dynamic reloading of the log4j configuration, IIRC.****
>>
>> ** **
>>
>> -Eric****
>>
>> ** **
>>
>> ** **
>>
>> On Wed, Apr 24, 2013 at 11:44 AM, Keith Turner <ke...@deenlo.com> wrote:*
>> ***
>>
>> ** **
>>
>> ** **
>>
>> On Wed, Apr 24, 2013 at 11:21 AM, Heath Abelson <
>> HAbelson@netcentricinc.com> wrote:****
>>
>> Hi, I am a colleague of Michael's, I can provide more info
>>
>> The following are the 5 threads that are not being cleaned up on an
>> undeploy. It's hard to get more stack info than this.
>>
>> Daemon Thread [FileWatchdog] (Running)
>> Daemon Thread [MSC service thread 1-5-SendThread(cbdbtestbox:2181)]
>> (Running)
>> Daemon Thread [MSC service thread 1-5-EventThread] (Running)
>> Daemon Thread [Thrift Connection Pool Checker] (Running)
>> Daemon Thread [FileWatchdog] (Running)
>>
>> Using Plumbr, the report states that the SendThread and EventThread above
>> both contain an instance of org.apache.zookeeper.ClientCnxn****
>>
>> ** **
>>
>> So two zookeeper threads. An Accumulo thread (Thrift Connection Pool
>> Checker). What are the FileWatchdog threads? Those could be related to
>> accumulo-start or hdfs.****
>>
>> ****
>>
>>
>> -Heath****
>>
>>
>> -----Original Message-----
>> From: Michael Giordano
>> Sent: Wednesday, April 24, 2013 9:48 AM
>> To: user@accumulo.apache.org
>> Subject: PermGen leak in JBoss
>>
>> Under version 1.3.7 we are using the following code to initialize a
>> cloudbase connection during initialization of our web app:
>>
>> ZooKeeperInstance instance = new
>> ZooKeeperInstance(instanceName, zooKeepers);
>> connector = instance.getConnector(userId,
>> password.getBytes());
>>
>> The problem is that under the hood, this call creates several (5) threads
>> that are not cleaned up when the app is undeployed in JBoss. This is
>> occurring without performing any scans or interacting with cloudbase in any
>> other way. After relatively few redeploys of the app, the PermGen Space is
>> OOM.
>>
>> I can't find any reference in the cloudbase API akin to a close() method
>> for the Connector object. This is a classloader leak effecting any webapp
>> that is accessing cloudbase directly. The result of this leak is not simply
>> orphaned threads, but thousands of classes not gc'd because the classloader
>> itself can't be gc'd. This is what is filling up PermGen.
>>
>> Has anyone discovered this particular issue? Has anyone discovered a
>> solution?
>>
>> Thanks,
>> Mike G.
>>
>> NetCentric Technology, Inc.
>> 3349 Route 138, Building A
>> Wall, NJ 07719
>> Phone: 732-544-0888
>>
>> ****
>>
>> ** **
>>
>> ** **
>>
>
>
Re: PermGen leak in JBoss
Posted by Keith Turner <ke...@deenlo.com>.
If you execute the following code in the before the app is undeployed and
in the correct classloader context, then it will cause the "Thrift
Connection Pool Checker" thread to die w/ a NPE being printed.
Field cacheField = ThriftTransportPool.class.getDeclaredField("cache");
cacheField.setAccessible(true);
cacheField.set(ThriftTransportPool.getInstance(), null);
On Wed, Apr 24, 2013 at 12:08 PM, Heath Abelson
<HA...@netcentricinc.com>wrote:
> As an additional point, we have a second webapp that also uses 1.3.7
> that also leaves the same threads – minus the 2 two watchdog threads****
>
> ** **
>
> *From:* Eric Newton [mailto:eric.newton@gmail.com]
> *Sent:* Wednesday, April 24, 2013 11:47 AM
> *To:* user@accumulo.apache.org
> *Subject:* Re: PermGen leak in JBoss****
>
> ** **
>
> One is for the dynamic reloading of the log4j configuration, IIRC.****
>
> ** **
>
> -Eric****
>
> ** **
>
> ** **
>
> On Wed, Apr 24, 2013 at 11:44 AM, Keith Turner <ke...@deenlo.com> wrote:**
> **
>
> ** **
>
> ** **
>
> On Wed, Apr 24, 2013 at 11:21 AM, Heath Abelson <
> HAbelson@netcentricinc.com> wrote:****
>
> Hi, I am a colleague of Michael's, I can provide more info
>
> The following are the 5 threads that are not being cleaned up on an
> undeploy. It's hard to get more stack info than this.
>
> Daemon Thread [FileWatchdog] (Running)
> Daemon Thread [MSC service thread 1-5-SendThread(cbdbtestbox:2181)]
> (Running)
> Daemon Thread [MSC service thread 1-5-EventThread] (Running)
> Daemon Thread [Thrift Connection Pool Checker] (Running)
> Daemon Thread [FileWatchdog] (Running)
>
> Using Plumbr, the report states that the SendThread and EventThread above
> both contain an instance of org.apache.zookeeper.ClientCnxn****
>
> ** **
>
> So two zookeeper threads. An Accumulo thread (Thrift Connection Pool
> Checker). What are the FileWatchdog threads? Those could be related to
> accumulo-start or hdfs.****
>
> ****
>
>
> -Heath****
>
>
> -----Original Message-----
> From: Michael Giordano
> Sent: Wednesday, April 24, 2013 9:48 AM
> To: user@accumulo.apache.org
> Subject: PermGen leak in JBoss
>
> Under version 1.3.7 we are using the following code to initialize a
> cloudbase connection during initialization of our web app:
>
> ZooKeeperInstance instance = new
> ZooKeeperInstance(instanceName, zooKeepers);
> connector = instance.getConnector(userId,
> password.getBytes());
>
> The problem is that under the hood, this call creates several (5) threads
> that are not cleaned up when the app is undeployed in JBoss. This is
> occurring without performing any scans or interacting with cloudbase in any
> other way. After relatively few redeploys of the app, the PermGen Space is
> OOM.
>
> I can't find any reference in the cloudbase API akin to a close() method
> for the Connector object. This is a classloader leak effecting any webapp
> that is accessing cloudbase directly. The result of this leak is not simply
> orphaned threads, but thousands of classes not gc'd because the classloader
> itself can't be gc'd. This is what is filling up PermGen.
>
> Has anyone discovered this particular issue? Has anyone discovered a
> solution?
>
> Thanks,
> Mike G.
>
> NetCentric Technology, Inc.
> 3349 Route 138, Building A
> Wall, NJ 07719
> Phone: 732-544-0888
>
> ****
>
> ** **
>
> ** **
>
RE: PermGen leak in JBoss
Posted by Heath Abelson <HA...@netcentricinc.com>.
As an additional point, we have a second webapp that also uses 1.3.7 that also leaves the same threads - minus the 2 two watchdog threads
From: Eric Newton [mailto:eric.newton@gmail.com]
Sent: Wednesday, April 24, 2013 11:47 AM
To: user@accumulo.apache.org
Subject: Re: PermGen leak in JBoss
One is for the dynamic reloading of the log4j configuration, IIRC.
-Eric
On Wed, Apr 24, 2013 at 11:44 AM, Keith Turner <ke...@deenlo.com>> wrote:
On Wed, Apr 24, 2013 at 11:21 AM, Heath Abelson <HA...@netcentricinc.com>> wrote:
Hi, I am a colleague of Michael's, I can provide more info
The following are the 5 threads that are not being cleaned up on an undeploy. It's hard to get more stack info than this.
Daemon Thread [FileWatchdog] (Running)
Daemon Thread [MSC service thread 1-5-SendThread(cbdbtestbox:2181)] (Running)
Daemon Thread [MSC service thread 1-5-EventThread] (Running)
Daemon Thread [Thrift Connection Pool Checker] (Running)
Daemon Thread [FileWatchdog] (Running)
Using Plumbr, the report states that the SendThread and EventThread above both contain an instance of org.apache.zookeeper.ClientCnxn
So two zookeeper threads. An Accumulo thread (Thrift Connection Pool Checker). What are the FileWatchdog threads? Those could be related to accumulo-start or hdfs.
-Heath
-----Original Message-----
From: Michael Giordano
Sent: Wednesday, April 24, 2013 9:48 AM
To: user@accumulo.apache.org<ma...@accumulo.apache.org>
Subject: PermGen leak in JBoss
Under version 1.3.7 we are using the following code to initialize a cloudbase connection during initialization of our web app:
ZooKeeperInstance instance = new ZooKeeperInstance(instanceName, zooKeepers);
connector = instance.getConnector(userId, password.getBytes());
The problem is that under the hood, this call creates several (5) threads that are not cleaned up when the app is undeployed in JBoss. This is occurring without performing any scans or interacting with cloudbase in any other way. After relatively few redeploys of the app, the PermGen Space is OOM.
I can't find any reference in the cloudbase API akin to a close() method for the Connector object. This is a classloader leak effecting any webapp that is accessing cloudbase directly. The result of this leak is not simply orphaned threads, but thousands of classes not gc'd because the classloader itself can't be gc'd. This is what is filling up PermGen.
Has anyone discovered this particular issue? Has anyone discovered a solution?
Thanks,
Mike G.
NetCentric Technology, Inc.
3349 Route 138, Building A
Wall, NJ 07719
Phone: 732-544-0888<tel:732-544-0888>
Re: PermGen leak in JBoss
Posted by Eric Newton <er...@gmail.com>.
One is for the dynamic reloading of the log4j configuration, IIRC.
-Eric
On Wed, Apr 24, 2013 at 11:44 AM, Keith Turner <ke...@deenlo.com> wrote:
>
>
>
> On Wed, Apr 24, 2013 at 11:21 AM, Heath Abelson <
> HAbelson@netcentricinc.com> wrote:
>
>> Hi, I am a colleague of Michael's, I can provide more info
>>
>> The following are the 5 threads that are not being cleaned up on an
>> undeploy. It's hard to get more stack info than this.
>>
>> Daemon Thread [FileWatchdog] (Running)
>> Daemon Thread [MSC service thread 1-5-SendThread(cbdbtestbox:2181)]
>> (Running)
>> Daemon Thread [MSC service thread 1-5-EventThread] (Running)
>> Daemon Thread [Thrift Connection Pool Checker] (Running)
>> Daemon Thread [FileWatchdog] (Running)
>>
>> Using Plumbr, the report states that the SendThread and EventThread above
>> both contain an instance of org.apache.zookeeper.ClientCnxn
>>
>
> So two zookeeper threads. An Accumulo thread (Thrift Connection Pool
> Checker). What are the FileWatchdog threads? Those could be related to
> accumulo-start or hdfs.
>
>
>>
>> -Heath
>>
>> -----Original Message-----
>> From: Michael Giordano
>> Sent: Wednesday, April 24, 2013 9:48 AM
>> To: user@accumulo.apache.org
>> Subject: PermGen leak in JBoss
>>
>> Under version 1.3.7 we are using the following code to initialize a
>> cloudbase connection during initialization of our web app:
>>
>> ZooKeeperInstance instance = new
>> ZooKeeperInstance(instanceName, zooKeepers);
>> connector = instance.getConnector(userId,
>> password.getBytes());
>>
>> The problem is that under the hood, this call creates several (5) threads
>> that are not cleaned up when the app is undeployed in JBoss. This is
>> occurring without performing any scans or interacting with cloudbase in any
>> other way. After relatively few redeploys of the app, the PermGen Space is
>> OOM.
>>
>> I can't find any reference in the cloudbase API akin to a close() method
>> for the Connector object. This is a classloader leak effecting any webapp
>> that is accessing cloudbase directly. The result of this leak is not simply
>> orphaned threads, but thousands of classes not gc'd because the classloader
>> itself can't be gc'd. This is what is filling up PermGen.
>>
>> Has anyone discovered this particular issue? Has anyone discovered a
>> solution?
>>
>> Thanks,
>> Mike G.
>>
>> NetCentric Technology, Inc.
>> 3349 Route 138, Building A
>> Wall, NJ 07719
>> Phone: 732-544-0888
>>
>>
>>
>
Re: PermGen leak in JBoss
Posted by Keith Turner <ke...@deenlo.com>.
On Wed, Apr 24, 2013 at 11:21 AM, Heath Abelson
<HA...@netcentricinc.com>wrote:
> Hi, I am a colleague of Michael's, I can provide more info
>
> The following are the 5 threads that are not being cleaned up on an
> undeploy. It's hard to get more stack info than this.
>
> Daemon Thread [FileWatchdog] (Running)
> Daemon Thread [MSC service thread 1-5-SendThread(cbdbtestbox:2181)]
> (Running)
> Daemon Thread [MSC service thread 1-5-EventThread] (Running)
> Daemon Thread [Thrift Connection Pool Checker] (Running)
> Daemon Thread [FileWatchdog] (Running)
>
> Using Plumbr, the report states that the SendThread and EventThread above
> both contain an instance of org.apache.zookeeper.ClientCnxn
>
So two zookeeper threads. An Accumulo thread (Thrift Connection Pool
Checker). What are the FileWatchdog threads? Those could be related to
accumulo-start or hdfs.
>
> -Heath
>
> -----Original Message-----
> From: Michael Giordano
> Sent: Wednesday, April 24, 2013 9:48 AM
> To: user@accumulo.apache.org
> Subject: PermGen leak in JBoss
>
> Under version 1.3.7 we are using the following code to initialize a
> cloudbase connection during initialization of our web app:
>
> ZooKeeperInstance instance = new
> ZooKeeperInstance(instanceName, zooKeepers);
> connector = instance.getConnector(userId,
> password.getBytes());
>
> The problem is that under the hood, this call creates several (5) threads
> that are not cleaned up when the app is undeployed in JBoss. This is
> occurring without performing any scans or interacting with cloudbase in any
> other way. After relatively few redeploys of the app, the PermGen Space is
> OOM.
>
> I can't find any reference in the cloudbase API akin to a close() method
> for the Connector object. This is a classloader leak effecting any webapp
> that is accessing cloudbase directly. The result of this leak is not simply
> orphaned threads, but thousands of classes not gc'd because the classloader
> itself can't be gc'd. This is what is filling up PermGen.
>
> Has anyone discovered this particular issue? Has anyone discovered a
> solution?
>
> Thanks,
> Mike G.
>
> NetCentric Technology, Inc.
> 3349 Route 138, Building A
> Wall, NJ 07719
> Phone: 732-544-0888
>
>
>
Re: PermGen leak in JBoss
Posted by Keith Turner <ke...@deenlo.com>.
On Fri, May 3, 2013 at 11:43 AM, Michael Giordano <
MGiordano@netcentricinc.com> wrote:
> Sorry for resurrecting this ‘old’ thread but I need to open a ticket for
> this.
>
https://issues.apache.org/jira/browse/ACCUMULO
> ****
>
> ** **
>
> How do I do that ?****
>
> ** **
>
> Thanks,****
>
> Mike G.****
>
> ** **
>
> *From:* Keith Turner [mailto:keith@deenlo.com]
> *Sent:* Wednesday, April 24, 2013 12:23 PM
>
> *To:* user@accumulo.apache.org
> *Subject:* Re: PermGen leak in JBoss****
>
> ** **
>
> ** **
>
> ** **
>
> On Wed, Apr 24, 2013 at 11:21 AM, Heath Abelson <
> HAbelson@netcentricinc.com> wrote:****
>
> Hi, I am a colleague of Michael's, I can provide more info
>
> The following are the 5 threads that are not being cleaned up on an
> undeploy. It's hard to get more stack info than this.
>
> Daemon Thread [FileWatchdog] (Running)
> Daemon Thread [MSC service thread 1-5-SendThread(cbdbtestbox:2181)]
> (Running)
> Daemon Thread [MSC service thread 1-5-EventThread] (Running)
> Daemon Thread [Thrift Connection Pool Checker] (Running)
> Daemon Thread [FileWatchdog] (Running)
>
> Using Plumbr, the report states that the SendThread and EventThread above
> both contain an instance of org.apache.zookeeper.ClientCnxn****
>
> ** **
>
> ** **
>
> I tried this on 1.4 and then ran jstack using following program. ****
>
> ** **
>
> ** **
>
> public class Test {****
>
> public static void main(String[] args) throws Exception {****
>
> ZooKeeperInstance instance = new ZooKeeperInstance("test14",
> "localhost");****
>
> Connector connector = instance.getConnector("root", "secret");****
>
> ****
>
> UtilWaitThread.sleep(600000);****
>
> ** **
>
> }****
>
> }****
>
> ** **
>
> I see the following threads after filtering out java threads and the
> thread calling main. I am running this code in a different way. I used
> the accumulo script to execute it, which uses Accumulo start, which
> introduces the AccumuloFilesystemAlterationMonitor thread. I am seeing
> three of the threads you see, I am not seeing the FileWatchDog threads.***
> *
>
> ** **
>
> "Thrift Connection Pool Checker" daemon prio=10 tid=0x00007f861839f800
> nid=0x61f0 waiting on condition [0x00007f8652a62000]****
>
> java.lang.Thread.State: TIMED_WAITING (sleeping)****
>
> at java.lang.Thread.sleep(Native Method)****
>
> at
> org.apache.accumulo.core.client.impl.ThriftTransportPool$Closer.run(ThriftTransportPool.java:129)
> ****
>
> at java.lang.Thread.run(Thread.java:662)****
>
> ** **
>
> "Test-EventThread" daemon prio=10 tid=0x00007f8618119000 nid=0x61ee
> waiting on condition [0x00007f8652d02000]****
>
> java.lang.Thread.State: WAITING (parking)****
>
> at sun.misc.Unsafe.park(Native Method)****
>
> - parking to wait for <0x00000000bb402270> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)****
>
> at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)****
>
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> ****
>
> at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> ****
>
> at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)****
>
> ** **
>
> "Test-SendThread(localhost:2181)" daemon prio=10 tid=0x00007f8618103000
> nid=0x61ed runnable [0x00007f8652e03000]****
>
> java.lang.Thread.State: RUNNABLE****
>
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)****
>
> at
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)****
>
> at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)****
>
> at
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)****
>
> - locked <0x00000000bb3e0e88> (a sun.nio.ch.Util$2)****
>
> - locked <0x00000000bb3e0e78> (a
> java.util.Collections$UnmodifiableSet)****
>
> - locked <0x00000000bb3e0aa8> (a sun.nio.ch.EPollSelectorImpl)
> ****
>
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)****
>
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1131)****
>
> ** **
>
> "org.apache.accumulo.start.classloader.AccumuloFilesystemAlterationMonitor"
> daemon prio=10 tid=0x00007f86a4387000 nid=0x61e9 waiting on condition
> [0x00007f865367c000]****
>
> java.lang.Thread.State: TIMED_WAITING (sleeping)****
>
> at java.lang.Thread.sleep(Native Method)****
>
> at
> org.apache.accumulo.start.classloader.AccumuloFilesystemAlterationMonitor.run(AccumuloFilesystemAlterationMonitor.java:127)
> ****
>
> at java.lang.Thread.run(Thread.java:662)****
>
> ** **
>
> ****
>
> Should probably open a ticket about this issue. We could probably do
> something about the Thrift Connection Pool Checker threads. I am not sure
> if the zookeeper threads are an issue w/ zookeeper or an issue w/ how
> accumulo is using zookeeper.****
>
> ** **
>
>
> -Heath****
>
>
> -----Original Message-----
> From: Michael Giordano
> Sent: Wednesday, April 24, 2013 9:48 AM
> To: user@accumulo.apache.org
> Subject: PermGen leak in JBoss
>
> Under version 1.3.7 we are using the following code to initialize a
> cloudbase connection during initialization of our web app:
>
> ZooKeeperInstance instance = new
> ZooKeeperInstance(instanceName, zooKeepers);
> connector = instance.getConnector(userId,
> password.getBytes());
>
> The problem is that under the hood, this call creates several (5) threads
> that are not cleaned up when the app is undeployed in JBoss. This is
> occurring without performing any scans or interacting with cloudbase in any
> other way. After relatively few redeploys of the app, the PermGen Space is
> OOM.
>
> I can't find any reference in the cloudbase API akin to a close() method
> for the Connector object. This is a classloader leak effecting any webapp
> that is accessing cloudbase directly. The result of this leak is not simply
> orphaned threads, but thousands of classes not gc'd because the classloader
> itself can't be gc'd. This is what is filling up PermGen.
>
> Has anyone discovered this particular issue? Has anyone discovered a
> solution?
>
> Thanks,
> Mike G.
>
> NetCentric Technology, Inc.
> 3349 Route 138, Building A
> Wall, NJ 07719
> Phone: 732-544-0888
>
> ****
>
> ** **
>
RE: PermGen leak in JBoss
Posted by Michael Giordano <MG...@netcentricinc.com>.
Sorry for resurrecting this 'old' thread but I need to open a ticket for this.
How do I do that ?
Thanks,
Mike G.
From: Keith Turner [mailto:keith@deenlo.com]
Sent: Wednesday, April 24, 2013 12:23 PM
To: user@accumulo.apache.org
Subject: Re: PermGen leak in JBoss
On Wed, Apr 24, 2013 at 11:21 AM, Heath Abelson <HA...@netcentricinc.com>> wrote:
Hi, I am a colleague of Michael's, I can provide more info
The following are the 5 threads that are not being cleaned up on an undeploy. It's hard to get more stack info than this.
Daemon Thread [FileWatchdog] (Running)
Daemon Thread [MSC service thread 1-5-SendThread(cbdbtestbox:2181)] (Running)
Daemon Thread [MSC service thread 1-5-EventThread] (Running)
Daemon Thread [Thrift Connection Pool Checker] (Running)
Daemon Thread [FileWatchdog] (Running)
Using Plumbr, the report states that the SendThread and EventThread above both contain an instance of org.apache.zookeeper.ClientCnxn
I tried this on 1.4 and then ran jstack using following program.
public class Test {
public static void main(String[] args) throws Exception {
ZooKeeperInstance instance = new ZooKeeperInstance("test14", "localhost");
Connector connector = instance.getConnector("root", "secret");
UtilWaitThread.sleep(600000);
}
}
I see the following threads after filtering out java threads and the thread calling main. I am running this code in a different way. I used the accumulo script to execute it, which uses Accumulo start, which introduces the AccumuloFilesystemAlterationMonitor thread. I am seeing three of the threads you see, I am not seeing the FileWatchDog threads.
"Thrift Connection Pool Checker" daemon prio=10 tid=0x00007f861839f800 nid=0x61f0 waiting on condition [0x00007f8652a62000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.accumulo.core.client.impl.ThriftTransportPool$Closer.run(ThriftTransportPool.java:129)
at java.lang.Thread.run(Thread.java:662)
"Test-EventThread" daemon prio=10 tid=0x00007f8618119000 nid=0x61ee waiting on condition [0x00007f8652d02000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000bb402270> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
"Test-SendThread(localhost:2181)" daemon prio=10 tid=0x00007f8618103000 nid=0x61ed runnable [0x00007f8652e03000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00000000bb3e0e88> (a sun.nio.ch.Util$2)
- locked <0x00000000bb3e0e78> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000bb3e0aa8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1131)
"org.apache.accumulo.start.classloader.AccumuloFilesystemAlterationMonitor" daemon prio=10 tid=0x00007f86a4387000 nid=0x61e9 waiting on condition [0x00007f865367c000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.accumulo.start.classloader.AccumuloFilesystemAlterationMonitor.run(AccumuloFilesystemAlterationMonitor.java:127)
at java.lang.Thread.run(Thread.java:662)
Should probably open a ticket about this issue. We could probably do something about the Thrift Connection Pool Checker threads. I am not sure if the zookeeper threads are an issue w/ zookeeper or an issue w/ how accumulo is using zookeeper.
-Heath
-----Original Message-----
From: Michael Giordano
Sent: Wednesday, April 24, 2013 9:48 AM
To: user@accumulo.apache.org<ma...@accumulo.apache.org>
Subject: PermGen leak in JBoss
Under version 1.3.7 we are using the following code to initialize a cloudbase connection during initialization of our web app:
ZooKeeperInstance instance = new ZooKeeperInstance(instanceName, zooKeepers);
connector = instance.getConnector(userId, password.getBytes());
The problem is that under the hood, this call creates several (5) threads that are not cleaned up when the app is undeployed in JBoss. This is occurring without performing any scans or interacting with cloudbase in any other way. After relatively few redeploys of the app, the PermGen Space is OOM.
I can't find any reference in the cloudbase API akin to a close() method for the Connector object. This is a classloader leak effecting any webapp that is accessing cloudbase directly. The result of this leak is not simply orphaned threads, but thousands of classes not gc'd because the classloader itself can't be gc'd. This is what is filling up PermGen.
Has anyone discovered this particular issue? Has anyone discovered a solution?
Thanks,
Mike G.
NetCentric Technology, Inc.
3349 Route 138, Building A
Wall, NJ 07719
Phone: 732-544-0888<tel:732-544-0888>
Re: PermGen leak in JBoss
Posted by Keith Turner <ke...@deenlo.com>.
On Wed, Apr 24, 2013 at 11:21 AM, Heath Abelson
<HA...@netcentricinc.com>wrote:
> Hi, I am a colleague of Michael's, I can provide more info
>
> The following are the 5 threads that are not being cleaned up on an
> undeploy. It's hard to get more stack info than this.
>
> Daemon Thread [FileWatchdog] (Running)
> Daemon Thread [MSC service thread 1-5-SendThread(cbdbtestbox:2181)]
> (Running)
> Daemon Thread [MSC service thread 1-5-EventThread] (Running)
> Daemon Thread [Thrift Connection Pool Checker] (Running)
> Daemon Thread [FileWatchdog] (Running)
>
> Using Plumbr, the report states that the SendThread and EventThread above
> both contain an instance of org.apache.zookeeper.ClientCnxn
>
I tried this on 1.4 and then ran jstack using following program.
public class Test {
public static void main(String[] args) throws Exception {
ZooKeeperInstance instance = new ZooKeeperInstance("test14",
"localhost");
Connector connector = instance.getConnector("root", "secret");
UtilWaitThread.sleep(600000);
}
}
I see the following threads after filtering out java threads and the thread
calling main. I am running this code in a different way. I used the
accumulo script to execute it, which uses Accumulo start, which introduces
the AccumuloFilesystemAlterationMonitor thread. I am seeing three of the
threads you see, I am not seeing the FileWatchDog threads.
"Thrift Connection Pool Checker" daemon prio=10 tid=0x00007f861839f800
nid=0x61f0 waiting on condition [0x00007f8652a62000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.accumulo.core.client.impl.ThriftTransportPool$Closer.run(ThriftTransportPool.java:129)
at java.lang.Thread.run(Thread.java:662)
"Test-EventThread" daemon prio=10 tid=0x00007f8618119000 nid=0x61ee waiting
on condition [0x00007f8652d02000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000bb402270> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
"Test-SendThread(localhost:2181)" daemon prio=10 tid=0x00007f8618103000
nid=0x61ed runnable [0x00007f8652e03000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00000000bb3e0e88> (a sun.nio.ch.Util$2)
- locked <0x00000000bb3e0e78> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000bb3e0aa8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1131)
"org.apache.accumulo.start.classloader.AccumuloFilesystemAlterationMonitor"
daemon prio=10 tid=0x00007f86a4387000 nid=0x61e9 waiting on condition
[0x00007f865367c000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at
org.apache.accumulo.start.classloader.AccumuloFilesystemAlterationMonitor.run(AccumuloFilesystemAlterationMonitor.java:127)
at java.lang.Thread.run(Thread.java:662)
Should probably open a ticket about this issue. We could probably do
something about the Thrift Connection Pool Checker threads. I am not sure
if the zookeeper threads are an issue w/ zookeeper or an issue w/ how
accumulo is using zookeeper.
> -Heath
>
> -----Original Message-----
> From: Michael Giordano
> Sent: Wednesday, April 24, 2013 9:48 AM
> To: user@accumulo.apache.org
> Subject: PermGen leak in JBoss
>
> Under version 1.3.7 we are using the following code to initialize a
> cloudbase connection during initialization of our web app:
>
> ZooKeeperInstance instance = new
> ZooKeeperInstance(instanceName, zooKeepers);
> connector = instance.getConnector(userId,
> password.getBytes());
>
> The problem is that under the hood, this call creates several (5) threads
> that are not cleaned up when the app is undeployed in JBoss. This is
> occurring without performing any scans or interacting with cloudbase in any
> other way. After relatively few redeploys of the app, the PermGen Space is
> OOM.
>
> I can't find any reference in the cloudbase API akin to a close() method
> for the Connector object. This is a classloader leak effecting any webapp
> that is accessing cloudbase directly. The result of this leak is not simply
> orphaned threads, but thousands of classes not gc'd because the classloader
> itself can't be gc'd. This is what is filling up PermGen.
>
> Has anyone discovered this particular issue? Has anyone discovered a
> solution?
>
> Thanks,
> Mike G.
>
> NetCentric Technology, Inc.
> 3349 Route 138, Building A
> Wall, NJ 07719
> Phone: 732-544-0888
>
>
>
RE: PermGen leak in JBoss
Posted by Heath Abelson <HA...@netcentricinc.com>.
Hi, I am a colleague of Michael's, I can provide more info
The following are the 5 threads that are not being cleaned up on an undeploy. It's hard to get more stack info than this.
Daemon Thread [FileWatchdog] (Running)
Daemon Thread [MSC service thread 1-5-SendThread(cbdbtestbox:2181)] (Running)
Daemon Thread [MSC service thread 1-5-EventThread] (Running)
Daemon Thread [Thrift Connection Pool Checker] (Running)
Daemon Thread [FileWatchdog] (Running)
Using Plumbr, the report states that the SendThread and EventThread above both contain an instance of org.apache.zookeeper.ClientCnxn
-Heath
-----Original Message-----
From: Michael Giordano
Sent: Wednesday, April 24, 2013 9:48 AM
To: user@accumulo.apache.org
Subject: PermGen leak in JBoss
Under version 1.3.7 we are using the following code to initialize a cloudbase connection during initialization of our web app:
ZooKeeperInstance instance = new ZooKeeperInstance(instanceName, zooKeepers);
connector = instance.getConnector(userId, password.getBytes());
The problem is that under the hood, this call creates several (5) threads that are not cleaned up when the app is undeployed in JBoss. This is occurring without performing any scans or interacting with cloudbase in any other way. After relatively few redeploys of the app, the PermGen Space is OOM.
I can't find any reference in the cloudbase API akin to a close() method for the Connector object. This is a classloader leak effecting any webapp that is accessing cloudbase directly. The result of this leak is not simply orphaned threads, but thousands of classes not gc'd because the classloader itself can't be gc'd. This is what is filling up PermGen.
Has anyone discovered this particular issue? Has anyone discovered a solution?
Thanks,
Mike G.
NetCentric Technology, Inc.
3349 Route 138, Building A
Wall, NJ 07719
Phone: 732-544-0888
Re: PermGen leak in JBoss
Posted by Keith Turner <ke...@deenlo.com>.
On Wed, Apr 24, 2013 at 10:12 AM, Josh Elser <jo...@gmail.com> wrote:
> Giving a quick glance at that code again, the only thing I see that would
> be active is the ZooKeeperInstance (ZooCache) which will try to stay
> connected/updated to ZooKeeper. However, that should also disconnect/expire
> itself.
>
> What's the time frame in which you're redeploying your webapp (within a
> few seconds, minutes)? Also, threadname/stacktrace for said 5 threads would
> be helpful/enlightening.
Yeah those stack traces would be nice if you have them. I suspect one
culprit may come from o.a.accumulo.core.client.impl.ThriftTransportPool,
but I am curious about the stack traces. I am going to try firing up a
client and doing what you did in 1.4 and looking at the stacks.
>
>
> On 4/24/13 9:48 AM, Michael Giordano wrote:
>
>> Under version 1.3.7 we are using the following code to initialize a
>> cloudbase connection during initialization of our web app:
>>
>> ZooKeeperInstance instance = new
>> ZooKeeperInstance(**instanceName, zooKeepers);
>> connector = instance.getConnector(userId,
>> password.getBytes());
>>
>> The problem is that under the hood, this call creates several (5) threads
>> that are not cleaned up when the app is undeployed in JBoss. This is
>> occurring without performing any scans or interacting with cloudbase in any
>> other way. After relatively few redeploys of the app, the PermGen Space is
>> OOM.
>>
>> I can't find any reference in the cloudbase API akin to a close() method
>> for the Connector object. This is a classloader leak effecting any webapp
>> that is accessing cloudbase directly. The result of this leak is not simply
>> orphaned threads, but thousands of classes not gc'd because the classloader
>> itself can't be gc'd. This is what is filling up PermGen.
>>
>> Has anyone discovered this particular issue? Has anyone discovered a
>> solution?
>>
>> Thanks,
>> Mike G.
>>
>> NetCentric Technology, Inc.
>> 3349 Route 138, Building A
>> Wall, NJ 07719
>> Phone: 732-544-0888
>>
>>
>>
>
Re: PermGen leak in JBoss
Posted by Josh Elser <jo...@gmail.com>.
Giving a quick glance at that code again, the only thing I see that
would be active is the ZooKeeperInstance (ZooCache) which will try to
stay connected/updated to ZooKeeper. However, that should also
disconnect/expire itself.
What's the time frame in which you're redeploying your webapp (within a
few seconds, minutes)? Also, threadname/stacktrace for said 5 threads
would be helpful/enlightening.
On 4/24/13 9:48 AM, Michael Giordano wrote:
> Under version 1.3.7 we are using the following code to initialize a cloudbase connection during initialization of our web app:
>
> ZooKeeperInstance instance = new ZooKeeperInstance(instanceName, zooKeepers);
> connector = instance.getConnector(userId, password.getBytes());
>
> The problem is that under the hood, this call creates several (5) threads that are not cleaned up when the app is undeployed in JBoss. This is occurring without performing any scans or interacting with cloudbase in any other way. After relatively few redeploys of the app, the PermGen Space is OOM.
>
> I can't find any reference in the cloudbase API akin to a close() method for the Connector object. This is a classloader leak effecting any webapp that is accessing cloudbase directly. The result of this leak is not simply orphaned threads, but thousands of classes not gc'd because the classloader itself can't be gc'd. This is what is filling up PermGen.
>
> Has anyone discovered this particular issue? Has anyone discovered a solution?
>
> Thanks,
> Mike G.
>
> NetCentric Technology, Inc.
> 3349 Route 138, Building A
> Wall, NJ 07719
> Phone: 732-544-0888
>
>