You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ofbiz.apache.org by "Jacques Le Roux (JIRA)" <ji...@apache.org> on 2010/12/15 14:03:03 UTC

[jira] Commented: (OFBIZ-4043) Memory leaks in (at least) stable demo

    [ https://issues.apache.org/jira/browse/OFBIZ-4043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12971670#action_12971670 ] 

Jacques Le Roux commented on OFBIZ-4043:
----------------------------------------

Another one today:

108 639 instances of "org.apache.derby.impl.jdbc.EmbedConnection40", loaded by "java.net.URLClassLoader @ 0x7fc419071150" occupy 408 594 832 (78,54%) bytes. These instances are referenced from one instance of "java.util.WeakHashMap$Entry[]", loaded by "<system class loader>"

Keywords
org.apache.derby.impl.jdbc.EmbedConnection40
java.net.URLClassLoader @ 0x7fc419071150
java.util.WeakHashMap$Entry[]


See also 2d screenshoot, this time it comes from a Timer, so I guess a job polling. It seems we don't cross such issues in trunk. The problem there seems mostly related to blocked threads. Not sure why yet, but at least here is a stack taken from a blocked state:
{code}
$ ps aux | grep ofbiz
ofbiz    29724 83.6 42.4 1468468 1092996 ?     Sl   Dec14 1111:26 java -Xms128M -Xmx768M -XX:MaxPermSize=192m -XX:+HeapDumpOnOutOfMemoryError -Dofbiz.admin.port=10523 -Dofbiz.admin.key=so3du5kasd5dn -jar ofbiz.jar

Trunk
$ jstack 29724
2010-12-15 07:25:15
Full thread dump Java HotSpot(TM) 64-Bit Server VM (17.1-b03 mixed mode):

"GenericXaResource-Thread" daemon prio=10 tid=0x00007f9a54107000 nid=0x1daa runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Attach Listener" daemon prio=10 tid=0x0000000041684000 nid=0x1da8 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"default-invoker-Thread-102545" prio=10 tid=0x00007f9a546ea800 nid=0x5ce0 waiting on condition [0x00007f9a6360f000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.ofbiz.service.job.JobInvoker.run(JobInvoker.java:226)
        - locked <0x00007f9a84464668> (a org.ofbiz.service.job.JobInvoker)
        at java.lang.Thread.run(Thread.java:662)

"default-invoker-Thread-102544" prio=10 tid=0x0000000042099000 nid=0x5cdf waiting on condition [0x00007f9a62ff9000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.ofbiz.service.job.JobInvoker.run(JobInvoker.java:226)
        - locked <0x00007f9a84464768> (a org.ofbiz.service.job.JobInvoker)
        at java.lang.Thread.run(Thread.java:662)

"default-invoker-Thread-102543" prio=10 tid=0x0000000042098800 nid=0x5cde waiting on condition [0x00007f9a58e59000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.ofbiz.service.job.JobInvoker.run(JobInvoker.java:226)
        - locked <0x00007f9a84464868> (a org.ofbiz.service.job.JobInvoker)
        at java.lang.Thread.run(Thread.java:662)

"default-invoker-Thread-102542" prio=10 tid=0x0000000041dec000 nid=0x5cdd waiting on condition [0x00007f9a58d58000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.ofbiz.service.job.JobInvoker.run(JobInvoker.java:226)
        - locked <0x00007f9a84464968> (a org.ofbiz.service.job.JobInvoker)
        at java.lang.Thread.run(Thread.java:662)

"default-invoker-Thread-102541" prio=10 tid=0x0000000042424000 nid=0x5cdc waiting on condition [0x00007f9a62ef8000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.ofbiz.service.job.JobInvoker.run(JobInvoker.java:226)
        - locked <0x00007f9a84464338> (a org.ofbiz.service.job.JobInvoker)
        at java.lang.Thread.run(Thread.java:662)

"http-0.0.0.0-8080-9" daemon prio=10 tid=0x000000004363a000 nid=0x3e88 in Object.wait() [0x00007f9a630fa000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f9a84225258> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:458)
        - locked <0x00007f9a84225258> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:484)
        at java.lang.Thread.run(Thread.java:662)

"TP-Processor35" daemon prio=10 tid=0x0000000042584800 nid=0x6ccb runnable [0x00007f9a5864f000]
   java.lang.Thread.State: RUNNABLE
        at java.lang.StringBuffer.append(StringBuffer.java:224)
        - locked <0x00007f9a828e7ed8> (a java.lang.StringBuffer)
        at java.io.StringWriter.write(StringWriter.java:84)
        at java.io.StringWriter.append(StringWriter.java:126)
        at java.io.StringWriter.append(StringWriter.java:24)
        at org.ofbiz.base.util.UtilIO.copy(UtilIO.java:111)
        at org.ofbiz.content.data.DataResourceWorker.renderFile(DataResourceWorker.java:899)
        at org.ofbiz.content.data.DataResourceWorker.writeDataResourceText(DataResourceWorker.java:820)
        at org.ofbiz.content.data.DataResourceWorker.renderDataResourceAsText(DataResourceWorker.java:626)
        at org.ofbiz.content.content.ContentWorker.renderContentAsText(ContentWorker.java:273)
        at org.ofbiz.content.content.ContentWorker.renderContentAsText(ContentWorker.java:330)
        at org.ofbiz.content.cms.CmsEvents.cms(CmsEvents.java:289)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.ofbiz.webapp.event.JavaEventHandler.invoke(JavaEventHandler.java:92)
        at org.ofbiz.webapp.event.JavaEventHandler.invoke(JavaEventHandler.java:78)
        at org.ofbiz.webapp.control.RequestHandler.runEvent(RequestHandler.java:635)
        at org.ofbiz.webapp.control.RequestHandler.doRequest(RequestHandler.java:381)
        at org.ofbiz.webapp.control.ControlServlet.doGet(ControlServlet.java:230)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.ofbiz.webapp.control.ContextFilter.doFilter(ContextFilter.java:338)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
        at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:190)
        at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)
        at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:774)
        at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:703)
        at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:896)
        at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
        at java.lang.Thread.run(Thread.java:662)

"http-0.0.0.0-8080-8" daemon prio=10 tid=0x0000000042343000 nid=0x6b1a in Object.wait() [0x00007f9a59560000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f9a82646da8> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:458)
        - locked <0x00007f9a82646da8> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:484)
        at java.lang.Thread.run(Thread.java:662)

"http-0.0.0.0-8080-7" daemon prio=10 tid=0x0000000042ab5800 nid=0x6b19 in Object.wait() [0x00007f9a58550000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f9a82648040> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:458)
        - locked <0x00007f9a82648040> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:484)
        at java.lang.Thread.run(Thread.java:662)

"http-0.0.0.0-8443-14" daemon prio=10 tid=0x00000000417cd000 nid=0x7ab0 in Object.wait() [0x00007f9a59762000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f9a81b95590> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:458)
        - locked <0x00007f9a81b95590> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:484)
        at java.lang.Thread.run(Thread.java:662)

"http-0.0.0.0-8443-13" daemon prio=10 tid=0x00000000435c3800 nid=0x7aaf in Object.wait() [0x00007f9a59661000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f9a81b999d8> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:458)
        - locked <0x00007f9a81b999d8> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:484)
        at java.lang.Thread.run(Thread.java:662)

"http-0.0.0.0-8443-12" daemon prio=10 tid=0x00000000417d3000 nid=0x7aae in Object.wait() [0x00007f9a5935e000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f9a81b9de20> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:458)
        - locked <0x00007f9a81b9de20> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:484)
        at java.lang.Thread.run(Thread.java:662)

"http-0.0.0.0-8443-11" daemon prio=10 tid=0x00000000428fc000 nid=0x7aad in Object.wait() [0x00007f9a59863000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f9a81b9df00> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:458)
        - locked <0x00007f9a81b9df00> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:484)
        at java.lang.Thread.run(Thread.java:662)

"http-0.0.0.0-8443-10" daemon prio=10 tid=0x00000000414be000 nid=0x7aac in Object.wait() [0x00007f9a5a3e2000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f9a81b9dfe0> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:458)
        - locked <0x00007f9a81b9dfe0> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:484)
        at java.lang.Thread.run(Thread.java:662)

"http-0.0.0.0-8443-9" daemon prio=10 tid=0x0000000041a18000 nid=0x7a3a in Object.wait() [0x00007f9a5a6e5000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f9a81b6e760> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:458)
        - locked <0x00007f9a81b6e760> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:484)
        at java.lang.Thread.run(Thread.java:662)

"http-0.0.0.0-8443-8" daemon prio=10 tid=0x0000000042900000 nid=0x79a2 in Object.wait() [0x00007f9a59a65000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f9a81b3c418> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:458)
        - locked <0x00007f9a81b3c418> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:484)
        at java.lang.Thread.run(Thread.java:662)

"TP-Processor31" daemon prio=10 tid=0x0000000041f30800 nid=0x49e5 runnable [0x00007f9a59b64000]
   java.lang.Thread.State: RUNNABLE
        at java.lang.StringBuffer.append(StringBuffer.java:224)
        - locked <0x00007f9a843a95e0> (a java.lang.StringBuffer)
        at java.io.StringWriter.write(StringWriter.java:84)
        at java.io.StringWriter.append(StringWriter.java:126)
        at java.io.StringWriter.append(StringWriter.java:24)
        at org.ofbiz.base.util.UtilIO.copy(UtilIO.java:111)
        at org.ofbiz.content.data.DataResourceWorker.renderFile(DataResourceWorker.java:899)
        at org.ofbiz.content.data.DataResourceWorker.writeDataResourceText(DataResourceWorker.java:820)
        at org.ofbiz.content.data.DataResourceWorker.renderDataResourceAsText(DataResourceWorker.java:626)
        at org.ofbiz.content.content.ContentWorker.renderContentAsText(ContentWorker.java:273)
        at org.ofbiz.content.content.ContentWorker.renderContentAsText(ContentWorker.java:330)
        at org.ofbiz.content.cms.CmsEvents.cms(CmsEvents.java:289)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.ofbiz.webapp.event.JavaEventHandler.invoke(JavaEventHandler.java:92)
        at org.ofbiz.webapp.event.JavaEventHandler.invoke(JavaEventHandler.java:78)
        at org.ofbiz.webapp.control.RequestHandler.runEvent(RequestHandler.java:635)
        at org.ofbiz.webapp.control.RequestHandler.doRequest(RequestHandler.java:381)
        at org.ofbiz.webapp.control.ControlServlet.doGet(ControlServlet.java:230)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.ofbiz.webapp.control.ContextFilter.doFilter(ContextFilter.java:338)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
        at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:190)
        at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)
        at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:774)
        at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:703)
        at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:896)
        at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
        at java.lang.Thread.run(Thread.java:662)

"http-0.0.0.0-8080-6" daemon prio=10 tid=0x0000000041624000 nid=0x4651 in Object.wait() [0x00007f9a5b4f3000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f9a81395698> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:458)
        - locked <0x00007f9a81395698> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:484)
        at java.lang.Thread.run(Thread.java:662)

"http-0.0.0.0-8080-5" daemon prio=10 tid=0x00000000415d1000 nid=0x4650 in Object.wait() [0x00007f9a5a4e3000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f9a81396930> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:458)
        - locked <0x00007f9a81396930> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:484)
        at java.lang.Thread.run(Thread.java:662)

"http-0.0.0.0-8080-4" daemon prio=10 tid=0x0000000041f34800 nid=0x464f in Object.wait() [0x00007f9a5adec000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f9a81399108> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:458)
        - locked <0x00007f9a81399108> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:484)
        at java.lang.Thread.run(Thread.java:662)

"http-0.0.0.0-8080-3" daemon prio=10 tid=0x0000000041c5f000 nid=0x464e in Object.wait() [0x00007f9a5a5e4000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f9a8139a3a0> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:458)
        - locked <0x00007f9a8139a3a0> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:484)
        at java.lang.Thread.run(Thread.java:662)

"http-0.0.0.0-8443-7" daemon prio=10 tid=0x0000000042077000 nid=0x32a6 in Object.wait() [0x00007f9a5a8e7000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f9a8112a680> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:458)
        - locked <0x00007f9a8112a680> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:484)
        at java.lang.Thread.run(Thread.java:662)

"http-0.0.0.0-8080-2" daemon prio=10 tid=0x0000000042fe0000 nid=0xe5d in Object.wait() [0x00007f9a5a9e8000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f9a800ff450> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:458)
        - locked <0x00007f9a800ff450> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:484)
        at java.lang.Thread.run(Thread.java:662)

"TP-Processor28" daemon prio=10 tid=0x0000000043664800 nid=0x22b runnable [0x00007f9a5b5f2000]
   java.lang.Thread.State: RUNNABLE
        at java.io.Reader.read(Reader.java:83)
        at org.ofbiz.base.util.UtilIO.copy(UtilIO.java:109)
        at org.ofbiz.content.data.DataResourceWorker.renderFile(DataResourceWorker.java:899)
        at org.ofbiz.content.data.DataResourceWorker.writeDataResourceText(DataResourceWorker.java:820)
        at org.ofbiz.content.data.DataResourceWorker.renderDataResourceAsText(DataResourceWorker.java:626)
        at org.ofbiz.content.content.ContentWorker.renderContentAsText(ContentWorker.java:273)
        at org.ofbiz.content.content.ContentWorker.renderContentAsText(ContentWorker.java:330)
        at org.ofbiz.content.cms.CmsEvents.cms(CmsEvents.java:289)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.ofbiz.webapp.event.JavaEventHandler.invoke(JavaEventHandler.java:92)
        at org.ofbiz.webapp.event.JavaEventHandler.invoke(JavaEventHandler.java:78)
        at org.ofbiz.webapp.control.RequestHandler.runEvent(RequestHandler.java:635)
        at org.ofbiz.webapp.control.RequestHandler.doRequest(RequestHandler.java:381)
        at org.ofbiz.webapp.control.ControlServlet.doGet(ControlServlet.java:230)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.ofbiz.webapp.control.ContextFilter.doFilter(ContextFilter.java:338)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
        at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:190)
        at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)
        at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:774)
        at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:703)
        at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:896)
        at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
        at java.lang.Thread.run(Thread.java:662)

"TP-Processor23" daemon prio=10 tid=0x000000004222e800 nid=0x157 runnable [0x00007f9a5ace9000]
   java.lang.Thread.State: RUNNABLE
        at java.lang.StringBuffer.append(StringBuffer.java:224)
        - locked <0x00007f9a80f38480> (a java.lang.StringBuffer)
        at java.io.StringWriter.write(StringWriter.java:84)
        at java.io.StringWriter.append(StringWriter.java:126)
        at java.io.StringWriter.append(StringWriter.java:24)
        at org.ofbiz.base.util.UtilIO.copy(UtilIO.java:111)
        at org.ofbiz.content.data.DataResourceWorker.renderFile(DataResourceWorker.java:899)
        at org.ofbiz.content.data.DataResourceWorker.writeDataResourceText(DataResourceWorker.java:820)
        at org.ofbiz.content.data.DataResourceWorker.renderDataResourceAsText(DataResourceWorker.java:626)
        at org.ofbiz.content.content.ContentWorker.renderContentAsText(ContentWorker.java:273)
        at org.ofbiz.content.content.ContentWorker.renderContentAsText(ContentWorker.java:330)
        at org.ofbiz.content.cms.CmsEvents.cms(CmsEvents.java:289)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.ofbiz.webapp.event.JavaEventHandler.invoke(JavaEventHandler.java:92)
        at org.ofbiz.webapp.event.JavaEventHandler.invoke(JavaEventHandler.java:78)
        at org.ofbiz.webapp.control.RequestHandler.runEvent(RequestHandler.java:635)
        at org.ofbiz.webapp.control.RequestHandler.doRequest(RequestHandler.java:381)
        at org.ofbiz.webapp.control.ControlServlet.doGet(ControlServlet.java:230)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.ofbiz.webapp.control.ContextFilter.doFilter(ContextFilter.java:338)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
        at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:190)
        at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)
        at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:774)
        at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:703)
        at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:896)
        at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
        at java.lang.Thread.run(Thread.java:662)

"TP-Processor22" daemon prio=10 tid=0x000000004222d800 nid=0x156 runnable [0x00007f9a5bcf9000]
   java.lang.Thread.State: RUNNABLE
        at java.lang.StringBuffer.append(StringBuffer.java:224)
        - locked <0x00007f9a7ffad3f8> (a java.lang.StringBuffer)
        at java.io.StringWriter.write(StringWriter.java:84)
        at java.io.StringWriter.append(StringWriter.java:126)
        at java.io.StringWriter.append(StringWriter.java:24)
        at org.ofbiz.base.util.UtilIO.copy(UtilIO.java:111)
        at org.ofbiz.content.data.DataResourceWorker.renderFile(DataResourceWorker.java:899)
        at org.ofbiz.content.data.DataResourceWorker.writeDataResourceText(DataResourceWorker.java:820)
        at org.ofbiz.content.data.DataResourceWorker.renderDataResourceAsText(DataResourceWorker.java:626)
        at org.ofbiz.content.content.ContentWorker.renderContentAsText(ContentWorker.java:273)
        at org.ofbiz.content.content.ContentWorker.renderContentAsText(ContentWorker.java:330)
        at org.ofbiz.content.cms.CmsEvents.cms(CmsEvents.java:289)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.ofbiz.webapp.event.JavaEventHandler.invoke(JavaEventHandler.java:92)
        at org.ofbiz.webapp.event.JavaEventHandler.invoke(JavaEventHandler.java:78)
        at org.ofbiz.webapp.control.RequestHandler.runEvent(RequestHandler.java:635)
        at org.ofbiz.webapp.control.RequestHandler.doRequest(RequestHandler.java:381)
        at org.ofbiz.webapp.control.ControlServlet.doGet(ControlServlet.java:230)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.ofbiz.webapp.control.ContextFilter.doFilter(ContextFilter.java:338)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
        at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:190)
        at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)
        at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:774)
        at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:703)
        at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:896)
        at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
        at java.lang.Thread.run(Thread.java:662)

"http-0.0.0.0-8443-6" daemon prio=10 tid=0x0000000042dbc000 nid=0x7970 in Object.wait() [0x00007f9a5b7f6000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f9a7f2bc138> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:458)
        - locked <0x00007f9a7f2bc138> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:484)
        at java.lang.Thread.run(Thread.java:662)

"http-0.0.0.0-8443-4" daemon prio=10 tid=0x00000000418e2000 nid=0x796e in Object.wait() [0x00007f9a5b9f8000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f9a7f2bd4b0> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:458)
        - locked <0x00007f9a7f2bd4b0> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:484)
        at java.lang.Thread.run(Thread.java:662)

"http-0.0.0.0-8443-3" daemon prio=10 tid=0x0000000042db9000 nid=0x796d in Object.wait() [0x00007f9a5bdfc000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f9a7f2be748> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:458)
        - locked <0x00007f9a7f2be748> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:484)
        at java.lang.Thread.run(Thread.java:662)

"http-0.0.0.0-8443-2" daemon prio=10 tid=0x00000000418df800 nid=0x796c in Object.wait() [0x00007f9a5b6f5000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f9a7f2bf9e0> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:458)
        - locked <0x00007f9a7f2bf9e0> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:484)
        at java.lang.Thread.run(Thread.java:662)

"TP-Processor16" daemon prio=10 tid=0x000000004149f800 nid=0x7932 runnable [0x00007f9a60129000]
   java.lang.Thread.State: RUNNABLE
        at java.lang.StringBuffer.append(StringBuffer.java:224)
        - locked <0x00007f9a8071f7d8> (a java.lang.StringBuffer)
        at java.io.StringWriter.write(StringWriter.java:84)
        at java.io.StringWriter.append(StringWriter.java:126)
        at java.io.StringWriter.append(StringWriter.java:24)
        at org.ofbiz.base.util.UtilIO.copy(UtilIO.java:111)
        at org.ofbiz.content.data.DataResourceWorker.renderFile(DataResourceWorker.java:899)
        at org.ofbiz.content.data.DataResourceWorker.writeDataResourceText(DataResourceWorker.java:820)
        at org.ofbiz.content.data.DataResourceWorker.renderDataResourceAsText(DataResourceWorker.java:626)
        at org.ofbiz.content.content.ContentWorker.renderContentAsText(ContentWorker.java:273)
        at org.ofbiz.content.content.ContentWorker.renderContentAsText(ContentWorker.java:330)
        at org.ofbiz.content.cms.CmsEvents.cms(CmsEvents.java:289)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.ofbiz.webapp.event.JavaEventHandler.invoke(JavaEventHandler.java:92)
        at org.ofbiz.webapp.event.JavaEventHandler.invoke(JavaEventHandler.java:78)
        at org.ofbiz.webapp.control.RequestHandler.runEvent(RequestHandler.java:635)
        at org.ofbiz.webapp.control.RequestHandler.doRequest(RequestHandler.java:381)
        at org.ofbiz.webapp.control.ControlServlet.doGet(ControlServlet.java:230)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.ofbiz.webapp.control.ContextFilter.doFilter(ContextFilter.java:338)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
        at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:190)
        at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)
        at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:774)
        at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:703)
        at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:896)
        at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
        at java.lang.Thread.run(Thread.java:662)

"TP-Processor13" daemon prio=10 tid=0x0000000041616000 nid=0x792f runnable [0x00007f9a5bbf8000]
   java.lang.Thread.State: RUNNABLE
        at java.lang.StringBuffer.append(StringBuffer.java:224)
        - locked <0x00007f9a8083eef8> (a java.lang.StringBuffer)
        at java.io.StringWriter.write(StringWriter.java:84)
        at java.io.StringWriter.append(StringWriter.java:126)
        at java.io.StringWriter.append(StringWriter.java:24)
        at org.ofbiz.base.util.UtilIO.copy(UtilIO.java:111)
        at org.ofbiz.content.data.DataResourceWorker.renderFile(DataResourceWorker.java:899)
        at org.ofbiz.content.data.DataResourceWorker.writeDataResourceText(DataResourceWorker.java:820)
        at org.ofbiz.content.data.DataResourceWorker.renderDataResourceAsText(DataResourceWorker.java:626)
        at org.ofbiz.content.content.ContentWorker.renderContentAsText(ContentWorker.java:273)
        at org.ofbiz.content.content.ContentWorker.renderContentAsText(ContentWorker.java:330)
        at org.ofbiz.content.cms.CmsEvents.cms(CmsEvents.java:289)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.ofbiz.webapp.event.JavaEventHandler.invoke(JavaEventHandler.java:92)
        at org.ofbiz.webapp.event.JavaEventHandler.invoke(JavaEventHandler.java:78)
        at org.ofbiz.webapp.control.RequestHandler.runEvent(RequestHandler.java:635)
        at org.ofbiz.webapp.control.RequestHandler.doRequest(RequestHandler.java:381)
        at org.ofbiz.webapp.control.ControlServlet.doGet(ControlServlet.java:230)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.ofbiz.webapp.control.ContextFilter.doFilter(ContextFilter.java:338)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
        at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:190)
        at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)
        at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:774)
        at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:703)
        at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:896)
        at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
        at java.lang.Thread.run(Thread.java:662)

"TP-Processor10" daemon prio=10 tid=0x0000000041dd7000 nid=0x77ff runnable [0x00007f9a60bd3000]
   java.lang.Thread.State: RUNNABLE
        at java.lang.StringBuffer.append(StringBuffer.java:224)
        - locked <0x00007f9a82819a20> (a java.lang.StringBuffer)
        at java.io.StringWriter.write(StringWriter.java:84)
        at java.io.StringWriter.append(StringWriter.java:126)
        at java.io.StringWriter.append(StringWriter.java:24)
        at org.ofbiz.base.util.UtilIO.copy(UtilIO.java:111)
        at org.ofbiz.content.data.DataResourceWorker.renderFile(DataResourceWorker.java:899)
        at org.ofbiz.content.data.DataResourceWorker.writeDataResourceText(DataResourceWorker.java:820)
        at org.ofbiz.content.data.DataResourceWorker.renderDataResourceAsText(DataResourceWorker.java:626)
        at org.ofbiz.content.content.ContentWorker.renderContentAsText(ContentWorker.java:273)
        at org.ofbiz.content.content.ContentWorker.renderContentAsText(ContentWorker.java:330)
        at org.ofbiz.content.cms.CmsEvents.cms(CmsEvents.java:289)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.ofbiz.webapp.event.JavaEventHandler.invoke(JavaEventHandler.java:92)
        at org.ofbiz.webapp.event.JavaEventHandler.invoke(JavaEventHandler.java:78)
        at org.ofbiz.webapp.control.RequestHandler.runEvent(RequestHandler.java:635)
        at org.ofbiz.webapp.control.RequestHandler.doRequest(RequestHandler.java:381)
        at org.ofbiz.webapp.control.ControlServlet.doGet(ControlServlet.java:230)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.ofbiz.webapp.control.ContextFilter.doFilter(ContextFilter.java:338)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
        at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:190)
        at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)
        at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:774)
        at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:703)
        at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:896)
        at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
        at java.lang.Thread.run(Thread.java:662)

"Java2D Disposer" daemon prio=10 tid=0x00007f9a6608b800 nid=0x7765 in Object.wait() [0x00007f9a629f3000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f9a7ec9dfa0> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
        - locked <0x00007f9a7ec9dfa0> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
        at sun.java2d.Disposer.run(Disposer.java:127)
        at java.lang.Thread.run(Thread.java:662)

"http-0.0.0.0-8443-1" daemon prio=10 tid=0x000000004197b800 nid=0x76e5 in Object.wait() [0x00007f9a60dd7000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f9a7ec3a260> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:458)
        - locked <0x00007f9a7ec3a260> (a org.apache.tomcat.util.net.JIoEndpoint$Worker)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:484)
        at java.lang.Thread.run(Thread.java:662)

"TP-Processor7" daemon prio=10 tid=0x0000000042c42000 nid=0x76cf runnable [0x00007f9a625ed000]
   java.lang.Thread.State: RUNNABLE
        at java.lang.StringBuffer.append(StringBuffer.java:224)
        - locked <0x00007f9a82a058f8> (a java.lang.StringBuffer)
        at java.io.StringWriter.write(StringWriter.java:84)
        at java.io.StringWriter.append(StringWriter.java:126)
        at java.io.StringWriter.append(StringWriter.java:24)
        at org.ofbiz.base.util.UtilIO.copy(UtilIO.java:111)
        at org.ofbiz.content.data.DataResourceWorker.renderFile(DataResourceWorker.java:899)
        at org.ofbiz.content.data.DataResourceWorker.writeDataResourceText(DataResourceWorker.java:820)
        at org.ofbiz.content.data.DataResourceWorker.renderDataResourceAsText(DataResourceWorker.java:626)
        at org.ofbiz.content.content.ContentWorker.renderContentAsText(ContentWorker.java:273)
        at org.ofbiz.content.content.ContentWorker.renderContentAsText(ContentWorker.java:330)
        at org.ofbiz.content.cms.CmsEvents.cms(CmsEvents.java:289)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.ofbiz.webapp.event.JavaEventHandler.invoke(JavaEventHandler.java:92)
        at org.ofbiz.webapp.event.JavaEventHandler.invoke(JavaEventHandler.java:78)
        at org.ofbiz.webapp.control.RequestHandler.runEvent(RequestHandler.java:635)
        at org.ofbiz.webapp.control.RequestHandler.doRequest(RequestHandler.java:381)
        at org.ofbiz.webapp.control.ControlServlet.doGet(ControlServlet.java:230)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.ofbiz.webapp.control.ContextFilter.doFilter(ContextFilter.java:338)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
        at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:190)
        at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)
        at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:774)
        at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:703)
        at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:896)
        at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
        at java.lang.Thread.run(Thread.java:662)

"DestroyJavaVM" prio=10 tid=0x00007f9a661e8000 nid=0x741d waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Thread-205" prio=10 tid=0x00007f9a66543800 nid=0x7546 runnable [0x00007f9a60fd9000]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
        - locked <0x00007f9a7e2f3600> (a java.net.SocksSocketImpl)
        at java.net.ServerSocket.implAccept(ServerSocket.java:453)
        at java.net.ServerSocket.accept(ServerSocket.java:421)
        at bsh.util.Sessiond.run(Unknown Source)
        at java.lang.Thread.run(Thread.java:662)

"Thread-203" prio=10 tid=0x00007f9a678d4800 nid=0x7545 runnable [0x00007f9a610da000]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
        - locked <0x00007f9a7e2f3ab0> (a java.net.SocksSocketImpl)
        at java.net.ServerSocket.implAccept(ServerSocket.java:453)
        at java.net.ServerSocket.accept(ServerSocket.java:421)
        at bsh.util.Httpd.run(Unknown Source)
        at java.lang.Thread.run(Thread.java:662)

"Worker-JM" prio=10 tid=0x00000000424d5800 nid=0x7534 in Object.wait() [0x00007f9a611db000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f9a7e1bbc28> (a java.util.ArrayList)
        at org.eclipse.core.internal.jobs.InternalWorker.run(InternalWorker.java:58)
        - locked <0x00007f9a7e1bbc28> (a java.util.ArrayList)

"Bundle File Closer" daemon prio=10 tid=0x0000000041fc2000 nid=0x7532 in Object.wait() [0x00007f9a612dc000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f9a7e182818> (a org.eclipse.osgi.framework.eventmgr.EventManager$EventThread)
        at java.lang.Object.wait(Object.java:485)
        at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.getNextEvent(EventManager.java:397)
        - locked <0x00007f9a7e182818> (a org.eclipse.osgi.framework.eventmgr.EventManager$EventThread)
        at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run(EventManager.java:333)

"Start Level Event Dispatcher" daemon prio=10 tid=0x00007f9a64620000 nid=0x752e in Object.wait() [0x00007f9a613dd000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f9a7e1c5830> (a org.eclipse.osgi.framework.eventmgr.EventManager$EventThread)
        at java.lang.Object.wait(Object.java:485)
        at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.getNextEvent(EventManager.java:397)
        - locked <0x00007f9a7e1c5830> (a org.eclipse.osgi.framework.eventmgr.EventManager$EventThread)
        at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run(EventManager.java:333)

"Framework Event Dispatcher" daemon prio=10 tid=0x00007f9a5c617800 nid=0x752d in Object.wait() [0x00007f9a614de000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f9a7e1c5b28> (a org.eclipse.osgi.framework.eventmgr.EventManager$EventThread)
        at java.lang.Object.wait(Object.java:485)
        at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.getNextEvent(EventManager.java:397)
        - locked <0x00007f9a7e1c5b28> (a org.eclipse.osgi.framework.eventmgr.EventManager$EventThread)
        at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run(EventManager.java:333)

"TP-Processor2" daemon prio=10 tid=0x00007f9a66744000 nid=0x7515 runnable [0x00007f9a6350c000]
   java.lang.Thread.State: RUNNABLE
        at java.io.Reader.read(Reader.java:83)
        at org.ofbiz.base.util.UtilIO.copy(UtilIO.java:109)
        at org.ofbiz.content.data.DataResourceWorker.renderFile(DataResourceWorker.java:899)
        at org.ofbiz.content.data.DataResourceWorker.writeDataResourceText(DataResourceWorker.java:820)
        at org.ofbiz.content.data.DataResourceWorker.renderDataResourceAsText(DataResourceWorker.java:626)
        at org.ofbiz.content.content.ContentWorker.renderContentAsText(ContentWorker.java:273)
        at org.ofbiz.content.content.ContentWorker.renderContentAsText(ContentWorker.java:330)
        at org.ofbiz.content.cms.CmsEvents.cms(CmsEvents.java:289)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.ofbiz.webapp.event.JavaEventHandler.invoke(JavaEventHandler.java:92)
        at org.ofbiz.webapp.event.JavaEventHandler.invoke(JavaEventHandler.java:78)
        at org.ofbiz.webapp.control.RequestHandler.runEvent(RequestHandler.java:635)
        at org.ofbiz.webapp.control.RequestHandler.doRequest(RequestHandler.java:381)
        at org.ofbiz.webapp.control.ControlServlet.doGet(ControlServlet.java:230)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.ofbiz.webapp.control.ContextFilter.doFilter(ContextFilter.java:338)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
        at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:190)
        at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)
        at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:774)
        at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:703)
        at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:896)
        at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
        at java.lang.Thread.run(Thread.java:662)

"TTLObject(async-update)-1" daemon prio=10 tid=0x00007f9a67327000 nid=0x7497 waiting on condition [0x00007f9a61de7000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00007f9a7db5d4a0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:662)

"TTLObject(async-update)-0" daemon prio=10 tid=0x00007f9a65cf1800 nid=0x7496 waiting on condition [0x00007f9a61ce6000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00007f9a7db5d4a0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:662)

"ThreadPool-1" daemon prio=10 tid=0x00000000418da000 nid=0x7495 waiting on condition [0x00007f9a621eb000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00007f9a7db5d7d0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:662)

"ThreadPool-0" daemon prio=10 tid=0x00000000415dc800 nid=0x7494 waiting on condition [0x00007f9a61be5000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00007f9a7db5d7d0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:662)

"ReferenceCleaner" daemon prio=10 tid=0x0000000041c58800 nid=0x748e in Object.wait() [0x00007f9a620ea000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f9a7db5db00> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
        - locked <0x00007f9a7db5db00> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
        at org.webslinger.lang.ReferenceCleaner.run(ReferenceCleaner.java:93)
        at org.webslinger.lang.ReferenceCleaner.access$100(ReferenceCleaner.java:10)
        at org.webslinger.lang.ReferenceCleaner$ReferenceCleanerBootstrapper$1.run(ReferenceCleaner.java:74)
        at java.lang.Thread.run(Thread.java:662)

"TTLObject(0) pulse" daemon prio=10 tid=0x00000000423a6800 nid=0x748a waiting on condition [0x00007f9a623ed000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00007f9a7db5dc00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at org.webslinger.concurrent.TTLObject$TTLObjectWorker.run(TTLObject.java:27)
        at java.lang.Thread.run(Thread.java:662)

"ReferenceCleaner" daemon prio=10 tid=0x00000000428f1000 nid=0x7473 in Object.wait() [0x00007f9a622ec000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f9a7da453f0> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
        - locked <0x00007f9a7da453f0> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
        at org.ofbiz.base.util.ReferenceCleaner$CleanerThread.run(ReferenceCleaner.java:45)

"ExecutionPoolPulseWorker(0)" daemon prio=10 tid=0x00007f9a646d5800 nid=0x745a waiting on condition [0x00007f9a624ee000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00007f9a7d2034b0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
        at org.ofbiz.base.concurrent.ExecutionPool$ExecutionPoolPulseWorker.run(ExecutionPool.java:106)
        at java.lang.Thread.run(Thread.java:662)

"RMI RenewClean-[140.211.11.41:42426,org.ofbiz.service.rmi.socket.ssl.SSLClientSocketFactory@2ebf13d5]" daemon prio=10 tid=0x0000000041c75000 nid=0x7457 in Object.wait() [0x00007f9a626f0000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f9a7d0f7788> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
        - locked <0x00007f9a7d0f7788> (a java.lang.ref.ReferenceQueue$Lock)
        at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:516)
        at java.lang.Thread.run(Thread.java:662)

"RMI Scheduler(0)" daemon prio=10 tid=0x00007f9a653c7800 nid=0x7456 waiting on condition [0x00007f9a627f1000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00007f9a7d0f78f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:662)

"GC Daemon" daemon prio=10 tid=0x00007f9a5c52a800 nid=0x7454 in Object.wait() [0x00007f9a62af4000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f9a7d0f7b38> (a sun.misc.GC$LatencyLock)
        at sun.misc.GC$Daemon.run(GC.java:100)
        - locked <0x00007f9a7d0f7b38> (a sun.misc.GC$LatencyLock)

"RMI Reaper" prio=10 tid=0x00007f9a66dde800 nid=0x7453 in Object.wait() [0x00007f9a62bf5000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f9a7d0f7bf8> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
        - locked <0x00007f9a7d0f7bf8> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
        at sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:333)
        at java.lang.Thread.run(Thread.java:662)

"RMI TCP Accept-0" daemon prio=10 tid=0x00007f9a65717000 nid=0x7452 runnable [0x00007f9a62cf6000]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
        - locked <0x00007f9a7d0f7d08> (a java.net.SocksSocketImpl)
        at java.net.ServerSocket.implAccept(ServerSocket.java:453)
        at com.sun.net.ssl.internal.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:261)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
        at java.lang.Thread.run(Thread.java:662)

"org.ofbiz.service.job.JobPoller@6c1c19c6" prio=10 tid=0x0000000042be7000 nid=0x744e runnable [0x00007f9a62df7000]
   java.lang.Thread.State: RUNNABLE
        at org.apache.log4j.AsyncAppender.append(AsyncAppender.java:223)
        at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
        - locked <0x00007f9a797edb80> (a org.apache.log4j.AsyncAppender)
        at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
        at org.apache.log4j.Category.callAppenders(Category.java:206)
        - locked <0x00007f9a797406e0> (a org.apache.log4j.spi.RootLogger)
        at org.apache.log4j.Category.forcedLog(Category.java:391)
        at org.apache.log4j.Category.log(Category.java:856)
        at org.ofbiz.base.util.Debug.log(Debug.java:166)
        at org.ofbiz.base.util.Debug.log(Debug.java:149)
        at org.ofbiz.base.util.Debug.logError(Debug.java:314)
        at org.ofbiz.entity.GenericDelegator.storeByCondition(GenericDelegator.java:1293)
        at org.ofbiz.entity.GenericDelegator.storeByCondition(GenericDelegator.java:1257)
        at org.ofbiz.service.job.JobManager.poll(JobManager.java:157)
        - locked <0x00007f9a7b864520> (a org.ofbiz.service.job.JobManager)
        - locked <0x00007f9a7b864520> (a org.ofbiz.service.job.JobManager)
        at org.ofbiz.service.job.JobPoller.run(JobPoller.java:90)
        - locked <0x00007f9a7b85c368> (a org.ofbiz.service.job.JobPoller)
        at java.lang.Thread.run(Thread.java:662)

"Thread-5" daemon prio=10 tid=0x0000000041553000 nid=0x7445 waiting on condition [0x00007f9a63710000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.apache.geronimo.transaction.manager.TransactionTimer$CurrentTime.run(TransactionTimer.java:38)

"derby.rawStoreDaemon" daemon prio=10 tid=0x00000000419e7800 nid=0x7443 in Object.wait() [0x00007f9a639f2000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f9a7b575950> (a org.apache.derby.impl.services.daemon.BasicDaemon)
        at org.apache.derby.impl.services.daemon.BasicDaemon.rest(Unknown Source)
        - locked <0x00007f9a7b575950> (a org.apache.derby.impl.services.daemon.BasicDaemon)
        at org.apache.derby.impl.services.daemon.BasicDaemon.run(Unknown Source)
        at java.lang.Thread.run(Thread.java:662)

"derby.rawStoreDaemon" daemon prio=10 tid=0x0000000041a87000 nid=0x7441 in Object.wait() [0x00007f9a63af3000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f9a7b575a58> (a org.apache.derby.impl.services.daemon.BasicDaemon)
        at org.apache.derby.impl.services.daemon.BasicDaemon.rest(Unknown Source)
        - locked <0x00007f9a7b575a58> (a org.apache.derby.impl.services.daemon.BasicDaemon)
        at org.apache.derby.impl.services.daemon.BasicDaemon.run(Unknown Source)
        at java.lang.Thread.run(Thread.java:662)

"derby.rawStoreDaemon" daemon prio=10 tid=0x00000000419ef800 nid=0x743b in Object.wait() [0x00007f9a63bf4000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f9a7b33d748> (a org.apache.derby.impl.services.daemon.BasicDaemon)
        at org.apache.derby.impl.services.daemon.BasicDaemon.rest(Unknown Source)
        - locked <0x00007f9a7b33d748> (a org.apache.derby.impl.services.daemon.BasicDaemon)
        at org.apache.derby.impl.services.daemon.BasicDaemon.run(Unknown Source)
        at java.lang.Thread.run(Thread.java:662)

"Timer-1" daemon prio=10 tid=0x00000000417a2800 nid=0x743a in Object.wait() [0x00007f9a63cf5000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f9a7b33d860> (a java.util.TaskQueue)
        at java.util.TimerThread.mainLoop(Timer.java:509)
        - locked <0x00007f9a7b33d860> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:462)

"Timer-0" daemon prio=10 tid=0x0000000041871800 nid=0x7439 in Object.wait() [0x00007f9a63efd000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f9a7b33d988> (a java.util.TaskQueue)
        at java.lang.Object.wait(Object.java:485)
        at java.util.TimerThread.mainLoop(Timer.java:483)
        - locked <0x00007f9a7b33d988> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:462)

"derby.antiGC" daemon prio=10 tid=0x00000000416fd000 nid=0x7437 in Object.wait() [0x00007f9a63ffe000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f9a7b33da58> (a org.apache.derby.impl.services.monitor.AntiGC)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.derby.impl.services.monitor.AntiGC.run(Unknown Source)
        - locked <0x00007f9a7b33da58> (a org.apache.derby.impl.services.monitor.AntiGC)
        at java.lang.Thread.run(Thread.java:662)

"Dispatcher-Thread-2" daemon prio=10 tid=0x00007f9a64490800 nid=0x7428 in Object.wait() [0x00007f9a684c5000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f9a797201b8> (a java.util.ArrayList)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.log4j.AsyncAppender$Dispatcher.run(AsyncAppender.java:545)
        - locked <0x00007f9a797201b8> (a java.util.ArrayList)
        at java.lang.Thread.run(Thread.java:662)

"org.ofbiz.base.start.Start@576a9d42" prio=10 tid=0x00007f9a64004000 nid=0x7426 runnable [0x00007f9a688ab000]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
        - locked <0x00007f9a797202d8> (a java.net.SocksSocketImpl)
        at java.net.ServerSocket.implAccept(ServerSocket.java:453)
        at java.net.ServerSocket.accept(ServerSocket.java:421)
        at org.ofbiz.base.start.Start.run(Start.java:114)
        at java.lang.Thread.run(Thread.java:662)

"Low Memory Detector" daemon prio=10 tid=0x0000000041415800 nid=0x7424 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"CompilerThread1" daemon prio=10 tid=0x0000000041410000 nid=0x7423 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"CompilerThread0" daemon prio=10 tid=0x000000004140e000 nid=0x7422 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=10 tid=0x000000004140c000 nid=0x7421 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=10 tid=0x00000000413e6800 nid=0x7420 in Object.wait() [0x00007f9a6900e000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f9a79720920> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
        - locked <0x00007f9a79720920> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

"Reference Handler" daemon prio=10 tid=0x00000000413e4800 nid=0x741f in Object.wait() [0x00007f9a6910f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f9a79720a08> (a java.lang.ref.Reference$Lock)
        at java.lang.Object.wait(Object.java:485)
        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
        - locked <0x00007f9a79720a08> (a java.lang.ref.Reference$Lock)

"VM Thread" prio=10 tid=0x00000000413e0000 nid=0x741e runnable

"VM Periodic Task Thread" prio=10 tid=0x0000000041420800 nid=0x7425 waiting on condition

JNI global references: 1821
{code}


> Memory leaks in (at least) stable demo
> --------------------------------------
>
>                 Key: OFBIZ-4043
>                 URL: https://issues.apache.org/jira/browse/OFBIZ-4043
>             Project: OFBiz
>          Issue Type: Bug
>    Affects Versions: Release Branch 09.04
>         Environment: On OFBiz demo stable
>            Reporter: Jacques Le Roux
>         Attachments: Leaks Suspect .jpg
>
>
> Using [Mat|http://www.eclipse.org/mat/]
> I generated these report
> 110 542 instances of "org.apache.derby.impl.jdbc.EmbedConnection40", loaded by "java.net.URLClassLoader @ 0x7f3681c111f0" occupy 415 680 440 (79,89%) bytes. These instances are referenced from one instance of "java.util.WeakHashMap$Entry[]", loaded by "<system class loader>"
> Keywords
> java.net.URLClassLoader @ 0x7f3681c111f0
> org.apache.derby.impl.jdbc.EmbedConnection40
> java.util.WeakHashMap$Entry[]
> It's not quite clear to me how this happends, but I'm quite sure it happens in stable demo and maybe also in trunk. I have also attached a screen-copy with more information

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.