You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by balvindar dhaliwal <ba...@hotmail.com> on 2012/09/03 15:12:33 UTC
Apache tomcat ( 7.0.19 ) stops processing user requests suddenly
but works fine after restart.
Hi ,
We are using tomcat 7.0.19, for a jsf(richfaces),seam, jpa web application.
Here are the software versions and hardware infrastructure details.
Apache Tomact # 7.0.19
Java run time # 1.6.0_12
Server infrastructure # SunOS xxxxxxx 5.10 Generic_142900-03 sun4v sparc SUNW,Sun-Fire-T200
OS version # solaris 10.
Issue Description #
We have noticed after about 2 months, the web requests stop being processed. We had observed this issue twice in the last 5 months.
We are using HTTP/1.1 Connector ( BIO ). The heap and permgen memory seem fine. There is no out of memory errors in the logs. We have used jvisualvm to monitor heap, permgen, threads and thread dump, AND THEY ALL LOOK OK. The thread dump looks fine and there are no deadlocks/stuck threads observed.
The localhost_access_log stops being updated. The connector port is still listening on SSL port 8443.
After we restart Tomcat, the new requests are processing fine.
Can you please advise other areas we should look into for investigation (any additional tracing).
Should we upgrade to tomcat 7.0.29? What could cause the Connector to stop processing requests on the port?
Further info: Heap(1/6th used), PermGen(1/2 used). GC is effectively reducing the heap. The fact that local access log is not being updated, suggests the request is not making to the Servlet container. We are using JPA Connection Pool, have not noticed any db connection issues. After the restart, everything works well.
The system logs seems to be fine.
Please find the attached sample thread dump copy when this issue was occurred.
Please let me know if you need more information.
Thanks
Subba.
Re: Apache tomcat ( 7.0.19 ) stops processing user requests suddenly
but works fine after restart.
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Balvindar,
On 9/3/12 9:12 AM, balvindar dhaliwal wrote:
> We are using tomcat 7.0.19, for a jsf(richfaces),seam, jpa web
> application. Here are the software versions and hardware
> infrastructure details. Apache Tomact # 7.0.19 Java run time #
> 1.6.0_12
If possible, it would be best to upgrade to the latest version of both
of these packages: Java 1.6.0_36 and Tomcat 7.0.30. There are several
important security fixes in both of those (relative to where you are now).
> We have noticed after about 2 months, the web requests stop being
> processed. We had observed this issue twice in the last 5 months.
> We are using HTTP/1.1 Connector ( BIO ). The heap and permgen
> memory seem fine. There is no out of memory errors in the logs. We
> have used jvisualvm to monitor heap, permgen, threads and thread
> dump, AND THEY ALL LOOK OK. The thread dump looks fine and there
> are no deadlocks/stuck threads observed.
Your thread dump attachment was stripped from the list. Can you
copy/paste it into a follow post?
> Please let me know if you need more information.
It's hard to guess without the thread dump, but my initial reaction is
that it's connection-pool exhaustion, but you said that your
connection pool seems healthy.
I see from your server.xml that you are using UserDatabaseRealm: is
that actually in use in production? It's probably an inappropriate
component to use for anything but the most trivial configuration (like
maybe the Tomcat manager app). Are you using any kind of <Realm> or
<Resource> specifically configured in in your webapp's context.xml?
That would also be good to know.
If you are using JDBCRealm, you may have hit an edge case that causes
some kind of deadlock (but again, you say there are no deadlocks).
There were some fixes to the NIO and APR connectors recently (and your
version of Tomcat is now more than a year out-of-date) that fixed
apparent deadlocks but I'm not sure if those extended into the BIO
realm. I can't see anything in the changelog that would indicate a fix
to the BIO connector that would affect you.
Again, it's very difficult to speculate without some more data, so
sending that thread dump (especially if you can give us several thread
dumps over a few minutes) will definitely help.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/
iEYEARECAAYFAlBKEXEACgkQ9CaO5/Lv0PA4dACgjDfHeAMIJClgb9LrwH/Lv2Oy
wIsAoKsu4c/ilXoEppKUX2YB2GgsMS1R
=X/Hd
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Apache tomcat ( 7.0.19 ) stops processing user requests
suddenly but works fine after restart.
Posted by balvindar dhaliwal <ba...@hotmail.com>.
Hi,
The issue seems similar to:
http://www.tomcatexpert.com/ask-the-experts/tomcat-hangs-production-and-doesn%E2%80%99t-respond-new-http-requests
https://issues.apache.org/bugzilla/show_bug.cgi?id=53173
Regards,Bal.
From: balvindard@hotmail.com
To: users@tomcat.apache.org
Subject: RE: Apache tomcat ( 7.0.19 ) stops processing user requests suddenly but works fine after restart.
Date: Tue, 11 Sep 2012 15:04:45 +0000
Hi,
I have pasted the first thread dump, and attached 3 other thread dumps.
Thread dump1:-------------------
2012-08-10 14:40:52
Full thread dump Java HotSpot(TM) Server VM (11.2-b01 mixed mode):
"RMI TCP Connection(23136)-149.184.142.160" - Thread t@37810
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
- locked java.io.BufferedInputStream@427137
at java.io.FilterInputStream.read(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- locked java.util.concurrent.locks.ReentrantLock$NonfairSync@17119c
"JMX server connection timeout 37809" - Thread t@37809
java.lang.Thread.State: TIMED_WAITING on [I@378cf5
at java.lang.Object.wait(Native Method)
at com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"JMX server connection timeout 37808" - Thread t@37808
java.lang.Thread.State: TIMED_WAITING on [I@1075926
at java.lang.Object.wait(Native Method)
at com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"RMI TCP Connection(23129)-149.177.35.85" - Thread t@37806
java.lang.Thread.State: RUNNABLE
at sun.management.ThreadImpl.dumpThreads0(Native Method)
at sun.management.ThreadImpl.dumpAllThreads(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(Unknown Source)
at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(Unknown Source)
at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(Unknown Source)
at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(Unknown Source)
at com.sun.jmx.mbeanserver.PerInterface.invoke(Unknown Source)
at com.sun.jmx.mbeanserver.MBeanSupport.invoke(Unknown Source)
at javax.management.StandardMBean.invoke(Unknown Source)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Unknown Source)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(Unknown Source)
at com.sun.jmx.remote.security.MBeanServerAccessController.invoke(Unknown Source)
at javax.management.remote.rmi.RMIConnectionImpl.doOperation(Unknown Source)
at javax.management.remote.rmi.RMIConnectionImpl.access$200(Unknown Source)
at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(Unknown Source)
at javax.management.remote.rmi.RMIConnectionImpl.invoke(Unknown Source)
at sun.reflect.GeneratedMethodAccessor5299.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source)
at sun.rmi.transport.Transport$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- locked java.util.concurrent.locks.ReentrantLock$NonfairSync@d0793c
"RMI TCP Connection(23134)-149.184.142.160" - Thread t@37803
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
- locked java.io.BufferedInputStream@754035
at java.io.FilterInputStream.read(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- locked java.util.concurrent.locks.ReentrantLock$NonfairSync@1ca727a
"RMI TCP Connection(23130)-149.184.142.160" - Thread t@37799
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
- locked java.io.BufferedInputStream@d3f8d
at java.io.FilterInputStream.read(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- locked java.util.concurrent.locks.ReentrantLock$NonfairSync@1c75893
"RMI TCP Connection(23115)-149.184.142.160" - Thread t@37795
java.lang.Thread.State: TIMED_WAITING on com.sun.jmx.remote.internal.ArrayNotificationBuffer@fbfe50
at java.lang.Object.wait(Native Method)
at com.sun.jmx.remote.internal.ArrayNotificationBuffer.fetchNotifications(Unknown Source)
at com.sun.jmx.remote.internal.ArrayNotificationBuffer$ShareBuffer.fetchNotifications(Unknown Source)
at com.sun.jmx.remote.internal.ServerNotifForwarder.fetchNotifs(Unknown Source)
at javax.management.remote.rmi.RMIConnectionImpl$2.run(Unknown Source)
at javax.management.remote.rmi.RMIConnectionImpl$2.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at javax.management.remote.rmi.RMIConnectionImpl.fetchNotifications(Unknown Source)
at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source)
at sun.rmi.transport.Transport$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- locked java.util.concurrent.locks.ReentrantLock$NonfairSync@88281d
"RMI TCP Connection(23135)-149.184.142.160" - Thread t@37787
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
- locked java.io.BufferedInputStream@9d3697
at java.io.FilterInputStream.read(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- locked java.util.concurrent.locks.ReentrantLock$NonfairSync@16af9f9
"RMI TCP Connection(23062)-149.184.142.160" - Thread t@37756
java.lang.Thread.State: TIMED_WAITING on com.sun.jmx.remote.internal.ArrayNotificationBuffer@fbfe50
at java.lang.Object.wait(Native Method)
at com.sun.jmx.remote.internal.ArrayNotificationBuffer.fetchNotifications(Unknown Source)
at com.sun.jmx.remote.internal.ArrayNotificationBuffer$ShareBuffer.fetchNotifications(Unknown Source)
at com.sun.jmx.remote.internal.ServerNotifForwarder.fetchNotifs(Unknown Source)
at javax.management.remote.rmi.RMIConnectionImpl$2.run(Unknown Source)
at javax.management.remote.rmi.RMIConnectionImpl$2.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at javax.management.remote.rmi.RMIConnectionImpl.fetchNotifications(Unknown Source)
at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source)
at sun.rmi.transport.Transport$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- locked java.util.concurrent.locks.ReentrantLock$NonfairSync@415da0
"http-bio-8443-exec-13182" - Thread t@37623
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
- locked java.lang.Object@169bcbd
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
- locked com.sun.net.ssl.internal.ssl.AppInputStream@e61d04
at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:445)
at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:430)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:215)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:515)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:302)
- locked org.apache.tomcat.util.net.SocketWrapper@17b14f2
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- locked java.util.concurrent.locks.ReentrantLock$NonfairSync@19c4cd7
"http-bio-8443-exec-13181" - Thread t@37619
java.lang.Thread.State: WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1f31168
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"http-bio-8443-exec-13180" - Thread t@37618
java.lang.Thread.State: WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1f31168
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"http-bio-8443-exec-13179" - Thread t@37615
java.lang.Thread.State: WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1f31168
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"http-bio-8443-exec-13176" - Thread t@37584
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
- locked java.lang.Object@9d2f3
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
- locked com.sun.net.ssl.internal.ssl.AppInputStream@f8902a
at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:445)
at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:430)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:215)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:515)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:302)
- locked org.apache.tomcat.util.net.SocketWrapper@c0d197
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- locked java.util.concurrent.locks.ReentrantLock$NonfairSync@1004e70
"http-bio-8443-exec-13175" - Thread t@37583
java.lang.Thread.State: WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1f31168
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"http-bio-8443-exec-13174" - Thread t@37582
java.lang.Thread.State: WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1f31168
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"http-bio-8443-exec-13172" - Thread t@37580
java.lang.Thread.State: WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1f31168
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"http-bio-8443-exec-13157" - Thread t@37493
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
- locked java.lang.Object@11db953
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
- locked com.sun.net.ssl.internal.ssl.AppInputStream@7200ec
at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:445)
at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:430)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:215)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:515)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:302)
- locked org.apache.tomcat.util.net.SocketWrapper@9aa1e0
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- locked java.util.concurrent.locks.ReentrantLock$NonfairSync@1f9b5f5
"http-bio-8443-exec-13153" - Thread t@37489
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
- locked java.lang.Object@118f940
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
- locked com.sun.net.ssl.internal.ssl.AppInputStream@93e184
at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:445)
at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:430)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:215)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:515)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:302)
- locked org.apache.tomcat.util.net.SocketWrapper@bae208
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- locked java.util.concurrent.locks.ReentrantLock$NonfairSync@14d8afa
"JMX server connection timeout 20278" - Thread t@20278
java.lang.Thread.State: TIMED_WAITING on [I@172a3d4
at java.lang.Object.wait(Native Method)
at com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"JMX server connection timeout 20277" - Thread t@20277
java.lang.Thread.State: TIMED_WAITING on [I@161b40a
at java.lang.Object.wait(Native Method)
at com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"TIBCO EMS TCPLink Reader (Server-8211461)" - Thread t@14697
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
- locked java.lang.Object@15c19ae
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
- locked com.sun.net.ssl.internal.ssl.AppInputStream@199ff9e
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
- locked java.io.BufferedInputStream@15cf15b
at java.io.DataInputStream.readInt(Unknown Source)
at com.tibco.tibjms.TibjmsxLinkTcp._readWireMsg(TibjmsxLinkTcp.java:629)
at com.tibco.tibjms.TibjmsxLinkSSL._readWireMsg(TibjmsxLinkSSL.java:651)
at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.work(TibjmsxLinkTcp.java:280)
at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.run(TibjmsxLinkTcp.java:259)
Locked ownable synchronizers:
- None
"TIBCO EMS TCPLink Reader (Server-7262243)" - Thread t@6265
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
- locked java.lang.Object@2e0c34
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
- locked com.sun.net.ssl.internal.ssl.AppInputStream@1233759
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
- locked java.io.BufferedInputStream@106d785
at java.io.DataInputStream.readInt(Unknown Source)
at com.tibco.tibjms.TibjmsxLinkTcp._readWireMsg(TibjmsxLinkTcp.java:629)
at com.tibco.tibjms.TibjmsxLinkSSL._readWireMsg(TibjmsxLinkSSL.java:651)
at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.work(TibjmsxLinkTcp.java:280)
at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.run(TibjmsxLinkTcp.java:259)
Locked ownable synchronizers:
- None
"TIBCO EMS TCPLink Reader (Server-34701031)" - Thread t@4657
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
- locked java.lang.Object@2f626a
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
- locked com.sun.net.ssl.internal.ssl.AppInputStream@e655a9
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
- locked java.io.BufferedInputStream@2000c
at java.io.DataInputStream.readInt(Unknown Source)
at com.tibco.tibjms.TibjmsxLinkTcp._readWireMsg(TibjmsxLinkTcp.java:629)
at com.tibco.tibjms.TibjmsxLinkSSL._readWireMsg(TibjmsxLinkSSL.java:651)
at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.work(TibjmsxLinkTcp.java:280)
at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.run(TibjmsxLinkTcp.java:259)
Locked ownable synchronizers:
- None
"TIBCO EMS TCPLink Reader (Server-10500556)" - Thread t@4580
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
- locked java.lang.Object@1514a33
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
- locked com.sun.net.ssl.internal.ssl.AppInputStream@1de6f76
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
- locked java.io.BufferedInputStream@67b838
at java.io.DataInputStream.readInt(Unknown Source)
at com.tibco.tibjms.TibjmsxLinkTcp._readWireMsg(TibjmsxLinkTcp.java:629)
at com.tibco.tibjms.TibjmsxLinkSSL._readWireMsg(TibjmsxLinkSSL.java:651)
at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.work(TibjmsxLinkTcp.java:280)
at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.run(TibjmsxLinkTcp.java:259)
Locked ownable synchronizers:
- None
"TIBCO EMS TCPLink Reader (Server-10479806)" - Thread t@3911
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
- locked java.lang.Object@16f6ae4
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
- locked com.sun.net.ssl.internal.ssl.AppInputStream@1d2d6dd
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
- locked java.io.BufferedInputStream@1185eab
at java.io.DataInputStream.readInt(Unknown Source)
at com.tibco.tibjms.TibjmsxLinkTcp._readWireMsg(TibjmsxLinkTcp.java:629)
at com.tibco.tibjms.TibjmsxLinkSSL._readWireMsg(TibjmsxLinkSSL.java:651)
at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.work(TibjmsxLinkTcp.java:280)
at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.run(TibjmsxLinkTcp.java:259)
Locked ownable synchronizers:
- None
"TIBCO EMS TCPLink Reader (Server-7530293)" - Thread t@3746
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
- locked java.lang.Object@ef6481
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
- locked com.sun.net.ssl.internal.ssl.AppInputStream@af7fcb
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
- locked java.io.BufferedInputStream@1eef61c
at java.io.DataInputStream.readInt(Unknown Source)
at com.tibco.tibjms.TibjmsxLinkTcp._readWireMsg(TibjmsxLinkTcp.java:629)
at com.tibco.tibjms.TibjmsxLinkSSL._readWireMsg(TibjmsxLinkSSL.java:651)
at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.work(TibjmsxLinkTcp.java:280)
at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.run(TibjmsxLinkTcp.java:259)
Locked ownable synchronizers:
- None
"TIBCO EMS TCPLink Reader (Server-7208323)" - Thread t@3705
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
- locked java.lang.Object@6966e3
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
- locked com.sun.net.ssl.internal.ssl.AppInputStream@1e1e638
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
- locked java.io.BufferedInputStream@1d1a81f
at java.io.DataInputStream.readInt(Unknown Source)
at com.tibco.tibjms.TibjmsxLinkTcp._readWireMsg(TibjmsxLinkTcp.java:629)
at com.tibco.tibjms.TibjmsxLinkSSL._readWireMsg(TibjmsxLinkSSL.java:651)
at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.work(TibjmsxLinkTcp.java:280)
at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.run(TibjmsxLinkTcp.java:259)
Locked ownable synchronizers:
- None
"TIBCO EMS Connections Pinger" - Thread t@3704
java.lang.Thread.State: TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at com.tibco.tibjms.TibjmsxLinkTcp$Pinger.run(TibjmsxLinkTcp.java:154)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"Timer-1" - Thread t@3630
java.lang.Thread.State: TIMED_WAITING on java.util.TaskQueue@1f564da
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Unknown Source)
at java.util.TimerThread.run(Unknown Source)
Locked ownable synchronizers:
- None
"TIBCO EMS TCPLink Reader (Server-7291923)" - Thread t@677
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
- locked java.lang.Object@174b00f
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
- locked com.sun.net.ssl.internal.ssl.AppInputStream@1d2b3ca
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
- locked java.io.BufferedInputStream@19a318
at java.io.DataInputStream.readInt(Unknown Source)
at com.tibco.tibjms.TibjmsxLinkTcp._readWireMsg(TibjmsxLinkTcp.java:629)
at com.tibco.tibjms.TibjmsxLinkSSL._readWireMsg(TibjmsxLinkSSL.java:651)
at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.work(TibjmsxLinkTcp.java:280)
at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.run(TibjmsxLinkTcp.java:259)
Locked ownable synchronizers:
- None
"TIBCO EMS TCPLink Reader (Server-7097900)" - Thread t@380
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
- locked java.lang.Object@1229e8f
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
- locked com.sun.net.ssl.internal.ssl.AppInputStream@1d7cf47
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
- locked java.io.BufferedInputStream@1697394
at java.io.DataInputStream.readInt(Unknown Source)
at com.tibco.tibjms.TibjmsxLinkTcp._readWireMsg(TibjmsxLinkTcp.java:629)
at com.tibco.tibjms.TibjmsxLinkSSL._readWireMsg(TibjmsxLinkSSL.java:651)
at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.work(TibjmsxLinkTcp.java:280)
at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.run(TibjmsxLinkTcp.java:259)
Locked ownable synchronizers:
- None
"TIBCO EMS TCPLink Reader (Server-10324836)" - Thread t@157
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
- locked java.lang.Object@22fe5f
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
- locked com.sun.net.ssl.internal.ssl.AppInputStream@16a4569
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
- locked java.io.BufferedInputStream@1f3009b
at java.io.DataInputStream.readInt(Unknown Source)
at com.tibco.tibjms.TibjmsxLinkTcp._readWireMsg(TibjmsxLinkTcp.java:629)
at com.tibco.tibjms.TibjmsxLinkSSL._readWireMsg(TibjmsxLinkSSL.java:651)
at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.work(TibjmsxLinkTcp.java:280)
at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.run(TibjmsxLinkTcp.java:259)
Locked ownable synchronizers:
- None
"Timer-0" - Thread t@155
java.lang.Thread.State: TIMED_WAITING on java.util.TaskQueue@1b0504c
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Unknown Source)
at java.util.TimerThread.run(Unknown Source)
Locked ownable synchronizers:
- None
"TIBCO EMS TCPLink Reader (Server-34276374)" - Thread t@97
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
- locked java.lang.Object@acfea9
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
- locked com.sun.net.ssl.internal.ssl.AppInputStream@1832a6b
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
- locked java.io.BufferedInputStream@55e5f0
at java.io.DataInputStream.readInt(Unknown Source)
at com.tibco.tibjms.TibjmsxLinkTcp._readWireMsg(TibjmsxLinkTcp.java:629)
at com.tibco.tibjms.TibjmsxLinkSSL._readWireMsg(TibjmsxLinkSSL.java:651)
at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.work(TibjmsxLinkTcp.java:280)
at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.run(TibjmsxLinkTcp.java:259)
Locked ownable synchronizers:
- None
"TIBCO EMS TCPLink Reader (Server-7093946)" - Thread t@91
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
- locked java.lang.Object@16af6ea
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
- locked com.sun.net.ssl.internal.ssl.AppInputStream@17891e4
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
- locked java.io.BufferedInputStream@b2270e
at java.io.DataInputStream.readInt(Unknown Source)
at com.tibco.tibjms.TibjmsxLinkTcp._readWireMsg(TibjmsxLinkTcp.java:629)
at com.tibco.tibjms.TibjmsxLinkSSL._readWireMsg(TibjmsxLinkSSL.java:651)
at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.work(TibjmsxLinkTcp.java:280)
at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.run(TibjmsxLinkTcp.java:259)
Locked ownable synchronizers:
- None
"TIBCO EMS Connections Pinger" - Thread t@90
java.lang.Thread.State: TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at com.tibco.tibjms.TibjmsxLinkTcp$Pinger.run(TibjmsxLinkTcp.java:154)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"Java2D Disposer" - Thread t@51
java.lang.Thread.State: WAITING on java.lang.ref.ReferenceQueue$Lock@67dfae
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(Unknown Source)
at java.lang.ref.ReferenceQueue.remove(Unknown Source)
at sun.java2d.Disposer.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"http-bio-8443-AsyncTimeout" - Thread t@32
java.lang.Thread.State: TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at org.apache.tomcat.util.net.JIoEndpoint$AsyncTimeout.run(JIoEndpoint.java:141)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"http-bio-8443-Acceptor-0" - Thread t@31
java.lang.Thread.State: WAITING on org.apache.tomcat.util.threads.LimitLatch$Sync@16dd685
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(Unknown Source)
at org.apache.tomcat.util.threads.LimitLatch.countUpOrAwait(LimitLatch.java:99)
at org.apache.tomcat.util.net.AbstractEndpoint.countUpOrAwaitConnection(AbstractEndpoint.java:581)
at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:204)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"ContainerBackgroundProcessor[StandardEngine[Catalina]]" - Thread t@30
java.lang.Thread.State: TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1369)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"GC Daemon" - Thread t@21
java.lang.Thread.State: TIMED_WAITING on sun.misc.GC$LatencyLock@18e5957
at java.lang.Object.wait(Native Method)
at sun.misc.GC$Daemon.run(Unknown Source)
Locked ownable synchronizers:
- None
"RMI Scheduler(0)" - Thread t@14
java.lang.Thread.State: TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@186b6cc
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
at java.util.concurrent.DelayQueue.take(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"RMI TCP Accept-7360" - Thread t@11
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(Unknown Source)
- locked java.net.SocksSocketImpl@1ad9253
at java.net.ServerSocket.implAccept(Unknown Source)
at java.net.ServerSocket.accept(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"RMI TCP Accept-0" - Thread t@10
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(Unknown Source)
- locked java.net.SocksSocketImpl@61c8a1
at java.net.ServerSocket.implAccept(Unknown Source)
at java.net.ServerSocket.accept(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"Signal Dispatcher" - Thread t@4
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Finalizer" - Thread t@3
java.lang.Thread.State: WAITING on java.lang.ref.ReferenceQueue$Lock@14b402d
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(Unknown Source)
at java.lang.ref.ReferenceQueue.remove(Unknown Source)
at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)
Locked ownable synchronizers:
- None
"Reference Handler" - Thread t@2
java.lang.Thread.State: WAITING on java.lang.ref.Reference$Lock@16269cb
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source)
Locked ownable synchronizers:
- None
"main" - Thread t@1
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(Unknown Source)
- locked java.net.SocksSocketImpl@94237f
at java.net.ServerSocket.implAccept(Unknown Source)
at java.net.ServerSocket.accept(Unknown Source)
at org.apache.catalina.core.StandardServer.await(StandardServer.java:447)
at org.apache.catalina.startup.Catalina.await(Catalina.java:709)
at org.apache.catalina.startup.Catalina.start(Catalina.java:654)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:322)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:450)
Locked ownable synchronizers:
- None
Also, we are using Default JPA Connection Pooling? Should we use Tomcat JDBC Connection Pooling?
There nothing additional configured in context.xml.
I see UserDatabaseRealm in server.xml, is this a default setting?
What connector is recommended for SSL Connections on Solaris 10?
Regards,Bal.
Balvindar,
On 9/3/12 9:12 AM, balvindar dhaliwal wrote:
We are using tomcat 7.0.19, for a jsf(richfaces),seam, jpa web
application. Here are the software versions and hardware
infrastructure details. Apache Tomact # 7.0.19 Java run time #
1.6.0_12
If possible, it would be best to upgrade to the latest version of both
of these packages: Java 1.6.0_36 and Tomcat 7.0.30. There are several
important security fixes in both of those (relative to where you are now).
We have noticed after about 2 months, the web requests stop being
processed. We had observed this issue twice in the last 5 months.
We are using HTTP/1.1 Connector ( BIO ). The heap and permgen
memory seem fine. There is no out of memory errors in the logs. We
have used jvisualvm to monitor heap, permgen, threads and thread
dump, AND THEY ALL LOOK OK. The thread dump looks fine and there
are no deadlocks/stuck threads observed.
Your thread dump attachment was stripped from the list. Can you
copy/paste it into a follow post?
Please let me know if you need more information.
It's hard to guess without the thread dump, but my initial reaction is
that it's connection-pool exhaustion, but you said that your
connection pool seems healthy.
I see from your server.xml that you are using UserDatabaseRealm: is
that actually in use in production? It's probably an inappropriate
component to use for anything but the most trivial configuration (like
maybe the Tomcat manager app). Are you using any kind of <Realm> or
<Resource> specifically configured in in your webapp's context.xml?
That would also be good to know.
If you are using JDBCRealm, you may have hit an edge case that causes
some kind of deadlock (but again, you say there are no deadlocks).
There were some fixes to the NIO and APR connectors recently (and your
version of Tomcat is now more than a year out-of-date) that fixed
apparent deadlocks but I'm not sure if those extended into the BIO
realm. I can't see anything in the changelog that would indicate a fix
to the BIO connector that would affect you.
Again, it's very difficult to speculate without some more data, so
sending that thread dump (especially if you can give us several thread
dumps over a few minutes) will definitely help.
> From: balvindard@hotmail.com
> To: users@tomcat.apache.org
> Subject: RE: Apache tomcat ( 7.0.19 ) stops processing user requests suddenly but works fine after restart.
> Date: Mon, 3 Sep 2012 13:47:50 +0000
>
>
>
> Hi,
> Q1:server.xml :------------<?xml version='1.0' encoding='utf-8'?><Server port="8009" shutdown="xxxxx"> <Listener className="org.apache.catalina.security.SecurityListener" /> <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" /> <Listener className="org.apache.catalina.core.JasperListener" /> <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" /> <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" /> <Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" /> <GlobalNamingResources> <Resource name="UserDatabase" auth="Container" type="org.apache.catalina.UserDatabase" description="User database that can be updated and saved" factory="org.apache.catalina.users.MemoryUserDatabaseFactory" pathname="conf/tomcat-users.xml" /> </GlobalNamingResources> <Service name="Catalina"> <Connector port="8443" protocol="HTTP/1.1" SSLEnabled="true" maxThreads="150" scheme="https" secure="true" clientAuth="false" sslProtocol="TLS" ciphers="TLS_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_DSS_WITH_AES_128_CBC_SHA,SSL_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA" keystoreFile="conf/xxxxxx-yy-zzzz.jks" keystorePass="xxxx" /> <Engine name="Catalina" defaultHost="localhost"> <Realm className="org.apache.catalina.realm.UserDatabaseRealm" digest="SHA" resourceName="UserDatabase"/> <Host name="localhost" appBase="webapps" unpackWARs="true" autoDeploy="true" xmlValidation="false" xmlNamespaceAware="false">
> <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs" prefix="localhost_access_log." suffix=".txt" pattern="common" resolveHosts="false"/>
> </Host> </Engine> </Service></Server>
> Q2:We are using HTTPS.
> Regards,Bal.
> > Date: Mon, 3 Sep 2012 15:35:31 +0200
> > From: aw@ice-sa.com
> > To: users@tomcat.apache.org
> > Subject: Re: Apache tomcat ( 7.0.19 ) stops processing user requests suddenly but works fine after restart.
> >
> > balvindar dhaliwal wrote:
> > >
> > >
> > >
> > >
> > > Hi ,
> > >
> > >
>
> > > We are using tomcat 7.0.19, for a jsf(richfaces),seam, jpa web application.
> > > Here are the software versions and hardware infrastructure details.
> > > Apache Tomact # 7.0.19
> > > Java run time # 1.6.0_12
> > > Server infrastructure # SunOS xxxxxxx 5.10 Generic_142900-03 sun4v sparc SUNW,Sun-Fire-T200
> > > OS version # solaris 10.
> > >
> > > Issue Description #
> > > We have noticed after about 2 months, the web requests stop being processed. We had observed this issue twice in the last 5 months.
> > > We are using HTTP/1.1 Connector ( BIO ). The heap and permgen memory seem fine. There is no out of memory errors in the logs. We have used jvisualvm to monitor heap, permgen, threads and thread dump, AND THEY ALL LOOK OK. The thread dump looks fine and there are no deadlocks/stuck threads observed.
> > > The localhost_access_log stops being updated. The connector port is still listening on SSL port 8443.
> > > After we restart Tomcat, the new requests are processing fine.
> > > Can you please advise other areas we should look into for investigation (any additional tracing).
> > > Should we upgrade to tomcat 7.0.29? What could cause the Connector to stop processing requests on the port?
> > > Further info: Heap(1/6th used), PermGen(1/2 used). GC is effectively reducing the heap. The fact that local access log is not being updated, suggests the request is not making to the Servlet container. We are using JPA Connection Pool, have not noticed any db connection issues. After the restart, everything works well.
> > > The system logs seems to be fine.
> > > Please find the attached sample thread dump copy when this issue was occurred.
> > >
> > > Please let me know if you need more information.
> > >
> >
> > Q1: Can you copy and paste the server.xml of Tomcat here ? (removing any private information).
> > Copy and paste in-line, the list strips most attachments.
> >
> > Q2: Above, you mention that the HTTPS connector on port 8443 is still listening. What
> > about the HTTP connector on port ??? (usually 80).
> > Or are all the requests HTTPS ?
> >
> >
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> > For additional commands, e-mail: users-help@tomcat.apache.org
> >
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Apache tomcat ( 7.0.19 ) stops processing user requests
suddenly but works fine after restart.
Posted by balvindar dhaliwal <ba...@hotmail.com>.
Hi,
I have pasted the first thread dump, and attached 3 other thread dumps.
Thread dump1:-------------------
2012-08-10 14:40:52
Full thread dump Java HotSpot(TM) Server VM (11.2-b01 mixed mode):
"RMI TCP Connection(23136)-149.184.142.160" - Thread t@37810
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
- locked java.io.BufferedInputStream@427137
at java.io.FilterInputStream.read(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- locked java.util.concurrent.locks.ReentrantLock$NonfairSync@17119c
"JMX server connection timeout 37809" - Thread t@37809
java.lang.Thread.State: TIMED_WAITING on [I@378cf5
at java.lang.Object.wait(Native Method)
at com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"JMX server connection timeout 37808" - Thread t@37808
java.lang.Thread.State: TIMED_WAITING on [I@1075926
at java.lang.Object.wait(Native Method)
at com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"RMI TCP Connection(23129)-149.177.35.85" - Thread t@37806
java.lang.Thread.State: RUNNABLE
at sun.management.ThreadImpl.dumpThreads0(Native Method)
at sun.management.ThreadImpl.dumpAllThreads(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(Unknown Source)
at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(Unknown Source)
at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(Unknown Source)
at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(Unknown Source)
at com.sun.jmx.mbeanserver.PerInterface.invoke(Unknown Source)
at com.sun.jmx.mbeanserver.MBeanSupport.invoke(Unknown Source)
at javax.management.StandardMBean.invoke(Unknown Source)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Unknown Source)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(Unknown Source)
at com.sun.jmx.remote.security.MBeanServerAccessController.invoke(Unknown Source)
at javax.management.remote.rmi.RMIConnectionImpl.doOperation(Unknown Source)
at javax.management.remote.rmi.RMIConnectionImpl.access$200(Unknown Source)
at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(Unknown Source)
at javax.management.remote.rmi.RMIConnectionImpl.invoke(Unknown Source)
at sun.reflect.GeneratedMethodAccessor5299.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source)
at sun.rmi.transport.Transport$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- locked java.util.concurrent.locks.ReentrantLock$NonfairSync@d0793c
"RMI TCP Connection(23134)-149.184.142.160" - Thread t@37803
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
- locked java.io.BufferedInputStream@754035
at java.io.FilterInputStream.read(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- locked java.util.concurrent.locks.ReentrantLock$NonfairSync@1ca727a
"RMI TCP Connection(23130)-149.184.142.160" - Thread t@37799
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
- locked java.io.BufferedInputStream@d3f8d
at java.io.FilterInputStream.read(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- locked java.util.concurrent.locks.ReentrantLock$NonfairSync@1c75893
"RMI TCP Connection(23115)-149.184.142.160" - Thread t@37795
java.lang.Thread.State: TIMED_WAITING on com.sun.jmx.remote.internal.ArrayNotificationBuffer@fbfe50
at java.lang.Object.wait(Native Method)
at com.sun.jmx.remote.internal.ArrayNotificationBuffer.fetchNotifications(Unknown Source)
at com.sun.jmx.remote.internal.ArrayNotificationBuffer$ShareBuffer.fetchNotifications(Unknown Source)
at com.sun.jmx.remote.internal.ServerNotifForwarder.fetchNotifs(Unknown Source)
at javax.management.remote.rmi.RMIConnectionImpl$2.run(Unknown Source)
at javax.management.remote.rmi.RMIConnectionImpl$2.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at javax.management.remote.rmi.RMIConnectionImpl.fetchNotifications(Unknown Source)
at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source)
at sun.rmi.transport.Transport$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- locked java.util.concurrent.locks.ReentrantLock$NonfairSync@88281d
"RMI TCP Connection(23135)-149.184.142.160" - Thread t@37787
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
- locked java.io.BufferedInputStream@9d3697
at java.io.FilterInputStream.read(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- locked java.util.concurrent.locks.ReentrantLock$NonfairSync@16af9f9
"RMI TCP Connection(23062)-149.184.142.160" - Thread t@37756
java.lang.Thread.State: TIMED_WAITING on com.sun.jmx.remote.internal.ArrayNotificationBuffer@fbfe50
at java.lang.Object.wait(Native Method)
at com.sun.jmx.remote.internal.ArrayNotificationBuffer.fetchNotifications(Unknown Source)
at com.sun.jmx.remote.internal.ArrayNotificationBuffer$ShareBuffer.fetchNotifications(Unknown Source)
at com.sun.jmx.remote.internal.ServerNotifForwarder.fetchNotifs(Unknown Source)
at javax.management.remote.rmi.RMIConnectionImpl$2.run(Unknown Source)
at javax.management.remote.rmi.RMIConnectionImpl$2.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at javax.management.remote.rmi.RMIConnectionImpl.fetchNotifications(Unknown Source)
at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source)
at sun.rmi.transport.Transport$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- locked java.util.concurrent.locks.ReentrantLock$NonfairSync@415da0
"http-bio-8443-exec-13182" - Thread t@37623
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
- locked java.lang.Object@169bcbd
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
- locked com.sun.net.ssl.internal.ssl.AppInputStream@e61d04
at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:445)
at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:430)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:215)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:515)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:302)
- locked org.apache.tomcat.util.net.SocketWrapper@17b14f2
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- locked java.util.concurrent.locks.ReentrantLock$NonfairSync@19c4cd7
"http-bio-8443-exec-13181" - Thread t@37619
java.lang.Thread.State: WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1f31168
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"http-bio-8443-exec-13180" - Thread t@37618
java.lang.Thread.State: WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1f31168
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"http-bio-8443-exec-13179" - Thread t@37615
java.lang.Thread.State: WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1f31168
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"http-bio-8443-exec-13176" - Thread t@37584
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
- locked java.lang.Object@9d2f3
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
- locked com.sun.net.ssl.internal.ssl.AppInputStream@f8902a
at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:445)
at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:430)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:215)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:515)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:302)
- locked org.apache.tomcat.util.net.SocketWrapper@c0d197
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- locked java.util.concurrent.locks.ReentrantLock$NonfairSync@1004e70
"http-bio-8443-exec-13175" - Thread t@37583
java.lang.Thread.State: WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1f31168
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"http-bio-8443-exec-13174" - Thread t@37582
java.lang.Thread.State: WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1f31168
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"http-bio-8443-exec-13172" - Thread t@37580
java.lang.Thread.State: WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1f31168
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104)
at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"http-bio-8443-exec-13157" - Thread t@37493
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
- locked java.lang.Object@11db953
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
- locked com.sun.net.ssl.internal.ssl.AppInputStream@7200ec
at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:445)
at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:430)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:215)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:515)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:302)
- locked org.apache.tomcat.util.net.SocketWrapper@9aa1e0
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- locked java.util.concurrent.locks.ReentrantLock$NonfairSync@1f9b5f5
"http-bio-8443-exec-13153" - Thread t@37489
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
- locked java.lang.Object@118f940
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
- locked com.sun.net.ssl.internal.ssl.AppInputStream@93e184
at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:445)
at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:430)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:215)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:515)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:302)
- locked org.apache.tomcat.util.net.SocketWrapper@bae208
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- locked java.util.concurrent.locks.ReentrantLock$NonfairSync@14d8afa
"JMX server connection timeout 20278" - Thread t@20278
java.lang.Thread.State: TIMED_WAITING on [I@172a3d4
at java.lang.Object.wait(Native Method)
at com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"JMX server connection timeout 20277" - Thread t@20277
java.lang.Thread.State: TIMED_WAITING on [I@161b40a
at java.lang.Object.wait(Native Method)
at com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"TIBCO EMS TCPLink Reader (Server-8211461)" - Thread t@14697
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
- locked java.lang.Object@15c19ae
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
- locked com.sun.net.ssl.internal.ssl.AppInputStream@199ff9e
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
- locked java.io.BufferedInputStream@15cf15b
at java.io.DataInputStream.readInt(Unknown Source)
at com.tibco.tibjms.TibjmsxLinkTcp._readWireMsg(TibjmsxLinkTcp.java:629)
at com.tibco.tibjms.TibjmsxLinkSSL._readWireMsg(TibjmsxLinkSSL.java:651)
at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.work(TibjmsxLinkTcp.java:280)
at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.run(TibjmsxLinkTcp.java:259)
Locked ownable synchronizers:
- None
"TIBCO EMS TCPLink Reader (Server-7262243)" - Thread t@6265
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
- locked java.lang.Object@2e0c34
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
- locked com.sun.net.ssl.internal.ssl.AppInputStream@1233759
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
- locked java.io.BufferedInputStream@106d785
at java.io.DataInputStream.readInt(Unknown Source)
at com.tibco.tibjms.TibjmsxLinkTcp._readWireMsg(TibjmsxLinkTcp.java:629)
at com.tibco.tibjms.TibjmsxLinkSSL._readWireMsg(TibjmsxLinkSSL.java:651)
at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.work(TibjmsxLinkTcp.java:280)
at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.run(TibjmsxLinkTcp.java:259)
Locked ownable synchronizers:
- None
"TIBCO EMS TCPLink Reader (Server-34701031)" - Thread t@4657
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
- locked java.lang.Object@2f626a
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
- locked com.sun.net.ssl.internal.ssl.AppInputStream@e655a9
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
- locked java.io.BufferedInputStream@2000c
at java.io.DataInputStream.readInt(Unknown Source)
at com.tibco.tibjms.TibjmsxLinkTcp._readWireMsg(TibjmsxLinkTcp.java:629)
at com.tibco.tibjms.TibjmsxLinkSSL._readWireMsg(TibjmsxLinkSSL.java:651)
at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.work(TibjmsxLinkTcp.java:280)
at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.run(TibjmsxLinkTcp.java:259)
Locked ownable synchronizers:
- None
"TIBCO EMS TCPLink Reader (Server-10500556)" - Thread t@4580
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
- locked java.lang.Object@1514a33
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
- locked com.sun.net.ssl.internal.ssl.AppInputStream@1de6f76
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
- locked java.io.BufferedInputStream@67b838
at java.io.DataInputStream.readInt(Unknown Source)
at com.tibco.tibjms.TibjmsxLinkTcp._readWireMsg(TibjmsxLinkTcp.java:629)
at com.tibco.tibjms.TibjmsxLinkSSL._readWireMsg(TibjmsxLinkSSL.java:651)
at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.work(TibjmsxLinkTcp.java:280)
at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.run(TibjmsxLinkTcp.java:259)
Locked ownable synchronizers:
- None
"TIBCO EMS TCPLink Reader (Server-10479806)" - Thread t@3911
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
- locked java.lang.Object@16f6ae4
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
- locked com.sun.net.ssl.internal.ssl.AppInputStream@1d2d6dd
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
- locked java.io.BufferedInputStream@1185eab
at java.io.DataInputStream.readInt(Unknown Source)
at com.tibco.tibjms.TibjmsxLinkTcp._readWireMsg(TibjmsxLinkTcp.java:629)
at com.tibco.tibjms.TibjmsxLinkSSL._readWireMsg(TibjmsxLinkSSL.java:651)
at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.work(TibjmsxLinkTcp.java:280)
at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.run(TibjmsxLinkTcp.java:259)
Locked ownable synchronizers:
- None
"TIBCO EMS TCPLink Reader (Server-7530293)" - Thread t@3746
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
- locked java.lang.Object@ef6481
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
- locked com.sun.net.ssl.internal.ssl.AppInputStream@af7fcb
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
- locked java.io.BufferedInputStream@1eef61c
at java.io.DataInputStream.readInt(Unknown Source)
at com.tibco.tibjms.TibjmsxLinkTcp._readWireMsg(TibjmsxLinkTcp.java:629)
at com.tibco.tibjms.TibjmsxLinkSSL._readWireMsg(TibjmsxLinkSSL.java:651)
at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.work(TibjmsxLinkTcp.java:280)
at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.run(TibjmsxLinkTcp.java:259)
Locked ownable synchronizers:
- None
"TIBCO EMS TCPLink Reader (Server-7208323)" - Thread t@3705
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
- locked java.lang.Object@6966e3
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
- locked com.sun.net.ssl.internal.ssl.AppInputStream@1e1e638
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
- locked java.io.BufferedInputStream@1d1a81f
at java.io.DataInputStream.readInt(Unknown Source)
at com.tibco.tibjms.TibjmsxLinkTcp._readWireMsg(TibjmsxLinkTcp.java:629)
at com.tibco.tibjms.TibjmsxLinkSSL._readWireMsg(TibjmsxLinkSSL.java:651)
at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.work(TibjmsxLinkTcp.java:280)
at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.run(TibjmsxLinkTcp.java:259)
Locked ownable synchronizers:
- None
"TIBCO EMS Connections Pinger" - Thread t@3704
java.lang.Thread.State: TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at com.tibco.tibjms.TibjmsxLinkTcp$Pinger.run(TibjmsxLinkTcp.java:154)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"Timer-1" - Thread t@3630
java.lang.Thread.State: TIMED_WAITING on java.util.TaskQueue@1f564da
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Unknown Source)
at java.util.TimerThread.run(Unknown Source)
Locked ownable synchronizers:
- None
"TIBCO EMS TCPLink Reader (Server-7291923)" - Thread t@677
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
- locked java.lang.Object@174b00f
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
- locked com.sun.net.ssl.internal.ssl.AppInputStream@1d2b3ca
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
- locked java.io.BufferedInputStream@19a318
at java.io.DataInputStream.readInt(Unknown Source)
at com.tibco.tibjms.TibjmsxLinkTcp._readWireMsg(TibjmsxLinkTcp.java:629)
at com.tibco.tibjms.TibjmsxLinkSSL._readWireMsg(TibjmsxLinkSSL.java:651)
at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.work(TibjmsxLinkTcp.java:280)
at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.run(TibjmsxLinkTcp.java:259)
Locked ownable synchronizers:
- None
"TIBCO EMS TCPLink Reader (Server-7097900)" - Thread t@380
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
- locked java.lang.Object@1229e8f
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
- locked com.sun.net.ssl.internal.ssl.AppInputStream@1d7cf47
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
- locked java.io.BufferedInputStream@1697394
at java.io.DataInputStream.readInt(Unknown Source)
at com.tibco.tibjms.TibjmsxLinkTcp._readWireMsg(TibjmsxLinkTcp.java:629)
at com.tibco.tibjms.TibjmsxLinkSSL._readWireMsg(TibjmsxLinkSSL.java:651)
at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.work(TibjmsxLinkTcp.java:280)
at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.run(TibjmsxLinkTcp.java:259)
Locked ownable synchronizers:
- None
"TIBCO EMS TCPLink Reader (Server-10324836)" - Thread t@157
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
- locked java.lang.Object@22fe5f
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
- locked com.sun.net.ssl.internal.ssl.AppInputStream@16a4569
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
- locked java.io.BufferedInputStream@1f3009b
at java.io.DataInputStream.readInt(Unknown Source)
at com.tibco.tibjms.TibjmsxLinkTcp._readWireMsg(TibjmsxLinkTcp.java:629)
at com.tibco.tibjms.TibjmsxLinkSSL._readWireMsg(TibjmsxLinkSSL.java:651)
at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.work(TibjmsxLinkTcp.java:280)
at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.run(TibjmsxLinkTcp.java:259)
Locked ownable synchronizers:
- None
"Timer-0" - Thread t@155
java.lang.Thread.State: TIMED_WAITING on java.util.TaskQueue@1b0504c
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Unknown Source)
at java.util.TimerThread.run(Unknown Source)
Locked ownable synchronizers:
- None
"TIBCO EMS TCPLink Reader (Server-34276374)" - Thread t@97
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
- locked java.lang.Object@acfea9
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
- locked com.sun.net.ssl.internal.ssl.AppInputStream@1832a6b
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
- locked java.io.BufferedInputStream@55e5f0
at java.io.DataInputStream.readInt(Unknown Source)
at com.tibco.tibjms.TibjmsxLinkTcp._readWireMsg(TibjmsxLinkTcp.java:629)
at com.tibco.tibjms.TibjmsxLinkSSL._readWireMsg(TibjmsxLinkSSL.java:651)
at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.work(TibjmsxLinkTcp.java:280)
at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.run(TibjmsxLinkTcp.java:259)
Locked ownable synchronizers:
- None
"TIBCO EMS TCPLink Reader (Server-7093946)" - Thread t@91
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
- locked java.lang.Object@16af6ea
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
- locked com.sun.net.ssl.internal.ssl.AppInputStream@17891e4
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
- locked java.io.BufferedInputStream@b2270e
at java.io.DataInputStream.readInt(Unknown Source)
at com.tibco.tibjms.TibjmsxLinkTcp._readWireMsg(TibjmsxLinkTcp.java:629)
at com.tibco.tibjms.TibjmsxLinkSSL._readWireMsg(TibjmsxLinkSSL.java:651)
at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.work(TibjmsxLinkTcp.java:280)
at com.tibco.tibjms.TibjmsxLinkTcp$LinkReader.run(TibjmsxLinkTcp.java:259)
Locked ownable synchronizers:
- None
"TIBCO EMS Connections Pinger" - Thread t@90
java.lang.Thread.State: TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at com.tibco.tibjms.TibjmsxLinkTcp$Pinger.run(TibjmsxLinkTcp.java:154)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"Java2D Disposer" - Thread t@51
java.lang.Thread.State: WAITING on java.lang.ref.ReferenceQueue$Lock@67dfae
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(Unknown Source)
at java.lang.ref.ReferenceQueue.remove(Unknown Source)
at sun.java2d.Disposer.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"http-bio-8443-AsyncTimeout" - Thread t@32
java.lang.Thread.State: TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at org.apache.tomcat.util.net.JIoEndpoint$AsyncTimeout.run(JIoEndpoint.java:141)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"http-bio-8443-Acceptor-0" - Thread t@31
java.lang.Thread.State: WAITING on org.apache.tomcat.util.threads.LimitLatch$Sync@16dd685
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(Unknown Source)
at org.apache.tomcat.util.threads.LimitLatch.countUpOrAwait(LimitLatch.java:99)
at org.apache.tomcat.util.net.AbstractEndpoint.countUpOrAwaitConnection(AbstractEndpoint.java:581)
at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:204)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"ContainerBackgroundProcessor[StandardEngine[Catalina]]" - Thread t@30
java.lang.Thread.State: TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1369)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"GC Daemon" - Thread t@21
java.lang.Thread.State: TIMED_WAITING on sun.misc.GC$LatencyLock@18e5957
at java.lang.Object.wait(Native Method)
at sun.misc.GC$Daemon.run(Unknown Source)
Locked ownable synchronizers:
- None
"RMI Scheduler(0)" - Thread t@14
java.lang.Thread.State: TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@186b6cc
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source)
at java.util.concurrent.DelayQueue.take(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"RMI TCP Accept-7360" - Thread t@11
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(Unknown Source)
- locked java.net.SocksSocketImpl@1ad9253
at java.net.ServerSocket.implAccept(Unknown Source)
at java.net.ServerSocket.accept(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"RMI TCP Accept-0" - Thread t@10
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(Unknown Source)
- locked java.net.SocksSocketImpl@61c8a1
at java.net.ServerSocket.implAccept(Unknown Source)
at java.net.ServerSocket.accept(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(Unknown Source)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Locked ownable synchronizers:
- None
"Signal Dispatcher" - Thread t@4
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Finalizer" - Thread t@3
java.lang.Thread.State: WAITING on java.lang.ref.ReferenceQueue$Lock@14b402d
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(Unknown Source)
at java.lang.ref.ReferenceQueue.remove(Unknown Source)
at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)
Locked ownable synchronizers:
- None
"Reference Handler" - Thread t@2
java.lang.Thread.State: WAITING on java.lang.ref.Reference$Lock@16269cb
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source)
Locked ownable synchronizers:
- None
"main" - Thread t@1
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(Unknown Source)
- locked java.net.SocksSocketImpl@94237f
at java.net.ServerSocket.implAccept(Unknown Source)
at java.net.ServerSocket.accept(Unknown Source)
at org.apache.catalina.core.StandardServer.await(StandardServer.java:447)
at org.apache.catalina.startup.Catalina.await(Catalina.java:709)
at org.apache.catalina.startup.Catalina.start(Catalina.java:654)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:322)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:450)
Locked ownable synchronizers:
- None
Also, we are using Default JPA Connection Pooling? Should we use Tomcat JDBC Connection Pooling?
There nothing additional configured in context.xml.
I see UserDatabaseRealm in server.xml, is this a default setting?
What connector is recommended for SSL Connections on Solaris 10?
Regards,Bal.
Balvindar,
On 9/3/12 9:12 AM, balvindar dhaliwal wrote:
We are using tomcat 7.0.19, for a jsf(richfaces),seam, jpa web
application. Here are the software versions and hardware
infrastructure details. Apache Tomact # 7.0.19 Java run time #
1.6.0_12
If possible, it would be best to upgrade to the latest version of both
of these packages: Java 1.6.0_36 and Tomcat 7.0.30. There are several
important security fixes in both of those (relative to where you are now).
We have noticed after about 2 months, the web requests stop being
processed. We had observed this issue twice in the last 5 months.
We are using HTTP/1.1 Connector ( BIO ). The heap and permgen
memory seem fine. There is no out of memory errors in the logs. We
have used jvisualvm to monitor heap, permgen, threads and thread
dump, AND THEY ALL LOOK OK. The thread dump looks fine and there
are no deadlocks/stuck threads observed.
Your thread dump attachment was stripped from the list. Can you
copy/paste it into a follow post?
Please let me know if you need more information.
It's hard to guess without the thread dump, but my initial reaction is
that it's connection-pool exhaustion, but you said that your
connection pool seems healthy.
I see from your server.xml that you are using UserDatabaseRealm: is
that actually in use in production? It's probably an inappropriate
component to use for anything but the most trivial configuration (like
maybe the Tomcat manager app). Are you using any kind of <Realm> or
<Resource> specifically configured in in your webapp's context.xml?
That would also be good to know.
If you are using JDBCRealm, you may have hit an edge case that causes
some kind of deadlock (but again, you say there are no deadlocks).
There were some fixes to the NIO and APR connectors recently (and your
version of Tomcat is now more than a year out-of-date) that fixed
apparent deadlocks but I'm not sure if those extended into the BIO
realm. I can't see anything in the changelog that would indicate a fix
to the BIO connector that would affect you.
Again, it's very difficult to speculate without some more data, so
sending that thread dump (especially if you can give us several thread
dumps over a few minutes) will definitely help.
> From: balvindard@hotmail.com
> To: users@tomcat.apache.org
> Subject: RE: Apache tomcat ( 7.0.19 ) stops processing user requests suddenly but works fine after restart.
> Date: Mon, 3 Sep 2012 13:47:50 +0000
>
>
>
> Hi,
> Q1:server.xml :------------<?xml version='1.0' encoding='utf-8'?><Server port="8009" shutdown="xxxxx"> <Listener className="org.apache.catalina.security.SecurityListener" /> <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" /> <Listener className="org.apache.catalina.core.JasperListener" /> <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" /> <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" /> <Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" /> <GlobalNamingResources> <Resource name="UserDatabase" auth="Container" type="org.apache.catalina.UserDatabase" description="User database that can be updated and saved" factory="org.apache.catalina.users.MemoryUserDatabaseFactory" pathname="conf/tomcat-users.xml" /> </GlobalNamingResources> <Service name="Catalina"> <Connector port="8443" protocol="HTTP/1.1" SSLEnabled="true" maxThreads="150" scheme="https" secure="true" clientAuth="false" sslProtocol="TLS" ciphers="TLS_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_DSS_WITH_AES_128_CBC_SHA,SSL_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA" keystoreFile="conf/xxxxxx-yy-zzzz.jks" keystorePass="xxxx" /> <Engine name="Catalina" defaultHost="localhost"> <Realm className="org.apache.catalina.realm.UserDatabaseRealm" digest="SHA" resourceName="UserDatabase"/> <Host name="localhost" appBase="webapps" unpackWARs="true" autoDeploy="true" xmlValidation="false" xmlNamespaceAware="false">
> <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs" prefix="localhost_access_log." suffix=".txt" pattern="common" resolveHosts="false"/>
> </Host> </Engine> </Service></Server>
> Q2:We are using HTTPS.
> Regards,Bal.
> > Date: Mon, 3 Sep 2012 15:35:31 +0200
> > From: aw@ice-sa.com
> > To: users@tomcat.apache.org
> > Subject: Re: Apache tomcat ( 7.0.19 ) stops processing user requests suddenly but works fine after restart.
> >
> > balvindar dhaliwal wrote:
> > >
> > >
> > >
> > >
> > > Hi ,
> > >
> > >
>
> > > We are using tomcat 7.0.19, for a jsf(richfaces),seam, jpa web application.
> > > Here are the software versions and hardware infrastructure details.
> > > Apache Tomact # 7.0.19
> > > Java run time # 1.6.0_12
> > > Server infrastructure # SunOS xxxxxxx 5.10 Generic_142900-03 sun4v sparc SUNW,Sun-Fire-T200
> > > OS version # solaris 10.
> > >
> > > Issue Description #
> > > We have noticed after about 2 months, the web requests stop being processed. We had observed this issue twice in the last 5 months.
> > > We are using HTTP/1.1 Connector ( BIO ). The heap and permgen memory seem fine. There is no out of memory errors in the logs. We have used jvisualvm to monitor heap, permgen, threads and thread dump, AND THEY ALL LOOK OK. The thread dump looks fine and there are no deadlocks/stuck threads observed.
> > > The localhost_access_log stops being updated. The connector port is still listening on SSL port 8443.
> > > After we restart Tomcat, the new requests are processing fine.
> > > Can you please advise other areas we should look into for investigation (any additional tracing).
> > > Should we upgrade to tomcat 7.0.29? What could cause the Connector to stop processing requests on the port?
> > > Further info: Heap(1/6th used), PermGen(1/2 used). GC is effectively reducing the heap. The fact that local access log is not being updated, suggests the request is not making to the Servlet container. We are using JPA Connection Pool, have not noticed any db connection issues. After the restart, everything works well.
> > > The system logs seems to be fine.
> > > Please find the attached sample thread dump copy when this issue was occurred.
> > >
> > > Please let me know if you need more information.
> > >
> >
> > Q1: Can you copy and paste the server.xml of Tomcat here ? (removing any private information).
> > Copy and paste in-line, the list strips most attachments.
> >
> > Q2: Above, you mention that the HTTPS connector on port 8443 is still listening. What
> > about the HTTP connector on port ??? (usually 80).
> > Or are all the requests HTTPS ?
> >
> >
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> > For additional commands, e-mail: users-help@tomcat.apache.org
> >
>
RE: Apache tomcat ( 7.0.19 ) stops processing user requests
suddenly but works fine after restart.
Posted by balvindar dhaliwal <ba...@hotmail.com>.
Hi,
Q1:server.xml :------------<?xml version='1.0' encoding='utf-8'?><Server port="8009" shutdown="xxxxx"> <Listener className="org.apache.catalina.security.SecurityListener" /> <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" /> <Listener className="org.apache.catalina.core.JasperListener" /> <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener" /> <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" /> <Listener className="org.apache.catalina.core.ThreadLocalLeakPreventionListener" /> <GlobalNamingResources> <Resource name="UserDatabase" auth="Container" type="org.apache.catalina.UserDatabase" description="User database that can be updated and saved" factory="org.apache.catalina.users.MemoryUserDatabaseFactory" pathname="conf/tomcat-users.xml" /> </GlobalNamingResources> <Service name="Catalina"> <Connector port="8443" protocol="HTTP/1.1" SSLEnabled="true" maxThreads="150" scheme="https" secure="true" clientAuth="false" sslProtocol="TLS" ciphers="TLS_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_DSS_WITH_AES_128_CBC_SHA,SSL_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA" keystoreFile="conf/xxxxxx-yy-zzzz.jks" keystorePass="xxxx" /> <Engine name="Catalina" defaultHost="localhost"> <Realm className="org.apache.catalina.realm.UserDatabaseRealm" digest="SHA" resourceName="UserDatabase"/> <Host name="localhost" appBase="webapps" unpackWARs="true" autoDeploy="true" xmlValidation="false" xmlNamespaceAware="false">
<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs" prefix="localhost_access_log." suffix=".txt" pattern="common" resolveHosts="false"/>
</Host> </Engine> </Service></Server>
Q2:We are using HTTPS.
Regards,Bal.
> Date: Mon, 3 Sep 2012 15:35:31 +0200
> From: aw@ice-sa.com
> To: users@tomcat.apache.org
> Subject: Re: Apache tomcat ( 7.0.19 ) stops processing user requests suddenly but works fine after restart.
>
> balvindar dhaliwal wrote:
> >
> >
> >
> >
> > Hi ,
> >
> >
> > We are using tomcat 7.0.19, for a jsf(richfaces),seam, jpa web application.
> > Here are the software versions and hardware infrastructure details.
> > Apache Tomact # 7.0.19
> > Java run time # 1.6.0_12
> > Server infrastructure # SunOS xxxxxxx 5.10 Generic_142900-03 sun4v sparc SUNW,Sun-Fire-T200
> > OS version # solaris 10.
> >
> > Issue Description #
> > We have noticed after about 2 months, the web requests stop being processed. We had observed this issue twice in the last 5 months.
> > We are using HTTP/1.1 Connector ( BIO ). The heap and permgen memory seem fine. There is no out of memory errors in the logs. We have used jvisualvm to monitor heap, permgen, threads and thread dump, AND THEY ALL LOOK OK. The thread dump looks fine and there are no deadlocks/stuck threads observed.
> > The localhost_access_log stops being updated. The connector port is still listening on SSL port 8443.
> > After we restart Tomcat, the new requests are processing fine.
> > Can you please advise other areas we should look into for investigation (any additional tracing).
> > Should we upgrade to tomcat 7.0.29? What could cause the Connector to stop processing requests on the port?
> > Further info: Heap(1/6th used), PermGen(1/2 used). GC is effectively reducing the heap. The fact that local access log is not being updated, suggests the request is not making to the Servlet container. We are using JPA Connection Pool, have not noticed any db connection issues. After the restart, everything works well.
> > The system logs seems to be fine.
> > Please find the attached sample thread dump copy when this issue was occurred.
> >
> > Please let me know if you need more information.
> >
>
> Q1: Can you copy and paste the server.xml of Tomcat here ? (removing any private information).
> Copy and paste in-line, the list strips most attachments.
>
> Q2: Above, you mention that the HTTPS connector on port 8443 is still listening. What
> about the HTTP connector on port ??? (usually 80).
> Or are all the requests HTTPS ?
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
Re: Apache tomcat ( 7.0.19 ) stops processing user requests suddenly
but works fine after restart.
Posted by André Warnier <aw...@ice-sa.com>.
balvindar dhaliwal wrote:
>
>
>
>
> Hi ,
>
>
>
>
>
>
>
>
>
>
>
> We are using tomcat 7.0.19, for a jsf(richfaces),seam, jpa web application.
> Here are the software versions and hardware infrastructure details.
> Apache Tomact # 7.0.19
> Java run time # 1.6.0_12
> Server infrastructure # SunOS xxxxxxx 5.10 Generic_142900-03 sun4v sparc SUNW,Sun-Fire-T200
> OS version # solaris 10.
>
> Issue Description #
> We have noticed after about 2 months, the web requests stop being processed. We had observed this issue twice in the last 5 months.
> We are using HTTP/1.1 Connector ( BIO ). The heap and permgen memory seem fine. There is no out of memory errors in the logs. We have used jvisualvm to monitor heap, permgen, threads and thread dump, AND THEY ALL LOOK OK. The thread dump looks fine and there are no deadlocks/stuck threads observed.
> The localhost_access_log stops being updated. The connector port is still listening on SSL port 8443.
> After we restart Tomcat, the new requests are processing fine.
> Can you please advise other areas we should look into for investigation (any additional tracing).
> Should we upgrade to tomcat 7.0.29? What could cause the Connector to stop processing requests on the port?
> Further info: Heap(1/6th used), PermGen(1/2 used). GC is effectively reducing the heap. The fact that local access log is not being updated, suggests the request is not making to the Servlet container. We are using JPA Connection Pool, have not noticed any db connection issues. After the restart, everything works well.
> The system logs seems to be fine.
> Please find the attached sample thread dump copy when this issue was occurred.
>
> Please let me know if you need more information.
>
Q1: Can you copy and paste the server.xml of Tomcat here ? (removing any private information).
Copy and paste in-line, the list strips most attachments.
Q2: Above, you mention that the HTTPS connector on port 8443 is still listening. What
about the HTTP connector on port ??? (usually 80).
Or are all the requests HTTPS ?
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org