You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by jikai <ji...@youxigu.com> on 2010/07/10 07:32:54 UTC
when memory leak, tomcat can't accept requests?
Hi all:
Out web site use Nginx infront of tomcat6.0.28, nio connector, in heavy
load running about 24 hours, tomcat can't accept request, we must restart
tomcat .
check nginx error log, there are lots of "Connection timed out", print
tomcat jstack, found most tomcat threads is "TIMED_WAITING", like this:
"http-10.132.23.74-8090-exec-54" daemon prio=10 tid=0x00002aab36ecec00
nid=0x3175 waiting on condition [0x00000000498c0000..0x00000000498c0e20]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00002aaaba3cf768> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
06)
at java.lang.Thread.run(Thread.java:619)
print jmap, we found we have lots of memory leak:
num #instances #bytes class name
----------------------------------------------
1: 7772927 336536416 [C
2: 7774657 310986280 java.lang.String
3: 398536 162602688 com.manu.dynasty.base.domain.User
4: 3460146 138405840 java.sql.Timestamp
5: 2334971 112078608 java.util.HashMap$Entry
6: 792095 101388160 com.manu.dynasty.function.domain.Friend
.
108: 437 80408 java.net.SocksSocketImpl
Objects num in memory like "com.manu.dynasty.base.domain.User",
"com.manu.dynasty.function.domain.Friend" is too big than normal situation,
And the num of "java.net.SocksSocketImpl" is too small.
So my question is the memory leak cause tomcat can't create Socket to accept
nginx's request?
Our app is running in product envionment now , and every day we have to
restart tomcat, so any help is very appreciated
Re: when memory leak, tomcat can't accept requests?
Posted by André Warnier <aw...@ice-sa.com>.
Pid wrote:
> On 10/07/2010 10:19, jikai wrote:
>>> Because it's already saturated with requests? No server has an infinite
>>> capacity. How many threads did jstack report were running?
>>>
>>> Can you connect with JMX and see what state the connector is in?
>>>
>>> Are you using an Executor in combination with your Connector?
>> Thanks for your quick reply.
>> we didn't use Executor, and there is no JMX with tomcat. When beginning of
>> error occurs, there is more than 600 worker threads(most of them is
>> waiting), an hour later, there's only 50 worker threads left(I think because
>> of idle, threads were killed by pool), BUT tomcat still can't process
>> requests, nginx still report connection timeout for most request, I can't
>> understand why this happened?
>
> Neither can I with the information you've provided so far. The thread
> dump you've posted doesn't have enough information in it to determine
> what's going on.
>
> Stick to using the JIO connector for now, and try using an Executor to
> manage the pool.
>
> Is your application dependant on a database?
>
> You didn't answer: are you storing the objects you referred to in the
> user session?
>
>
And maybe worth having a look at the connectionTimeout and keepAliveTimeout settings.
See definitions here : http://tomcat.apache.org/tomcat-6.0-doc/config/http.html
By default, keepAliveTimeout is set the same as connectionTimeout (in this case 30 s).
That means that, after serving the last request of any given client, each thread is going
to sit waiting for a possible next request on the same connection, for up to 30 s.
That is ok if the pages you serve, typically, have lots of image links for example. The
keepAlive then allows the browser to request the embedded images using the same
connection, since it is still alive.
But if your pages typically do not have such embedded objects, then it is finally a waste
of threads (and associated resources), who could be better used than just waiting for
requests that will never come.
I personally find 30 s way too high for most cases. If a browser is going to request an
inline image, it will do so within maximum a few seconds of receiving the original page.
KeepAliveTimeout applies to the time between the initial TCP connection setup by the
browser, and the moment it consents to send its HTTP request over that connection. I
would think that unless you have *very* slow clients, a time of a few seconds should
suffice. Otherwise, you also open yourself to DoS attacks : a client makes a connection,
and then willfully waits before sending a request. Enough of those and your server would
be paralysed.
Summary : what about setting both connectionTimeout and KeepAliveTimeout to "5000" (5 s)
explicitly, and see how many waiting threads you still have then ?
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: when memory leak, tomcat can't accept requests?
Posted by Pid <pi...@pidster.com>.
On 10/07/2010 10:19, jikai wrote:
>> Because it's already saturated with requests? No server has an infinite
>> capacity. How many threads did jstack report were running?
>>
>> Can you connect with JMX and see what state the connector is in?
>>
>> Are you using an Executor in combination with your Connector?
>
> Thanks for your quick reply.
> we didn't use Executor, and there is no JMX with tomcat. When beginning of
> error occurs, there is more than 600 worker threads(most of them is
> waiting), an hour later, there's only 50 worker threads left(I think because
> of idle, threads were killed by pool), BUT tomcat still can't process
> requests, nginx still report connection timeout for most request, I can't
> understand why this happened?
Neither can I with the information you've provided so far. The thread
dump you've posted doesn't have enough information in it to determine
what's going on.
Stick to using the JIO connector for now, and try using an Executor to
manage the pool.
Is your application dependant on a database?
You didn't answer: are you storing the objects you referred to in the
user session?
p
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
Re: when memory leak, tomcat can't accept requests?
Posted by jikai <ji...@youxigu.com>.
> Because it's already saturated with requests? No server has an infinite
> capacity. How many threads did jstack report were running?
>
> Can you connect with JMX and see what state the connector is in?
>
> Are you using an Executor in combination with your Connector?
Thanks for your quick reply.
we didn't use Executor, and there is no JMX with tomcat. When beginning of
error occurs, there is more than 600 worker threads(most of them is
waiting), an hour later, there's only 50 worker threads left(I think because
of idle, threads were killed by pool), BUT tomcat still can't process
requests, nginx still report connection timeout for most request, I can't
understand why this happened?
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: when memory leak, tomcat can't accept requests?
Posted by Pid <pi...@pidster.com>.
On 10/07/2010 08:55, jikai wrote:
> I give more details when error occurs:
> 1) nginx suddently report lots of " Connection timed out" like this
> 2010/07/09 19:26:37 [error] 11558#0: *3067544729 connect() failed (110:
> Connection timed out) while connecting to upstream, ...
>
> 2) there are still a few requests can connect tomcat and well processed, but
> most requests from nginx is connection timeout. the cpu and net traffic of
> tomcat server decreased a lot
>
> 3) tomcat acceptor and poller thread is RUNNABLE, but most worker thread is
> "TIMED_WAITING"
> "http-10.132.23.74-8090-ClientPoller" daemon prio=10 tid=0x00002aab355e2c00
> nid=0x5afd runnable [0x0000000044169000..0x0000000044169e20]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
> at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x00002aaaba1d8c90> (a sun.nio.ch.Util$1)
> - locked <0x00002aaaba1d8c78> (a
> java.util.Collections$UnmodifiableSet)
> - locked <0x00002aaaba1d8a98> (a sun.nio.ch.EPollSelectorImpl)
> at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> at
> org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1473)
> at java.lang.Thread.run(Thread.java:619)
>
> "http-10.132.23.74-8090-Acceptor-0" daemon prio=10 tid=0x00002aab355df800
> nid=0x5afc runnable [0x0000000044068000..0x0000000044068ca0]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
> - locked <0x00002aaab8936bd8> (a java.lang.Object)
> at
> org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:1198)
> at java.lang.Thread.run(Thread.java:619)
>
> 4) there is no "java.lang.OutOfMemoryError" in catalina.out
>
> 5) I tried JIO instead of NIO, but error still here
>
> 6) my NIO configuration is below:
> <Connector port="8090" enableLookups="false" redirectPort="8443"
> protocol="org.apache.coyote.http11.Http11NioProtocol"
> connectionTimeout="30000" maxThreads="1000" acceptCount="800"
> URIEncoding="UTF-8" />
>
> any other reason cause tomcat can't accept requests?
Because it's already saturated with requests? No server has an infinite
capacity. How many threads did jstack report were running?
Can you connect with JMX and see what state the connector is in?
Are you using an Executor in combination with your Connector?
p
> On 10/07/2010 06:32, jikai wrote:
>> Hi all:
>>
>> Out web site use Nginx infront of tomcat6.0.28, nio connector, in
> heavy
>> load running about 24 hours, tomcat can't accept request, we must restart
>> tomcat .
>>
>> check nginx error log, there are lots of "Connection timed out", print
>> tomcat jstack, found most tomcat threads is "TIMED_WAITING", like this:
>>
>>
>>
>> "http-10.132.23.74-8090-exec-54" daemon prio=10 tid=0x00002aab36ecec00
>> nid=0x3175 waiting on condition [0x00000000498c0000..0x00000000498c0e20]
>>
>> java.lang.Thread.State: TIMED_WAITING (parking)
>>
>> at sun.misc.Unsafe.park(Native Method)
>>
>> - parking to wait for <0x00002aaaba3cf768> (a
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>
>> at
>> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>>
>> at
>>
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
>> anos(AbstractQueuedSynchronizer.java:1963)
>>
>> at
>>
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>>
>> at
>>
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>>
>> at
>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
>> 06)
>>
>> at java.lang.Thread.run(Thread.java:619)
>>
>>
>>
>> print jmap, we found we have lots of memory leak:
>>
>>
>>
>> num #instances #bytes class name
>>
>> ----------------------------------------------
>>
>> 1: 7772927 336536416 [C
>>
>> 2: 7774657 310986280 java.lang.String
>>
>> 3: 398536 162602688 com.manu.dynasty.base.domain.User
>>
>> 4: 3460146 138405840 java.sql.Timestamp
>>
>> 5: 2334971 112078608 java.util.HashMap$Entry
>>
>> 6: 792095 101388160
> com.manu.dynasty.function.domain.Friend
>>
>> .
>>
>> 108: 437 80408 java.net.SocksSocketImpl
>>
>>
>>
>> Objects num in memory like "com.manu.dynasty.base.domain.User",
>> "com.manu.dynasty.function.domain.Friend" is too big than normal
> situation,
>>
>> And the num of "java.net.SocksSocketImpl" is too small.
>>
>>
>>
>> So my question is the memory leak cause tomcat can't create Socket to
> accept
>> nginx's request?
>
> A memory leak will mean that eventually the JVM is unable to allocate
> memory for the creation of any new objects, and yes that will include
> network connections.
>
> Work out what's holding onto those objects. Are you storing them in the
> user session perhaps?
>
>
> p
>
>> Our app is running in product envionment now , and every day we have to
>> restart tomcat, so any help is very appreciated
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
Re: when memory leak, tomcat can't accept requests?
Posted by jikai <ji...@youxigu.com>.
I give more details when error occurs:
1) nginx suddently report lots of " Connection timed out" like this
2010/07/09 19:26:37 [error] 11558#0: *3067544729 connect() failed (110:
Connection timed out) while connecting to upstream, ...
2) there are still a few requests can connect tomcat and well processed, but
most requests from nginx is connection timeout. the cpu and net traffic of
tomcat server decreased a lot
3) tomcat acceptor and poller thread is RUNNABLE, but most worker thread is
"TIMED_WAITING"
"http-10.132.23.74-8090-ClientPoller" daemon prio=10 tid=0x00002aab355e2c00
nid=0x5afd runnable [0x0000000044169000..0x0000000044169e20]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x00002aaaba1d8c90> (a sun.nio.ch.Util$1)
- locked <0x00002aaaba1d8c78> (a
java.util.Collections$UnmodifiableSet)
- locked <0x00002aaaba1d8a98> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
at
org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1473)
at java.lang.Thread.run(Thread.java:619)
"http-10.132.23.74-8090-Acceptor-0" daemon prio=10 tid=0x00002aab355df800
nid=0x5afc runnable [0x0000000044068000..0x0000000044068ca0]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
- locked <0x00002aaab8936bd8> (a java.lang.Object)
at
org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:1198)
at java.lang.Thread.run(Thread.java:619)
4) there is no "java.lang.OutOfMemoryError" in catalina.out
5) I tried JIO instead of NIO, but error still here
6) my NIO configuration is below:
<Connector port="8090" enableLookups="false" redirectPort="8443"
protocol="org.apache.coyote.http11.Http11NioProtocol"
connectionTimeout="30000" maxThreads="1000" acceptCount="800"
URIEncoding="UTF-8" />
any other reason cause tomcat can't accept requests?
On 10/07/2010 06:32, jikai wrote:
> Hi all:
>
> Out web site use Nginx infront of tomcat6.0.28, nio connector, in
heavy
> load running about 24 hours, tomcat can't accept request, we must restart
> tomcat .
>
> check nginx error log, there are lots of "Connection timed out", print
> tomcat jstack, found most tomcat threads is "TIMED_WAITING", like this:
>
>
>
> "http-10.132.23.74-8090-exec-54" daemon prio=10 tid=0x00002aab36ecec00
> nid=0x3175 waiting on condition [0x00000000498c0000..0x00000000498c0e20]
>
> java.lang.Thread.State: TIMED_WAITING (parking)
>
> at sun.misc.Unsafe.park(Native Method)
>
> - parking to wait for <0x00002aaaba3cf768> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>
> at
>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
>
> at
>
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>
> at
>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>
> at
>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
>
> at java.lang.Thread.run(Thread.java:619)
>
>
>
> print jmap, we found we have lots of memory leak:
>
>
>
> num #instances #bytes class name
>
> ----------------------------------------------
>
> 1: 7772927 336536416 [C
>
> 2: 7774657 310986280 java.lang.String
>
> 3: 398536 162602688 com.manu.dynasty.base.domain.User
>
> 4: 3460146 138405840 java.sql.Timestamp
>
> 5: 2334971 112078608 java.util.HashMap$Entry
>
> 6: 792095 101388160
com.manu.dynasty.function.domain.Friend
>
> .
>
> 108: 437 80408 java.net.SocksSocketImpl
>
>
>
> Objects num in memory like "com.manu.dynasty.base.domain.User",
> "com.manu.dynasty.function.domain.Friend" is too big than normal
situation,
>
> And the num of "java.net.SocksSocketImpl" is too small.
>
>
>
> So my question is the memory leak cause tomcat can't create Socket to
accept
> nginx's request?
A memory leak will mean that eventually the JVM is unable to allocate
memory for the creation of any new objects, and yes that will include
network connections.
Work out what's holding onto those objects. Are you storing them in the
user session perhaps?
p
> Our app is running in product envionment now , and every day we have to
> restart tomcat, so any help is very appreciated
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: when memory leak, tomcat can't accept requests?
Posted by Pid <pi...@pidster.com>.
On 10/07/2010 06:32, jikai wrote:
> Hi all:
>
> Out web site use Nginx infront of tomcat6.0.28, nio connector, in heavy
> load running about 24 hours, tomcat can't accept request, we must restart
> tomcat .
>
> check nginx error log, there are lots of "Connection timed out", print
> tomcat jstack, found most tomcat threads is "TIMED_WAITING", like this:
>
>
>
> "http-10.132.23.74-8090-exec-54" daemon prio=10 tid=0x00002aab36ecec00
> nid=0x3175 waiting on condition [0x00000000498c0000..0x00000000498c0e20]
>
> java.lang.Thread.State: TIMED_WAITING (parking)
>
> at sun.misc.Unsafe.park(Native Method)
>
> - parking to wait for <0x00002aaaba3cf768> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
>
> at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>
> at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:944)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 06)
>
> at java.lang.Thread.run(Thread.java:619)
>
>
>
> print jmap, we found we have lots of memory leak:
>
>
>
> num #instances #bytes class name
>
> ----------------------------------------------
>
> 1: 7772927 336536416 [C
>
> 2: 7774657 310986280 java.lang.String
>
> 3: 398536 162602688 com.manu.dynasty.base.domain.User
>
> 4: 3460146 138405840 java.sql.Timestamp
>
> 5: 2334971 112078608 java.util.HashMap$Entry
>
> 6: 792095 101388160 com.manu.dynasty.function.domain.Friend
>
> .
>
> 108: 437 80408 java.net.SocksSocketImpl
>
>
>
> Objects num in memory like "com.manu.dynasty.base.domain.User",
> "com.manu.dynasty.function.domain.Friend" is too big than normal situation,
>
> And the num of "java.net.SocksSocketImpl" is too small.
>
>
>
> So my question is the memory leak cause tomcat can't create Socket to accept
> nginx's request?
A memory leak will mean that eventually the JVM is unable to allocate
memory for the creation of any new objects, and yes that will include
network connections.
Work out what's holding onto those objects. Are you storing them in the
user session perhaps?
p
> Our app is running in product envionment now , and every day we have to
> restart tomcat, so any help is very appreciated
>
>
>
>