You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Michal Singer <Mi...@expand.com> on 2008/12/07 06:43:18 UTC

tomcat gets stuck after a load for streams writing

I use tomcat 6 with nio connectors. i run a heavy load of data and after a
while the tomcat gets stuck
and stops processing requests from client.
I posted before regarding this and was asnwered that the problem may be in
my server
(http://www.nabble.com/Tomcat-request-processing-gets-stuck-td20526036.html)

However, i tested this and saw that during the stuck:

after tomcat stops receiving messages it will receive some kinds and some
not: 
1. if i use http client and write a string directly the tomcat will receive
the data 
2. if i use http client and write a stream, the tomcat will not receive the
data 
3. if i use wget and send a message to tomcat it will receive data 

which means, that after tomcat works with a load of data it gets stuck to
streams 
and not to strings (maybe related to the fact that i use nio connector??) 

Does any one have any idea on this problem with tomcat???? 
any help will be highly appreciated. 

Thanks, Michal 
-- 
View this message in context: http://www.nabble.com/tomcat-gets-stuck-after-a-load-for-streams-writing-tp20878338p20878338.html
Sent from the Tomcat - User mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Re: tomcat gets stuck after a load for streams writing

Posted by Filip Hanik - Dev Lists <de...@hanik.com>.
6.0.18
tons of bug fixes (and security)

Filip

Michal Singer wrote:
> * I see that the requests processors increase but it does not stop the stuck
> * The data base is slow on get connection but i checked the connections and
> i still have many left that can be used 
> * i see that the active threads count get to the limit before the stuck
> (according to jconsole) but
> after the stuck it goes down to 0 back again and it is still stuck so i
> don't believe the problem
> is no threads available for work
> * as for the stream requests: i use jmeter to create soap requests. i saw
> that it uses org.apache.commons.httpclient.methods.RequestEntity to write
> the requests as streams.
> when i use the jmeter to send after tomcat stops processing requests, it
> does not get the request
> when i use my own simulator which writes a regular string using http client
> - the tomcat does receive the request
>
>
> Caldarale, Charles R wrote:
>   
>>> From: Michal Singer [mailto:Michal.Singer@expand.com]
>>> Subject: RE: tomcat gets stuck after a load for streams writing
>>>
>>> i configured the maxThreads on the executor and didn't help.
>>>       
>> It should have let you get to 400 request processors, rather than the
>> default of 200 that you were seeing.
>>
>>     
>>> i also checked the thread dumps.
>>>       
>> And where do they show the threads are?  If you can force the hang with
>> any number of threads, then set maxThreads to a low value, and see where
>> they are executing when the stall occurs; things will be easier to figure
>> out when there's less to look at.
>>
>>     
>>> i still don't understand why it would cause
>>> a completey unrecovable tomcat stuck.
>>>       
>> If all the possible threads are already working on requests, no more
>> requests will be honored.  You could also be out of data base connections
>> (assuming you are using one), resulting in all threads waiting until a
>> connection becomes available.  If you webapp fails to return connections
>> to the pool in *all* circumstances, eventually you'll hang.
>>
>>     
>>> What also is weird is that it gets stuck only for stream
>>> writing. this does not seem like a problem in our server.
>>> why can i still send string messages?
>>>       
>> I don't know what you mean by "stream writing" vs "string messages".  If
>> these are actions performed by the client you're using to test Tomcat
>> with, it's irrelevant how the requests are constructed on the client end -
>> it's all HTTP by the time it comes over the wire.  If one of the
>> mechanisms is passing in bad headers, that's a different matter.  An
>> improperly terminated chunked request could tie up a thread for quite some
>> time.
>>
>>  - Chuck
>>
>>
>> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
>> MATERIAL and is thus for use only by the intended recipient. If you
>> received this in error, please contact the sender and delete the e-mail
>> and its attachments from all computers.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>>
>>     
>
>   


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


RE: tomcat gets stuck after a load for streams writing

Posted by Michal Singer <Mi...@expand.com>.
* I see that the requests processors increase but it does not stop the stuck
* The data base is slow on get connection but i checked the connections and
i still have many left that can be used 
* i see that the active threads count get to the limit before the stuck
(according to jconsole) but
after the stuck it goes down to 0 back again and it is still stuck so i
don't believe the problem
is no threads available for work
* as for the stream requests: i use jmeter to create soap requests. i saw
that it uses org.apache.commons.httpclient.methods.RequestEntity to write
the requests as streams.
when i use the jmeter to send after tomcat stops processing requests, it
does not get the request
when i use my own simulator which writes a regular string using http client
- the tomcat does receive the request


Caldarale, Charles R wrote:
> 
>> From: Michal Singer [mailto:Michal.Singer@expand.com]
>> Subject: RE: tomcat gets stuck after a load for streams writing
>>
>> i configured the maxThreads on the executor and didn't help.
> 
> It should have let you get to 400 request processors, rather than the
> default of 200 that you were seeing.
> 
>> i also checked the thread dumps.
> 
> And where do they show the threads are?  If you can force the hang with
> any number of threads, then set maxThreads to a low value, and see where
> they are executing when the stall occurs; things will be easier to figure
> out when there's less to look at.
> 
>> i still don't understand why it would cause
>> a completey unrecovable tomcat stuck.
> 
> If all the possible threads are already working on requests, no more
> requests will be honored.  You could also be out of data base connections
> (assuming you are using one), resulting in all threads waiting until a
> connection becomes available.  If you webapp fails to return connections
> to the pool in *all* circumstances, eventually you'll hang.
> 
>> What also is weird is that it gets stuck only for stream
>> writing. this does not seem like a problem in our server.
>> why can i still send string messages?
> 
> I don't know what you mean by "stream writing" vs "string messages".  If
> these are actions performed by the client you're using to test Tomcat
> with, it's irrelevant how the requests are constructed on the client end -
> it's all HTTP by the time it comes over the wire.  If one of the
> mechanisms is passing in bad headers, that's a different matter.  An
> improperly terminated chunked request could tie up a thread for quite some
> time.
> 
>  - Chuck
> 
> 
> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
> MATERIAL and is thus for use only by the intended recipient. If you
> received this in error, please contact the sender and delete the e-mail
> and its attachments from all computers.
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
> 
> 
> 

-- 
View this message in context: http://www.nabble.com/tomcat-gets-stuck-after-a-load-for-streams-writing-tp20878338p20890106.html
Sent from the Tomcat - User mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


RE: tomcat gets stuck after a load for streams writing

Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: Michal Singer [mailto:Michal.Singer@expand.com]
> Subject: RE: tomcat gets stuck after a load for streams writing
>
> i configured the maxThreads on the executor and didn't help.

It should have let you get to 400 request processors, rather than the default of 200 that you were seeing.

> i also checked the thread dumps.

And where do they show the threads are?  If you can force the hang with any number of threads, then set maxThreads to a low value, and see where they are executing when the stall occurs; things will be easier to figure out when there's less to look at.

> i still don't understand why it would cause
> a completey unrecovable tomcat stuck.

If all the possible threads are already working on requests, no more requests will be honored.  You could also be out of data base connections (assuming you are using one), resulting in all threads waiting until a connection becomes available.  If you webapp fails to return connections to the pool in *all* circumstances, eventually you'll hang.

> What also is weird is that it gets stuck only for stream
> writing. this does not seem like a problem in our server.
> why can i still send string messages?

I don't know what you mean by "stream writing" vs "string messages".  If these are actions performed by the client you're using to test Tomcat with, it's irrelevant how the requests are constructed on the client end - it's all HTTP by the time it comes over the wire.  If one of the mechanisms is passing in bad headers, that's a different matter.  An improperly terminated chunked request could tie up a thread for quite some time.

 - Chuck


THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


RE: tomcat gets stuck after a load for streams writing

Posted by Michal Singer <Mi...@expand.com>.
Hi. I did all that.
i configured the maxThreads on the executor and didn't help.
i also checked the thread dumps. the work on the server is heavy but i still
don't understand why it would
cause a completey unrecovable tomcat stuck.
even if the work is heavy on our server, the tomcat should not get stuck
receiving messages in stream.
after the stuck it never recovers until i restart the tomcat.
What also is weird is that it gets stuck only for stream writing. this does
not seem like a problem in our server. why can i still send string messages?
Thanks, Michal


Caldarale, Charles R wrote:
> 
>> From: Michal Singer [mailto:Michal.Singer@expand.com]
>> Subject: Re: tomcat gets stuck after a load for streams writing
>>
>> why? was there a bug regarding this issue which was fixed there?
> 
> Possibly, notably this one:
> https://issues.apache.org/bugzilla/show_bug.cgi?id=44620
> 
> However, if you're not burning up a CPU or two, that won't be the problem.
> 
> As you were told on the first go-round, the maxThreads setting in the
> <Connector> is being ignored, since you have an executor attribute for it. 
> The <Executor> is where you want the maxThreads specification.  RTFM.
> 
> Also as we asked for previously, where are the thread dumps?  You're
> unlikely to definitively resolve this unless you actually take the dumps
> and look at them.
> 
>  - Chuck
> 
> 
> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
> MATERIAL and is thus for use only by the intended recipient. If you
> received this in error, please contact the sender and delete the e-mail
> and its attachments from all computers.
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
> 
> 
> 

-- 
View this message in context: http://www.nabble.com/tomcat-gets-stuck-after-a-load-for-streams-writing-tp20878338p20889736.html
Sent from the Tomcat - User mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


RE: tomcat gets stuck after a load for streams writing

Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: Michal Singer [mailto:Michal.Singer@expand.com]
> Subject: Re: tomcat gets stuck after a load for streams writing
>
> why? was there a bug regarding this issue which was fixed there?

Possibly, notably this one:
https://issues.apache.org/bugzilla/show_bug.cgi?id=44620

However, if you're not burning up a CPU or two, that won't be the problem.

As you were told on the first go-round, the maxThreads setting in the <Connector> is being ignored, since you have an executor attribute for it.  The <Executor> is where you want the maxThreads specification.  RTFM.

Also as we asked for previously, where are the thread dumps?  You're unlikely to definitively resolve this unless you actually take the dumps and look at them.

 - Chuck


THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Re: tomcat gets stuck after a load for streams writing

Posted by Michal Singer <Mi...@expand.com>.
why? was there a bug regarding this issue which was fixed there?
thanks, Michal



markt-2 wrote:
> 
> Michal Singer wrote:
>> Tomcat version? 6.0.14
> 
> Try 6.0.18
> 
> Mark
> 
>> JDK version? 1.5.0_12
>> OS? Microsoft Windows Server 2003
>> Connector configuration settings?
>> <Connector port="8080" executor="AgentExecutor"
>> protocol="org.apache.coyote.http11.Http11NioProtocol"
>> maxKeepAliveRequests="1" maxThreads="400" processCache="400"
>> connectionTimeout="20000" acceptorThreadCount="1" redirectPort="8443" />
>> <Executor name="AgentExecutor" threadPriority="Thread.NORM_PRIORITY"
>> namePrefix="agent"/>
>> 
>> 
>> markt-2 wrote:
>>> Michal Singer wrote:
>>>> I use tomcat 6 with nio connectors. i run a heavy load of data and
>>>> after
>>>> a
>>>> while the tomcat gets stuck
>>>> and stops processing requests from client.
>>>> I posted before regarding this and was asnwered that the problem may be
>>>> in
>>>> my server
>>>> (http://www.nabble.com/Tomcat-request-processing-gets-stuck-td20526036.html)
>>>>
>>>> However, i tested this and saw that during the stuck:
>>>>
>>>> after tomcat stops receiving messages it will receive some kinds and
>>>> some
>>>> not: 
>>>> 1. if i use http client and write a string directly the tomcat will
>>>> receive
>>>> the data 
>>>> 2. if i use http client and write a stream, the tomcat will not receive
>>>> the
>>>> data 
>>>> 3. if i use wget and send a message to tomcat it will receive data 
>>>>
>>>> which means, that after tomcat works with a load of data it gets stuck
>>>> to
>>>> streams 
>>>> and not to strings (maybe related to the fact that i use nio
>>>> connector??) 
>>>>
>>>> Does any one have any idea on this problem with tomcat???? 
>>>> any help will be highly appreciated.
>>> Tomcat version?
>>> JDK version?
>>> OS?
>>> Connector configuration settings?
>>>
>>> Mark
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>>
>>>
>> 
> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
> 
> 
> 

-- 
View this message in context: http://www.nabble.com/tomcat-gets-stuck-after-a-load-for-streams-writing-tp20878338p20881911.html
Sent from the Tomcat - User mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Re: tomcat gets stuck after a load for streams writing

Posted by Mark Thomas <ma...@apache.org>.
Michal Singer wrote:
> Tomcat version? 6.0.14

Try 6.0.18

Mark

> JDK version? 1.5.0_12
> OS? Microsoft Windows Server 2003
> Connector configuration settings?
> <Connector port="8080" executor="AgentExecutor"
> protocol="org.apache.coyote.http11.Http11NioProtocol"
> maxKeepAliveRequests="1" maxThreads="400" processCache="400"
> connectionTimeout="20000" acceptorThreadCount="1" redirectPort="8443" />
> <Executor name="AgentExecutor" threadPriority="Thread.NORM_PRIORITY"
> namePrefix="agent"/>
> 
> 
> markt-2 wrote:
>> Michal Singer wrote:
>>> I use tomcat 6 with nio connectors. i run a heavy load of data and after
>>> a
>>> while the tomcat gets stuck
>>> and stops processing requests from client.
>>> I posted before regarding this and was asnwered that the problem may be
>>> in
>>> my server
>>> (http://www.nabble.com/Tomcat-request-processing-gets-stuck-td20526036.html)
>>>
>>> However, i tested this and saw that during the stuck:
>>>
>>> after tomcat stops receiving messages it will receive some kinds and some
>>> not: 
>>> 1. if i use http client and write a string directly the tomcat will
>>> receive
>>> the data 
>>> 2. if i use http client and write a stream, the tomcat will not receive
>>> the
>>> data 
>>> 3. if i use wget and send a message to tomcat it will receive data 
>>>
>>> which means, that after tomcat works with a load of data it gets stuck to
>>> streams 
>>> and not to strings (maybe related to the fact that i use nio connector??) 
>>>
>>> Does any one have any idea on this problem with tomcat???? 
>>> any help will be highly appreciated.
>> Tomcat version?
>> JDK version?
>> OS?
>> Connector configuration settings?
>>
>> Mark
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>>
> 



---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Re: tomcat gets stuck after a load for streams writing

Posted by Michal Singer <Mi...@expand.com>.
Tomcat version? 6.0.14
JDK version? 1.5.0_12
OS? Microsoft Windows Server 2003
Connector configuration settings?
<Connector port="8080" executor="AgentExecutor"
protocol="org.apache.coyote.http11.Http11NioProtocol"
maxKeepAliveRequests="1" maxThreads="400" processCache="400"
connectionTimeout="20000" acceptorThreadCount="1" redirectPort="8443" />
<Executor name="AgentExecutor" threadPriority="Thread.NORM_PRIORITY"
namePrefix="agent"/>


markt-2 wrote:
> 
> Michal Singer wrote:
>> I use tomcat 6 with nio connectors. i run a heavy load of data and after
>> a
>> while the tomcat gets stuck
>> and stops processing requests from client.
>> I posted before regarding this and was asnwered that the problem may be
>> in
>> my server
>> (http://www.nabble.com/Tomcat-request-processing-gets-stuck-td20526036.html)
>> 
>> However, i tested this and saw that during the stuck:
>> 
>> after tomcat stops receiving messages it will receive some kinds and some
>> not: 
>> 1. if i use http client and write a string directly the tomcat will
>> receive
>> the data 
>> 2. if i use http client and write a stream, the tomcat will not receive
>> the
>> data 
>> 3. if i use wget and send a message to tomcat it will receive data 
>> 
>> which means, that after tomcat works with a load of data it gets stuck to
>> streams 
>> and not to strings (maybe related to the fact that i use nio connector??) 
>> 
>> Does any one have any idea on this problem with tomcat???? 
>> any help will be highly appreciated.
> 
> Tomcat version?
> JDK version?
> OS?
> Connector configuration settings?
> 
> Mark
> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
> 
> 
> 

-- 
View this message in context: http://www.nabble.com/tomcat-gets-stuck-after-a-load-for-streams-writing-tp20878338p20881721.html
Sent from the Tomcat - User mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Re: tomcat gets stuck after a load for streams writing

Posted by Mark Thomas <ma...@apache.org>.
Michal Singer wrote:
> I use tomcat 6 with nio connectors. i run a heavy load of data and after a
> while the tomcat gets stuck
> and stops processing requests from client.
> I posted before regarding this and was asnwered that the problem may be in
> my server
> (http://www.nabble.com/Tomcat-request-processing-gets-stuck-td20526036.html)
> 
> However, i tested this and saw that during the stuck:
> 
> after tomcat stops receiving messages it will receive some kinds and some
> not: 
> 1. if i use http client and write a string directly the tomcat will receive
> the data 
> 2. if i use http client and write a stream, the tomcat will not receive the
> data 
> 3. if i use wget and send a message to tomcat it will receive data 
> 
> which means, that after tomcat works with a load of data it gets stuck to
> streams 
> and not to strings (maybe related to the fact that i use nio connector??) 
> 
> Does any one have any idea on this problem with tomcat???? 
> any help will be highly appreciated.

Tomcat version?
JDK version?
OS?
Connector configuration settings?

Mark



---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org