You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Radha Krishnan D <Ra...@ivycomptech.com> on 2013/02/18 17:56:52 UTC
HTTP rejections and too many open files exception on tomcat
Hi,
We are running into too many open issues exception on our production systems for the last
This email and any attachments are confidential, and may be legally privileged and protected by copyright. If you are not the intended recipient dissemination or copying of this email is prohibited. If you have received this in error, please notify the sender by replying by email and then delete the email completely from your system. Any views or opinions are solely those of the sender. This communication is not intended to form a binding contract unless expressly indicated to the contrary and properly authorised. Any actions taken on the basis of this email are at the recipient's own risk.
Re: HTTP rejections and too many open files exception on tomcat
Posted by Mark Thomas <ma...@apache.org>.
On 18/02/2013 21:14, Pid wrote:
> On 18/02/2013 17:05, Radha Krishnan D wrote:
>> Hi
>>
>> Am sorry, clicked submit by mistake
>>
>> The below is the actual problem
>>
>>
>> We are running into too many open files exception on our production tomcat server for the last 2-3 days.
>>
>> We have max threads as 150 and acceptCount as 100 and at one point of time all the 250 threads are busy processing requests (due to application related issues)
>
> You only have 150 threads, not 250. 150 is not a large number & is in
> fact lower than the default.
>
> At 50-60 requests per second, you must serve each request within 0.3s
That number (0.3) is around an order of magnitude too small.
Mark
> to
> avoid queuing requests (ie, using the acceptCount queue). You should
> avoid queueing requests.
>
>
>> At peak times, when all the threads are busy, new requests are rejected by tomcat.
>
> Expected.
>
>
>> Nearly 40 mins to 1 hr after the rejections begin, there are too many open files exception coming on the tomcat server..
>
> Maybe this is the real peak.
>
>
>> The number of file descriptors is 16384 and we are getting 50-60 requests per sec.
>>
>> What can be the root cause here.
>
> Too many possible options, not enough info.
>
>
>> After rejecting a request, is the socket not closed immediately and this results in pile up of open files or is there some other reason ?
>>
>> Can any one help us in this regard ?
>
> Please post the Connector configuration from server.xml. You may
> obscure/remove passwords if present.
>
>
> p
>
>> Thanks,
>> D.Radhakrishnan
>>
>>
>> -----Original Message-----
>> From: André Warnier [mailto:aw@ice-sa.com]
>> Sent: 18 February 2013 22:32
>> To: Tomcat Users List
>> Subject: Re: HTTP rejections and too many open files exception on tomcat
>>
>> Radha Krishnan D wrote:
>>> Hi,
>>>
>>> We are running into too many open issues exception on our production systems for the last
>>
>> With a lot of guesswork ..
>>
>> Google for "ulimit" ?
>> Get a bigger server ?
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>> This email and any attachments are confidential, and may be legally privileged and protected by copyright. If you are not the intended recipient dissemination or copying of this email is prohibited. If you have received this in error, please notify the sender by replying by email and then delete the email completely from your system. Any views or opinions are solely those of the sender. This communication is not intended to form a binding contract unless expressly indicated to the contrary and properly authorised. Any actions taken on the basis of this email are at the recipient's own risk.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: HTTP rejections and too many open files exception on tomcat
Posted by Pid <pi...@pidster.com>.
On 18/02/2013 17:05, Radha Krishnan D wrote:
> Hi
>
> Am sorry, clicked submit by mistake
>
> The below is the actual problem
>
>
> We are running into too many open files exception on our production tomcat server for the last 2-3 days.
>
> We have max threads as 150 and acceptCount as 100 and at one point of time all the 250 threads are busy processing requests (due to application related issues)
You only have 150 threads, not 250. 150 is not a large number & is in
fact lower than the default.
At 50-60 requests per second, you must serve each request within 0.3s to
avoid queuing requests (ie, using the acceptCount queue). You should
avoid queueing requests.
> At peak times, when all the threads are busy, new requests are rejected by tomcat.
Expected.
> Nearly 40 mins to 1 hr after the rejections begin, there are too many open files exception coming on the tomcat server..
Maybe this is the real peak.
> The number of file descriptors is 16384 and we are getting 50-60 requests per sec.
>
> What can be the root cause here.
Too many possible options, not enough info.
> After rejecting a request, is the socket not closed immediately and this results in pile up of open files or is there some other reason ?
>
> Can any one help us in this regard ?
Please post the Connector configuration from server.xml. You may
obscure/remove passwords if present.
p
> Thanks,
> D.Radhakrishnan
>
>
> -----Original Message-----
> From: André Warnier [mailto:aw@ice-sa.com]
> Sent: 18 February 2013 22:32
> To: Tomcat Users List
> Subject: Re: HTTP rejections and too many open files exception on tomcat
>
> Radha Krishnan D wrote:
>> Hi,
>>
>> We are running into too many open issues exception on our production systems for the last
>
> With a lot of guesswork ..
>
> Google for "ulimit" ?
> Get a bigger server ?
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
> This email and any attachments are confidential, and may be legally privileged and protected by copyright. If you are not the intended recipient dissemination or copying of this email is prohibited. If you have received this in error, please notify the sender by replying by email and then delete the email completely from your system. Any views or opinions are solely those of the sender. This communication is not intended to form a binding contract unless expressly indicated to the contrary and properly authorised. Any actions taken on the basis of this email are at the recipient's own risk.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
--
[key:62590808]
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: HTTP rejections and too many open files exception on tomcat
Posted by André Warnier <aw...@ice-sa.com>.
André Warnier wrote:
> Radha Krishnan D wrote:
>> Hi
>>
>> Am sorry, clicked submit by mistake
>>
>> The below is the actual problem
>>
>>
>> We are running into too many open files exception on our production
>> tomcat server for the last 2-3 days.
>>
>> We have max threads as 150 and acceptCount as 100 and at one point of
>> time all the 250 threads are busy processing requests (due to
>> application related issues)
>>
>> At peak times, when all the threads are busy, new requests are
>> rejected by tomcat.
>>
>> Nearly 40 mins to 1 hr after the rejections begin, there are too many
>> open files exception coming on the tomcat server..
>>
>> The number of file descriptors is 16384 and we are getting 50-60
>> requests per sec.
>>
>> What can be the root cause here. After rejecting a request, is the
>> socket not closed immediately and this results in pile up of open
>> files or is there some other reason ?
>>
>> Can any one help us in this regard ?
>>
>>
>
> platform/OS version ?
> Java version ?
> Tomcat version (x.y.z) ?
>
> "The number of file descriptors is 16384.." : where does that number
> come from ?
> And are they sockets or other things ?
>
> "After rejecting a request, is the socket not closed immediately and
> this results in pile up of open files or is there some other reason ?"
>
> There are some "states" in TCP/IP, that a connection goes through even
> after a close() has been issued by the server side. There are some
> delays there, and maybe that is a source for your problem. Google for
> "TCP state diagram".
> Use "netstat" and "lsof" (under Unix/Linux) to find out more.
>
See also this
http://stackoverflow.com/questions/34588/how-do-i-change-the-number-of-open-files-limit-in-linux
The point is : your system(s) may really be too small to handle the load that comes to
them; or your application may be too slow to respond; or you may have KeepAlive set too
high for your own good, or any of the parameters here that can be used to "modulate" what
happens could be wrong for your case :
https://tomcat.apache.org/tomcat-7.0-doc/config/http.html#Standard_Implementation
But all these parameters are to some extent interdependent, and before "tuning" any of
them, you would need to have (and provide us with) a better picture of what really
happens. If you start playing around with them without really understanding what is
happening and what you are doing, you'll probably make the situation worse, not better.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: HTTP rejections and too many open files exception on tomcat
Posted by André Warnier <aw...@ice-sa.com>.
Radha Krishnan D wrote:
> Hi
>
> Am sorry, clicked submit by mistake
>
> The below is the actual problem
>
>
> We are running into too many open files exception on our production tomcat server for the last 2-3 days.
>
> We have max threads as 150 and acceptCount as 100 and at one point of time all the 250 threads are busy processing requests (due to application related issues)
>
> At peak times, when all the threads are busy, new requests are rejected by tomcat.
>
> Nearly 40 mins to 1 hr after the rejections begin, there are too many open files exception coming on the tomcat server..
>
> The number of file descriptors is 16384 and we are getting 50-60 requests per sec.
>
> What can be the root cause here. After rejecting a request, is the socket not closed immediately and this results in pile up of open files or is there some other reason ?
>
> Can any one help us in this regard ?
>
>
platform/OS version ?
Java version ?
Tomcat version (x.y.z) ?
"The number of file descriptors is 16384.." : where does that number come from ?
And are they sockets or other things ?
"After rejecting a request, is the socket not closed immediately and this results in pile
up of open files or is there some other reason ?"
There are some "states" in TCP/IP, that a connection goes through even after a close() has
been issued by the server side. There are some delays there, and maybe that is a source
for your problem. Google for "TCP state diagram".
Use "netstat" and "lsof" (under Unix/Linux) to find out more.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: HTTP rejections and too many open files exception on tomcat
Posted by Radha Krishnan D <Ra...@ivycomptech.com>.
Hi
Am sorry, clicked submit by mistake
The below is the actual problem
We are running into too many open files exception on our production tomcat server for the last 2-3 days.
We have max threads as 150 and acceptCount as 100 and at one point of time all the 250 threads are busy processing requests (due to application related issues)
At peak times, when all the threads are busy, new requests are rejected by tomcat.
Nearly 40 mins to 1 hr after the rejections begin, there are too many open files exception coming on the tomcat server..
The number of file descriptors is 16384 and we are getting 50-60 requests per sec.
What can be the root cause here. After rejecting a request, is the socket not closed immediately and this results in pile up of open files or is there some other reason ?
Can any one help us in this regard ?
Thanks,
D.Radhakrishnan
-----Original Message-----
From: André Warnier [mailto:aw@ice-sa.com]
Sent: 18 February 2013 22:32
To: Tomcat Users List
Subject: Re: HTTP rejections and too many open files exception on tomcat
Radha Krishnan D wrote:
> Hi,
>
> We are running into too many open issues exception on our production systems for the last
With a lot of guesswork ..
Google for "ulimit" ?
Get a bigger server ?
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
This email and any attachments are confidential, and may be legally privileged and protected by copyright. If you are not the intended recipient dissemination or copying of this email is prohibited. If you have received this in error, please notify the sender by replying by email and then delete the email completely from your system. Any views or opinions are solely those of the sender. This communication is not intended to form a binding contract unless expressly indicated to the contrary and properly authorised. Any actions taken on the basis of this email are at the recipient's own risk.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: HTTP rejections and too many open files exception on tomcat
Posted by André Warnier <aw...@ice-sa.com>.
Radha Krishnan D wrote:
> Hi,
>
> We are running into too many open issues exception on our production systems for the last
With a lot of guesswork ..
Google for "ulimit" ?
Get a bigger server ?
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org