You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by feedly team <fe...@gmail.com> on 2009/04/08 08:23:35 UTC
apache/tomcat communication issues (502 response)
We are running apache and tomcat on the same machine (using the http
connector) and logging requests in both. Occasionally (maybe 1% of
requests) we see 502 response in the apache log spread fairly evenly
throughout the day. these requests don't appear in the tomcat log
produced by the access valve. in the apache error log I see:
proxy: error reading status line from remote server 127.0.0.1
proxy: Error reading from remote server returned by [url]
The requests are a mix of methods (GET/POST/PUT) of pretty small
bodies and responses, generally under 1k.
We added request time (%T) in the apache logs and many of the requests
have 0 response time so I don't think it's a timeout issue. Reading
the tomcat connector documentation, I tried increasing the maxThreads
to 100 and backlog to 1000 in the http connector since it seemed like
with the default settings a burst of traffic may cause this issue.
This didn't solve the problem. I have done periodic thread dumps and
it looks like the number of http threads doesn't get past 80.
concerning machine resources, cpu is not loaded at all (~10%
utilization). i checked file descriptor usage, it's about 400 and
there is a per process limit of 1024.
I am not sure if this should be posted to the apache list or tomcat
list, but thought i would start here since to me it seems more likely
that I am not configuring tomcat correctly as opposed to some problem
in apache. I have searched the web quite a bit, but nothing seems to
fit. The next thing I will try is to use the AJP connector. But even
if it gets rid of the problem, I am worried it may reappear once
traffic increases. Has anyone experienced a similar problem?
other stuff:
OS: CentOS
apache: 2.2.3, using mod_proxy
tomcat: 6.0.16
using netstat, i see a moderate number (~80) of tomcat's sockets in
the CLOSE_WAIT state, not sure if this is relevant.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: apache/tomcat communication issues (502 response)
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: André Warnier [mailto:aw@ice-sa.com]
> Subject: Re: apache/tomcat communication issues (502 response)
>
> You seem to imply that idle threads use less resources in the 64-bit
> world than (presumably) in a lesser-number-of-bit world.
In a 64-bit environment, each thread uses *relatively* less of the resource pool, simply because the pool is so much bigger than in a 32-bit world; 2^47 (or thereabouts) vs 2^31. You do need RAM to hold whatever you use of course, but that's incredibly cheap these days.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: apache/tomcat communication issues (502 response)
Posted by André Warnier <aw...@ice-sa.com>.
Caldarale, Charles R wrote:
>> From: André Warnier [mailto:aw@ice-sa.com]
>> Subject: Re: apache/tomcat communication issues (502 response)
>>
>> That is because a KeepAlive connection will keep a server child (or
>> thread) tied to that particular client connection, until the KeepAlive
>> expires, probably doing nothing most of the time.
>
> But think about this in the context of a modern 64-bit server environment: what's wrong with keeping these idle thread(s) around? When process space was at a premium, it made sense to trade off a little CPU time rebuilding connections versus chewing up the limited address space; now that the resource limits are essentially gone, you might as well take advantage of it.
>
>> It would be better in most cases to have the client re-build a
>> connection more often, and allow those server children/threads
>> to take care of other requests in the meantime.
>
> An idle thread costs very little (at least in a 64-bit world) - configure more of them.
>
Mere curiosity :
You seem to imply that idle threads use less resources in the 64-bit
world than (presumably) in a lesser-number-of-bit world.
Why is that ? Can you elaborate a little bit ?
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: apache/tomcat communication issues (502 response)
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: André Warnier [mailto:aw@ice-sa.com]
> Subject: Re: apache/tomcat communication issues (502 response)
>
> That is because a KeepAlive connection will keep a server child (or
> thread) tied to that particular client connection, until the KeepAlive
> expires, probably doing nothing most of the time.
But think about this in the context of a modern 64-bit server environment: what's wrong with keeping these idle thread(s) around? When process space was at a premium, it made sense to trade off a little CPU time rebuilding connections versus chewing up the limited address space; now that the resource limits are essentially gone, you might as well take advantage of it.
> It would be better in most cases to have the client re-build a
> connection more often, and allow those server children/threads
> to take care of other requests in the meantime.
An idle thread costs very little (at least in a 64-bit world) - configure more of them.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: apache/tomcat communication issues (502 response)
Posted by André Warnier <aw...@ice-sa.com>.
Caldarale, Charles R wrote:
[...]
>
> When you catch threads doing socket I/O, they are almost always waiting for input from the client. If you're using keep-alives (highly likely), that's the normal state of a thread. This likely means you don't have enough requests coming in to keep the threads you already have busy; adding more threads won't change that.
>
If I may add something :
During the recent ApacheCON, one of the gurus there made a comment
related to KeepAlive, which as far as I recall was to the following
effect : the default KeepAlive settings of webservers (many requests,
long duration) are basically a relic of the past, when the setup and
teardown of a TCP connection tended to be significant in terms of
resources, compared to the time actually spent responding to the
request. With modern servers, it is worth having a good look at the
settings, and in most cases reduce them rather drastically.
That is because a KeepAlive connection will keep a server child (or
thread) tied to that particular client connection, until the KeepAlive
expires, probably doing nothing most of the time. It would be better in
most cases to have the client re-build a connection more often, and
allow those server children/threads to take care of other requests in
the meantime.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: apache/tomcat communication issues (502 response)
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: feedly team [mailto:feedlydev@gmail.com]
> Subject: Re: apache/tomcat communication issues (502 response)
>
> I have generated stack dumps throughout the day and
> practically all running threads are doing socket i/o.
> almost none are performing application logic.
When you catch threads doing socket I/O, they are almost always waiting for input from the client. If you're using keep-alives (highly likely), that's the normal state of a thread. This likely means you don't have enough requests coming in to keep the threads you already have busy; adding more threads won't change that.
Using the NIO connector will avoid having the threads sitting around waiting for socket input, but it won't make your CPU busier if you have nothing to do.
Why do you think having a lot of threads is a Bad Thing? Are you running out of process memory space?
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: apache/tomcat communication issues (502 response)
Posted by feedly team <fe...@gmail.com>.
Thanks for all the tips. Today I tried the AJP connector and we had no
problems. After watching a very informative presentation on tomcat
performance tuning by Filip Hanik and reading the docs, I am guessing
this was more of a function of the higher default max thread count of
the ajp connector. The higher default seemed more inline with the
recommendations of the presentation. My next step is to go back to the
http connector with a max thread count of 200 ( the ajp default).
However, the fact that my system already needs 200 threads for
processing with minimal CPU usage makes me think that I will not be
able to effectively utilize a large % of available CPU with the thread
per request model. i.e., i would need to set the max thread count
excessively high to utilize most of the cpu. I have generated stack
dumps throughout the day and practically all running threads are doing
socket i/o. almost none are performing application logic.
I know tomcat has some comet support, but in my limited investigation
that seems designed for ajax heavy applications. Would it still be
useful in my scenario (highly concurrent small requests served very
fast from cache). It seems like it would still have value by allowing
async reads and writes.
On Wed, Apr 8, 2009 at 3:08 PM, Rainer Jung <ra...@kippdata.de> wrote:
> On 08.04.2009 08:23, feedly team wrote:
>> We are running apache and tomcat on the same machine (using the http
>> connector) and logging requests in both. Occasionally (maybe 1% of
>> requests) we see 502 response in the apache log spread fairly evenly
>> throughout the day. these requests don't appear in the tomcat log
>> produced by the access valve. in the apache error log I see:
>>
>> proxy: error reading status line from remote server 127.0.0.1
>> proxy: Error reading from remote server returned by [url]
>>
>> The requests are a mix of methods (GET/POST/PUT) of pretty small
>> bodies and responses, generally under 1k.
>>
>> We added request time (%T) in the apache logs and many of the requests
>> have 0 response time so I don't think it's a timeout issue. Reading
>> the tomcat connector documentation, I tried increasing the maxThreads
>> to 100 and backlog to 1000 in the http connector since it seemed like
>> with the default settings a burst of traffic may cause this issue.
>> This didn't solve the problem. I have done periodic thread dumps and
>> it looks like the number of http threads doesn't get past 80.
>>
>> concerning machine resources, cpu is not loaded at all (~10%
>> utilization). i checked file descriptor usage, it's about 400 and
>> there is a per process limit of 1024.
>>
>> I am not sure if this should be posted to the apache list or tomcat
>> list, but thought i would start here since to me it seems more likely
>> that I am not configuring tomcat correctly as opposed to some problem
>> in apache. I have searched the web quite a bit, but nothing seems to
>> fit. The next thing I will try is to use the AJP connector. But even
>> if it gets rid of the problem, I am worried it may reappear once
>> traffic increases. Has anyone experienced a similar problem?
>
> I guess it's the usual HTTP Keep-Alive race with mod_proxy_http.
> The connections between Apache and Tomcat are only reused for a certain
> amount of time. What happens every now and then is that httpd thinks it
> has a valid connection and sends the next data, but in parallel Tomcat
> detects the HTTP KeepAlive has timed out and closes the connection.
>
> There have been improvements in newer 2.2.x versions (latest is 2.2.11).
> You should especially look at the various pool size and timeout
> configuration attributes on the mod_proxy documentation page and also at
> the Keepalive configuration values for the Tomcat HTTP connector you are
> using.
>
>> other stuff:
>> OS: CentOS
>> apache: 2.2.3, using mod_proxy
>> tomcat: 6.0.16
>> using netstat, i see a moderate number (~80) of tomcat's sockets in
>> the CLOSE_WAIT state, not sure if this is relevant.
>
> I assume Apache and Tomcat are on separate systems, and you notice
> CLOSE_WAIT when doing netstat on the Tomcat system and the connections
> are proxy connections? Then this means Apache has closed a connection
> and Tomcat would still use it. Investigate the related timeout
> parameters as suggested above.
>
> Regards,
>
> Rainer
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: apache/tomcat communication issues (502 response)
Posted by Rainer Jung <ra...@kippdata.de>.
On 08.04.2009 08:23, feedly team wrote:
> We are running apache and tomcat on the same machine (using the http
> connector) and logging requests in both. Occasionally (maybe 1% of
> requests) we see 502 response in the apache log spread fairly evenly
> throughout the day. these requests don't appear in the tomcat log
> produced by the access valve. in the apache error log I see:
>
> proxy: error reading status line from remote server 127.0.0.1
> proxy: Error reading from remote server returned by [url]
>
> The requests are a mix of methods (GET/POST/PUT) of pretty small
> bodies and responses, generally under 1k.
>
> We added request time (%T) in the apache logs and many of the requests
> have 0 response time so I don't think it's a timeout issue. Reading
> the tomcat connector documentation, I tried increasing the maxThreads
> to 100 and backlog to 1000 in the http connector since it seemed like
> with the default settings a burst of traffic may cause this issue.
> This didn't solve the problem. I have done periodic thread dumps and
> it looks like the number of http threads doesn't get past 80.
>
> concerning machine resources, cpu is not loaded at all (~10%
> utilization). i checked file descriptor usage, it's about 400 and
> there is a per process limit of 1024.
>
> I am not sure if this should be posted to the apache list or tomcat
> list, but thought i would start here since to me it seems more likely
> that I am not configuring tomcat correctly as opposed to some problem
> in apache. I have searched the web quite a bit, but nothing seems to
> fit. The next thing I will try is to use the AJP connector. But even
> if it gets rid of the problem, I am worried it may reappear once
> traffic increases. Has anyone experienced a similar problem?
I guess it's the usual HTTP Keep-Alive race with mod_proxy_http.
The connections between Apache and Tomcat are only reused for a certain
amount of time. What happens every now and then is that httpd thinks it
has a valid connection and sends the next data, but in parallel Tomcat
detects the HTTP KeepAlive has timed out and closes the connection.
There have been improvements in newer 2.2.x versions (latest is 2.2.11).
You should especially look at the various pool size and timeout
configuration attributes on the mod_proxy documentation page and also at
the Keepalive configuration values for the Tomcat HTTP connector you are
using.
> other stuff:
> OS: CentOS
> apache: 2.2.3, using mod_proxy
> tomcat: 6.0.16
> using netstat, i see a moderate number (~80) of tomcat's sockets in
> the CLOSE_WAIT state, not sure if this is relevant.
I assume Apache and Tomcat are on separate systems, and you notice
CLOSE_WAIT when doing netstat on the Tomcat system and the connections
are proxy connections? Then this means Apache has closed a connection
and Tomcat would still use it. Investigate the related timeout
parameters as suggested above.
Regards,
Rainer
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
[Mildly OT] RE: apache/tomcat communication issues (502 response)
Posted by Peter Crowther <Pe...@melandra.com>.
> From: André Warnier [mailto:aw@ice-sa.com]
> a TCP CLOSE_WAIT state happens when the writing side of a TCP
> connection
> has finished writing and (nicely) closes its side of the socket to
> indicate the fact,
Yes.
> but the reading side of the connection does not read
> what is left in the buffers, so there is still some data unread in the
> pipeline,
No. It merely means that the reader has not yet closed the socket. You're putting more context behind that than there really is - the reader may, for example, have read all the data, but never have issued the close() call.
> and the reading side never closes the socket.
"Has not yet closed" :-).
http://www.cs.northwestern.edu/~agupta/cs340/project2/TCPIP_State_Transition_Diagram.pdf (itself reproduced from "TCP/IP Illustrated, Volume 2") is a useful resource if you want to know what the states mean. If you're seeing CLOSE_WAITs on the Tomcat server, it means the client has sent a FIN and Tomcat's ACKed that. If you can, look on the client to see what's happening. If you see connections in FIN_WAIT_1, the client never got the ACK from Tomcat, if you see connections in FIN_WAIT_2, the client got the ACK and is sitting around waiting for Tomcat to close the connection and send a FIN.
- Peter
P.S. Yes, I used to teach this stuff ;-).
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
[Mildly OT] Re: apache/tomcat communication issues (502 response)
Posted by André Warnier <aw...@ice-sa.com>.
Alan Chaney wrote:
> Hi Andre
>
Hi. And thanks, to Peter also.
As this might start to look like a thread hijack, I'll repost with a new
CLOSE_WAIT subject to continue this discussion.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: apache/tomcat communication issues (502 response)
Posted by Alan Chaney <al...@compulsivecreative.com>.
Hi Andre
André Warnier wrote:
> feedly team wrote:
> [...]
>> using netstat, i see a moderate number (~80) of tomcat's sockets in
>> the CLOSE_WAIT state, not sure if this is relevant.
>>
> Approximately, because I am not sure I have this really understood yet
> : a TCP CLOSE_WAIT state happens when the writing side of a TCP
> connection has finished writing and (nicely) closes its side of the
> socket to indicate the fact, but the reading side of the connection
> does not read what is left in the buffers, so there is still some data
> unread in the pipeline, and the reading side never closes the socket.
> And now I'm stuck in my explanation, because I am not sure which side
> is seeing the CLOSE_WAIT... ;-)
I think that you are indicating one condition in which you can see a
CLOSE_WAIT but there are many others. I also think that the condition
you indicate is appropriate when the CLOSE_WAIT is observed at the
receiving end of a communication, but is possible for a socket to be in
this state when it has sent data as well, but of course there will be no
outstanding data to send.
More generally CLOSE_WAIT is the state in which a socket is left AFTER
the "other end" says its finished and BEFORE the application which is
using the socket actually closes the socket. The "WAIT" refers to the
operating system waiting for the application to finish using the socket.
I think a socket can be in a CLOSE_WAIT state without there being any
further data to read or write - literally just waiting for the calling
application to close it.
Having written socket handling code for both Java and C++ on a variety
of platforms I don't think there is any particular reason why Java
should be better or worse (in fact, code which uses sockets in Java is
generally pretty easy.) I suspect that your observations may be affected
by "local conditions" eg one application is badly written but represents
a lot of your network activity, so its behavior is predominant in
conditioning your thinking. Or not! :)
regards
Alan Chaney
> But anyway, it indicates a problem somewhere in one of the two
> applications, my guess being the reading one.
> It should do more reads to exhaust the remaining data, get an
> end-of-stream, then close its side of the connection, but it never does.
> There is apparently no timeout for that, so the OS can never get rid
> of that socket, which probably leaves a bunch of things hanging around
> and consuming resources that they shouldn't.
> On one of our systems, I have occasionally seen this issue grow until
> the point where the system seemed unable to accept new connections.
> Now whether that has any bearing on your particular issue, I don't
> know. But it sure indicates a logical problem somewhere.
>
> There is quite a bit on the subject on Google, of unequal quality.
> If someone knows a more rigorous explanation, please go ahead.
>
> I will still add a purely sibjective and personal note : from personal
> experience, this phenomenon seems to happens more frequently whith
> Java applications than with others, so I would guess that there might
> be something in the Java handling of sockets that makes it a bit
> harder to write correct socket-handling code.
> A Java expert may want to comment on that too.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
>
> !DSPAM:49dc6634305142136417547!
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: apache/tomcat communication issues (502 response)
Posted by André Warnier <aw...@ice-sa.com>.
feedly team wrote:
[...]
> using netstat, i see a moderate number (~80) of tomcat's sockets in
> the CLOSE_WAIT state, not sure if this is relevant.
>
Approximately, because I am not sure I have this really understood yet :
a TCP CLOSE_WAIT state happens when the writing side of a TCP connection
has finished writing and (nicely) closes its side of the socket to
indicate the fact, but the reading side of the connection does not read
what is left in the buffers, so there is still some data unread in the
pipeline, and the reading side never closes the socket.
And now I'm stuck in my explanation, because I am not sure which side is
seeing the CLOSE_WAIT... ;-)
But anyway, it indicates a problem somewhere in one of the two
applications, my guess being the reading one.
It should do more reads to exhaust the remaining data, get an
end-of-stream, then close its side of the connection, but it never does.
There is apparently no timeout for that, so the OS can never get rid of
that socket, which probably leaves a bunch of things hanging around and
consuming resources that they shouldn't.
On one of our systems, I have occasionally seen this issue grow until
the point where the system seemed unable to accept new connections.
Now whether that has any bearing on your particular issue, I don't know.
But it sure indicates a logical problem somewhere.
There is quite a bit on the subject on Google, of unequal quality.
If someone knows a more rigorous explanation, please go ahead.
I will still add a purely sibjective and personal note : from personal
experience, this phenomenon seems to happens more frequently whith Java
applications than with others, so I would guess that there might be
something in the Java handling of sockets that makes it a bit harder to
write correct socket-handling code.
A Java expert may want to comment on that too.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org