You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Jamie Jackson <ja...@gmail.com> on 2015/10/07 22:20:30 UTC
Monitoring Connections
Hi Folks,
I had a server crash on Friday, and I had the following symptoms:
Response time worsened, and when I looked at my real-time monitoring (an
app called FusionReactor), I saw that the app *seemed* to be
single-threading (I only saw one request process at a time), then after a
few minutes, even though resources still looked good, the app no longer
seemed to receive any requests.
My first guess was that somehow, connections were slowly getting clogged
until there were none left, then my server was completely unresponsive.
I front Tomcat (7.0.59) with Apache HTTPD, and connect with mod_proxy. (The
end app is built in Lucee, and is running on RHEL 6.7 and Oracle Java
1.7.0_76.)
My instinct is to monitor the connection (pools?) in, I guess, Tomcat and
mod_proxy, but:
1. I don't know if that's sane.
2. I have no clue how one would go about that.
I could use some input.
Thanks,
Jamie
Re: Monitoring Connections
Posted by Aurélien Terrestris <at...@gmail.com>.
> I know mod_jk will complain if it can't
> make a connection or if there is a timeout... I suspect mod_proxy_http
> will do the same.
They both are supposed to log 502, while Tomcat will raise a "connection
reset by peer" when answering to an already closed connection.
Timeouts for both are buggy until, if I remember well, Apache 2.2.17 (see :
http://www.spinics.net/lists/apache-users/msg93765.html ), behaving with a
timeout 4 times too short. I suppose this was hard-coded for test reasons,
and just forgotten. They never replied on this anyway, and please check if
you're not running this old bugged Apache.
As said by Chris, your server seems quiet, and this means the problem might
be out there.
Uploading files ? Please check if there is any virus scanner software on
the server, this might be an explanation. Have a look on the server if you
can find the temporary file which is being uploaded, and what is its size
(more than 100 megs ??)
When the problem happens again, don't forget to do a 'netstat -an'. There,
it will be important to check if there is any SYN_SENT connections, showing
a network problem.
Another thing more difficult if the problem is related to DNS solving which
could be stalled for any reason. You can try some nslookup commands when
the problem happens and compare with normal nslookup (what DNS server
answered ?) ; you also can install a BIND on your server and make it log
everything, this would give you logs for after crash analysis. Or let run a
'tcpdump port 53' and save its capture for later analysis.
regards
A.T.
2015-10-21 20:58 GMT+02:00 Christopher Schultz <chris@christopherschultz.net
>:
> Jamie,
>
> On 10/21/15 2:37 PM, Jamie Jackson wrote:
> > On Wed, Oct 21, 2015 at 1:03 PM, Christopher Schultz wrote:
> >
> >> Jamie,
> >>
> >>
> >>
> >> Your mostly-default <Connector> will default to a maximum of 200
> >> incoming connections with 200 threads to handle them. You are only using
> >> 12, so something else must be going on. You have no obvious limits on
> >> httpd, so you are probably using the default there as well
> >> (coincidentally, also in the 200-connection range).
> >>
> >> That's a high connection timeout: 93 seconds (why 93?). Note that the
> >> connectionTimeout sets the amount of time Tomcat will wait for a client
> >> to send the request line (the "GET /foo HTTP/1.1"), not the amount of
> >> time the request is allowed to run -- like for an upload, etc. I usually
> >> lower this setting from the default of 60 seconds to more like 5 or 10
> >> seconds. Clients shouldn't be waiting a long time between making a
> >> connection and sending a request.
> >>
> >> This timeout also applies to subsequent requests on a keep-alive
> >> connection. So if the browser opens a connection and sends 1, 2, 3
> >> requests, Tomcat will hold that thread+connection open for 93 seconds
> >> after the last request (assuming the client doesn't terminate the
> >> connection, which it might NOT) before allowing other clients to be
> >> serviced by that thread. This is a BIO-Connector-only behavior. The
> >> NIO/NIO2 and APR connectors don't hold-up the request thread waiting for
> >> a follow-up keep-alive request from a client.
> >>
> >
> > Thanks for the info. It seems as if connectionTimeout is almost
> universally
> > misunderstood to mean something like "request timeout," (which is why it
> > had been high--to accommodate things like long responses and file
> uploads).
> > It seems possible that we could be using up too many threads for too long
> > because of the effect of this long timeout on keep-alives.
>
> While that's true, you should something like 185 threads "in reserve"
> and so the server shouldn't grind to a halt and not let anyone else in.
> If there are other components in the mix, those could prevent more
> connections (e.g. load-balancer, QOS component, etc.) or even if you are
> trying to connect from a single web browser with a 4-connection limit,
> you'll obviously only be able to upload 4 files at a time.
>
> But you didn't say anything about that kind of thing, so I assume it's
> not the issue.
>
> > The only time I can think of that a client would be taking any kind of
> time
> > between connection and sending the request URI line is if someone is
> > manually interacting (say, via telnet). I'm going to follow your lead and
> > reduce this.
>
> I wouldn't reduce it past the default of 60 seconds (60000ms) unless you
> are observing client-starvation.
>
> > I doubt that this is the *sole* culprit, but it *is* something for me to
> > tweak.
>
> I would read the whole HTTP-Connector configuration reference --
> especially the "timeout" related items -- and make sure you understand
> them all before setting any of them. The defaults are reasonable, but
> every environment has its own special set of requirements.
>
> I don't think the timeouts are the issue. What else can you tell us
> about the behavior of the server when it "crashes"? I don't think you
> have really described the actual problem, yet.
>
> -chris
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: Monitoring Connections
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Jamie,
On 10/21/15 2:37 PM, Jamie Jackson wrote:
> On Wed, Oct 21, 2015 at 1:03 PM, Christopher Schultz wrote:
>
>> Jamie,
>>
>>
>>
>> Your mostly-default <Connector> will default to a maximum of 200
>> incoming connections with 200 threads to handle them. You are only using
>> 12, so something else must be going on. You have no obvious limits on
>> httpd, so you are probably using the default there as well
>> (coincidentally, also in the 200-connection range).
>>
>> That's a high connection timeout: 93 seconds (why 93?). Note that the
>> connectionTimeout sets the amount of time Tomcat will wait for a client
>> to send the request line (the "GET /foo HTTP/1.1"), not the amount of
>> time the request is allowed to run -- like for an upload, etc. I usually
>> lower this setting from the default of 60 seconds to more like 5 or 10
>> seconds. Clients shouldn't be waiting a long time between making a
>> connection and sending a request.
>>
>> This timeout also applies to subsequent requests on a keep-alive
>> connection. So if the browser opens a connection and sends 1, 2, 3
>> requests, Tomcat will hold that thread+connection open for 93 seconds
>> after the last request (assuming the client doesn't terminate the
>> connection, which it might NOT) before allowing other clients to be
>> serviced by that thread. This is a BIO-Connector-only behavior. The
>> NIO/NIO2 and APR connectors don't hold-up the request thread waiting for
>> a follow-up keep-alive request from a client.
>>
>
> Thanks for the info. It seems as if connectionTimeout is almost universally
> misunderstood to mean something like "request timeout," (which is why it
> had been high--to accommodate things like long responses and file uploads).
> It seems possible that we could be using up too many threads for too long
> because of the effect of this long timeout on keep-alives.
While that's true, you should something like 185 threads "in reserve"
and so the server shouldn't grind to a halt and not let anyone else in.
If there are other components in the mix, those could prevent more
connections (e.g. load-balancer, QOS component, etc.) or even if you are
trying to connect from a single web browser with a 4-connection limit,
you'll obviously only be able to upload 4 files at a time.
But you didn't say anything about that kind of thing, so I assume it's
not the issue.
> The only time I can think of that a client would be taking any kind of time
> between connection and sending the request URI line is if someone is
> manually interacting (say, via telnet). I'm going to follow your lead and
> reduce this.
I wouldn't reduce it past the default of 60 seconds (60000ms) unless you
are observing client-starvation.
> I doubt that this is the *sole* culprit, but it *is* something for me to
> tweak.
I would read the whole HTTP-Connector configuration reference --
especially the "timeout" related items -- and make sure you understand
them all before setting any of them. The defaults are reasonable, but
every environment has its own special set of requirements.
I don't think the timeouts are the issue. What else can you tell us
about the behavior of the server when it "crashes"? I don't think you
have really described the actual problem, yet.
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Monitoring Connections
Posted by Jamie Jackson <ja...@gmail.com>.
On Wed, Oct 21, 2015 at 1:03 PM, Christopher Schultz wrote:
> Jamie,
>
>
>
> Your mostly-default <Connector> will default to a maximum of 200
> incoming connections with 200 threads to handle them. You are only using
> 12, so something else must be going on. You have no obvious limits on
> httpd, so you are probably using the default there as well
> (coincidentally, also in the 200-connection range).
>
> That's a high connection timeout: 93 seconds (why 93?). Note that the
> connectionTimeout sets the amount of time Tomcat will wait for a client
> to send the request line (the "GET /foo HTTP/1.1"), not the amount of
> time the request is allowed to run -- like for an upload, etc. I usually
> lower this setting from the default of 60 seconds to more like 5 or 10
> seconds. Clients shouldn't be waiting a long time between making a
> connection and sending a request.
>
> This timeout also applies to subsequent requests on a keep-alive
> connection. So if the browser opens a connection and sends 1, 2, 3
> requests, Tomcat will hold that thread+connection open for 93 seconds
> after the last request (assuming the client doesn't terminate the
> connection, which it might NOT) before allowing other clients to be
> serviced by that thread. This is a BIO-Connector-only behavior. The
> NIO/NIO2 and APR connectors don't hold-up the request thread waiting for
> a follow-up keep-alive request from a client.
>
Thanks for the info. It seems as if connectionTimeout is almost universally
misunderstood to mean something like "request timeout," (which is why it
had been high--to accommodate things like long responses and file uploads).
It seems possible that we could be using up too many threads for too long
because of the effect of this long timeout on keep-alives.
The only time I can think of that a client would be taking any kind of time
between connection and sending the request URI line is if someone is
manually interacting (say, via telnet). I'm going to follow your lead and
reduce this.
I doubt that this is the *sole* culprit, but it *is* something for me to
tweak.
Thanks,
Jamie
> -chris
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: Monitoring Connections
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Jamie,
On 10/16/15 3:12 PM, Jamie Jackson wrote:
> On Tue, Oct 13, 2015 at 9:34 AM, Christopher Schultz <
> chris@christopherschultz.net> wrote:
>
> Jamie,
>
> On 10/9/15 10:03 AM, Jamie Jackson wrote:
>>>> Here's the stack trace dump:
>>>> https://gist.github.com/jamiejackson/ca2a49d2c8afac496067
>>>>
>>>> FWIW, I've been trying to come up with a reliable test case to
>>>> trigger the problem, but I haven't nailed it yet. I've suspected
>>>> that it's related to file (large or slow) HTTP file uploads, and
>>>> that's what I was running at the time, which helps to explain the
>>>> java.net.SocketInputStream.socketRead0 traces.
>
> Your server looks mostly idle. It also looks like it's running both
> Tomcat and Jetty in the same instance. What's going on?
>
>
>> I'm not sure yet. I'm asking around, and I'll let you know.
>
> About the appearance of Jetty in the stack trace dump: It's part of
> FusionReactor (the JVM monitor)--it uses Jetty to serve its interface.
Okay.
>>>> "http-bio-8888-exec-19" Id=27307 RUNNABLE (in native)
>>>> java.lang.Thread.State: RUNNABLE at
>>>> java.net.SocketInputStream.socketRead0(Native Method) at
>>>> java.net.SocketInputStream.read(SocketInputStream.java:152) at
>>>> java.net.SocketInputStream.read(SocketInputStream.java:122) at
>>>> [...]
> org.apache.commons.fileupload.MultipartStream$ItemInputStream.read(MultipartStream.java:887)
>>>> at java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>>>> at java.io.BufferedInputStream.read(BufferedInputStream.java:334) -
>>>> locked java.io.BufferedInputStream@6e81abfc at
>>>> java.io.FilterInputStream.read(FilterInputStream.java:107)
>
> Obviously, this thread is reading a file upload. Also these:
>
> "http-bio-8888-exec-21" Id=27309 RUNNABLE (in native)
> java.lang.Thread.State: RUNNABLE
> "http-bio-8888-exec-26" Id=27314 RUNNABLE (in native)
> java.lang.Thread.State: RUNNABLE
>
> This thread is making an outgoing HTTP request, which could be hanging:
>
>>>> "Thread-27550" Id=27623 RUNNABLE (in native)
>>>> java.lang.Thread.State: RUNNABLE at
>>>> java.net.SocketInputStream.socketRead0(Native Method) at
>>>> java.net.SocketInputStream.read(SocketInputStream.java:152) at
>>>> java.net.SocketInputStream.read(SocketInputStream.java:122) at
>>>> sun.security.ssl.InputRecord.readFully(InputRecord.java:442) at
>>>> sun.security.ssl.InputRecord.read(InputRecord.java:480) at
>>>> sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:934) -
>>>> locked java.lang.Object@659eed19 at
>>>> sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:891)
>
> Same with this one:
>
> "Thread-27551" Id=27624 RUNNABLE (in native)
> java.lang.Thread.State: RUNNABLE
>
> But neither of the above threads making HTTP requests are
> request-processing threads: they are independent and shouldn't block
> incoming requests.
>
> You have 12 total request-processing threads: the ones named
> "http-bio-8888-exec-[#]". Most of them aren't doing anything -- just
> waiting on a new request to come in.
>
> You have an AJP connector defined, but it looks like it's not being used.
>
> Initially, you said that you used mod_proxy from httpd. Are you
> expecting to use mod_proxy_http or mod_proxy_ajp?
>
>
>> mod_proxy_http, as far as I know, but see below...
Yes, you are using mod_proxy_http.
> When you try to make a connection during these incidents, do you get
> any errors on the httpd side?
>
>
>> Unfortunately, at the time, I thought this would be easy to reproduce, so I
>> didn't take enough notes (including the time of the problem on my load test
>> box). Here are some errors that I think were just created by the app timing
>> out requests. I don't think these were associated with the failed requests,
>> so take these entries (from httpd's error_log) with a grain of salt:
>> https://gist.github.com/jamiejackson/2f16ca83bdfc9c8f8795#file-errors-txt
There's not a whole lot in there, other than the fact that something
"bad" happened.
> Can you post your <Connector> configuration from conf/server.xml?
> Remember to remove any sensitive information that you might have in there.
>
>
>> I think this is what you're after:
>
>> <Connector port="8888" protocol="HTTP/1.1"
>> connectionTimeout="93000"
>> redirectPort="8443" />
Your mostly-default <Connector> will default to a maximum of 200
incoming connections with 200 threads to handle them. You are only using
12, so something else must be going on. You have no obvious limits on
httpd, so you are probably using the default there as well
(coincidentally, also in the 200-connection range).
That's a high connection timeout: 93 seconds (why 93?). Note that the
connectionTimeout sets the amount of time Tomcat will wait for a client
to send the request line (the "GET /foo HTTP/1.1"), not the amount of
time the request is allowed to run -- like for an upload, etc. I usually
lower this setting from the default of 60 seconds to more like 5 or 10
seconds. Clients shouldn't be waiting a long time between making a
connection and sending a request.
This timeout also applies to subsequent requests on a keep-alive
connection. So if the browser opens a connection and sends 1, 2, 3
requests, Tomcat will hold that thread+connection open for 93 seconds
after the last request (assuming the client doesn't terminate the
connection, which it might NOT) before allowing other clients to be
serviced by that thread. This is a BIO-Connector-only behavior. The
NIO/NIO2 and APR connectors don't hold-up the request thread waiting for
a follow-up keep-alive request from a client.
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Monitoring Connections
Posted by Jamie Jackson <ja...@gmail.com>.
About the appearance of Jetty in the stack trace dump: It's part of
FusionReactor (the JVM monitor)--it uses Jetty to serve its interface.
Thanks,
Jamie
On Fri, Oct 16, 2015 at 3:12 PM, Jamie Jackson <ja...@gmail.com> wrote:
>
>
> On Tue, Oct 13, 2015 at 9:34 AM, Christopher Schultz <
> chris@christopherschultz.net> wrote:
>
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA256
>>
>> Jamie,
>>
>> On 10/9/15 10:03 AM, Jamie Jackson wrote:
>> > Here's the stack trace dump:
>> > https://gist.github.com/jamiejackson/ca2a49d2c8afac496067
>> >
>> > FWIW, I've been trying to come up with a reliable test case to
>> > trigger the problem, but I haven't nailed it yet. I've suspected
>> > that it's related to file (large or slow) HTTP file uploads, and
>> > that's what I was running at the time, which helps to explain the
>> > java.net.SocketInputStream.socketRead0 traces.
>>
>> Your server looks mostly idle. It also looks like it's running both
>> Tomcat and Jetty in the same instance. What's going on?
>
>
> I'm not sure yet. I'm asking around, and I'll let you know.
>
>
>> > "http-bio-8888-exec-19" Id=27307 RUNNABLE (in native)
>> > java.lang.Thread.State: RUNNABLE at
>> > java.net.SocketInputStream.socketRead0(Native Method) at
>> > java.net.SocketInputStream.read(SocketInputStream.java:152) at
>> > java.net.SocketInputStream.read(SocketInputStream.java:122) at
>> >
>> org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:535)
>> >
>> >
>> at
>>
>> org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:504)
>> > at
>> >
>> org.apache.coyote.http11.InternalInputBuffer$InputStreamInputBuffer.doRead(InternalInputBuffer.java:566)
>> >
>> >
>> at
>>
>> org.apache.coyote.http11.filters.IdentityInputFilter.doRead(IdentityInputFilter.java:137)
>> > at
>> >
>> org.apache.coyote.http11.AbstractInputBuffer.doRead(AbstractInputBuffer.java:348)
>> >
>> >
>> at org.apache.coyote.Request.doRead(Request.java:422)
>> > at
>> >
>> org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:290)
>> >
>> >
>> at org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:390)
>> > at
>> > org.apache.catalina.connector.InputBuffer.readByte(InputBuffer.java:304)
>> >
>> >
>> at
>>
>> org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:106)
>> > at
>> >
>> com.intergral.fusionreactor.j2ee.filter.surrogate.FusionReactorServletRequestProxy$1.read(FusionReactorServletRequestProxy.java:123)
>> >
>> >
>> at java.io.InputStream.read(InputStream.java:179)
>> > at
>> >
>> org.apache.commons.fileupload.MultipartStream$ItemInputStream.makeAvailable(MultipartStream.java:977)
>> >
>> >
>> at
>>
>> org.apache.commons.fileupload.MultipartStream$ItemInputStream.read(MultipartStream.java:887)
>> > at java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
>> > at java.io.BufferedInputStream.read(BufferedInputStream.java:334) -
>> > locked java.io.BufferedInputStream@6e81abfc at
>> > java.io.FilterInputStream.read(FilterInputStream.java:107)
>>
>> Obviously, this thread is reading a file upload. Also these:
>>
>> "http-bio-8888-exec-21" Id=27309 RUNNABLE (in native)
>> java.lang.Thread.State: RUNNABLE
>> "http-bio-8888-exec-26" Id=27314 RUNNABLE (in native)
>> java.lang.Thread.State: RUNNABLE
>>
>> This thread is making an outgoing HTTP request, which could be hanging:
>>
>> > "Thread-27550" Id=27623 RUNNABLE (in native)
>> > java.lang.Thread.State: RUNNABLE at
>> > java.net.SocketInputStream.socketRead0(Native Method) at
>> > java.net.SocketInputStream.read(SocketInputStream.java:152) at
>> > java.net.SocketInputStream.read(SocketInputStream.java:122) at
>> > sun.security.ssl.InputRecord.readFully(InputRecord.java:442) at
>> > sun.security.ssl.InputRecord.read(InputRecord.java:480) at
>> > sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:934) -
>> > locked java.lang.Object@659eed19 at
>> > sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:891)
>> >
>> >
>> at sun.security.ssl.AppInputStream.read(AppInputStream.java:102)
>> > - locked sun.security.ssl.AppInputStream@7c98d0ab at
>> >
>> org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
>> >
>> >
>> at
>>
>> org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
>> > at
>> >
>> org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
>> >
>> >
>> at
>>
>> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
>> > at
>> >
>> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
>> >
>> >
>> at
>>
>> org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
>> > at
>> >
>> org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
>> >
>> >
>> at
>>
>> org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
>> > at
>> >
>> org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
>> >
>> >
>> at
>>
>> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
>> > at
>> >
>> org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
>> >
>> >
>> at
>>
>> org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
>> > at
>> >
>> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
>> >
>> >
>> at
>>
>> org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
>> > at
>> >
>> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
>> >
>> >
>> at sun.reflect.GeneratedMethodAccessor98.invoke(Unknown Source)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> >
>> >
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> > at
>> >
>> lucee.commons.net.http.httpclient4.HTTPEngine4Impl.execute(HTTPEngine4Impl.java:416)
>> >
>> >
>> at
>>
>> lucee.commons.net.http.httpclient4.HTTPEngine4Impl._invoke(HTTPEngine4Impl.java:252)
>> > at
>> >
>> lucee.commons.net.http.httpclient4.HTTPEngine4Impl.get(HTTPEngine4Impl.java:112)
>> >
>> >
>> at lucee.commons.net.http.HTTPEngine.get(HTTPEngine.java:86)
>> > at
>> > lucee.runtime.schedule.ExecutionThread.execute(ExecutionThread.java:108)
>> >
>> >
>> at lucee.runtime.schedule.ExecutionThread.run(ExecutionThread.java:58)
>>
>>
>> Same with this one:
>>
>> "Thread-27551" Id=27624 RUNNABLE (in native)
>> java.lang.Thread.State: RUNNABLE
>>
>> But neither of the above threads making HTTP requests are
>> request-processing threads: they are independent and shouldn't block
>> incoming requests.
>>
>> You have 12 total request-processing threads: the ones named
>> "http-bio-8888-exec-[#]". Most of them aren't doing anything -- just
>> waiting on a new request to come in.
>>
>> You have an AJP connector defined, but it looks like it's not being used.
>>
>> Initially, you said that you used mod_proxy from httpd. Are you
>> expecting to use mod_proxy_http or mod_proxy_ajp?
>
>
> mod_proxy_http, as far as I know, but see below...
>
>
>> It certainly looks
>> like you are using the HTTP flavor of mod_proxy since your
>> file-uploads are going to the HTTP connector on port 8888 instead of
>> the AJP connector on port 8009. If you don't need the AJP connector,
>> you might want to disable it and get a small amount of resources back.
>>
>
> Okay, thanks.
>
> When you try to make a connection during these incidents, do you get
>> any errors on the httpd side?
>
>
> Unfortunately, at the time, I thought this would be easy to reproduce, so
> I didn't take enough notes (including the time of the problem on my load
> test box). Here are some errors that I think were just created by the app
> timing out requests. I don't think these were associated with the failed
> requests, so take these entries (from httpd's error_log) with a grain of
> salt:
> https://gist.github.com/jamiejackson/2f16ca83bdfc9c8f8795#file-errors-txt
>
> I'll capture this information next time I trigger a crash.
>
>
>> I know mod_jk will complain if it can't
>> make a connection or if there is a timeout... I suspect mod_proxy_http
>> will do the same.
>>
>> Can you post your <Connector> configuration from conf/server.xml?
>> Remember to remove any sensitive information that you might have in there.
>>
>
> I think this is what you're after:
>
> <Connector port="8888" protocol="HTTP/1.1"
> connectionTimeout="93000"
> redirectPort="8443" />
>
> FWIW, I'm also noticing this:
>
> <Host name="127.0.0.1" appBase="webapps"
> unpackWARs="false" autoDeploy="false">
>
> <!-- Access log processes all example.
> Documentation at: /docs/config/valve.html
> Note: The pattern used is equivalent to using
> pattern="common" -->
> <Valve className="org.apache.catalina.valves.AccessLogValve"
> directory="logs"
> prefix="localhost_access_log." suffix=".txt"
> pattern="%h %l %u %t "%r" %s %b"
> resolveHosts="false"/>
>
> <!-- visit modcfml.org for details on mod_cfml configuration
> options -->
> <Valve className="mod_cfml.core"
> loggingEnabled="false"
> waitForContext="3"
> maxContexts="200"
> timeBetweenContexts="30000"
> />
> </Host>
>
>
> Also, what do your ProxyPass/ProxyPassReverse configuration directives
>> look like in httpd.conf?
>
>
> The apache config has:
>
> LoadModule proxy_module modules/mod_proxy.so
> LoadModule proxy_balancer_module modules/mod_proxy_balancer.so
> LoadModule proxy_ftp_module modules/mod_proxy_ftp.so
> LoadModule proxy_http_module modules/mod_proxy_http.so
> LoadModule proxy_ajp_module modules/mod_proxy_ajp.so
> LoadModule proxy_connect_module modules/mod_proxy_connect.so
>
> And the virtual host has:
> https://gist.github.com/jamiejackson/2f16ca83bdfc9c8f8795#file-virtual-host
>
>
>
>> - -chris
>> -----BEGIN PGP SIGNATURE-----
>> Comment: GPGTools - http://gpgtools.org
>>
>> iQIcBAEBCAAGBQJWHQhoAAoJEBzwKT+lPKRYoVwP/08YV4m1q3jXp+ixsPaPReEf
>> m9wnqPlWpX9tGjwVcsYvnqMPTJe8Ew+yzIFZ5NkPsmQk/eeKQFMYBsr0yDje4Nl0
>> Suk+OfNXhdL7RUn/PZg1Rd1xl7hcVeJmgOD6HRIcXAsnB+iTpCYpR+FtTnoGosia
>> PQ64AKEZR7KliP0bkM04NY4hHa+eA5vp2kGyGAAkA2g+sxUUXgiAVoUYMbLYwetv
>> CKqwlCFcG+JF9M2qi4n8vVwpsTqdLrInlO+yAiejZoQuHaSefnolIkFE0y+08bnz
>> Yv1nhNXwIOJAIvsl2BFoQIXXnpVh2ncILLIzhrcBBsazBy0RxccKsOJR0F6SKEWg
>> 3JLD6z/3CxtuW28KU2nFIQiKEQHzhZ+fV1NAp3XrHaQPncFGlNGPOUgy4Ggqxy3j
>> eAOkN1si3GFRm+ILELDO4ekKQCoAAdWs3MMpVTOKdL6mMO1OiYGPHzVTclgLloxc
>> o6FVgy81s6Hk9UJlu5enZJu9I+5tFxlD1gXrBmG7ryCOyiDn9pHmZ68EBpe3j+Zf
>> HPUtt7XP5eTVZp8vlVS3ebBHjrMkxUYGuBKYZF7prGfA5LE42nctdMslneiyZJJc
>> AOrcp32bgJFCcXmdBovD2UrJu0Evc4su3vV4JW9PuFSE1i3ntFLKuqL4FiArf1ls
>> +0KkHZDkU8C3MP3MCySA
>> =5Mun
>> -----END PGP SIGNATURE-----
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>
Re: Monitoring Connections
Posted by Jamie Jackson <ja...@gmail.com>.
On Tue, Oct 13, 2015 at 9:34 AM, Christopher Schultz <
chris@christopherschultz.net> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Jamie,
>
> On 10/9/15 10:03 AM, Jamie Jackson wrote:
> > Here's the stack trace dump:
> > https://gist.github.com/jamiejackson/ca2a49d2c8afac496067
> >
> > FWIW, I've been trying to come up with a reliable test case to
> > trigger the problem, but I haven't nailed it yet. I've suspected
> > that it's related to file (large or slow) HTTP file uploads, and
> > that's what I was running at the time, which helps to explain the
> > java.net.SocketInputStream.socketRead0 traces.
>
> Your server looks mostly idle. It also looks like it's running both
> Tomcat and Jetty in the same instance. What's going on?
I'm not sure yet. I'm asking around, and I'll let you know.
> > "http-bio-8888-exec-19" Id=27307 RUNNABLE (in native)
> > java.lang.Thread.State: RUNNABLE at
> > java.net.SocketInputStream.socketRead0(Native Method) at
> > java.net.SocketInputStream.read(SocketInputStream.java:152) at
> > java.net.SocketInputStream.read(SocketInputStream.java:122) at
> >
> org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:535)
> >
> >
> at
>
> org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:504)
> > at
> >
> org.apache.coyote.http11.InternalInputBuffer$InputStreamInputBuffer.doRead(InternalInputBuffer.java:566)
> >
> >
> at
>
> org.apache.coyote.http11.filters.IdentityInputFilter.doRead(IdentityInputFilter.java:137)
> > at
> >
> org.apache.coyote.http11.AbstractInputBuffer.doRead(AbstractInputBuffer.java:348)
> >
> >
> at org.apache.coyote.Request.doRead(Request.java:422)
> > at
> >
> org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:290)
> >
> >
> at org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:390)
> > at
> > org.apache.catalina.connector.InputBuffer.readByte(InputBuffer.java:304)
> >
> >
> at
>
> org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:106)
> > at
> >
> com.intergral.fusionreactor.j2ee.filter.surrogate.FusionReactorServletRequestProxy$1.read(FusionReactorServletRequestProxy.java:123)
> >
> >
> at java.io.InputStream.read(InputStream.java:179)
> > at
> >
> org.apache.commons.fileupload.MultipartStream$ItemInputStream.makeAvailable(MultipartStream.java:977)
> >
> >
> at
>
> org.apache.commons.fileupload.MultipartStream$ItemInputStream.read(MultipartStream.java:887)
> > at java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
> > at java.io.BufferedInputStream.read(BufferedInputStream.java:334) -
> > locked java.io.BufferedInputStream@6e81abfc at
> > java.io.FilterInputStream.read(FilterInputStream.java:107)
>
> Obviously, this thread is reading a file upload. Also these:
>
> "http-bio-8888-exec-21" Id=27309 RUNNABLE (in native)
> java.lang.Thread.State: RUNNABLE
> "http-bio-8888-exec-26" Id=27314 RUNNABLE (in native)
> java.lang.Thread.State: RUNNABLE
>
> This thread is making an outgoing HTTP request, which could be hanging:
>
> > "Thread-27550" Id=27623 RUNNABLE (in native)
> > java.lang.Thread.State: RUNNABLE at
> > java.net.SocketInputStream.socketRead0(Native Method) at
> > java.net.SocketInputStream.read(SocketInputStream.java:152) at
> > java.net.SocketInputStream.read(SocketInputStream.java:122) at
> > sun.security.ssl.InputRecord.readFully(InputRecord.java:442) at
> > sun.security.ssl.InputRecord.read(InputRecord.java:480) at
> > sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:934) -
> > locked java.lang.Object@659eed19 at
> > sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:891)
> >
> >
> at sun.security.ssl.AppInputStream.read(AppInputStream.java:102)
> > - locked sun.security.ssl.AppInputStream@7c98d0ab at
> >
> org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
> >
> >
> at
>
> org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
> > at
> >
> org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
> >
> >
> at
>
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
> > at
> >
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
> >
> >
> at
>
> org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
> > at
> >
> org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
> >
> >
> at
>
> org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
> > at
> >
> org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
> >
> >
> at
>
> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
> > at
> >
> org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
> >
> >
> at
>
> org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
> > at
> >
> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
> >
> >
> at
>
> org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
> > at
> >
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
> >
> >
> at sun.reflect.GeneratedMethodAccessor98.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >
> >
> at java.lang.reflect.Method.invoke(Method.java:606)
> > at
> >
> lucee.commons.net.http.httpclient4.HTTPEngine4Impl.execute(HTTPEngine4Impl.java:416)
> >
> >
> at
>
> lucee.commons.net.http.httpclient4.HTTPEngine4Impl._invoke(HTTPEngine4Impl.java:252)
> > at
> >
> lucee.commons.net.http.httpclient4.HTTPEngine4Impl.get(HTTPEngine4Impl.java:112)
> >
> >
> at lucee.commons.net.http.HTTPEngine.get(HTTPEngine.java:86)
> > at
> > lucee.runtime.schedule.ExecutionThread.execute(ExecutionThread.java:108)
> >
> >
> at lucee.runtime.schedule.ExecutionThread.run(ExecutionThread.java:58)
>
>
> Same with this one:
>
> "Thread-27551" Id=27624 RUNNABLE (in native)
> java.lang.Thread.State: RUNNABLE
>
> But neither of the above threads making HTTP requests are
> request-processing threads: they are independent and shouldn't block
> incoming requests.
>
> You have 12 total request-processing threads: the ones named
> "http-bio-8888-exec-[#]". Most of them aren't doing anything -- just
> waiting on a new request to come in.
>
> You have an AJP connector defined, but it looks like it's not being used.
>
> Initially, you said that you used mod_proxy from httpd. Are you
> expecting to use mod_proxy_http or mod_proxy_ajp?
mod_proxy_http, as far as I know, but see below...
> It certainly looks
> like you are using the HTTP flavor of mod_proxy since your
> file-uploads are going to the HTTP connector on port 8888 instead of
> the AJP connector on port 8009. If you don't need the AJP connector,
> you might want to disable it and get a small amount of resources back.
>
Okay, thanks.
When you try to make a connection during these incidents, do you get
> any errors on the httpd side?
Unfortunately, at the time, I thought this would be easy to reproduce, so I
didn't take enough notes (including the time of the problem on my load test
box). Here are some errors that I think were just created by the app timing
out requests. I don't think these were associated with the failed requests,
so take these entries (from httpd's error_log) with a grain of salt:
https://gist.github.com/jamiejackson/2f16ca83bdfc9c8f8795#file-errors-txt
I'll capture this information next time I trigger a crash.
> I know mod_jk will complain if it can't
> make a connection or if there is a timeout... I suspect mod_proxy_http
> will do the same.
>
> Can you post your <Connector> configuration from conf/server.xml?
> Remember to remove any sensitive information that you might have in there.
>
I think this is what you're after:
<Connector port="8888" protocol="HTTP/1.1"
connectionTimeout="93000"
redirectPort="8443" />
FWIW, I'm also noticing this:
<Host name="127.0.0.1" appBase="webapps"
unpackWARs="false" autoDeploy="false">
<!-- Access log processes all example.
Documentation at: /docs/config/valve.html
Note: The pattern used is equivalent to using pattern="common"
-->
<Valve className="org.apache.catalina.valves.AccessLogValve"
directory="logs"
prefix="localhost_access_log." suffix=".txt"
pattern="%h %l %u %t "%r" %s %b"
resolveHosts="false"/>
<!-- visit modcfml.org for details on mod_cfml configuration
options -->
<Valve className="mod_cfml.core"
loggingEnabled="false"
waitForContext="3"
maxContexts="200"
timeBetweenContexts="30000"
/>
</Host>
Also, what do your ProxyPass/ProxyPassReverse configuration directives
> look like in httpd.conf?
The apache config has:
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_balancer_module modules/mod_proxy_balancer.so
LoadModule proxy_ftp_module modules/mod_proxy_ftp.so
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule proxy_ajp_module modules/mod_proxy_ajp.so
LoadModule proxy_connect_module modules/mod_proxy_connect.so
And the virtual host has:
https://gist.github.com/jamiejackson/2f16ca83bdfc9c8f8795#file-virtual-host
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Comment: GPGTools - http://gpgtools.org
>
> iQIcBAEBCAAGBQJWHQhoAAoJEBzwKT+lPKRYoVwP/08YV4m1q3jXp+ixsPaPReEf
> m9wnqPlWpX9tGjwVcsYvnqMPTJe8Ew+yzIFZ5NkPsmQk/eeKQFMYBsr0yDje4Nl0
> Suk+OfNXhdL7RUn/PZg1Rd1xl7hcVeJmgOD6HRIcXAsnB+iTpCYpR+FtTnoGosia
> PQ64AKEZR7KliP0bkM04NY4hHa+eA5vp2kGyGAAkA2g+sxUUXgiAVoUYMbLYwetv
> CKqwlCFcG+JF9M2qi4n8vVwpsTqdLrInlO+yAiejZoQuHaSefnolIkFE0y+08bnz
> Yv1nhNXwIOJAIvsl2BFoQIXXnpVh2ncILLIzhrcBBsazBy0RxccKsOJR0F6SKEWg
> 3JLD6z/3CxtuW28KU2nFIQiKEQHzhZ+fV1NAp3XrHaQPncFGlNGPOUgy4Ggqxy3j
> eAOkN1si3GFRm+ILELDO4ekKQCoAAdWs3MMpVTOKdL6mMO1OiYGPHzVTclgLloxc
> o6FVgy81s6Hk9UJlu5enZJu9I+5tFxlD1gXrBmG7ryCOyiDn9pHmZ68EBpe3j+Zf
> HPUtt7XP5eTVZp8vlVS3ebBHjrMkxUYGuBKYZF7prGfA5LE42nctdMslneiyZJJc
> AOrcp32bgJFCcXmdBovD2UrJu0Evc4su3vV4JW9PuFSE1i3ntFLKuqL4FiArf1ls
> +0KkHZDkU8C3MP3MCySA
> =5Mun
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: Monitoring Connections
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Jamie,
On 10/9/15 10:03 AM, Jamie Jackson wrote:
> Here's the stack trace dump:
> https://gist.github.com/jamiejackson/ca2a49d2c8afac496067
>
> FWIW, I've been trying to come up with a reliable test case to
> trigger the problem, but I haven't nailed it yet. I've suspected
> that it's related to file (large or slow) HTTP file uploads, and
> that's what I was running at the time, which helps to explain the
> java.net.SocketInputStream.socketRead0 traces.
Your server looks mostly idle. It also looks like it's running both
Tomcat and Jetty in the same instance. What's going on?
> "http-bio-8888-exec-19" Id=27307 RUNNABLE (in native)
> java.lang.Thread.State: RUNNABLE at
> java.net.SocketInputStream.socketRead0(Native Method) at
> java.net.SocketInputStream.read(SocketInputStream.java:152) at
> java.net.SocketInputStream.read(SocketInputStream.java:122) at
> org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:535)
>
>
at
org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:504)
> at
> org.apache.coyote.http11.InternalInputBuffer$InputStreamInputBuffer.doRead(InternalInputBuffer.java:566)
>
>
at
org.apache.coyote.http11.filters.IdentityInputFilter.doRead(IdentityInputFilter.java:137)
> at
> org.apache.coyote.http11.AbstractInputBuffer.doRead(AbstractInputBuffer.java:348)
>
>
at org.apache.coyote.Request.doRead(Request.java:422)
> at
> org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:290)
>
>
at org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:390)
> at
> org.apache.catalina.connector.InputBuffer.readByte(InputBuffer.java:304)
>
>
at
org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:106)
> at
> com.intergral.fusionreactor.j2ee.filter.surrogate.FusionReactorServletRequestProxy$1.read(FusionReactorServletRequestProxy.java:123)
>
>
at java.io.InputStream.read(InputStream.java:179)
> at
> org.apache.commons.fileupload.MultipartStream$ItemInputStream.makeAvailable(MultipartStream.java:977)
>
>
at
org.apache.commons.fileupload.MultipartStream$ItemInputStream.read(MultipartStream.java:887)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:273)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:334) -
> locked java.io.BufferedInputStream@6e81abfc at
> java.io.FilterInputStream.read(FilterInputStream.java:107)
Obviously, this thread is reading a file upload. Also these:
"http-bio-8888-exec-21" Id=27309 RUNNABLE (in native)
java.lang.Thread.State: RUNNABLE
"http-bio-8888-exec-26" Id=27314 RUNNABLE (in native)
java.lang.Thread.State: RUNNABLE
This thread is making an outgoing HTTP request, which could be hanging:
> "Thread-27550" Id=27623 RUNNABLE (in native)
> java.lang.Thread.State: RUNNABLE at
> java.net.SocketInputStream.socketRead0(Native Method) at
> java.net.SocketInputStream.read(SocketInputStream.java:152) at
> java.net.SocketInputStream.read(SocketInputStream.java:122) at
> sun.security.ssl.InputRecord.readFully(InputRecord.java:442) at
> sun.security.ssl.InputRecord.read(InputRecord.java:480) at
> sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:934) -
> locked java.lang.Object@659eed19 at
> sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:891)
>
>
at sun.security.ssl.AppInputStream.read(AppInputStream.java:102)
> - locked sun.security.ssl.AppInputStream@7c98d0ab at
> org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
>
>
at
org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
> at
> org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
>
>
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
> at
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
>
>
at
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
> at
> org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
>
>
at
org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
> at
> org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
>
>
at
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
> at
> org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
>
>
at
org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
> at
> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
>
>
at
org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
> at
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
>
>
at sun.reflect.GeneratedMethodAccessor98.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
>
at java.lang.reflect.Method.invoke(Method.java:606)
> at
> lucee.commons.net.http.httpclient4.HTTPEngine4Impl.execute(HTTPEngine4Impl.java:416)
>
>
at
lucee.commons.net.http.httpclient4.HTTPEngine4Impl._invoke(HTTPEngine4Impl.java:252)
> at
> lucee.commons.net.http.httpclient4.HTTPEngine4Impl.get(HTTPEngine4Impl.java:112)
>
>
at lucee.commons.net.http.HTTPEngine.get(HTTPEngine.java:86)
> at
> lucee.runtime.schedule.ExecutionThread.execute(ExecutionThread.java:108)
>
>
at lucee.runtime.schedule.ExecutionThread.run(ExecutionThread.java:58)
Same with this one:
"Thread-27551" Id=27624 RUNNABLE (in native)
java.lang.Thread.State: RUNNABLE
But neither of the above threads making HTTP requests are
request-processing threads: they are independent and shouldn't block
incoming requests.
You have 12 total request-processing threads: the ones named
"http-bio-8888-exec-[#]". Most of them aren't doing anything -- just
waiting on a new request to come in.
You have an AJP connector defined, but it looks like it's not being used.
Initially, you said that you used mod_proxy from httpd. Are you
expecting to use mod_proxy_http or mod_proxy_ajp? It certainly looks
like you are using the HTTP flavor of mod_proxy since your
file-uploads are going to the HTTP connector on port 8888 instead of
the AJP connector on port 8009. If you don't need the AJP connector,
you might want to disable it and get a small amount of resources back.
When you try to make a connection during these incidents, do you get
any errors on the httpd side? I know mod_jk will complain if it can't
make a connection or if there is a timeout... I suspect mod_proxy_http
will do the same.
Can you post your <Connector> configuration from conf/server.xml?
Remember to remove any sensitive information that you might have in there.
Also, what do your ProxyPass/ProxyPassReverse configuration directives
look like in httpd.conf?
- -chris
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org
iQIcBAEBCAAGBQJWHQhoAAoJEBzwKT+lPKRYoVwP/08YV4m1q3jXp+ixsPaPReEf
m9wnqPlWpX9tGjwVcsYvnqMPTJe8Ew+yzIFZ5NkPsmQk/eeKQFMYBsr0yDje4Nl0
Suk+OfNXhdL7RUn/PZg1Rd1xl7hcVeJmgOD6HRIcXAsnB+iTpCYpR+FtTnoGosia
PQ64AKEZR7KliP0bkM04NY4hHa+eA5vp2kGyGAAkA2g+sxUUXgiAVoUYMbLYwetv
CKqwlCFcG+JF9M2qi4n8vVwpsTqdLrInlO+yAiejZoQuHaSefnolIkFE0y+08bnz
Yv1nhNXwIOJAIvsl2BFoQIXXnpVh2ncILLIzhrcBBsazBy0RxccKsOJR0F6SKEWg
3JLD6z/3CxtuW28KU2nFIQiKEQHzhZ+fV1NAp3XrHaQPncFGlNGPOUgy4Ggqxy3j
eAOkN1si3GFRm+ILELDO4ekKQCoAAdWs3MMpVTOKdL6mMO1OiYGPHzVTclgLloxc
o6FVgy81s6Hk9UJlu5enZJu9I+5tFxlD1gXrBmG7ryCOyiDn9pHmZ68EBpe3j+Zf
HPUtt7XP5eTVZp8vlVS3ebBHjrMkxUYGuBKYZF7prGfA5LE42nctdMslneiyZJJc
AOrcp32bgJFCcXmdBovD2UrJu0Evc4su3vV4JW9PuFSE1i3ntFLKuqL4FiArf1ls
+0KkHZDkU8C3MP3MCySA
=5Mun
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Monitoring Connections
Posted by Jamie Jackson <ja...@gmail.com>.
Here's the stack trace dump:
https://gist.github.com/jamiejackson/ca2a49d2c8afac496067
FWIW, I've been trying to come up with a reliable test case to trigger the
problem, but I haven't nailed it yet. I've suspected that it's related to
file (large or slow) HTTP file uploads, and that's what I was running at
the time, which helps to explain the java.net.SocketInputStream.socketRead0
traces.
Thanks,
Jamie
On Fri, Oct 9, 2015 at 7:32 AM, Jamie Jackson <ja...@gmail.com> wrote:
> Thanks, so far, everyone.
>
> I do already have a JVM monitoring tool in action--FusionReactor. I'm not
> very familiar with some of the other offerings, but FR has seemed
> full-featured to me, so I'm going to continue with FR until I find out
> if/that it's missing some key aspect.
>
> I managed to trigger the problem on my load test server yesterday, and
> grabbed a stack trace from FR. Sometimes, the problem identified by a stack
> trace is obvious (to me), but I didn't spot anything in this one, at a
> glance.
>
> I"ll post the stack trace once I get back to work, and we can start there.
>
> Thanks,
> Jamie
>
> On Thu, Oct 8, 2015 at 3:25 PM, Leon Rosenberg <ro...@gmail.com>
> wrote:
>
>> On Wed, Oct 7, 2015 at 11:59 PM, Aurélien Terrestris <
>> aterrestris@gmail.com>
>> wrote:
>>
>> > Hi Jamie,
>> > ....
>> >
>> > If you enjoy live monitoring, you need to have a look on Christopher's
>> > presentation (
>> >
>> >
>> http://events.linuxfoundation.org/sites/events/files/slides/Monitoring%20Apache%20Tomcat%20with%20JMX.pdf
>> > ) who posts many answers to this mailing list.
>> >
>> >
>> > or if you want to go deeper, try MoSKito (http://www.moskito.org)
>>
>> regards
>> Leon
>>
>>
>> >
>> >
>>
>
>
Re: Monitoring Connections
Posted by Jamie Jackson <ja...@gmail.com>.
Thanks, so far, everyone.
I do already have a JVM monitoring tool in action--FusionReactor. I'm not
very familiar with some of the other offerings, but FR has seemed
full-featured to me, so I'm going to continue with FR until I find out
if/that it's missing some key aspect.
I managed to trigger the problem on my load test server yesterday, and
grabbed a stack trace from FR. Sometimes, the problem identified by a stack
trace is obvious (to me), but I didn't spot anything in this one, at a
glance.
I"ll post the stack trace once I get back to work, and we can start there.
Thanks,
Jamie
On Thu, Oct 8, 2015 at 3:25 PM, Leon Rosenberg <ro...@gmail.com>
wrote:
> On Wed, Oct 7, 2015 at 11:59 PM, Aurélien Terrestris <
> aterrestris@gmail.com>
> wrote:
>
> > Hi Jamie,
> > ....
> >
> > If you enjoy live monitoring, you need to have a look on Christopher's
> > presentation (
> >
> >
> http://events.linuxfoundation.org/sites/events/files/slides/Monitoring%20Apache%20Tomcat%20with%20JMX.pdf
> > ) who posts many answers to this mailing list.
> >
> >
> > or if you want to go deeper, try MoSKito (http://www.moskito.org)
>
> regards
> Leon
>
>
> >
> >
>
Re: Monitoring Connections
Posted by Leon Rosenberg <ro...@gmail.com>.
On Wed, Oct 7, 2015 at 11:59 PM, Aurélien Terrestris <at...@gmail.com>
wrote:
> Hi Jamie,
> ....
>
> If you enjoy live monitoring, you need to have a look on Christopher's
> presentation (
>
> http://events.linuxfoundation.org/sites/events/files/slides/Monitoring%20Apache%20Tomcat%20with%20JMX.pdf
> ) who posts many answers to this mailing list.
>
>
> or if you want to go deeper, try MoSKito (http://www.moskito.org)
regards
Leon
>
>
Re: Monitoring Connections
Posted by Aurélien Terrestris <at...@gmail.com>.
Chris,
I disagree in some way with you ;)
Instrumentation is difficult because you (as a developer ?) want to go
deep, but it is not necessary for a production team to go that deep to know
when something is going wrong. It's a developer challenge, and he could
choose to rely only on good logging. Whatever happens, there should be at
least some logs for the production team, then they can 'grep' for SEVERE or
Exceptions before asking more help to the N2 or N3 team.
"1. Java doesn't directly support SNMP;"
Sorry, but it does (
http://docs.oracle.com/javase/7/docs/technotes/guides/management/snmp.html
) and we use it with good results (deltas on GCs for example). Maybe are
you confusing with the fact that Java has no built-in API for SNMP software
developers.
"2. If the JVM is braindead, the SNMP agent can't announce any state
to the managers."
The SNMP server will be unable to check the JVM, it will raise an alert.
Easy, cheap, and in most cases enough ;)
best regards
A.T.
2015-10-24 14:18 GMT+02:00 Christopher Schultz <chris@christopherschultz.net
>:
> Aurélien,
>
> On 10/23/15 6:47 PM, Aurélien Terrestris wrote:
> > "Live monitoring is the only way to go, unless you want your customers
> > to surprise you with performance problems on your own site. :)"
> >
> > Monitoring, ok, but alerting is critical for large scale production
> > environments.
>
> Apologies... I already considered "alerting" to be a critical component
> of "monitoring", so I didn't mention it specifically. Instrumentation is
> the hard part; alerting is relatively easy once instrumentation has been
> done.
>
> > You mention JMX callbacks for alerting, but it would be worth
> > that you mention SNMP since it's very easy to set up and efficient.
>
> SNMP is great, but there are a few problems with using it with Java:
>
> 1. Java doesn't directly support SNMP; instead, you have to use
> a 3rd-party library (which isn't difficult to find). This may be
> a barrier for certain organizations who only want to use their
> own code in their own applications.
>
> 2. If the JVM is braindead, the SNMP agent can't announce any state
> to the managers. This requires the use of probe-based instrumentation
> (active probing from the outside of the application), unless you want
> to have an agent send a heartbeat for every status change.
>
> > We also can raise alerts on deltas with this method, and see a
> > problem before the CPU says something bad is going to happen.
>
> There's nothing stopping you from doing deltas with other techniques. My
> presentation even includes a tool that allows you to do deltas.
>
> -chris
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: Monitoring Connections
Posted by Jamie Jackson <ja...@gmail.com>.
By the way, I had to back-burner this investigation due to other things
that have come up. When those are out of the way, I'll pick up where I left
off. Thanks for the discussion so far.
(I don't know anything about SNMP, etc., so I'm only following that thread
very loosely until I have time to concentrate on it.)
On Wed, Nov 4, 2015 at 3:25 AM, Aurélien Terrestris <at...@gmail.com>
wrote:
> Hi Chris
>
> I still recommend SNMP, maybe I expressed myself incorrectly but Java SNMP
> support is made for monitoring. I also provided a bad link, here is the
> good one :
> http://docs.oracle.com/javase/7/docs/technotes/guides/management/snmp.html
>
> When using the right server, the SNMP will also provide alerting, both for
> counters and deltas. This is the critical point for large scale teams with
> N1 support, although developers often want something else and need more
> (logs, instrumentation).
>
> In "Effective Monitoring and Alerting, O'Reilly" we find these definitions
> that we already agree, I believe :
>
> "The main purpose of monitoring is to gain near real-time insight into the
> current state of the system, in the context of its recent performance"
>
> "The core functionality of an alarm is to trigger on detection of abnormal
> timeseries behavior, but the alerting system should also support the
> aggregation and conditional suppression of alarms."
>
> In my opinion, SNMP could be more used, and I notice that nobody is talking
> about this technology for years on the mailing-list. It's a pity to ignore
> solutions.
>
>
> best regards
>
>
>
>
> 2015-11-03 16:05 GMT+01:00 Christopher Schultz <
> chris@christopherschultz.net
> >:
>
> > Aurelien,
> >
> > On 11/2/15 5:54 PM, Aurélien Terrestris wrote:
> > > Either my reply was not read, or I'm surprised nobody is answering
> here.
> > >
> > > "1. Java doesn't directly support SNMP;"
> > >
> > > It does, officially :
> > http://docs.oracle.com/javase/7/docs/technotes/guides/
> > > management/snmp.html and it's working pretty well with Tomcat besides
> > being
> > > easy to set up.
> >
> > I meant there was no client API
> >
> > > "2. If the JVM is braindead, the SNMP agent can't announce any state
> to
> > > the managers."
> > >
> > > Agent ? This is the SNMP server which is polling the Tomcat here. When
> > > Tomcat is dead, the server cannot poll anymore and raises an alert.
> >
> > You were recommending the use of SNMP as an "alerting" alternative to
> > "monitoring". I assumed you had meant that more passive "alerting" (e.g.
> > only take action when a problem is occurring) was more efficient than
> > "monitoring" (which I took to mean polling a service to check its
> status).
> >
> > So polling is less efficient but more reliable. Whether you use SNMP,
> > JMX, HTTP, or whatever, *polling* a component is better than having a
> > component try to report that it's failing, because part of the failure
> > could be its inability to report the failure state.
> >
> > This isn't about SNMP versus some other technology. It's about probing
> > versus ... not doing so, I guess.
> >
> > > You see how alerting is even easier without instrumenting, but maybe
> you
> > > want to go deeper. But then, you probably start mixing what's necessary
> > for
> > > N1 support with what's helpful for developers. They have different
> jobs.
> >
> > What is your definition of "instrumentation"? What about "probing" or
> > "alerting"? I'm certainly confused with your use of terms, especially
> > when you equate one of them with a particular technology.
> >
> > -chris
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> > For additional commands, e-mail: users-help@tomcat.apache.org
> >
> >
>
Re: Monitoring Connections
Posted by Aurélien Terrestris <at...@gmail.com>.
Hi Chris
I still recommend SNMP, maybe I expressed myself incorrectly but Java SNMP
support is made for monitoring. I also provided a bad link, here is the
good one :
http://docs.oracle.com/javase/7/docs/technotes/guides/management/snmp.html
When using the right server, the SNMP will also provide alerting, both for
counters and deltas. This is the critical point for large scale teams with
N1 support, although developers often want something else and need more
(logs, instrumentation).
In "Effective Monitoring and Alerting, O'Reilly" we find these definitions
that we already agree, I believe :
"The main purpose of monitoring is to gain near real-time insight into the
current state of the system, in the context of its recent performance"
"The core functionality of an alarm is to trigger on detection of abnormal
timeseries behavior, but the alerting system should also support the
aggregation and conditional suppression of alarms."
In my opinion, SNMP could be more used, and I notice that nobody is talking
about this technology for years on the mailing-list. It's a pity to ignore
solutions.
best regards
2015-11-03 16:05 GMT+01:00 Christopher Schultz <chris@christopherschultz.net
>:
> Aurelien,
>
> On 11/2/15 5:54 PM, Aurélien Terrestris wrote:
> > Either my reply was not read, or I'm surprised nobody is answering here.
> >
> > "1. Java doesn't directly support SNMP;"
> >
> > It does, officially :
> http://docs.oracle.com/javase/7/docs/technotes/guides/
> > management/snmp.html and it's working pretty well with Tomcat besides
> being
> > easy to set up.
>
> I meant there was no client API
>
> > "2. If the JVM is braindead, the SNMP agent can't announce any state to
> > the managers."
> >
> > Agent ? This is the SNMP server which is polling the Tomcat here. When
> > Tomcat is dead, the server cannot poll anymore and raises an alert.
>
> You were recommending the use of SNMP as an "alerting" alternative to
> "monitoring". I assumed you had meant that more passive "alerting" (e.g.
> only take action when a problem is occurring) was more efficient than
> "monitoring" (which I took to mean polling a service to check its status).
>
> So polling is less efficient but more reliable. Whether you use SNMP,
> JMX, HTTP, or whatever, *polling* a component is better than having a
> component try to report that it's failing, because part of the failure
> could be its inability to report the failure state.
>
> This isn't about SNMP versus some other technology. It's about probing
> versus ... not doing so, I guess.
>
> > You see how alerting is even easier without instrumenting, but maybe you
> > want to go deeper. But then, you probably start mixing what's necessary
> for
> > N1 support with what's helpful for developers. They have different jobs.
>
> What is your definition of "instrumentation"? What about "probing" or
> "alerting"? I'm certainly confused with your use of terms, especially
> when you equate one of them with a particular technology.
>
> -chris
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: Monitoring Connections
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Aurelien,
On 11/2/15 5:54 PM, Aurélien Terrestris wrote:
> Either my reply was not read, or I'm surprised nobody is answering here.
>
> "1. Java doesn't directly support SNMP;"
>
> It does, officially : http://docs.oracle.com/javase/7/docs/technotes/guides/
> management/snmp.html and it's working pretty well with Tomcat besides being
> easy to set up.
I meant there was no client API
> "2. If the JVM is braindead, the SNMP agent can't announce any state to
> the managers."
>
> Agent ? This is the SNMP server which is polling the Tomcat here. When
> Tomcat is dead, the server cannot poll anymore and raises an alert.
You were recommending the use of SNMP as an "alerting" alternative to
"monitoring". I assumed you had meant that more passive "alerting" (e.g.
only take action when a problem is occurring) was more efficient than
"monitoring" (which I took to mean polling a service to check its status).
So polling is less efficient but more reliable. Whether you use SNMP,
JMX, HTTP, or whatever, *polling* a component is better than having a
component try to report that it's failing, because part of the failure
could be its inability to report the failure state.
This isn't about SNMP versus some other technology. It's about probing
versus ... not doing so, I guess.
> You see how alerting is even easier without instrumenting, but maybe you
> want to go deeper. But then, you probably start mixing what's necessary for
> N1 support with what's helpful for developers. They have different jobs.
What is your definition of "instrumentation"? What about "probing" or
"alerting"? I'm certainly confused with your use of terms, especially
when you equate one of them with a particular technology.
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Monitoring Connections
Posted by Aurélien Terrestris <at...@gmail.com>.
Either my reply was not read, or I'm surprised nobody is answering here.
"1. Java doesn't directly support SNMP;"
It does, officially : http://docs.oracle.com/javase/7/docs/technotes/guides/
management/snmp.html and it's working pretty well with Tomcat besides being
easy to set up.
"2. If the JVM is braindead, the SNMP agent can't announce any state to
the managers."
Agent ? This is the SNMP server which is polling the Tomcat here. When
Tomcat is dead, the server cannot poll anymore and raises an alert.
You see how alerting is even easier without instrumenting, but maybe you
want to go deeper. But then, you probably start mixing what's necessary for
N1 support with what's helpful for developers. They have different jobs.
best regards
A.T.
2015-10-24 14:18 GMT+02:00 Christopher Schultz <chris@christopherschultz.net
>:
> Aurélien,
>
> On 10/23/15 6:47 PM, Aurélien Terrestris wrote:
> > "Live monitoring is the only way to go, unless you want your customers
> > to surprise you with performance problems on your own site. :)"
> >
> > Monitoring, ok, but alerting is critical for large scale production
> > environments.
>
> Apologies... I already considered "alerting" to be a critical component
> of "monitoring", so I didn't mention it specifically. Instrumentation is
> the hard part; alerting is relatively easy once instrumentation has been
> done.
>
> > You mention JMX callbacks for alerting, but it would be worth
> > that you mention SNMP since it's very easy to set up and efficient.
>
> SNMP is great, but there are a few problems with using it with Java:
>
> 1. Java doesn't directly support SNMP; instead, you have to use
> a 3rd-party library (which isn't difficult to find). This may be
> a barrier for certain organizations who only want to use their
> own code in their own applications.
>
> 2. If the JVM is braindead, the SNMP agent can't announce any state
> to the managers. This requires the use of probe-based instrumentation
> (active probing from the outside of the application), unless you want
> to have an agent send a heartbeat for every status change.
>
> > We also can raise alerts on deltas with this method, and see a
> > problem before the CPU says something bad is going to happen.
>
> There's nothing stopping you from doing deltas with other techniques. My
> presentation even includes a tool that allows you to do deltas.
>
> -chris
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: Monitoring Connections
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Aurélien,
On 10/23/15 6:47 PM, Aurélien Terrestris wrote:
> "Live monitoring is the only way to go, unless you want your customers
> to surprise you with performance problems on your own site. :)"
>
> Monitoring, ok, but alerting is critical for large scale production
> environments.
Apologies... I already considered "alerting" to be a critical component
of "monitoring", so I didn't mention it specifically. Instrumentation is
the hard part; alerting is relatively easy once instrumentation has been
done.
> You mention JMX callbacks for alerting, but it would be worth
> that you mention SNMP since it's very easy to set up and efficient.
SNMP is great, but there are a few problems with using it with Java:
1. Java doesn't directly support SNMP; instead, you have to use
a 3rd-party library (which isn't difficult to find). This may be
a barrier for certain organizations who only want to use their
own code in their own applications.
2. If the JVM is braindead, the SNMP agent can't announce any state
to the managers. This requires the use of probe-based instrumentation
(active probing from the outside of the application), unless you want
to have an agent send a heartbeat for every status change.
> We also can raise alerts on deltas with this method, and see a
> problem before the CPU says something bad is going to happen.
There's nothing stopping you from doing deltas with other techniques. My
presentation even includes a tool that allows you to do deltas.
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Monitoring Connections
Posted by Aurélien Terrestris <at...@gmail.com>.
Chris,
"Live monitoring is the only way to go, unless you want your customers
to surprise you with performance problems on your own site. :)"
Monitoring, ok, but alerting is critical for large scale production
environments. You mention JMX callbacks for alerting, but it would be worth
that you mention SNMP since it's very easy to set up and efficient. We also
can raise alerts on deltas with this method, and see a problem before the
CPU says something bad is going to happen. Just my feedback ;)
regards
2015-10-08 19:27 GMT+02:00 Christopher Schultz <chris@christopherschultz.net
>:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Aurélien,
>
> On 10/7/15 5:59 PM, Aurélien Terrestris wrote:
> > when this happens you can do a thread-dump (kill -3 pid on Linux
> > platforms) and you would see if there is a lock on JDBC objects, or
> > anything else synchronized (from the Collections like Hashtable).
> > Not easy for beginners to understand a dump, but worth learning.
>
> +1
>
> This is the only way to find out what's going on. All other monitoring
> solutions will just tell you that *something* is wrong, but this will
> give you details.
>
> > Very often an application will slow down because Garbage
> > Collection, and the GC thread will be the only one working at that
> > time. Please use the verbose settings to have all your Tomcats log
> > GC activity. This is useful for both live and after time analysis.
>
> This is pretty unlikely for the reported observations, since Tomcat
> slows down and stops responding (presumably until restarted). That is
> more likely to be something like database connection pool being
> exhausted due to poor resource management rather than out-of-control
> GC activity.
>
> > You won't monitor mod_proxy, this has no sense.
>
> I disagree. The connection can go down for various reasons. But I
> suspect that's not the problem in this case.
>
> > If you suspect a network flood, do a 'netstat -an' to see how many
> > connections are used by your Apache (both ESTABLISHED and
> > TIME_WAIT). However, you might need to set timeout and flushpacket
> > on your proxypass directive when your application has such
> > requirements (see HTTPD doc for this settings). To know if you're
> > flooded, you can do a 'ps -eaf | grep [h]ttp' and you would see
> > many httpd process, it is similar to the netstat result more or
> > less.
>
> +1
>
> > If you enjoy live monitoring, you need to have a look on
> > Christopher's presentation (
> > http://events.linuxfoundation.org/sites/events/files/slides/Monitoring
> %20Apache%20Tomcat%20with%20JMX.pdf
> >
> >
> ) who posts many answers to this mailing list.
>
> Live monitoring is the only way to go, unless you want your customers
> to surprise you with performance problems on your own site. :)
>
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Comment: GPGTools - http://gpgtools.org
>
> iQIcBAEBCAAGBQJWFqd5AAoJEBzwKT+lPKRYi9AQAIpHkOgXoz2Ntfdf8aH1mjTB
> gdKDu1B5fEgpTYRhV+mQ5rEegkszJ0jUEQdCVjwp2ZjldgsLPP6wXaBw6LdUG3UB
> l63K4tCkDwIiVI8rB0syAeQQ/4uUS7SmiJaLZAwLHl08mDZHs9e/LiKJ47rJjh3+
> 4pkejtbHhobF2fcH61ZM7M5qxPKlzP49dDzXyppbvEG/kk7T9FkIl2TVOT349zkP
> c1jrtC19fKDhbz72c/03/Z1V/clZ+Q3BScYHddlRw0ZXuOEu8LQJRD0VPmEY4pM5
> DTPH+z5spnYUO0hH0FHx/WGHgQ5iCrQHYG8pdkfhDEqSaDnqm4qIFxC4fChIFSuh
> c9eqor7axyOcacHvTCwPudXGuWBxK0LCnigikd83OM/EaKdRwR0ahMqXKEQkRdTn
> QW3vfsOQXXuy0vAFz6fphOkWXCr4SJWVOsAG3f1vlStMs2to4q66hpNQpLFJpcdN
> ectoRaCRvQMLdw9UhdG3z5wTCvPPj9RP8IA/Lxi48QisYOMZwABgh7mMh2+mE7pH
> tTHun4fE2HhdVPqORonHhTWldvu4xYqWav9FNBgg2ywyekAWVcrcFZRLgYhG+T37
> 0neZbSdwd/iLdsZ+sKhGY04CSwAsUZdTh87pZ+8NWH7tvi71JIt7nlI0JgRI5ipw
> Jdty982ew2WiaY0rhspW
> =Ra47
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: Monitoring Connections
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Aurélien,
On 10/7/15 5:59 PM, Aurélien Terrestris wrote:
> when this happens you can do a thread-dump (kill -3 pid on Linux
> platforms) and you would see if there is a lock on JDBC objects, or
> anything else synchronized (from the Collections like Hashtable).
> Not easy for beginners to understand a dump, but worth learning.
+1
This is the only way to find out what's going on. All other monitoring
solutions will just tell you that *something* is wrong, but this will
give you details.
> Very often an application will slow down because Garbage
> Collection, and the GC thread will be the only one working at that
> time. Please use the verbose settings to have all your Tomcats log
> GC activity. This is useful for both live and after time analysis.
This is pretty unlikely for the reported observations, since Tomcat
slows down and stops responding (presumably until restarted). That is
more likely to be something like database connection pool being
exhausted due to poor resource management rather than out-of-control
GC activity.
> You won't monitor mod_proxy, this has no sense.
I disagree. The connection can go down for various reasons. But I
suspect that's not the problem in this case.
> If you suspect a network flood, do a 'netstat -an' to see how many
> connections are used by your Apache (both ESTABLISHED and
> TIME_WAIT). However, you might need to set timeout and flushpacket
> on your proxypass directive when your application has such
> requirements (see HTTPD doc for this settings). To know if you're
> flooded, you can do a 'ps -eaf | grep [h]ttp' and you would see
> many httpd process, it is similar to the netstat result more or
> less.
+1
> If you enjoy live monitoring, you need to have a look on
> Christopher's presentation (
> http://events.linuxfoundation.org/sites/events/files/slides/Monitoring
%20Apache%20Tomcat%20with%20JMX.pdf
>
>
) who posts many answers to this mailing list.
Live monitoring is the only way to go, unless you want your customers
to surprise you with performance problems on your own site. :)
- -chris
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org
iQIcBAEBCAAGBQJWFqd5AAoJEBzwKT+lPKRYi9AQAIpHkOgXoz2Ntfdf8aH1mjTB
gdKDu1B5fEgpTYRhV+mQ5rEegkszJ0jUEQdCVjwp2ZjldgsLPP6wXaBw6LdUG3UB
l63K4tCkDwIiVI8rB0syAeQQ/4uUS7SmiJaLZAwLHl08mDZHs9e/LiKJ47rJjh3+
4pkejtbHhobF2fcH61ZM7M5qxPKlzP49dDzXyppbvEG/kk7T9FkIl2TVOT349zkP
c1jrtC19fKDhbz72c/03/Z1V/clZ+Q3BScYHddlRw0ZXuOEu8LQJRD0VPmEY4pM5
DTPH+z5spnYUO0hH0FHx/WGHgQ5iCrQHYG8pdkfhDEqSaDnqm4qIFxC4fChIFSuh
c9eqor7axyOcacHvTCwPudXGuWBxK0LCnigikd83OM/EaKdRwR0ahMqXKEQkRdTn
QW3vfsOQXXuy0vAFz6fphOkWXCr4SJWVOsAG3f1vlStMs2to4q66hpNQpLFJpcdN
ectoRaCRvQMLdw9UhdG3z5wTCvPPj9RP8IA/Lxi48QisYOMZwABgh7mMh2+mE7pH
tTHun4fE2HhdVPqORonHhTWldvu4xYqWav9FNBgg2ywyekAWVcrcFZRLgYhG+T37
0neZbSdwd/iLdsZ+sKhGY04CSwAsUZdTh87pZ+8NWH7tvi71JIt7nlI0JgRI5ipw
Jdty982ew2WiaY0rhspW
=Ra47
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Monitoring Connections
Posted by Aurélien Terrestris <at...@gmail.com>.
Hi Jamie,
when this happens you can do a thread-dump (kill -3 pid on Linux platforms)
and you would see if there is a lock on JDBC objects, or anything else
synchronized (from the Collections like Hashtable). Not easy for beginners
to understand a dump, but worth learning.
Very often an application will slow down because Garbage Collection, and
the GC thread will be the only one working at that time. Please use the
verbose settings to have all your Tomcats log GC activity. This is useful
for both live and after time analysis.
You won't monitor mod_proxy, this has no sense. If you suspect a network
flood, do a 'netstat -an' to see how many connections are used by your
Apache (both ESTABLISHED and TIME_WAIT). However, you might need to set
timeout and flushpacket on your proxypass directive when your application
has such requirements (see HTTPD doc for this settings). To know if you're
flooded, you can do a 'ps -eaf | grep [h]ttp' and you would see many httpd
process, it is similar to the netstat result more or less.
If you enjoy live monitoring, you need to have a look on Christopher's
presentation (
http://events.linuxfoundation.org/sites/events/files/slides/Monitoring%20Apache%20Tomcat%20with%20JMX.pdf
) who posts many answers to this mailing list.
regards
2015-10-07 22:20 GMT+02:00 Jamie Jackson <ja...@gmail.com>:
> Hi Folks,
>
> I had a server crash on Friday, and I had the following symptoms:
>
> Response time worsened, and when I looked at my real-time monitoring (an
> app called FusionReactor), I saw that the app *seemed* to be
> single-threading (I only saw one request process at a time), then after a
> few minutes, even though resources still looked good, the app no longer
> seemed to receive any requests.
>
> My first guess was that somehow, connections were slowly getting clogged
> until there were none left, then my server was completely unresponsive.
>
> I front Tomcat (7.0.59) with Apache HTTPD, and connect with mod_proxy. (The
> end app is built in Lucee, and is running on RHEL 6.7 and Oracle Java
> 1.7.0_76.)
>
> My instinct is to monitor the connection (pools?) in, I guess, Tomcat and
> mod_proxy, but:
>
>
> 1. I don't know if that's sane.
> 2. I have no clue how one would go about that.
>
> I could use some input.
>
> Thanks,
> Jamie
>