You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by David Kumar <dk...@telestar.de> on 2013/03/12 07:53:13 UTC
AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections
keept open
Hey,
we are still having that issue.
But we could manage to figure out some more stuff.
We made a Tomcat and Java update since that time we had our problem a few times again, also we did some reconfiguration with connectors etc.
The last 2 times we where able to see, that both tomcat by them self where alive. Just ajp on both where dead. We couldn't make a connection either trough 7009 nor 9009. An with our openFiles trick we found a lot of close_wait again, e.g. 200 for 9009. I left the second tomcat on this state for a few ours just to see, what happens. The count of 200 connection with close_wait was kept until a reboot of the tomcat.
I would say with some of our reconfiguration we managed to stop increasing connections. But still we are not sure why our ajp connections dying..
Here is our connector out of Server.xml:
<Connector port="9009" protocol="AJP/1.3" redirectPort="9443" maxThreads="200" connectionTimeout="600000" />
worker.properties:
worker.tomcatX.host=localhost
worker.tomcatX.type=ajp13
worker.tomcatX.fail_on_status=404
worker.tomcatX.lbfactor=1
worker.tomcatX.ping_timeout=1000
worker.tomcatX.ping_mode=A
worker.tomcatX.socket_timeout=10
worker.tomcatX.connection_pool_timeout=600
worker.tomcat1.reference=worker.tomcatX
worker.tomcat1.port=7009
worker.tomcat2.reference=worker.tomcatX
worker.tomcat2.port=9009
worker.loadbalancer.type=lb
worker.loadbalancer.balance_workers=tomcat1,tomcat2
worker.status.type=status
Hopefully one of you guys can give us a hint to fix that problem.
Mit freundlichen Grüßen
David Kumar
Softwareentwickler, B. Sc.
Abteilung Infotech - Interaktiv
TELESTAR-DIGITAL GmbH
Am Weiher 14
D-56766 Ulmen
http://www.telestar.de/
-----Ursprüngliche Nachricht-----
Von: David Kumar [mailto:dkumar@telestar.de]
Gesendet: Dienstag, 22. Januar 2013 07:36
An: Tomcat Users List
Betreff: AW: AW: AW: ajp on 7009 and 9009 : connections keept open
Hey,
last friday I changed our configuration to use a executor.
Here is what I did:
<Connector port="7009" protocol="AJP/1.3" redirectPort="8443" maxThreads="200" executor="active-executor" />
<Executor name="active-executor" namePrefix="activeThread-" maxThreads="200" minSpareThreads="30" maxIdleTime="60000" />
<Connector port="7080" protocol="HTTP/1.1"
connectionTimeout="20000"
redirectPort="8443" executor="active-executor" />
The second tomcat has same configuration besides ports..
Until yesterday it worked like a charm. But at late afternoon one of the tomcats failed again..
I couldn't start the garbagecollection so far..
Any other ideas?
Thanks
Mit freundlichen Grüßen
David Kumar
Softwareentwickler, B. Sc.
Abteilung Infotech - Interaktiv
TELESTAR-DIGITAL GmbH
Am Weiher 14
D-56766 Ulmen
http://www.telestar.de/
-----Ursprüngliche Nachricht-----
Von: David Kumar [mailto:dkumar@telestar.de]
Gesendet: Freitag, 18. Januar 2013 11:19
An: Tomcat Users List; Tomcat Users List
Betreff: AW: AW: AW: ( ajp on 7009 and 9009 not afs3-rmtsys): connections keept open
Hey,
I do that at next deployment. --> I Thursday...
So far I'm trying executor for tomcat. As far I read, when I'm using connectors idle process are forced to be close..
Mit freundlichen Grüßen
David Kumar
Softwareentwickler, B. Sc.
Abteilung Infotech - Interaktiv
TELESTAR-DIGITAL GmbH
Am Weiher 14
D-56766 Ulmen
http://www.telestar.de/
-----Ursprüngliche Nachricht-----
Von: André Warnier [mailto:aw@ice-sa.com]
Gesendet: Freitag, 18. Januar 2013 11:10
An: Tomcat Users List
Betreff: Re: AW: AW: ( ajp on 7009 and 9009 not afs3-rmtsys): connections keept open
David Kumar wrote:
> Hey André,
>
> are you talking about running System.gc()?
Yes.
> That should be possible..
>
> Mit freundlichen Grüßen
> David Kumar
> Softwareentwickler, B. Sc.
> Abteilung Infotech - Interaktiv
> TELESTAR-DIGITAL GmbH
> Am Weiher 14
> D-56766 Ulmen
>
> http://www.telestar.de/
>
>
>
>
> -----Ursprüngliche Nachricht-----
> Von: André Warnier [mailto:aw@ice-sa.com]
> Gesendet: Freitag, 18. Januar 2013 10:07
> An: Tomcat Users List
> Betreff: Re: AW: ( ajp on 7009 and 9009 not afs3-rmtsys): connections keept open
>
> David,
> (and sorry for top-posting here)
>
> just to verify something.
> Can you trigger a Major Garbage Collection at the Tomcat JVM level, at a moment when you
> have all these connections in CLOSE_WAIT, and see if they disappear after the GC ?
>
> If yes, it may give a good clue about where all these CLOSE_WAITs are coming from.
>
>
> David Kumar wrote:
>> Just read this email.. :-)
>>
>> I figured out we are not using executor connector...
>>
>>
>> Mit freundlichen Grüßen
>> David Kumar
>> Softwareentwickler, B. Sc.
>> Abteilung Infotech - Interaktiv
>> TELESTAR-DIGITAL GmbH
>> Am Weiher 14
>> D-56766 Ulmen
>> http://www.telestar.de/
>>
>>
>>
>>
>> -----Ursprüngliche Nachricht-----
>> Von: David Kumar [mailto:dkumar@telestar.de]
>> Gesendet: Freitag, 18. Januar 2013 09:11
>> An: Tomcat Users List
>> Betreff: AW: ( ajp on 7009 and 9009 not afs3-rmtsys): connections keept open
>>
>> here you are with attachment :-)
>>
>>
>> btw: in mod_jk.log I found some
>> [Thu Jan 17 23:00:08 2013] [11196:140336689317632] [error] ajp_get_reply::jk_ajp_common.c (2055): (tomcat2) Tomcat is down or refused connection. No response has been sent to the client (yet)
>> [Thu Jan 17 23:00:08 2013] [11196:140336689317632] [error] ajp_service::jk_ajp_common.c (2559): (tomcat2) connecting to tomcat failed.
>>
>>
>> but realy just a few one...
>>
>>
>> Mit freundlichen Grüßen
>> David Kumar
>> Softwareentwickler, B. Sc.
>> Abteilung Infotech - Interaktiv
>> TELESTAR-DIGITAL GmbH
>> Am Weiher 14
>> D-56766 Ulmen
>>
>> http://www.telestar.de/
>>
>>
>>
>>
>> -----Ursprüngliche Nachricht-----
>> Von: David Kumar
>> Gesendet: Freitag, 18. Januar 2013 09:08
>> An: 'Tomcat Users List'
>> Betreff: ( ajp on 7009 and 9009 not afs3-rmtsys): connections keept open
>>
>> Hey,
>>
>> thanks for reply. I got that about the Apache configuration. Since we had our problem yesterday, again and there was no error at the apache logs I'm willing to say that is not the main problem and I have to check that, when my main problem is solved.. :-)
>>
>>
>>
>> I agree with you about wrong reporting of service. Its just shown up as afs3 because these service uses 7009 per default. But I'm using 7009 and 9009 for ajp.
>>
>>
>> So doesn't this mean there is a connection problems between my Apache and the tomcats?
>>
>> You're right, both Webapps doing the same and are configured identically besides the ports.
>>
>> I'm using more than one database, but all of them are used through a database pool. If there is a bug, I think I should have found some error at my logs like no free connection or something like that. As there is no such log entry I'm willing to say that my database connections processing like they should.
>>
>> Basically on each tomcat there are running two services. One is a axis2 project. Our CRM is posting customer data to this webapp. This data will be persisted into a database. Depending on the information given by our CRM axis sends a email.
>>
>> The second one is basically a cache for our websites. We have a PIM with all our product data. These app is gathering all the data from PIM and a CMS and is merging these information together so that the data can be displayed.
>> All the mentioned data is hold in different "cache objects". Also some communication with our ERP and some databases are made trough this app.
>>
>> The second app is a REST service. Information will be posted as POST or GET request to it. Most likely the responses are JSON Object.
>>
>> When ever one webApp is reloading (automatically or manually) itself, the result will be posted to the other tomcat/webapp as a serialized object, so the other on do not need to reload it self.
>>
>> I can't say how many SMB files there are, it is depending on some other stuff so it is dynamic.
>>
>> Attached you can find a printed list by lsof.
>>
>> There you can see a really strange thing. Yesterday just tomcat2 had the problem with to many open files. A few days before it was just tomcat1 having this problem.
>>
>> Now let my answer your question:
>>
>> 1. That is hard to say, I guess I have to do some more investigation on our logfiles.
>>
>> 2. / 3. Here is my httpd.conf:
>> <IfModule mpm_worker_module>
>> ThreadLimit 25
>> StartServers 2
>> MaxClients 150
>> MinSpareThreads 25
>> MaxSpareThreads 75
>> ThreadsPerChild 25
>> MaxRequestsPerChild 4000
>> </IfModule>
>>
>> we are using worker ....
>>
>> And here are our tomcat connectors again:
>> tomcat1:
>>
>> <Connector port="7080" protocol="HTTP/1.1" connectionTimeout="20000" redirectPort="8443"/>
>>
>>
>> <Connector port="7009" protocol="AJP/1.3" redirectPort="8443"/>
>>
>>
>> tomcat2:
>> <Connector port="9080" protocol="HTTP/1.1" connectionTimeout="20000" redirectPort="9443"/>
>>
>> <Connector port="9009" protocol="AJP/1.3" redirectPort="9443"/>
>>
>>
>> Okay we are not using executor.. I will check that..
>>
>> You probably read my copy-paste error. I did copy some comments out of out server config --> Sry again.
>>
>> 4. we are using..
>> 5. via a multipart message sending to the other tomcat.
>> 6. I don't think so also because of that the connections are kept open on our ajp ports.
>>
>> I know that "CLOSE_WAIT" means, waiting for connections to be closed, but wondering that it is not closing..
>>
>>
>> Thanks again
>>
>> Mit freundlichen Grüßen
>> David Kumar
>> Softwareentwickler, B. Sc.
>> Abteilung Infotech - Interaktiv
>> TELESTAR-DIGITAL GmbH
>> Am Weiher 14
>> D-56766 Ulmen
>> http://www.telestar.de/
>>
>>
>>
>>
>> -----Ursprüngliche Nachricht-----
>> Von: Christopher Schultz [mailto:chris@christopherschultz.net]
>> Gesendet: Donnerstag, 17. Januar 2013 18:38
>> An: Tomcat Users List
>> Betreff: Re: AW: AW: afs3-rmtsys: connections keept open
>>
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA256
>>
>> David,
>>
>> On 1/17/13 1:49 AM, David Kumar wrote:
>>> I just checked /var/logs/apache2/error.logs. And found following
>>> errors:
>>>
>>> [Wed Jan 16 15:14:46 2013] [error] server is within MinSpareThreads
>>> of MaxClients, consider raising the MaxClients setting [Wed Jan 16
>>> 15:14:56 2013] [error] server reached MaxClients setting, consider
>>> raising the MaxClients setting
>> So you are maxing-out your connections: you are experiencing enough
>> load that your configuration cannot handle any more connections:
>> requests are being queued by the TCP/IP stack and some requests may be
>> rejected entirely depending upon the queue length of the socket.
>>
>> The first question to ask yourself is whether or not your hardware can
>> take more than you have it configured to accept. For instance, if your
>> load average, memory usage, and response time are all reasonable, then
>> you could probably afford to raise your MaxClients setting in httpd.
>>
>> Note that the above has almost nothing to do with Tomcat: it only has
>> to do with Apache httpd.
>>
>>> Yesterday my problem occurred about the same time.
>> So, the problem is that Tomcat cannot handle your peak load due to a
>> file handle limitation. IIRC, your current file handle limit for the
>> Tomcat process is 4096.
>>
>>> I'm checking every five minutes how many open files there are:
>>>
>>> count open files started: 01-16-2013_15:10: Count: 775 count open
>>> files started: 01-16-2013_15:15: Count: 1092
>> Okay. lsof will help you determine how many of those are "real" files
>> versus sockets. Limiting socket usage might be somewhat easier
>> depending upon what your application actually does.
>>
>>> But maybe the afs3 connection causing the Apache error?
>> afs3 is a red herring: you are using port 7009 for AJP communication
>> between httpd and Tomcat and it's being reported as afs3. This has
>> nothing to do with afs3 unless you know for a fact that your web
>> application uses that protocol for something. I don't see any evidence
>> that afs3 is related to your environment in the slightest. I do see
>> every indication that you are using port 7009 yourself for AJP so
>> let's assume that's the truth.
>>
>> Let's recap what your webapp(s) actually do to see if we can't figure
>> out where all your file handles are being used. I'll assume that each
>> Tomcat is configured (reasonably) identically, other than port numbers
>> and such. I'll also assume that you are running the same webapp using
>> the same (virtually) identical configuration and that nothing
>> pathological is happening (like one process totally going crazy and
>> making thousands of socket connections due to an application bug).
>>
>> First, all processes need access to stdin, stdout, stderr: that's 3
>> file handles. Plus all shared libraries required to get the process
>> and JVM started. Plus everything Java needs. Depending on the OS,
>> that's about 30 or so to begin with. Then, Tomcat uses /dev/random (or
>> /dev/urandom) plus it needs to load all of its own libraries from JAR
>> files. There are about 25 of them, and they generally stay open. So,
>> we're up to about 55 file handles. Don't worry: we won't be counting
>> these things one-at-a-time for long. Next, Tomcat has two <Connector>s
>> defined with default connection sizes. At peak load, they will both be
>> maxed-out at 200 connections each for a total of 402 file handles (1
>> bind file handle + 200 file handles for the connections * 2
>> connectors). So, we're up to 457.
>>
>> Now, onto your web application. You have to count the number of JAR
>> files that your web application provides: each one of those likely
>> consumes another file handle that will stay open. Does your webapp use
>> a database? If so, do you use a connection pool? How big is the
>> connection pool? Do you have any leaks? If you use a connection pool
>> and have no leaks, then you can add 'maxActive' file handles to our
>> running count. If you don't use a connection pool, then you can add
>> 400 file handles to your count, because any incoming request on either
>> of those two connectors could result in a database connection. (I
>> highly recommend using a connection pool if you aren't already).
>>
>> Next, you said this:
>>
>>> Both of the tomcats are "synchronising" them self. The send some
>>> serialized objects via http to each other.
>> So the webapps make requests to each other? How? Is there a limit to
>> the number of connections directly from one Tomcat to another? If not,
>> then you can add another 400 file handles because any incoming
>> connection could trigger an HTTP connection to the other Tomcat. (What
>> happens if an incoming client connection causes a connection to the
>> other Tomcat... will that Tomcat ever call-back to the first one and
>> set-up a communication storm?).
>>
>>> And both of them getting some file from SMB shares.
>> How many files? Every file you open consumes a file handle. If you
>> close the file, you can reduce your fd footprint, but if you keep lots
>> of files open...
>>
>> If you have a dbcp with size=50 and you limit your cross-Tomcat
>> connections to, say another 50 and your webapp uses 50 JAR files then
>> you are looking at 600 or so file handles required to run your webapp
>> under peak load, not including files that must be opened to satisfy a
>> particular request.
>>
>> So the question is: where are all your fds going? Use lsof to
>> determine what they are being used for.
>>
>> Some suggestions:
>>
>> 1. Consider the number of connections you actually need to be able to
>> handle: for both connectors. Maybe you don't need 200 possible
>> connections for your HTTP connector.
>>
>> 2. Make sure your MaxClients in httpd matches make sense with what
>> you've got in Tomcat's AJP connector: you want to make sure that you
>> have enough connections available from httpd->Tomcat that you aren't
>> making users wait. If you're using prefork MPM that means that
>> MaxClients should be the same as your <Connector>'s maxThreads setting
>> (or, better yet, use an <Executor>).
>>
>> 3. Use an <Executor>. Right now, you might allocate up to 400 threads
>> to handle connections from both AJP and HTTP. Maybe you don't need
>> that. You can share request-processing threads by using an <Executor>
>> and have both connectors share the same pool.
>>
>> 4. Use a DBCP. Just in case you aren't.
>>
>> 5. Check to see how you are communicating Tomcat-to-Tomcat: you may
>> have a problem where too many connections are being opened.
>>
>> 6. Check to make sure you don't have any resource leaks: JDBC
>> connections that aren't closed, files not being closed, etc. etc.
>> Check to make sure you are closing files that don't need to be open
>> after they are read.
>>
>>> But I can't imagine that might be the problem? I'm wondering why
>>> the tcp connections with state "CLOSE_WAIT" doesn't get closed.
>> http://en.wikipedia.org/wiki/Transmission_Control_Protocol
>>
>> - -chris
>> -----BEGIN PGP SIGNATURE-----
>> Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
>> Comment: GPGTools - http://gpgtools.org
>> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>>
>> iEYEAREIAAYFAlD4NvsACgkQ9CaO5/Lv0PC4EACfURhDENZPf28HDIazwPqAqri5
>> KqYAni9AOSQZVIdsBtQLoEfDcYkpuf7f
>> =dEDY
>> -----END PGP SIGNATURE-----
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>>
>> ------------------------------------------------------------------------
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections
keept open
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
David,
On 3/12/13 6:58 AM, David Kumar wrote:
> We are using Apache 2.2.16 on Debian. Therefore MaxRequestWorkers
> is called MaxClients, isn't it?
That depends upon the MPM you are using. I have Debian-based systems
with both prefork and worker MPMs configured (separately).
> Currently it is set to 312. So if we have two tomcats with 200
> thread each MaxClients is to low? I either should reduce the Thread
> at the connector or increase MaxClients?
If you have MaxClients=312, then your Tomcat configuration should be
at least that big, even if you have two back-end Tomcats. What happens
if one of your Tomcats dies (or gets a 404 LOL)? All your traffic will
go to the other Tomcat: that means 312 connections to a single
back-end node.
So you either need to adjust your MaxClients/MaxRequestWorkers /down/
so that a single Tomcat can handle those connections, or you need to
increase your maxThreads on Tomcat to handle the potential client load
from httpd.
If you have multiple httpd instances, then you need to multiple
everything by N: MaxClients * N <= maxThreads. I would set maxThreads
to be a bit bigger, anyway, because then you can do nice things like
test to see if a Tomcat is available outside of httpd. You don't want
MaxClients == maxThreads and then not be able to ping your server due
to thread starvation.
Finally, if you have more than 8192 possible incoming connections from
all your httpd instances, you may have to adjust your maxConnections
setting (default is 8192 for APR and 10000 for NIO... default is
maxThreads for BIO which is the only thing that makes any sense at all).
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iEYEAREIAAYFAlE/U3gACgkQ9CaO5/Lv0PD1XACdG0l/hyIBXh1CoE5Yp6XRf81P
D+kAniaCJT0KSKfSl7SSs6WLETCAB1Zx
=hqwF
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections
keept open
Posted by Mark Thomas <ma...@apache.org>.
On 12/03/2013 10:58, David Kumar wrote:
> Hey Mark,
>
> thanks for reply.
>
> I do have some more questions as the main configuration i not done by myself.
> We are using Apache 2.2.16 on Debian. Therefore MaxRequestWorkers is called MaxClients, isn't it?
Correct.
> Currently it is set to 312. So if we have two tomcats with 200 thread each MaxClients is to low? I either should reduce the Thread at the connector or increase MaxClients?
Increase maxThreads to 400.
Mark
> We got the connection_pool_timeout from here:
> https://community.jboss.org/wiki/OptimalModjk12Configuration
>
> I will have a look on the other recommended options
>
> Thanks..
>
>
> Mit freundlichen Grüßen
> David Kumar
> Softwareentwickler, B. Sc.
> Abteilung Infotech - Interaktiv
> TELESTAR-DIGITAL GmbH
> Am Weiher 14
> D-56766 Ulmen
>
>
> http://www.telestar.de/
>
>
>
>
> -----Ursprüngliche Nachricht-----
> Von: Mark Thomas [mailto:markt@apache.org]
> Gesendet: Dienstag, 12. März 2013 10:25
> An: Tomcat Users List
> Betreff: Re: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
>
> On 12/03/2013 06:53, David Kumar wrote:
>>
>> Hey,
>>
>> we are still having that issue. But we could manage to figure out
>> some more stuff. We made a Tomcat and Java update since that time we
>> had our problem a few times again, also we did some reconfiguration
>> with connectors etc. The last 2 times we where able to see, that both
>> tomcat by them self where alive. Just ajp on both where dead. We
>> couldn't make a connection either trough 7009 nor 9009. An with our
>> openFiles trick we found a lot of close_wait again, e.g. 200 for
>> 9009. I left the second tomcat on this state for a few ours just to
>> see, what happens. The count of 200 connection with close_wait was
>> kept until a reboot of the tomcat. I would say with some of our
>> reconfiguration we managed to stop increasing connections. But still
>> we are not sure why our ajp connections dying..
>>
>> Here is our connector out of Server.xml:
>>
>> <Connector port="9009" protocol="AJP/1.3" redirectPort="9443"
>> maxThreads="200" connectionTimeout="600000" />
>
> Only 200 threads on the Tomcat side. If httpd's
> MaxClients/MaxRequestWorkers is greater than 200 you may get thread
> starvation in Tomcat.
>
>> worker.properties:
>>
>> worker.tomcatX.host=localhost
>> worker.tomcatX.type=ajp13
>> worker.tomcatX.fail_on_status=404
>
> That is a really bad idea. A single 404 and the entire Tomcat instance
> gets taken out of the loadbalancer for 60 seconds. Hello DOS attack.
>
>> worker.tomcatX.lbfactor=1
>> worker.tomcatX.ping_timeout=1000
>> worker.tomcatX.ping_mode=A
>> worker.tomcatX.socket_timeout=10
>> worker.tomcatX.connection_pool_timeout=600
>
> 10 minutes is a long time to keep a persistent connection around. With
> even a moderate load you'll easily get to 200 connections in a 10 minute
> period.
>
>> worker.tomcat1.reference=worker.tomcatX worker.tomcat1.port=7009
>> worker.tomcat2.reference=worker.tomcatX worker.tomcat2.port=9009
>>
>> worker.loadbalancer.type=lb
>> worker.loadbalancer.balance_workers=tomcat1,tomcat2
>>
>> worker.status.type=status
>>
>> Hopefully one of you guys can give us a hint to fix that problem.
>
> Do one of the following:
>
> 1. Increase maxThreads in Tomcat's connector to > MaxRequestWorkers
>
> 2. Use JkOptions +DisableReuse (ignore the performance warnings)
>
> 3. Reduce the connection_pool_timeout
>
> Mark
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections
keept open
Posted by Rainer Jung <ra...@kippdata.de>.
On 12.03.2013 11:58, David Kumar wrote:
> We got the connection_pool_timeout from here:
> https://community.jboss.org/wiki/OptimalModjk12Configuration
>
> I will have a look on the other recommended options
Pick a source download of mod_jk. It contains an example configuration
that should work pretty well. Don't choose the "minimal" example config,
just the normal one.
Regards,
Rainer
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections
keept open
Posted by David Kumar <dk...@telestar.de>.
Hey Mark,
thanks for reply.
I do have some more questions as the main configuration i not done by myself.
We are using Apache 2.2.16 on Debian. Therefore MaxRequestWorkers is called MaxClients, isn't it?
Currently it is set to 312. So if we have two tomcats with 200 thread each MaxClients is to low? I either should reduce the Thread at the connector or increase MaxClients?
We got the connection_pool_timeout from here:
https://community.jboss.org/wiki/OptimalModjk12Configuration
I will have a look on the other recommended options
Thanks..
Mit freundlichen Grüßen
David Kumar
Softwareentwickler, B. Sc.
Abteilung Infotech - Interaktiv
TELESTAR-DIGITAL GmbH
Am Weiher 14
D-56766 Ulmen
http://www.telestar.de/
-----Ursprüngliche Nachricht-----
Von: Mark Thomas [mailto:markt@apache.org]
Gesendet: Dienstag, 12. März 2013 10:25
An: Tomcat Users List
Betreff: Re: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
On 12/03/2013 06:53, David Kumar wrote:
>
> Hey,
>
> we are still having that issue. But we could manage to figure out
> some more stuff. We made a Tomcat and Java update since that time we
> had our problem a few times again, also we did some reconfiguration
> with connectors etc. The last 2 times we where able to see, that both
> tomcat by them self where alive. Just ajp on both where dead. We
> couldn't make a connection either trough 7009 nor 9009. An with our
> openFiles trick we found a lot of close_wait again, e.g. 200 for
> 9009. I left the second tomcat on this state for a few ours just to
> see, what happens. The count of 200 connection with close_wait was
> kept until a reboot of the tomcat. I would say with some of our
> reconfiguration we managed to stop increasing connections. But still
> we are not sure why our ajp connections dying..
>
> Here is our connector out of Server.xml:
>
> <Connector port="9009" protocol="AJP/1.3" redirectPort="9443"
> maxThreads="200" connectionTimeout="600000" />
Only 200 threads on the Tomcat side. If httpd's
MaxClients/MaxRequestWorkers is greater than 200 you may get thread
starvation in Tomcat.
> worker.properties:
>
> worker.tomcatX.host=localhost
> worker.tomcatX.type=ajp13
> worker.tomcatX.fail_on_status=404
That is a really bad idea. A single 404 and the entire Tomcat instance
gets taken out of the loadbalancer for 60 seconds. Hello DOS attack.
> worker.tomcatX.lbfactor=1
> worker.tomcatX.ping_timeout=1000
> worker.tomcatX.ping_mode=A
> worker.tomcatX.socket_timeout=10
> worker.tomcatX.connection_pool_timeout=600
10 minutes is a long time to keep a persistent connection around. With
even a moderate load you'll easily get to 200 connections in a 10 minute
period.
> worker.tomcat1.reference=worker.tomcatX worker.tomcat1.port=7009
> worker.tomcat2.reference=worker.tomcatX worker.tomcat2.port=9009
>
> worker.loadbalancer.type=lb
> worker.loadbalancer.balance_workers=tomcat1,tomcat2
>
> worker.status.type=status
>
> Hopefully one of you guys can give us a hint to fix that problem.
Do one of the following:
1. Increase maxThreads in Tomcat's connector to > MaxRequestWorkers
2. Use JkOptions +DisableReuse (ignore the performance warnings)
3. Reduce the connection_pool_timeout
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections
keept open
Posted by Mark Thomas <ma...@apache.org>.
On 12/03/2013 06:53, David Kumar wrote:
>
> Hey,
>
> we are still having that issue. But we could manage to figure out
> some more stuff. We made a Tomcat and Java update since that time we
> had our problem a few times again, also we did some reconfiguration
> with connectors etc. The last 2 times we where able to see, that both
> tomcat by them self where alive. Just ajp on both where dead. We
> couldn't make a connection either trough 7009 nor 9009. An with our
> openFiles trick we found a lot of close_wait again, e.g. 200 for
> 9009. I left the second tomcat on this state for a few ours just to
> see, what happens. The count of 200 connection with close_wait was
> kept until a reboot of the tomcat. I would say with some of our
> reconfiguration we managed to stop increasing connections. But still
> we are not sure why our ajp connections dying..
>
> Here is our connector out of Server.xml:
>
> <Connector port="9009" protocol="AJP/1.3" redirectPort="9443"
> maxThreads="200" connectionTimeout="600000" />
Only 200 threads on the Tomcat side. If httpd's
MaxClients/MaxRequestWorkers is greater than 200 you may get thread
starvation in Tomcat.
> worker.properties:
>
> worker.tomcatX.host=localhost
> worker.tomcatX.type=ajp13
> worker.tomcatX.fail_on_status=404
That is a really bad idea. A single 404 and the entire Tomcat instance
gets taken out of the loadbalancer for 60 seconds. Hello DOS attack.
> worker.tomcatX.lbfactor=1
> worker.tomcatX.ping_timeout=1000
> worker.tomcatX.ping_mode=A
> worker.tomcatX.socket_timeout=10
> worker.tomcatX.connection_pool_timeout=600
10 minutes is a long time to keep a persistent connection around. With
even a moderate load you'll easily get to 200 connections in a 10 minute
period.
> worker.tomcat1.reference=worker.tomcatX worker.tomcat1.port=7009
> worker.tomcat2.reference=worker.tomcatX worker.tomcat2.port=9009
>
> worker.loadbalancer.type=lb
> worker.loadbalancer.balance_workers=tomcat1,tomcat2
>
> worker.status.type=status
>
> Hopefully one of you guys can give us a hint to fix that problem.
Do one of the following:
1. Increase maxThreads in Tomcat's connector to > MaxRequestWorkers
2. Use JkOptions +DisableReuse (ignore the performance warnings)
3. Reduce the connection_pool_timeout
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections
keept open
Posted by David Kumar <dk...@telestar.de>.
Hey,
thanks..
I downloaded jmxsh... I keep you guy up to date..
thanks...
Mit freundlichen Grüßen
David Kumar
Softwareentwickler, B. Sc.
Abteilung Infotech - Interaktiv
TELESTAR-DIGITAL GmbH
Am Weiher 14
D-56766 Ulmen
http://www.telestar.de/
-----Ursprüngliche Nachricht-----
Von: André Warnier [mailto:aw@ice-sa.com]
Gesendet: Dienstag, 12. März 2013 09:54
An: Tomcat Users List
Betreff: Re: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
David Kumar wrote:
> Hey,
>
> we are still having that issue.
> But we could manage to figure out some more stuff.
> We made a Tomcat and Java update since that time we had our problem a few times again, also we did some reconfiguration with connectors etc.
> The last 2 times we where able to see, that both tomcat by them self where alive. Just ajp on both where dead. We couldn't make a connection either trough 7009 nor 9009. An with our openFiles trick we found a lot of close_wait again, e.g. 200 for 9009. I left the second tomcat on this state for a few ours just to see, what happens. The count of 200 connection with close_wait was kept until a reboot of the tomcat.
Instead of rebooting Tomcat, try to force the Tomcat JVM to do a Major Garbage Collection.
There are a number of tools that allow to do that.
One command-line one which I found practical is jmxsh, here :
http://code.google.com/p/jmxsh/
If when you do a Major GC, these CLOSE_WAIT connections disappear, you will have learned
something about their origin.
And if then - without restarting Tomcat - you can connect again via the AJP ports, you'll
have learned something else.
Go do it and report.
> I would say with some of our reconfiguration we managed to stop increasing connections. But still we are not sure why our ajp connections dying..
>
> Here is our connector out of Server.xml:
>
> <Connector port="9009" protocol="AJP/1.3" redirectPort="9443" maxThreads="200" connectionTimeout="600000" />
>
>
> worker.properties:
>
> worker.tomcatX.host=localhost
> worker.tomcatX.type=ajp13
> worker.tomcatX.fail_on_status=404
> worker.tomcatX.lbfactor=1
> worker.tomcatX.ping_timeout=1000
> worker.tomcatX.ping_mode=A
> worker.tomcatX.socket_timeout=10
> worker.tomcatX.connection_pool_timeout=600
>
>
> worker.tomcat1.reference=worker.tomcatX
> worker.tomcat1.port=7009
>
> worker.tomcat2.reference=worker.tomcatX
> worker.tomcat2.port=9009
>
> worker.loadbalancer.type=lb
> worker.loadbalancer.balance_workers=tomcat1,tomcat2
>
> worker.status.type=status
>
> Hopefully one of you guys can give us a hint to fix that problem.
>
> Mit freundlichen Grüßen
> David Kumar
> Softwareentwickler, B. Sc.
> Abteilung Infotech - Interaktiv
> TELESTAR-DIGITAL GmbH
> Am Weiher 14
> D-56766 Ulmen
>
> http://www.telestar.de/
>
>
>
>
> -----Ursprüngliche Nachricht-----
> Von: David Kumar [mailto:dkumar@telestar.de]
> Gesendet: Dienstag, 22. Januar 2013 07:36
> An: Tomcat Users List
> Betreff: AW: AW: AW: ajp on 7009 and 9009 : connections keept open
>
> Hey,
>
> last friday I changed our configuration to use a executor.
> Here is what I did:
>
> <Connector port="7009" protocol="AJP/1.3" redirectPort="8443" maxThreads="200" executor="active-executor" />
> <Executor name="active-executor" namePrefix="activeThread-" maxThreads="200" minSpareThreads="30" maxIdleTime="60000" />
> <Connector port="7080" protocol="HTTP/1.1"
> connectionTimeout="20000"
> redirectPort="8443" executor="active-executor" />
>
> The second tomcat has same configuration besides ports..
>
> Until yesterday it worked like a charm. But at late afternoon one of the tomcats failed again..
>
> I couldn't start the garbagecollection so far..
>
> Any other ideas?
>
> Thanks
>
> Mit freundlichen Grüßen
> David Kumar
> Softwareentwickler, B. Sc.
> Abteilung Infotech - Interaktiv
> TELESTAR-DIGITAL GmbH
> Am Weiher 14
> D-56766 Ulmen
> http://www.telestar.de/
>
>
>
>
> -----Ursprüngliche Nachricht-----
> Von: David Kumar [mailto:dkumar@telestar.de]
> Gesendet: Freitag, 18. Januar 2013 11:19
> An: Tomcat Users List; Tomcat Users List
> Betreff: AW: AW: AW: ( ajp on 7009 and 9009 not afs3-rmtsys): connections keept open
>
> Hey,
>
> I do that at next deployment. --> I Thursday...
>
> So far I'm trying executor for tomcat. As far I read, when I'm using connectors idle process are forced to be close..
>
>
> Mit freundlichen Grüßen
> David Kumar
> Softwareentwickler, B. Sc.
> Abteilung Infotech - Interaktiv
> TELESTAR-DIGITAL GmbH
> Am Weiher 14
> D-56766 Ulmen
>
> http://www.telestar.de/
>
>
>
>
> -----Ursprüngliche Nachricht-----
> Von: André Warnier [mailto:aw@ice-sa.com]
> Gesendet: Freitag, 18. Januar 2013 11:10
> An: Tomcat Users List
> Betreff: Re: AW: AW: ( ajp on 7009 and 9009 not afs3-rmtsys): connections keept open
>
> David Kumar wrote:
>> Hey André,
>>
>> are you talking about running System.gc()?
> Yes.
>
>> That should be possible..
>>
>> Mit freundlichen Grüßen
>> David Kumar
>> Softwareentwickler, B. Sc.
>> Abteilung Infotech - Interaktiv
>> TELESTAR-DIGITAL GmbH
>> Am Weiher 14
>> D-56766 Ulmen
>>
>> http://www.telestar.de/
>>
>>
>>
>>
>> -----Ursprüngliche Nachricht-----
>> Von: André Warnier [mailto:aw@ice-sa.com]
>> Gesendet: Freitag, 18. Januar 2013 10:07
>> An: Tomcat Users List
>> Betreff: Re: AW: ( ajp on 7009 and 9009 not afs3-rmtsys): connections keept open
>>
>> David,
>> (and sorry for top-posting here)
>>
>> just to verify something.
>> Can you trigger a Major Garbage Collection at the Tomcat JVM level, at a moment when you
>> have all these connections in CLOSE_WAIT, and see if they disappear after the GC ?
>>
>> If yes, it may give a good clue about where all these CLOSE_WAITs are coming from.
>>
>>
>> David Kumar wrote:
>>> Just read this email.. :-)
>>>
>>> I figured out we are not using executor connector...
>>>
>>>
>>> Mit freundlichen Grüßen
>>> David Kumar
>>> Softwareentwickler, B. Sc.
>>> Abteilung Infotech - Interaktiv
>>> TELESTAR-DIGITAL GmbH
>>> Am Weiher 14
>>> D-56766 Ulmen
>>> http://www.telestar.de/
>>>
>>>
>>>
>>>
>>> -----Ursprüngliche Nachricht-----
>>> Von: David Kumar [mailto:dkumar@telestar.de]
>>> Gesendet: Freitag, 18. Januar 2013 09:11
>>> An: Tomcat Users List
>>> Betreff: AW: ( ajp on 7009 and 9009 not afs3-rmtsys): connections keept open
>>>
>>> here you are with attachment :-)
>>>
>>>
>>> btw: in mod_jk.log I found some
>>> [Thu Jan 17 23:00:08 2013] [11196:140336689317632] [error] ajp_get_reply::jk_ajp_common.c (2055): (tomcat2) Tomcat is down or refused connection. No response has been sent to the client (yet)
>>> [Thu Jan 17 23:00:08 2013] [11196:140336689317632] [error] ajp_service::jk_ajp_common.c (2559): (tomcat2) connecting to tomcat failed.
>>>
>>>
>>> but realy just a few one...
>>>
>>>
>>> Mit freundlichen Grüßen
>>> David Kumar
>>> Softwareentwickler, B. Sc.
>>> Abteilung Infotech - Interaktiv
>>> TELESTAR-DIGITAL GmbH
>>> Am Weiher 14
>>> D-56766 Ulmen
>>>
>>> http://www.telestar.de/
>>>
>>>
>>>
>>>
>>> -----Ursprüngliche Nachricht-----
>>> Von: David Kumar
>>> Gesendet: Freitag, 18. Januar 2013 09:08
>>> An: 'Tomcat Users List'
>>> Betreff: ( ajp on 7009 and 9009 not afs3-rmtsys): connections keept open
>>>
>>> Hey,
>>>
>>> thanks for reply. I got that about the Apache configuration. Since we had our problem yesterday, again and there was no error at the apache logs I'm willing to say that is not the main problem and I have to check that, when my main problem is solved.. :-)
>>>
>>>
>>>
>>> I agree with you about wrong reporting of service. Its just shown up as afs3 because these service uses 7009 per default. But I'm using 7009 and 9009 for ajp.
>>>
>>>
>>> So doesn't this mean there is a connection problems between my Apache and the tomcats?
>>>
>>> You're right, both Webapps doing the same and are configured identically besides the ports.
>>>
>>> I'm using more than one database, but all of them are used through a database pool. If there is a bug, I think I should have found some error at my logs like no free connection or something like that. As there is no such log entry I'm willing to say that my database connections processing like they should.
>>>
>>> Basically on each tomcat there are running two services. One is a axis2 project. Our CRM is posting customer data to this webapp. This data will be persisted into a database. Depending on the information given by our CRM axis sends a email.
>>>
>>> The second one is basically a cache for our websites. We have a PIM with all our product data. These app is gathering all the data from PIM and a CMS and is merging these information together so that the data can be displayed.
>>> All the mentioned data is hold in different "cache objects". Also some communication with our ERP and some databases are made trough this app.
>>>
>>> The second app is a REST service. Information will be posted as POST or GET request to it. Most likely the responses are JSON Object.
>>>
>>> When ever one webApp is reloading (automatically or manually) itself, the result will be posted to the other tomcat/webapp as a serialized object, so the other on do not need to reload it self.
>>>
>>> I can't say how many SMB files there are, it is depending on some other stuff so it is dynamic.
>>>
>>> Attached you can find a printed list by lsof.
>>>
>>> There you can see a really strange thing. Yesterday just tomcat2 had the problem with to many open files. A few days before it was just tomcat1 having this problem.
>>>
>>> Now let my answer your question:
>>>
>>> 1. That is hard to say, I guess I have to do some more investigation on our logfiles.
>>>
>>> 2. / 3. Here is my httpd.conf:
>>> <IfModule mpm_worker_module>
>>> ThreadLimit 25
>>> StartServers 2
>>> MaxClients 150
>>> MinSpareThreads 25
>>> MaxSpareThreads 75
>>> ThreadsPerChild 25
>>> MaxRequestsPerChild 4000
>>> </IfModule>
>>>
>>> we are using worker ....
>>>
>>> And here are our tomcat connectors again:
>>> tomcat1:
>>>
>>> <Connector port="7080" protocol="HTTP/1.1" connectionTimeout="20000" redirectPort="8443"/>
>>>
>>>
>>> <Connector port="7009" protocol="AJP/1.3" redirectPort="8443"/>
>>>
>>>
>>> tomcat2:
>>> <Connector port="9080" protocol="HTTP/1.1" connectionTimeout="20000" redirectPort="9443"/>
>>>
>>> <Connector port="9009" protocol="AJP/1.3" redirectPort="9443"/>
>>>
>>>
>>> Okay we are not using executor.. I will check that..
>>>
>>> You probably read my copy-paste error. I did copy some comments out of out server config --> Sry again.
>>>
>>> 4. we are using..
>>> 5. via a multipart message sending to the other tomcat.
>>> 6. I don't think so also because of that the connections are kept open on our ajp ports.
>>>
>>> I know that "CLOSE_WAIT" means, waiting for connections to be closed, but wondering that it is not closing..
>>>
>>>
>>> Thanks again
>>>
>>> Mit freundlichen Grüßen
>>> David Kumar
>>> Softwareentwickler, B. Sc.
>>> Abteilung Infotech - Interaktiv
>>> TELESTAR-DIGITAL GmbH
>>> Am Weiher 14
>>> D-56766 Ulmen
>>> http://www.telestar.de/
>>>
>>>
>>>
>>>
>>> -----Ursprüngliche Nachricht-----
>>> Von: Christopher Schultz [mailto:chris@christopherschultz.net]
>>> Gesendet: Donnerstag, 17. Januar 2013 18:38
>>> An: Tomcat Users List
>>> Betreff: Re: AW: AW: afs3-rmtsys: connections keept open
>>>
>>> -----BEGIN PGP SIGNED MESSAGE-----
>>> Hash: SHA256
>>>
>>> David,
>>>
>>> On 1/17/13 1:49 AM, David Kumar wrote:
>>>> I just checked /var/logs/apache2/error.logs. And found following
>>>> errors:
>>>>
>>>> [Wed Jan 16 15:14:46 2013] [error] server is within MinSpareThreads
>>>> of MaxClients, consider raising the MaxClients setting [Wed Jan 16
>>>> 15:14:56 2013] [error] server reached MaxClients setting, consider
>>>> raising the MaxClients setting
>>> So you are maxing-out your connections: you are experiencing enough
>>> load that your configuration cannot handle any more connections:
>>> requests are being queued by the TCP/IP stack and some requests may be
>>> rejected entirely depending upon the queue length of the socket.
>>>
>>> The first question to ask yourself is whether or not your hardware can
>>> take more than you have it configured to accept. For instance, if your
>>> load average, memory usage, and response time are all reasonable, then
>>> you could probably afford to raise your MaxClients setting in httpd.
>>>
>>> Note that the above has almost nothing to do with Tomcat: it only has
>>> to do with Apache httpd.
>>>
>>>> Yesterday my problem occurred about the same time.
>>> So, the problem is that Tomcat cannot handle your peak load due to a
>>> file handle limitation. IIRC, your current file handle limit for the
>>> Tomcat process is 4096.
>>>
>>>> I'm checking every five minutes how many open files there are:
>>>>
>>>> count open files started: 01-16-2013_15:10: Count: 775 count open
>>>> files started: 01-16-2013_15:15: Count: 1092
>>> Okay. lsof will help you determine how many of those are "real" files
>>> versus sockets. Limiting socket usage might be somewhat easier
>>> depending upon what your application actually does.
>>>
>>>> But maybe the afs3 connection causing the Apache error?
>>> afs3 is a red herring: you are using port 7009 for AJP communication
>>> between httpd and Tomcat and it's being reported as afs3. This has
>>> nothing to do with afs3 unless you know for a fact that your web
>>> application uses that protocol for something. I don't see any evidence
>>> that afs3 is related to your environment in the slightest. I do see
>>> every indication that you are using port 7009 yourself for AJP so
>>> let's assume that's the truth.
>>>
>>> Let's recap what your webapp(s) actually do to see if we can't figure
>>> out where all your file handles are being used. I'll assume that each
>>> Tomcat is configured (reasonably) identically, other than port numbers
>>> and such. I'll also assume that you are running the same webapp using
>>> the same (virtually) identical configuration and that nothing
>>> pathological is happening (like one process totally going crazy and
>>> making thousands of socket connections due to an application bug).
>>>
>>> First, all processes need access to stdin, stdout, stderr: that's 3
>>> file handles. Plus all shared libraries required to get the process
>>> and JVM started. Plus everything Java needs. Depending on the OS,
>>> that's about 30 or so to begin with. Then, Tomcat uses /dev/random (or
>>> /dev/urandom) plus it needs to load all of its own libraries from JAR
>>> files. There are about 25 of them, and they generally stay open. So,
>>> we're up to about 55 file handles. Don't worry: we won't be counting
>>> these things one-at-a-time for long. Next, Tomcat has two <Connector>s
>>> defined with default connection sizes. At peak load, they will both be
>>> maxed-out at 200 connections each for a total of 402 file handles (1
>>> bind file handle + 200 file handles for the connections * 2
>>> connectors). So, we're up to 457.
>>>
>>> Now, onto your web application. You have to count the number of JAR
>>> files that your web application provides: each one of those likely
>>> consumes another file handle that will stay open. Does your webapp use
>>> a database? If so, do you use a connection pool? How big is the
>>> connection pool? Do you have any leaks? If you use a connection pool
>>> and have no leaks, then you can add 'maxActive' file handles to our
>>> running count. If you don't use a connection pool, then you can add
>>> 400 file handles to your count, because any incoming request on either
>>> of those two connectors could result in a database connection. (I
>>> highly recommend using a connection pool if you aren't already).
>>>
>>> Next, you said this:
>>>
>>>> Both of the tomcats are "synchronising" them self. The send some
>>>> serialized objects via http to each other.
>>> So the webapps make requests to each other? How? Is there a limit to
>>> the number of connections directly from one Tomcat to another? If not,
>>> then you can add another 400 file handles because any incoming
>>> connection could trigger an HTTP connection to the other Tomcat. (What
>>> happens if an incoming client connection causes a connection to the
>>> other Tomcat... will that Tomcat ever call-back to the first one and
>>> set-up a communication storm?).
>>>
>>>> And both of them getting some file from SMB shares.
>>> How many files? Every file you open consumes a file handle. If you
>>> close the file, you can reduce your fd footprint, but if you keep lots
>>> of files open...
>>>
>>> If you have a dbcp with size=50 and you limit your cross-Tomcat
>>> connections to, say another 50 and your webapp uses 50 JAR files then
>>> you are looking at 600 or so file handles required to run your webapp
>>> under peak load, not including files that must be opened to satisfy a
>>> particular request.
>>>
>>> So the question is: where are all your fds going? Use lsof to
>>> determine what they are being used for.
>>>
>>> Some suggestions:
>>>
>>> 1. Consider the number of connections you actually need to be able to
>>> handle: for both connectors. Maybe you don't need 200 possible
>>> connections for your HTTP connector.
>>>
>>> 2. Make sure your MaxClients in httpd matches make sense with what
>>> you've got in Tomcat's AJP connector: you want to make sure that you
>>> have enough connections available from httpd->Tomcat that you aren't
>>> making users wait. If you're using prefork MPM that means that
>>> MaxClients should be the same as your <Connector>'s maxThreads setting
>>> (or, better yet, use an <Executor>).
>>>
>>> 3. Use an <Executor>. Right now, you might allocate up to 400 threads
>>> to handle connections from both AJP and HTTP. Maybe you don't need
>>> that. You can share request-processing threads by using an <Executor>
>>> and have both connectors share the same pool.
>>>
>>> 4. Use a DBCP. Just in case you aren't.
>>>
>>> 5. Check to see how you are communicating Tomcat-to-Tomcat: you may
>>> have a problem where too many connections are being opened.
>>>
>>> 6. Check to make sure you don't have any resource leaks: JDBC
>>> connections that aren't closed, files not being closed, etc. etc.
>>> Check to make sure you are closing files that don't need to be open
>>> after they are read.
>>>
>>>> But I can't imagine that might be the problem? I'm wondering why
>>>> the tcp connections with state "CLOSE_WAIT" doesn't get closed.
>>> http://en.wikipedia.org/wiki/Transmission_Control_Protocol
>>>
>>> - -chris
>>> -----BEGIN PGP SIGNATURE-----
>>> Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
>>> Comment: GPGTools - http://gpgtools.org
>>> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>>>
>>> iEYEAREIAAYFAlD4NvsACgkQ9CaO5/Lv0PC4EACfURhDENZPf28HDIazwPqAqri5
>>> KqYAni9AOSQZVIdsBtQLoEfDcYkpuf7f
>>> =dEDY
>>> -----END PGP SIGNATURE-----
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>>
>>>
>>> ------------------------------------------------------------------------
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: AW: AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections
keept open
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
DAvid,
On 3/13/13 3:01 PM, David Kumar wrote:
> Hey,
>
> right no we're having our Problem. I tried gc through jconsole -->
> no changes and still a lot CLOSE_WAIT. So it is not a GC Problem
> and disablereuse doesn't work either.. Any other ideas?
>
> What do you guy think about proxy_ajp instead of jk? What are the
> advantages of proxy_ajp?
mod_proxy_ajp uses the identical protocol to mod_jk: they both use
AJP, and the code on the Tomcat side is identical. So, if you think
this is a Java problem, switching from mod_jk -> mod_proxy_ajp is
unlikely to change anything at all.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iEYEAREIAAYFAlFDemcACgkQ9CaO5/Lv0PAt0ACZAZg5UqqAeyEhpuOcssa02p4d
DjYAn0IwtT+RJ6pdw+jMX0cy6A8gDhai
=A38V
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
AW: AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 :
connections keept open
Posted by David Kumar <dk...@telestar.de>.
Hey,
right no we're having our Problem.
I tried gc through jconsole --> no changes and still a lot CLOSE_WAIT.
So it is not a GC Problem and disablereuse doesn't work either..
Any other ideas?
What do you guy think about proxy_ajp instead of jk? What are the advantages of proxy_ajp?
Thanks a lot..
Mit freundlichen Grüßen
David Kumar
Softwareentwickler, B. Sc.
Abteilung Infotech - Interaktiv
TELESTAR-DIGITAL GmbH
Am Weiher 14
D-56766 Ulmen
http://www.telestar.de/
-----Ursprüngliche Nachricht-----
Von: David Kumar [mailto:dkumar@telestar.de]
Gesendet: Mittwoch, 13. März 2013 14:32
An: Tomcat Users List
Betreff: AW: AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
Hey,
I'm really not sure if that tool does anything, because there is no feedback..
Funny: Just a few minutes ago a colleague showed ma a tool called something like VM monitor on his mac. It look a kind of similar to jconsole.exe. Next time I will try to make GC with this tool.
Meanwhile I reduced the connectionTimeout, reconfigured maxClients and added -DisableReuse to jkpropertys.
Lets see what happens...
Thanks a lot..
Mit freundlichen Grüßen
David Kumar
Softwareentwickler, B. Sc.
Abteilung Infotech - Interaktiv
TELESTAR-DIGITAL GmbH
Am Weiher 14
D-56766 Ulmen
http://www.telestar.de/
-----Ursprüngliche Nachricht-----
Von: André Warnier [mailto:aw@ice-sa.com]
Gesendet: Mittwoch, 13. März 2013 12:20
An: Tomcat Users List
Betreff: Re: AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
David Kumar wrote:
> Hey,
>
> we just had that problem again. I did that GC trick, it didn't work out well.
> I used:
>
> set hosts [list]
> lappend hosts {localhost:7008}
> lappend hosts {localhost:9008}
> # ...add as many as you want...
>
> foreach {host} $hosts {
> set parts [split $host ":"]
> set hostname [lindex $parts 0]
> set port [lindex $parts 1]
>
> # for each host...
>
> # Connect to it.
> jmx_connect -h $hostname -p $port
>
> # Invoke the garbage collector.
> jmx_invoke -n -m java.lang:type=Memory gc
>
> # Close this connection
> jmx_close
> }
>
> I found that while googleing.
> But it doesn't have any effect. Any other ideas?
>
Did it really connect with the Tomcat JVM and really do anything ?
Is your JVM started with options that allow JMX interaction ?
Also, there are ways to start the JVM with parameters telling it to log the Garbage
Collections to a logfile. Make sure that you do that, and check the logfile to see if it
really does a GC when you ask it to.
Note that there are also other ways to trogger a GC than using jmxsh. I was only pointing
you to that tool as an easy-to-script, non-GUI tool.
But in your java directory (at least the JDK/bin), there should also be a "jconsole"
program which allows you to do pretty much the same interactively with a GUI.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections
keept open
Posted by David Kumar <dk...@telestar.de>.
sry I was wrong default is -1 what menas it is disabled, so I'm fine..?!?
>Interesting.
>
>If the problem was too many sockets in CLOSE_WAIT, consider looking at
>the connectionLinger setting on your AJP connector's in Tomcat.
>Mark
Mhh.. I haven't heard about the options before.. But it sound like a great idea.. Default is -1. So what would be a good time, one minute?
Thanks
Mit freundlichen Grüßen
David Kumar
Softwareentwickler, B. Sc.
Abteilung Infotech - Interaktiv
TELESTAR-DIGITAL GmbH
Am Weiher 14
D-56766 Ulmen
http://www.telestar.de/
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections
keept open
Posted by David Kumar <dk...@telestar.de>.
>Interesting.
>
>If the problem was too many sockets in CLOSE_WAIT, consider looking at
>the connectionLinger setting on your AJP connector's in Tomcat.
>Mark
Mhh.. I haven't heard about the options before.. But it sound like a great idea.. Default is -1. So what would be a good time, one minute?
Thanks
Mit freundlichen Grüßen
David Kumar
Softwareentwickler, B. Sc.
Abteilung Infotech - Interaktiv
TELESTAR-DIGITAL GmbH
Am Weiher 14
D-56766 Ulmen
http://www.telestar.de/
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: AW: AW: AW: AW: AJP suddenly Stopps acting: ajp on 7009 and 9009
: connections keept open
Posted by Mark Thomas <ma...@apache.org>.
On 14/03/2013 13:10, David Kumar wrote:
> I tried to just restart the Apache, didn't work. As suggested I
> tried to use the option -DisableReuse. That did make our situation a
> lot more horrible instead of days / week we it took just minutes to
> shut all AJP ports down.
Interesting.
If the problem was too many sockets in CLOSE_WAIT, consider looking at
the connectionLinger setting on your AJP connector's in Tomcat.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
AW: AW: AW: AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 :
connections keept open
Posted by David Kumar <dk...@telestar.de>.
Hey André,
André Warnier wrote:
>The "jsvc" process here is the wrapper which wraps the JVM and Tomcat to >allow them to use
>orts below 1024 and still run as non-root. For practical purposes,
>consider it as "tomcat".
>
>As you can see, there are some AJP connections (local port 8009) in the >CLOSE_WAIT state
>also. It is a normal state of a TCP connection.
>What is less normal, is that they would remain in that state for a long >time. That means
>that one of the sides of the TCP connection is not doing what it should do.
>
>In my experience, under Linux, this can become a problem if you have >hundreds of these.
>At some point, the TCP stack becomes unresponsive and does not allow any >new connection.
>
>Sometimes, it can happen because the "client" side in the connection (the >one who
>initially creates the connection to a server's "listening" socket), >discards a Java object
>which contains an open low-level OS socket object with a connection still >open, without
>explicitly "closing" this connection first.
>The discarded (and unreachable) Java object is then still sitting in the >heap for a while,
>and only a GC will actually destroy it and as a side-effect close the >underlying connection.
>
>That is why I was asking you to force a GC, to see if this made your >CLOSE_WAIT
>connections disappear. Apparently in your case it doesn't, which mean that >yours is
>another case.
>
>Now, instead of restarting Tomcat or doing a GC, have you tried to restart >Apache ?
>or maybe just do a /etc/init.d/apache2 reload. That may kill off and >restart the Apache
>children processes, and clean up their mod_jk connections to Tomcat.
I tried to just restart the Apache, didn't work. As suggested I tried to use the option -DisableReuse. That did make our situation a lot more horrible instead of days / week we it took just minutes to shut all AJP ports down.
Mit freundlichen Grüßen
David Kumar
Softwareentwickler, B. Sc.
Abteilung Infotech - Interaktiv
TELESTAR-DIGITAL GmbH
Am Weiher 14
D-56766 Ulmen
http://www.telestar.de/
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: AW: AW: AW: AJP suddenly Stopps acting: ajp on 7009 and 9009
: connections keept open
Posted by André Warnier <aw...@ice-sa.com>.
David Kumar wrote:
> Hey,
>
> I'm using:
>
> lsof -u tomcat
>
example of "netstat" output on one of our own systems right now :
1) first one
vovm1:~# netstat -t -pan | grep -v ESTAB
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
PID/Program name
tcp 0 0 0.0.0.0:32032 0.0.0.0:* LISTEN 2999/usbsrvd
tcp 0 0 0.0.0.0:20224 0.0.0.0:* LISTEN 2965/inetd
tcp 0 0 0.0.0.0:11022 0.0.0.0:* LISTEN 9179/perl
tcp 0 0 127.0.0.1:111 0.0.0.0:* LISTEN 2172/portmap
tcp 0 0 0.0.0.0:54321 0.0.0.0:* LISTEN 18279/python
tcp 0 0 0.0.0.0:4949 0.0.0.0:* LISTEN
23225/munin-node
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 2431/sshd
tcp 0 0 0.0.0.0:23 0.0.0.0:* LISTEN 2965/inetd
tcp 0 0 0.0.0.0:43961 0.0.0.0:* LISTEN -
tcp 0 0 192.168.20.43:25 0.0.0.0:* LISTEN 2946/exim4
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 2946/exim4
tcp 0 0 0.0.0.0:11002 0.0.0.0:* LISTEN 2965/inetd
tcp 0 0 0.0.0.0:46911 0.0.0.0:* LISTEN 2183/rpc.statd
tcp 0 0 127.0.0.1:11002 127.0.0.1:40739 TIME_WAIT -
tcp 0 0 127.0.0.1:11002 127.0.0.1:40735 TIME_WAIT -
tcp 0 0 127.0.0.1:11002 127.0.0.1:40728 TIME_WAIT -
tcp 0 0 127.0.0.1:11002 127.0.0.1:40724 TIME_WAIT -
tcp 1 0 127.0.0.1:54321 127.0.0.1:37916 CLOSE_WAIT 18279/python
tcp 0 0 127.0.0.1:43797 127.0.0.1:8009 TIME_WAIT -
tcp 0 0 192.168.20.43:56603 192.168.20.80:111 TIME_WAIT -
tcp6 0 0 :::8009 :::* LISTEN 9206/jsvc
tcp6 0 0 :::80 :::* LISTEN 10730/apache2
tcp6 0 0 :::8180 :::* LISTEN 9206/jsvc
tcp6 0 0 :::22 :::* LISTEN 2431/sshd
tcp6 0 0 :::11100 :::* LISTEN 9133/java
tcp6 0 0 :::11101 :::* LISTEN 9160/java
tcp6 0 0 127.0.0.1:11100 127.0.0.1:38143 TIME_WAIT -
tcp6 0 0 127.0.0.1:40725 127.0.0.1:11002 TIME_WAIT -
tcp6 0 0 127.0.0.1:11101 127.0.0.1:54664 TIME_WAIT -
tcp6 1 0 127.0.0.1:8009 127.0.0.1:35156 CLOSE_WAIT 9206/jsvc
tcp6 0 0 212.85.38.148:80 62.99.208.50:49516 TIME_WAIT -
tcp6 0 0 212.85.38.148:80 62.155.235.89:50654 TIME_WAIT -
tcp6 0 0 127.0.0.1:11101 127.0.0.1:54649 TIME_WAIT -
tcp6 1 0 127.0.0.1:8009 127.0.0.1:33159 CLOSE_WAIT 9206/jsvc
tcp6 1 0 127.0.0.1:8009 127.0.0.1:60682 CLOSE_WAIT 9206/jsvc
tcp6 1 0 127.0.0.1:8009 127.0.0.1:33138 CLOSE_WAIT 9206/jsvc
tcp6 0 0 127.0.0.1:40743 127.0.0.1:11002 TIME_WAIT -
tcp6 0 0 127.0.0.1:40733 127.0.0.1:11002 TIME_WAIT -
tcp6 0 0 127.0.0.1:40740 127.0.0.1:11002 TIME_WAIT -
tcp6 0 0 127.0.0.1:11100 127.0.0.1:38136 TIME_WAIT -
tcp6 0 0 127.0.0.1:11101 127.0.0.1:54656 TIME_WAIT -
tcp6 0 0 127.0.0.1:11101 127.0.0.1:54652 TIME_WAIT -
tcp6 0 0 127.0.0.1:11100 127.0.0.1:50778 TIME_WAIT -
tcp6 0 0 127.0.0.1:40732 127.0.0.1:11002 TIME_WAIT -
tcp6 0 0 127.0.0.1:40742 127.0.0.1:11002 TIME_WAIT -
tcp6 0 0 127.0.0.1:11101 127.0.0.1:54667 TIME_WAIT -
tcp6 0 0 127.0.0.1:40729 127.0.0.1:11002 TIME_WAIT -
tcp6 0 0 127.0.0.1:40736 127.0.0.1:11002 TIME_WAIT -
tcp6 0 0 127.0.0.1:11100 127.0.0.1:38140 TIME_WAIT -
vovm1:~#
(The "grep -v ESTAB" is to eliminate the "ESTABLISHED" connections, which are maybe
irrelevant here).
The "jsvc" process here is the wrapper which wraps the JVM and Tomcat to allow them to use
ports below 1024 and still run as non-root. For practical purposes, consider it as "tomcat".
As you can see, there are some AJP connections (local port 8009) in the CLOSE_WAIT state
also. It is a normal state of a TCP connection.
What is less normal, is that they would remain in that state for a long time. That means
that one of the sides of the TCP connection is not doing what it should do.
In my experience, under Linux, this can become a problem if you have hundreds of these.
At some point, the TCP stack becomes unresponsive and does not allow any new connection.
Sometimes, it can happen because the "client" side in the connection (the one who
initially creates the connection to a server's "listening" socket), discards a Java object
which contains an open low-level OS socket object with a connection still open, without
explicitly "closing" this connection first.
The discarded (and unreachable) Java object is then still sitting in the heap for a while,
and only a GC will actually destroy it and as a side-effect close the underlying connection.
That is why I was asking you to force a GC, to see if this made your CLOSE_WAIT
connections disappear. Apparently in your case it doesn't, which mean that yours is
another case.
Now, instead of restarting Tomcat or doing a GC, have you tried to restart Apache ?
or maybe just do a /etc/init.d/apache2 reload. That may kill off and restart the Apache
children processes, and clean up their mod_jk connections to Tomcat.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections
keept open
Posted by David Kumar <dk...@telestar.de>.
Hey,
>Please please before restarting take two thread dumps. On Linux/Unix a
>"kill -QUIT" will do that (and write the result to STDOUT, which
>depending on your setup likely goes to catalina.out). The QUIT signal
>does *not* terminate the JVM, it just quickly writes out the stacks of
>all threads. The interruption for the running JVM is minimal, so you can
>already try how this works before the process gets into the critical state.
great I just tried a kill -QUIT in our watchdog --> pretty fast. Next time we got our problem I will send the threaddump to you guys.
thanks a lot
David
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections
keept open
Posted by David Kumar <dk...@telestar.de>.
Hi all,
I just want to give you an update about my problem....
THANKS TO ALL!!!!
Felix gave me the final hint... I really had some connection not given back to the pool, so the connections kept open.. -> :-( I thought I checked that.
Since I fixed that and use removeAbandoned there was no more crash!!! Finally.. :-)
I REALY appreciate your help. You guys are doing a really GREAT job!
Thanks again..
David Kumar
>Hi Felix,
>1. You are abandoning db connections and thus are draining the pool
> (most likely in my eyes)
>2. The pool is too small (unlikely, since no work seems to be done other
>than waiting for new connections)
>3. Lock contention while going through borrowObject (unlikely, since no
>thread is doing a validation)
>
>So I would configure 'logAbandoned="true"' and even
>'removeAbandoned="true"' and 'removeAbandonedTimeout="60"' as stated on
>http://tomcat.apache.org/tomcat-7.0-doc/jndi-datasource-examples->howto.html.
>When your pool is getting drained, you will see a message in your log
>files, where you have taken connections out of the pool, but haven used
>them for a while. Than you can start to look for places, where those
>connections might have leaked.
>>thanks for that hint, I will have a look at all of my databasepools..
>>Hopefully you're right and that is our problem..
>>thanks
>>avid
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections
keept open
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
David,
On 3/18/13 2:07 AM, David Kumar wrote:
>> 1. You are abandoning db connections and thus are draining the
>> pool (most likely in my eyes) 2. The pool is too small (unlikely,
>> since no work seems to be done other than waiting for new
>> connections) 3. Lock contention while going through borrowObject
>> (unlikely, since no thread is doing a validation)
>>
>> So I would configure 'logAbandoned="true"' and even
>> 'removeAbandoned="true"' and 'removeAbandonedTimeout="60"' as
>> stated on
>> http://tomcat.apache.org/tomcat-7.0-doc/jndi-datasource-examples->howto.html.
>
>> When your pool is getting drained, you will see a message in
>> your log files, where you have taken connections out of the pool,
>> but haven used them for a while. Than you can start to look for
>> places, where those connections might have leaked.
>
> thanks for that hint, I will have a look at all of my
> databasepools.. Hopefully you're right and that is our problem..
Recommended reading:
http://blog.christopherschultz.net/index.php/2009/03/16/properly-handling-pooled-jdbc-connections/
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iEYEAREIAAYFAlFHMhEACgkQ9CaO5/Lv0PAgVwCfUoMRvSSqqrti566y/YipPTm8
b2QAoLIa3huGEVhwXEj+BnVdWAcrxUL7
=NhLI
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections
keept open
Posted by David Kumar <dk...@telestar.de>.
Hi Felix,
>1. You are abandoning db connections and thus are draining the pool
> (most likely in my eyes)
>2. The pool is too small (unlikely, since no work seems to be done other
>than waiting for new connections)
>3. Lock contention while going through borrowObject (unlikely, since no
>thread is doing a validation)
>
>So I would configure 'logAbandoned="true"' and even
>'removeAbandoned="true"' and 'removeAbandonedTimeout="60"' as stated on
>http://tomcat.apache.org/tomcat-7.0-doc/jndi-datasource-examples->howto.html.
>When your pool is getting drained, you will see a message in your log
>files, where you have taken connections out of the pool, but haven used
>them for a while. Than you can start to look for places, where those
>connections might have leaked.
thanks for that hint, I will have a look at all of my databasepools..
Hopefully you're right and that is our problem..
thanks
avid
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections
keept open
Posted by Felix Schumacher <fe...@internetallee.de>.
Hi David,
Am 17.03.2013 16:17, schrieb David Kumar:
> Hey Rainer,
>
>> It got stripped by the list. Can you post it somewhere and make the URL
>> available?
>
> here you go:
>
> http://www.telestar.de/tmp10356/threaddump.txt
almost all of your threads are waiting to get a database connection out
of the pool. There are three possible scenarios, where this might happen.
1. You are abandoning db connections and thus are draining the pool
(most likely in my eyes)
2. The pool is too small (unlikely, since no work seems to be done other
than waiting for new connections)
3. Lock contention while going through borrowObject (unlikely, since no
thread is doing a validation)
So I would configure 'logAbandoned="true"' and even
'removeAbandoned="true"' and 'removeAbandonedTimeout="60"' as stated on
http://tomcat.apache.org/tomcat-7.0-doc/jndi-datasource-examples-howto.html.
When your pool is getting drained, you will see a message in your log
files, where you have taken connections out of the pool, but haven used
them for a while. Than you can start to look for places, where those
connections might have leaked.
Regards
Felix
>
> thanks
>
>
> David
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections
keept open
Posted by David Kumar <dk...@telestar.de>.
Hey Rainer,
>It got stripped by the list. Can you post it somewhere and make the URL
>available?
here you go:
http://www.telestar.de/tmp10356/threaddump.txt
thanks
David
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections
keept open
Posted by David Kumar <dk...@telestar.de>.
Hey Rainer,
>It got stripped by the list. Can you post it somewhere and make the URL
available?
here you go:
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections
keept open
Posted by Rainer Jung <ra...@kippdata.de>.
On 15.03.2013 15:44, David Kumar wrote:
> Hey Rainer,
>
> attached you can find a Threaddump. Just rename it to .zip.
> I'm not sure waht all the stuff at the dump means. but I'm sure you know.. :-)
It got stripped by the list. Can you post it somewhere and make the URL
available?
Regards,
Rainer
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections
keept open
Posted by David Kumar <dk...@telestar.de>.
Hey Rainer,
attached you can find a Threaddump. Just rename it to .zip.
I'm not sure waht all the stuff at the dump means. but I'm sure you know.. :-)
thanks
>Please please before restarting take two thread dumps. On Linux/Unix a
>"kill -QUIT" will do that (and write the result to STDOUT, which
>depending on your setup likely goes to catalina.out). The QUIT signal
>does *not* terminate the JVM, it just quickly writes out the stacks of
>all threads. The interruption for the running JVM is minimal, so you can
>already try how this works before the process gets into the critical state.
Re: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections
keept open
Posted by Rainer Jung <ra...@kippdata.de>.
On 15.03.2013 11:57, David Kumar wrote:
>> It could be, that your requests in Tomcat got stuck and Tomcat still is
>> in the state of working on the requests, therefore keeping the
>> connection open to send back stuff finally, whereas mod_jk has already
>> timed out. To check for that, take a couöple of threa dumps (not: heap
>> dumps) of the running Tomcat process while the close_wait problem is
>> visible. Check what your Tomcat threads are currently doing, e.g. are
>> they mostly sitting idle in the thread pool or executor, or are many of
>> them deep in your application stacks and waiting for database, locks or
>> other stuff.
>
> I try to, currently we have a "watchdog" running, checking every 2 minutes if the tomcats are available at 7009/9009 and http. If one of the tomcats are not responding it will be restarted....
Please please before restarting take two thread dumps. On Linux/Unix a
"kill -QUIT" will do that (and write the result to STDOUT, which
depending on your setup likely goes to catalina.out). The QUIT signal
does *not* terminate the JVM, it just quickly writes out the stacks of
all threads. The interruption for the running JVM is minimal, so you can
already try how this works before the process gets into the critical state.
Regards,
Rainer
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections
keept open
Posted by David Kumar <dk...@telestar.de>.
>close_wait for an AJP connection seen from Tomcat means the other side -
>mod_jk - has closed the connection, but not Tomcat.
>
>This is often due to a shorter Timeout on the mod_jk side than on the
>Tomcat side. It is not a problem per se, but it is if it happens for too
>many connections for a too long time.
>
>I vaguely remember that you have a 10 second socket_timeout in your
>workers.properties. That's typically bad. Look at the example config in
>the source mod_jk download.
I will do that.
>It could be, that your requests in Tomcat got stuck and Tomcat still is
>in the state of working on the requests, therefore keeping the
>connection open to send back stuff finally, whereas mod_jk has already
>timed out. To check for that, take a couöple of threa dumps (not: heap
>dumps) of the running Tomcat process while the close_wait problem is
>visible. Check what your Tomcat threads are currently doing, e.g. are
>they mostly sitting idle in the thread pool or executor, or are many of
>them deep in your application stacks and waiting for database, locks or
>other stuff.
I try to, currently we have a "watchdog" running, checking every 2 minutes if the tomcats are available at 7009/9009 and http. If one of the tomcats are not responding it will be restarted....
Thanks
David
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections
keept open
Posted by Rainer Jung <ra...@kippdata.de>.
On 15.03.2013 11:47, André Warnier wrote:
> Rainer Jung wrote:
>> On 14.03.2013 10:04, David Kumar wrote:
>>> Hey,
>>>
>>> thanks for note..
>>> Attached you can find a new list.
>>> So, java is keeping these connections in close_wait.
>>
>> close_wait for an AJP connection seen from Tomcat means the other side -
>> mod_jk - has closed the connection, but not Tomcat.
>>
>> This is often due to a shorter Timeout on the mod_jk side than on the
>> Tomcat side. It is not a problem per se, but it is if it happens for too
>> many connections for a too long time.
>>
>> I vaguely remember that you have a 10 second socket_timeout in your
>> workers.properties. That's typically bad. Look at the example config in
>> the source mod_jk download.
>>
>> It could be, that your requests in Tomcat got stuck and Tomcat still is
>> in the state of working on the requests, therefore keeping the
>> connection open to send back stuff finally, whereas mod_jk has already
>> timed out. To check for that, take a couöple of threa dumps (not: heap
>> dumps) of the running Tomcat process while the close_wait problem is
>> visible. Check what your Tomcat threads are currently doing, e.g. are
>> they mostly sitting idle in the thread pool or executor, or are many of
>> them deep in your application stacks and waiting for database, locks or
>> other stuff.
>>
>
> Hi Rainer, a question to you :
>
> In a previous post, David posted the output of "netstat -t -pan".
> In that output, there are about 1900 connections from Apache to Tomcat's
> AJP connectors, in state "TIME_WAIT".
> As far as I know, this indicates that the connection is closed from the
> point of view of Apache, and this TIME_WAIT should last only a few sec.
> maximum, and then should go away.
> Why does he have so many though ? I can't see anything like that on any
> of my servers.
TIME_WAIT is a state which means the connection is closed by both sides
but the TCP stack still blocks the (server port, server ip, client port,
client IP) quadruple for reuse to prevent long overdue packets of the
old connection to disturb the new connection. It doesn't prevent new
connections from being created nor are there still process resources on
client or server being used for those connections (threds, buffers). It
is just an entry in the OS TCP connection hash.
The TIME_WAIT state is only used on the side of the connection that
started the connection shutdown, i.e. that first send the FIN. The other
side doesn't go into TIME_WAIT. If your clients typically start the
connection shutdown you won't see the TIME_WAIT.
Connections are kept in TIME_WAIT for seconds to about a minute
depending on your OS and TCP tuning. For a service which uses a lot of
new connections per second and initiates the shutdown for them itself it
is not uncommon to see several hundreds to thousands of TIME_WAIT
connections. If the count goes up into the range of 10.000-50.000 you
could start running into inefficiencies of the TCP connection hash
handling (depending on the OS).
Regards,
Rainer
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections
keept open
Posted by André Warnier <aw...@ice-sa.com>.
Rainer Jung wrote:
> On 14.03.2013 10:04, David Kumar wrote:
>> Hey,
>>
>> thanks for note..
>> Attached you can find a new list.
>> So, java is keeping these connections in close_wait.
>
> close_wait for an AJP connection seen from Tomcat means the other side -
> mod_jk - has closed the connection, but not Tomcat.
>
> This is often due to a shorter Timeout on the mod_jk side than on the
> Tomcat side. It is not a problem per se, but it is if it happens for too
> many connections for a too long time.
>
> I vaguely remember that you have a 10 second socket_timeout in your
> workers.properties. That's typically bad. Look at the example config in
> the source mod_jk download.
>
> It could be, that your requests in Tomcat got stuck and Tomcat still is
> in the state of working on the requests, therefore keeping the
> connection open to send back stuff finally, whereas mod_jk has already
> timed out. To check for that, take a couöple of threa dumps (not: heap
> dumps) of the running Tomcat process while the close_wait problem is
> visible. Check what your Tomcat threads are currently doing, e.g. are
> they mostly sitting idle in the thread pool or executor, or are many of
> them deep in your application stacks and waiting for database, locks or
> other stuff.
>
Hi Rainer, a question to you :
In a previous post, David posted the output of "netstat -t -pan".
In that output, there are about 1900 connections from Apache to Tomcat's AJP connectors,
in state "TIME_WAIT".
As far as I know, this indicates that the connection is closed from the point of view of
Apache, and this TIME_WAIT should last only a few sec. maximum, and then should go away.
Why does he have so many though ? I can't see anything like that on any of my servers.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections
keept open
Posted by Rainer Jung <ra...@kippdata.de>.
On 14.03.2013 10:04, David Kumar wrote:
>
> Hey,
>
> thanks for note..
> Attached you can find a new list.
> So, java is keeping these connections in close_wait.
close_wait for an AJP connection seen from Tomcat means the other side -
mod_jk - has closed the connection, but not Tomcat.
This is often due to a shorter Timeout on the mod_jk side than on the
Tomcat side. It is not a problem per se, but it is if it happens for too
many connections for a too long time.
I vaguely remember that you have a 10 second socket_timeout in your
workers.properties. That's typically bad. Look at the example config in
the source mod_jk download.
It could be, that your requests in Tomcat got stuck and Tomcat still is
in the state of working on the requests, therefore keeping the
connection open to send back stuff finally, whereas mod_jk has already
timed out. To check for that, take a couöple of threa dumps (not: heap
dumps) of the running Tomcat process while the close_wait problem is
visible. Check what your Tomcat threads are currently doing, e.g. are
they mostly sitting idle in the thread pool or executor, or are many of
them deep in your application stacks and waiting for database, locks or
other stuff.
Regards,
Rainer
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: AW:AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections
keept open
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
André,
On 3/14/13 11:02 AM, André Warnier wrote:
> But this is the request/response connection, so I doubt that there
> would be a bug there, otherwise we'd have problems reports filling
> this list every day.
>
> Might there be that there is somewhere a discrepancy between the
> keep-alive settings, between Apache and Tomcat ?
I'd take a look at the connection timeout settings between mod_jk and
Tomcat: if they are mismatched, you might get some weirdness where
connections are going stale on the Apache side and not the Tomcat side
(or vice-versa). It's easy to do since one side expresses timeouts in
ms and the other side in seconds. :)
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iEYEAREIAAYFAlFDfBsACgkQ9CaO5/Lv0PD+ggCguhh5KOW+hoxxhwRFIY0AsH8N
/30AoJ2sfyMb+xSqewoI0iOSBJRXq0hg
=Up57
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
AW: AW:AJP suddenly Stopps acting: ajp on 7009 and 9009 :
connections keept open
Posted by David Kumar <dk...@telestar.de>.
Hey André,
> According to vmtools the server is boring most times..
>What do you mean, "boring" ? (like German "langweilig" ?).
--> Jep prety idle.. :-)
>If you have 2000+ connections there in one state or another, that should >mean that the
>system is usually pretty busy, I would think. On my production machines > (vSphere VM's,
>Linux, Apache+Tomcat, +database), some of them pretty busy, I never see >more than 200-300
>connections with netstat. That is all states together, with "netstat -t ->pan | wc -l").
>Granted, I am not running Yahoo or Facebook, but still, these are public >Internet servers,
>fairly sollicited.
>
> I almost totally agree, but I'm not sure if it is a AJP fault. Because >most time everything is alright, so all of theses steps are done. Just >sometimes after a arbitrarily amount of time ajp / tomcat runs crazy. And >in my sample there is the sourceport 9009 / java. It looks for me that >apache didn't received the ACK from tomcat after CLOSE_WAIT or it just >don't care because apache thought this connections is already closed? >Let's assume that is the case.. Isn't there a option to tell tomcat to wait >max. time for a response and after that time kill the connection? As far as >I understand connectionTimeout is just for the initial request, so after >Apache did answer in first correctly connectionTimeout is valueless?
>
>Let's look at the various options in the Connector :
>First, the "-1" in some options means "unlimited" or "forever", it doesn't >mean "disabled".
>Second, some parameters say that if they are not specified, they default to >the value of
>another parameter, so you should check really what this other parameter is >set to.
According to Tomcat Configuration Reference 5.5 :
connectionLinger
The number of milliseconds during which the sockets used by this Connector will linger when they are closed. The default value is -1 (socket linger is disabled).
According to Tomcat Configuration Reference 7.0.37 :
connectionLinger
The number of seconds during which the sockets used by this Connector will linger when they are closed. If not specified, the JVM default will be used.
As I'm at 7.0.37 I have to figure out where an how to set linger options at the JVM.
> I'll have a look.
thx
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
AW:AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections
keept open
Posted by David Kumar <dk...@telestar.de>.
Mit freundlichen Grüßen
David Kumar
Softwareentwickler, B. Sc.
Abteilung Infotech - Interaktiv
TELESTAR-DIGITAL GmbH
Am Weiher 14
D-56766 Ulmen
Tel.: + 49 (0) 6592 / 712 -2826
Tel.: + 49 (0) 2676 / 9520 -183
Fax: + 49 (0) 6592 / 712 -2829
http://www.telestar.de/
-----Ursprüngliche Nachricht-----
Von: André Warnier [mailto:aw@ice-sa.com]
Gesendet: Freitag, 15. März 2013 10:45
An: Tomcat Users List
Betreff: Re: AW:AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
>Some notes :
>- according to the comments in your Apache config file, you are using the >"worker MPM".
>But are you sure ? what does "/usr/sbin/apache2ctl -l" say ?
Compiled in modules:
core.c
mod_log_config.c
mod_logio.c
worker.c
http_core.c
mod_so.c
>- we are missing the contents of your "/etc/apache2/workers.properties" file
Sry didn't rename properties, see attached..
>- we are missing the JkMounts or equivalent that you are doing from Apache >to Tomcat.
Same to our default website
> related questions : you have 2 tomcats. Are you doing load-balancing ? or >are you just
>sending some URLs to tomcat1 and other to tomcat2 ?
We doing loadbalacning
>Are you proxying everything to the tomcats, or is the Apache httpd front->end serving some URLs on its own ? if so, what proportion ?
Everything goes to the tomcats
>- how many hits per second/minute/hour (any of them, approx.) is your >server handling ?
according to our logs yesterday between 8pm and 9pm we had 73667 hits together on both tomcats.
>- you do not have any specific timeout parameters set in your Tomcat AJP ><Connectors>,
> which is basically a good thing : better to leave the defaults in place, >than to start
>playing with settings that you do not really understand, and make things >worse.
But you do set one :
connectionTimeout="200000". Which sounds extremely high to me.
Re: http://tomcat.apache.org/tomcat-7.0-doc/config/ajp.html
>It means that when a client connects to that Connector, a Tomcat thread >will be started to
>handle this connection; then the thread will wait on the connection, to >read the request.
>If the request does not appear, this thread will still wait, up to 200 >seconds (more than
>3 minutes !) for this request to appear, before it gives up, closes that >connection, and
>goes back to the pool of available threads.
>A normal client would not do that, but a badly-intentioned client that >tries to create a
>DOS attack on your server, will do that, just to block threads on your >server until there
>are no more threads available.
>
>In this case, it is even worse, because other parameters are using this >same value as a
>default, like : keepAliveTimeout.
>This one means that once one request has been processed by this Tomcat >thread on that
>connection, the thread will not close this connection, but wait to see if >any other
>request appears on that connection from the same client, within the timeout >given.
>In this case, because it defaults to connectionTimeout, the thread will >wait more than 3
>minutes to see if there is another request. If the client (browser) does >not send any
>additional request on that connection, you have a thread that is blocked >doing nothing,
>for more than 3 minutes.
>In this case, the client is in reality the mod_jk module under Apache. So >the settings of
>Apache and mod_jk will matter, and I cannot tell exactly at this point what >will happen.
>But if this was a HTTP Connector open to the external WWW world, what would >certainly
>heppen is that you would run out of threads in Tomcat within a couple of >minutes, and your
>server would stop accepting new requests.
>
>
>Or to put this another way : if this was a HTTP connector open to the >outside world, and I
>knew the address of your server, I could bring it down in 3 seconds, using >the standard
>"ab" program that comes with Apache.
that is how our servers are connected together:
We have another Webserver (different server) connecting to our Apache - Proxy which is loadbalacning our two tomcats. (Apache and tomcat are together at on server)
I will go end reduce the timeout on both worker.properties and server.xml. What will be a value? 30 Seconds?
Thanks a lot..
Re: AW:AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections
keept open
Posted by André Warnier <aw...@ice-sa.com>.
> David Kumar wrote:
>
>> Attached you can find our configuration files, maybe you can find a /
>> some misconfigurations?
>>
>
> I'll have a look.
>
Some notes :
- according to the comments in your Apache config file, you are using the "worker MPM".
But are you sure ? what does "/usr/sbin/apache2ctl -l" say ?
- we are missing the contents of your "/etc/apache2/workers.properties" file
- we are missing the JkMounts or equivalent that you are doing from Apache to Tomcat.
related questions : you have 2 tomcats. Are you doing load-balancing ? or are you just
sending some URLs to tomcat1 and other to tomcat2 ?
Are you proxying everything to the tomcats, or is the Apache httpd front-end serving some
URLs on its own ? if so, what proportion ?
- how many hits per second/minute/hour (any of them, approx.) is your server handling ?
- you do not have any specific timeout parameters set in your Tomcat AJP <Connectors>,
which is basically a good thing : better to leave the defaults in place, than to start
playing with settings that you do not really understand, and make things worse.
But you do set one :
connectionTimeout="200000". Which sounds extremely high to me.
Re: http://tomcat.apache.org/tomcat-7.0-doc/config/ajp.html
It means that when a client connects to that Connector, a Tomcat thread will be started to
handle this connection; then the thread will wait on the connection, to read the request.
If the request does not appear, this thread will still wait, up to 200 seconds (more than
3 minutes !) for this request to appear, before it gives up, closes that connection, and
goes back to the pool of available threads.
A normal client would not do that, but a badly-intentioned client that tries to create a
DOS attack on your server, will do that, just to block threads on your server until there
are no more threads available.
In this case, it is even worse, because other parameters are using this same value as a
default, like : keepAliveTimeout.
This one means that once one request has been processed by this Tomcat thread on that
connection, the thread will not close this connection, but wait to see if any other
request appears on that connection from the same client, within the timeout given.
In this case, because it defaults to connectionTimeout, the thread will wait more than 3
minutes to see if there is another request. If the client (browser) does not send any
additional request on that connection, you have a thread that is blocked doing nothing,
for more than 3 minutes.
In this case, the client is in reality the mod_jk module under Apache. So the settings of
Apache and mod_jk will matter, and I cannot tell exactly at this point what will happen.
But if this was a HTTP Connector open to the external WWW world, what would certainly
heppen is that you would run out of threads in Tomcat within a couple of minutes, and your
server would stop accepting new requests.
Or to put this another way : if this was a HTTP connector open to the outside world, and I
knew the address of your server, I could bring it down in 3 seconds, using the standard
"ab" program that comes with Apache.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: AW:AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections
keept open
Posted by André Warnier <aw...@ice-sa.com>.
David Kumar wrote:
> hey,
>> Yes. Whether this is the java that runs tomcat is not immediately evident.
>> But it also seems that they concern port numbers like 7009, 9009 etc. which >must be your
>> AJP Connectors, so let's assume the CLOSE-WAITs all involve Tomcat.
>>
>> The first observation hat I would make is that this looks like a pretty >busy system.
>>
>> Then, there are 2100+ lines in that list. Let's assume first that they are >all internal
>> connections (which probable they are not, but for simplicity), so that >there are actually
>> 2 lines for the same connection, like these :
>>
>> tcp 0 0 127.0.0.1:39872 127.0.0.1:7009 >FIN_WAIT2
>> 20766/apache2
>> . ..
>> tcp6 1 0 127.0.0.1:7009 127.0.0.1:39872 >CLOSE_WAIT 20564/java
>>
>> ..
>>
>> So let's say that we have in total about 1000 connections active in some >state.
>> Compared to that, there are 54 connections in CLOSE_WAIT, which is like 5%.
>> That does not really sound "pathological" to me, considering that >CLOSE_WAIT is a normal
>> state through which any TCP connection goes at some point.
>>
>> If you look again at the above connection, and if I am not confusing my >clients and
>> servers again, it looks here like :
>> - Apache http (PID 20766), as a client, has established a connection to >port 7009 of the
>> server Tomcat (PID 20564). That must be an AJP connection, from the mod_jk >module of
>> Apache, to the AJP Connector in Tomcat.
>> - A while later, Apache closes the connection. It sends a "FIN" packet to >Tomcat, and
>> waits for Tomcat to acknowledge this FIN packet.
>> - when it receives the FIN packet from Apache, Tomcat responds with an ACK, >and then goes
>> to the CLOSE_WAIT state.
>> - when Apache receives the ACK from Tomcat, it goes to state FIN_WAIT2.
>> (and that is the current state of the above connection)
>> - now Tomcat is supposed to send a FIN to Apache, and receive an ACK in >return.
>> - if Apache receives the FIN from Tomcat, it sends back an ACK, and goes >into state
>> TIME_WAIT (where it doesn't expect anything anymore), and then after a >short moment, it
>> discards that connection.
>>
>> If there are connections that remain in the CLOSE_WAIT state for very long, >it means that
>> something in the last 2 steps above is not working.
>>
>> Now in your listing, we see about 1900 lines in the TIME_WAIT state, all of >them *to* an
>> AJP port of Tomcat. This seems to suggest that the Apache side is working >as it should.
>> But if the Tomcat side stays in CLOSE_WAIT for a long time, then it would >suggest that on
>> the Tomcat side, that connection is never properly close()'d.
>
>> But this is the request/response connection, so I doubt that there would be >a bug there,
>> otherwise we'd have problems reports filling this list every day.
>>
>> Might there be that there is somewhere a discrepancy between the keep-alive >settings,
>> between Apache and Tomcat ?
>>
>> or maybe your server is just not strong enough for the load you are putting >on it ?
>
> According to vmtools the server is boring most times..
What do you mean, "boring" ? (like German "langweilig" ?).
If you have 2000+ connections there in one state or another, that should mean that the
system is usually pretty busy, I would think. On my production machines (vSphere VM's,
Linux, Apache+Tomcat, +database), some of them pretty busy, I never see more than 200-300
connections with netstat. That is all states together, with "netstat -t -pan | wc -l").
Granted, I am not running Yahoo or Facebook, but still, these are public Internet servers,
fairly sollicited.
>
> I almost totally agree, but I'm not sure if it is a AJP fault. Because most time everything is alright, so all of theses steps are done. Just sometimes after a arbitrarily amount of time ajp / tomcat runs crazy. And in my sample there is the sourceport 9009 / java. It looks for me that apache didn't received the ACK from tomcat after CLOSE_WAIT or it just don't care because apache thought this connections is already closed? Let's assume that is the case.. Isn't there a option to tell tomcat to wait max. time for a response and after that time kill the connection? As far as I understand connectionTimeout is just for the initial request, so after Apache did answer in first correctly connectionTimeout is valueless?
>
Let's look at the various options in the Connector :
First, the "-1" in some options means "unlimited" or "forever", it doesn't mean "disabled".
Second, some parameters say that if they are not specified, they default to the value of
another parameter, so you should check really what this other parameter is set to.
> Attached you can find our configuration files, maybe you can find a / some misconfigurations?
>
I'll have a look.
> I would agree that we have maybe a kind of misconfigured timeout problem...
>
>
> thanks
>
>
>
> Mit freundlichen Grüßen
> David Kumar
> Softwareentwickler, B. Sc.
> Abteilung Infotech - Interaktiv
> TELESTAR-DIGITAL GmbH
> Am Weiher 14
> D-56766 Ulmen
>
> http://www.telestar.de/
>
>
> ------------------------------------------------------------------------
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
AW:AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections
keept open
Posted by David Kumar <dk...@telestar.de>.
hey,
>Yes. Whether this is the java that runs tomcat is not immediately evident.
>But it also seems that they concern port numbers like 7009, 9009 etc. which >must be your
>AJP Connectors, so let's assume the CLOSE-WAITs all involve Tomcat.
>
>The first observation hat I would make is that this looks like a pretty >busy system.
>
>Then, there are 2100+ lines in that list. Let's assume first that they are >all internal
>connections (which probable they are not, but for simplicity), so that >there are actually
>2 lines for the same connection, like these :
>
>tcp 0 0 127.0.0.1:39872 127.0.0.1:7009 >FIN_WAIT2
>20766/apache2
>. ..
>tcp6 1 0 127.0.0.1:7009 127.0.0.1:39872 >CLOSE_WAIT 20564/java
>
>..
>
>So let's say that we have in total about 1000 connections active in some >state.
>Compared to that, there are 54 connections in CLOSE_WAIT, which is like 5%.
>That does not really sound "pathological" to me, considering that >CLOSE_WAIT is a normal
>state through which any TCP connection goes at some point.
>
>If you look again at the above connection, and if I am not confusing my >clients and
>servers again, it looks here like :
>- Apache http (PID 20766), as a client, has established a connection to >port 7009 of the
>server Tomcat (PID 20564). That must be an AJP connection, from the mod_jk >module of
>Apache, to the AJP Connector in Tomcat.
>- A while later, Apache closes the connection. It sends a "FIN" packet to >Tomcat, and
>waits for Tomcat to acknowledge this FIN packet.
>- when it receives the FIN packet from Apache, Tomcat responds with an ACK, >and then goes
>to the CLOSE_WAIT state.
>- when Apache receives the ACK from Tomcat, it goes to state FIN_WAIT2.
> (and that is the current state of the above connection)
>- now Tomcat is supposed to send a FIN to Apache, and receive an ACK in >return.
>- if Apache receives the FIN from Tomcat, it sends back an ACK, and goes >into state
>TIME_WAIT (where it doesn't expect anything anymore), and then after a >short moment, it
>discards that connection.
>
>If there are connections that remain in the CLOSE_WAIT state for very long, >it means that
>something in the last 2 steps above is not working.
>
>Now in your listing, we see about 1900 lines in the TIME_WAIT state, all of >them *to* an
>AJP port of Tomcat. This seems to suggest that the Apache side is working >as it should.
>But if the Tomcat side stays in CLOSE_WAIT for a long time, then it would >suggest that on
>the Tomcat side, that connection is never properly close()'d.
>But this is the request/response connection, so I doubt that there would be >a bug there,
>otherwise we'd have problems reports filling this list every day.
>
>Might there be that there is somewhere a discrepancy between the keep-alive >settings,
>between Apache and Tomcat ?
>
>or maybe your server is just not strong enough for the load you are putting >on it ?
According to vmtools the server is boring most times..
I almost totally agree, but I'm not sure if it is a AJP fault. Because most time everything is alright, so all of theses steps are done. Just sometimes after a arbitrarily amount of time ajp / tomcat runs crazy. And in my sample there is the sourceport 9009 / java. It looks for me that apache didn't received the ACK from tomcat after CLOSE_WAIT or it just don't care because apache thought this connections is already closed? Let's assume that is the case.. Isn't there a option to tell tomcat to wait max. time for a response and after that time kill the connection? As far as I understand connectionTimeout is just for the initial request, so after Apache did answer in first correctly connectionTimeout is valueless?
Attached you can find our configuration files, maybe you can find a / some misconfigurations?
I would agree that we have maybe a kind of misconfigured timeout problem...
thanks
Mit freundlichen Grüßen
David Kumar
Softwareentwickler, B. Sc.
Abteilung Infotech - Interaktiv
TELESTAR-DIGITAL GmbH
Am Weiher 14
D-56766 Ulmen
http://www.telestar.de/
Re: AW:AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections
keept open
Posted by André Warnier <aw...@ice-sa.com>.
David Kumar wrote:
> Hey,
>
> thanks for note..
> Attached you can find a new list.
> So, java is keeping these connections in close_wait.
>
Yes. Whether this is the java that runs tomcat is not immediately evident.
But it also seems that they concern port numbers like 7009, 9009 etc. which must be your
AJP Connectors, so let's assume the CLOSE-WAITs all involve Tomcat.
The first observation hat I would make is that this looks like a pretty busy system.
Then, there are 2100+ lines in that list. Let's assume first that they are all internal
connections (which probable they are not, but for simplicity), so that there are actually
2 lines for the same connection, like these :
tcp 0 0 127.0.0.1:39872 127.0.0.1:7009 FIN_WAIT2
20766/apache2
...
tcp6 1 0 127.0.0.1:7009 127.0.0.1:39872 CLOSE_WAIT 20564/java
..
So let's say that we have in total about 1000 connections active in some state.
Compared to that, there are 54 connections in CLOSE_WAIT, which is like 5%.
That does not really sound "pathological" to me, considering that CLOSE_WAIT is a normal
state through which any TCP connection goes at some point.
If you look again at the above connection, and if I am not confusing my clients and
servers again, it looks here like :
- Apache http (PID 20766), as a client, has established a connection to port 7009 of the
server Tomcat (PID 20564). That must be an AJP connection, from the mod_jk module of
Apache, to the AJP Connector in Tomcat.
- A while later, Apache closes the connection. It sends a "FIN" packet to Tomcat, and
waits for Tomcat to acknowledge this FIN packet.
- when it receives the FIN packet from Apache, Tomcat responds with an ACK, and then goes
to the CLOSE_WAIT state.
- when Apache receives the ACK from Tomcat, it goes to state FIN_WAIT2.
(and that is the current state of the above connection)
- now Tomcat is supposed to send a FIN to Apache, and receive an ACK in return.
- if Apache receives the FIN from Tomcat, it sends back an ACK, and goes into state
TIME_WAIT (where it doesn't expect anything anymore), and then after a short moment, it
discards that connection.
If there are connections that remain in the CLOSE_WAIT state for very long, it means that
something in the last 2 steps above is not working.
Now in your listing, we see about 1900 lines in the TIME_WAIT state, all of them *to* an
AJP port of Tomcat. This seems to suggest that the Apache side is working as it should.
But if the Tomcat side stays in CLOSE_WAIT for a long time, then it would suggest that on
the Tomcat side, that connection is never properly close()'d.
But this is the request/response connection, so I doubt that there would be a bug there,
otherwise we'd have problems reports filling this list every day.
Might there be that there is somewhere a discrepancy between the keep-alive settings,
between Apache and Tomcat ?
or maybe your server is just not strong enough for the load you are putting on it ?
>
>
> How about a "netstat -t -pan" ? it's a bit easier to find the relevant connections.
> If needed, tell us which processes are Tomcat instances/threads and which are Apache httpd
> (if it is not evident from the list).
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
> Mit freundlichen Grüßen
> David Kumar
> Softwareentwickler, B. Sc.
> Abteilung Infotech - Interaktiv
> TELESTAR-DIGITAL GmbH
> Am Weiher 14
> D-56766 Ulmen
>
> http://www.telestar.de/
>
>
>
> ------------------------------------------------------------------------
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
AW:AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections
keept open
Posted by David Kumar <dk...@telestar.de>.
Hey,
thanks for note..
Attached you can find a new list.
So, java is keeping these connections in close_wait.
How about a "netstat -t -pan" ? it's a bit easier to find the relevant connections.
If needed, tell us which processes are Tomcat instances/threads and which are Apache httpd
(if it is not evident from the list).
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Mit freundlichen Grüßen
David Kumar
Softwareentwickler, B. Sc.
Abteilung Infotech - Interaktiv
TELESTAR-DIGITAL GmbH
Am Weiher 14
D-56766 Ulmen
http://www.telestar.de/
Re: AW: AW: AW: AJP suddenly Stopps acting: ajp on 7009 and 9009
: connections keept open
Posted by André Warnier <aw...@ice-sa.com>.
Hi.
Note:
try to avoid top-posting, on this list.
(top-posting means responding on top of the previous message, like I am doing just now).
The reason why we ask not to do that, is that it forces whoever is reading your message,
to scroll down and try to figure out to what question(s) you are answering.
Answer in the text, just below the question, or at the end (and remove what is no longer
relevant). This way, someone else can easily follow the logic flow of the matter.
End of Note.
David Kumar wrote:
> Hey,
>
> I'm using:
>
> lsof -u tomcat
>
> Both of our tomcats are runing wiht user tomcat..
> Attached you can find a sample from today 07:00 am (GMT+1)
>
>
How about a "netstat -t -pan" ? it's a bit easier to find the relevant connections.
If needed, tell us which processes are Tomcat instances/threads and which are Apache httpd
(if it is not evident from the list).
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
AW: AW: AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 :
connections keept open
Posted by David Kumar <dk...@telestar.de>.
Hey,
I'm using:
lsof -u tomcat
Both of our tomcats are runing wiht user tomcat..
Attached you can find a sample from today 07:00 am (GMT+1)
Mit freundlichen Grüßen
David Kumar
Softwareentwickler, B. Sc.
Abteilung Infotech - Interaktiv
TELESTAR-DIGITAL GmbH
Am Weiher 14
D-56766 Ulmen
http://www.telestar.de/
-----Ursprüngliche Nachricht-----
Von: André Warnier [mailto:aw@ice-sa.com]
Gesendet: Mittwoch, 13. März 2013 21:41
An: Tomcat Users List
Betreff: Re: AW: AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
David Kumar wrote:
> BTW... after a while CLOSE_WAIT changes to "can't identify protocol"
Huh ?
Which command are you using to see the connections in CLOSE_WAIT state ?
(and maybe paste a sample here)
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: AW: AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections
keept open
Posted by André Warnier <aw...@ice-sa.com>.
David Kumar wrote:
> BTW... after a while CLOSE_WAIT changes to "can't identify protocol"
Huh ?
Which command are you using to see the connections in CLOSE_WAIT state ?
(and maybe paste a sample here)
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
AW: AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 :
connections keept open
Posted by David Kumar <dk...@telestar.de>.
BTW... after a while CLOSE_WAIT changes to "can't identify protocol"
Mit freundlichen Grüßen
David Kumar
Softwareentwickler, B. Sc.
Abteilung Infotech - Interaktiv
TELESTAR-DIGITAL GmbH
Am Weiher 14
D-56766 Ulmen
Tel.: + 49 (0) 6592 / 712 -2826
Tel.: + 49 (0) 2676 / 9520 -183
Fax: + 49 (0) 6592 / 712 -2829
http://www.telestar.de/
-----Ursprüngliche Nachricht-----
Von: David Kumar
Gesendet: Mittwoch, 13. März 2013 20:01
An: 'Tomcat Users List'
Betreff: AW: AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
Hey,
right no we're having our Problem.
I tried gc through jconsole --> no changes and still a lot CLOSE_WAIT.
So it is not a GC Problem and disablereuse doesn't work either..
Any other ideas?
What do you guy think about proxy_ajp instead of jk? What are the advantages of proxy_ajp?
Thanks a lot..
Mit freundlichen Grüßen
David Kumar
Softwareentwickler, B. Sc.
Abteilung Infotech - Interaktiv
TELESTAR-DIGITAL GmbH
Am Weiher 14
D-56766 Ulmen
http://www.telestar.de/
-----Ursprüngliche Nachricht-----
Von: David Kumar [mailto:dkumar@telestar.de]
Gesendet: Mittwoch, 13. März 2013 14:32
An: Tomcat Users List
Betreff: AW: AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
Hey,
I'm really not sure if that tool does anything, because there is no feedback..
Funny: Just a few minutes ago a colleague showed ma a tool called something like VM monitor on his mac. It look a kind of similar to jconsole.exe. Next time I will try to make GC with this tool.
Meanwhile I reduced the connectionTimeout, reconfigured maxClients and added -DisableReuse to jkpropertys.
Lets see what happens...
Thanks a lot..
Mit freundlichen Grüßen
David Kumar
Softwareentwickler, B. Sc.
Abteilung Infotech - Interaktiv
TELESTAR-DIGITAL GmbH
Am Weiher 14
D-56766 Ulmen
http://www.telestar.de/
-----Ursprüngliche Nachricht-----
Von: André Warnier [mailto:aw@ice-sa.com]
Gesendet: Mittwoch, 13. März 2013 12:20
An: Tomcat Users List
Betreff: Re: AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
David Kumar wrote:
> Hey,
>
> we just had that problem again. I did that GC trick, it didn't work out well.
> I used:
>
> set hosts [list]
> lappend hosts {localhost:7008}
> lappend hosts {localhost:9008}
> # ...add as many as you want...
>
> foreach {host} $hosts {
> set parts [split $host ":"]
> set hostname [lindex $parts 0]
> set port [lindex $parts 1]
>
> # for each host...
>
> # Connect to it.
> jmx_connect -h $hostname -p $port
>
> # Invoke the garbage collector.
> jmx_invoke -n -m java.lang:type=Memory gc
>
> # Close this connection
> jmx_close
> }
>
> I found that while googleing.
> But it doesn't have any effect. Any other ideas?
>
Did it really connect with the Tomcat JVM and really do anything ?
Is your JVM started with options that allow JMX interaction ?
Also, there are ways to start the JVM with parameters telling it to log the Garbage
Collections to a logfile. Make sure that you do that, and check the logfile to see if it
really does a GC when you ask it to.
Note that there are also other ways to trogger a GC than using jmxsh. I was only pointing
you to that tool as an easy-to-script, non-GUI tool.
But in your java directory (at least the JDK/bin), there should also be a "jconsole"
program which allows you to do pretty much the same interactively with a GUI.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
AW: AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 :
connections keept open
Posted by David Kumar <dk...@telestar.de>.
Hey,
I'm really not sure if that tool does anything, because there is no feedback..
Funny: Just a few minutes ago a colleague showed ma a tool called something like VM monitor on his mac. It look a kind of similar to jconsole.exe. Next time I will try to make GC with this tool.
Meanwhile I reduced the connectionTimeout, reconfigured maxClients and added -DisableReuse to jkpropertys.
Lets see what happens...
Thanks a lot..
Mit freundlichen Grüßen
David Kumar
Softwareentwickler, B. Sc.
Abteilung Infotech - Interaktiv
TELESTAR-DIGITAL GmbH
Am Weiher 14
D-56766 Ulmen
http://www.telestar.de/
-----Ursprüngliche Nachricht-----
Von: André Warnier [mailto:aw@ice-sa.com]
Gesendet: Mittwoch, 13. März 2013 12:20
An: Tomcat Users List
Betreff: Re: AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
David Kumar wrote:
> Hey,
>
> we just had that problem again. I did that GC trick, it didn't work out well.
> I used:
>
> set hosts [list]
> lappend hosts {localhost:7008}
> lappend hosts {localhost:9008}
> # ...add as many as you want...
>
> foreach {host} $hosts {
> set parts [split $host ":"]
> set hostname [lindex $parts 0]
> set port [lindex $parts 1]
>
> # for each host...
>
> # Connect to it.
> jmx_connect -h $hostname -p $port
>
> # Invoke the garbage collector.
> jmx_invoke -n -m java.lang:type=Memory gc
>
> # Close this connection
> jmx_close
> }
>
> I found that while googleing.
> But it doesn't have any effect. Any other ideas?
>
Did it really connect with the Tomcat JVM and really do anything ?
Is your JVM started with options that allow JMX interaction ?
Also, there are ways to start the JVM with parameters telling it to log the Garbage
Collections to a logfile. Make sure that you do that, and check the logfile to see if it
really does a GC when you ask it to.
Note that there are also other ways to trogger a GC than using jmxsh. I was only pointing
you to that tool as an easy-to-script, non-GUI tool.
But in your java directory (at least the JDK/bin), there should also be a "jconsole"
program which allows you to do pretty much the same interactively with a GUI.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections
keept open
Posted by André Warnier <aw...@ice-sa.com>.
David Kumar wrote:
> Hey,
>
> we just had that problem again. I did that GC trick, it didn't work out well.
> I used:
>
> set hosts [list]
> lappend hosts {localhost:7008}
> lappend hosts {localhost:9008}
> # ...add as many as you want...
>
> foreach {host} $hosts {
> set parts [split $host ":"]
> set hostname [lindex $parts 0]
> set port [lindex $parts 1]
>
> # for each host...
>
> # Connect to it.
> jmx_connect -h $hostname -p $port
>
> # Invoke the garbage collector.
> jmx_invoke -n -m java.lang:type=Memory gc
>
> # Close this connection
> jmx_close
> }
>
> I found that while googleing.
> But it doesn't have any effect. Any other ideas?
>
Did it really connect with the Tomcat JVM and really do anything ?
Is your JVM started with options that allow JMX interaction ?
Also, there are ways to start the JVM with parameters telling it to log the Garbage
Collections to a logfile. Make sure that you do that, and check the logfile to see if it
really does a GC when you ask it to.
Note that there are also other ways to trogger a GC than using jmxsh. I was only pointing
you to that tool as an easy-to-script, non-GUI tool.
But in your java directory (at least the JDK/bin), there should also be a "jconsole"
program which allows you to do pretty much the same interactively with a GUI.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections
keept open
Posted by David Kumar <dk...@telestar.de>.
Hey,
we just had that problem again. I did that GC trick, it didn't work out well.
I used:
set hosts [list]
lappend hosts {localhost:7008}
lappend hosts {localhost:9008}
# ...add as many as you want...
foreach {host} $hosts {
set parts [split $host ":"]
set hostname [lindex $parts 0]
set port [lindex $parts 1]
# for each host...
# Connect to it.
jmx_connect -h $hostname -p $port
# Invoke the garbage collector.
jmx_invoke -n -m java.lang:type=Memory gc
# Close this connection
jmx_close
}
I found that while googleing.
But it doesn't have any effect. Any other ideas?
Thanks....
Mit freundlichen Grüßen
David Kumar
Softwareentwickler, B. Sc.
Abteilung Infotech - Interaktiv
TELESTAR-DIGITAL GmbH
Am Weiher 14
D-56766 Ulmen
http://www.telestar.de/
-----Ursprüngliche Nachricht-----
Von: André Warnier [mailto:aw@ice-sa.com]
Gesendet: Dienstag, 12. März 2013 09:54
An: Tomcat Users List
Betreff: Re: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
David Kumar wrote:
> Hey,
>
> we are still having that issue.
> But we could manage to figure out some more stuff.
> We made a Tomcat and Java update since that time we had our problem a few times again, also we did some reconfiguration with connectors etc.
> The last 2 times we where able to see, that both tomcat by them self where alive. Just ajp on both where dead. We couldn't make a connection either trough 7009 nor 9009. An with our openFiles trick we found a lot of close_wait again, e.g. 200 for 9009. I left the second tomcat on this state for a few ours just to see, what happens. The count of 200 connection with close_wait was kept until a reboot of the tomcat.
Instead of rebooting Tomcat, try to force the Tomcat JVM to do a Major Garbage Collection.
There are a number of tools that allow to do that.
One command-line one which I found practical is jmxsh, here :
http://code.google.com/p/jmxsh/
If when you do a Major GC, these CLOSE_WAIT connections disappear, you will have learned
something about their origin.
And if then - without restarting Tomcat - you can connect again via the AJP ports, you'll
have learned something else.
Go do it and report.
> I would say with some of our reconfiguration we managed to stop increasing connections. But still we are not sure why our ajp connections dying..
>
> Here is our connector out of Server.xml:
>
> <Connector port="9009" protocol="AJP/1.3" redirectPort="9443" maxThreads="200" connectionTimeout="600000" />
>
>
> worker.properties:
>
> worker.tomcatX.host=localhost
> worker.tomcatX.type=ajp13
> worker.tomcatX.fail_on_status=404
> worker.tomcatX.lbfactor=1
> worker.tomcatX.ping_timeout=1000
> worker.tomcatX.ping_mode=A
> worker.tomcatX.socket_timeout=10
> worker.tomcatX.connection_pool_timeout=600
>
>
> worker.tomcat1.reference=worker.tomcatX
> worker.tomcat1.port=7009
>
> worker.tomcat2.reference=worker.tomcatX
> worker.tomcat2.port=9009
>
> worker.loadbalancer.type=lb
> worker.loadbalancer.balance_workers=tomcat1,tomcat2
>
> worker.status.type=status
>
> Hopefully one of you guys can give us a hint to fix that problem.
>
> Mit freundlichen Grüßen
> David Kumar
> Softwareentwickler, B. Sc.
> Abteilung Infotech - Interaktiv
> TELESTAR-DIGITAL GmbH
> Am Weiher 14
> D-56766 Ulmen
>
> http://www.telestar.de/
>
>
>
>
> -----Ursprüngliche Nachricht-----
> Von: David Kumar [mailto:dkumar@telestar.de]
> Gesendet: Dienstag, 22. Januar 2013 07:36
> An: Tomcat Users List
> Betreff: AW: AW: AW: ajp on 7009 and 9009 : connections keept open
>
> Hey,
>
> last friday I changed our configuration to use a executor.
> Here is what I did:
>
> <Connector port="7009" protocol="AJP/1.3" redirectPort="8443" maxThreads="200" executor="active-executor" />
> <Executor name="active-executor" namePrefix="activeThread-" maxThreads="200" minSpareThreads="30" maxIdleTime="60000" />
> <Connector port="7080" protocol="HTTP/1.1"
> connectionTimeout="20000"
> redirectPort="8443" executor="active-executor" />
>
> The second tomcat has same configuration besides ports..
>
> Until yesterday it worked like a charm. But at late afternoon one of the tomcats failed again..
>
> I couldn't start the garbagecollection so far..
>
> Any other ideas?
>
> Thanks
>
> Mit freundlichen Grüßen
> David Kumar
> Softwareentwickler, B. Sc.
> Abteilung Infotech - Interaktiv
> TELESTAR-DIGITAL GmbH
> Am Weiher 14
> D-56766 Ulmen
> http://www.telestar.de/
>
>
>
>
> -----Ursprüngliche Nachricht-----
> Von: David Kumar [mailto:dkumar@telestar.de]
> Gesendet: Freitag, 18. Januar 2013 11:19
> An: Tomcat Users List; Tomcat Users List
> Betreff: AW: AW: AW: ( ajp on 7009 and 9009 not afs3-rmtsys): connections keept open
>
> Hey,
>
> I do that at next deployment. --> I Thursday...
>
> So far I'm trying executor for tomcat. As far I read, when I'm using connectors idle process are forced to be close..
>
>
> Mit freundlichen Grüßen
> David Kumar
> Softwareentwickler, B. Sc.
> Abteilung Infotech - Interaktiv
> TELESTAR-DIGITAL GmbH
> Am Weiher 14
> D-56766 Ulmen
>
> http://www.telestar.de/
>
>
>
>
> -----Ursprüngliche Nachricht-----
> Von: André Warnier [mailto:aw@ice-sa.com]
> Gesendet: Freitag, 18. Januar 2013 11:10
> An: Tomcat Users List
> Betreff: Re: AW: AW: ( ajp on 7009 and 9009 not afs3-rmtsys): connections keept open
>
> David Kumar wrote:
>> Hey André,
>>
>> are you talking about running System.gc()?
> Yes.
>
>> That should be possible..
>>
>> Mit freundlichen Grüßen
>> David Kumar
>> Softwareentwickler, B. Sc.
>> Abteilung Infotech - Interaktiv
>> TELESTAR-DIGITAL GmbH
>> Am Weiher 14
>> D-56766 Ulmen
>>
>> http://www.telestar.de/
>>
>>
>>
>>
>> -----Ursprüngliche Nachricht-----
>> Von: André Warnier [mailto:aw@ice-sa.com]
>> Gesendet: Freitag, 18. Januar 2013 10:07
>> An: Tomcat Users List
>> Betreff: Re: AW: ( ajp on 7009 and 9009 not afs3-rmtsys): connections keept open
>>
>> David,
>> (and sorry for top-posting here)
>>
>> just to verify something.
>> Can you trigger a Major Garbage Collection at the Tomcat JVM level, at a moment when you
>> have all these connections in CLOSE_WAIT, and see if they disappear after the GC ?
>>
>> If yes, it may give a good clue about where all these CLOSE_WAITs are coming from.
>>
>>
>> David Kumar wrote:
>>> Just read this email.. :-)
>>>
>>> I figured out we are not using executor connector...
>>>
>>>
>>> Mit freundlichen Grüßen
>>> David Kumar
>>> Softwareentwickler, B. Sc.
>>> Abteilung Infotech - Interaktiv
>>> TELESTAR-DIGITAL GmbH
>>> Am Weiher 14
>>> D-56766 Ulmen
>>> http://www.telestar.de/
>>>
>>>
>>>
>>>
>>> -----Ursprüngliche Nachricht-----
>>> Von: David Kumar [mailto:dkumar@telestar.de]
>>> Gesendet: Freitag, 18. Januar 2013 09:11
>>> An: Tomcat Users List
>>> Betreff: AW: ( ajp on 7009 and 9009 not afs3-rmtsys): connections keept open
>>>
>>> here you are with attachment :-)
>>>
>>>
>>> btw: in mod_jk.log I found some
>>> [Thu Jan 17 23:00:08 2013] [11196:140336689317632] [error] ajp_get_reply::jk_ajp_common.c (2055): (tomcat2) Tomcat is down or refused connection. No response has been sent to the client (yet)
>>> [Thu Jan 17 23:00:08 2013] [11196:140336689317632] [error] ajp_service::jk_ajp_common.c (2559): (tomcat2) connecting to tomcat failed.
>>>
>>>
>>> but realy just a few one...
>>>
>>>
>>> Mit freundlichen Grüßen
>>> David Kumar
>>> Softwareentwickler, B. Sc.
>>> Abteilung Infotech - Interaktiv
>>> TELESTAR-DIGITAL GmbH
>>> Am Weiher 14
>>> D-56766 Ulmen
>>>
>>> http://www.telestar.de/
>>>
>>>
>>>
>>>
>>> -----Ursprüngliche Nachricht-----
>>> Von: David Kumar
>>> Gesendet: Freitag, 18. Januar 2013 09:08
>>> An: 'Tomcat Users List'
>>> Betreff: ( ajp on 7009 and 9009 not afs3-rmtsys): connections keept open
>>>
>>> Hey,
>>>
>>> thanks for reply. I got that about the Apache configuration. Since we had our problem yesterday, again and there was no error at the apache logs I'm willing to say that is not the main problem and I have to check that, when my main problem is solved.. :-)
>>>
>>>
>>>
>>> I agree with you about wrong reporting of service. Its just shown up as afs3 because these service uses 7009 per default. But I'm using 7009 and 9009 for ajp.
>>>
>>>
>>> So doesn't this mean there is a connection problems between my Apache and the tomcats?
>>>
>>> You're right, both Webapps doing the same and are configured identically besides the ports.
>>>
>>> I'm using more than one database, but all of them are used through a database pool. If there is a bug, I think I should have found some error at my logs like no free connection or something like that. As there is no such log entry I'm willing to say that my database connections processing like they should.
>>>
>>> Basically on each tomcat there are running two services. One is a axis2 project. Our CRM is posting customer data to this webapp. This data will be persisted into a database. Depending on the information given by our CRM axis sends a email.
>>>
>>> The second one is basically a cache for our websites. We have a PIM with all our product data. These app is gathering all the data from PIM and a CMS and is merging these information together so that the data can be displayed.
>>> All the mentioned data is hold in different "cache objects". Also some communication with our ERP and some databases are made trough this app.
>>>
>>> The second app is a REST service. Information will be posted as POST or GET request to it. Most likely the responses are JSON Object.
>>>
>>> When ever one webApp is reloading (automatically or manually) itself, the result will be posted to the other tomcat/webapp as a serialized object, so the other on do not need to reload it self.
>>>
>>> I can't say how many SMB files there are, it is depending on some other stuff so it is dynamic.
>>>
>>> Attached you can find a printed list by lsof.
>>>
>>> There you can see a really strange thing. Yesterday just tomcat2 had the problem with to many open files. A few days before it was just tomcat1 having this problem.
>>>
>>> Now let my answer your question:
>>>
>>> 1. That is hard to say, I guess I have to do some more investigation on our logfiles.
>>>
>>> 2. / 3. Here is my httpd.conf:
>>> <IfModule mpm_worker_module>
>>> ThreadLimit 25
>>> StartServers 2
>>> MaxClients 150
>>> MinSpareThreads 25
>>> MaxSpareThreads 75
>>> ThreadsPerChild 25
>>> MaxRequestsPerChild 4000
>>> </IfModule>
>>>
>>> we are using worker ....
>>>
>>> And here are our tomcat connectors again:
>>> tomcat1:
>>>
>>> <Connector port="7080" protocol="HTTP/1.1" connectionTimeout="20000" redirectPort="8443"/>
>>>
>>>
>>> <Connector port="7009" protocol="AJP/1.3" redirectPort="8443"/>
>>>
>>>
>>> tomcat2:
>>> <Connector port="9080" protocol="HTTP/1.1" connectionTimeout="20000" redirectPort="9443"/>
>>>
>>> <Connector port="9009" protocol="AJP/1.3" redirectPort="9443"/>
>>>
>>>
>>> Okay we are not using executor.. I will check that..
>>>
>>> You probably read my copy-paste error. I did copy some comments out of out server config --> Sry again.
>>>
>>> 4. we are using..
>>> 5. via a multipart message sending to the other tomcat.
>>> 6. I don't think so also because of that the connections are kept open on our ajp ports.
>>>
>>> I know that "CLOSE_WAIT" means, waiting for connections to be closed, but wondering that it is not closing..
>>>
>>>
>>> Thanks again
>>>
>>> Mit freundlichen Grüßen
>>> David Kumar
>>> Softwareentwickler, B. Sc.
>>> Abteilung Infotech - Interaktiv
>>> TELESTAR-DIGITAL GmbH
>>> Am Weiher 14
>>> D-56766 Ulmen
>>> http://www.telestar.de/
>>>
>>>
>>>
>>>
>>> -----Ursprüngliche Nachricht-----
>>> Von: Christopher Schultz [mailto:chris@christopherschultz.net]
>>> Gesendet: Donnerstag, 17. Januar 2013 18:38
>>> An: Tomcat Users List
>>> Betreff: Re: AW: AW: afs3-rmtsys: connections keept open
>>>
>>> -----BEGIN PGP SIGNED MESSAGE-----
>>> Hash: SHA256
>>>
>>> David,
>>>
>>> On 1/17/13 1:49 AM, David Kumar wrote:
>>>> I just checked /var/logs/apache2/error.logs. And found following
>>>> errors:
>>>>
>>>> [Wed Jan 16 15:14:46 2013] [error] server is within MinSpareThreads
>>>> of MaxClients, consider raising the MaxClients setting [Wed Jan 16
>>>> 15:14:56 2013] [error] server reached MaxClients setting, consider
>>>> raising the MaxClients setting
>>> So you are maxing-out your connections: you are experiencing enough
>>> load that your configuration cannot handle any more connections:
>>> requests are being queued by the TCP/IP stack and some requests may be
>>> rejected entirely depending upon the queue length of the socket.
>>>
>>> The first question to ask yourself is whether or not your hardware can
>>> take more than you have it configured to accept. For instance, if your
>>> load average, memory usage, and response time are all reasonable, then
>>> you could probably afford to raise your MaxClients setting in httpd.
>>>
>>> Note that the above has almost nothing to do with Tomcat: it only has
>>> to do with Apache httpd.
>>>
>>>> Yesterday my problem occurred about the same time.
>>> So, the problem is that Tomcat cannot handle your peak load due to a
>>> file handle limitation. IIRC, your current file handle limit for the
>>> Tomcat process is 4096.
>>>
>>>> I'm checking every five minutes how many open files there are:
>>>>
>>>> count open files started: 01-16-2013_15:10: Count: 775 count open
>>>> files started: 01-16-2013_15:15: Count: 1092
>>> Okay. lsof will help you determine how many of those are "real" files
>>> versus sockets. Limiting socket usage might be somewhat easier
>>> depending upon what your application actually does.
>>>
>>>> But maybe the afs3 connection causing the Apache error?
>>> afs3 is a red herring: you are using port 7009 for AJP communication
>>> between httpd and Tomcat and it's being reported as afs3. This has
>>> nothing to do with afs3 unless you know for a fact that your web
>>> application uses that protocol for something. I don't see any evidence
>>> that afs3 is related to your environment in the slightest. I do see
>>> every indication that you are using port 7009 yourself for AJP so
>>> let's assume that's the truth.
>>>
>>> Let's recap what your webapp(s) actually do to see if we can't figure
>>> out where all your file handles are being used. I'll assume that each
>>> Tomcat is configured (reasonably) identically, other than port numbers
>>> and such. I'll also assume that you are running the same webapp using
>>> the same (virtually) identical configuration and that nothing
>>> pathological is happening (like one process totally going crazy and
>>> making thousands of socket connections due to an application bug).
>>>
>>> First, all processes need access to stdin, stdout, stderr: that's 3
>>> file handles. Plus all shared libraries required to get the process
>>> and JVM started. Plus everything Java needs. Depending on the OS,
>>> that's about 30 or so to begin with. Then, Tomcat uses /dev/random (or
>>> /dev/urandom) plus it needs to load all of its own libraries from JAR
>>> files. There are about 25 of them, and they generally stay open. So,
>>> we're up to about 55 file handles. Don't worry: we won't be counting
>>> these things one-at-a-time for long. Next, Tomcat has two <Connector>s
>>> defined with default connection sizes. At peak load, they will both be
>>> maxed-out at 200 connections each for a total of 402 file handles (1
>>> bind file handle + 200 file handles for the connections * 2
>>> connectors). So, we're up to 457.
>>>
>>> Now, onto your web application. You have to count the number of JAR
>>> files that your web application provides: each one of those likely
>>> consumes another file handle that will stay open. Does your webapp use
>>> a database? If so, do you use a connection pool? How big is the
>>> connection pool? Do you have any leaks? If you use a connection pool
>>> and have no leaks, then you can add 'maxActive' file handles to our
>>> running count. If you don't use a connection pool, then you can add
>>> 400 file handles to your count, because any incoming request on either
>>> of those two connectors could result in a database connection. (I
>>> highly recommend using a connection pool if you aren't already).
>>>
>>> Next, you said this:
>>>
>>>> Both of the tomcats are "synchronising" them self. The send some
>>>> serialized objects via http to each other.
>>> So the webapps make requests to each other? How? Is there a limit to
>>> the number of connections directly from one Tomcat to another? If not,
>>> then you can add another 400 file handles because any incoming
>>> connection could trigger an HTTP connection to the other Tomcat. (What
>>> happens if an incoming client connection causes a connection to the
>>> other Tomcat... will that Tomcat ever call-back to the first one and
>>> set-up a communication storm?).
>>>
>>>> And both of them getting some file from SMB shares.
>>> How many files? Every file you open consumes a file handle. If you
>>> close the file, you can reduce your fd footprint, but if you keep lots
>>> of files open...
>>>
>>> If you have a dbcp with size=50 and you limit your cross-Tomcat
>>> connections to, say another 50 and your webapp uses 50 JAR files then
>>> you are looking at 600 or so file handles required to run your webapp
>>> under peak load, not including files that must be opened to satisfy a
>>> particular request.
>>>
>>> So the question is: where are all your fds going? Use lsof to
>>> determine what they are being used for.
>>>
>>> Some suggestions:
>>>
>>> 1. Consider the number of connections you actually need to be able to
>>> handle: for both connectors. Maybe you don't need 200 possible
>>> connections for your HTTP connector.
>>>
>>> 2. Make sure your MaxClients in httpd matches make sense with what
>>> you've got in Tomcat's AJP connector: you want to make sure that you
>>> have enough connections available from httpd->Tomcat that you aren't
>>> making users wait. If you're using prefork MPM that means that
>>> MaxClients should be the same as your <Connector>'s maxThreads setting
>>> (or, better yet, use an <Executor>).
>>>
>>> 3. Use an <Executor>. Right now, you might allocate up to 400 threads
>>> to handle connections from both AJP and HTTP. Maybe you don't need
>>> that. You can share request-processing threads by using an <Executor>
>>> and have both connectors share the same pool.
>>>
>>> 4. Use a DBCP. Just in case you aren't.
>>>
>>> 5. Check to see how you are communicating Tomcat-to-Tomcat: you may
>>> have a problem where too many connections are being opened.
>>>
>>> 6. Check to make sure you don't have any resource leaks: JDBC
>>> connections that aren't closed, files not being closed, etc. etc.
>>> Check to make sure you are closing files that don't need to be open
>>> after they are read.
>>>
>>>> But I can't imagine that might be the problem? I'm wondering why
>>>> the tcp connections with state "CLOSE_WAIT" doesn't get closed.
>>> http://en.wikipedia.org/wiki/Transmission_Control_Protocol
>>>
>>> - -chris
>>> -----BEGIN PGP SIGNATURE-----
>>> Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
>>> Comment: GPGTools - http://gpgtools.org
>>> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>>>
>>> iEYEAREIAAYFAlD4NvsACgkQ9CaO5/Lv0PC4EACfURhDENZPf28HDIazwPqAqri5
>>> KqYAni9AOSQZVIdsBtQLoEfDcYkpuf7f
>>> =dEDY
>>> -----END PGP SIGNATURE-----
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>>
>>>
>>> ------------------------------------------------------------------------
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections
keept open
Posted by André Warnier <aw...@ice-sa.com>.
David Kumar wrote:
> Hey,
>
> we are still having that issue.
> But we could manage to figure out some more stuff.
> We made a Tomcat and Java update since that time we had our problem a few times again, also we did some reconfiguration with connectors etc.
> The last 2 times we where able to see, that both tomcat by them self where alive. Just ajp on both where dead. We couldn't make a connection either trough 7009 nor 9009. An with our openFiles trick we found a lot of close_wait again, e.g. 200 for 9009. I left the second tomcat on this state for a few ours just to see, what happens. The count of 200 connection with close_wait was kept until a reboot of the tomcat.
Instead of rebooting Tomcat, try to force the Tomcat JVM to do a Major Garbage Collection.
There are a number of tools that allow to do that.
One command-line one which I found practical is jmxsh, here :
http://code.google.com/p/jmxsh/
If when you do a Major GC, these CLOSE_WAIT connections disappear, you will have learned
something about their origin.
And if then - without restarting Tomcat - you can connect again via the AJP ports, you'll
have learned something else.
Go do it and report.
> I would say with some of our reconfiguration we managed to stop increasing connections. But still we are not sure why our ajp connections dying..
>
> Here is our connector out of Server.xml:
>
> <Connector port="9009" protocol="AJP/1.3" redirectPort="9443" maxThreads="200" connectionTimeout="600000" />
>
>
> worker.properties:
>
> worker.tomcatX.host=localhost
> worker.tomcatX.type=ajp13
> worker.tomcatX.fail_on_status=404
> worker.tomcatX.lbfactor=1
> worker.tomcatX.ping_timeout=1000
> worker.tomcatX.ping_mode=A
> worker.tomcatX.socket_timeout=10
> worker.tomcatX.connection_pool_timeout=600
>
>
> worker.tomcat1.reference=worker.tomcatX
> worker.tomcat1.port=7009
>
> worker.tomcat2.reference=worker.tomcatX
> worker.tomcat2.port=9009
>
> worker.loadbalancer.type=lb
> worker.loadbalancer.balance_workers=tomcat1,tomcat2
>
> worker.status.type=status
>
> Hopefully one of you guys can give us a hint to fix that problem.
>
> Mit freundlichen Grüßen
> David Kumar
> Softwareentwickler, B. Sc.
> Abteilung Infotech - Interaktiv
> TELESTAR-DIGITAL GmbH
> Am Weiher 14
> D-56766 Ulmen
>
> http://www.telestar.de/
>
>
>
>
> -----Ursprüngliche Nachricht-----
> Von: David Kumar [mailto:dkumar@telestar.de]
> Gesendet: Dienstag, 22. Januar 2013 07:36
> An: Tomcat Users List
> Betreff: AW: AW: AW: ajp on 7009 and 9009 : connections keept open
>
> Hey,
>
> last friday I changed our configuration to use a executor.
> Here is what I did:
>
> <Connector port="7009" protocol="AJP/1.3" redirectPort="8443" maxThreads="200" executor="active-executor" />
> <Executor name="active-executor" namePrefix="activeThread-" maxThreads="200" minSpareThreads="30" maxIdleTime="60000" />
> <Connector port="7080" protocol="HTTP/1.1"
> connectionTimeout="20000"
> redirectPort="8443" executor="active-executor" />
>
> The second tomcat has same configuration besides ports..
>
> Until yesterday it worked like a charm. But at late afternoon one of the tomcats failed again..
>
> I couldn't start the garbagecollection so far..
>
> Any other ideas?
>
> Thanks
>
> Mit freundlichen Grüßen
> David Kumar
> Softwareentwickler, B. Sc.
> Abteilung Infotech - Interaktiv
> TELESTAR-DIGITAL GmbH
> Am Weiher 14
> D-56766 Ulmen
> http://www.telestar.de/
>
>
>
>
> -----Ursprüngliche Nachricht-----
> Von: David Kumar [mailto:dkumar@telestar.de]
> Gesendet: Freitag, 18. Januar 2013 11:19
> An: Tomcat Users List; Tomcat Users List
> Betreff: AW: AW: AW: ( ajp on 7009 and 9009 not afs3-rmtsys): connections keept open
>
> Hey,
>
> I do that at next deployment. --> I Thursday...
>
> So far I'm trying executor for tomcat. As far I read, when I'm using connectors idle process are forced to be close..
>
>
> Mit freundlichen Grüßen
> David Kumar
> Softwareentwickler, B. Sc.
> Abteilung Infotech - Interaktiv
> TELESTAR-DIGITAL GmbH
> Am Weiher 14
> D-56766 Ulmen
>
> http://www.telestar.de/
>
>
>
>
> -----Ursprüngliche Nachricht-----
> Von: André Warnier [mailto:aw@ice-sa.com]
> Gesendet: Freitag, 18. Januar 2013 11:10
> An: Tomcat Users List
> Betreff: Re: AW: AW: ( ajp on 7009 and 9009 not afs3-rmtsys): connections keept open
>
> David Kumar wrote:
>> Hey André,
>>
>> are you talking about running System.gc()?
> Yes.
>
>> That should be possible..
>>
>> Mit freundlichen Grüßen
>> David Kumar
>> Softwareentwickler, B. Sc.
>> Abteilung Infotech - Interaktiv
>> TELESTAR-DIGITAL GmbH
>> Am Weiher 14
>> D-56766 Ulmen
>>
>> http://www.telestar.de/
>>
>>
>>
>>
>> -----Ursprüngliche Nachricht-----
>> Von: André Warnier [mailto:aw@ice-sa.com]
>> Gesendet: Freitag, 18. Januar 2013 10:07
>> An: Tomcat Users List
>> Betreff: Re: AW: ( ajp on 7009 and 9009 not afs3-rmtsys): connections keept open
>>
>> David,
>> (and sorry for top-posting here)
>>
>> just to verify something.
>> Can you trigger a Major Garbage Collection at the Tomcat JVM level, at a moment when you
>> have all these connections in CLOSE_WAIT, and see if they disappear after the GC ?
>>
>> If yes, it may give a good clue about where all these CLOSE_WAITs are coming from.
>>
>>
>> David Kumar wrote:
>>> Just read this email.. :-)
>>>
>>> I figured out we are not using executor connector...
>>>
>>>
>>> Mit freundlichen Grüßen
>>> David Kumar
>>> Softwareentwickler, B. Sc.
>>> Abteilung Infotech - Interaktiv
>>> TELESTAR-DIGITAL GmbH
>>> Am Weiher 14
>>> D-56766 Ulmen
>>> http://www.telestar.de/
>>>
>>>
>>>
>>>
>>> -----Ursprüngliche Nachricht-----
>>> Von: David Kumar [mailto:dkumar@telestar.de]
>>> Gesendet: Freitag, 18. Januar 2013 09:11
>>> An: Tomcat Users List
>>> Betreff: AW: ( ajp on 7009 and 9009 not afs3-rmtsys): connections keept open
>>>
>>> here you are with attachment :-)
>>>
>>>
>>> btw: in mod_jk.log I found some
>>> [Thu Jan 17 23:00:08 2013] [11196:140336689317632] [error] ajp_get_reply::jk_ajp_common.c (2055): (tomcat2) Tomcat is down or refused connection. No response has been sent to the client (yet)
>>> [Thu Jan 17 23:00:08 2013] [11196:140336689317632] [error] ajp_service::jk_ajp_common.c (2559): (tomcat2) connecting to tomcat failed.
>>>
>>>
>>> but realy just a few one...
>>>
>>>
>>> Mit freundlichen Grüßen
>>> David Kumar
>>> Softwareentwickler, B. Sc.
>>> Abteilung Infotech - Interaktiv
>>> TELESTAR-DIGITAL GmbH
>>> Am Weiher 14
>>> D-56766 Ulmen
>>>
>>> http://www.telestar.de/
>>>
>>>
>>>
>>>
>>> -----Ursprüngliche Nachricht-----
>>> Von: David Kumar
>>> Gesendet: Freitag, 18. Januar 2013 09:08
>>> An: 'Tomcat Users List'
>>> Betreff: ( ajp on 7009 and 9009 not afs3-rmtsys): connections keept open
>>>
>>> Hey,
>>>
>>> thanks for reply. I got that about the Apache configuration. Since we had our problem yesterday, again and there was no error at the apache logs I'm willing to say that is not the main problem and I have to check that, when my main problem is solved.. :-)
>>>
>>>
>>>
>>> I agree with you about wrong reporting of service. Its just shown up as afs3 because these service uses 7009 per default. But I'm using 7009 and 9009 for ajp.
>>>
>>>
>>> So doesn't this mean there is a connection problems between my Apache and the tomcats?
>>>
>>> You're right, both Webapps doing the same and are configured identically besides the ports.
>>>
>>> I'm using more than one database, but all of them are used through a database pool. If there is a bug, I think I should have found some error at my logs like no free connection or something like that. As there is no such log entry I'm willing to say that my database connections processing like they should.
>>>
>>> Basically on each tomcat there are running two services. One is a axis2 project. Our CRM is posting customer data to this webapp. This data will be persisted into a database. Depending on the information given by our CRM axis sends a email.
>>>
>>> The second one is basically a cache for our websites. We have a PIM with all our product data. These app is gathering all the data from PIM and a CMS and is merging these information together so that the data can be displayed.
>>> All the mentioned data is hold in different "cache objects". Also some communication with our ERP and some databases are made trough this app.
>>>
>>> The second app is a REST service. Information will be posted as POST or GET request to it. Most likely the responses are JSON Object.
>>>
>>> When ever one webApp is reloading (automatically or manually) itself, the result will be posted to the other tomcat/webapp as a serialized object, so the other on do not need to reload it self.
>>>
>>> I can't say how many SMB files there are, it is depending on some other stuff so it is dynamic.
>>>
>>> Attached you can find a printed list by lsof.
>>>
>>> There you can see a really strange thing. Yesterday just tomcat2 had the problem with to many open files. A few days before it was just tomcat1 having this problem.
>>>
>>> Now let my answer your question:
>>>
>>> 1. That is hard to say, I guess I have to do some more investigation on our logfiles.
>>>
>>> 2. / 3. Here is my httpd.conf:
>>> <IfModule mpm_worker_module>
>>> ThreadLimit 25
>>> StartServers 2
>>> MaxClients 150
>>> MinSpareThreads 25
>>> MaxSpareThreads 75
>>> ThreadsPerChild 25
>>> MaxRequestsPerChild 4000
>>> </IfModule>
>>>
>>> we are using worker ....
>>>
>>> And here are our tomcat connectors again:
>>> tomcat1:
>>>
>>> <Connector port="7080" protocol="HTTP/1.1" connectionTimeout="20000" redirectPort="8443"/>
>>>
>>>
>>> <Connector port="7009" protocol="AJP/1.3" redirectPort="8443"/>
>>>
>>>
>>> tomcat2:
>>> <Connector port="9080" protocol="HTTP/1.1" connectionTimeout="20000" redirectPort="9443"/>
>>>
>>> <Connector port="9009" protocol="AJP/1.3" redirectPort="9443"/>
>>>
>>>
>>> Okay we are not using executor.. I will check that..
>>>
>>> You probably read my copy-paste error. I did copy some comments out of out server config --> Sry again.
>>>
>>> 4. we are using..
>>> 5. via a multipart message sending to the other tomcat.
>>> 6. I don't think so also because of that the connections are kept open on our ajp ports.
>>>
>>> I know that "CLOSE_WAIT" means, waiting for connections to be closed, but wondering that it is not closing..
>>>
>>>
>>> Thanks again
>>>
>>> Mit freundlichen Grüßen
>>> David Kumar
>>> Softwareentwickler, B. Sc.
>>> Abteilung Infotech - Interaktiv
>>> TELESTAR-DIGITAL GmbH
>>> Am Weiher 14
>>> D-56766 Ulmen
>>> http://www.telestar.de/
>>>
>>>
>>>
>>>
>>> -----Ursprüngliche Nachricht-----
>>> Von: Christopher Schultz [mailto:chris@christopherschultz.net]
>>> Gesendet: Donnerstag, 17. Januar 2013 18:38
>>> An: Tomcat Users List
>>> Betreff: Re: AW: AW: afs3-rmtsys: connections keept open
>>>
>>> -----BEGIN PGP SIGNED MESSAGE-----
>>> Hash: SHA256
>>>
>>> David,
>>>
>>> On 1/17/13 1:49 AM, David Kumar wrote:
>>>> I just checked /var/logs/apache2/error.logs. And found following
>>>> errors:
>>>>
>>>> [Wed Jan 16 15:14:46 2013] [error] server is within MinSpareThreads
>>>> of MaxClients, consider raising the MaxClients setting [Wed Jan 16
>>>> 15:14:56 2013] [error] server reached MaxClients setting, consider
>>>> raising the MaxClients setting
>>> So you are maxing-out your connections: you are experiencing enough
>>> load that your configuration cannot handle any more connections:
>>> requests are being queued by the TCP/IP stack and some requests may be
>>> rejected entirely depending upon the queue length of the socket.
>>>
>>> The first question to ask yourself is whether or not your hardware can
>>> take more than you have it configured to accept. For instance, if your
>>> load average, memory usage, and response time are all reasonable, then
>>> you could probably afford to raise your MaxClients setting in httpd.
>>>
>>> Note that the above has almost nothing to do with Tomcat: it only has
>>> to do with Apache httpd.
>>>
>>>> Yesterday my problem occurred about the same time.
>>> So, the problem is that Tomcat cannot handle your peak load due to a
>>> file handle limitation. IIRC, your current file handle limit for the
>>> Tomcat process is 4096.
>>>
>>>> I'm checking every five minutes how many open files there are:
>>>>
>>>> count open files started: 01-16-2013_15:10: Count: 775 count open
>>>> files started: 01-16-2013_15:15: Count: 1092
>>> Okay. lsof will help you determine how many of those are "real" files
>>> versus sockets. Limiting socket usage might be somewhat easier
>>> depending upon what your application actually does.
>>>
>>>> But maybe the afs3 connection causing the Apache error?
>>> afs3 is a red herring: you are using port 7009 for AJP communication
>>> between httpd and Tomcat and it's being reported as afs3. This has
>>> nothing to do with afs3 unless you know for a fact that your web
>>> application uses that protocol for something. I don't see any evidence
>>> that afs3 is related to your environment in the slightest. I do see
>>> every indication that you are using port 7009 yourself for AJP so
>>> let's assume that's the truth.
>>>
>>> Let's recap what your webapp(s) actually do to see if we can't figure
>>> out where all your file handles are being used. I'll assume that each
>>> Tomcat is configured (reasonably) identically, other than port numbers
>>> and such. I'll also assume that you are running the same webapp using
>>> the same (virtually) identical configuration and that nothing
>>> pathological is happening (like one process totally going crazy and
>>> making thousands of socket connections due to an application bug).
>>>
>>> First, all processes need access to stdin, stdout, stderr: that's 3
>>> file handles. Plus all shared libraries required to get the process
>>> and JVM started. Plus everything Java needs. Depending on the OS,
>>> that's about 30 or so to begin with. Then, Tomcat uses /dev/random (or
>>> /dev/urandom) plus it needs to load all of its own libraries from JAR
>>> files. There are about 25 of them, and they generally stay open. So,
>>> we're up to about 55 file handles. Don't worry: we won't be counting
>>> these things one-at-a-time for long. Next, Tomcat has two <Connector>s
>>> defined with default connection sizes. At peak load, they will both be
>>> maxed-out at 200 connections each for a total of 402 file handles (1
>>> bind file handle + 200 file handles for the connections * 2
>>> connectors). So, we're up to 457.
>>>
>>> Now, onto your web application. You have to count the number of JAR
>>> files that your web application provides: each one of those likely
>>> consumes another file handle that will stay open. Does your webapp use
>>> a database? If so, do you use a connection pool? How big is the
>>> connection pool? Do you have any leaks? If you use a connection pool
>>> and have no leaks, then you can add 'maxActive' file handles to our
>>> running count. If you don't use a connection pool, then you can add
>>> 400 file handles to your count, because any incoming request on either
>>> of those two connectors could result in a database connection. (I
>>> highly recommend using a connection pool if you aren't already).
>>>
>>> Next, you said this:
>>>
>>>> Both of the tomcats are "synchronising" them self. The send some
>>>> serialized objects via http to each other.
>>> So the webapps make requests to each other? How? Is there a limit to
>>> the number of connections directly from one Tomcat to another? If not,
>>> then you can add another 400 file handles because any incoming
>>> connection could trigger an HTTP connection to the other Tomcat. (What
>>> happens if an incoming client connection causes a connection to the
>>> other Tomcat... will that Tomcat ever call-back to the first one and
>>> set-up a communication storm?).
>>>
>>>> And both of them getting some file from SMB shares.
>>> How many files? Every file you open consumes a file handle. If you
>>> close the file, you can reduce your fd footprint, but if you keep lots
>>> of files open...
>>>
>>> If you have a dbcp with size=50 and you limit your cross-Tomcat
>>> connections to, say another 50 and your webapp uses 50 JAR files then
>>> you are looking at 600 or so file handles required to run your webapp
>>> under peak load, not including files that must be opened to satisfy a
>>> particular request.
>>>
>>> So the question is: where are all your fds going? Use lsof to
>>> determine what they are being used for.
>>>
>>> Some suggestions:
>>>
>>> 1. Consider the number of connections you actually need to be able to
>>> handle: for both connectors. Maybe you don't need 200 possible
>>> connections for your HTTP connector.
>>>
>>> 2. Make sure your MaxClients in httpd matches make sense with what
>>> you've got in Tomcat's AJP connector: you want to make sure that you
>>> have enough connections available from httpd->Tomcat that you aren't
>>> making users wait. If you're using prefork MPM that means that
>>> MaxClients should be the same as your <Connector>'s maxThreads setting
>>> (or, better yet, use an <Executor>).
>>>
>>> 3. Use an <Executor>. Right now, you might allocate up to 400 threads
>>> to handle connections from both AJP and HTTP. Maybe you don't need
>>> that. You can share request-processing threads by using an <Executor>
>>> and have both connectors share the same pool.
>>>
>>> 4. Use a DBCP. Just in case you aren't.
>>>
>>> 5. Check to see how you are communicating Tomcat-to-Tomcat: you may
>>> have a problem where too many connections are being opened.
>>>
>>> 6. Check to make sure you don't have any resource leaks: JDBC
>>> connections that aren't closed, files not being closed, etc. etc.
>>> Check to make sure you are closing files that don't need to be open
>>> after they are read.
>>>
>>>> But I can't imagine that might be the problem? I'm wondering why
>>>> the tcp connections with state "CLOSE_WAIT" doesn't get closed.
>>> http://en.wikipedia.org/wiki/Transmission_Control_Protocol
>>>
>>> - -chris
>>> -----BEGIN PGP SIGNATURE-----
>>> Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
>>> Comment: GPGTools - http://gpgtools.org
>>> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>>>
>>> iEYEAREIAAYFAlD4NvsACgkQ9CaO5/Lv0PC4EACfURhDENZPf28HDIazwPqAqri5
>>> KqYAni9AOSQZVIdsBtQLoEfDcYkpuf7f
>>> =dEDY
>>> -----END PGP SIGNATURE-----
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>>
>>>
>>> ------------------------------------------------------------------------
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org