You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Isaac Gonzalez <ig...@autoreturn.com> on 2014/02/24 20:27:57 UTC
tomcat 6 refuses mod_jk connections after server runs for a couple
of days
Hello all,
I'm running tomcat 6.0.32 on Cent OS 6 with 2 front end apache load balancers with a firewall in between the tomcat and load balancers using mod_jk v. 1.2.37 under apache 2.2.10 to connect the backend tomcat. I have had this running ok for a few years but our user traffic has increased significantly.
A few months ago, the tomcat server seemed to refuse or not accept any new connections from either load balancer and required a restart on the tomcat end, even though I could easily connect to tomcat on port 8080(manager). I can intermittently telnet to port 8009, but am denied a bit as well both inside and outside the firewall.
I proceeded to split the tomcats up into their own instances, hoping when this issue recurred that it would only affect a particular tomcat app. It also gave our developers the ability to patch a single tomcat app without downing all of our apps.
Unfortunately, this issue has recurred several times and I have spent most of my days researching and digging for hope of someone with a similar experience that may have resolved it. Last Friday the problem was so bad, I had to completely restart the tomcat server(reboot it).
So far I am at a loss...I have installed psi-probe on all tomcat instances to give me more in depth analysis to tomcat threads and related server metadata when the problem is occuring. I have made a few modifications to workers.properties, in particular to decrease the connection timeout as well as the tomcat ajp connector from 10 minutes to 5 minutes and added the ping timeout and socket timeout. I also increased my apache prefork MPM client connections to 500 on each load balancer. Below is my relevant configs...any suggestions to help remedy this would help... I have also increased threads from 200 to 500 on all tomcat instances.
Workers.properties:
worker.list=jkstatus,server1,server2,server3,server4,server5,server6,server7,server8
worker.jkstatus.type=status
# Let's define some defaults
worker.basic.port=8009
worker.basic.type=ajp13
worker.basic.socket_keepalive=True
worker.basic.connection_pool_timeout=300
worker.basic.ping_timeout=1000
worker.basic.ping_mode=A
worker.basic.socket_timeout=10
worker.lb1.distance=0
worker.lb1.reference=worker.basic
worker.server1.host= server1hostname
worker.server1.reference=worker.lb1
worker.server2.host=server2hostname
worker.server2.reference=worker.lb1
worker.server3.host=server3hostname
worker.server3.reference=worker.lb1
worker.server4.host= server4hostname
worker.server4.reference=worker.lb1
worker.server5.host= server5hostname
worker.server5.reference=worker.lb1
worker.server6.host= server6hostname
worker.server6.reference=worker.lb1
worker.server7.host= server7hostname
worker.server7.reference=worker.lb1
worker.server8.host= server7hostname
worker.server8.reference=worker.lb1
httpd.conf:
KeepAlive Off
MaxKeepAliveRequests 100
KeepAliveTimeout 15
# prefork MPM
# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# ServerLimit: maximum value for MaxClients for the lifetime of the server
# MaxClients: maximum number of server processes allowed to start
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule prefork.c>
StartServers 8
MinSpareServers 5
MaxSpareServers 20
ServerLimit 500
MaxClients 500
MaxRequestsPerChild 5000
</IfModule>
Tomcat server.xml:
<!-- Define an AJP 1.3 Connector on port 8009 -->
<Connector port="8009" address="x.x.x.x" protocol="AJP/1.3" redirectPort="8443" connectionTimeout="300000" maxThreads="500" />
RE: tomcat 6 refuses mod_jk connections after server runs for a
couple of days
Posted by Isaac Gonzalez <ig...@autoreturn.com>.
Hi Konstantin,
I can try to upgrade to tomcat 6.0.39 or tomcat 7...It should be a simple enough upgrade....would possibly help out a bit.
I have the worker.basic.socket_keepalive=True set so according to the tomcat connector documentation, this should help with the firewall dropping open connections. When I have this problem and check the AJP threads in PSI-Probe for each tomcat instance, for example, its always well under the maximum threads. Perhaps I should be looking at AJP sessions instead?...I'm not sure there is a way to maximize or minimize this.
-Isaac
-----Original Message-----
From: Konstantin Kolinko [mailto:knst.kolinko@gmail.com]
Sent: Tuesday, February 25, 2014 3:33 PM
To: Tomcat Users List
Subject: Re: tomcat 6 refuses mod_jk connections after server runs for a couple of days
2014-02-24 23:27 GMT+04:00 Isaac Gonzalez <ig...@autoreturn.com>:
> Hello all,
>
> I'm running tomcat 6.0.32
Can you upgrade to 6.0.39 or 7.0.52?
> on Cent OS 6 with 2 front end apache load balancers with a firewall in
> between the tomcat and load balancers
A firewall between Apache HTTPD Server and Apache Tomcat?
Sometimes a firewall may drop a TCP connection without properly terminating is. So your Tomcat might still think that it has 500 AJP connections open and refuse new ones.
There have been several discussions on such issues over the years.
An old thread,
http://marc.info/?t=121818607600002&r=1&w=2
Best regards,
Konstantin Kolinko
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: tomcat 6 refuses mod_jk connections after server runs for a
couple of days
Posted by Konstantin Kolinko <kn...@gmail.com>.
2014-02-24 23:27 GMT+04:00 Isaac Gonzalez <ig...@autoreturn.com>:
> Hello all,
>
> I'm running tomcat 6.0.32
Can you upgrade to 6.0.39 or 7.0.52?
> on Cent OS 6 with 2 front end apache load balancers with a firewall in between the tomcat and load balancers
A firewall between Apache HTTPD Server and Apache Tomcat?
Sometimes a firewall may drop a TCP connection without properly
terminating is. So your Tomcat might still think that it has 500 AJP
connections open and refuse new ones.
There have been several discussions on such issues over the years.
An old thread,
http://marc.info/?t=121818607600002&r=1&w=2
Best regards,
Konstantin Kolinko
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: tomcat 6 refuses mod_jk connections after server runs for a couple
of days
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Issac,
On 3/1/14, 12:41 AM, Isaac Gonzalez wrote:
> ________________________________________ From: Christopher Schultz
> [chris@christopherschultz.net] Sent: Friday, February 28, 2014
> 11:40 AM To: Tomcat Users List Subject: Re: tomcat 6 refuses mod_jk
> connections after server runs for a couple of days
>
>>> pipe size (512 bytes, -p) 8 POSIX message queues
>>> (bytes, -q) 819200 real-time priority (-r) 0
>>> stack size (kbytes, -s) 10240 cpu time (seconds,
>>> -t) unlimited max user processes (-u) 1024
>>
>> You might want to increase this number. How many processes is
>> "tomcat" running outside of the JVM? This is likely to be the
>> limit you are hitting.
>
> Tomcat is only running about 7 processes total, one for each
> JVM...but nothing else...unless I need to look beyond ps... Don't
> think this is it...but you never know....
Some *NIXs count individual threads as processes. You'll have to check
in your own environment.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBCAAGBQJTFKTgAAoJEBzwKT+lPKRYhK0P/0XBtTjcmIFE+tRIsR+SxTN2
T5wpkzqmjO7o1Q2+BRQXJyWIFB3EshlKjB4e3P6RkiqB9yWL5PV6N+7LqZGlJZSn
AMWQDUIkszG7wP6E1lTMNm3IJRba0PGJlhsYb4BtTuxQyKJlqMNdzKw3OyETghbl
ibOAPRHZZK9apQKSQKaO7CGOUv0+VAGTwwf1tcXv7UQVOqpryApuCWIrVNNMyrOD
wiXioTj6qhbcl7i+x+qW2Nqk4ldkNGVKgS+3wfAjzUbFuXWfPMKRG7YCAltySP8W
tZ+4SFAQ0GYTP0M2yoEC8+m8kXTOlDWHNeLPvhlU3NhFOCl25W9u1GHJyvFvjzKP
IV+nglZ47qZryQYWYeiZOquM25hjuvCGT+r5o0enrca3VFxw7TCdYyDOq/CHLQjU
+MaY9yXEuRYbXphdmWj0hEWyY+TirTEdumjFB1tVKrY87jr7biq2gyCtRC/mey8t
xLLi3nYhHEqVDCPCv8pt+5nQg6AgQRqv3veBt8Z86p21JuuBcPoRYJy6aI0KpIms
11Xxe9M05M8W4b7wY+4U6vrxYVsEx6GVPEA8ZTsARTpwDjYeHRSQ5Tp9nRf9NrAf
vCYtws9OkOfy7UOkEGQt4OLpuGHdld4rCEQfNwC+REuqW68r7+hE/bpqZwFFalP4
rf60cH024jR32N8gJvqn
=ZKmN
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: tomcat 6 refuses mod_jk connections after server runs for a
couple of days
Posted by Isaac Gonzalez <ig...@autoreturn.com>.
Christopher,
________________________________________
From: Christopher Schultz [chris@christopherschultz.net]
Sent: Friday, February 28, 2014 11:40 AM
To: Tomcat Users List
Subject: Re: tomcat 6 refuses mod_jk connections after server runs for a couple of days
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Issac,
On 2/27/14, 6:23 PM, Isaac Gonzalez wrote:
>
> -----Original Message----- From: Konstantin Kolinko
> [mailto:knst.kolinko@gmail.com] Sent: Thursday, February 27, 2014
> 2:40 PM To: Tomcat Users List Subject: Re: tomcat 6 refuses mod_jk
> connections after server runs for a couple of days
>
> 2014-02-28 2:06 GMT+04:00 Isaac Gonzalez
> <ig...@autoreturn.com>:
>> Hi Christopher(and Konstantin), attached is a couple of thread
>> dumps of when we experienced the issue again today. I also
>> noticed we get this message right before the problem occurs: Feb
>> 27, 2014 12:47:15 PM
>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable run
>> SEVERE: Caught exception (java.lang.OutOfMemoryError: unable to
>> create new native thread) executing
>> org.apache.jk.common.ChannelSocket$SocketAcceptor@177ddea,
>> terminating thread
>
> That explains why a connection cannot be accepted.
>
> I wonder are you hitting an "ulimit" limit, or there is just not
> enough free memory to allocate stack area for a new thread (which
> size is set by -Xss option to java executable).
>
> I thought of the ulimit settings and increased it to the upward
> limit allowed at the end of last weekend: [root@server ~]# su
> tomcat [tomcat@server root]$ ulimit -a core file size
> (blocks, -c) 0 data seg size (kbytes, -d) unlimited
> scheduling priority (-e) 0 file size
> (blocks, -f) unlimited pending signals (-i) 62835
> max locked memory (kbytes, -l) 64 max memory size
> (kbytes, -m) unlimited open files (-n) 65535
Open-files can sometimes be a problem. This setting looks just fine to
me, though.
> pipe size (512 bytes, -p) 8 POSIX message queues
> (bytes, -q) 819200 real-time priority (-r) 0 stack
> size (kbytes, -s) 10240 cpu time
> (seconds, -t) unlimited max user processes (-u) 1024
You might want to increase this number. How many processes is "tomcat"
running outside of the JVM? This is likely to be the limit you are
hitting.
Tomcat is only running about 7 processes total, one for each JVM...but nothing else...unless I need to look beyond ps...
Don't think this is it...but you never know....
-Isaac
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBCAAGBQJTEOY2AAoJEBzwKT+lPKRYmAwP+wVF9fwj88SOXxWTEKXdbk01
a9slFh4LmnDn0tacrAmA3m47VsndF3ewHwIEF1yP7wNRYf3NXSCzVp0OWuji6ZU1
yfjPpBT3pI8dVPqu9hkWVxkvQxA5xL0sm+9L2BeVBW0QbLs69L0g/v3xt2LwMxvF
j/9mZNqW6A177ZG1o5wcdexRhzV2566Z3idWdc8Zp9uISwFdZXdzYxJtTiku9k6q
nV3gQ8ICAwI+VGBKc1DwbL6QqUwpY8O7OjmQ5OEJaqHYEXjVNkdgo87oY+2BXRkV
9BW9J1zHLPAi8UhdfumDeqRKBQ7JPRhRLGGrhAHsmmA+G0XzShzU2zY84s5PSGU8
GwNiNZ/NJpTPtYjV5viY3GdWWbyeO9J4VDUBsBbs8k1XN7a44OjmKpRhnVIlQT6z
XLYfg3GpWjK8Xdd2L81RB/O6Q2xn9jY5FMik8jh0HgDm38Wf4AeymhVdEaEfVT5Z
TdAQECOFeYGDgLHNY7sFr/QQfJkLAFhfNM9xcgDTx4WcUH9V4QMn8S2qjOeFPbgx
hqwg+p2au18JMTb+RkmnHAVIcqtiFtUU/dN9Xap/vH+bc8UKimE87brBlnTnD/pk
uW0ea5m4f6MDcX2hDSh4+1ZU5uI0ZqTMvcp445UE0GW/4ITu9iauVedvM9fUlVlt
fzDyTTTEUHZ10n+yF5XC
=kOiz
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: tomcat 6 refuses mod_jk connections after server runs for a couple
of days
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Issac,
On 2/27/14, 6:23 PM, Isaac Gonzalez wrote:
>
> -----Original Message----- From: Konstantin Kolinko
> [mailto:knst.kolinko@gmail.com] Sent: Thursday, February 27, 2014
> 2:40 PM To: Tomcat Users List Subject: Re: tomcat 6 refuses mod_jk
> connections after server runs for a couple of days
>
> 2014-02-28 2:06 GMT+04:00 Isaac Gonzalez
> <ig...@autoreturn.com>:
>> Hi Christopher(and Konstantin), attached is a couple of thread
>> dumps of when we experienced the issue again today. I also
>> noticed we get this message right before the problem occurs: Feb
>> 27, 2014 12:47:15 PM
>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable run
>> SEVERE: Caught exception (java.lang.OutOfMemoryError: unable to
>> create new native thread) executing
>> org.apache.jk.common.ChannelSocket$SocketAcceptor@177ddea,
>> terminating thread
>
> That explains why a connection cannot be accepted.
>
> I wonder are you hitting an "ulimit" limit, or there is just not
> enough free memory to allocate stack area for a new thread (which
> size is set by -Xss option to java executable).
>
> I thought of the ulimit settings and increased it to the upward
> limit allowed at the end of last weekend: [root@server ~]# su
> tomcat [tomcat@server root]$ ulimit -a core file size
> (blocks, -c) 0 data seg size (kbytes, -d) unlimited
> scheduling priority (-e) 0 file size
> (blocks, -f) unlimited pending signals (-i) 62835
> max locked memory (kbytes, -l) 64 max memory size
> (kbytes, -m) unlimited open files (-n) 65535
Open-files can sometimes be a problem. This setting looks just fine to
me, though.
> pipe size (512 bytes, -p) 8 POSIX message queues
> (bytes, -q) 819200 real-time priority (-r) 0 stack
> size (kbytes, -s) 10240 cpu time
> (seconds, -t) unlimited max user processes (-u) 1024
You might want to increase this number. How many processes is "tomcat"
running outside of the JVM? This is likely to be the limit you are
hitting.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBCAAGBQJTEOY2AAoJEBzwKT+lPKRYmAwP+wVF9fwj88SOXxWTEKXdbk01
a9slFh4LmnDn0tacrAmA3m47VsndF3ewHwIEF1yP7wNRYf3NXSCzVp0OWuji6ZU1
yfjPpBT3pI8dVPqu9hkWVxkvQxA5xL0sm+9L2BeVBW0QbLs69L0g/v3xt2LwMxvF
j/9mZNqW6A177ZG1o5wcdexRhzV2566Z3idWdc8Zp9uISwFdZXdzYxJtTiku9k6q
nV3gQ8ICAwI+VGBKc1DwbL6QqUwpY8O7OjmQ5OEJaqHYEXjVNkdgo87oY+2BXRkV
9BW9J1zHLPAi8UhdfumDeqRKBQ7JPRhRLGGrhAHsmmA+G0XzShzU2zY84s5PSGU8
GwNiNZ/NJpTPtYjV5viY3GdWWbyeO9J4VDUBsBbs8k1XN7a44OjmKpRhnVIlQT6z
XLYfg3GpWjK8Xdd2L81RB/O6Q2xn9jY5FMik8jh0HgDm38Wf4AeymhVdEaEfVT5Z
TdAQECOFeYGDgLHNY7sFr/QQfJkLAFhfNM9xcgDTx4WcUH9V4QMn8S2qjOeFPbgx
hqwg+p2au18JMTb+RkmnHAVIcqtiFtUU/dN9Xap/vH+bc8UKimE87brBlnTnD/pk
uW0ea5m4f6MDcX2hDSh4+1ZU5uI0ZqTMvcp445UE0GW/4ITu9iauVedvM9fUlVlt
fzDyTTTEUHZ10n+yF5XC
=kOiz
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: tomcat 6 refuses mod_jk connections after server runs for a
couple of days
Posted by Isaac Gonzalez <ig...@autoreturn.com>.
-----Original Message-----
From: Konstantin Kolinko [mailto:knst.kolinko@gmail.com]
Sent: Thursday, February 27, 2014 2:40 PM
To: Tomcat Users List
Subject: Re: tomcat 6 refuses mod_jk connections after server runs for a couple of days
2014-02-28 2:06 GMT+04:00 Isaac Gonzalez <ig...@autoreturn.com>:
> Hi Christopher(and Konstantin), attached is a couple of thread dumps of when we experienced the issue again today. I also noticed we get this message right before the problem occurs:
> Feb 27, 2014 12:47:15 PM
> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable run
> SEVERE: Caught exception (java.lang.OutOfMemoryError: unable to create
> new native thread) executing
> org.apache.jk.common.ChannelSocket$SocketAcceptor@177ddea, terminating
> thread
That explains why a connection cannot be accepted.
I wonder are you hitting an "ulimit" limit, or there is just not enough free memory to allocate stack area for a new thread (which size is set by -Xss option to java executable).
I thought of the ulimit settings and increased it to the upward limit allowed at the end of last weekend:
[root@server ~]# su tomcat
[tomcat@server root]$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 62835
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 65535
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 1024
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
Here is my options I am passing to Java in my init script....(I'm not using the -Xss option)
-Dbuild.compiler.emacs=true -XX:MaxPermSize=256M -Xmx2048m -Duser.timezone=America/Los_Angeles -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8089 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Xdebug -Xrunjdwp:transport=dt_socket,address=8003,server=y,suspend=n
Your thread dumps contain 149 threads each.
So I am not maxing out threads...seems like each tomcat node is running out of memory at the same time....
Best regards,
Konstantin Kolinko
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: tomcat 6 refuses mod_jk connections after server runs for a
couple of days
Posted by Isaac Gonzalez <ig...@autoreturn.com>.
Christopher
________________________________________
From: Christopher Schultz [chris@christopherschultz.net]
Sent: Friday, February 28, 2014 11:38 AM
To: Tomcat Users List
Subject: Re: tomcat 6 refuses mod_jk connections after server runs for a couple of days
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Konstantin,
On 2/27/14, 5:40 PM, Konstantin Kolinko wrote:
> 2014-02-28 2:06 GMT+04:00 Isaac Gonzalez
> <ig...@autoreturn.com>:
>> Hi Christopher(and Konstantin), attached is a couple of thread
>> dumps of when we experienced the issue again today. I also
>> noticed we get this message right before the problem occurs: Feb
>> 27, 2014 12:47:15 PM
>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable run
>> SEVERE: Caught exception (java.lang.OutOfMemoryError: unable to
>> create new native thread) executing
>> org.apache.jk.common.ChannelSocket$SocketAcceptor@177ddea,
>> terminating thread
>
> That explains why a connection cannot be accepted.
>
> I wonder are you hitting an "ulimit" limit, or there is just not
> enough free memory to allocate stack area for a new thread (which
> size is set by -Xss option to java executable).
>
> Your thread dumps contain 149 threads each.
While it does explain why one (Tomcat) server would become
unresponsive, it doesn't really explain why the entire cluster would
become unresponsive.
Issac, are you using sticky-sessions or anything like that, or does
your load-balancing mod_jk configuration choose arbitrarily between a
backend server? You initially gave an abridged configuration, so I
can't tell.
As you indicate below, I am not clustering. There is only one backend tomcat.
>> After the "split", did both Tomcats appear to lock-up
>> simultaneously, or did only one of them have a problem and the
>> other one stayed up?
>
> Isaac: They all appear to lock-up simultaneously, if users try to
> access that JK mount point.
>
> [...]
>
> Isaac: I am not load-balancing the tomcat servers...I only have
> one...I do "load balance" the apache front end servers via dns
> round-robin....
Oh, you only have a single back-end server? Well, then that why they
"all" go down at once, so you seem to have found your problem: the
server itself is going down because you don't have enough resources to
keep it up.
Indeed I have! Seems like I underallocated server memory...the machine had only 8 Gigs with 7 tomcat instances running that all had up to a maximum of 2 gigs of maximum memory heap size, plus OS stuff running, RabbitMQ, and other things. I am wondering though if something else could be the underlying root cause of this issue, or was I simply under allocating memory..such as connections not being closed, either by the client mod_jk connector, or the db connector...We'll see in the next few days I guess....
thanks again Chris, you and Konstantin pointed me to the issue...
-Isaac
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBCAAGBQJTEOWeAAoJEBzwKT+lPKRYUxYQAItxQZTOWjubWA32fDwGhcGy
Y4cuBDU7OZiw/I4oQTCEY0Uk62eKf6g0dXQL4VqM8j4jz2VLhP10uPDA3c/+CBhU
kpVJg2ifLD0YJmUqnxuTHYIdwwvPrPa/PNCAWYJCcDXE1DVEz8HLAoYlQY5oJ5Cf
P+wtYTgWqJyzddtB2sjB+YQcVj+83aWkfKuipednSdm0utCPP5PQzPiF7agoP9qt
vDB0preG0GFQXTShYqMRKeEt3hu+BdLXugp7kJA5KDMEcSbWyPzefxWl0CtKhcJB
d/ntEtoYbR0gWGO4Qajio6NVmw9TWzBf4spbg8scBz8ijE314VNsw6mdT9F55TZZ
43iYSnDAK1dNfs7guqAAk7z5Gf+fChy28zFmOm0lSzs1/o5HHvJFqKse94hzjJW6
R4uCUVktbvoJPfot6zoG3ofsYm+PVcibPOj4Xh0m7nBPKvqTZ5BVyeLBRR/E+KRy
O4jJ0DshRnhy3qL9l5uO6h7miIb+GMwjpc3A6lbbITHVDKspaq8kll+m6sAn1ppV
Z8PnNysSMTGHY6azjMisZlp4b/i9r+Nc+HabtbjRrt1StfuWrHTeIwQ46n7XcMrr
biATXUpo94kM2eGJecP0jtyBrgYwkaz22NtbW3i2l47XQKa/dhhP6IzlgzK8Fmki
eKiBf5+iNyhc9dB3adQ2
=Uchz
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: tomcat 6 refuses mod_jk connections after server runs for a couple
of days
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Konstantin,
On 2/27/14, 5:40 PM, Konstantin Kolinko wrote:
> 2014-02-28 2:06 GMT+04:00 Isaac Gonzalez
> <ig...@autoreturn.com>:
>> Hi Christopher(and Konstantin), attached is a couple of thread
>> dumps of when we experienced the issue again today. I also
>> noticed we get this message right before the problem occurs: Feb
>> 27, 2014 12:47:15 PM
>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable run
>> SEVERE: Caught exception (java.lang.OutOfMemoryError: unable to
>> create new native thread) executing
>> org.apache.jk.common.ChannelSocket$SocketAcceptor@177ddea,
>> terminating thread
>
> That explains why a connection cannot be accepted.
>
> I wonder are you hitting an "ulimit" limit, or there is just not
> enough free memory to allocate stack area for a new thread (which
> size is set by -Xss option to java executable).
>
> Your thread dumps contain 149 threads each.
While it does explain why one (Tomcat) server would become
unresponsive, it doesn't really explain why the entire cluster would
become unresponsive.
Issac, are you using sticky-sessions or anything like that, or does
your load-balancing mod_jk configuration choose arbitrarily between a
backend server? You initially gave an abridged configuration, so I
can't tell.
>> After the "split", did both Tomcats appear to lock-up
>> simultaneously, or did only one of them have a problem and the
>> other one stayed up?
>
> Isaac: They all appear to lock-up simultaneously, if users try to
> access that JK mount point.
>
> [...]
>
> Isaac: I am not load-balancing the tomcat servers...I only have
> one...I do "load balance" the apache front end servers via dns
> round-robin....
Oh, you only have a single back-end server? Well, then that why they
"all" go down at once, so you seem to have found your problem: the
server itself is going down because you don't have enough resources to
keep it up.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBCAAGBQJTEOWeAAoJEBzwKT+lPKRYUxYQAItxQZTOWjubWA32fDwGhcGy
Y4cuBDU7OZiw/I4oQTCEY0Uk62eKf6g0dXQL4VqM8j4jz2VLhP10uPDA3c/+CBhU
kpVJg2ifLD0YJmUqnxuTHYIdwwvPrPa/PNCAWYJCcDXE1DVEz8HLAoYlQY5oJ5Cf
P+wtYTgWqJyzddtB2sjB+YQcVj+83aWkfKuipednSdm0utCPP5PQzPiF7agoP9qt
vDB0preG0GFQXTShYqMRKeEt3hu+BdLXugp7kJA5KDMEcSbWyPzefxWl0CtKhcJB
d/ntEtoYbR0gWGO4Qajio6NVmw9TWzBf4spbg8scBz8ijE314VNsw6mdT9F55TZZ
43iYSnDAK1dNfs7guqAAk7z5Gf+fChy28zFmOm0lSzs1/o5HHvJFqKse94hzjJW6
R4uCUVktbvoJPfot6zoG3ofsYm+PVcibPOj4Xh0m7nBPKvqTZ5BVyeLBRR/E+KRy
O4jJ0DshRnhy3qL9l5uO6h7miIb+GMwjpc3A6lbbITHVDKspaq8kll+m6sAn1ppV
Z8PnNysSMTGHY6azjMisZlp4b/i9r+Nc+HabtbjRrt1StfuWrHTeIwQ46n7XcMrr
biATXUpo94kM2eGJecP0jtyBrgYwkaz22NtbW3i2l47XQKa/dhhP6IzlgzK8Fmki
eKiBf5+iNyhc9dB3adQ2
=Uchz
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: tomcat 6 refuses mod_jk connections after server runs for a
couple of days
Posted by Konstantin Kolinko <kn...@gmail.com>.
2014-02-28 2:06 GMT+04:00 Isaac Gonzalez <ig...@autoreturn.com>:
> Hi Christopher(and Konstantin), attached is a couple of thread dumps of when we experienced the issue again today. I also noticed we get this message right before the problem occurs:
> Feb 27, 2014 12:47:15 PM org.apache.tomcat.util.threads.ThreadPool$ControlRunnable run
> SEVERE: Caught exception (java.lang.OutOfMemoryError: unable to create new native thread) executing org.apache.jk.common.ChannelSocket$SocketAcceptor@177ddea, terminating thread
That explains why a connection cannot be accepted.
I wonder are you hitting an "ulimit" limit,
or there is just not enough free memory to allocate stack area for a
new thread (which size is set by -Xss option to java executable).
Your thread dumps contain 149 threads each.
Best regards,
Konstantin Kolinko
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: tomcat 6 refuses mod_jk connections after server runs for a couple
of days
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Isaac,
On 3/11/14, 6:56 PM, Isaac Gonzalez wrote:
>
>
> -----Original Message----- From: Christopher Schultz
> [mailto:chris@christopherschultz.net] Sent: Friday, March 07, 2014
> 8:18 AM To: Tomcat Users List Subject: Re: tomcat 6 refuses mod_jk
> connections after server runs for a couple of days
>
> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
>
>
>
> On 3/6/14, 7:39 AM, Daniel Mikusa wrote:
>> On Mar 5, 2014, at 4:51 PM, Isaac Gonzalez
>> <ig...@autoreturn.com> wrote:
>>
>>>
>>>
>>> -----Original Message----- From: Daniel Mikusa
>>> [mailto:dmikusa@gopivotal.com] Sent: Tuesday, March 04, 2014
>>> 12:42 PM To: Tomcat Users List Subject: Re: tomcat 6 refuses
>>> mod_jk connections after server runs for a couple of days
>>>
>>> On Mar 4, 2014, at 1:55 PM, Isaac Gonzalez
>>> <ig...@autoreturn.com> wrote:
>>>
>>>> Dan,
>>>>
>>>> ________________________________________ From: Daniel Mikusa
>>>> [dmikusa@gopivotal.com] Sent: Tuesday, March 04, 2014 6:20
>>>> AM To: Tomcat Users List Subject: Re: tomcat 6 refuses mod_jk
>>>> connections after server runs for a couple of days
>>>>
>>>> On Mar 4, 2014, at 6:32 AM, Rainer Jung
>>>> <ra...@kippdata.de> wrote:
>>>>
>>>>> On 27.02.2014 23:06, Isaac Gonzalez wrote:
>>>>>> Hi Christopher(and Konstantin), attached is a couple of
>>>>>> thread dumps of when we experienced the issue again
>>>>>> today. I also noticed we get this message right before
>>>>>> the problem occurs: Feb 27, 2014 12:47:15 PM
>>>>>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable
>>>>>>
>>>>>>
run SEVERE: Caught exception (java.lang.OutOfMemoryError:
>>>>>> unable to create new native thread) executing
>>>>>> org.apache.jk.common.ChannelSocket$SocketAcceptor@177ddea,
>>>>>>
>>>>>>
terminating thread
>>>>>
>>>>> Is it a 32Bit system? You have 2GB of heap plus Perm plus
>>>>> native memory needed by the process plus thread stacks. Not
>>>>> unlikely, that you ran out of memory address space for a 32
>>>>> bit process.
>>>>>
>>>>> The only fixes would then be:
>>>>>
>>>>> - switch to a 64 bit system
>>>>>
>>>>> - reduce heap if the app can work with less
>>>>>
>>>>> - improve performance or eliminate bottlenecks so that the
>>>>> app works with less threads
>>>>>
>>>>> - limit you connector thread pool size. That will still
>>>>> mean that if requests begin to queue because of performance
>>>>> problems, the web server can't create additional
>>>>> connections, but you won't get in an irregular situation as
>>>>> you experience now. In that case you would need to
>>>>> configure a low idle timeout for the connections on the JK
>>>>> and TC side.
>>>>
>>>> It may also be possible to lower the thread stack size with
>>>> the -Xss option.
>>>>
>>>> Ok so we are 64 bit Linux with 1024k in the 64-bit
>>>> VM....would lowering it to 64k be a bit too low? What sort of
>>>> repercussions would we run into? Very helpful information by
>>>> the way.
>>>
>>> It depends on your apps, so you'll need to test and see. If
>>> you go too low, you'll get StackOverflow exceptions. If you
>>> see those, just gradually increase until they go away.
>>>
>>> Dan
>>>
>>>
>>>>
>>>> -Isaac
>>>>
>>>>
>>>> http://www.oracle.com/technetwork/java/hotspotfaq-138619.html#thread
>>>>
>>>>
s_
>>>>
>>>>
> oom
>>>>
>>>> Might buy you some room for a few additional threads.
>>>>
>>>> Dan
>>>>
>>>>>
>>>>> Regards,
>>>>>
>>>>> Rainer
>>>>>
>>>>>
>>>>> -------------------------------------------------------------------
>>>>>
>>>>>
- --
>>>>>
>>>>>
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>> For additional commands, e-mail:
>>>>> users-help@tomcat.apache.org
>>>>>
>>>>
>>>>
>>>> --------------------------------------------------------------------
>>>>
>>>>
- -
>>>>
>>>>
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>> For additional commands, e-mail:
>>>> users-help@tomcat.apache.org
>>>> --------------------------------------------------------------------
>>>>
>>>>
- -
>>>>
>>>>
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>> For additional commands, e-mail:
>>>> users-help@tomcat.apache.org
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>>
>>>
>
>>>
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>>
>>> Ok so the problem just happened again just now. Dan, Can you
>>> elaborate on how to configure limiting the connector thread
>>> pool size. I am also going to lower the thread stack size as
>>> you recommended. It seems like this problem creeps up when we
>>> have a hiccup in connectivity at our data center. Perhaps I
>>> also need to lower the idle timeout some more between tomcat
>>> and mod_jk. They are also between a firewall by the way, so I
>>> can configure a timeout between the two there as well. We
>>> aren't experiencing too many idle disconnects there.
>>
>> See maxConnections / maxThreads on the Connector tag.
>>
>> http://tomcat.apache.org/tomcat-7.0-doc/config/http.html#Standard_Impl
>>
>>
ementation
>>
>> or Executor if you’re using an executor.
>>
>> http://tomcat.apache.org/tomcat-7.0-doc/config/executor.html
>
>> ... and you definitely *should* be using a manually-configured
>> Executor.
>
>> - -chris
>
> Chris, why should I be using a connector since we are only having
> users use the single 8009 AJP connection on each tomcat instance? I
> am the only one that uses the 8080 connector for troubleshooting
> and monitoring purposes. Is it mainly to help recycle unused
> threads?
Yes, you can better-manage all the thread-based resources using an
<Executor> rather than having one (or two) created for you
automatically by the connector.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBCAAGBQJTIKA/AAoJEBzwKT+lPKRYSHgQALvACFo5ayaSkX/d8EuXwBUl
wMd9L0XBUOpc+gnM+nERqa29/uTmZisWCifo0VlI7p6rbmFQOAZSl7eqFHpT1eZF
DZgeahpLCRIABTmXwKfzExXue7OEu+Ry4JPhmPM/6aO0vZK+pFQGyl/v6V8b+Hfo
kGsaYlNFxTiuYqyOWLKoWMbUNjhwAVlz/hqZkO+QpuVf1DtrI9Srnc2OBcC6pNSf
xiP/aybrMYRH2rZWkU1qafAqo3lVBus3T32HBGwPJ1MSvHDoMNluBMMrzeSxECFl
Pv0tgdAzStLCE+UPwyeSMkZlotC/fIz+l2e0V0fCxYBZZuWIWMOkyaw5qUGnrI/f
Ur7PQM7El77yHKn+0MrTC7kXlRISdO7GQRDn0lTUqLDpLjTGF/qdS4gJLYR+fqHS
n5rLXnuOzEThCpFKYV8wBm9ePtGqk65jl59828yQ9x3QcchYU9byzIfoGI8KQf/O
SG8OdVV1G0z1axnnukhnqUl65teA6ST9h1qrrli/CP8NRO5PNQ4ftHwVhWkxVRPH
wfis6YSCnqNPBG8JwsHYWaRpNqeDnHRXaGdKOGd/GEhZrxNwd3UNzA81NDQI1yoE
hZfvaOi0xUFOQeX+PVhhPvFeGD8HqNlh02GTPKkfBZSeG6mENM/cEtFbG69mfmoG
IskslmrAlsqtm3igHHoX
=EWbM
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: tomcat 6 refuses mod_jk connections after server runs for a
couple of days
Posted by Isaac Gonzalez <ig...@autoreturn.com>.
-----Original Message-----
From: Christopher Schultz [mailto:chris@christopherschultz.net]
Sent: Friday, March 07, 2014 8:18 AM
To: Tomcat Users List
Subject: Re: tomcat 6 refuses mod_jk connections after server runs for a couple of days
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
On 3/6/14, 7:39 AM, Daniel Mikusa wrote:
> On Mar 5, 2014, at 4:51 PM, Isaac Gonzalez <ig...@autoreturn.com>
> wrote:
>
>>
>>
>> -----Original Message----- From: Daniel Mikusa
>> [mailto:dmikusa@gopivotal.com] Sent: Tuesday, March 04, 2014
>> 12:42 PM To: Tomcat Users List Subject: Re: tomcat 6 refuses mod_jk
>> connections after server runs for a couple of days
>>
>> On Mar 4, 2014, at 1:55 PM, Isaac Gonzalez <ig...@autoreturn.com>
>> wrote:
>>
>>> Dan,
>>>
>>> ________________________________________ From: Daniel Mikusa
>>> [dmikusa@gopivotal.com] Sent: Tuesday, March 04, 2014 6:20 AM
>>> To: Tomcat Users List Subject: Re: tomcat 6 refuses mod_jk
>>> connections after server runs for a couple of days
>>>
>>> On Mar 4, 2014, at 6:32 AM, Rainer Jung <ra...@kippdata.de>
>>> wrote:
>>>
>>>> On 27.02.2014 23:06, Isaac Gonzalez wrote:
>>>>> Hi Christopher(and Konstantin), attached is a couple of thread
>>>>> dumps of when we experienced the issue again today.
>>>>> I also noticed we get this message right before the problem
>>>>> occurs: Feb 27, 2014 12:47:15 PM
>>>>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable
>>>>> run SEVERE: Caught exception (java.lang.OutOfMemoryError:
>>>>> unable to create new native thread) executing
>>>>> org.apache.jk.common.ChannelSocket$SocketAcceptor@177ddea,
>>>>> terminating thread
>>>>
>>>> Is it a 32Bit system? You have 2GB of heap plus Perm plus native
>>>> memory needed by the process plus thread stacks. Not unlikely, that
>>>> you ran out of memory address space for a 32 bit process.
>>>>
>>>> The only fixes would then be:
>>>>
>>>> - switch to a 64 bit system
>>>>
>>>> - reduce heap if the app can work with less
>>>>
>>>> - improve performance or eliminate bottlenecks so that the app
>>>> works with less threads
>>>>
>>>> - limit you connector thread pool size. That will still mean that
>>>> if requests begin to queue because of performance problems, the web
>>>> server can't create additional connections, but you won't get in an
>>>> irregular situation as you experience now. In that case you would
>>>> need to configure a low idle timeout for the connections on the JK
>>>> and TC side.
>>>
>>> It may also be possible to lower the thread stack size with the -Xss
>>> option.
>>>
>>> Ok so we are 64 bit Linux with 1024k in the 64-bit VM....would
>>> lowering it to 64k be a bit too low? What sort of repercussions
>>> would we run into? Very helpful information by the way.
>>
>> It depends on your apps, so you'll need to test and see. If you go
>> too low, you'll get StackOverflow exceptions. If you see those, just
>> gradually increase until they go away.
>>
>> Dan
>>
>>
>>>
>>> -Isaac
>>>
>>>
>>> http://www.oracle.com/technetwork/java/hotspotfaq-138619.html#thread
>>> s_
>>>
>>>
oom
>>>
>>> Might buy you some room for a few additional threads.
>>>
>>> Dan
>>>
>>>>
>>>> Regards,
>>>>
>>>> Rainer
>>>>
>>>>
>>>> -------------------------------------------------------------------
>>>> --
>>>>
>>>>
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>> For additional commands, e-mail:
>>>> users-help@tomcat.apache.org
>>>>
>>>
>>>
>>> --------------------------------------------------------------------
>>> -
>>>
>>>
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>> --------------------------------------------------------------------
>>> -
>>>
>>>
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>
>>
>> ---------------------------------------------------------------------
>>
>>
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>> Ok so the problem just happened again just now. Dan, Can you
>> elaborate on how to configure limiting the connector thread pool
>> size. I am also going to lower the thread stack size as you
>> recommended. It seems like this problem creeps up when we have a
>> hiccup in connectivity at our data center. Perhaps I also need to
>> lower the idle timeout some more between tomcat and mod_jk. They are
>> also between a firewall by the way, so I can configure a timeout
>> between the two there as well. We aren't experiencing too many idle
>> disconnects there.
>
> See maxConnections / maxThreads on the Connector tag.
>
> http://tomcat.apache.org/tomcat-7.0-doc/config/http.html#Standard_Impl
> ementation
>
> or Executor if you’re using an executor.
>
> http://tomcat.apache.org/tomcat-7.0-doc/config/executor.html
>... and you definitely *should* be using a manually-configured Executor.
>- -chris
Chris, why should I be using a connector since we are only having users use the single 8009 AJP connection on each tomcat instance? I am the only one that uses the 8080 connector for troubleshooting and monitoring purposes. Is it mainly to help recycle unused threads?
-Isaac
RE: tomcat 6 refuses mod_jk connections after server runs for a
couple of days
Posted by Isaac Gonzalez <ig...@autoreturn.com>.
-----Original Message-----
From: Christopher Schultz [mailto:chris@christopherschultz.net]
Sent: Friday, March 07, 2014 8:18 AM
To: Tomcat Users List
Subject: Re: tomcat 6 refuses mod_jk connections after server runs for a couple of days
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
On 3/6/14, 7:39 AM, Daniel Mikusa wrote:
> On Mar 5, 2014, at 4:51 PM, Isaac Gonzalez <ig...@autoreturn.com>
> wrote:
>
>>
>>
>> -----Original Message----- From: Daniel Mikusa
>> [mailto:dmikusa@gopivotal.com] Sent: Tuesday, March 04, 2014
>> 12:42 PM To: Tomcat Users List Subject: Re: tomcat 6 refuses mod_jk
>> connections after server runs for a couple of days
>>
>> On Mar 4, 2014, at 1:55 PM, Isaac Gonzalez <ig...@autoreturn.com>
>> wrote:
>>
>>> Dan,
>>>
>>> ________________________________________ From: Daniel Mikusa
>>> [dmikusa@gopivotal.com] Sent: Tuesday, March 04, 2014 6:20 AM
>>> To: Tomcat Users List Subject: Re: tomcat 6 refuses mod_jk
>>> connections after server runs for a couple of days
>>>
>>> On Mar 4, 2014, at 6:32 AM, Rainer Jung <ra...@kippdata.de>
>>> wrote:
>>>
>>>> On 27.02.2014 23:06, Isaac Gonzalez wrote:
>>>>> Hi Christopher(and Konstantin), attached is a couple of thread
>>>>> dumps of when we experienced the issue again today.
>>>>> I also noticed we get this message right before the problem
>>>>> occurs: Feb 27, 2014 12:47:15 PM
>>>>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable
>>>>> run SEVERE: Caught exception (java.lang.OutOfMemoryError:
>>>>> unable to create new native thread) executing
>>>>> org.apache.jk.common.ChannelSocket$SocketAcceptor@177ddea,
>>>>> terminating thread
>>>>
>>>> Is it a 32Bit system? You have 2GB of heap plus Perm plus native
>>>> memory needed by the process plus thread stacks. Not unlikely, that
>>>> you ran out of memory address space for a 32 bit process.
>>>>
>>>> The only fixes would then be:
>>>>
>>>> - switch to a 64 bit system
>>>>
>>>> - reduce heap if the app can work with less
>>>>
>>>> - improve performance or eliminate bottlenecks so that the app
>>>> works with less threads
>>>>
>>>> - limit you connector thread pool size. That will still mean that
>>>> if requests begin to queue because of performance problems, the web
>>>> server can't create additional connections, but you won't get in an
>>>> irregular situation as you experience now. In that case you would
>>>> need to configure a low idle timeout for the connections on the JK
>>>> and TC side.
>>>
>>> It may also be possible to lower the thread stack size with the -Xss
>>> option.
>>>
>>> Ok so we are 64 bit Linux with 1024k in the 64-bit VM....would
>>> lowering it to 64k be a bit too low? What sort of repercussions
>>> would we run into? Very helpful information by the way.
>>
>> It depends on your apps, so you'll need to test and see. If you go
>> too low, you'll get StackOverflow exceptions. If you see those, just
>> gradually increase until they go away.
>>
>> Dan
>>
>>
>>>
>>> -Isaac
>>>
>>>
>>> http://www.oracle.com/technetwork/java/hotspotfaq-138619.html#thread
>>> s_
>>>
>>>
oom
>>>
>>> Might buy you some room for a few additional threads.
>>>
>>> Dan
>>>
>>>>
>>>> Regards,
>>>>
>>>> Rainer
>>>>
>>>>
>>>> -------------------------------------------------------------------
>>>> --
>>>>
>>>>
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>> For additional commands, e-mail:
>>>> users-help@tomcat.apache.org
>>>>
>>>
>>>
>>> --------------------------------------------------------------------
>>> -
>>>
>>>
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>> --------------------------------------------------------------------
>>> -
>>>
>>>
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>
>>
>> ---------------------------------------------------------------------
>>
>>
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>> Ok so the problem just happened again just now. Dan, Can you
>> elaborate on how to configure limiting the connector thread pool
>> size. I am also going to lower the thread stack size as you
>> recommended. It seems like this problem creeps up when we have a
>> hiccup in connectivity at our data center. Perhaps I also need to
>> lower the idle timeout some more between tomcat and mod_jk. They are
>> also between a firewall by the way, so I can configure a timeout
>> between the two there as well. We aren't experiencing too many idle
>> disconnects there.
>
> See maxConnections / maxThreads on the Connector tag.
>
> http://tomcat.apache.org/tomcat-7.0-doc/config/http.html#Standard_Impl
> ementation
>
> or Executor if you’re using an executor.
>
> http://tomcat.apache.org/tomcat-7.0-doc/config/executor.html
>... and you definitely *should* be using a manually-configured Executor.
>- -chris
Ok so move to a manually configured executor and set the thread stack size to be smaller than the 1024k default sounds like it should definitely help out a bit.
-Isaac
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: tomcat 6 refuses mod_jk connections after server runs for a couple
of days
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
On 3/6/14, 7:39 AM, Daniel Mikusa wrote:
> On Mar 5, 2014, at 4:51 PM, Isaac Gonzalez
> <ig...@autoreturn.com> wrote:
>
>>
>>
>> -----Original Message----- From: Daniel Mikusa
>> [mailto:dmikusa@gopivotal.com] Sent: Tuesday, March 04, 2014
>> 12:42 PM To: Tomcat Users List Subject: Re: tomcat 6 refuses
>> mod_jk connections after server runs for a couple of days
>>
>> On Mar 4, 2014, at 1:55 PM, Isaac Gonzalez
>> <ig...@autoreturn.com> wrote:
>>
>>> Dan,
>>>
>>> ________________________________________ From: Daniel Mikusa
>>> [dmikusa@gopivotal.com] Sent: Tuesday, March 04, 2014 6:20 AM
>>> To: Tomcat Users List Subject: Re: tomcat 6 refuses mod_jk
>>> connections after server runs for a couple of days
>>>
>>> On Mar 4, 2014, at 6:32 AM, Rainer Jung
>>> <ra...@kippdata.de> wrote:
>>>
>>>> On 27.02.2014 23:06, Isaac Gonzalez wrote:
>>>>> Hi Christopher(and Konstantin), attached is a couple of
>>>>> thread dumps of when we experienced the issue again today.
>>>>> I also noticed we get this message right before the problem
>>>>> occurs: Feb 27, 2014 12:47:15 PM
>>>>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable
>>>>> run SEVERE: Caught exception (java.lang.OutOfMemoryError:
>>>>> unable to create new native thread) executing
>>>>> org.apache.jk.common.ChannelSocket$SocketAcceptor@177ddea,
>>>>> terminating thread
>>>>
>>>> Is it a 32Bit system? You have 2GB of heap plus Perm plus
>>>> native memory needed by the process plus thread stacks. Not
>>>> unlikely, that you ran out of memory address space for a 32
>>>> bit process.
>>>>
>>>> The only fixes would then be:
>>>>
>>>> - switch to a 64 bit system
>>>>
>>>> - reduce heap if the app can work with less
>>>>
>>>> - improve performance or eliminate bottlenecks so that the
>>>> app works with less threads
>>>>
>>>> - limit you connector thread pool size. That will still mean
>>>> that if requests begin to queue because of performance
>>>> problems, the web server can't create additional connections,
>>>> but you won't get in an irregular situation as you experience
>>>> now. In that case you would need to configure a low idle
>>>> timeout for the connections on the JK and TC side.
>>>
>>> It may also be possible to lower the thread stack size with the
>>> -Xss option.
>>>
>>> Ok so we are 64 bit Linux with 1024k in the 64-bit VM....would
>>> lowering it to 64k be a bit too low? What sort of repercussions
>>> would we run into? Very helpful information by the way.
>>
>> It depends on your apps, so you'll need to test and see. If you
>> go too low, you'll get StackOverflow exceptions. If you see
>> those, just gradually increase until they go away.
>>
>> Dan
>>
>>
>>>
>>> -Isaac
>>>
>>>
>>> http://www.oracle.com/technetwork/java/hotspotfaq-138619.html#threads_
>>>
>>>
oom
>>>
>>> Might buy you some room for a few additional threads.
>>>
>>> Dan
>>>
>>>>
>>>> Regards,
>>>>
>>>> Rainer
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>>
>>>>
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>> For additional commands, e-mail:
>>>> users-help@tomcat.apache.org
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>>
>>>
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>> ---------------------------------------------------------------------
>>>
>>>
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>
>>
>> ---------------------------------------------------------------------
>>
>>
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>> Ok so the problem just happened again just now. Dan, Can you
>> elaborate on how to configure limiting the connector thread pool
>> size. I am also going to lower the thread stack size as you
>> recommended. It seems like this problem creeps up when we have a
>> hiccup in connectivity at our data center. Perhaps I also need to
>> lower the idle timeout some more between tomcat and mod_jk. They
>> are also between a firewall by the way, so I can configure a
>> timeout between the two there as well. We aren't experiencing too
>> many idle disconnects there.
>
> See maxConnections / maxThreads on the Connector tag.
>
> http://tomcat.apache.org/tomcat-7.0-doc/config/http.html#Standard_Implementation
>
> or Executor if you’re using an executor.
>
> http://tomcat.apache.org/tomcat-7.0-doc/config/executor.html
... and you definitely *should* be using a manually-configured Executor.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBCAAGBQJTGfE9AAoJEBzwKT+lPKRYz8AQAJZj3JaI2wsYt62N7s6L+DHB
En9FT1J3o96rvrrqhiLT7Et5R3o6kR++eeajVgHprbLLMYnDVH+k4+rvKDoW5JUp
IBGeti5hukvojAEtBjVQdGBB0zE3v632E9mUHepyelC5Y3v5hKnQ7hLMYrJSbgmk
+V1Dustvg3BQ4Dgzrn28OaDMrWd/9Lt2zDZAtxaGKH7DKkkkIvCQ6KWq7KGLd+4H
fUx013uE5Pdd1VlqjicTXGLP1WtjiafFXjK4TQtwBiNhocCXFIBhCa/fHGJOOPv2
NRhM0UBkJ3JKUGLBP8XX4YOsl367gyTmdL1DoHT7XH7XZeefRAYKd2Py314wdAYR
MXpDVtfOAszt2Ezae+9mXzOmyxBGMYm8pCsDK5BVF1yFEgfASRDMhjcyOdzb9vzL
E5Op4BVgZ26KohQjyOujd7sX/mKp7eOA3JB7fGNzfM/VF4kw94atriKp+Sgbysez
29BGOeIpOdHrdavgEbnZJt5BNQqdSNVJsGVQWI3PThAEY80dXYbFknsUpjzeS/Rc
V4cqH9OWk7MTH90PBj4ywz0UXPVf+hCpYXma75vM6SbS2eHJCQwD42rqIwaSGUVM
0Wq80J60OmWPw0JZdFfw8evKQrYUegCkJTtsome3gKomh5ed0Lunl6Ow6ALfXf2U
s7bowgPr+Xt131YB9iPL
=iUDj
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: tomcat 6 refuses mod_jk connections after server runs for a couple of days
Posted by Daniel Mikusa <dm...@gopivotal.com>.
On Mar 5, 2014, at 4:51 PM, Isaac Gonzalez <ig...@autoreturn.com> wrote:
>
>
> -----Original Message-----
> From: Daniel Mikusa [mailto:dmikusa@gopivotal.com]
> Sent: Tuesday, March 04, 2014 12:42 PM
> To: Tomcat Users List
> Subject: Re: tomcat 6 refuses mod_jk connections after server runs for a couple of days
>
> On Mar 4, 2014, at 1:55 PM, Isaac Gonzalez <ig...@autoreturn.com> wrote:
>
>> Dan,
>>
>> ________________________________________
>> From: Daniel Mikusa [dmikusa@gopivotal.com]
>> Sent: Tuesday, March 04, 2014 6:20 AM
>> To: Tomcat Users List
>> Subject: Re: tomcat 6 refuses mod_jk connections after server runs for
>> a couple of days
>>
>> On Mar 4, 2014, at 6:32 AM, Rainer Jung <ra...@kippdata.de> wrote:
>>
>>> On 27.02.2014 23:06, Isaac Gonzalez wrote:
>>>> Hi Christopher(and Konstantin), attached is a couple of thread dumps of when we experienced the issue again today. I also noticed we get this message right before the problem occurs:
>>>> Feb 27, 2014 12:47:15 PM
>>>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable run
>>>> SEVERE: Caught exception (java.lang.OutOfMemoryError: unable to
>>>> create new native thread) executing
>>>> org.apache.jk.common.ChannelSocket$SocketAcceptor@177ddea,
>>>> terminating thread
>>>
>>> Is it a 32Bit system? You have 2GB of heap plus Perm plus native
>>> memory needed by the process plus thread stacks. Not unlikely, that
>>> you ran out of memory address space for a 32 bit process.
>>>
>>> The only fixes would then be:
>>>
>>> - switch to a 64 bit system
>>>
>>> - reduce heap if the app can work with less
>>>
>>> - improve performance or eliminate bottlenecks so that the app works
>>> with less threads
>>>
>>> - limit you connector thread pool size. That will still mean that if
>>> requests begin to queue because of performance problems, the web
>>> server can't create additional connections, but you won't get in an
>>> irregular situation as you experience now. In that case you would
>>> need to configure a low idle timeout for the connections on the JK and TC side.
>>
>> It may also be possible to lower the thread stack size with the -Xss option.
>>
>> Ok so we are 64 bit Linux with 1024k in the 64-bit VM....would lowering it to 64k be a bit too low? What sort of repercussions would we run into?
>> Very helpful information by the way.
>
> It depends on your apps, so you'll need to test and see. If you go too low, you'll get StackOverflow exceptions. If you see those, just gradually increase until they go away.
>
> Dan
>
>
>>
>> -Isaac
>>
>>
>> http://www.oracle.com/technetwork/java/hotspotfaq-138619.html#threads_
>> oom
>>
>> Might buy you some room for a few additional threads.
>>
>> Dan
>>
>>>
>>> Regards,
>>>
>>> Rainer
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
> Ok so the problem just happened again just now. Dan, Can you elaborate on how to configure limiting the connector thread pool size. I am also going to lower the thread stack size as you recommended. It seems like this problem creeps up when we have a hiccup in connectivity at our data center. Perhaps I also need to lower the idle timeout some more between tomcat and mod_jk. They are also between a firewall by the way, so I can configure a timeout between the two there as well. We aren't experiencing too many idle disconnects there.
See maxConnections / maxThreads on the Connector tag.
http://tomcat.apache.org/tomcat-7.0-doc/config/http.html#Standard_Implementation
or Executor if you’re using an executor.
http://tomcat.apache.org/tomcat-7.0-doc/config/executor.html
Dan
>
> -Isaac
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: tomcat 6 refuses mod_jk connections after server runs for a
couple of days
Posted by Isaac Gonzalez <ig...@autoreturn.com>.
-----Original Message-----
From: Daniel Mikusa [mailto:dmikusa@gopivotal.com]
Sent: Tuesday, March 04, 2014 12:42 PM
To: Tomcat Users List
Subject: Re: tomcat 6 refuses mod_jk connections after server runs for a couple of days
On Mar 4, 2014, at 1:55 PM, Isaac Gonzalez <ig...@autoreturn.com> wrote:
> Dan,
>
> ________________________________________
> From: Daniel Mikusa [dmikusa@gopivotal.com]
> Sent: Tuesday, March 04, 2014 6:20 AM
> To: Tomcat Users List
> Subject: Re: tomcat 6 refuses mod_jk connections after server runs for
> a couple of days
>
> On Mar 4, 2014, at 6:32 AM, Rainer Jung <ra...@kippdata.de> wrote:
>
>> On 27.02.2014 23:06, Isaac Gonzalez wrote:
>>> Hi Christopher(and Konstantin), attached is a couple of thread dumps of when we experienced the issue again today. I also noticed we get this message right before the problem occurs:
>>> Feb 27, 2014 12:47:15 PM
>>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable run
>>> SEVERE: Caught exception (java.lang.OutOfMemoryError: unable to
>>> create new native thread) executing
>>> org.apache.jk.common.ChannelSocket$SocketAcceptor@177ddea,
>>> terminating thread
>>
>> Is it a 32Bit system? You have 2GB of heap plus Perm plus native
>> memory needed by the process plus thread stacks. Not unlikely, that
>> you ran out of memory address space for a 32 bit process.
>>
>> The only fixes would then be:
>>
>> - switch to a 64 bit system
>>
>> - reduce heap if the app can work with less
>>
>> - improve performance or eliminate bottlenecks so that the app works
>> with less threads
>>
>> - limit you connector thread pool size. That will still mean that if
>> requests begin to queue because of performance problems, the web
>> server can't create additional connections, but you won't get in an
>> irregular situation as you experience now. In that case you would
>> need to configure a low idle timeout for the connections on the JK and TC side.
>
> It may also be possible to lower the thread stack size with the -Xss option.
>
> Ok so we are 64 bit Linux with 1024k in the 64-bit VM....would lowering it to 64k be a bit too low? What sort of repercussions would we run into?
> Very helpful information by the way.
It depends on your apps, so you'll need to test and see. If you go too low, you'll get StackOverflow exceptions. If you see those, just gradually increase until they go away.
Dan
>
> -Isaac
>
>
> http://www.oracle.com/technetwork/java/hotspotfaq-138619.html#threads_
> oom
>
> Might buy you some room for a few additional threads.
>
> Dan
>
>>
>> Regards,
>>
>> Rainer
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Ok so the problem just happened again just now. Dan, Can you elaborate on how to configure limiting the connector thread pool size. I am also going to lower the thread stack size as you recommended. It seems like this problem creeps up when we have a hiccup in connectivity at our data center. Perhaps I also need to lower the idle timeout some more between tomcat and mod_jk. They are also between a firewall by the way, so I can configure a timeout between the two there as well. We aren't experiencing too many idle disconnects there.
-Isaac
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: tomcat 6 refuses mod_jk connections after server runs for a couple of days
Posted by Daniel Mikusa <dm...@gopivotal.com>.
On Mar 4, 2014, at 1:55 PM, Isaac Gonzalez <ig...@autoreturn.com> wrote:
> Dan,
>
> ________________________________________
> From: Daniel Mikusa [dmikusa@gopivotal.com]
> Sent: Tuesday, March 04, 2014 6:20 AM
> To: Tomcat Users List
> Subject: Re: tomcat 6 refuses mod_jk connections after server runs for a couple of days
>
> On Mar 4, 2014, at 6:32 AM, Rainer Jung <ra...@kippdata.de> wrote:
>
>> On 27.02.2014 23:06, Isaac Gonzalez wrote:
>>> Hi Christopher(and Konstantin), attached is a couple of thread dumps of when we experienced the issue again today. I also noticed we get this message right before the problem occurs:
>>> Feb 27, 2014 12:47:15 PM org.apache.tomcat.util.threads.ThreadPool$ControlRunnable run
>>> SEVERE: Caught exception (java.lang.OutOfMemoryError: unable to create new native thread) executing org.apache.jk.common.ChannelSocket$SocketAcceptor@177ddea, terminating thread
>>
>> Is it a 32Bit system? You have 2GB of heap plus Perm plus native memory
>> needed by the process plus thread stacks. Not unlikely, that you ran out
>> of memory address space for a 32 bit process.
>>
>> The only fixes would then be:
>>
>> - switch to a 64 bit system
>>
>> - reduce heap if the app can work with less
>>
>> - improve performance or eliminate bottlenecks so that the app works
>> with less threads
>>
>> - limit you connector thread pool size. That will still mean that if
>> requests begin to queue because of performance problems, the web server
>> can't create additional connections, but you won't get in an irregular
>> situation as you experience now. In that case you would need to
>> configure a low idle timeout for the connections on the JK and TC side.
>
> It may also be possible to lower the thread stack size with the -Xss option.
>
> Ok so we are 64 bit Linux with 1024k in the 64-bit VM....would lowering it to 64k be a bit too low? What sort of repercussions would we run into?
> Very helpful information by the way.
It depends on your apps, so you’ll need to test and see. If you go too low, you’ll get StackOverflow exceptions. If you see those, just gradually increase until they go away.
Dan
>
> -Isaac
>
> http://www.oracle.com/technetwork/java/hotspotfaq-138619.html#threads_oom
>
> Might buy you some room for a few additional threads.
>
> Dan
>
>>
>> Regards,
>>
>> Rainer
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: tomcat 6 refuses mod_jk connections after server runs for a
couple of days
Posted by Isaac Gonzalez <ig...@autoreturn.com>.
Dan,
________________________________________
From: Daniel Mikusa [dmikusa@gopivotal.com]
Sent: Tuesday, March 04, 2014 6:20 AM
To: Tomcat Users List
Subject: Re: tomcat 6 refuses mod_jk connections after server runs for a couple of days
On Mar 4, 2014, at 6:32 AM, Rainer Jung <ra...@kippdata.de> wrote:
> On 27.02.2014 23:06, Isaac Gonzalez wrote:
>> Hi Christopher(and Konstantin), attached is a couple of thread dumps of when we experienced the issue again today. I also noticed we get this message right before the problem occurs:
>> Feb 27, 2014 12:47:15 PM org.apache.tomcat.util.threads.ThreadPool$ControlRunnable run
>> SEVERE: Caught exception (java.lang.OutOfMemoryError: unable to create new native thread) executing org.apache.jk.common.ChannelSocket$SocketAcceptor@177ddea, terminating thread
>
> Is it a 32Bit system? You have 2GB of heap plus Perm plus native memory
> needed by the process plus thread stacks. Not unlikely, that you ran out
> of memory address space for a 32 bit process.
>
> The only fixes would then be:
>
> - switch to a 64 bit system
>
> - reduce heap if the app can work with less
>
> - improve performance or eliminate bottlenecks so that the app works
> with less threads
>
> - limit you connector thread pool size. That will still mean that if
> requests begin to queue because of performance problems, the web server
> can't create additional connections, but you won't get in an irregular
> situation as you experience now. In that case you would need to
> configure a low idle timeout for the connections on the JK and TC side.
It may also be possible to lower the thread stack size with the -Xss option.
Ok so we are 64 bit Linux with 1024k in the 64-bit VM....would lowering it to 64k be a bit too low? What sort of repercussions would we run into?
Very helpful information by the way.
-Isaac
http://www.oracle.com/technetwork/java/hotspotfaq-138619.html#threads_oom
Might buy you some room for a few additional threads.
Dan
>
> Regards,
>
> Rainer
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: tomcat 6 refuses mod_jk connections after server runs for a couple of days
Posted by Daniel Mikusa <dm...@gopivotal.com>.
On Mar 4, 2014, at 6:32 AM, Rainer Jung <ra...@kippdata.de> wrote:
> On 27.02.2014 23:06, Isaac Gonzalez wrote:
>> Hi Christopher(and Konstantin), attached is a couple of thread dumps of when we experienced the issue again today. I also noticed we get this message right before the problem occurs:
>> Feb 27, 2014 12:47:15 PM org.apache.tomcat.util.threads.ThreadPool$ControlRunnable run
>> SEVERE: Caught exception (java.lang.OutOfMemoryError: unable to create new native thread) executing org.apache.jk.common.ChannelSocket$SocketAcceptor@177ddea, terminating thread
>
> Is it a 32Bit system? You have 2GB of heap plus Perm plus native memory
> needed by the process plus thread stacks. Not unlikely, that you ran out
> of memory address space for a 32 bit process.
>
> The only fixes would then be:
>
> - switch to a 64 bit system
>
> - reduce heap if the app can work with less
>
> - improve performance or eliminate bottlenecks so that the app works
> with less threads
>
> - limit you connector thread pool size. That will still mean that if
> requests begin to queue because of performance problems, the web server
> can't create additional connections, but you won't get in an irregular
> situation as you experience now. In that case you would need to
> configure a low idle timeout for the connections on the JK and TC side.
It may also be possible to lower the thread stack size with the -Xss option.
http://www.oracle.com/technetwork/java/hotspotfaq-138619.html#threads_oom
Might buy you some room for a few additional threads.
Dan
>
> Regards,
>
> Rainer
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: tomcat 6 refuses mod_jk connections after server runs for a
couple of days
Posted by Isaac Gonzalez <ig...@autoreturn.com>.
Rainer,
________________________________________
From: Rainer Jung [rainer.jung@kippdata.de]
Sent: Tuesday, March 04, 2014 3:32 AM
To: Tomcat Users List
Subject: Re: tomcat 6 refuses mod_jk connections after server runs for a couple of days
On 27.02.2014 23:06, Isaac Gonzalez wrote:
> Hi Christopher(and Konstantin), attached is a couple of thread dumps of when we experienced the issue again today. I also noticed we get this message right before the problem occurs:
> Feb 27, 2014 12:47:15 PM org.apache.tomcat.util.threads.ThreadPool$ControlRunnable run
> SEVERE: Caught exception (java.lang.OutOfMemoryError: unable to create new native thread) executing org.apache.jk.common.ChannelSocket$SocketAcceptor@177ddea, terminating thread
Is it a 32Bit system? You have 2GB of heap plus Perm plus native memory
needed by the process plus thread stacks. Not unlikely, that you ran out
of memory address space for a 32 bit process.
No we are running on 64 bit
The only fixes would then be:
- switch to a 64 bit system
- reduce heap if the app can work with less
I'd like to keep the heap the same...most of our apps need it.
- improve performance or eliminate bottlenecks so that the app works
with less threads
Can you give an example of such a bottleneck? Ie: open unfinished connections to the backend database from tomcat?
- limit you connector thread pool size. That will still mean that if
requests begin to queue because of performance problems, the web server
can't create additional connections, but you won't get in an irregular
situation as you experience now. In that case you would need to
configure a low idle timeout for the connections on the JK and TC side.
I'm not sure I want to do this because that would cause hiccups on the client UI and not allow them to connect. It would keep active ones open I imagine. I already have a 5 minute idle timeout on JK and TC...Guess I need to lower it down to like a minute or so... Just wondering if that would be too low.
-Isaac
Regards,
Rainer
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: tomcat 6 refuses mod_jk connections after server runs for a couple
of days
Posted by Rainer Jung <ra...@kippdata.de>.
On 27.02.2014 23:06, Isaac Gonzalez wrote:
> Hi Christopher(and Konstantin), attached is a couple of thread dumps of when we experienced the issue again today. I also noticed we get this message right before the problem occurs:
> Feb 27, 2014 12:47:15 PM org.apache.tomcat.util.threads.ThreadPool$ControlRunnable run
> SEVERE: Caught exception (java.lang.OutOfMemoryError: unable to create new native thread) executing org.apache.jk.common.ChannelSocket$SocketAcceptor@177ddea, terminating thread
Is it a 32Bit system? You have 2GB of heap plus Perm plus native memory
needed by the process plus thread stacks. Not unlikely, that you ran out
of memory address space for a 32 bit process.
The only fixes would then be:
- switch to a 64 bit system
- reduce heap if the app can work with less
- improve performance or eliminate bottlenecks so that the app works
with less threads
- limit you connector thread pool size. That will still mean that if
requests begin to queue because of performance problems, the web server
can't create additional connections, but you won't get in an irregular
situation as you experience now. In that case you would need to
configure a low idle timeout for the connections on the JK and TC side.
Regards,
Rainer
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: tomcat 6 refuses mod_jk connections after server runs for a
couple of days
Posted by Isaac Gonzalez <ig...@autoreturn.com>.
Hi Christopher(and Konstantin), attached is a couple of thread dumps of when we experienced the issue again today. I also noticed we get this message right before the problem occurs:
Feb 27, 2014 12:47:15 PM org.apache.tomcat.util.threads.ThreadPool$ControlRunnable run
SEVERE: Caught exception (java.lang.OutOfMemoryError: unable to create new native thread) executing org.apache.jk.common.ChannelSocket$SocketAcceptor@177ddea, terminating thread
Thanks,
Isaac
-----Original Message-----
From: Isaac Gonzalez [mailto:igonzalez@autoreturn.com]
Sent: Tuesday, February 25, 2014 9:13 AM
To: Tomcat Users List
Subject: RE: tomcat 6 refuses mod_jk connections after server runs for a couple of days
Hi Christopher thanks so much for your replies..., I am responding with inline comments below ________________________________________
From: Christopher Schultz [chris@christopherschultz.net]
Sent: Monday, February 24, 2014 9:56 PM
To: Tomcat Users List
Subject: Re: tomcat 6 refuses mod_jk connections after server runs for a couple of days
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Isaac,
On 2/24/14, 2:27 PM, Isaac Gonzalez wrote:
> Hello all,
>
> I'm running tomcat 6.0.32 on Cent OS 6 with 2 front end apache load
> balancers with a firewall in between the tomcat and load balancers
> using mod_jk v. 1.2.37 under apache 2.2.10 to connect the backend
> tomcat. I have had this running ok for a few years but our user
> traffic has increased significantly. A few months ago, the tomcat
> server seemed to refuse or not accept any new connections from either
> load balancer and required a restart on the tomcat end, even though I
> could easily connect to tomcat on port 8080(manager). I can
> intermittently telnet to port 8009, but am denied a bit as well both
> inside and outside the firewall.
>
> I proceeded to split the tomcats up into their own instances, hoping
> when this issue recurred that it would only affect a particular tomcat
> app. It also gave our developers the ability to patch a single tomcat
> app without downing all of our apps.
>
> Unfortunately, this issue has recurred several times and I have spent
> most of my days researching and digging for hope of someone with a
> similar experience that may have resolved it. Last Friday the problem
> was so bad, I had to completely restart the tomcat server(reboot it).
>
> So far I am at a loss...I have installed psi-probe on all tomcat
> instances to give me more in depth analysis to tomcat threads and
> related server metadata when the problem is occuring. I have made a
> few modifications to workers.properties, in particular to decrease the
> connection timeout as well as the tomcat ajp connector from 10 minutes
> to 5 minutes and added the ping timeout and socket timeout.
> I also increased my apache prefork MPM client connections to 500 on
> each load balancer. Below is my relevant configs...any suggestions to
> help remedy this would help... I have also increased threads from 200
> to 500 on all tomcat instances.
I'd be interested to see a thread dump on a "stuck" Tomcat to see what it's doing. If it happens again, please take a thread dump (or, better yet, 3 or so maybe 5-10 seconds apart) and post them back to the list.
http://wiki.apache.org/tomcat/HowTo#How_do_I_obtain_a_thread_dump_of_my_running_webapp_.3F
Isaac: Ok, I will submit one...PSI Probe shows them all but I have to click on each one at a time...
Does restarting the Tomcat instance fix everything, or do you have to also bounce httpd? What happens if you bounce only httpd?
Isaac: Restarting the Tomcat instance fixes it. Bouncing httpd has no affect.
After the "split", did both Tomcats appear to lock-up simultaneously, or did only one of them have a problem and the other one stayed up?
Isaac: They all appear to lock-up simultaneously, if users try to access that JK mount point.
Do the lock-ups appear to be related to anything you can observe, such as particularly high-load, etc.?
I have seen the lock-up appear when we had some network latency and other network issues going on all externally facing traffic at this datacenter. I have also seen it happen when there is some database connectivity issues within the applications. Other times I have just seen it appear with possibly a high load.
> Workers.properties:
>
> worker.list=jkstatus,server1,server2,server3,server4,server5,server6,s
> erver7,server8
>
>
worker.jkstatus.type=status
>
> # Let's define some defaults worker.basic.port=8009
> worker.basic.type=ajp13 worker.basic.socket_keepalive=True
> worker.basic.connection_pool_timeout=300
> worker.basic.ping_timeout=1000 worker.basic.ping_mode=A
> worker.basic.socket_timeout=10
>
> worker.lb1.distance=0 worker.lb1.reference=worker.basic
>
> worker.server1.host= server1hostname
> worker.server1.reference=worker.lb1
> worker.server2.host=server2hostname
> worker.server2.reference=worker.lb1
> worker.server3.host=server3hostname
> worker.server3.reference=worker.lb1 worker.server4.host=
> server4hostname worker.server4.reference=worker.lb1
> worker.server5.host= server5hostname
> worker.server5.reference=worker.lb1 worker.server6.host=
> server6hostname worker.server6.reference=worker.lb1
> worker.server7.host= server7hostname
> worker.server7.reference=worker.lb1 worker.server8.host=
> server7hostname worker.server8.reference=worker.lb1
You didn't show any JkMounts in your httpd.conf file. What worker are you using? It sounded like you were load-balancing the servers, but your "lb1" worker does not have any balance_workers setting so it doesn't look like it's going to work.
Isaac: I am not load-balancing the tomcat servers...I only have one...I do "load balance" the apache front end servers via dns round-robin....
JkWorkersFile /etc/httpd/conf/workers.properties
JkLogFile logs/mod_jk.log
JkLogLevel info
JkLogStampFormat "[%a %b %d %H:%M:%S %Y] "
# JkOptions indicate to send SSL KEY SIZE, JkOptions +ForwardKeySize +ForwardURICompat -ForwardDirectories JkRequestLogFormat "%w %V %T"
JkMount /appnam4escrubbed server4
JkMount /appnames4crubbed/* server4
JkMount /appname5scrubbed server5
JkMount /appname5scrubbed/* server5
JkMount /appname7scrubbed server7
JkMount /appname7scrubbed/* server7
JkMount /appname2scrubbed server2
JkMount /appname2scrubbed/* server2
JkMount /appname6scrubbed server6
JkMount /appname6scrubbed/* server6
JkMount /appname3scrubbed server3
JkMount /appname3scrubbed/* server3
JkMount /appname1scrubbed server1
JkMount /appname1scrubbed/* server1
JkMount /appname8scrubbed server8
JkMount /appname8scrubbed/* server8
JkMount /jkmanager/* jkstatus
> httpd.conf:
>
> KeepAlive Off MaxKeepAliveRequests 100 KeepAliveTimeout 15
>
>
>
> # prefork MPM # StartServers: number of server processes to start #
> MinSpareServers: minimum number of server processes which are kept
> spare # MaxSpareServers: maximum number of server processes which are
> kept spare # ServerLimit: maximum value for MaxClients for the
> lifetime of the server # MaxClients: maximum number of server
> processes allowed to start # MaxRequestsPerChild: maximum number of
> requests a server process serves <IfModule prefork.c> StartServers
> 8 MinSpareServers 5 MaxSpareServers 20 ServerLimit 500
> MaxClients 500 MaxRequestsPerChild 5000 </IfModule>
It would be good to see your Jk* setting as well.
Isaac: See above
> Tomcat server.xml:
>
> <!-- Define an AJP 1.3 Connector on port 8009 --> <Connector
> port="8009" address="x.x.x.x" protocol="AJP/1.3"
> redirectPort="8443" connectionTimeout="300000" maxThreads="500" />
Why do you both having a connectionTimeout on an AJP connection? httpd should only send a request to you once the request line has been received by the client, so there isn't really any legitimate reason for an AJP request to time out before the request-line comes-through.
I have seen many with this same problem and it has been suggested, even in the apache connector documentation to have the connection timeout in the AJP connector match the connection_pool_timeout in the workers.properties file. Here I have them both set to 5 minutes. They were set to 10 minutes before. I also recently added the socket_timeout in workers.properties after reading a helpful how-to from some jboss documentation. It did not help with the problem at all.
IMO, you should define an <Executor> and share threads between your AJP connector and your HTTP connector (which you didn't show config for, but mentioned you had one above running on port 8080). Otherwise, your Tomcat configuration looks fine.
Isaac: Hmmm...here is my <executor> connector below which is commented out...:
<Service name="Catalina">
<!--The connectors can use a shared executor, you can define one or more named thread pools-->
<!--
<Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
maxThreads="150" minSpareThreads="4"/>
-->
<!-- A "Connector" represents an endpoint by which requests are received
and responses are returned. Documentation at :
Java HTTP Connector: /docs/config/http.html (blocking & non-blocking)
Java AJP Connector: /docs/config/ajp.html
APR (HTTP/AJP) Connector: /docs/apr.html
Define a non-SSL HTTP/1.1 Connector on port 8080
-->
<Connector port="8080" protocol="HTTP/1.1"
address="x.x.x.x" connectionTimeout="20000"
redirectPort="8443" />
<!-- A "Connector" using the shared thread pool-->
<!--
<Connector executor="tomcatThreadPool"
port="8080" protocol="HTTP/1.1"
connectionTimeout="20000"
redirectPort="8443" />
-->
If you get despirate, you can set mod_jk's log level to DEBUG and watch your disk fill up. You'll get great information if/when things start to go south.
Isaac: I am very desperate Chris, I guess I have no choice...I'm unsure of how I should implement the executor...so any suggestions on that would be helpful...
-Isaac
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBCAAGBQJTDDCXAAoJEBzwKT+lPKRYfm0P/jx9Dqimk06Aihjpm1Qv9e6r
G7wgpmKjpPYOULNk6TVsz0Gs4YljOTklviDvTH2dL1bHszaJNKR5ZOKXlgXiaWRG
3w6HpH2ujgJQczXxVi43eF3bbYnPHJguGEN6pVG4ig5qYGuwPVHNYsvWQBRjVITS
IRxnTdCsBtBOgdfAS8WEJNZKK/Ep5s37nehtcZ4Hq8T/q9TkWGnZ7rABglUN0Mex
JKnSJR3/VYS1FFH7efV+GDdtlPRFEebXXzjA+DNAYyo2HH4zV0XAfF4zaMAmTHxK
MgvP8sQA6klWH3XSkNQRm+SJMD5Kl32JLsIi3osGP7pLowlg/jvI66lcZnkVKhGi
SIhgzUSEHLAgBZOO+wIRyTEBU5ALJ6kW3ySZqdL6fF4wDsON1ZV/gXCshFlByJ9q
i+jth+jtovmDsLgiwYhKsZ3zr+lfeVVQWjZ3cY5l0kGxaPYpJt+EhEjLtjSMg8Lg
06ZMSBi8hZ+R+fyz1jBzcGadRMTRmN6aaolf/XZOom4i+5qcjrJ5r0cSXFqVBY1D
XBfnDRwAAjOfWLCwiQ9XWhDWfyOQZghJ2mBT2vaQbCnZelcK4e3V0Ix8iYUatnTu
o2lKiOxsRlkIXRgBfutXu7z3IpdNK1YaotpFtz4hb4lz0dSmDsNP5llpw3zyuQ9l
ZMKc94GOw+ZY2Tt2XFYD
=tdbH
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: tomcat 6 refuses mod_jk connections after server runs for a
couple of days
Posted by Isaac Gonzalez <ig...@autoreturn.com>.
Hi Christopher thanks so much for your replies...,
I am responding with inline comments below
________________________________________
From: Christopher Schultz [chris@christopherschultz.net]
Sent: Monday, February 24, 2014 9:56 PM
To: Tomcat Users List
Subject: Re: tomcat 6 refuses mod_jk connections after server runs for a couple of days
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Isaac,
On 2/24/14, 2:27 PM, Isaac Gonzalez wrote:
> Hello all,
>
> I'm running tomcat 6.0.32 on Cent OS 6 with 2 front end apache load
> balancers with a firewall in between the tomcat and load balancers
> using mod_jk v. 1.2.37 under apache 2.2.10 to connect the backend
> tomcat. I have had this running ok for a few years but our user
> traffic has increased significantly. A few months ago, the tomcat
> server seemed to refuse or not accept any new connections from
> either load balancer and required a restart on the tomcat end, even
> though I could easily connect to tomcat on port 8080(manager). I
> can intermittently telnet to port 8009, but am denied a bit as well
> both inside and outside the firewall.
>
> I proceeded to split the tomcats up into their own instances,
> hoping when this issue recurred that it would only affect a
> particular tomcat app. It also gave our developers the ability to
> patch a single tomcat app without downing all of our apps.
>
> Unfortunately, this issue has recurred several times and I have
> spent most of my days researching and digging for hope of someone
> with a similar experience that may have resolved it. Last Friday
> the problem was so bad, I had to completely restart the tomcat
> server(reboot it).
>
> So far I am at a loss...I have installed psi-probe on all tomcat
> instances to give me more in depth analysis to tomcat threads and
> related server metadata when the problem is occuring. I have made a
> few modifications to workers.properties, in particular to decrease
> the connection timeout as well as the tomcat ajp connector from 10
> minutes to 5 minutes and added the ping timeout and socket timeout.
> I also increased my apache prefork MPM client connections to 500 on
> each load balancer. Below is my relevant configs...any suggestions
> to help remedy this would help... I have also increased threads
> from 200 to 500 on all tomcat instances.
I'd be interested to see a thread dump on a "stuck" Tomcat to see what
it's doing. If it happens again, please take a thread dump (or, better
yet, 3 or so maybe 5-10 seconds apart) and post them back to the list.
http://wiki.apache.org/tomcat/HowTo#How_do_I_obtain_a_thread_dump_of_my_running_webapp_.3F
Isaac: Ok, I will submit one...PSI Probe shows them all but I have to click on each one at a time...
Does restarting the Tomcat instance fix everything, or do you have to
also bounce httpd? What happens if you bounce only httpd?
Isaac: Restarting the Tomcat instance fixes it. Bouncing httpd has no affect.
After the "split", did both Tomcats appear to lock-up simultaneously,
or did only one of them have a problem and the other one stayed up?
Isaac: They all appear to lock-up simultaneously, if users try to access that JK mount point.
Do the lock-ups appear to be related to anything you can observe, such
as particularly high-load, etc.?
I have seen the lock-up appear when we had some network latency and other network issues going on all externally facing traffic at this datacenter. I have also seen it happen when there is some database connectivity issues within the applications. Other times I have just seen it appear with possibly a high load.
> Workers.properties:
>
> worker.list=jkstatus,server1,server2,server3,server4,server5,server6,server7,server8
>
>
worker.jkstatus.type=status
>
> # Let's define some defaults worker.basic.port=8009
> worker.basic.type=ajp13 worker.basic.socket_keepalive=True
> worker.basic.connection_pool_timeout=300
> worker.basic.ping_timeout=1000 worker.basic.ping_mode=A
> worker.basic.socket_timeout=10
>
> worker.lb1.distance=0 worker.lb1.reference=worker.basic
>
> worker.server1.host= server1hostname
> worker.server1.reference=worker.lb1
> worker.server2.host=server2hostname
> worker.server2.reference=worker.lb1
> worker.server3.host=server3hostname
> worker.server3.reference=worker.lb1 worker.server4.host=
> server4hostname worker.server4.reference=worker.lb1
> worker.server5.host= server5hostname
> worker.server5.reference=worker.lb1 worker.server6.host=
> server6hostname worker.server6.reference=worker.lb1
> worker.server7.host= server7hostname
> worker.server7.reference=worker.lb1 worker.server8.host=
> server7hostname worker.server8.reference=worker.lb1
You didn't show any JkMounts in your httpd.conf file. What worker are
you using? It sounded like you were load-balancing the servers, but
your "lb1" worker does not have any balance_workers setting so it
doesn't look like it's going to work.
Isaac: I am not load-balancing the tomcat servers...I only have one...I do "load balance" the apache front end servers via dns round-robin....
JkWorkersFile /etc/httpd/conf/workers.properties
JkLogFile logs/mod_jk.log
JkLogLevel info
JkLogStampFormat "[%a %b %d %H:%M:%S %Y] "
# JkOptions indicate to send SSL KEY SIZE,
JkOptions +ForwardKeySize +ForwardURICompat -ForwardDirectories
JkRequestLogFormat "%w %V %T"
JkMount /appnam4escrubbed server4
JkMount /appnames4crubbed/* server4
JkMount /appname5scrubbed server5
JkMount /appname5scrubbed/* server5
JkMount /appname7scrubbed server7
JkMount /appname7scrubbed/* server7
JkMount /appname2scrubbed server2
JkMount /appname2scrubbed/* server2
JkMount /appname6scrubbed server6
JkMount /appname6scrubbed/* server6
JkMount /appname3scrubbed server3
JkMount /appname3scrubbed/* server3
JkMount /appname1scrubbed server1
JkMount /appname1scrubbed/* server1
JkMount /appname8scrubbed server8
JkMount /appname8scrubbed/* server8
JkMount /jkmanager/* jkstatus
> httpd.conf:
>
> KeepAlive Off MaxKeepAliveRequests 100 KeepAliveTimeout 15
>
>
>
> # prefork MPM # StartServers: number of server processes to start #
> MinSpareServers: minimum number of server processes which are kept
> spare # MaxSpareServers: maximum number of server processes which
> are kept spare # ServerLimit: maximum value for MaxClients for the
> lifetime of the server # MaxClients: maximum number of server
> processes allowed to start # MaxRequestsPerChild: maximum number of
> requests a server process serves <IfModule prefork.c> StartServers
> 8 MinSpareServers 5 MaxSpareServers 20 ServerLimit 500
> MaxClients 500 MaxRequestsPerChild 5000 </IfModule>
It would be good to see your Jk* setting as well.
Isaac: See above
> Tomcat server.xml:
>
> <!-- Define an AJP 1.3 Connector on port 8009 --> <Connector
> port="8009" address="x.x.x.x" protocol="AJP/1.3"
> redirectPort="8443" connectionTimeout="300000" maxThreads="500" />
Why do you both having a connectionTimeout on an AJP connection? httpd
should only send a request to you once the request line has been
received by the client, so there isn't really any legitimate reason
for an AJP request to time out before the request-line comes-through.
I have seen many with this same problem and it has been suggested, even in the apache connector documentation to have the connection timeout in the AJP connector match the connection_pool_timeout in the workers.properties file. Here I have them both set to 5 minutes. They were set to 10 minutes before. I also recently added the socket_timeout in workers.properties after reading a helpful how-to from some jboss documentation. It did not help with the problem at all.
IMO, you should define an <Executor> and share threads between your
AJP connector and your HTTP connector (which you didn't show config
for, but mentioned you had one above running on port 8080). Otherwise,
your Tomcat configuration looks fine.
Isaac: Hmmm...here is my <executor> connector below which is commented out...:
<Service name="Catalina">
<!--The connectors can use a shared executor, you can define one or more named thread pools-->
<!--
<Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
maxThreads="150" minSpareThreads="4"/>
-->
<!-- A "Connector" represents an endpoint by which requests are received
and responses are returned. Documentation at :
Java HTTP Connector: /docs/config/http.html (blocking & non-blocking)
Java AJP Connector: /docs/config/ajp.html
APR (HTTP/AJP) Connector: /docs/apr.html
Define a non-SSL HTTP/1.1 Connector on port 8080
-->
<Connector port="8080" protocol="HTTP/1.1"
address="x.x.x.x" connectionTimeout="20000"
redirectPort="8443" />
<!-- A "Connector" using the shared thread pool-->
<!--
<Connector executor="tomcatThreadPool"
port="8080" protocol="HTTP/1.1"
connectionTimeout="20000"
redirectPort="8443" />
-->
If you get despirate, you can set mod_jk's log level to DEBUG and
watch your disk fill up. You'll get great information if/when things
start to go south.
Isaac: I am very desperate Chris, I guess I have no choice...I'm unsure of how I should implement the executor...so any suggestions on that would be helpful...
-Isaac
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBCAAGBQJTDDCXAAoJEBzwKT+lPKRYfm0P/jx9Dqimk06Aihjpm1Qv9e6r
G7wgpmKjpPYOULNk6TVsz0Gs4YljOTklviDvTH2dL1bHszaJNKR5ZOKXlgXiaWRG
3w6HpH2ujgJQczXxVi43eF3bbYnPHJguGEN6pVG4ig5qYGuwPVHNYsvWQBRjVITS
IRxnTdCsBtBOgdfAS8WEJNZKK/Ep5s37nehtcZ4Hq8T/q9TkWGnZ7rABglUN0Mex
JKnSJR3/VYS1FFH7efV+GDdtlPRFEebXXzjA+DNAYyo2HH4zV0XAfF4zaMAmTHxK
MgvP8sQA6klWH3XSkNQRm+SJMD5Kl32JLsIi3osGP7pLowlg/jvI66lcZnkVKhGi
SIhgzUSEHLAgBZOO+wIRyTEBU5ALJ6kW3ySZqdL6fF4wDsON1ZV/gXCshFlByJ9q
i+jth+jtovmDsLgiwYhKsZ3zr+lfeVVQWjZ3cY5l0kGxaPYpJt+EhEjLtjSMg8Lg
06ZMSBi8hZ+R+fyz1jBzcGadRMTRmN6aaolf/XZOom4i+5qcjrJ5r0cSXFqVBY1D
XBfnDRwAAjOfWLCwiQ9XWhDWfyOQZghJ2mBT2vaQbCnZelcK4e3V0Ix8iYUatnTu
o2lKiOxsRlkIXRgBfutXu7z3IpdNK1YaotpFtz4hb4lz0dSmDsNP5llpw3zyuQ9l
ZMKc94GOw+ZY2Tt2XFYD
=tdbH
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: tomcat 6 refuses mod_jk connections after server runs for a couple
of days
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Isaac,
On 2/24/14, 2:27 PM, Isaac Gonzalez wrote:
> Hello all,
>
> I'm running tomcat 6.0.32 on Cent OS 6 with 2 front end apache load
> balancers with a firewall in between the tomcat and load balancers
> using mod_jk v. 1.2.37 under apache 2.2.10 to connect the backend
> tomcat. I have had this running ok for a few years but our user
> traffic has increased significantly. A few months ago, the tomcat
> server seemed to refuse or not accept any new connections from
> either load balancer and required a restart on the tomcat end, even
> though I could easily connect to tomcat on port 8080(manager). I
> can intermittently telnet to port 8009, but am denied a bit as well
> both inside and outside the firewall.
>
> I proceeded to split the tomcats up into their own instances,
> hoping when this issue recurred that it would only affect a
> particular tomcat app. It also gave our developers the ability to
> patch a single tomcat app without downing all of our apps.
>
> Unfortunately, this issue has recurred several times and I have
> spent most of my days researching and digging for hope of someone
> with a similar experience that may have resolved it. Last Friday
> the problem was so bad, I had to completely restart the tomcat
> server(reboot it).
>
> So far I am at a loss...I have installed psi-probe on all tomcat
> instances to give me more in depth analysis to tomcat threads and
> related server metadata when the problem is occuring. I have made a
> few modifications to workers.properties, in particular to decrease
> the connection timeout as well as the tomcat ajp connector from 10
> minutes to 5 minutes and added the ping timeout and socket timeout.
> I also increased my apache prefork MPM client connections to 500 on
> each load balancer. Below is my relevant configs...any suggestions
> to help remedy this would help... I have also increased threads
> from 200 to 500 on all tomcat instances.
I'd be interested to see a thread dump on a "stuck" Tomcat to see what
it's doing. If it happens again, please take a thread dump (or, better
yet, 3 or so maybe 5-10 seconds apart) and post them back to the list.
http://wiki.apache.org/tomcat/HowTo#How_do_I_obtain_a_thread_dump_of_my_running_webapp_.3F
Does restarting the Tomcat instance fix everything, or do you have to
also bounce httpd? What happens if you bounce only httpd?
After the "split", did both Tomcats appear to lock-up simultaneously,
or did only one of them have a problem and the other one stayed up?
Do the lock-ups appear to be related to anything you can observe, such
as particularly high-load, etc.?
> Workers.properties:
>
> worker.list=jkstatus,server1,server2,server3,server4,server5,server6,server7,server8
>
>
worker.jkstatus.type=status
>
> # Let's define some defaults worker.basic.port=8009
> worker.basic.type=ajp13 worker.basic.socket_keepalive=True
> worker.basic.connection_pool_timeout=300
> worker.basic.ping_timeout=1000 worker.basic.ping_mode=A
> worker.basic.socket_timeout=10
>
> worker.lb1.distance=0 worker.lb1.reference=worker.basic
>
> worker.server1.host= server1hostname
> worker.server1.reference=worker.lb1
> worker.server2.host=server2hostname
> worker.server2.reference=worker.lb1
> worker.server3.host=server3hostname
> worker.server3.reference=worker.lb1 worker.server4.host=
> server4hostname worker.server4.reference=worker.lb1
> worker.server5.host= server5hostname
> worker.server5.reference=worker.lb1 worker.server6.host=
> server6hostname worker.server6.reference=worker.lb1
> worker.server7.host= server7hostname
> worker.server7.reference=worker.lb1 worker.server8.host=
> server7hostname worker.server8.reference=worker.lb1
You didn't show any JkMounts in your httpd.conf file. What worker are
you using? It sounded like you were load-balancing the servers, but
your "lb1" worker does not have any balance_workers setting so it
doesn't look like it's going to work.
> httpd.conf:
>
> KeepAlive Off MaxKeepAliveRequests 100 KeepAliveTimeout 15
>
>
>
> # prefork MPM # StartServers: number of server processes to start #
> MinSpareServers: minimum number of server processes which are kept
> spare # MaxSpareServers: maximum number of server processes which
> are kept spare # ServerLimit: maximum value for MaxClients for the
> lifetime of the server # MaxClients: maximum number of server
> processes allowed to start # MaxRequestsPerChild: maximum number of
> requests a server process serves <IfModule prefork.c> StartServers
> 8 MinSpareServers 5 MaxSpareServers 20 ServerLimit 500
> MaxClients 500 MaxRequestsPerChild 5000 </IfModule>
It would be good to see your Jk* setting as well.
> Tomcat server.xml:
>
> <!-- Define an AJP 1.3 Connector on port 8009 --> <Connector
> port="8009" address="x.x.x.x" protocol="AJP/1.3"
> redirectPort="8443" connectionTimeout="300000" maxThreads="500" />
Why do you both having a connectionTimeout on an AJP connection? httpd
should only send a request to you once the request line has been
received by the client, so there isn't really any legitimate reason
for an AJP request to time out before the request-line comes-through.
IMO, you should define an <Executor> and share threads between your
AJP connector and your HTTP connector (which you didn't show config
for, but mentioned you had one above running on port 8080). Otherwise,
your Tomcat configuration looks fine.
If you get despirate, you can set mod_jk's log level to DEBUG and
watch your disk fill up. You'll get great information if/when things
start to go south.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBCAAGBQJTDDCXAAoJEBzwKT+lPKRYfm0P/jx9Dqimk06Aihjpm1Qv9e6r
G7wgpmKjpPYOULNk6TVsz0Gs4YljOTklviDvTH2dL1bHszaJNKR5ZOKXlgXiaWRG
3w6HpH2ujgJQczXxVi43eF3bbYnPHJguGEN6pVG4ig5qYGuwPVHNYsvWQBRjVITS
IRxnTdCsBtBOgdfAS8WEJNZKK/Ep5s37nehtcZ4Hq8T/q9TkWGnZ7rABglUN0Mex
JKnSJR3/VYS1FFH7efV+GDdtlPRFEebXXzjA+DNAYyo2HH4zV0XAfF4zaMAmTHxK
MgvP8sQA6klWH3XSkNQRm+SJMD5Kl32JLsIi3osGP7pLowlg/jvI66lcZnkVKhGi
SIhgzUSEHLAgBZOO+wIRyTEBU5ALJ6kW3ySZqdL6fF4wDsON1ZV/gXCshFlByJ9q
i+jth+jtovmDsLgiwYhKsZ3zr+lfeVVQWjZ3cY5l0kGxaPYpJt+EhEjLtjSMg8Lg
06ZMSBi8hZ+R+fyz1jBzcGadRMTRmN6aaolf/XZOom4i+5qcjrJ5r0cSXFqVBY1D
XBfnDRwAAjOfWLCwiQ9XWhDWfyOQZghJ2mBT2vaQbCnZelcK4e3V0Ix8iYUatnTu
o2lKiOxsRlkIXRgBfutXu7z3IpdNK1YaotpFtz4hb4lz0dSmDsNP5llpw3zyuQ9l
ZMKc94GOw+ZY2Tt2XFYD
=tdbH
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org