You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Hildegard Meier <da...@gmx.de> on 2016/08/31 12:10:38 UTC
mod_jk/1.2.37 Apache/2.4.7 (Ubuntu): AJP established connections
pile up, connection_pool_timeout has no effect?
Running:
Ubuntu Trusty 14.0.4 LTS 64 Bit
mod_jk/1.2.37
Apache/2.4.7 (Ubuntu)
Hello,
I have much investigated into this, but have no idea, why the AJP connections from the Apache (MPM worker) to the AJP backends pile up.
They do not get closed, they stay in state ESTABLISHED.
Over night, the idle connections should be closed from the pool, but the do not get closed, they get more and more every day, and the RAM of the Apache server gets eaten up.
We have the following /etc/libapache2-mod-jk/workers.properties:
---
# not used
workers.tomcat_home=/usr/share/tomcat6
# not used
workers.java_home=/usr/lib/jvm/default-java
ps=/
# General worker template
worker.template_01.type=ajp13
worker.template_01.socket_connect_timeout=3000
worker.template_01.ping_mode=A
worker.template_01.connection_pool_timeout=600
worker.template_01.connection_pool_minsize=1
worker.template_01.socket_keepalive=true
worker.template_01.reply_timeout=1800000
---
and on the AJP backends (Jboss / EAP), we have also set the connectionTimeout to 600000 ms , as written in
https://tomcat.apache.org/connectors-doc/common_howto/timeouts.html
But even a Backend with low traffic, (maybe one request per minute) gets 300 established AJP connections after some days.
Here the content of /etc/apache2/mods-enabled/jk.conf:
<IfModule jk_module>
JkWorkersFile /etc/libapache2-mod-jk/workers.properties
JkLogFile /var/log/apache2/mod_jk.log
JkLogLevel info
JkShmFile /var/log/apache2/jk-runtime-status
JkWatchdogInterval 60
</IfModule>
It looks like the JkWatchdog is not running? Or do the connections do not idle because of some non-Apache requests like cping/cpong?
Your support is much appreciated.
Thanks
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: mod_jk/1.2.37 Apache/2.4.7 (Ubuntu): AJP established
connections pile up, connection_pool_timeout has no effect?
Posted by Jäkel,
Guido <G....@dnb.de>.
Dear Hildegard,
I wrote:
>Note that the AJP connection pool is used per child (and per tomcat backend definition if you're using more than one). You
>use connection_pool_minsize=1. Please also note, that the spare threads will no gather to the lowest possible number of
>childs. May this facts explain the "remaining idle" AJP connections?
You can get a good picture of what's up if you watch the scoreboard of the Apache server status page (mod_status) with the option refresh=1
As an example on one of my Apache frontends I'm using 16 children with 64 threads. And a minimum pool of 128 (= 2*64), i.e. there may be 6 connections keep to each Tomcat after first usage of any thread in a newborn child as long as the child is alive.
The actual scoreboard of an more or less busy of my Apache looks like
................................................................
_KC_KKKKKKKKR___WK___K_K_KK_RRR__K_C_K__KK_K______K_K__KRKK_KR__
K_WKRR_KRK_KCK_KCR_K__RK_KK_W_KKW__K__KK___KRRK_RKW_WKR_KK_RKR_K
__KK_RWRRK_CC__KW_KKRRKKK___KKKKK____KW_KKR_RK_R__KW__KCKCK___R_
K_K_KKRKR_CR__KR_K_K__K__KR_K___KK__KK__WK__KK_K_KRK_KKR__KRK__K
_RK__R_K_CR__K__K__K__KKK__K_KKKRK____W_K_RK_RKK_KK___RKRK_KRKKK
R_R__R_KKK_KW_R_____RKKKKRW__RKK_K_CKK__KC___KKKKKRKKKC__R__KR_K
R____CKRWKKKK_KR_K_K___KK_R_K_K___K_R______RK_KRK__K_K_KRRK___RK
................................................................
___KK_R__K___RWK_R_K_KK_K_K_KKK____RKK_K______KRRKKR_RRRR__KKK_R
................................................................
KK__RRKKRRKK_K_K____KK_K__KRKKKRK___RK_KKK_KKKK____R_KWRK_K_KC_K
K_KC_RK__KRKK___RKKR___KK_KR_KK__KK_KKK_WKKK__KRK__K__K_RKKKKKKK
................................................................
................................................................
................................................................
and the same for a much more lazy one
K____K______KK_K_KKK__K__________K__K___________K__K__W_KK__KKKK
___KW_K___K__K__KK___KK__K__K_KK_K____K______KK____W_W__K_____K_
................................................................
K_K__K____K____KK__K___K_K__K__K______K__K_K_________W_____KK___
___KKK_____K_____________________K_K___K__KK__K__________K______
K_W________K____KK_____K__K__K_______K___K______K________K______
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
where '_' represents an idle thread but a line of '.' a closed child.
Guido
RE: [SOLVED] mod_jk/1.2.37 Apache/2.4.7 (Ubuntu): AJP established
connections pile up, connection_pool_timeout has no effect?
Posted by Jäkel,
Guido <G....@dnb.de>.
>From: Hildegard Meier [mailto:daku8938@gmx.de]
>Sent: Thursday, September 01, 2016 10:14 AM
>To: users@tomcat.apache.org
>Subject: [SOLVED] mod_jk/1.2.37 Apache/2.4.7 (Ubuntu): AJP established connections pile up, connection_pool_timeout has
>no effect?
>
>Dear Guido,
>
>many thanks for Your answer.
many thanks for your feedback, too.
>
>It looks, like the parameter reply_timeout=1800000 was the solution. I added that parameter yesterday and was not sure if
>that would help and since I go to vacation in short time, I asked in this list in parallel to maybe solve the problem before
>vacation in another way.
>[...]
>Why the backend applications so often do not give any response, I think is another issue for the developers.
>
>But maybe the importance of changing the reply_timeout parameter from default value "infinit" to a even very high value of
>30 min should be stated even more prominent in the worker documentation.
I set this even lower (to 120s) because it's said it's the timeout for reading of each packet and no application on the Tomcat should take so much time to send any part of an answers.
I might be of worth if you pass the URIs of the dangling requests to developers. Because not finished, they are not logged. But you may take a look on the scoreboards of the Apache or the Tomcat. On Tomcat, it available via an MBean, you may read out it via JMX or via the http-proxy of the Tomcat Manager application. I may send you a script as a pointer, if you like.
Guido
[SOLVED] mod_jk/1.2.37 Apache/2.4.7 (Ubuntu): AJP established
connections pile up, connection_pool_timeout has no effect?
Posted by Hildegard Meier <da...@gmx.de>.
Dear Guido,
many thanks for Your answer.
It looks, like the parameter reply_timeout=1800000 was the solution. I added that parameter yesterday and was not sure if that would help and since I go to vacation in short time, I asked in this list in parallel to maybe solve the problem before vacation in another way.
The documentation for reply_timeout (https://tomcat.apache.org/connectors-doc/reference/workers.html) says "By default (value zero) the webserver will wait forever which could be an issue for you." and it looks, like this really was an issue for us. It looks to me, like the JBoss backends do not give any reply to ajp requests sometimes(often?) and they are then kept established forever and pile up. They are not interpreted as idle, and therefore they do not get removed by JkWatchDog. Why the backend applications so often do not give any response, I think is another issue for the developers.
But maybe the importance of changing the reply_timeout parameter from default value "infinit" to a even very high value of 30 min should be stated even more prominent in the worker documentation.
> you wrote, you're using the worker MPM. What are the pool sizes for child workers and it's threads, i.e. the values for ServerLimit, ThreadsPerChild? And what's the minimum thread pool size, i.e. MinSpareThreads?
/etc/apache2/mods-enabled/mpm_worker.conf
<IfModule mpm_worker_module>
StartServers 2
MinSpareThreads 25
MaxSpareThreads 75
ThreadLimit 64
ThreadsPerChild 35
MaxRequestWorkers 560
MaxConnectionsPerChild 10000
</IfModule>
> But even if the number of connections is unexpected high -- this will not consome more memory time by time after establishing.
I don't know, but I started investigating the Apache because it nearly run out of RAM some days ago, as Nagios informed. And the RAM usage of Apache clearly correlates with the count of AJP connections, as the Nagios graphs show.
Now, with connection pool cleanup working since yesterday, the count of AJP connections is radical lower then before, and also the RAM usage is radical lower in parallel :-)
Many thanks again, and
best regards,
Hilde
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: mod_jk/1.2.37 Apache/2.4.7 (Ubuntu): AJP established
connections pile up, connection_pool_timeout has no effect?
Posted by Jäkel,
Guido <G....@dnb.de>.
Dear Hildegard,
you wrote, you're using the worker MPM. What are the pool sizes for child workers and it's threads, i.e. the values for ServerLimit, ThreadsPerChild? And what's the minimum thread pool size, i.e. MinSpareThreads?
Note that the AJP connection pool is used per child (and per tomcat backend definition if you're using more than one). You use connection_pool_minsize=1. Please also note, that the spare threads will no gather to the lowest possible number of childs. May this facts explain the "remaining idle" AJP connections?
But even if the number of connections is unexpected high -- this will not consome more memory time by time after establishing.
If you see a real leak, this will have another reason. One from my lessons learned: If you're write access and error logs by a piped process like rotatelogs, this will "slowly and continuous eat up memory", if it will not consume. This will e.g. happen, if rotatelogs can't create files because of a broken path or lack of rights.
Greetings
Guido
>-----Original Message-----
>From: Hildegard Meier [mailto:daku8938@gmx.de]
>Sent: Wednesday, August 31, 2016 2:11 PM
>To: users@tomcat.apache.org
>Subject: mod_jk/1.2.37 Apache/2.4.7 (Ubuntu): AJP established connections pile up, connection_pool_timeout has no effect?
>
>Running:
>Ubuntu Trusty 14.0.4 LTS 64 Bit
>mod_jk/1.2.37
>Apache/2.4.7 (Ubuntu)
>
>Hello,
>
>I have much investigated into this, but have no idea, why the AJP connections from the Apache (MPM worker) to the AJP
>backends pile up.
>They do not get closed, they stay in state ESTABLISHED.
>
>Over night, the idle connections should be closed from the pool, but the do not get closed, they get more and more every
>day, and the RAM of the Apache server gets eaten up.
>
>We have the following /etc/libapache2-mod-jk/workers.properties:
>
>---
>
># not used
>workers.tomcat_home=/usr/share/tomcat6
># not used
>workers.java_home=/usr/lib/jvm/default-java
>ps=/
>
># General worker template
>worker.template_01.type=ajp13
>worker.template_01.socket_connect_timeout=3000
>worker.template_01.ping_mode=A
>worker.template_01.connection_pool_timeout=600
>worker.template_01.connection_pool_minsize=1
>worker.template_01.socket_keepalive=true
>worker.template_01.reply_timeout=1800000
>
>---
>
>and on the AJP backends (Jboss / EAP), we have also set the connectionTimeout to 600000 ms , as written in
>https://tomcat.apache.org/connectors-doc/common_howto/timeouts.html
>
>But even a Backend with low traffic, (maybe one request per minute) gets 300 established AJP connections after some days.
>
>Here the content of /etc/apache2/mods-enabled/jk.conf:
>
><IfModule jk_module>
>
> JkWorkersFile /etc/libapache2-mod-jk/workers.properties
> JkLogFile /var/log/apache2/mod_jk.log
> JkLogLevel info
> JkShmFile /var/log/apache2/jk-runtime-status
> JkWatchdogInterval 60
></IfModule>
>
>It looks like the JkWatchdog is not running? Or do the connections do not idle because of some non-Apache requests like
>cping/cpong?
>
>Your support is much appreciated.
>
>Thanks
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Aw: mod_jk/1.2.37 Apache/2.4.7 (Ubuntu): AJP established
connections pile up, connection_pool_timeout has no effect?
Posted by Hildegard Meier <da...@gmx.de>.
P.S. in workers.properties we use the template then with
worker.myhost.reference=worker.template_01
worker.myhost.host=10.20.30.40
worker.myhost.port=8809
> Gesendet: Mittwoch, 31. August 2016 um 14:10 Uhr
> Von: "Hildegard Meier" <da...@gmx.de>
> An: users@tomcat.apache.org
> Betreff: mod_jk/1.2.37 Apache/2.4.7 (Ubuntu): AJP established connections pile up, connection_pool_timeout has no effect?
>
> We have the following /etc/libapache2-mod-jk/workers.properties:
>
> ---
>
> # not used
> workers.tomcat_home=/usr/share/tomcat6
> # not used
> workers.java_home=/usr/lib/jvm/default-java
> ps=/
>
> # General worker template
> worker.template_01.type=ajp13
> worker.template_01.socket_connect_timeout=3000
> worker.template_01.ping_mode=A
> worker.template_01.connection_pool_timeout=600
> worker.template_01.connection_pool_minsize=1
> worker.template_01.socket_keepalive=true
> worker.template_01.reply_timeout=1800000
>
> ---
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org