mod_jk: connection_pool_timeout vs. connection_ping_interval

[Martin Knoblauch <kn...@gmail.com>]

Hi,

 this is a maybe esoteric question, but I would like to understand the
behavior of "mod_jk" when using both "connection_pool_timeout" and
"connection_ping_interval".

My setup is Apache-2.4.10, mod_jk-1.2.41-dev(r1621292) and Tomcat 7.042
(could be newer, I know).

Tomcat "server.xml" has:

    <Connector port="9397" protocol="AJP/1.3"
        connectionTimeout="600000"
        maxThreads="300"
        minSpareThreads="10"
        redirectPort="8443" />

workers.properties has:

worker.cb2tpl.prepost_timeout=15000
worker.cb2tpl.connect_timeout=15000
worker.cb2tpl.ping_timeout=15000
worker.cb2tpl.ping_mode=A
worker.cb2tpl.connection_ping_interval=180
worker.cb2tpl.connection_pool_timeout=600

So, I was [naively] assuming that established idle backend connections
would be closed after 10+ minutes. To my surprise, nothing happened. I
opened 6 backend connections and they stayed for that for hours.

So I started looking at the code. First thing I found was the code in
"jk_ajp_common.c:3334-3341" will basically disable the connection retiring
if the number of active backend connections is less than
"connection_pool_minsize"
(13 in my case). Is this correct? Makes actually some sense, but came
unexpected.

So, as a n experiment I took out the break in line 3340 and redid the
experiment. Still nothing happened after considerable time.

Then I looked at the code handling the worker keep alive (3346-3378). So,
if I understand it right, this will trigger after a few maintenance cycles
when last_accessed gets larger than the connection ping interval. Now my
question is, will performing the connection ping test reset the connection
timeout on the tomcat side? If yes, this would explain the observed
behavior: the connector connectionTimeout will never trigger if it is
larger than (connection_ping_interval + maintain_time). Actually, I think
it will only *reliably* trigger if smaller than connection_ping_interval.
Experiments seem to confirm this.

So, basically three questions:

a) are my observations correct?
b) if yes, is this the desired behavior?
c1) if if yes, what is the recommendation on sane values
connection_pool_timeout and connection_ping_interval?
c2) if not, what is the desired behavior?

Cheers
Martin

-- 
------------------------------------------------------
Martin Knoblauch
email: k n o b i AT knobisoft DOT de
www: http://www.knobisoft.de

Re: mod_jk: connection_pool_timeout vs. connection_ping_interval

[Rainer Jung <ra...@kippdata.de>]

Good findings! See below.

Am 29.08.2014 um 15:12 schrieb Martin Knoblauch:
> Hi,
>
>   this is a maybe esoteric question, but I would like to understand the
> behavior of "mod_jk" when using both "connection_pool_timeout" and
> "connection_ping_interval".
>
> My setup is Apache-2.4.10, mod_jk-1.2.41-dev(r1621292) and Tomcat 7.042
> (could be newer, I know).
>
> Tomcat "server.xml" has:
>
>      <Connector port="9397" protocol="AJP/1.3"
>          connectionTimeout="600000"
>          maxThreads="300"
>          minSpareThreads="10"
>          redirectPort="8443" />
>
> workers.properties has:
>
> worker.cb2tpl.prepost_timeout=15000
> worker.cb2tpl.connect_timeout=15000
> worker.cb2tpl.ping_timeout=15000
> worker.cb2tpl.ping_mode=A
> worker.cb2tpl.connection_ping_interval=180
> worker.cb2tpl.connection_pool_timeout=600
>
> So, I was [naively] assuming that established idle backend connections
> would be closed after 10+ minutes. To my surprise, nothing happened. I
> opened 6 backend connections and they stayed for that for hours.

> So I started looking at the code. First thing I found was the code in
> "jk_ajp_common.c:3334-3341" will basically disable the connection retiring
> if the number of active backend connections is less than
> "connection_pool_minsize"
> (13 in my case). Is this correct? Makes actually some sense, but came
> unexpected.

The best description probably can be found on

http://tomcat.apache.org/connectors-doc/generic_howto/timeouts.html#Connection%20Pools%20and%20Idle%20Timeouts

which I wrote a couple of years ago.

- yes, "connection_pool_minsize" is the minimum number of connections to 
which mod_jk will shrink the pool. If you want to close as many 
connections as possible set to "0".

- pool shrinking is done in the internal maintenance method. Until 
1.2.26 that method only ran when a request was being processed. Since 
connection pools are per web server process, it could well happen, that 
some forked processes didn't work on a request for a long time and their 
connections were then kept open even much longer than 
connection_pool_timeout.

- starting with 1.2.27 there's a separate optional watchdog thread, that 
you can and should activate with JkWatchdogInterval, see 
http://tomcat.apache.org/connectors-doc/reference/apache.html.

- all this works even without doing cping on idle connections.

- again starting with 1.2.27 there's a new feature that will check idle 
connections from time to time, whether they still work. It can be seen 
as an addition to checking at the moment a request comes in, because if 
you check in advance and find hanging or buggy connections, you save 
some latency and error handling for real requests. This feature is 
activated by setting ping_mode to "A" or at least "I" and can be 
controlled by ping_timeout (how long to wait for a cpong response) and 
connection_ping_interval. If the feature is activated, connections which 
are to fresh for connection_pool_timeout but already longer idle than 
the connection_ping_interval, are checked by a cping/cpong whether they 
still work (and close if not). This check will update the last access 
timer though, so connections checked with cping interval check will 
typicaly never be closed due to connection_pool_timeout (which I just 
realized right now).

> So, as a n experiment I took out the break in line 3340 and redid the
> experiment. Still nothing happened after considerable time.

Maybe the connections you observed belonged to processes that didn't 
have to handle requests and the watchdg thrad was off as it is by default.

> Then I looked at the code handling the worker keep alive (3346-3378). So,
> if I understand it right, this will trigger after a few maintenance cycles
> when last_accessed gets larger than the connection ping interval. Now my
> question is, will performing the connection ping test reset the connection
> timeout on the tomcat side?

I had a quick look at least at Tomcat 7 code for the BIO AJP connector, 
the default for TC 7, and yes: CPING is a normal first message and it 
will reset the connectionTimeout timer.

> If yes, this would explain the observed
> behavior: the connector connectionTimeout will never trigger if it is
> larger than (connection_ping_interval + maintain_time). Actually, I think
> it will only *reliably* trigger if smaller than connection_ping_interval.
> Experiments seem to confirm this.

Agreed, currently you need to choose between one of the two features 
"interval cping" or "pool timeout".

> So, basically three questions:
>
> a) are my observations correct?

Yes with some additions above.

> b) if yes, is this the desired behavior?

I'd say "no". Interval cping was added much later and we didn't probably 
think about its implications on the keep-alive status and timeout of the 
connection.

On the mod_jk side fixing this would be trivial, by not updating 
last_access after the cping in the ajp_maintain method. On the Tomcat 
side we would need to adjust the socket timeout while waiting for the 
first packet of the next request a bit more dynamically. Likely possible 
with a few lines of code.

> c1) if if yes, what is the recommendation on sane values
> connection_pool_timeout and connection_ping_interval?

In the meantime: I'd say in most situations a pool timeout is more 
important than interval cping. connection_pool_timeout values often are 
between 60 and 600 seconds.

> c2) if not, what is the desired behavior?

IMHO interval cping should not reset the keep alive timeout.

Regards,

Rainer


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org