You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Christopher Schultz <ch...@christopherschultz.net> on 2025/10/30 14:06:35 UTC

Getting intermittent mod_jk errors on 1 / 3 web servers

All,

I've got 3 web servers in AWS EC2 connected to a pair of back-end Tomcat 
instances using mod_jk. I'm also using stunnel. So the connections look 
like this:

ALB -> web server -> [AJP over stunnel] -> Tomcat AjpNioProtocol

One of these web servers is newly-built and it seems to be suffering 
from connection errors. This is a load-balancer connection using sticky 
sessions to the back-end nodes. The jk-status page shows e.g. a small 
number of errors:

Name	Act	State	Err
node1	ACT	OK	5
node2	ACT	OK	6

Note: these are NOT "client errors". I pretty much always ignore those.

This number is small because I reset the balancer member stats this 
morning to get a better handle on how often the errors occur. It's not 
all that often, but the other two web servers are pretty much never 
registering ANY errors. So this is definitely a problem I'd like to solve.

Some notable differences between the existing web servers and the new one:

1. Old web servers are x86-64 based, the new one is aarch64
2. New web server goes through an AWS NAT gateway for IPv4

The new web server has only IPv6 public, and uses a NAT gateway to get 
out to the internet over IPv4. The other two web servers have public 
IPv4 addresses. Neither back-end server has IPv6, so all communication 
for AJP/stunnel is over IPv4.

The mod_jk log contains logs like these:

[Thu Oct 30 13:46:15.893 2025] [aQNsJ08YUCb0jK7JrtZe3gAAAAA] 
[1447:281472310505664] [info] 
ajp_connection_tcp_get_message::jk_ajp_common.c (1376): (node1) can't 
receive the response header message from tomcat, network problems or 
tomcat (127.0.0.1:7015) is down (errno=104)

[Thu Oct 30 13:46:15.893 2025] [aQNsJ08YUCb0jK7JrtZe3gAAAAA] 
[1447:281472310505664] [error] ajp_get_reply::jk_ajp_common.c (2346): 
(node1) Tomcat is down or refused connection. No response has been sent 
to the client (yet)

[Thu Oct 30 13:46:15.893 2025] [aQNsJ08YUCb0jK7JrtZe3gAAAAA] 
[1447:281472310505664] [info] ajp_service::jk_ajp_common.c (2892): 
(node1) sending request to tomcat failed (recoverable),  (attempt=1)

[Thu Oct 30 13:46:15.995 2025] [aQNsJ08YUCb0jK7JrtZe3gAAAAA] 
[1447:281472310505664] [info] 
ajp_connection_tcp_get_message::jk_ajp_common.c (1376): (node1) can't 
receive the response header message from tomcat, network problems or 
tomcat (127.0.0.1:7015) is down (errno=104)

[Thu Oct 30 13:46:15.995 2025] [aQNsJ08YUCb0jK7JrtZe3gAAAAA] 
[1447:281472310505664] [error] ajp_get_reply::jk_ajp_common.c (2346): 
(node1) Tomcat is down or refused connection. No response has been sent 
to the client (yet)

[Thu Oct 30 13:46:15.995 2025] [aQNsJ08YUCb0jK7JrtZe3gAAAAA] 
[1447:281472310505664] [info] ajp_service::jk_ajp_common.c (2892): 
(node1) sending request to tomcat failed (recoverable),  (attempt=2)

[Thu Oct 30 13:46:15.995 2025] [aQNsJ08YUCb0jK7JrtZe3gAAAAA] 
[1447:281472310505664] [error] ajp_service::jk_ajp_common.c (2913): 
(node1) connecting to tomcat failed (rc=0, errors=6, client_errors=2).

[Thu Oct 30 13:46:15.997 2025] [aQNsJ08YUCb0jK7JrtZe3gAAAAA] 
[1447:281472310505664] [info] service::jk_lb_worker.c (1602): service 
failed, worker node1 is in local error state

127.0.0.1:7015 is the port number where stunnel is listening.

stunnel contains these logs:

Oct 30 13:46:15 ip-10-2-0-166.ec2.internal stunnel[1444]: LOG3[423]: 
SSL_read: ssl/record/rec_layer_s3.c:689: error:0A000126:SSL 
routines::unexpected eof while reading

Oct 30 13:46:15 ip-10-2-0-166.ec2.internal stunnel[1444]: LOG3[373]: 
SSL_read: ssl/record/rec_layer_s3.c:689: error:0A000126:SSL 
routines::unexpected eof while reading

Oct 30 13:46:15 ip-10-2-0-166.ec2.internal stunnel[1444]: LOG3[375]: 
SSL_read: ssl/record/rec_layer_s3.c:689: error:0A000126:SSL 
routines::unexpected eof while reading

Oct 30 13:46:15 ip-10-2-0-166.ec2.internal stunnel[1444]: LOG3[374]: 
SSL_read: ssl/record/rec_layer_s3.c:689: error:0A000126:SSL 
routines::unexpected eof while reading

Given that these timestamps are correlated, it seems that they are 
reporting the same event.

When stunnel reports "unexpected eof" it typically means that the remote 
server (or some network gear) closed the connection without tearing-down 
the TLS connection cleanly.

That symptom, plus the "this is the only server using a NAT gateway" 
would surely point to one place: the NAT gateway is killing connections 
that are idle and surprising both stunnel and mod_jk. I can also see a 
graph of non-zero numbers of "Idle Timeouts" on the NAT gateway. It 
doesn't tell me more details about those timeouts, but they are almost 
certainly outgoing AJP/stunnel connections.

But.

Here is my mod_jk workers configuration:

# Template worker
worker.template.type=ajp13
worker.template.host=localhost
worker.template.connection_pool_timeout=60
worker.template.socket_timeout=300
worker.template.max_packet_size=65536

worker.node1.reference=worker.template
worker.node1.port=7015
worker.node1.route=node1

My expectation is that connection_pool_timeout of 60 (seconds) will 
close connections which have been idle for 60 seconds. If mod_jk closes 
a connection, stunnel will also close that connection. (Note: I have no 
explicit connectionTimeout or keepAliveTimeout on the Tomcat side. But 
this doesn't seem to be any problem for the other two web servers.)

Checking my configuration for the NAT gateway, it has a fixed idle 
timeout of 350 seconds, which is much longer than the 60 seconds I 
(believe I) have set for idle AJP connections.

I do not use servlet async or Websocket for anything in my application, 
so I do not expect long-lasting connections between client and server.

Is there anything I haven't checked at this point?

-chris


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Re: [EXTERNAL] Getting intermittent mod_jk errors on 1 / 3 web servers

Posted by Joey Cochran <Jo...@mtsu.edu>.
Just taking a stab.... these in the $CATALINA_BASE/conf/web.xml  ...might be something to ensure

    <filter>
        <filter-name>httpHeaderSecurity</filter-name>
        <filter-class>org.apache.catalina.filters.HttpHeaderSecurityFilter</filter-class>
        <async-supported>true</async-supported>
        <init-param>
                <param-name>antiClickJackingEnabled</param-name>
                <param-value>false</param-value>
        </init-param>
    </filter>

    <filter>
        <filter-name>cors</filter-name>
        <filter-class>org.apache.catalina.filters.CorsFilter</filter-class>
        <async-supported>true</async-supported>
        <init-param>
                <param-name>cors.allowed.origins</param-name>
                <param-value>https://youDrCrossDomainDependecy.com</param-value>
                <param-value>https://yourOtherCrossDomainDependency.com</param-value>
        </init-param>
        <init-param>
                <param-name>cors.allowed.methods</param-name>
                <param-value>GET,POST,HEAD,OPTIONS,PUT</param-value>
        </init-param>
        <init-param>
                <param-name>cors.allowed.headers</param-name>
                <param-value>Content-Type,X-Requested-With,accept,Origin,Access-Control-Request-Method,Access-Control-Request-Headers</param-value>
        </init-param>
        <init-param>
                <param-name>cors.exposed.headers</param-name>
                <param-value>Access-Control-Allow-Origin,Access-Control-Allow-Credentials</param-value>
        </init-param>
        <init-param>
                <param-name>cors.support.credentials</param-name>
                <param-value>true</param-value>
        </init-param>
        <init-param>
                <param-name>cors.preflight.maxage</param-name>
                <param-value>10</param-value>
        </init-param>
    </filter>
    <filter-mapping>

    <filter-mapping>
        <filter-name>httpHeaderSecurity</filter-name>
        <url-pattern>/*</url-pattern>
        <dispatcher>REQUEST</dispatcher>
    </filter-mapping>

Hope this helps...


Thanks!

-Joey



           [cid:3578188d-8568-4324-8ab3-47cdee98ad1d]

Joey Cochran

Systems Administrator II

Middleware Developer

Information Technology Division

Middle Tennessee State University

615-898-2714

www.mtsu.edu





________________________________
From: Christopher Schultz <ch...@christopherschultz.net>
Sent: Thursday, October 30, 2025 9:06 AM
To: Tomcat Users List <us...@tomcat.apache.org>
Subject: [EXTERNAL] Getting intermittent mod_jk errors on 1 / 3 web servers

All,

I've got 3 web servers in AWS EC2 connected to a pair of back-end Tomcat
instances using mod_jk. I'm also using stunnel. So the connections look
like this:

ALB -> web server -> [AJP over stunnel] -> Tomcat AjpNioProtocol

One of these web servers is newly-built and it seems to be suffering
from connection errors. This is a load-balancer connection using sticky
sessions to the back-end nodes. The jk-status page shows e.g. a small
number of errors:

Name    Act     State   Err
node1   ACT     OK      5
node2   ACT     OK      6

Note: these are NOT "client errors". I pretty much always ignore those.

This number is small because I reset the balancer member stats this
morning to get a better handle on how often the errors occur. It's not
all that often, but the other two web servers are pretty much never
registering ANY errors. So this is definitely a problem I'd like to solve.

Some notable differences between the existing web servers and the new one:

1. Old web servers are x86-64 based, the new one is aarch64
2. New web server goes through an AWS NAT gateway for IPv4

The new web server has only IPv6 public, and uses a NAT gateway to get
out to the internet over IPv4. The other two web servers have public
IPv4 addresses. Neither back-end server has IPv6, so all communication
for AJP/stunnel is over IPv4.

The mod_jk log contains logs like these:

[Thu Oct 30 13:46:15.893 2025] [aQNsJ08YUCb0jK7JrtZe3gAAAAA]
[1447:281472310505664] [info]
ajp_connection_tcp_get_message::jk_ajp_common.c (1376): (node1) can't
receive the response header message from tomcat, network problems or
tomcat (127.0.0.1:7015) is down (errno=104)

[Thu Oct 30 13:46:15.893 2025] [aQNsJ08YUCb0jK7JrtZe3gAAAAA]
[1447:281472310505664] [error] ajp_get_reply::jk_ajp_common.c (2346):
(node1) Tomcat is down or refused connection. No response has been sent
to the client (yet)

[Thu Oct 30 13:46:15.893 2025] [aQNsJ08YUCb0jK7JrtZe3gAAAAA]
[1447:281472310505664] [info] ajp_service::jk_ajp_common.c (2892):
(node1) sending request to tomcat failed (recoverable),  (attempt=1)

[Thu Oct 30 13:46:15.995 2025] [aQNsJ08YUCb0jK7JrtZe3gAAAAA]
[1447:281472310505664] [info]
ajp_connection_tcp_get_message::jk_ajp_common.c (1376): (node1) can't
receive the response header message from tomcat, network problems or
tomcat (127.0.0.1:7015) is down (errno=104)

[Thu Oct 30 13:46:15.995 2025] [aQNsJ08YUCb0jK7JrtZe3gAAAAA]
[1447:281472310505664] [error] ajp_get_reply::jk_ajp_common.c (2346):
(node1) Tomcat is down or refused connection. No response has been sent
to the client (yet)

[Thu Oct 30 13:46:15.995 2025] [aQNsJ08YUCb0jK7JrtZe3gAAAAA]
[1447:281472310505664] [info] ajp_service::jk_ajp_common.c (2892):
(node1) sending request to tomcat failed (recoverable),  (attempt=2)

[Thu Oct 30 13:46:15.995 2025] [aQNsJ08YUCb0jK7JrtZe3gAAAAA]
[1447:281472310505664] [error] ajp_service::jk_ajp_common.c (2913):
(node1) connecting to tomcat failed (rc=0, errors=6, client_errors=2).

[Thu Oct 30 13:46:15.997 2025] [aQNsJ08YUCb0jK7JrtZe3gAAAAA]
[1447:281472310505664] [info] service::jk_lb_worker.c (1602): service
failed, worker node1 is in local error state

127.0.0.1:7015 is the port number where stunnel is listening.

stunnel contains these logs:

Oct 30 13:46:15 ip-10-2-0-166.ec2.internal stunnel[1444]: LOG3[423]:
SSL_read: ssl/record/rec_layer_s3.c:689: error:0A000126:SSL
routines::unexpected eof while reading

Oct 30 13:46:15 ip-10-2-0-166.ec2.internal stunnel[1444]: LOG3[373]:
SSL_read: ssl/record/rec_layer_s3.c:689: error:0A000126:SSL
routines::unexpected eof while reading

Oct 30 13:46:15 ip-10-2-0-166.ec2.internal stunnel[1444]: LOG3[375]:
SSL_read: ssl/record/rec_layer_s3.c:689: error:0A000126:SSL
routines::unexpected eof while reading

Oct 30 13:46:15 ip-10-2-0-166.ec2.internal stunnel[1444]: LOG3[374]:
SSL_read: ssl/record/rec_layer_s3.c:689: error:0A000126:SSL
routines::unexpected eof while reading

Given that these timestamps are correlated, it seems that they are
reporting the same event.

When stunnel reports "unexpected eof" it typically means that the remote
server (or some network gear) closed the connection without tearing-down
the TLS connection cleanly.

That symptom, plus the "this is the only server using a NAT gateway"
would surely point to one place: the NAT gateway is killing connections
that are idle and surprising both stunnel and mod_jk. I can also see a
graph of non-zero numbers of "Idle Timeouts" on the NAT gateway. It
doesn't tell me more details about those timeouts, but they are almost
certainly outgoing AJP/stunnel connections.

But.

Here is my mod_jk workers configuration:

# Template worker
worker.template.type=ajp13
worker.template.host=localhost
worker.template.connection_pool_timeout=60
worker.template.socket_timeout=300
worker.template.max_packet_size=65536

worker.node1.reference=worker.template
worker.node1.port=7015
worker.node1.route=node1

My expectation is that connection_pool_timeout of 60 (seconds) will
close connections which have been idle for 60 seconds. If mod_jk closes
a connection, stunnel will also close that connection. (Note: I have no
explicit connectionTimeout or keepAliveTimeout on the Tomcat side. But
this doesn't seem to be any problem for the other two web servers.)

Checking my configuration for the NAT gateway, it has a fixed idle
timeout of 350 seconds, which is much longer than the 60 seconds I
(believe I) have set for idle AJP connections.

I do not use servlet async or Websocket for anything in my application,
so I do not expect long-lasting connections between client and server.

Is there anything I haven't checked at this point?

-chris


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Re: Getting intermittent mod_jk errors on 1 / 3 web servers

Posted by Christopher Schultz <ch...@christopherschultz.net>.
Mark,

On 10/31/25 4:04 AM, Mark Thomas wrote:
> On 30/10/2025 14:06, Christopher Schultz wrote:
> 
> <snip/>
> 
>> That symptom, plus the "this is the only server using a NAT gateway" 
>> would surely point to one place: the NAT gateway is killing 
>> connections that are idle and surprising both stunnel and mod_jk. I 
>> can also see a graph of non-zero numbers of "Idle Timeouts" on the NAT 
>> gateway. It doesn't tell me more details about those timeouts, but 
>> they are almost certainly outgoing AJP/stunnel connections.
> 
> Your reasoning above looks sound to me.

Thanks for the second brain's opinion on this.

>> But.
>>
>> Here is my mod_jk workers configuration:
>>
>> # Template worker
>> worker.template.type=ajp13
>> worker.template.host=localhost
>> worker.template.connection_pool_timeout=60
>> worker.template.socket_timeout=300
>> worker.template.max_packet_size=65536
>>
>> worker.node1.reference=worker.template
>> worker.node1.port=7015
>> worker.node1.route=node1
>>
>> My expectation is that connection_pool_timeout of 60 (seconds) will 
>> close connections which have been idle for 60 seconds. If mod_jk 
>> closes a connection, stunnel will also close that connection. (Note: I 
>> have no explicit connectionTimeout or keepAliveTimeout on the Tomcat 
>> side. But this doesn't seem to be any problem for the other two web 
>> servers.)
>>
>> Checking my configuration for the NAT gateway, it has a fixed idle 
>> timeout of 350 seconds, which is much longer than the 60 seconds I 
>> (believe I) have set for idle AJP connections.
>>
>> I do not use servlet async or Websocket for anything in my 
>> application, so I do not expect long-lasting connections between 
>> client and server.
>>
>> Is there anything I haven't checked at this point?
> 
> You might want to check how connection_pool_timeout interacts with 
> connection_pool_size and connection_pool_minsize.

My current connection_pool_size and connection_pool_minsize are the 
defaults, so I suspect they will be:

connection_pool_size = ThreadsPerChild = 25

connection_pool_minsize = 25 + 1 / 2 = 13

So, basically, default everything.

Looking at ps, I see that there are 6 httpd processes running as the 
apache user, and one running as root (the control process).

The mod_status page says I have 124 idle workers and 1 in-flight 
request. So believe I do in fact have the default configuration of 25 
threads per process; with 6 processes I have 125 total threads.

netstat tells me that I currently have 112 ESTABLISHED connections owned 
by stunnel.

> I am wondering if the current size is at minsize will an idle connection 
> be closed if it exceeds the timeout?
> 
> I think the code that handles this is here:
> 
> https://github.com/apache/tomcat-connectors/blob/main/native/common/ 
> jk_ajp_common.c#L3510
 >
> and I think it only closes idle connections until the pool reaches
 > minsize.

I'll have to do a lot of reading to determine how that's going to 
behave. So many configuration properties have changed names over time, 
it's not 100% clear how the code will behave, but ...

https://github.com/apache/tomcat-connectors/blob/main/native/common/jk_ajp_common.c#L3567

This certainly seems like it will only "clean" connections until it gets 
to the min size.

> Which isn't what the docs suggest. Which in turn might explain why you 
> are seeing connections open longer than the NAT gateway timeout.
> 
> Rainer understands this code far better than I do. It would be good to 
> get his view on this. If I am right, I think we either need to update 
> the docs or we need to fix the code so idle connections below minsize 
> are closed and then re-opened to refresh them.
So it looks like I have a few options for immediate relief, here, though 
if there is a bug (or missing feature) in mod_jk, then I think that's 
the best long-term solution for me: I'd like AJP connections that have 
been open for "too long" to simply be recycled.

1. Set connection_pool_minsize=0. This seems ... potentially problematic?

2. Set socket_keepalive on the worker. The documentation is not 
encouraging, as it suggests that the keepalives may be send on the order 
of hours and not minutes.

3. Use stunnel's TIMEOUTidle=300 (or similar) to get stunnel to kill 
idle connections after too much time has passed (but less than the NAT 
idle timeout). I think this just moves the problem from the NAT router 
surprising stunnel and mod_jk to stunnel surprising mod_jk.

4. Use stunnel's keepalive capabilities.

socket = l:SO_KEEPALIVE=1
socket = r:SO_KEEPALIVE=1

I believe this is similar to mod_jk's keepalives, which rely on the OS. 
Modifying the global OS settings for how keepalives behave doesn't feel 
right, so I'm going to drop these two options from my list.

5. Set Tomcat's keepAliveTimeout instead of leaving it at the default. I 
think this will close the connection but mod_jk won't know about it 
until it attempts another write. (Right?)

4. Use ping_mode=P or ping_mode=I with an appropriate 
connection_ping_interval and ping_timeout. This seems the most 
promising, because it will probe the connection before using it.

This kind of happens already, because when mod_jk attempts to use a 
connection and discovers it's unusable, it will roll-over to another 
connection. But I think it will only do it 3 times before it (1) gives 
up and (2) marks the worker as ERR.

If I use ping_mode=P or I, will this change the above behavior at all? I 
would like mod_jk in this case to ping each connection, discover it's 
unusable, and recycle it but not mark the worker as being in the ERR 
state, even if mod_jk needs to recycle a whole bunch of connections in 
order to find one that works.

Are there any other options I haven't identified above?

I have other nuclear options, including allocating a public IP address 
for this instance and using that directly. My goal was actually to push 
everything behind this NAT router for a couple of reasons, so I would 
like to figure out how to get it working.

As the subject says, I do have 2 other web servers, so I can experiment 
with this one, including putting it in debug/trace log mode, removing it 
from the hw load-balancer, poking it with traffic only generated by me, etc.

-chris


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Re: Getting intermittent mod_jk errors on 1 / 3 web servers

Posted by Christopher Schultz <ch...@christopherschultz.net>.
Rainer,

On 10/31/25 1:08 PM, Rainer Jung wrote:
> Am 31.10.25 um 18:02 schrieb Rainer Jung:
>> Am 31.10.25 um 09:04 schrieb Mark Thomas:
>>> On 30/10/2025 14:06, Christopher Schultz wrote:
>>>
>>> <snip/>
>>>
>>>> That symptom, plus the "this is the only server using a NAT gateway" 
>>>> would surely point to one place: the NAT gateway is killing 
>>>> connections that are idle and surprising both stunnel and mod_jk. I 
>>>> can also see a graph of non-zero numbers of "Idle Timeouts" on the 
>>>> NAT gateway. It doesn't tell me more details about those timeouts, 
>>>> but they are almost certainly outgoing AJP/stunnel connections.
>>>
>>> Your reasoning above looks sound to me.
>>>
>>>> But.
>>>>
>>>> Here is my mod_jk workers configuration:
>>>>
>>>> # Template worker
>>>> worker.template.type=ajp13
>>>> worker.template.host=localhost
>>>> worker.template.connection_pool_timeout=60
>>>> worker.template.socket_timeout=300
>>>> worker.template.max_packet_size=65536
>>>>
>>>> worker.node1.reference=worker.template
>>>> worker.node1.port=7015
>>>> worker.node1.route=node1
>>>>
>>>> My expectation is that connection_pool_timeout of 60 (seconds) will 
>>>> close connections which have been idle for 60 seconds. If mod_jk 
>>>> closes a connection, stunnel will also close that connection. (Note: 
>>>> I have no explicit connectionTimeout or keepAliveTimeout on the 
>>>> Tomcat side. But this doesn't seem to be any problem for the other 
>>>> two web servers.)
>>>>
>>>> Checking my configuration for the NAT gateway, it has a fixed idle 
>>>> timeout of 350 seconds, which is much longer than the 60 seconds I 
>>>> (believe I) have set for idle AJP connections.
>>>>
>>>> I do not use servlet async or Websocket for anything in my 
>>>> application, so I do not expect long-lasting connections between 
>>>> client and server.
>>>>
>>>> Is there anything I haven't checked at this point?
>>>
>>> You might want to check how connection_pool_timeout interacts with 
>>> connection_pool_size and connection_pool_minsize.
>>>
>>> I am wondering if the current size is at minsize will an idle 
>>> connection be closed if it exceeds the timeout?
>>>
>>> I think the code that handles this is here:
>>>
>>> https://github.com/apache/tomcat-connectors/blob/main/native/common/ 
>>> jk_ajp_common.c#L3510
>>>
>>> and I think it only closes idle connections until the pool reaches 
>>> minsize.
>>>
>>> Which isn't what the docs suggest. Which in turn might explain why 
>>> you are seeing connections open longer than the NAT gateway timeout.
>>>
>>> Rainer understands this code far better than I do. It would be good 
>>> to get his view on this. If I am right, I think we either need to 
>>> update the docs or we need to fix the code so idle connections below 
>>> minsize are closed and then re-opened to refresh them.
>>>
>>> Mark
>>
>> It has been quite a long time that I touched this code. But Mark is 
>> right, idle connections will not get closed below the min pool size.
>>
>> Most of the code default values for configuration never reflected the 
>> code progress and were kept to keep old behavior in place. I typically 
>> suggest to start with the config from here:
>>
>> https://github.com/apache/tomcat-connectors/blob/main/conf/ 
>> workers.properties
>>
>> That should have better default params (like min pool size 0) and 
>> comments. See also the other files in that directory.
>>
>> I like most of our jk timeouts and the cping/cpong, but I generally do 
>> not like the socket_timeout, because it is a complex to understand low 
>> level timeout which can kick in in any communication context.
>>
>> In addition when doing observation/monitoring keep in mind, that the 
>> connection pools are per child process, not per Apache httpd instance.
> 
> One other thing: the idle timeout is not tracked continuously but only 
> checked during the maintenance run. Generally that happens once per 
> minute, but hooked on request processing (no requests, no maintenance 
> runs), but nowadays you can (but have to) configure using the Apache 
> httpd watchdog thread to run it even when no requests are handled.

It looks like ping_mode=I plus the watchdog can help significantly.

What would happen if I set the connection_pool_minsize to something very 
low like ... zero? This would allow a watchdog wakeup to completely 
remove all connections which had been invalidated.

And, I believe, connections removed this way won't trip the 
failure-detector which sets the worker into the ERR state, right?

If I set ping_mode=I and there are zero request handled by the server 
for like 10 minutes, will any connection from the pool be tested as if 
ping_mode=P since the idle-timeout would have passed? When we have a lot 
of requests, testing the viability of a connection for every request 
seems wasteful.

When cping/cpong fails, does it have any effect on the status of the 
worker? In a terrible case, I might expect that all ~125 connections 
have gone stale. When a new request comes in, it will fail cping and 
then ... what happens? Do we just need trying connections in the pool 
until we get one that works? And if the pool is exhausted we create new 
connections? If so, that's great, and the behavior I'd like.

What I'm really trying to avoid is a situation where I have a connection 
pool with a bunch of stale connections but new connections would 
succeed, and servicing a request hits these stale connections, killing 
the whole worker. Since I'm using sticky sessions, this basically 
logs-out 1/6 of my users (3 web servers, 2 back-end workers) when it 
happens *to a single request*.

How can I make this setup bullet-proof? I don't need to be able to 
handle a million requests per second, and I don't need to worry about 
complicated fail-over situations, etc. I just want to clean-out these 
stale connections so they don't cause this kind of error.

-chris


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Re: Getting intermittent mod_jk errors on 1 / 3 web servers

Posted by Rainer Jung <ra...@kippdata.de>.
Am 31.10.25 um 18:02 schrieb Rainer Jung:
> Am 31.10.25 um 09:04 schrieb Mark Thomas:
>> On 30/10/2025 14:06, Christopher Schultz wrote:
>>
>> <snip/>
>>
>>> That symptom, plus the "this is the only server using a NAT gateway" 
>>> would surely point to one place: the NAT gateway is killing 
>>> connections that are idle and surprising both stunnel and mod_jk. I 
>>> can also see a graph of non-zero numbers of "Idle Timeouts" on the 
>>> NAT gateway. It doesn't tell me more details about those timeouts, 
>>> but they are almost certainly outgoing AJP/stunnel connections.
>>
>> Your reasoning above looks sound to me.
>>
>>> But.
>>>
>>> Here is my mod_jk workers configuration:
>>>
>>> # Template worker
>>> worker.template.type=ajp13
>>> worker.template.host=localhost
>>> worker.template.connection_pool_timeout=60
>>> worker.template.socket_timeout=300
>>> worker.template.max_packet_size=65536
>>>
>>> worker.node1.reference=worker.template
>>> worker.node1.port=7015
>>> worker.node1.route=node1
>>>
>>> My expectation is that connection_pool_timeout of 60 (seconds) will 
>>> close connections which have been idle for 60 seconds. If mod_jk 
>>> closes a connection, stunnel will also close that connection. (Note: 
>>> I have no explicit connectionTimeout or keepAliveTimeout on the 
>>> Tomcat side. But this doesn't seem to be any problem for the other 
>>> two web servers.)
>>>
>>> Checking my configuration for the NAT gateway, it has a fixed idle 
>>> timeout of 350 seconds, which is much longer than the 60 seconds I 
>>> (believe I) have set for idle AJP connections.
>>>
>>> I do not use servlet async or Websocket for anything in my 
>>> application, so I do not expect long-lasting connections between 
>>> client and server.
>>>
>>> Is there anything I haven't checked at this point?
>>
>> You might want to check how connection_pool_timeout interacts with 
>> connection_pool_size and connection_pool_minsize.
>>
>> I am wondering if the current size is at minsize will an idle 
>> connection be closed if it exceeds the timeout?
>>
>> I think the code that handles this is here:
>>
>> https://github.com/apache/tomcat-connectors/blob/main/native/common/ 
>> jk_ajp_common.c#L3510
>>
>> and I think it only closes idle connections until the pool reaches 
>> minsize.
>>
>> Which isn't what the docs suggest. Which in turn might explain why you 
>> are seeing connections open longer than the NAT gateway timeout.
>>
>> Rainer understands this code far better than I do. It would be good to 
>> get his view on this. If I am right, I think we either need to update 
>> the docs or we need to fix the code so idle connections below minsize 
>> are closed and then re-opened to refresh them.
>>
>> Mark
> 
> It has been quite a long time that I touched this code. But Mark is 
> right, idle connections will not get closed below the min pool size.
> 
> Most of the code default values for configuration never reflected the 
> code progress and were kept to keep old behavior in place. I typically 
> suggest to start with the config from here:
> 
> https://github.com/apache/tomcat-connectors/blob/main/conf/ 
> workers.properties
> 
> That should have better default params (like min pool size 0) and 
> comments. See also the other files in that directory.
> 
> I like most of our jk timeouts and the cping/cpong, but I generally do 
> not like the socket_timeout, because it is a complex to understand low 
> level timeout which can kick in in any communication context.
> 
> In addition when doing observation/monitoring keep in mind, that the 
> connection pools are per child process, not per Apache httpd instance.

One other thing: the idle timeout is not tracked continuously but only 
checked during the maintenance run. Generally that happens once per 
minute, but hooked on request processing (no requests, no maintenance 
runs), but nowadays you can (but have to) configure using the Apache 
httpd watchdog thread to run it even when no requests are handled.

Two docs pages which should contain most f the things I explained on 
list a few years ago are:

https://tomcat.apache.org/connectors-doc/common_howto/timeouts.html

and

https://tomcat.apache.org/connectors-doc/common_howto/proxy.html

(and of course also the other few jk docs pages).

Regards,

Rainer


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Re: Getting intermittent mod_jk errors on 1 / 3 web servers

Posted by Rainer Jung <ra...@kippdata.de>.
Am 31.10.25 um 09:04 schrieb Mark Thomas:
> On 30/10/2025 14:06, Christopher Schultz wrote:
> 
> <snip/>
> 
>> That symptom, plus the "this is the only server using a NAT gateway" 
>> would surely point to one place: the NAT gateway is killing 
>> connections that are idle and surprising both stunnel and mod_jk. I 
>> can also see a graph of non-zero numbers of "Idle Timeouts" on the NAT 
>> gateway. It doesn't tell me more details about those timeouts, but 
>> they are almost certainly outgoing AJP/stunnel connections.
> 
> Your reasoning above looks sound to me.
> 
>> But.
>>
>> Here is my mod_jk workers configuration:
>>
>> # Template worker
>> worker.template.type=ajp13
>> worker.template.host=localhost
>> worker.template.connection_pool_timeout=60
>> worker.template.socket_timeout=300
>> worker.template.max_packet_size=65536
>>
>> worker.node1.reference=worker.template
>> worker.node1.port=7015
>> worker.node1.route=node1
>>
>> My expectation is that connection_pool_timeout of 60 (seconds) will 
>> close connections which have been idle for 60 seconds. If mod_jk 
>> closes a connection, stunnel will also close that connection. (Note: I 
>> have no explicit connectionTimeout or keepAliveTimeout on the Tomcat 
>> side. But this doesn't seem to be any problem for the other two web 
>> servers.)
>>
>> Checking my configuration for the NAT gateway, it has a fixed idle 
>> timeout of 350 seconds, which is much longer than the 60 seconds I 
>> (believe I) have set for idle AJP connections.
>>
>> I do not use servlet async or Websocket for anything in my 
>> application, so I do not expect long-lasting connections between 
>> client and server.
>>
>> Is there anything I haven't checked at this point?
> 
> You might want to check how connection_pool_timeout interacts with 
> connection_pool_size and connection_pool_minsize.
> 
> I am wondering if the current size is at minsize will an idle connection 
> be closed if it exceeds the timeout?
> 
> I think the code that handles this is here:
> 
> https://github.com/apache/tomcat-connectors/blob/main/native/common/ 
> jk_ajp_common.c#L3510
> 
> and I think it only closes idle connections until the pool reaches minsize.
> 
> Which isn't what the docs suggest. Which in turn might explain why you 
> are seeing connections open longer than the NAT gateway timeout.
> 
> Rainer understands this code far better than I do. It would be good to 
> get his view on this. If I am right, I think we either need to update 
> the docs or we need to fix the code so idle connections below minsize 
> are closed and then re-opened to refresh them.
> 
> Mark

It has been quite a long time that I touched this code. But Mark is 
right, idle connections will not get closed below the min pool size.

Most of the code default values for configuration never reflected the 
code progress and were kept to keep old behavior in place. I typically 
suggest to start with the config from here:

https://github.com/apache/tomcat-connectors/blob/main/conf/workers.properties

That should have better default params (like min pool size 0) and 
comments. See also the other files in that directory.

I like most of our jk timeouts and the cping/cpong, but I generally do 
not like the socket_timeout, because it is a complex to understand low 
level timeout which can kick in in any communication context.

In addition when doing observation/monitoring keep in mind, that the 
connection pools are per child process, not per Apache httpd instance.

Hth a bit.

Rainer


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Re: Getting intermittent mod_jk errors on 1 / 3 web servers

Posted by Mark Thomas <ma...@apache.org>.
On 30/10/2025 14:06, Christopher Schultz wrote:

<snip/>

> That symptom, plus the "this is the only server using a NAT gateway" 
> would surely point to one place: the NAT gateway is killing connections 
> that are idle and surprising both stunnel and mod_jk. I can also see a 
> graph of non-zero numbers of "Idle Timeouts" on the NAT gateway. It 
> doesn't tell me more details about those timeouts, but they are almost 
> certainly outgoing AJP/stunnel connections.

Your reasoning above looks sound to me.

> But.
> 
> Here is my mod_jk workers configuration:
> 
> # Template worker
> worker.template.type=ajp13
> worker.template.host=localhost
> worker.template.connection_pool_timeout=60
> worker.template.socket_timeout=300
> worker.template.max_packet_size=65536
> 
> worker.node1.reference=worker.template
> worker.node1.port=7015
> worker.node1.route=node1
> 
> My expectation is that connection_pool_timeout of 60 (seconds) will 
> close connections which have been idle for 60 seconds. If mod_jk closes 
> a connection, stunnel will also close that connection. (Note: I have no 
> explicit connectionTimeout or keepAliveTimeout on the Tomcat side. But 
> this doesn't seem to be any problem for the other two web servers.)
> 
> Checking my configuration for the NAT gateway, it has a fixed idle 
> timeout of 350 seconds, which is much longer than the 60 seconds I 
> (believe I) have set for idle AJP connections.
> 
> I do not use servlet async or Websocket for anything in my application, 
> so I do not expect long-lasting connections between client and server.
> 
> Is there anything I haven't checked at this point?

You might want to check how connection_pool_timeout interacts with 
connection_pool_size and connection_pool_minsize.

I am wondering if the current size is at minsize will an idle connection 
be closed if it exceeds the timeout?

I think the code that handles this is here:

https://github.com/apache/tomcat-connectors/blob/main/native/common/jk_ajp_common.c#L3510

and I think it only closes idle connections until the pool reaches minsize.

Which isn't what the docs suggest. Which in turn might explain why you 
are seeing connections open longer than the NAT gateway timeout.

Rainer understands this code far better than I do. It would be good to 
get his view on this. If I am right, I think we either need to update 
the docs or we need to fix the code so idle connections below minsize 
are closed and then re-opened to refresh them.

Mark

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org