Maximum connections and thread notifications

["M. Manna" <ma...@gmail.com>]

Hello all,

Since yesterday all of our loadbalanced tomcat servers have been refusing
further connection intermittently. During our internal stress testing,
we’ve managed to reproduce  the issue where maximum number of
threads/connections to Catalina were reached for one server, but then the
rest were loadbalanced to others. Eventually, everyone starts failing
health checks the Citrix network health monitor on port 80 and 8443.
Because all of them were eventually running out.

We wanted to pinpoint the time of the first hit observers  with Catalina
logs to see when max connections are reached or maxThreads are used. Is
there any log level for Catalina/localhost logs which we can use to see
this message ? If so which level should we be using ?

Also, we are using 8.5.37 with our own TSL cipher suites etc. this started
happening only since last few days. We would need to find out whether any
application threads are doubling up and exhausting the APR connectors
capacity for max threads/connections by holding them indefinitely.

Any suggestion/help you can give here would be appreciated.

Thanks,

Re: Maximum connections and thread notifications

[Mark Thomas <ma...@apache.org>]

On 05/04/2019 11:15, M. Manna wrote:
> After a few more checks, we have identified a mismatch (our side) where
> 1.2.18 APR (tcnative-1.dll) was being used instead of 1.2.19 (packaged with
> 8.5.37.
> 
> Usually, we get an error with tomcat startup if the APR version isn't
> meeting the minimum version requirements. But for 8.5.37 when started with
> *1.2.18*, we didn't get this error.
> 
> Could someone confirm whether 1.2.19 was the minimum version requirement
> for 8.5.37?

Since Tomcat is open source you are able to look this up yourself.

https://github.com/apache/tomcat/blob/8.5.37/java/org/apache/catalina/core/AprLifecycleListener.java#L67

Note the difference between minimum required version and minimum
recommended version.

Generally, I'd recommend running the latest version of Tomcat-Native
available.



> On Fri, 5 Apr 2019 at 01:39, M. Manna <ma...@gmail.com> wrote:
> 
>> Hello all,
>>
>> Since yesterday all of our loadbalanced tomcat servers have been refusing
>> further connection intermittently. During our internal stress testing,
>> we’ve managed to reproduce  the issue where maximum number of
>> threads/connections to Catalina were reached for one server, but then the
>> rest were loadbalanced to others. Eventually, everyone starts failing
>> health checks the Citrix network health monitor on port 80 and 8443.
>> Because all of them were eventually running out.
>>
>> We wanted to pinpoint the time of the first hit observers  with Catalina
>> logs to see when max connections are reached or maxThreads are used. Is
>> there any log level for Catalina/localhost logs which we can use to see
>> this message ? If so which level should we be using ?

INFO should be fine.

>> Also, we are using 8.5.37 with our own TSL cipher suites etc. this started
>> happening only since last few days. We would need to find out whether any
>> application threads are doubling up and exhausting the APR connectors
>> capacity for max threads/connections by holding them indefinitely.
>>
>> Any suggestion/help you can give here would be appreciated.

Take 3 thread dumps ~5s apart and compare them.

Mark

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: Maximum connections and thread notifications

["M. Manna" <ma...@gmail.com>]

After a few more checks, we have identified a mismatch (our side) where
1.2.18 APR (tcnative-1.dll) was being used instead of 1.2.19 (packaged with
8.5.37.

Usually, we get an error with tomcat startup if the APR version isn't
meeting the minimum version requirements. But for 8.5.37 when started with
*1.2.18*, we didn't get this error.

Could someone confirm whether 1.2.19 was the minimum version requirement
for 8.5.37?

Thanks,

On Fri, 5 Apr 2019 at 01:39, M. Manna <ma...@gmail.com> wrote:

> Hello all,
>
> Since yesterday all of our loadbalanced tomcat servers have been refusing
> further connection intermittently. During our internal stress testing,
> we’ve managed to reproduce  the issue where maximum number of
> threads/connections to Catalina were reached for one server, but then the
> rest were loadbalanced to others. Eventually, everyone starts failing
> health checks the Citrix network health monitor on port 80 and 8443.
> Because all of them were eventually running out.
>
> We wanted to pinpoint the time of the first hit observers  with Catalina
> logs to see when max connections are reached or maxThreads are used. Is
> there any log level for Catalina/localhost logs which we can use to see
> this message ? If so which level should we be using ?
>
> Also, we are using 8.5.37 with our own TSL cipher suites etc. this started
> happening only since last few days. We would need to find out whether any
> application threads are doubling up and exhausting the APR connectors
> capacity for max threads/connections by holding them indefinitely.
>
> Any suggestion/help you can give here would be appreciated.
>
> Thanks,
>