Intermittent 502 errors from Load Balancer to Tomcat instance.

[Todd Patch <tp...@loftware.com>]

Hello,

We have a setup in AWS with an ALB sitting in front of 2 Tomcat servers hosting our application.  We are having sporadic issues where the ALB will reject some calls with a "502" error.  The system will be fine for days or weeks, and then suddenly for a few hours there will be several (~100) 502 errors.  The volume of requests hitting the ALB is elevated at this time but not very heavy (400 - 800 req/min, with some spikes to 1500).  

We initially had the well known issue of the ALB timeout being longer than the KeepAliveTimeout causing 502 errors, but this is not the current case.  We adjusted our settings and the errors went away for a longer period of time. 

Are there any other reasons for these 502 errors from a Tomcat perspective?  
The calls do not register with the localhost_access logs, are there any recommendations on how to see if or why Tomcat is rejecting these calls?

Our current settings are the following:

ALB 
Idle Timeout - 60 Seconds 
HTTP/2 - Enabled

Tomcat Server.xml
<Executor name="tomcatThreadPool" namePrefix="tomcat-"
maxThreads="400" minSpareThreads="10"/>

    <Connector port="8081"
                                maxHttpHeaderSize="65535"
                                executor="tomcatThreadPool"
                                protocol="org.apache.coyote.http11.Http11NioProtocol"
	    connectionTimeout="20000"
	    keepAliveTimeout="180000"
                                scheme="https"
                                proxyPort="443"
                                acceptorThreadCount="2"
                                acceptCount="2000"
                                compression="on"
                                compressableMimeType="text/html,text/xml,text/plain,text/css,text/javascript,application/json,application/javascript,font/otf,font/ttf,font/woff,font/woff2"
        />

Tomcat Server Version information
Server version name:   Apache Tomcat/8.5.54
Server built:          Apr 3 2020 14:06:10 UTC
Server version number: 8.5.54.0
OS Name:               Linux
OS Version:            4.14.173-137.228.amzn2.x86_64
Architecture:          amd64
JVM Version:           1.8.0_202-b08

Thank you,

Todd


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org