Tomcat 8: Random 404 and 505 errors

[Pavan Veginati <pa...@oracle.com>]

Hi,
We are seeing random 404 and 505 errors with GET and POST requests.

Out of the 10 million daily requests in one cluster, there are 2-3 such 404 errors. In another cluster with around 100 million daily requests, we are seeing 20-30 404s on average per day.
Requests with a 505 error are rare and only happen once or twice a week.

The same requests were successful a few milliseconds before and after the error. The servlet that handles these requests always returns a 200 with an appropriate error response if it fails, but never a 404. So, it is surprising to see a 404 response.

We've only had this issue in our test environment (similar to prod) a couple of times, but we can't reproduce it again. 
I've found a similar issue reported here: https://lists.apache.org/thread/rxypsggr36joq8fcd44khjky9w4qf4xx

Deployment Details:  
  Tomcat Version:  
	8.5.88. Built from source.
  Java Version:  
	JDK 17.
  Network Topology:  
	Direct server to server internal calls with no load balancers in between. 

Are there any known issues that could be causing this, and are there any specific logs we can enable that might help us debug this issue further ? 

-- 
Thanks
Pavan

Re: Tomcat 8: Random 404 and 505 errors

[Mark Thomas <ma...@apache.org>]

On 16/11/2023 22:53, Pavan Veginati wrote:
> Hi,
> We are seeing random 404 and 505 errors with GET and POST requests.
> 
> Out of the 10 million daily requests in one cluster, there are 2-3 such 404 errors. In another cluster with around 100 million daily requests, we are seeing 20-30 404s on average per day.
> Requests with a 505 error are rare and only happen once or twice a week.
> 
> The same requests were successful a few milliseconds before and after the error. The servlet that handles these requests always returns a 200 with an appropriate error response if it fails, but never a 404. So, it is surprising to see a 404 response.

Hmm. Tomcat returns a 505 response in a single location. It does this 
when it doesn't understand the HTTP version that has been received in 
the request line.

Assuming the clients aren't sending malformed HTTP requests that 
suggests the input buffer is getting corrupted. That form of request 
mix-up is rare but has happened due to both Tomcat bugs and application 
bugs in the past.

I don't think there are any fixes since 8.5.88 that relate to this but 
it would be worth considering updating to 8.5.96.

Do you use any of:
- WebSocket
- Servlet asynchronous API
- Sendfile
- HTTP/2

I'm just trying to narrow down the areas of code we may need to look at.

I think the 505s offer the best opportunity to track this down as they 
only happen in one place and debug logging is available.

> We've only had this issue in our test environment (similar to prod) a couple of times, but we can't reproduce it again.
> I've found a similar issue reported here: https://lists.apache.org/thread/rxypsggr36joq8fcd44khjky9w4qf4xx
> 
> Deployment Details:
>    Tomcat Version:
> 	8.5.88. Built from source.
>    Java Version:
> 	JDK 17.
>    Network Topology:
> 	Direct server to server internal calls with no load balancers in between.
> 
> Are there any known issues that could be causing this,

No.

> and are there any specific logs we can enable that might help us debug this issue further ?

Enable debug logging for the Http11Processor. Add this to 
logging.properties:

org.apache.coyote.http11.Http11Processor.level=FINE

That will log additional detail for most failed requests (but not 404s).

Add %r in the access log pattern which will log the entire request line 
for every request.


For each 404, check the access log and see whether the request line is 
as expected.

For each 505, check both the access log and the standard log files.

If you can provide us with the request lines from the access log for 
each 404 (along with whether or not they are as you expect) and the 
request line and debug log entry for each 505 we will hopefully be able 
draw some further conclusions.

Given you are building from source, one possible investigative option is 
providing you with a custom patch that adds additional debug logging. If 
we reach the point where that would be helpful, is that something you'd 
consider?

Mark

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org