Tomcat closes connections on unexpected status codes

[Stefan Ansing <st...@gmail.com>]

Hi,

We've observed some unexpected behaviour in Apache Tomcat (version 10.1.19)
where we see that HTTP/1.1 connections are closed whenever a servlet
application returns the following status codes: 400, 408, 411, 414, 500,
503, 501. This causes client applications to rapidly reconnect and induce
high server-side CPU load due to doing TLS handshakes.

The 400 and 500 status codes are commonly used in RESTful microservices to
communicate errors. Reviewing RFC 9112 I couldn't find any requirement for
closing connections on these status codes.

After testing with Undertow (version 2.3.12), where we didn't see the same
behaviour, we believe that these status codes do not necessitate a new
connection.

Checking the Tomcat sources makes me believe that the behaviour is
hard-coded[1]. I'm reaching out here to re-evaluate the list of status
codes and to discuss the possibilities of making the behaviour configurable.

A colleague of mine reported a bug for this issue:
https://bz.apache.org/bugzilla/show_bug.cgi?id=68901

Kind regards,
Stefan Ansing

[1]:
https://github.com/apache/tomcat/blame/bc900e0100de9879604b93af4722c272ab3d1a24/java/org/apache/coyote/http11/Http11Processor.java#L604-L617

Re: Tomcat closes connections on unexpected status codes

[Christopher Schultz <ch...@christopherschultz.net>]

Pawel,

On 4/18/24 20:32, Pawel Veselov wrote:
> On Thu, Apr 18, 2024 at 9:40 AM Adwait Kumar Singh <ad...@gmail.com> wrote:
>>>   I'm not (yet) convinced distinguishing between those scenarios is always
>>> going to be possible.
>> I have a Tomcat patch which we use at work to do this, i.e always close the
>> connection if HTTP parsing fails but not if it's a user set status. I can
>> create a PR for feedback.
> 
> It can't be that straightforward though. The HTTP parsing can very well fail
> well past the point the application was invoked, and returned a status, during
> chunked request parsing. The server can't respond with another error though,
> as the headers are/can be committed, it has no recourse but to try to send them
> if possible, and then shutdown the connection.

Assuming that the container is still in charge of the HTTP part of 
things, then the container definitely knows if the request has 
experienced some problem with e.g. chunked content.

Tomcat has private information about requests that it keeps separate 
from what applications can (usually) get access to, and it could simply 
mark a request as "requiring connection termination" and doing that at 
the appropriate time when execution finishes with the application and 
returns to the container.

> And handling incomplete body reads from the application is also
> going to be a nuisance that has to be dealt with. Tomcat will have to
> consume the remainder of the body without application doing so
(Tomcat already does this.)

> , which may not be desirable in all cases, and will require either an
> API change to be supported as being signalled by the application
> (here is my status, but PLEASE don't break the HTTP connection), or
> requiring the application to diligently consume the body till the end
> before returning service back to Tomcat, or having that as a
> configuration parameter.
I'm not sure this is different from today.

A new API isn't strictly necessary, here. The application can 
communicate with the container using request attributes. But I'm not 
sure anything like that is needed.

> I actually don't know how Tomcat deals with bodies that aren't fully read by
> the app right now. I would very much like to know for sure.

Tomcat does one of two things:

1. Reads the remainder of the request entity and discards it

2. Closes the connection without finishing the read

I believe it tries (1) for a while and, after some configurable number 
of bytes (<Connector> maxSwallowSize), it switches to (2) and gives up.

-chris

>> On Thu, Apr 18, 2024 at 8:42 AM Mark Thomas <ma...@apache.org> wrote:
>>
>>> On 18/04/2024 15:18, Stefan Ansing wrote:
>>>> Hi Rémy, Mark,
>>>>
>>>>
>>>>
>>>> I just want to make sure that we’re understanding each other. I can see
>>>> that the connection needs to be closed in certain conditions to prevent
>>>> request smuggling attacks. I certainly don’t want to change that
>>> behaviour.
>>>>
>>>> However, I’m facing a scenario where an application is responding to a
>>>> valid request (from HTTP perspective), with a valid response using these
>>>> status codes (more specifically status codes 400 and 500).
>>>
>>> If the request is a valid HTTP request then a 400 status doesn't seem
>>> appropriate to me.
>>>
>>> If the server is correctly handling that request to generate the
>>> response, a 500 status doesn't seem right either.
>>>
>>>>
>>>> I don’t think that in this scenario a request smuggling attack could be
>>>> executed, or am I missing something?
>>>
>>> The main issue is if the original request is invalid HTTP there is no
>>> way to determine where the next HTTP request starts.
>>>
>>> If there is a proxy in the mix then the risks of something going wrong
>>> tend to go up.
>>>
>>> Mark
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: Tomcat closes connections on unexpected status codes

[Pawel Veselov <pa...@gmail.com>]

On Thu, Apr 18, 2024 at 9:40 AM Adwait Kumar Singh <ad...@gmail.com> wrote:
> >  I'm not (yet) convinced distinguishing between those scenarios is always
> > going to be possible.
> I have a Tomcat patch which we use at work to do this, i.e always close the
> connection if HTTP parsing fails but not if it's a user set status. I can
> create a PR for feedback.

It can't be that straightforward though. The HTTP parsing can very well fail
well past the point the application was invoked, and returned a status, during
chunked request parsing. The server can't respond with another error though,
as the headers are/can be committed, it has no recourse but to try to send them
if possible, and then shutdown the connection.

And handling incomplete body reads from the application is also going
to be a nuisance
that has to be dealt with. Tomcat will have to consume the remainder of the body
without application doing so, which may not be desirable in all cases, and
will require either an API change to be supported as being signalled by the
application (here is my status, but PLEASE don't break the HTTP connection),
or requiring the application to diligently consume the body till the end before
returning service back to Tomcat, or having that as a configuration parameter.

I actually don't know how Tomcat deals with bodies that aren't fully read by
the app right now. I would very much like to know for sure.

>
> On Thu, Apr 18, 2024 at 8:42 AM Mark Thomas <ma...@apache.org> wrote:
>
> > On 18/04/2024 15:18, Stefan Ansing wrote:
> > > Hi Rémy, Mark,
> > >
> > >
> > >
> > > I just want to make sure that we’re understanding each other. I can see
> > > that the connection needs to be closed in certain conditions to prevent
> > > request smuggling attacks. I certainly don’t want to change that
> > behaviour.
> > >
> > > However, I’m facing a scenario where an application is responding to a
> > > valid request (from HTTP perspective), with a valid response using these
> > > status codes (more specifically status codes 400 and 500).
> >
> > If the request is a valid HTTP request then a 400 status doesn't seem
> > appropriate to me.
> >
> > If the server is correctly handling that request to generate the
> > response, a 500 status doesn't seem right either.
> >
> > >
> > > I don’t think that in this scenario a request smuggling attack could be
> > > executed, or am I missing something?
> >
> > The main issue is if the original request is invalid HTTP there is no
> > way to determine where the next HTTP request starts.
> >
> > If there is a proxy in the mix then the risks of something going wrong
> > tend to go up.
> >
> > Mark

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: Tomcat closes connections on unexpected status codes

[Adwait Kumar Singh <ad...@gmail.com>]

>
>  I'm not (yet) convinced distinguishing between those scenarios is always
> going to be possible.


I have a Tomcat patch which we use at work to do this, i.e always close the
connection if HTTP parsing fails but not if it's a user set status. I can
create a PR for feedback.

On Thu, Apr 18, 2024 at 8:42 AM Mark Thomas <ma...@apache.org> wrote:

> On 18/04/2024 15:18, Stefan Ansing wrote:
> > Hi Rémy, Mark,
> >
> >
> >
> > I just want to make sure that we’re understanding each other. I can see
> > that the connection needs to be closed in certain conditions to prevent
> > request smuggling attacks. I certainly don’t want to change that
> behaviour.
> >
> > However, I’m facing a scenario where an application is responding to a
> > valid request (from HTTP perspective), with a valid response using these
> > status codes (more specifically status codes 400 and 500).
>
> If the request is a valid HTTP request then a 400 status doesn't seem
> appropriate to me.
>
> If the server is correctly handling that request to generate the
> response, a 500 status doesn't seem right either.
>
> >
> > I don’t think that in this scenario a request smuggling attack could be
> > executed, or am I missing something?
>
> The main issue is if the original request is invalid HTTP there is no
> way to determine where the next HTTP request starts.
>
> If there is a proxy in the mix then the risks of something going wrong
> tend to go up.
>
> Mark
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>

Re: Tomcat closes connections on unexpected status codes

[Pawel Veselov <pa...@gmail.com>]

Chris,

On Fri, Apr 19, 2024 at 4:40 AM Christopher Schultz
<ch...@christopherschultz.net> wrote:
>
> Pawel,
>
> On 4/18/24 20:21, Pawel Veselov wrote:
> >> On 18/04/2024 15:18, Stefan Ansing wrote:
> >>> Hi Rémy, Mark,
> >>> I just want to make sure that we’re understanding each other. I can see
> >>> that the connection needs to be closed in certain conditions to prevent
> >>> request smuggling attacks. I certainly don’t want to change that behaviour.
> >>> However, I’m facing a scenario where an application is responding to a
> >>> valid request (from HTTP perspective), with a valid response using these
> >>> status codes (more specifically status codes 400 and 500).
> >> If the request is a valid HTTP request then a 400 status doesn't seem
> >> appropriate to me.
> >
> > It's by now, however, a de-facto standard. Every time I try to
> > determine "which HTTP response should I send back in case of issues
> > with the data", I find myself scrolling through the list of defined
> > codes and not finding anything that would otherwise fit. The HTTP
> > spec states what should the server do in case of HTTP protocol errors
> > (respond with an appropriate 4xx), but that's all that the spec
> > covers for the protocol, and it doesn't prohibit use of 400 for
> > application-level errors. Out of the entire 4xx codes, 400 is (maybe
> > also 414?) the only one that is used for protocol problems, others
> > are for application level errors, but they are very specific and
> > limiting (IMHO).
> When you say "protocol problems", what protocol are you referring to?

In this instance, I meant the actual problems with the HTTP protocol contents.
400 is normally returned by the container because it couldn't
understand the HTTP.
414 is probably as well (because container ran out of maximum
available space to read
in the URI). The rest of the error codes are expected to be produced by the
application, but they were devised for that application being a web server.

> Because if the request is readable and syntactically correct, there is
> no protocol problem. Everything else is ... something else. If you are
> establishing a protocol ON TOP OF HTTP then it's a violation of whatever
> protocol THAT is, not HTTP. So it's better to return { "error" : "Foo
> Protocol violation" } with an appropriate HTTP status code, possibly
> even *200*.

Yes. But (again, assuming web-service-like implementation), the application can
return other HTTP codes for other data (on top of HTTP) issues, i.e., when an
endpoint is not found, 404, when data type is wrong - 415; there are also those
exuberant codes like 409 and 410. So, on the surface, it looks like
the application
can use 4xx codes to signal an error, so one can lay down a contract that says
"2xx means OK response from the application's perspective, and if there is an
error, we'll return some 4xx code", but there *was* (see below) no good 4xx
code for saying "Hey, you forgot a JSON property in your input".

You're right, it probably should have been 2xx, but it's also cringy
to say things like
"there are some errors that we will return 4xx for, but for some it will be 2xx,
and here is how you differentiate successful 2xx from error response 2xx".
I did do that in at least one w/s implementation, and had the customer
yell at me for my trouble. This path is also not well supported by standards
like OAS/RAML (last time I checked, at least). The description of the responses
becomes significantly more complicated in this case.

And, again, there is AWS that is a rather large supplier of web services,
and they did decide to use 400 for this. The web-service client still needs
to distinguish between "400 because the header is mangled", or "400 because
a JSON property was missing". The RFC doesn't explicitly limit the use of
400 to HTTP protocol problems, even though all the examples that are listed
in the spec *are* such.

I did look again at the specs just now, and lo and behold, there is now a
code 422. Which, at least on the surface, looks like exactly the code to use
for responding with application-related content problems. It wasn't there in
RFC#7231 (and neither in #2616 or #2068), but #9110 does have it (yet
the language around 400 is still overly broad), which makes it about 2
years old.
Now, of course, existing w/s contracts aren't likely to switch to using that,
but for any new ones I'll certainly consider it.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: Tomcat closes connections on unexpected status codes

[Christopher Schultz <ch...@christopherschultz.net>]

Pawel,

On 4/18/24 20:21, Pawel Veselov wrote:
>> On 18/04/2024 15:18, Stefan Ansing wrote:
>>> Hi Rémy, Mark,
>>> I just want to make sure that we’re understanding each other. I can see
>>> that the connection needs to be closed in certain conditions to prevent
>>> request smuggling attacks. I certainly don’t want to change that behaviour.
>>> However, I’m facing a scenario where an application is responding to a
>>> valid request (from HTTP perspective), with a valid response using these
>>> status codes (more specifically status codes 400 and 500).
>> If the request is a valid HTTP request then a 400 status doesn't seem
>> appropriate to me.
> 
> It's by now, however, a de-facto standard. Every time I try to
> determine "which HTTP response should I send back in case of issues
> with the data", I find myself scrolling through the list of defined
> codes and not finding anything that would otherwise fit. The HTTP
> spec states what should the server do in case of HTTP protocol errors
> (respond with an appropriate 4xx), but that's all that the spec
> covers for the protocol, and it doesn't prohibit use of 400 for
> application-level errors. Out of the entire 4xx codes, 400 is (maybe
> also 414?) the only one that is used for protocol problems, others
> are for application level errors, but they are very specific and
> limiting (IMHO).
When you say "protocol problems", what protocol are you referring to? 
Because if the request is readable and syntactically correct, there is 
no protocol problem. Everything else is ... something else. If you are 
establishing a protocol ON TOP OF HTTP then it's a violation of whatever 
protocol THAT is, not HTTP. So it's better to return { "error" : "Foo 
Protocol violation" } with an appropriate HTTP status code, possibly 
even *200*.

> Then, how do I return error paths, when implementing things like a web service?
> Either returning it with 200 and custom headers that indicate it's an
> error response,
> or a 400 with the same. In which case, the difference on the client side is -
> does it have to logically differentiate valid response from an error based just
> on these custom headers/mime type, or differentiate protocol-related 400 from
> application-related 400, and can always treat 200 as proper.
> 
> Then I look at "what do others do", and I see AWS that uses 400 for all client
> errors, and the way a client knows it's an application error is because there is
> content that wouldn't be typically returned if there is a protocol problem.
> So I follow suit.
> 
>> If the server is correctly handling that request to generate the
>> response, a 500 status doesn't seem right either.
> 
> Right. 5xx - my fault, 4xx - your fault.
> 
>>> I don’t think that in this scenario a request smuggling attack could be
>>> executed, or am I missing something?
>> The main issue is if the original request is invalid HTTP there is no
>> way to determine where the next HTTP request starts.
> 
> I'd imagine it reasonable that as long as Tomcat doesn't find any issues
> with the HTTP protocol itself - closing connections wouldn't serve any
> helpful purpose in that case, except for not having to read the remainder
> of the message. I'd like to know how the connection can be mucked up
> otherwise, whether a proxy is involved or not.
> 
>>
>> If there is a proxy in the mix then the risks of something going wrong
>> tend to go up.

-chris


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: Tomcat closes connections on unexpected status codes

[Pawel Veselov <pa...@gmail.com>]

> On 18/04/2024 15:18, Stefan Ansing wrote:
> > Hi Rémy, Mark,
> > I just want to make sure that we’re understanding each other. I can see
> > that the connection needs to be closed in certain conditions to prevent
> > request smuggling attacks. I certainly don’t want to change that behaviour.
> > However, I’m facing a scenario where an application is responding to a
> > valid request (from HTTP perspective), with a valid response using these
> > status codes (more specifically status codes 400 and 500).
> If the request is a valid HTTP request then a 400 status doesn't seem
> appropriate to me.

It's by now, however, a de-facto standard. Every time I try to determine
"which HTTP response should I send back in case of issues with the data",
I find myself scrolling through the list of defined codes and not
finding anything
that would otherwise fit. The HTTP spec states what should the server
do in case of HTTP protocol errors (respond with an appropriate 4xx), but
that's all that the spec covers for the protocol, and it doesn't prohibit use
of 400 for application-level errors. Out of the entire 4xx codes, 400
is (maybe also
414?) the only one that is used for protocol problems, others are for
application
level errors, but they are very specific and limiting (IMHO).

Then, how do I return error paths, when implementing things like a web service?
Either returning it with 200 and custom headers that indicate it's an
error response,
or a 400 with the same. In which case, the difference on the client side is -
does it have to logically differentiate valid response from an error based just
on these custom headers/mime type, or differentiate protocol-related 400 from
application-related 400, and can always treat 200 as proper.

Then I look at "what do others do", and I see AWS that uses 400 for all client
errors, and the way a client knows it's an application error is because there is
content that wouldn't be typically returned if there is a protocol problem.
So I follow suit.

> If the server is correctly handling that request to generate the
> response, a 500 status doesn't seem right either.

Right. 5xx - my fault, 4xx - your fault.

> > I don’t think that in this scenario a request smuggling attack could be
> > executed, or am I missing something?
> The main issue is if the original request is invalid HTTP there is no
> way to determine where the next HTTP request starts.

I'd imagine it reasonable that as long as Tomcat doesn't find any issues
with the HTTP protocol itself - closing connections wouldn't serve any
helpful purpose in that case, except for not having to read the remainder
of the message. I'd like to know how the connection can be mucked up
otherwise, whether a proxy is involved or not.

>
> If there is a proxy in the mix then the risks of something going wrong
> tend to go up.
>
> Mark

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: Tomcat closes connections on unexpected status codes

[Adwait Kumar Singh <ad...@gmail.com>]

> Assuming it's easy for Tomcat to differentiate between errors generated

My PR was based on the assumption that it is easy, since Tomcat always
invokes this method[1] if it's a badRequest.


[1]
https://github.com/apache/tomcat/blob/main/java/org/apache/coyote/http11/Http11Processor.java#L849-L850

On Wed, Apr 24, 2024 at 11:51 AM Christopher Schultz <
chris@christopherschultz.net> wrote:

> Stefan,
>
> On 4/24/24 13:58, Stefan Ansing wrote:
> > Op do 18 apr 2024 om 17:42 schreef Mark Thomas <ma...@apache.org>:
> >
> >> On 18/04/2024 15:18, Stefan Ansing wrote:
> >>> Hi Rémy, Mark,
> >>>
> >>>
> >>>
> >>> I just want to make sure that we’re understanding each other. I can see
> >>> that the connection needs to be closed in certain conditions to prevent
> >>> request smuggling attacks. I certainly don’t want to change that
> >> behaviour.
> >>>
> >>> However, I’m facing a scenario where an application is responding to a
> >>> valid request (from HTTP perspective), with a valid response using
> these
> >>> status codes (more specifically status codes 400 and 500).
> >>
> >> If the request is a valid HTTP request then a 400 status doesn't seem
> >> appropriate to me.
> >>
> >> If the server is correctly handling that request to generate the
> >> response, a 500 status doesn't seem right either.
> >>
> >>>
> >>> I don’t think that in this scenario a request smuggling attack could be
> >>> executed, or am I missing something?
> >>
> >> The main issue is if the original request is invalid HTTP there is no
> >> way to determine where the next HTTP request starts.
> >>
> >> If there is a proxy in the mix then the risks of something going wrong
> >> tend to go up.
> >>
> >> Mark
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >> For additional commands, e-mail: users-help@tomcat.apache.org
> >>
> >>
> > Hi Mark,
> >
> > I can see your point of view regarding the use of the status codes for
> > application errors. Unfortunately the HTTP spec doesn't clearly
> > differentiate the use of status codes for protocol or application errors.
> > Which is probably why these status codes are now also commonly used for
> > application errors.
> >
> > Tomcat (and other servlet containers) currently allow applications to set
> > any status code, but with the current behaviour of Tomcat this leads to
> > unexpected side effects for some status codes.
> >
> > This behaviour makes it so that Tomcat might not be fit for our purpose
> > (Spring Boot services to services communications). I think the way to
> > resolve that is to alter the behaviour in Tomcat to differentiate between
> > protocol and application errors when using these status codes (and to
> make
> > this behaviour potentially configurable). I also think that this change
> > would benefit most users of Tomcat since the behaviour in this scenario
> is
> > unnecessary. Would the Tomcat developers be willing to do that?
>
> Assuming it's easy for Tomcat to differentiate between errors generated
> by Tomcat (e.g. "real" 400 responses) and those generated by the
> application, I think this is a good idea. HTTP 400 indicates a protocol
> error, but if the application is generating it, then Tomcat need not
> close the connection.
>
> Theoretically this could also be true for other status codes as well. I
> chose 400 because it means the connection MUST be closed for security if
> Tomcat is the one detecting that there is actually an HTTP protocol
> violation.
>
> -chris
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>

Re: Tomcat closes connections on unexpected status codes

[Christopher Schultz <ch...@christopherschultz.net>]

Stefan,

On 4/24/24 13:58, Stefan Ansing wrote:
> Op do 18 apr 2024 om 17:42 schreef Mark Thomas <ma...@apache.org>:
> 
>> On 18/04/2024 15:18, Stefan Ansing wrote:
>>> Hi Rémy, Mark,
>>>
>>>
>>>
>>> I just want to make sure that we’re understanding each other. I can see
>>> that the connection needs to be closed in certain conditions to prevent
>>> request smuggling attacks. I certainly don’t want to change that
>> behaviour.
>>>
>>> However, I’m facing a scenario where an application is responding to a
>>> valid request (from HTTP perspective), with a valid response using these
>>> status codes (more specifically status codes 400 and 500).
>>
>> If the request is a valid HTTP request then a 400 status doesn't seem
>> appropriate to me.
>>
>> If the server is correctly handling that request to generate the
>> response, a 500 status doesn't seem right either.
>>
>>>
>>> I don’t think that in this scenario a request smuggling attack could be
>>> executed, or am I missing something?
>>
>> The main issue is if the original request is invalid HTTP there is no
>> way to determine where the next HTTP request starts.
>>
>> If there is a proxy in the mix then the risks of something going wrong
>> tend to go up.
>>
>> Mark
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
> Hi Mark,
> 
> I can see your point of view regarding the use of the status codes for
> application errors. Unfortunately the HTTP spec doesn't clearly
> differentiate the use of status codes for protocol or application errors.
> Which is probably why these status codes are now also commonly used for
> application errors.
> 
> Tomcat (and other servlet containers) currently allow applications to set
> any status code, but with the current behaviour of Tomcat this leads to
> unexpected side effects for some status codes.
> 
> This behaviour makes it so that Tomcat might not be fit for our purpose
> (Spring Boot services to services communications). I think the way to
> resolve that is to alter the behaviour in Tomcat to differentiate between
> protocol and application errors when using these status codes (and to make
> this behaviour potentially configurable). I also think that this change
> would benefit most users of Tomcat since the behaviour in this scenario is
> unnecessary. Would the Tomcat developers be willing to do that?

Assuming it's easy for Tomcat to differentiate between errors generated 
by Tomcat (e.g. "real" 400 responses) and those generated by the 
application, I think this is a good idea. HTTP 400 indicates a protocol 
error, but if the application is generating it, then Tomcat need not 
close the connection.

Theoretically this could also be true for other status codes as well. I 
chose 400 because it means the connection MUST be closed for security if 
Tomcat is the one detecting that there is actually an HTTP protocol 
violation.

-chris

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: Tomcat closes connections on unexpected status codes

[Stefan Ansing <st...@gmail.com>]

Op do 18 apr 2024 om 17:42 schreef Mark Thomas <ma...@apache.org>:

> On 18/04/2024 15:18, Stefan Ansing wrote:
> > Hi Rémy, Mark,
> >
> >
> >
> > I just want to make sure that we’re understanding each other. I can see
> > that the connection needs to be closed in certain conditions to prevent
> > request smuggling attacks. I certainly don’t want to change that
> behaviour.
> >
> > However, I’m facing a scenario where an application is responding to a
> > valid request (from HTTP perspective), with a valid response using these
> > status codes (more specifically status codes 400 and 500).
>
> If the request is a valid HTTP request then a 400 status doesn't seem
> appropriate to me.
>
> If the server is correctly handling that request to generate the
> response, a 500 status doesn't seem right either.
>
> >
> > I don’t think that in this scenario a request smuggling attack could be
> > executed, or am I missing something?
>
> The main issue is if the original request is invalid HTTP there is no
> way to determine where the next HTTP request starts.
>
> If there is a proxy in the mix then the risks of something going wrong
> tend to go up.
>
> Mark
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Hi Mark,

I can see your point of view regarding the use of the status codes for
application errors. Unfortunately the HTTP spec doesn't clearly
differentiate the use of status codes for protocol or application errors.
Which is probably why these status codes are now also commonly used for
application errors.

Tomcat (and other servlet containers) currently allow applications to set
any status code, but with the current behaviour of Tomcat this leads to
unexpected side effects for some status codes.

This behaviour makes it so that Tomcat might not be fit for our purpose
(Spring Boot services to services communications). I think the way to
resolve that is to alter the behaviour in Tomcat to differentiate between
protocol and application errors when using these status codes (and to make
this behaviour potentially configurable). I also think that this change
would benefit most users of Tomcat since the behaviour in this scenario is
unnecessary. Would the Tomcat developers be willing to do that?

Stefan

Re: Tomcat closes connections on unexpected status codes

[Mark Thomas <ma...@apache.org>]

On 18/04/2024 15:18, Stefan Ansing wrote:
> Hi Rémy, Mark,
> 
> 
> 
> I just want to make sure that we’re understanding each other. I can see
> that the connection needs to be closed in certain conditions to prevent
> request smuggling attacks. I certainly don’t want to change that behaviour.
> 
> However, I’m facing a scenario where an application is responding to a
> valid request (from HTTP perspective), with a valid response using these
> status codes (more specifically status codes 400 and 500).

If the request is a valid HTTP request then a 400 status doesn't seem 
appropriate to me.

If the server is correctly handling that request to generate the 
response, a 500 status doesn't seem right either.

> 
> I don’t think that in this scenario a request smuggling attack could be
> executed, or am I missing something?

The main issue is if the original request is invalid HTTP there is no 
way to determine where the next HTTP request starts.

If there is a proxy in the mix then the risks of something going wrong 
tend to go up.

Mark

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: Tomcat closes connections on unexpected status codes

[Stefan Ansing <st...@gmail.com>]

Op do 18 apr 2024 om 15:41 schreef Rémy Maucherat <re...@apache.org>:

> On Thu, Apr 18, 2024 at 1:17 PM Mark Thomas <ma...@apache.org> wrote:
> >
> > On 18/04/2024 09:07, Stefan Ansing wrote:
> > > Hi,
> > >
> > > We've observed some unexpected behaviour in Apache Tomcat (version
> 10.1.19)
> > > where we see that HTTP/1.1 connections are closed whenever a servlet
> > > application returns the following status codes: 400, 408, 411, 414,
> 500,
> > > 503, 501. This causes client applications to rapidly reconnect and
> induce
> > > high server-side CPU load due to doing TLS handshakes.
> > >
> > > The 400 and 500 status codes are commonly used in RESTful
> microservices to
> > > communicate errors. Reviewing RFC 9112 I couldn't find any requirement
> for
> > > closing connections on these status codes.
> > >
> > > After testing with Undertow (version 2.3.12), where we didn't see the
> same
> > > behaviour, we believe that these status codes do not necessitate a new
> > > connection.
> >
> > The Tomcat developers disagree. Connections are closed after these
> > status codes to avoid various forms of request smuggling attacks.
> >
> > > Checking the Tomcat sources makes me believe that the behaviour is
> > > hard-coded[1]. I'm reaching out here to re-evaluate the list of status
> > > codes and to discuss the possibilities of making the behaviour
> configurable.
> >
> > Making this list of status codes configurable seems reasonable. The
> > default can stay as current and if users want to change it then they
> > have to accept the associated security risks.
>
> If it's insecure, then it would still be a valid CVE even if the
> configuration is optional. We don't have an "allowSmuggling" attribute
> on the connector to relax header or status line parsing, even though
> many would have wanted it in the past ...
>
> Rémy
>
> > Mark
> >
> >
> > >
> > > A colleague of mine reported a bug for this issue:
> > > https://bz.apache.org/bugzilla/show_bug.cgi?id=68901
> > >
> > > Kind regards,
> > > Stefan Ansing
> > >
> > > [1]:
> > >
> https://github.com/apache/tomcat/blame/bc900e0100de9879604b93af4722c272ab3d1a24/java/org/apache/coyote/http11/Http11Processor.java#L604-L617
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> > For additional commands, e-mail: users-help@tomcat.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Hi Rémy, Mark,



I just want to make sure that we’re understanding each other. I can see
that the connection needs to be closed in certain conditions to prevent
request smuggling attacks. I certainly don’t want to change that behaviour.

However, I’m facing a scenario where an application is responding to a
valid request (from HTTP perspective), with a valid response using these
status codes (more specifically status codes 400 and 500).

I don’t think that in this scenario a request smuggling attack could be
executed, or am I missing something?



Stefan

Re: Tomcat closes connections on unexpected status codes

[Mark Thomas <ma...@apache.org>]

On 18/04/2024 14:41, Rémy Maucherat wrote:
> On Thu, Apr 18, 2024 at 1:17 PM Mark Thomas <ma...@apache.org> wrote:
>>
>> On 18/04/2024 09:07, Stefan Ansing wrote:
>>> Hi,
>>>
>>> We've observed some unexpected behaviour in Apache Tomcat (version 10.1.19)
>>> where we see that HTTP/1.1 connections are closed whenever a servlet
>>> application returns the following status codes: 400, 408, 411, 414, 500,
>>> 503, 501. This causes client applications to rapidly reconnect and induce
>>> high server-side CPU load due to doing TLS handshakes.
>>>
>>> The 400 and 500 status codes are commonly used in RESTful microservices to
>>> communicate errors. Reviewing RFC 9112 I couldn't find any requirement for
>>> closing connections on these status codes.
>>>
>>> After testing with Undertow (version 2.3.12), where we didn't see the same
>>> behaviour, we believe that these status codes do not necessitate a new
>>> connection.
>>
>> The Tomcat developers disagree. Connections are closed after these
>> status codes to avoid various forms of request smuggling attacks.
>>
>>> Checking the Tomcat sources makes me believe that the behaviour is
>>> hard-coded[1]. I'm reaching out here to re-evaluate the list of status
>>> codes and to discuss the possibilities of making the behaviour configurable.
>>
>> Making this list of status codes configurable seems reasonable. The
>> default can stay as current and if users want to change it then they
>> have to accept the associated security risks.
> 
> If it's insecure, then it would still be a valid CVE even if the
> configuration is optional. We don't have an "allowSmuggling" attribute
> on the connector to relax header or status line parsing, even though
> many would have wanted it in the past ...

I don't think it is quite that black and white.

If the client connects directly to the server (no proxy) then there 
request smuggling is a lot less likely and arguably impossible (assuming 
no Tomcat bugs).

If there is a proxy that is when there is a risk of request smuggling.

Given that there are circumstances where the risk of request smuggling 
is low (possibly none) it seems reasonable to allow users to configure 
the status codes that trigger connection close.

Mark

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: Tomcat closes connections on unexpected status codes

[Adwait Kumar Singh <ad...@gmail.com>]

Any chance someone took a look at the PR? Do you guys think this is a
viable solution?

On Sun, Apr 21, 2024 at 12:54 PM Adwait Kumar Singh <ad...@gmail.com>
wrote:

> https://github.com/apache/tomcat/pull/723 is a draft PR of the idea I was
> talking about earlier, i.e close the connection on a bad request but
> otherwise allow it to be configurable by the user.
>
> Currently you need to subclass Http11Processor and override
> statusDropsConnections, but this is just to get early feedback. If people
> are aligned with this approach, I can polish the PR (add tests, make the
> status code list actually configurable etc etc)
>
> On Fri, Apr 19, 2024 at 5:51 AM Christopher Schultz <
> chris@christopherschultz.net> wrote:
>
>> Harri,
>>
>> On 4/19/24 08:10, Harri Pesonen wrote:
>> > I have developed a restful web service, which uses HTTP response codes
>> 200 OK, 201 Created, 204 No Content and 404 Not Found.
>> > It does not use 400 Bad Request or 500 Internal Server Error normally.
>> > 400 Bad Request is more common than 500 Internal Server Error, which
>> should basically never happen.
>>
>> I disagree. There aren't many good 5xx errors, so sometimes "500 Oops"
>> is the best you can do. HTTP 5xx doesn't even really allow you to
>> differentiate between "server error" and "application error".
>>
>> > 400 Bad Request is the best response in many cases, if client gives
>> some query parameter which is not supported by the application logic.
>>
>> -1
>>
>> 400 means that the HTTP request is bad. In your situation, the HTTP
>> request is 100% valid. It's your application that is saying "sorry,
>> client, you did something we don't allow". This is an application-level
>> error, not a protocol-level error. To me, 400 means "HTTP protocol
>> error" and should never be substituted for "application protocol error".
>>
>> > I think that it would be better not to close connection in this case,
>> if the error comes from application.
>> > I wonder if there is option for application to define that connection
>> should be closed or not after the response has been sent?
>> > Or is the option only from the client.
>> >
>> > For me this 404 Not Found is also a small problem, as it is error, but
>> it can happen quite often.
>> > HTTP errors are not nice in logs.
>> > Normally if you try to fetch some restful resource, which does not
>> exist, then it returns 404 Not Found.
>> > GET /service/resource/id => 404 Not Found
>> > If I now had an option to rewrite the service, I would probably use 204
>> No Content in this case as well, to avoid errors.
>>
>> 404 and 204 are fundamentally different responses. 404 means "this
>> resource does not exist" and 204 means "this resource DOES EXIST but it
>> doesn't contain anything". Your application may not differentiate
>> between those two cases, but as a client I would be confused if "Not
>> Found" was replaced by "Found to be empty" in all cases.
>>
>> > 204 No Content is normally used with PUT and DELETE requests.
>>
>> Yes, you can use those. 200 would also make sense and, of course 201 for
>> new resources.
>>
>> -chris
>>
>> > -----Original Message-----
>> > From: Christopher Schultz <ch...@christopherschultz.net>
>> > Sent: perjantai 19. huhtikuuta 2024 14.27
>> > To: users@tomcat.apache.org
>> > Subject: Re: Tomcat closes connections on unexpected status codes
>> >
>> > Mark,
>> >
>> > On 4/18/24 11:38, Mark Thomas wrote:
>> >> On 18/04/2024 15:16, Adwait Kumar Singh wrote:
>> >>> I think we should *always* close connections in cases where it can
>> >>> lead to request smuggling vulnerabilities like when there is an error
>> >>> during header or request line parsing, but allowing the user to
>> >>> control connection close when the status is being set by the user,
>> >>> should be safe?
>> >>
>> >> I'm not (yet) convinced distinguishing between those scenarios is
>> >> always going to be possible.
>> >>
>> >>> It allows users to send back responses like InvalidInputException
>> >>> with a
>> >>> 400 status without being forced to close the connection.
>> >>
>> >> I appreciate why a 400 is used here but 400 has always struck me as
>> >> more for protocol level issues rather than application level issues.
>> >
>> > Didn't someone tell me recently that, technically, ANY client-error is
>> allowed to trigger a 400 response without being more specific?
>> >
>> >> That is the fundamental problem here. The status codes are being used
>> >> for two completely different purposes.
>> >
>> > +1
>> >
>> > I've always found it distasteful when REST services do this. To me, 400
>> means "the request was actually malformed". If you need authentication,
>> that's a 401. If you aren't allowed, that's 403. If you didn't provide a
>> required header, that's a 412, etc. I've usually found the "correct"
>> > response code to use for every situation and I've never written an
>> application that returns a 400 response directly.
>> >
>> > -chris
>> >
>> >>> On Thu, Apr 18, 2024 at 6:41 AM Rémy Maucherat <re...@apache.org>
>> wrote:
>> >>>
>> >>>> On Thu, Apr 18, 2024 at 1:17 PM Mark Thomas <ma...@apache.org>
>> wrote:
>> >>>>>
>> >>>>> On 18/04/2024 09:07, Stefan Ansing wrote:
>> >>>>>> Hi,
>> >>>>>>
>> >>>>>> We've observed some unexpected behaviour in Apache Tomcat (version
>> >>>> 10.1.19)
>> >>>>>> where we see that HTTP/1.1 connections are closed whenever a
>> >>>>>> servlet application returns the following status codes: 400, 408,
>> >>>>>> 411, 414,
>> >>>> 500,
>> >>>>>> 503, 501. This causes client applications to rapidly reconnect and
>> >>>> induce
>> >>>>>> high server-side CPU load due to doing TLS handshakes.
>> >>>>>>
>> >>>>>> The 400 and 500 status codes are commonly used in RESTful
>> >>>> microservices to
>> >>>>>> communicate errors. Reviewing RFC 9112 I couldn't find any
>> >>>>>> requirement
>> >>>> for
>> >>>>>> closing connections on these status codes.
>> >>>>>>
>> >>>>>> After testing with Undertow (version 2.3.12), where we didn't see
>> >>>>>> the
>> >>>> same
>> >>>>>> behaviour, we believe that these status codes do not necessitate a
>> >>>>>> new connection.
>> >>>>>
>> >>>>> The Tomcat developers disagree. Connections are closed after these
>> >>>>> status codes to avoid various forms of request smuggling attacks.
>> >>>>>
>> >>>>>> Checking the Tomcat sources makes me believe that the behaviour is
>> >>>>>> hard-coded[1]. I'm reaching out here to re-evaluate the list of
>> >>>>>> status codes and to discuss the possibilities of making the
>> >>>>>> behaviour
>> >>>> configurable.
>> >>>>>
>> >>>>> Making this list of status codes configurable seems reasonable. The
>> >>>>> default can stay as current and if users want to change it then
>> >>>>> they have to accept the associated security risks.
>> >>>>
>> >>>> If it's insecure, then it would still be a valid CVE even if the
>> >>>> configuration is optional. We don't have an "allowSmuggling"
>> >>>> attribute on the connector to relax header or status line parsing,
>> >>>> even though many would have wanted it in the past ...
>> >>>>
>> >>>> Rémy
>> >>>>
>> >>>>> Mark
>> >>>>>
>> >>>>>
>> >>>>>>
>> >>>>>> A colleague of mine reported a bug for this issue:
>> >>>>>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2F
>> >>>>>> bz.apache.org%2Fbugzilla%2Fshow_bug.cgi%3Fid%3D68901&data=05%7C02%
>> >>>>>> 7Charri.pesonen%40sinch.com%7C66e83fa3469b43288fe608dc6063a357%7C3
>> >>>>>> b518aae89214a7b8497619d756ce20e%7C0%7C0%7C638491228268558870%7CUnk
>> >>>>>> nown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1
>> >>>>>> haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=l%2B6E%2BRgRxQmicXQ9ZOQpkIc
>> >>>>>> kaFzl70rI7O8ltNNvbSs%3D&reserved=0
>> >>>>>>
>> >>>>>> Kind regards,
>> >>>>>> Stefan Ansing
>> >>>>>>
>> >>>>>> [1]:
>> >>>>>>
>> >>>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgi
>> >>>> thub.com%2Fapache%2Ftomcat%2Fblame%2Fbc900e0100de9879604b93af4722c27
>> >>>> 2ab3d1a24%2Fjava%2Forg%2Fapache%2Fcoyote%2Fhttp11%2FHttp11Processor.
>> >>>> java%23L604-L617&data=05%7C02%7Charri.pesonen%40sinch.com%7C66e83fa3
>> >>>> 469b43288fe608dc6063a357%7C3b518aae89214a7b8497619d756ce20e%7C0%7C0%
>> >>>> 7C638491228268567937%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJ
>> >>>> QIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=fcgz%
>> >>>> 2Bw473MyShaMac3v9yN%2BEnNfnX39x919bajNtC1U%3D&reserved=0
>> >>>>>>
>> >>>>>
>> >>>>> -------------------------------------------------------------------
>> >>>>> -- To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> >>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>> >>>>>
>> >>>>
>> >>>> --------------------------------------------------------------------
>> >>>> - To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> >>>> For additional commands, e-mail: users-help@tomcat.apache.org
>> >>>>
>> >>>>
>> >>>
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> >> For additional commands, e-mail: users-help@tomcat.apache.org
>> >>
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> > For additional commands, e-mail: users-help@tomcat.apache.org
>> >
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> > For additional commands, e-mail: users-help@tomcat.apache.org
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>

Re: Tomcat closes connections on unexpected status codes

[Adwait Kumar Singh <ad...@gmail.com>]

https://github.com/apache/tomcat/pull/723 is a draft PR of the idea I was
talking about earlier, i.e close the connection on a bad request but
otherwise allow it to be configurable by the user.

Currently you need to subclass Http11Processor and override
statusDropsConnections, but this is just to get early feedback. If people
are aligned with this approach, I can polish the PR (add tests, make the
status code list actually configurable etc etc)

On Fri, Apr 19, 2024 at 5:51 AM Christopher Schultz <
chris@christopherschultz.net> wrote:

> Harri,
>
> On 4/19/24 08:10, Harri Pesonen wrote:
> > I have developed a restful web service, which uses HTTP response codes
> 200 OK, 201 Created, 204 No Content and 404 Not Found.
> > It does not use 400 Bad Request or 500 Internal Server Error normally.
> > 400 Bad Request is more common than 500 Internal Server Error, which
> should basically never happen.
>
> I disagree. There aren't many good 5xx errors, so sometimes "500 Oops"
> is the best you can do. HTTP 5xx doesn't even really allow you to
> differentiate between "server error" and "application error".
>
> > 400 Bad Request is the best response in many cases, if client gives some
> query parameter which is not supported by the application logic.
>
> -1
>
> 400 means that the HTTP request is bad. In your situation, the HTTP
> request is 100% valid. It's your application that is saying "sorry,
> client, you did something we don't allow". This is an application-level
> error, not a protocol-level error. To me, 400 means "HTTP protocol
> error" and should never be substituted for "application protocol error".
>
> > I think that it would be better not to close connection in this case, if
> the error comes from application.
> > I wonder if there is option for application to define that connection
> should be closed or not after the response has been sent?
> > Or is the option only from the client.
> >
> > For me this 404 Not Found is also a small problem, as it is error, but
> it can happen quite often.
> > HTTP errors are not nice in logs.
> > Normally if you try to fetch some restful resource, which does not
> exist, then it returns 404 Not Found.
> > GET /service/resource/id => 404 Not Found
> > If I now had an option to rewrite the service, I would probably use 204
> No Content in this case as well, to avoid errors.
>
> 404 and 204 are fundamentally different responses. 404 means "this
> resource does not exist" and 204 means "this resource DOES EXIST but it
> doesn't contain anything". Your application may not differentiate
> between those two cases, but as a client I would be confused if "Not
> Found" was replaced by "Found to be empty" in all cases.
>
> > 204 No Content is normally used with PUT and DELETE requests.
>
> Yes, you can use those. 200 would also make sense and, of course 201 for
> new resources.
>
> -chris
>
> > -----Original Message-----
> > From: Christopher Schultz <ch...@christopherschultz.net>
> > Sent: perjantai 19. huhtikuuta 2024 14.27
> > To: users@tomcat.apache.org
> > Subject: Re: Tomcat closes connections on unexpected status codes
> >
> > Mark,
> >
> > On 4/18/24 11:38, Mark Thomas wrote:
> >> On 18/04/2024 15:16, Adwait Kumar Singh wrote:
> >>> I think we should *always* close connections in cases where it can
> >>> lead to request smuggling vulnerabilities like when there is an error
> >>> during header or request line parsing, but allowing the user to
> >>> control connection close when the status is being set by the user,
> >>> should be safe?
> >>
> >> I'm not (yet) convinced distinguishing between those scenarios is
> >> always going to be possible.
> >>
> >>> It allows users to send back responses like InvalidInputException
> >>> with a
> >>> 400 status without being forced to close the connection.
> >>
> >> I appreciate why a 400 is used here but 400 has always struck me as
> >> more for protocol level issues rather than application level issues.
> >
> > Didn't someone tell me recently that, technically, ANY client-error is
> allowed to trigger a 400 response without being more specific?
> >
> >> That is the fundamental problem here. The status codes are being used
> >> for two completely different purposes.
> >
> > +1
> >
> > I've always found it distasteful when REST services do this. To me, 400
> means "the request was actually malformed". If you need authentication,
> that's a 401. If you aren't allowed, that's 403. If you didn't provide a
> required header, that's a 412, etc. I've usually found the "correct"
> > response code to use for every situation and I've never written an
> application that returns a 400 response directly.
> >
> > -chris
> >
> >>> On Thu, Apr 18, 2024 at 6:41 AM Rémy Maucherat <re...@apache.org>
> wrote:
> >>>
> >>>> On Thu, Apr 18, 2024 at 1:17 PM Mark Thomas <ma...@apache.org> wrote:
> >>>>>
> >>>>> On 18/04/2024 09:07, Stefan Ansing wrote:
> >>>>>> Hi,
> >>>>>>
> >>>>>> We've observed some unexpected behaviour in Apache Tomcat (version
> >>>> 10.1.19)
> >>>>>> where we see that HTTP/1.1 connections are closed whenever a
> >>>>>> servlet application returns the following status codes: 400, 408,
> >>>>>> 411, 414,
> >>>> 500,
> >>>>>> 503, 501. This causes client applications to rapidly reconnect and
> >>>> induce
> >>>>>> high server-side CPU load due to doing TLS handshakes.
> >>>>>>
> >>>>>> The 400 and 500 status codes are commonly used in RESTful
> >>>> microservices to
> >>>>>> communicate errors. Reviewing RFC 9112 I couldn't find any
> >>>>>> requirement
> >>>> for
> >>>>>> closing connections on these status codes.
> >>>>>>
> >>>>>> After testing with Undertow (version 2.3.12), where we didn't see
> >>>>>> the
> >>>> same
> >>>>>> behaviour, we believe that these status codes do not necessitate a
> >>>>>> new connection.
> >>>>>
> >>>>> The Tomcat developers disagree. Connections are closed after these
> >>>>> status codes to avoid various forms of request smuggling attacks.
> >>>>>
> >>>>>> Checking the Tomcat sources makes me believe that the behaviour is
> >>>>>> hard-coded[1]. I'm reaching out here to re-evaluate the list of
> >>>>>> status codes and to discuss the possibilities of making the
> >>>>>> behaviour
> >>>> configurable.
> >>>>>
> >>>>> Making this list of status codes configurable seems reasonable. The
> >>>>> default can stay as current and if users want to change it then
> >>>>> they have to accept the associated security risks.
> >>>>
> >>>> If it's insecure, then it would still be a valid CVE even if the
> >>>> configuration is optional. We don't have an "allowSmuggling"
> >>>> attribute on the connector to relax header or status line parsing,
> >>>> even though many would have wanted it in the past ...
> >>>>
> >>>> Rémy
> >>>>
> >>>>> Mark
> >>>>>
> >>>>>
> >>>>>>
> >>>>>> A colleague of mine reported a bug for this issue:
> >>>>>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2F
> >>>>>> bz.apache.org%2Fbugzilla%2Fshow_bug.cgi%3Fid%3D68901&data=05%7C02%
> >>>>>> 7Charri.pesonen%40sinch.com%7C66e83fa3469b43288fe608dc6063a357%7C3
> >>>>>> b518aae89214a7b8497619d756ce20e%7C0%7C0%7C638491228268558870%7CUnk
> >>>>>> nown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1
> >>>>>> haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=l%2B6E%2BRgRxQmicXQ9ZOQpkIc
> >>>>>> kaFzl70rI7O8ltNNvbSs%3D&reserved=0
> >>>>>>
> >>>>>> Kind regards,
> >>>>>> Stefan Ansing
> >>>>>>
> >>>>>> [1]:
> >>>>>>
> >>>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgi
> >>>> thub.com%2Fapache%2Ftomcat%2Fblame%2Fbc900e0100de9879604b93af4722c27
> >>>> 2ab3d1a24%2Fjava%2Forg%2Fapache%2Fcoyote%2Fhttp11%2FHttp11Processor.
> >>>> java%23L604-L617&data=05%7C02%7Charri.pesonen%40sinch.com%7C66e83fa3
> >>>> 469b43288fe608dc6063a357%7C3b518aae89214a7b8497619d756ce20e%7C0%7C0%
> >>>> 7C638491228268567937%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJ
> >>>> QIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=fcgz%
> >>>> 2Bw473MyShaMac3v9yN%2BEnNfnX39x919bajNtC1U%3D&reserved=0
> >>>>>>
> >>>>>
> >>>>> -------------------------------------------------------------------
> >>>>> -- To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >>>>> For additional commands, e-mail: users-help@tomcat.apache.org
> >>>>>
> >>>>
> >>>> --------------------------------------------------------------------
> >>>> - To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >>>> For additional commands, e-mail: users-help@tomcat.apache.org
> >>>>
> >>>>
> >>>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >> For additional commands, e-mail: users-help@tomcat.apache.org
> >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> > For additional commands, e-mail: users-help@tomcat.apache.org
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> > For additional commands, e-mail: users-help@tomcat.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>

Re: Tomcat closes connections on unexpected status codes

[Christopher Schultz <ch...@christopherschultz.net>]

Harri,

On 4/19/24 08:10, Harri Pesonen wrote:
> I have developed a restful web service, which uses HTTP response codes 200 OK, 201 Created, 204 No Content and 404 Not Found.
> It does not use 400 Bad Request or 500 Internal Server Error normally.
> 400 Bad Request is more common than 500 Internal Server Error, which should basically never happen.

I disagree. There aren't many good 5xx errors, so sometimes "500 Oops" 
is the best you can do. HTTP 5xx doesn't even really allow you to 
differentiate between "server error" and "application error".

> 400 Bad Request is the best response in many cases, if client gives some query parameter which is not supported by the application logic.

-1

400 means that the HTTP request is bad. In your situation, the HTTP 
request is 100% valid. It's your application that is saying "sorry, 
client, you did something we don't allow". This is an application-level 
error, not a protocol-level error. To me, 400 means "HTTP protocol 
error" and should never be substituted for "application protocol error".

> I think that it would be better not to close connection in this case, if the error comes from application.
> I wonder if there is option for application to define that connection should be closed or not after the response has been sent?
> Or is the option only from the client.
> 
> For me this 404 Not Found is also a small problem, as it is error, but it can happen quite often.
> HTTP errors are not nice in logs.
> Normally if you try to fetch some restful resource, which does not exist, then it returns 404 Not Found.
> GET /service/resource/id => 404 Not Found
> If I now had an option to rewrite the service, I would probably use 204 No Content in this case as well, to avoid errors.

404 and 204 are fundamentally different responses. 404 means "this 
resource does not exist" and 204 means "this resource DOES EXIST but it 
doesn't contain anything". Your application may not differentiate 
between those two cases, but as a client I would be confused if "Not 
Found" was replaced by "Found to be empty" in all cases.

> 204 No Content is normally used with PUT and DELETE requests.

Yes, you can use those. 200 would also make sense and, of course 201 for 
new resources.

-chris

> -----Original Message-----
> From: Christopher Schultz <ch...@christopherschultz.net>
> Sent: perjantai 19. huhtikuuta 2024 14.27
> To: users@tomcat.apache.org
> Subject: Re: Tomcat closes connections on unexpected status codes
> 
> Mark,
> 
> On 4/18/24 11:38, Mark Thomas wrote:
>> On 18/04/2024 15:16, Adwait Kumar Singh wrote:
>>> I think we should *always* close connections in cases where it can
>>> lead to request smuggling vulnerabilities like when there is an error
>>> during header or request line parsing, but allowing the user to
>>> control connection close when the status is being set by the user,
>>> should be safe?
>>
>> I'm not (yet) convinced distinguishing between those scenarios is
>> always going to be possible.
>>
>>> It allows users to send back responses like InvalidInputException
>>> with a
>>> 400 status without being forced to close the connection.
>>
>> I appreciate why a 400 is used here but 400 has always struck me as
>> more for protocol level issues rather than application level issues.
> 
> Didn't someone tell me recently that, technically, ANY client-error is allowed to trigger a 400 response without being more specific?
> 
>> That is the fundamental problem here. The status codes are being used
>> for two completely different purposes.
> 
> +1
> 
> I've always found it distasteful when REST services do this. To me, 400 means "the request was actually malformed". If you need authentication, that's a 401. If you aren't allowed, that's 403. If you didn't provide a required header, that's a 412, etc. I've usually found the "correct"
> response code to use for every situation and I've never written an application that returns a 400 response directly.
> 
> -chris
> 
>>> On Thu, Apr 18, 2024 at 6:41 AM Rémy Maucherat <re...@apache.org> wrote:
>>>
>>>> On Thu, Apr 18, 2024 at 1:17 PM Mark Thomas <ma...@apache.org> wrote:
>>>>>
>>>>> On 18/04/2024 09:07, Stefan Ansing wrote:
>>>>>> Hi,
>>>>>>
>>>>>> We've observed some unexpected behaviour in Apache Tomcat (version
>>>> 10.1.19)
>>>>>> where we see that HTTP/1.1 connections are closed whenever a
>>>>>> servlet application returns the following status codes: 400, 408,
>>>>>> 411, 414,
>>>> 500,
>>>>>> 503, 501. This causes client applications to rapidly reconnect and
>>>> induce
>>>>>> high server-side CPU load due to doing TLS handshakes.
>>>>>>
>>>>>> The 400 and 500 status codes are commonly used in RESTful
>>>> microservices to
>>>>>> communicate errors. Reviewing RFC 9112 I couldn't find any
>>>>>> requirement
>>>> for
>>>>>> closing connections on these status codes.
>>>>>>
>>>>>> After testing with Undertow (version 2.3.12), where we didn't see
>>>>>> the
>>>> same
>>>>>> behaviour, we believe that these status codes do not necessitate a
>>>>>> new connection.
>>>>>
>>>>> The Tomcat developers disagree. Connections are closed after these
>>>>> status codes to avoid various forms of request smuggling attacks.
>>>>>
>>>>>> Checking the Tomcat sources makes me believe that the behaviour is
>>>>>> hard-coded[1]. I'm reaching out here to re-evaluate the list of
>>>>>> status codes and to discuss the possibilities of making the
>>>>>> behaviour
>>>> configurable.
>>>>>
>>>>> Making this list of status codes configurable seems reasonable. The
>>>>> default can stay as current and if users want to change it then
>>>>> they have to accept the associated security risks.
>>>>
>>>> If it's insecure, then it would still be a valid CVE even if the
>>>> configuration is optional. We don't have an "allowSmuggling"
>>>> attribute on the connector to relax header or status line parsing,
>>>> even though many would have wanted it in the past ...
>>>>
>>>> Rémy
>>>>
>>>>> Mark
>>>>>
>>>>>
>>>>>>
>>>>>> A colleague of mine reported a bug for this issue:
>>>>>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2F
>>>>>> bz.apache.org%2Fbugzilla%2Fshow_bug.cgi%3Fid%3D68901&data=05%7C02%
>>>>>> 7Charri.pesonen%40sinch.com%7C66e83fa3469b43288fe608dc6063a357%7C3
>>>>>> b518aae89214a7b8497619d756ce20e%7C0%7C0%7C638491228268558870%7CUnk
>>>>>> nown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1
>>>>>> haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=l%2B6E%2BRgRxQmicXQ9ZOQpkIc
>>>>>> kaFzl70rI7O8ltNNvbSs%3D&reserved=0
>>>>>>
>>>>>> Kind regards,
>>>>>> Stefan Ansing
>>>>>>
>>>>>> [1]:
>>>>>>
>>>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgi
>>>> thub.com%2Fapache%2Ftomcat%2Fblame%2Fbc900e0100de9879604b93af4722c27
>>>> 2ab3d1a24%2Fjava%2Forg%2Fapache%2Fcoyote%2Fhttp11%2FHttp11Processor.
>>>> java%23L604-L617&data=05%7C02%7Charri.pesonen%40sinch.com%7C66e83fa3
>>>> 469b43288fe608dc6063a357%7C3b518aae89214a7b8497619d756ce20e%7C0%7C0%
>>>> 7C638491228268567937%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJ
>>>> QIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=fcgz%
>>>> 2Bw473MyShaMac3v9yN%2BEnNfnX39x919bajNtC1U%3D&reserved=0
>>>>>>
>>>>>
>>>>> -------------------------------------------------------------------
>>>>> -- To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>>
>>>>
>>>> --------------------------------------------------------------------
>>>> - To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>
>>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

RE: Tomcat closes connections on unexpected status codes

[Harri Pesonen <ha...@sinch.com>]

I have developed a restful web service, which uses HTTP response codes 200 OK, 201 Created, 204 No Content and 404 Not Found.
It does not use 400 Bad Request or 500 Internal Server Error normally.
400 Bad Request is more common than 500 Internal Server Error, which should basically never happen.
400 Bad Request is the best response in many cases, if client gives some query parameter which is not supported by the application logic.
I think that it would be better not to close connection in this case, if the error comes from application.
I wonder if there is option for application to define that connection should be closed or not after the response has been sent? 
Or is the option only from the client.

For me this 404 Not Found is also a small problem, as it is error, but it can happen quite often.
HTTP errors are not nice in logs.
Normally if you try to fetch some restful resource, which does not exist, then it returns 404 Not Found.
GET /service/resource/id => 404 Not Found
If I now had an option to rewrite the service, I would probably use 204 No Content in this case as well, to avoid errors.
204 No Content is normally used with PUT and DELETE requests.

-Harri

-----Original Message-----
From: Christopher Schultz <ch...@christopherschultz.net> 
Sent: perjantai 19. huhtikuuta 2024 14.27
To: users@tomcat.apache.org
Subject: Re: Tomcat closes connections on unexpected status codes

Mark,

On 4/18/24 11:38, Mark Thomas wrote:
> On 18/04/2024 15:16, Adwait Kumar Singh wrote:
>> I think we should *always* close connections in cases where it can 
>> lead to request smuggling vulnerabilities like when there is an error 
>> during header or request line parsing, but allowing the user to 
>> control connection close when the status is being set by the user, 
>> should be safe?
> 
> I'm not (yet) convinced distinguishing between those scenarios is 
> always going to be possible.
> 
>> It allows users to send back responses like InvalidInputException 
>> with a
>> 400 status without being forced to close the connection.
> 
> I appreciate why a 400 is used here but 400 has always struck me as 
> more for protocol level issues rather than application level issues.

Didn't someone tell me recently that, technically, ANY client-error is allowed to trigger a 400 response without being more specific?

> That is the fundamental problem here. The status codes are being used 
> for two completely different purposes.

+1

I've always found it distasteful when REST services do this. To me, 400 means "the request was actually malformed". If you need authentication, that's a 401. If you aren't allowed, that's 403. If you didn't provide a required header, that's a 412, etc. I've usually found the "correct" 
response code to use for every situation and I've never written an application that returns a 400 response directly.

-chris

>> On Thu, Apr 18, 2024 at 6:41 AM Rémy Maucherat <re...@apache.org> wrote:
>>
>>> On Thu, Apr 18, 2024 at 1:17 PM Mark Thomas <ma...@apache.org> wrote:
>>>>
>>>> On 18/04/2024 09:07, Stefan Ansing wrote:
>>>>> Hi,
>>>>>
>>>>> We've observed some unexpected behaviour in Apache Tomcat (version
>>> 10.1.19)
>>>>> where we see that HTTP/1.1 connections are closed whenever a 
>>>>> servlet application returns the following status codes: 400, 408, 
>>>>> 411, 414,
>>> 500,
>>>>> 503, 501. This causes client applications to rapidly reconnect and
>>> induce
>>>>> high server-side CPU load due to doing TLS handshakes.
>>>>>
>>>>> The 400 and 500 status codes are commonly used in RESTful
>>> microservices to
>>>>> communicate errors. Reviewing RFC 9112 I couldn't find any 
>>>>> requirement
>>> for
>>>>> closing connections on these status codes.
>>>>>
>>>>> After testing with Undertow (version 2.3.12), where we didn't see 
>>>>> the
>>> same
>>>>> behaviour, we believe that these status codes do not necessitate a 
>>>>> new connection.
>>>>
>>>> The Tomcat developers disagree. Connections are closed after these 
>>>> status codes to avoid various forms of request smuggling attacks.
>>>>
>>>>> Checking the Tomcat sources makes me believe that the behaviour is 
>>>>> hard-coded[1]. I'm reaching out here to re-evaluate the list of 
>>>>> status codes and to discuss the possibilities of making the 
>>>>> behaviour
>>> configurable.
>>>>
>>>> Making this list of status codes configurable seems reasonable. The 
>>>> default can stay as current and if users want to change it then 
>>>> they have to accept the associated security risks.
>>>
>>> If it's insecure, then it would still be a valid CVE even if the 
>>> configuration is optional. We don't have an "allowSmuggling" 
>>> attribute on the connector to relax header or status line parsing, 
>>> even though many would have wanted it in the past ...
>>>
>>> Rémy
>>>
>>>> Mark
>>>>
>>>>
>>>>>
>>>>> A colleague of mine reported a bug for this issue:
>>>>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2F
>>>>> bz.apache.org%2Fbugzilla%2Fshow_bug.cgi%3Fid%3D68901&data=05%7C02%
>>>>> 7Charri.pesonen%40sinch.com%7C66e83fa3469b43288fe608dc6063a357%7C3
>>>>> b518aae89214a7b8497619d756ce20e%7C0%7C0%7C638491228268558870%7CUnk
>>>>> nown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1
>>>>> haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=l%2B6E%2BRgRxQmicXQ9ZOQpkIc
>>>>> kaFzl70rI7O8ltNNvbSs%3D&reserved=0
>>>>>
>>>>> Kind regards,
>>>>> Stefan Ansing
>>>>>
>>>>> [1]:
>>>>>
>>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgi
>>> thub.com%2Fapache%2Ftomcat%2Fblame%2Fbc900e0100de9879604b93af4722c27
>>> 2ab3d1a24%2Fjava%2Forg%2Fapache%2Fcoyote%2Fhttp11%2FHttp11Processor.
>>> java%23L604-L617&data=05%7C02%7Charri.pesonen%40sinch.com%7C66e83fa3
>>> 469b43288fe608dc6063a357%7C3b518aae89214a7b8497619d756ce20e%7C0%7C0%
>>> 7C638491228268567937%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJ
>>> QIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=fcgz%
>>> 2Bw473MyShaMac3v9yN%2BEnNfnX39x919bajNtC1U%3D&reserved=0
>>>>>
>>>>
>>>> -------------------------------------------------------------------
>>>> -- To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>
>>>
>>> --------------------------------------------------------------------
>>> - To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>>
>>
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: Tomcat closes connections on unexpected status codes

[Christopher Schultz <ch...@christopherschultz.net>]

Mark,

On 4/18/24 11:38, Mark Thomas wrote:
> On 18/04/2024 15:16, Adwait Kumar Singh wrote:
>> I think we should *always* close connections in cases where it can 
>> lead to
>> request smuggling vulnerabilities like when there is an error during 
>> header
>> or request line parsing, but allowing the user to control connection 
>> close
>> when the status is being set by the user, should be safe?
> 
> I'm not (yet) convinced distinguishing between those scenarios is always 
> going to be possible.
> 
>> It allows users to send back responses like InvalidInputException with a
>> 400 status without being forced to close the connection.
> 
> I appreciate why a 400 is used here but 400 has always struck me as more 
> for protocol level issues rather than application level issues.

Didn't someone tell me recently that, technically, ANY client-error is 
allowed to trigger a 400 response without being more specific?

> That is the fundamental problem here. The status codes are being used 
> for two completely different purposes.

+1

I've always found it distasteful when REST services do this. To me, 400 
means "the request was actually malformed". If you need authentication, 
that's a 401. If you aren't allowed, that's 403. If you didn't provide a 
required header, that's a 412, etc. I've usually found the "correct" 
response code to use for every situation and I've never written an 
application that returns a 400 response directly.

-chris

>> On Thu, Apr 18, 2024 at 6:41 AM Rémy Maucherat <re...@apache.org> wrote:
>>
>>> On Thu, Apr 18, 2024 at 1:17 PM Mark Thomas <ma...@apache.org> wrote:
>>>>
>>>> On 18/04/2024 09:07, Stefan Ansing wrote:
>>>>> Hi,
>>>>>
>>>>> We've observed some unexpected behaviour in Apache Tomcat (version
>>> 10.1.19)
>>>>> where we see that HTTP/1.1 connections are closed whenever a servlet
>>>>> application returns the following status codes: 400, 408, 411, 414,
>>> 500,
>>>>> 503, 501. This causes client applications to rapidly reconnect and
>>> induce
>>>>> high server-side CPU load due to doing TLS handshakes.
>>>>>
>>>>> The 400 and 500 status codes are commonly used in RESTful
>>> microservices to
>>>>> communicate errors. Reviewing RFC 9112 I couldn't find any requirement
>>> for
>>>>> closing connections on these status codes.
>>>>>
>>>>> After testing with Undertow (version 2.3.12), where we didn't see the
>>> same
>>>>> behaviour, we believe that these status codes do not necessitate a new
>>>>> connection.
>>>>
>>>> The Tomcat developers disagree. Connections are closed after these
>>>> status codes to avoid various forms of request smuggling attacks.
>>>>
>>>>> Checking the Tomcat sources makes me believe that the behaviour is
>>>>> hard-coded[1]. I'm reaching out here to re-evaluate the list of status
>>>>> codes and to discuss the possibilities of making the behaviour
>>> configurable.
>>>>
>>>> Making this list of status codes configurable seems reasonable. The
>>>> default can stay as current and if users want to change it then they
>>>> have to accept the associated security risks.
>>>
>>> If it's insecure, then it would still be a valid CVE even if the
>>> configuration is optional. We don't have an "allowSmuggling" attribute
>>> on the connector to relax header or status line parsing, even though
>>> many would have wanted it in the past ...
>>>
>>> Rémy
>>>
>>>> Mark
>>>>
>>>>
>>>>>
>>>>> A colleague of mine reported a bug for this issue:
>>>>> https://bz.apache.org/bugzilla/show_bug.cgi?id=68901
>>>>>
>>>>> Kind regards,
>>>>> Stefan Ansing
>>>>>
>>>>> [1]:
>>>>>
>>> https://github.com/apache/tomcat/blame/bc900e0100de9879604b93af4722c272ab3d1a24/java/org/apache/coyote/http11/Http11Processor.java#L604-L617
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>>
>>
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: Tomcat closes connections on unexpected status codes

[Mark Thomas <ma...@apache.org>]

On 18/04/2024 15:16, Adwait Kumar Singh wrote:
> I think we should *always* close connections in cases where it can lead to
> request smuggling vulnerabilities like when there is an error during header
> or request line parsing, but allowing the user to control connection close
> when the status is being set by the user, should be safe?

I'm not (yet) convinced distinguishing between those scenarios is always 
going to be possible.

> It allows users to send back responses like InvalidInputException with a
> 400 status without being forced to close the connection.

I appreciate why a 400 is used here but 400 has always struck me as more 
for protocol level issues rather than application level issues.

That is the fundamental problem here. The status codes are being used 
for two completely different purposes.

Mark



> 
> On Thu, Apr 18, 2024 at 6:41 AM Rémy Maucherat <re...@apache.org> wrote:
> 
>> On Thu, Apr 18, 2024 at 1:17 PM Mark Thomas <ma...@apache.org> wrote:
>>>
>>> On 18/04/2024 09:07, Stefan Ansing wrote:
>>>> Hi,
>>>>
>>>> We've observed some unexpected behaviour in Apache Tomcat (version
>> 10.1.19)
>>>> where we see that HTTP/1.1 connections are closed whenever a servlet
>>>> application returns the following status codes: 400, 408, 411, 414,
>> 500,
>>>> 503, 501. This causes client applications to rapidly reconnect and
>> induce
>>>> high server-side CPU load due to doing TLS handshakes.
>>>>
>>>> The 400 and 500 status codes are commonly used in RESTful
>> microservices to
>>>> communicate errors. Reviewing RFC 9112 I couldn't find any requirement
>> for
>>>> closing connections on these status codes.
>>>>
>>>> After testing with Undertow (version 2.3.12), where we didn't see the
>> same
>>>> behaviour, we believe that these status codes do not necessitate a new
>>>> connection.
>>>
>>> The Tomcat developers disagree. Connections are closed after these
>>> status codes to avoid various forms of request smuggling attacks.
>>>
>>>> Checking the Tomcat sources makes me believe that the behaviour is
>>>> hard-coded[1]. I'm reaching out here to re-evaluate the list of status
>>>> codes and to discuss the possibilities of making the behaviour
>> configurable.
>>>
>>> Making this list of status codes configurable seems reasonable. The
>>> default can stay as current and if users want to change it then they
>>> have to accept the associated security risks.
>>
>> If it's insecure, then it would still be a valid CVE even if the
>> configuration is optional. We don't have an "allowSmuggling" attribute
>> on the connector to relax header or status line parsing, even though
>> many would have wanted it in the past ...
>>
>> Rémy
>>
>>> Mark
>>>
>>>
>>>>
>>>> A colleague of mine reported a bug for this issue:
>>>> https://bz.apache.org/bugzilla/show_bug.cgi?id=68901
>>>>
>>>> Kind regards,
>>>> Stefan Ansing
>>>>
>>>> [1]:
>>>>
>> https://github.com/apache/tomcat/blame/bc900e0100de9879604b93af4722c272ab3d1a24/java/org/apache/coyote/http11/Http11Processor.java#L604-L617
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: Tomcat closes connections on unexpected status codes

[Christopher Schultz <ch...@christopherschultz.net>]

All,

On 4/18/24 10:16, Adwait Kumar Singh wrote:
> I think we should *always* close connections in cases where it can lead to
> request smuggling vulnerabilities like when there is an error during header
> or request line parsing, but allowing the user to control connection close
> when the status is being set by the user, should be safe?
> 
> It allows users to send back responses like InvalidInputException with a
> 400 status without being forced to close the connection.

Maybe the behavior depends upon the source of the response code. If 
Tomcat itself determines that the connection should be terminated, it 
should. But if the application is the one returning the 4xx error, 
Tomcat should not terminate the connection.

Does that sound like a decent compromise and is it even possible to 
differentiate?

-chris


> On Thu, Apr 18, 2024 at 6:41 AM Rémy Maucherat <re...@apache.org> wrote:
> 
>> On Thu, Apr 18, 2024 at 1:17 PM Mark Thomas <ma...@apache.org> wrote:
>>>
>>> On 18/04/2024 09:07, Stefan Ansing wrote:
>>>> Hi,
>>>>
>>>> We've observed some unexpected behaviour in Apache Tomcat (version
>> 10.1.19)
>>>> where we see that HTTP/1.1 connections are closed whenever a servlet
>>>> application returns the following status codes: 400, 408, 411, 414,
>> 500,
>>>> 503, 501. This causes client applications to rapidly reconnect and
>> induce
>>>> high server-side CPU load due to doing TLS handshakes.
>>>>
>>>> The 400 and 500 status codes are commonly used in RESTful
>> microservices to
>>>> communicate errors. Reviewing RFC 9112 I couldn't find any requirement
>> for
>>>> closing connections on these status codes.
>>>>
>>>> After testing with Undertow (version 2.3.12), where we didn't see the
>> same
>>>> behaviour, we believe that these status codes do not necessitate a new
>>>> connection.
>>>
>>> The Tomcat developers disagree. Connections are closed after these
>>> status codes to avoid various forms of request smuggling attacks.
>>>
>>>> Checking the Tomcat sources makes me believe that the behaviour is
>>>> hard-coded[1]. I'm reaching out here to re-evaluate the list of status
>>>> codes and to discuss the possibilities of making the behaviour
>> configurable.
>>>
>>> Making this list of status codes configurable seems reasonable. The
>>> default can stay as current and if users want to change it then they
>>> have to accept the associated security risks.
>>
>> If it's insecure, then it would still be a valid CVE even if the
>> configuration is optional. We don't have an "allowSmuggling" attribute
>> on the connector to relax header or status line parsing, even though
>> many would have wanted it in the past ...
>>
>> Rémy
>>
>>> Mark
>>>
>>>
>>>>
>>>> A colleague of mine reported a bug for this issue:
>>>> https://bz.apache.org/bugzilla/show_bug.cgi?id=68901
>>>>
>>>> Kind regards,
>>>> Stefan Ansing
>>>>
>>>> [1]:
>>>>
>> https://github.com/apache/tomcat/blame/bc900e0100de9879604b93af4722c272ab3d1a24/java/org/apache/coyote/http11/Http11Processor.java#L604-L617
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: Tomcat closes connections on unexpected status codes

[Adwait Kumar Singh <ad...@gmail.com>]

I think we should *always* close connections in cases where it can lead to
request smuggling vulnerabilities like when there is an error during header
or request line parsing, but allowing the user to control connection close
when the status is being set by the user, should be safe?

It allows users to send back responses like InvalidInputException with a
400 status without being forced to close the connection.

On Thu, Apr 18, 2024 at 6:41 AM Rémy Maucherat <re...@apache.org> wrote:

> On Thu, Apr 18, 2024 at 1:17 PM Mark Thomas <ma...@apache.org> wrote:
> >
> > On 18/04/2024 09:07, Stefan Ansing wrote:
> > > Hi,
> > >
> > > We've observed some unexpected behaviour in Apache Tomcat (version
> 10.1.19)
> > > where we see that HTTP/1.1 connections are closed whenever a servlet
> > > application returns the following status codes: 400, 408, 411, 414,
> 500,
> > > 503, 501. This causes client applications to rapidly reconnect and
> induce
> > > high server-side CPU load due to doing TLS handshakes.
> > >
> > > The 400 and 500 status codes are commonly used in RESTful
> microservices to
> > > communicate errors. Reviewing RFC 9112 I couldn't find any requirement
> for
> > > closing connections on these status codes.
> > >
> > > After testing with Undertow (version 2.3.12), where we didn't see the
> same
> > > behaviour, we believe that these status codes do not necessitate a new
> > > connection.
> >
> > The Tomcat developers disagree. Connections are closed after these
> > status codes to avoid various forms of request smuggling attacks.
> >
> > > Checking the Tomcat sources makes me believe that the behaviour is
> > > hard-coded[1]. I'm reaching out here to re-evaluate the list of status
> > > codes and to discuss the possibilities of making the behaviour
> configurable.
> >
> > Making this list of status codes configurable seems reasonable. The
> > default can stay as current and if users want to change it then they
> > have to accept the associated security risks.
>
> If it's insecure, then it would still be a valid CVE even if the
> configuration is optional. We don't have an "allowSmuggling" attribute
> on the connector to relax header or status line parsing, even though
> many would have wanted it in the past ...
>
> Rémy
>
> > Mark
> >
> >
> > >
> > > A colleague of mine reported a bug for this issue:
> > > https://bz.apache.org/bugzilla/show_bug.cgi?id=68901
> > >
> > > Kind regards,
> > > Stefan Ansing
> > >
> > > [1]:
> > >
> https://github.com/apache/tomcat/blame/bc900e0100de9879604b93af4722c272ab3d1a24/java/org/apache/coyote/http11/Http11Processor.java#L604-L617
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> > For additional commands, e-mail: users-help@tomcat.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>

Re: Tomcat closes connections on unexpected status codes

[Rémy Maucherat <re...@apache.org>]

On Thu, Apr 18, 2024 at 1:17 PM Mark Thomas <ma...@apache.org> wrote:
>
> On 18/04/2024 09:07, Stefan Ansing wrote:
> > Hi,
> >
> > We've observed some unexpected behaviour in Apache Tomcat (version 10.1.19)
> > where we see that HTTP/1.1 connections are closed whenever a servlet
> > application returns the following status codes: 400, 408, 411, 414, 500,
> > 503, 501. This causes client applications to rapidly reconnect and induce
> > high server-side CPU load due to doing TLS handshakes.
> >
> > The 400 and 500 status codes are commonly used in RESTful microservices to
> > communicate errors. Reviewing RFC 9112 I couldn't find any requirement for
> > closing connections on these status codes.
> >
> > After testing with Undertow (version 2.3.12), where we didn't see the same
> > behaviour, we believe that these status codes do not necessitate a new
> > connection.
>
> The Tomcat developers disagree. Connections are closed after these
> status codes to avoid various forms of request smuggling attacks.
>
> > Checking the Tomcat sources makes me believe that the behaviour is
> > hard-coded[1]. I'm reaching out here to re-evaluate the list of status
> > codes and to discuss the possibilities of making the behaviour configurable.
>
> Making this list of status codes configurable seems reasonable. The
> default can stay as current and if users want to change it then they
> have to accept the associated security risks.

If it's insecure, then it would still be a valid CVE even if the
configuration is optional. We don't have an "allowSmuggling" attribute
on the connector to relax header or status line parsing, even though
many would have wanted it in the past ...

Rémy

> Mark
>
>
> >
> > A colleague of mine reported a bug for this issue:
> > https://bz.apache.org/bugzilla/show_bug.cgi?id=68901
> >
> > Kind regards,
> > Stefan Ansing
> >
> > [1]:
> > https://github.com/apache/tomcat/blame/bc900e0100de9879604b93af4722c272ab3d1a24/java/org/apache/coyote/http11/Http11Processor.java#L604-L617
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: Tomcat closes connections on unexpected status codes

[Mark Thomas <ma...@apache.org>]

On 18/04/2024 09:07, Stefan Ansing wrote:
> Hi,
> 
> We've observed some unexpected behaviour in Apache Tomcat (version 10.1.19)
> where we see that HTTP/1.1 connections are closed whenever a servlet
> application returns the following status codes: 400, 408, 411, 414, 500,
> 503, 501. This causes client applications to rapidly reconnect and induce
> high server-side CPU load due to doing TLS handshakes.
> 
> The 400 and 500 status codes are commonly used in RESTful microservices to
> communicate errors. Reviewing RFC 9112 I couldn't find any requirement for
> closing connections on these status codes.
> 
> After testing with Undertow (version 2.3.12), where we didn't see the same
> behaviour, we believe that these status codes do not necessitate a new
> connection.

The Tomcat developers disagree. Connections are closed after these 
status codes to avoid various forms of request smuggling attacks.

> Checking the Tomcat sources makes me believe that the behaviour is
> hard-coded[1]. I'm reaching out here to re-evaluate the list of status
> codes and to discuss the possibilities of making the behaviour configurable.

Making this list of status codes configurable seems reasonable. The 
default can stay as current and if users want to change it then they 
have to accept the associated security risks.

Mark


> 
> A colleague of mine reported a bug for this issue:
> https://bz.apache.org/bugzilla/show_bug.cgi?id=68901
> 
> Kind regards,
> Stefan Ansing
> 
> [1]:
> https://github.com/apache/tomcat/blame/bc900e0100de9879604b93af4722c272ab3d1a24/java/org/apache/coyote/http11/Http11Processor.java#L604-L617
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org