You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Leon Atherton <le...@idrsolutions.com> on 2021/05/06 10:25:56 UTC
Firefox triggers HTTP2 overhead protection - known issue?
We are seeing that Firefox triggers the HTTP2 overhead protection with
multipart file uploads. About 1MB is uploaded before overhead protection
is triggered. I believe a few weeks ago Chrome was triggering this too,
but it looks like a recent update may have resolved it.
This is on Tomcat 9.0.45 (though it applies to earlier versions too)
using the org.apache.coyote.http11.Http11NioProtocol connector with
upgrade to org.apache.coyote.http2.Http2Protocol.
With http2 debug logging enabled, there are occasional Payload size [1]
interspersed between larger payloads, though it appears to be a couple
of back-to-back Payload size [1]'s which may be triggering the
org.apache.coyote.http2.ConnectionException: Connection [0], Too much
overhead so the connection will be closed.
I've found earlier Tomcat conversations which seem to indicate it's a
client issue. I've searched through the Firefox bug tracker but have not
found anything that looks like this (feels like looking for a needle in
a haystack).
Is this a known issue in Tomcat or Firefox? It can be resolved by
disabling the protection or potentially just tweaking the config values,
but if it's a known Firefox issue I'd like to know if something has been
raised on their bug tracker I can follow?
Thanks
Leon
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Firefox triggers HTTP2 overhead protection - known issue?
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Leon,
On 5/6/21 12:35, Leon Atherton wrote:
> On 06/05/2021 17:13, Leon Atherton wrote:
>> On 06/05/2021 16:06, Christopher Schultz wrote:
>>> On 5/6/21 09:36, Mark Thomas wrote:
>>>> On 06/05/2021 13:33, Christopher Schultz wrote:
>>>>> Leon,
>>>>>
>>>>> On 5/6/21 06:25, Leon Atherton wrote:
>>>>>> We are seeing that Firefox triggers the HTTP2 overhead protection
>>>>>> with multipart file uploads. About 1MB is uploaded before overhead
>>>>>> protection is triggered. I believe a few weeks ago Chrome was
>>>>>> triggering this too, but it looks like a recent update may have
>>>>>> resolved it.
>>>>>>
>>>>>> This is on Tomcat 9.0.45 (though it applies to earlier versions
>>>>>> too) using the org.apache.coyote.http11.Http11NioProtocol
>>>>>> connector with upgrade to org.apache.coyote.http2.Http2Protocol.
>>>>>>
>>>>>> With http2 debug logging enabled, there are occasional Payload
>>>>>> size [1] interspersed between larger payloads, though it appears
>>>>>> to be a couple of back-to-back Payload size [1]'s which may be
>>>>>> triggering the org.apache.coyote.http2.ConnectionException:
>>>>>> Connection [0], Too much overhead so the connection will be closed.
>>>>>>
>>>>>> I've found earlier Tomcat conversations which seem to indicate
>>>>>> it's a client issue. I've searched through the Firefox bug tracker
>>>>>> but have not found anything that looks like this (feels like
>>>>>> looking for a needle in a haystack).
>>>>>>
>>>>>> Is this a known issue in Tomcat or Firefox? It can be resolved by
>>>>>> disabling the protection or potentially just tweaking the config
>>>>>> values, but if it's a known Firefox issue I'd like to know if
>>>>>> something has been raised on their bug tracker I can follow?
>>>>>
>>>>> Nothing in the tracker that I know of off-hand, but there is this
>>>>> informative answer on SO by markt which may help you a little:
>>>>>
>>>>> https://stackoverflow.com/a/61454503/276232
>>>>>
>>>>> Using the information in that answer may allow you to reconfigure
>>>>> your server to allow Firefox's behavior.
>>>>>
>>>>> It's probably worth us taking some time to adapt markt's SO answer
>>>>> there into a whole section on "Protocol Abuse and Protection
>>>>> Features" in the HTTP/2 configuration guide.
>>>>
>>>> There is an open issue for Chrome:
>>>> https://bugs.chromium.org/p/chromium/issues/detail?id=1000809
>>>
>>> ACK
>>>
>>>> but there hasn't been much movement.
>>>>
>>>> I'll add something to my TODO list to see what can be done to avoid
>>>> false positives like this as it looks like short sequences of small
>>>> packets can occur depending on how the client buffers data.
>>>
>>> Maybe a tweak to the default settings can improve things for "the
>>> status quo" for current web browser behavior.
>>
>> Thanks guys. I'm currently unable to reproduce the problem in Chrome
>> (not seeing any 1 byte payloads). I thought it reproduced a few weeks
>> ago, but perhaps I'm misremembering.
>>
>> Firefox's market share isn't what it was, but it would be good if this
>> could work out of the box - either by tweaking the Tomcat code or
>> default config to prevent the false-positive or opening a ticket so
>> the Firefox team know what to fix.
>>
>> Happy to help with the latter, though I'd need some assistance to
>> explain the problem and expected solution. I could bore you to tears
>> with the ins and outs of the PDF spec, but HTTP2 is not one of my
>> specialist subjects.
>>
>> Leon
>
> Just to confirm that I have been able to reproduce this on an older
> build of Chromium (88.0.4324.0).
Sounds like more recent builds of Chrome have addressed this issue, or
have at least made the h2 component more well-behaved.
> Seems like it's the same issue as the current version of Firefox -
> consequtive 1 byte payloads triggering the overhead protection.
Firefox may be working on tweaking this stuff. I would file a bug
against ff or comment on any existing related ones you may be able to find.
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Firefox triggers HTTP2 overhead protection - known issue?
Posted by Leon Atherton <le...@idrsolutions.com>.
On 06/05/2021 17:13, Leon Atherton wrote:
> On 06/05/2021 16:06, Christopher Schultz wrote:
>> On 5/6/21 09:36, Mark Thomas wrote:
>>> On 06/05/2021 13:33, Christopher Schultz wrote:
>>>> Leon,
>>>>
>>>> On 5/6/21 06:25, Leon Atherton wrote:
>>>>> We are seeing that Firefox triggers the HTTP2 overhead protection
>>>>> with multipart file uploads. About 1MB is uploaded before overhead
>>>>> protection is triggered. I believe a few weeks ago Chrome was
>>>>> triggering this too, but it looks like a recent update may have
>>>>> resolved it.
>>>>>
>>>>> This is on Tomcat 9.0.45 (though it applies to earlier versions
>>>>> too) using the org.apache.coyote.http11.Http11NioProtocol
>>>>> connector with upgrade to org.apache.coyote.http2.Http2Protocol.
>>>>>
>>>>> With http2 debug logging enabled, there are occasional Payload
>>>>> size [1] interspersed between larger payloads, though it appears
>>>>> to be a couple of back-to-back Payload size [1]'s which may be
>>>>> triggering the org.apache.coyote.http2.ConnectionException:
>>>>> Connection [0], Too much overhead so the connection will be closed.
>>>>>
>>>>> I've found earlier Tomcat conversations which seem to indicate
>>>>> it's a client issue. I've searched through the Firefox bug tracker
>>>>> but have not found anything that looks like this (feels like
>>>>> looking for a needle in a haystack).
>>>>>
>>>>> Is this a known issue in Tomcat or Firefox? It can be resolved by
>>>>> disabling the protection or potentially just tweaking the config
>>>>> values, but if it's a known Firefox issue I'd like to know if
>>>>> something has been raised on their bug tracker I can follow?
>>>>
>>>> Nothing in the tracker that I know of off-hand, but there is this
>>>> informative answer on SO by markt which may help you a little:
>>>>
>>>> https://stackoverflow.com/a/61454503/276232
>>>>
>>>> Using the information in that answer may allow you to reconfigure
>>>> your server to allow Firefox's behavior.
>>>>
>>>> It's probably worth us taking some time to adapt markt's SO answer
>>>> there into a whole section on "Protocol Abuse and Protection
>>>> Features" in the HTTP/2 configuration guide.
>>>
>>> There is an open issue for Chrome:
>>> https://bugs.chromium.org/p/chromium/issues/detail?id=1000809
>>
>> ACK
>>
>>> but there hasn't been much movement.
>>>
>>> I'll add something to my TODO list to see what can be done to avoid
>>> false positives like this as it looks like short sequences of small
>>> packets can occur depending on how the client buffers data.
>>
>> Maybe a tweak to the default settings can improve things for "the
>> status quo" for current web browser behavior.
>
> Thanks guys. I'm currently unable to reproduce the problem in Chrome
> (not seeing any 1 byte payloads). I thought it reproduced a few weeks
> ago, but perhaps I'm misremembering.
>
> Firefox's market share isn't what it was, but it would be good if this
> could work out of the box - either by tweaking the Tomcat code or
> default config to prevent the false-positive or opening a ticket so
> the Firefox team know what to fix.
>
> Happy to help with the latter, though I'd need some assistance to
> explain the problem and expected solution. I could bore you to tears
> with the ins and outs of the PDF spec, but HTTP2 is not one of my
> specialist subjects.
>
> Leon
Just to confirm that I have been able to reproduce this on an older
build of Chromium (88.0.4324.0).
Seems like it's the same issue as the current version of Firefox -
consequtive 1 byte payloads triggering the overhead protection.
Glad I'm not going crazy.
Leon
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Firefox triggers HTTP2 overhead protection - known issue?
Posted by Leon Atherton <le...@idrsolutions.com>.
On 06/05/2021 16:06, Christopher Schultz wrote:
> On 5/6/21 09:36, Mark Thomas wrote:
>> On 06/05/2021 13:33, Christopher Schultz wrote:
>>> Leon,
>>>
>>> On 5/6/21 06:25, Leon Atherton wrote:
>>>> We are seeing that Firefox triggers the HTTP2 overhead protection
>>>> with multipart file uploads. About 1MB is uploaded before overhead
>>>> protection is triggered. I believe a few weeks ago Chrome was
>>>> triggering this too, but it looks like a recent update may have
>>>> resolved it.
>>>>
>>>> This is on Tomcat 9.0.45 (though it applies to earlier versions
>>>> too) using the org.apache.coyote.http11.Http11NioProtocol connector
>>>> with upgrade to org.apache.coyote.http2.Http2Protocol.
>>>>
>>>> With http2 debug logging enabled, there are occasional Payload size
>>>> [1] interspersed between larger payloads, though it appears to be a
>>>> couple of back-to-back Payload size [1]'s which may be triggering
>>>> the org.apache.coyote.http2.ConnectionException: Connection [0],
>>>> Too much overhead so the connection will be closed.
>>>>
>>>> I've found earlier Tomcat conversations which seem to indicate it's
>>>> a client issue. I've searched through the Firefox bug tracker but
>>>> have not found anything that looks like this (feels like looking
>>>> for a needle in a haystack).
>>>>
>>>> Is this a known issue in Tomcat or Firefox? It can be resolved by
>>>> disabling the protection or potentially just tweaking the config
>>>> values, but if it's a known Firefox issue I'd like to know if
>>>> something has been raised on their bug tracker I can follow?
>>>
>>> Nothing in the tracker that I know of off-hand, but there is this
>>> informative answer on SO by markt which may help you a little:
>>>
>>> https://stackoverflow.com/a/61454503/276232
>>>
>>> Using the information in that answer may allow you to reconfigure
>>> your server to allow Firefox's behavior.
>>>
>>> It's probably worth us taking some time to adapt markt's SO answer
>>> there into a whole section on "Protocol Abuse and Protection
>>> Features" in the HTTP/2 configuration guide.
>>
>> There is an open issue for Chrome:
>> https://bugs.chromium.org/p/chromium/issues/detail?id=1000809
>
> ACK
>
>> but there hasn't been much movement.
>>
>> I'll add something to my TODO list to see what can be done to avoid
>> false positives like this as it looks like short sequences of small
>> packets can occur depending on how the client buffers data.
>
> Maybe a tweak to the default settings can improve things for "the
> status quo" for current web browser behavior.
Thanks guys. I'm currently unable to reproduce the problem in Chrome
(not seeing any 1 byte payloads). I thought it reproduced a few weeks
ago, but perhaps I'm misremembering.
Firefox's market share isn't what it was, but it would be good if this
could work out of the box - either by tweaking the Tomcat code or
default config to prevent the false-positive or opening a ticket so the
Firefox team know what to fix.
Happy to help with the latter, though I'd need some assistance to
explain the problem and expected solution. I could bore you to tears
with the ins and outs of the PDF spec, but HTTP2 is not one of my
specialist subjects.
Leon
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Firefox triggers HTTP2 overhead protection - known issue?
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Mark,
On 5/6/21 09:36, Mark Thomas wrote:
> On 06/05/2021 13:33, Christopher Schultz wrote:
>> Leon,
>>
>> On 5/6/21 06:25, Leon Atherton wrote:
>>> We are seeing that Firefox triggers the HTTP2 overhead protection
>>> with multipart file uploads. About 1MB is uploaded before overhead
>>> protection is triggered. I believe a few weeks ago Chrome was
>>> triggering this too, but it looks like a recent update may have
>>> resolved it.
>>>
>>> This is on Tomcat 9.0.45 (though it applies to earlier versions too)
>>> using the org.apache.coyote.http11.Http11NioProtocol connector with
>>> upgrade to org.apache.coyote.http2.Http2Protocol.
>>>
>>> With http2 debug logging enabled, there are occasional Payload size
>>> [1] interspersed between larger payloads, though it appears to be a
>>> couple of back-to-back Payload size [1]'s which may be triggering the
>>> org.apache.coyote.http2.ConnectionException: Connection [0], Too much
>>> overhead so the connection will be closed.
>>>
>>> I've found earlier Tomcat conversations which seem to indicate it's a
>>> client issue. I've searched through the Firefox bug tracker but have
>>> not found anything that looks like this (feels like looking for a
>>> needle in a haystack).
>>>
>>> Is this a known issue in Tomcat or Firefox? It can be resolved by
>>> disabling the protection or potentially just tweaking the config
>>> values, but if it's a known Firefox issue I'd like to know if
>>> something has been raised on their bug tracker I can follow?
>>
>> Nothing in the tracker that I know of off-hand, but there is this
>> informative answer on SO by markt which may help you a little:
>>
>> https://stackoverflow.com/a/61454503/276232
>>
>> Using the information in that answer may allow you to reconfigure your
>> server to allow Firefox's behavior.
>>
>> It's probably worth us taking some time to adapt markt's SO answer
>> there into a whole section on "Protocol Abuse and Protection Features"
>> in the HTTP/2 configuration guide.
>
> There is an open issue for Chrome:
> https://bugs.chromium.org/p/chromium/issues/detail?id=1000809
ACK
> but there hasn't been much movement.
>
> I'll add something to my TODO list to see what can be done to avoid
> false positives like this as it looks like short sequences of small
> packets can occur depending on how the client buffers data.
Maybe a tweak to the default settings can improve things for "the status
quo" for current web browser behavior.
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Firefox triggers HTTP2 overhead protection - known issue?
Posted by Mark Thomas <ma...@apache.org>.
On 11/06/2021 12:40, Mark Thomas wrote:
> On 06/05/2021 14:36, Mark Thomas wrote:
>
> <snip/>
>
>>> It's probably worth us taking some time to adapt markt's SO answer
>>> there into a whole section on "Protocol Abuse and Protection
>>> Features" in the HTTP/2 configuration guide.
>>
>> There is an open issue for Chrome:
>> https://bugs.chromium.org/p/chromium/issues/detail?id=1000809
>>
>> but there hasn't been much movement.
>>
>> I'll add something to my TODO list to see what can be done to avoid
>> false positives like this as it looks like short sequences of small
>> packets can occur depending on how the client buffers data.
>
> I've been doing some testing with both Chrome and FireFox and it appears
> that the behaviour of both has changed and that Tomcat's current
> overhead protection is likely to get triggered in normal usage.
>
> I'm currently investigating different overhead settings. I am trying to
> achieve the following:
>
> - a steady stream of normal requests should result in a steady fall in
> the overhead count (it currently results in no change over time)
>
> - the occasional "silly" frame (such as 2 30 byte data frames in the
> middle of a 700K upload) should not trigger the excessive overhead
> protection
>
> - there should be a greater range of possible values for
> overheadCountFactor other than 1 (normal), 2 (aggresive) and (>2 nothing
> will work)
>
> - additional debug logging so it is easier to track the current overhead
> count and what is driving it
>
> I hope to have something in time for the July release round.
I think there is merit in the above approach but it doesn't fully
address the issue with FireFox.
After some further investigation I have opted for a different approach
to address this issue.
Clients tend to send small amounts of data when they reach the end of
the buffer. If the client buffer size and the HTTP/2 initial window size
are similar, these writes can be very small.
These small writes are what Tomcat thinks are abusive (because small
writes in the middle of a request body are inefficient and some servers
could be DoS'd with lots of them).
Tomcat currently responds with a WINDOW_UPDATE after every read. Small
client writes mean small window updates.
Clients also appear to write until the current flow control window is
empty and then look to see if there is a window update. If there is,
they apply that Window update and carry on writing. If the window update
is small, so is the subsequent write.
The fix that seems to be working well for Chrome and Firefox is to delay
any small window updates for the stream, combine them with subsequent
window updates and only send the window update when there is a
reasonable (> overheadWindowUpdateThreshold) sized update to send.
I also have some additional debug logging for sending Window updates and
for tracking the current overhead count that I'll add.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Firefox triggers HTTP2 overhead protection - known issue?
Posted by Mark Thomas <ma...@apache.org>.
On 06/05/2021 14:36, Mark Thomas wrote:
<snip/>
>> It's probably worth us taking some time to adapt markt's SO answer
>> there into a whole section on "Protocol Abuse and Protection Features"
>> in the HTTP/2 configuration guide.
>
> There is an open issue for Chrome:
> https://bugs.chromium.org/p/chromium/issues/detail?id=1000809
>
> but there hasn't been much movement.
>
> I'll add something to my TODO list to see what can be done to avoid
> false positives like this as it looks like short sequences of small
> packets can occur depending on how the client buffers data.
I've been doing some testing with both Chrome and FireFox and it appears
that the behaviour of both has changed and that Tomcat's current
overhead protection is likely to get triggered in normal usage.
I'm currently investigating different overhead settings. I am trying to
achieve the following:
- a steady stream of normal requests should result in a steady fall in
the overhead count (it currently results in no change over time)
- the occasional "silly" frame (such as 2 30 byte data frames in the
middle of a 700K upload) should not trigger the excessive overhead
protection
- there should be a greater range of possible values for
overheadCountFactor other than 1 (normal), 2 (aggresive) and (>2 nothing
will work)
- additional debug logging so it is easier to track the current overhead
count and what is driving it
I hope to have something in time for the July release round.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Firefox triggers HTTP2 overhead protection - known issue?
Posted by Mark Thomas <ma...@apache.org>.
On 06/05/2021 13:33, Christopher Schultz wrote:
> Leon,
>
> On 5/6/21 06:25, Leon Atherton wrote:
>> We are seeing that Firefox triggers the HTTP2 overhead protection with
>> multipart file uploads. About 1MB is uploaded before overhead
>> protection is triggered. I believe a few weeks ago Chrome was
>> triggering this too, but it looks like a recent update may have
>> resolved it.
>>
>> This is on Tomcat 9.0.45 (though it applies to earlier versions too)
>> using the org.apache.coyote.http11.Http11NioProtocol connector with
>> upgrade to org.apache.coyote.http2.Http2Protocol.
>>
>> With http2 debug logging enabled, there are occasional Payload size
>> [1] interspersed between larger payloads, though it appears to be a
>> couple of back-to-back Payload size [1]'s which may be triggering the
>> org.apache.coyote.http2.ConnectionException: Connection [0], Too much
>> overhead so the connection will be closed.
>>
>> I've found earlier Tomcat conversations which seem to indicate it's a
>> client issue. I've searched through the Firefox bug tracker but have
>> not found anything that looks like this (feels like looking for a
>> needle in a haystack).
>>
>> Is this a known issue in Tomcat or Firefox? It can be resolved by
>> disabling the protection or potentially just tweaking the config
>> values, but if it's a known Firefox issue I'd like to know if
>> something has been raised on their bug tracker I can follow?
>
> Nothing in the tracker that I know of off-hand, but there is this
> informative answer on SO by markt which may help you a little:
>
> https://stackoverflow.com/a/61454503/276232
>
> Using the information in that answer may allow you to reconfigure your
> server to allow Firefox's behavior.
>
> It's probably worth us taking some time to adapt markt's SO answer there
> into a whole section on "Protocol Abuse and Protection Features" in the
> HTTP/2 configuration guide.
There is an open issue for Chrome:
https://bugs.chromium.org/p/chromium/issues/detail?id=1000809
but there hasn't been much movement.
I'll add something to my TODO list to see what can be done to avoid
false positives like this as it looks like short sequences of small
packets can occur depending on how the client buffers data.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Firefox triggers HTTP2 overhead protection - known issue?
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Leon,
On 5/6/21 06:25, Leon Atherton wrote:
> We are seeing that Firefox triggers the HTTP2 overhead protection with
> multipart file uploads. About 1MB is uploaded before overhead protection
> is triggered. I believe a few weeks ago Chrome was triggering this too,
> but it looks like a recent update may have resolved it.
>
> This is on Tomcat 9.0.45 (though it applies to earlier versions too)
> using the org.apache.coyote.http11.Http11NioProtocol connector with
> upgrade to org.apache.coyote.http2.Http2Protocol.
>
> With http2 debug logging enabled, there are occasional Payload size [1]
> interspersed between larger payloads, though it appears to be a couple
> of back-to-back Payload size [1]'s which may be triggering the
> org.apache.coyote.http2.ConnectionException: Connection [0], Too much
> overhead so the connection will be closed.
>
> I've found earlier Tomcat conversations which seem to indicate it's a
> client issue. I've searched through the Firefox bug tracker but have not
> found anything that looks like this (feels like looking for a needle in
> a haystack).
>
> Is this a known issue in Tomcat or Firefox? It can be resolved by
> disabling the protection or potentially just tweaking the config values,
> but if it's a known Firefox issue I'd like to know if something has been
> raised on their bug tracker I can follow?
Nothing in the tracker that I know of off-hand, but there is this
informative answer on SO by markt which may help you a little:
https://stackoverflow.com/a/61454503/276232
Using the information in that answer may allow you to reconfigure your
server to allow Firefox's behavior.
It's probably worth us taking some time to adapt markt's SO answer there
into a whole section on "Protocol Abuse and Protection Features" in the
HTTP/2 configuration guide.
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org