You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Baron Fujimoto <ba...@hawaii.edu> on 2024/04/09 20:33:18 UTC
Tomcat log warnings for connection parameter limits?
I'm investigating occasional 503 errors for our CAS service running in a
Tomcat 10.1.x container. The 503s appear to correlate with some traffic
spikes at the same time.
The connector is configured as follows:
<Connector protocol="org.apache.coyote.http11.Http11NioProtocol"
port="8443"
maxThreads="2500"
maxConnections="50000"
maxPostSize="100000"
maxParameterCount="1000"
scheme="https" secure="true"
SSLEnabled="true"
>
Can Tomcat log info such as when the maxThreads or maxConnections limits
are reached? I'm basically trying to see if there is a good way to
more definitively determine what may have caused the 503s and what may be
feasible to mitigate them.
--
Baron Fujimoto <ba...@hawaii.edu> ::: UH Information Technology Services
minutas cantorum, minutas balorum, minutas carboratum descendus pantorum
Re: Tomcat log warnings for connection parameter limits?
Posted by Baron Fujimoto <ba...@hawaii.edu>.
Thanks, Mark. Submitted <
https://bz.apache.org/bugzilla/show_bug.cgi?id=68934>
On Tue, Apr 16, 2024 at 4:44 AM Mark Thomas <ma...@apache.org> wrote:
> It would be worth creating an enhancement request for this in Bugzilla
> to ensure the request doesn't get forgotten about.
>
> Mark
>
>
> On 16/04/2024 01:06, Baron Fujimoto wrote:
> > From our perspective, it needn't be super timely. It would be more for
> > forensic confirmation that there's something we should consider. I think
> a
> > hysteresis behavior would be compatible with this.
> >
> > On Mon, Apr 15, 2024 at 12:00 AM Mark Thomas <ma...@apache.org> wrote:
> >
> >> On 11/04/2024 21:28, Baron Fujimoto wrote> I was thinking it would be
> >> something that would be left on in a live> system. We can set these
> >> parameters, so it would be useful to know if we
> >>> were hitting the set limits.
> >>
> >> For the connection limit:
> >>
> >> How timely do you need the information to be? It is much easier to add a
> >> check when the connection is closed since a) it returns the current
> >> count and b) we are already checking the return value for mis-counting.
> >>
> >> For the thread limit:
> >>
> >> We have a couple of options here. (We use a custom queue which
> >> effectively changes the way threads in the pool re used.) The simplest
> >> is probably to add log in TaskQueue.offer() if all threads are currently
> >> being used.
> >>
> >>> I'm not sure I fully grasp how this additional logging presents a
> >>> significant incremental DOS risk. I mean, if an attacker is flooding
> you
> >>> with enough traffic or connections where this becomes an issue, aren't
> >> you
> >>> already logging various aspects of the attempts anyway (e.g., access
> >> logs,
> >>> etc)? At that point aren't your logs already being filled anyway?
> >>
> >> It is an area we need to be careful with. Log messages that include
> >> exceptions are of greatest concern since they are relatively expensive
> >> to generate. If an invalid HTTP message triggers a log message with an
> >> exception (it does, but only when configured for debug logging) then
> >> there is a degree of DoS risk there.
> >>
> >> Access logs are a special case and one that we have taken care to try
> >> and optimise. Still, if you want to make a performance test look good
> >> turning off access logging is one of the first things to do (and of
> >> course results in totally meaningless results that don't reflect real
> >> world usage).
> >>
> >> I think the concern Chris was trying to express (although I'm sure he'll
> >> correct me if I am wrong) is that when these limits are reached there is
> >> a risk of a lot of log messages being generated and that can cause
> >> issues both from a performance issue and from just filling the logs with
> >> the same message over and over again.
> >>
> >> I think what we would need is some form of hysteresis. Log a message
> >> when the limit is reached, log another message when usage falls below
> >> 80%? of the limit. That way, administrators see the peaks in usage but
> >> aren't inundated with log messages.
> >>
> >> My only concern is that at least some this code is going to run for
> >> every single request. It needs to be efficient. We should measure the
> >> impact of adding this code before we decide on whether to include it.
> >>
> >> Mark
> >>
> >>
> >>>
> >>> On Thu, Apr 11, 2024 at 1:44 AM Christopher Schultz <
> >>> chris@christopherschultz.net> wrote:
> >>>
> >>>> Baron,
> >>>>
> >>>> On 4/9/24 16:33, Baron Fujimoto wrote:
> >>>>> I'm investigating occasional 503 errors for our CAS service running
> in
> >> a
> >>>>> Tomcat 10.1.x container. The 503s appear to correlate with some
> traffic
> >>>>> spikes at the same time.
> >>>>>
> >>>>> The connector is configured as follows:
> >>>>>
> >>>>> <Connector
> protocol="org.apache.coyote.http11.Http11NioProtocol"
> >>>>> port="8443"
> >>>>> maxThreads="2500"
> >>>>> maxConnections="50000"
> >>>>> maxPostSize="100000"
> >>>>> maxParameterCount="1000"
> >>>>> scheme="https" secure="true"
> >>>>> SSLEnabled="true"
> >>>>> >
> >>>>>
> >>>>> Can Tomcat log info such as when the maxThreads or maxConnections
> >> limits
> >>>>> are reached? I'm basically trying to see if there is a good way to
> >>>>> more definitively determine what may have caused the 503s and what
> may
> >> be
> >>>>> feasible to mitigate them.
> >>>>
> >>>> Are you thinking of a debugging feature or something to be left-on
> for a
> >>>> live production system?
> >>>>
> >>>> Such logging might be considered a DOS vector for a live system: you
> can
> >>>> fill the logs by asking lots of trivial requests.
> >>>>
> >>>> -chris
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >>>> For additional commands, e-mail: users-help@tomcat.apache.org
> >>>>
> >>>>
> >>>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >> For additional commands, e-mail: users-help@tomcat.apache.org
> >>
> >>
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
--
Baron Fujimoto <ba...@hawaii.edu> ::: UH Information Technology Services
minutas cantorum, minutas balorum, minutas carboratum descendus pantorum
Re: Tomcat log warnings for connection parameter limits?
Posted by Mark Thomas <ma...@apache.org>.
It would be worth creating an enhancement request for this in Bugzilla
to ensure the request doesn't get forgotten about.
Mark
On 16/04/2024 01:06, Baron Fujimoto wrote:
> From our perspective, it needn't be super timely. It would be more for
> forensic confirmation that there's something we should consider. I think a
> hysteresis behavior would be compatible with this.
>
> On Mon, Apr 15, 2024 at 12:00 AM Mark Thomas <ma...@apache.org> wrote:
>
>> On 11/04/2024 21:28, Baron Fujimoto wrote> I was thinking it would be
>> something that would be left on in a live> system. We can set these
>> parameters, so it would be useful to know if we
>>> were hitting the set limits.
>>
>> For the connection limit:
>>
>> How timely do you need the information to be? It is much easier to add a
>> check when the connection is closed since a) it returns the current
>> count and b) we are already checking the return value for mis-counting.
>>
>> For the thread limit:
>>
>> We have a couple of options here. (We use a custom queue which
>> effectively changes the way threads in the pool re used.) The simplest
>> is probably to add log in TaskQueue.offer() if all threads are currently
>> being used.
>>
>>> I'm not sure I fully grasp how this additional logging presents a
>>> significant incremental DOS risk. I mean, if an attacker is flooding you
>>> with enough traffic or connections where this becomes an issue, aren't
>> you
>>> already logging various aspects of the attempts anyway (e.g., access
>> logs,
>>> etc)? At that point aren't your logs already being filled anyway?
>>
>> It is an area we need to be careful with. Log messages that include
>> exceptions are of greatest concern since they are relatively expensive
>> to generate. If an invalid HTTP message triggers a log message with an
>> exception (it does, but only when configured for debug logging) then
>> there is a degree of DoS risk there.
>>
>> Access logs are a special case and one that we have taken care to try
>> and optimise. Still, if you want to make a performance test look good
>> turning off access logging is one of the first things to do (and of
>> course results in totally meaningless results that don't reflect real
>> world usage).
>>
>> I think the concern Chris was trying to express (although I'm sure he'll
>> correct me if I am wrong) is that when these limits are reached there is
>> a risk of a lot of log messages being generated and that can cause
>> issues both from a performance issue and from just filling the logs with
>> the same message over and over again.
>>
>> I think what we would need is some form of hysteresis. Log a message
>> when the limit is reached, log another message when usage falls below
>> 80%? of the limit. That way, administrators see the peaks in usage but
>> aren't inundated with log messages.
>>
>> My only concern is that at least some this code is going to run for
>> every single request. It needs to be efficient. We should measure the
>> impact of adding this code before we decide on whether to include it.
>>
>> Mark
>>
>>
>>>
>>> On Thu, Apr 11, 2024 at 1:44 AM Christopher Schultz <
>>> chris@christopherschultz.net> wrote:
>>>
>>>> Baron,
>>>>
>>>> On 4/9/24 16:33, Baron Fujimoto wrote:
>>>>> I'm investigating occasional 503 errors for our CAS service running in
>> a
>>>>> Tomcat 10.1.x container. The 503s appear to correlate with some traffic
>>>>> spikes at the same time.
>>>>>
>>>>> The connector is configured as follows:
>>>>>
>>>>> <Connector protocol="org.apache.coyote.http11.Http11NioProtocol"
>>>>> port="8443"
>>>>> maxThreads="2500"
>>>>> maxConnections="50000"
>>>>> maxPostSize="100000"
>>>>> maxParameterCount="1000"
>>>>> scheme="https" secure="true"
>>>>> SSLEnabled="true"
>>>>> >
>>>>>
>>>>> Can Tomcat log info such as when the maxThreads or maxConnections
>> limits
>>>>> are reached? I'm basically trying to see if there is a good way to
>>>>> more definitively determine what may have caused the 503s and what may
>> be
>>>>> feasible to mitigate them.
>>>>
>>>> Are you thinking of a debugging feature or something to be left-on for a
>>>> live production system?
>>>>
>>>> Such logging might be considered a DOS vector for a live system: you can
>>>> fill the logs by asking lots of trivial requests.
>>>>
>>>> -chris
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>>>
>>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat log warnings for connection parameter limits?
Posted by Baron Fujimoto <ba...@hawaii.edu>.
From our perspective, it needn't be super timely. It would be more for
forensic confirmation that there's something we should consider. I think a
hysteresis behavior would be compatible with this.
On Mon, Apr 15, 2024 at 12:00 AM Mark Thomas <ma...@apache.org> wrote:
> On 11/04/2024 21:28, Baron Fujimoto wrote> I was thinking it would be
> something that would be left on in a live> system. We can set these
> parameters, so it would be useful to know if we
> > were hitting the set limits.
>
> For the connection limit:
>
> How timely do you need the information to be? It is much easier to add a
> check when the connection is closed since a) it returns the current
> count and b) we are already checking the return value for mis-counting.
>
> For the thread limit:
>
> We have a couple of options here. (We use a custom queue which
> effectively changes the way threads in the pool re used.) The simplest
> is probably to add log in TaskQueue.offer() if all threads are currently
> being used.
>
> > I'm not sure I fully grasp how this additional logging presents a
> > significant incremental DOS risk. I mean, if an attacker is flooding you
> > with enough traffic or connections where this becomes an issue, aren't
> you
> > already logging various aspects of the attempts anyway (e.g., access
> logs,
> > etc)? At that point aren't your logs already being filled anyway?
>
> It is an area we need to be careful with. Log messages that include
> exceptions are of greatest concern since they are relatively expensive
> to generate. If an invalid HTTP message triggers a log message with an
> exception (it does, but only when configured for debug logging) then
> there is a degree of DoS risk there.
>
> Access logs are a special case and one that we have taken care to try
> and optimise. Still, if you want to make a performance test look good
> turning off access logging is one of the first things to do (and of
> course results in totally meaningless results that don't reflect real
> world usage).
>
> I think the concern Chris was trying to express (although I'm sure he'll
> correct me if I am wrong) is that when these limits are reached there is
> a risk of a lot of log messages being generated and that can cause
> issues both from a performance issue and from just filling the logs with
> the same message over and over again.
>
> I think what we would need is some form of hysteresis. Log a message
> when the limit is reached, log another message when usage falls below
> 80%? of the limit. That way, administrators see the peaks in usage but
> aren't inundated with log messages.
>
> My only concern is that at least some this code is going to run for
> every single request. It needs to be efficient. We should measure the
> impact of adding this code before we decide on whether to include it.
>
> Mark
>
>
> >
> > On Thu, Apr 11, 2024 at 1:44 AM Christopher Schultz <
> > chris@christopherschultz.net> wrote:
> >
> >> Baron,
> >>
> >> On 4/9/24 16:33, Baron Fujimoto wrote:
> >>> I'm investigating occasional 503 errors for our CAS service running in
> a
> >>> Tomcat 10.1.x container. The 503s appear to correlate with some traffic
> >>> spikes at the same time.
> >>>
> >>> The connector is configured as follows:
> >>>
> >>> <Connector protocol="org.apache.coyote.http11.Http11NioProtocol"
> >>> port="8443"
> >>> maxThreads="2500"
> >>> maxConnections="50000"
> >>> maxPostSize="100000"
> >>> maxParameterCount="1000"
> >>> scheme="https" secure="true"
> >>> SSLEnabled="true"
> >>> >
> >>>
> >>> Can Tomcat log info such as when the maxThreads or maxConnections
> limits
> >>> are reached? I'm basically trying to see if there is a good way to
> >>> more definitively determine what may have caused the 503s and what may
> be
> >>> feasible to mitigate them.
> >>
> >> Are you thinking of a debugging feature or something to be left-on for a
> >> live production system?
> >>
> >> Such logging might be considered a DOS vector for a live system: you can
> >> fill the logs by asking lots of trivial requests.
> >>
> >> -chris
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >> For additional commands, e-mail: users-help@tomcat.apache.org
> >>
> >>
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
--
Baron Fujimoto <ba...@hawaii.edu> ::: UH Information Technology Services
minutas cantorum, minutas balorum, minutas carboratum descendus pantorum
Re: Tomcat log warnings for connection parameter limits?
Posted by Mark Thomas <ma...@apache.org>.
On 11/04/2024 21:28, Baron Fujimoto wrote> I was thinking it would be
something that would be left on in a live> system. We can set these
parameters, so it would be useful to know if we
> were hitting the set limits.
For the connection limit:
How timely do you need the information to be? It is much easier to add a
check when the connection is closed since a) it returns the current
count and b) we are already checking the return value for mis-counting.
For the thread limit:
We have a couple of options here. (We use a custom queue which
effectively changes the way threads in the pool re used.) The simplest
is probably to add log in TaskQueue.offer() if all threads are currently
being used.
> I'm not sure I fully grasp how this additional logging presents a
> significant incremental DOS risk. I mean, if an attacker is flooding you
> with enough traffic or connections where this becomes an issue, aren't you
> already logging various aspects of the attempts anyway (e.g., access logs,
> etc)? At that point aren't your logs already being filled anyway?
It is an area we need to be careful with. Log messages that include
exceptions are of greatest concern since they are relatively expensive
to generate. If an invalid HTTP message triggers a log message with an
exception (it does, but only when configured for debug logging) then
there is a degree of DoS risk there.
Access logs are a special case and one that we have taken care to try
and optimise. Still, if you want to make a performance test look good
turning off access logging is one of the first things to do (and of
course results in totally meaningless results that don't reflect real
world usage).
I think the concern Chris was trying to express (although I'm sure he'll
correct me if I am wrong) is that when these limits are reached there is
a risk of a lot of log messages being generated and that can cause
issues both from a performance issue and from just filling the logs with
the same message over and over again.
I think what we would need is some form of hysteresis. Log a message
when the limit is reached, log another message when usage falls below
80%? of the limit. That way, administrators see the peaks in usage but
aren't inundated with log messages.
My only concern is that at least some this code is going to run for
every single request. It needs to be efficient. We should measure the
impact of adding this code before we decide on whether to include it.
Mark
>
> On Thu, Apr 11, 2024 at 1:44 AM Christopher Schultz <
> chris@christopherschultz.net> wrote:
>
>> Baron,
>>
>> On 4/9/24 16:33, Baron Fujimoto wrote:
>>> I'm investigating occasional 503 errors for our CAS service running in a
>>> Tomcat 10.1.x container. The 503s appear to correlate with some traffic
>>> spikes at the same time.
>>>
>>> The connector is configured as follows:
>>>
>>> <Connector protocol="org.apache.coyote.http11.Http11NioProtocol"
>>> port="8443"
>>> maxThreads="2500"
>>> maxConnections="50000"
>>> maxPostSize="100000"
>>> maxParameterCount="1000"
>>> scheme="https" secure="true"
>>> SSLEnabled="true"
>>> >
>>>
>>> Can Tomcat log info such as when the maxThreads or maxConnections limits
>>> are reached? I'm basically trying to see if there is a good way to
>>> more definitively determine what may have caused the 503s and what may be
>>> feasible to mitigate them.
>>
>> Are you thinking of a debugging feature or something to be left-on for a
>> live production system?
>>
>> Such logging might be considered a DOS vector for a live system: you can
>> fill the logs by asking lots of trivial requests.
>>
>> -chris
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat log warnings for connection parameter limits?
Posted by Baron Fujimoto <ba...@hawaii.edu>.
I was thinking it would be something that would be left on in a live
system. We can set these parameters, so it would be useful to know if we
were hitting the set limits.
I'm not sure I fully grasp how this additional logging presents a
significant incremental DOS risk. I mean, if an attacker is flooding you
with enough traffic or connections where this becomes an issue, aren't you
already logging various aspects of the attempts anyway (e.g., access logs,
etc)? At that point aren't your logs already being filled anyway?
On Thu, Apr 11, 2024 at 1:44 AM Christopher Schultz <
chris@christopherschultz.net> wrote:
> Baron,
>
> On 4/9/24 16:33, Baron Fujimoto wrote:
> > I'm investigating occasional 503 errors for our CAS service running in a
> > Tomcat 10.1.x container. The 503s appear to correlate with some traffic
> > spikes at the same time.
> >
> > The connector is configured as follows:
> >
> > <Connector protocol="org.apache.coyote.http11.Http11NioProtocol"
> > port="8443"
> > maxThreads="2500"
> > maxConnections="50000"
> > maxPostSize="100000"
> > maxParameterCount="1000"
> > scheme="https" secure="true"
> > SSLEnabled="true"
> > >
> >
> > Can Tomcat log info such as when the maxThreads or maxConnections limits
> > are reached? I'm basically trying to see if there is a good way to
> > more definitively determine what may have caused the 503s and what may be
> > feasible to mitigate them.
>
> Are you thinking of a debugging feature or something to be left-on for a
> live production system?
>
> Such logging might be considered a DOS vector for a live system: you can
> fill the logs by asking lots of trivial requests.
>
> -chris
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
--
Baron Fujimoto <ba...@hawaii.edu> ::: UH Information Technology Services
minutas cantorum, minutas balorum, minutas carboratum descendus pantorum
Re: Tomcat log warnings for connection parameter limits?
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Baron,
On 4/9/24 16:33, Baron Fujimoto wrote:
> I'm investigating occasional 503 errors for our CAS service running in a
> Tomcat 10.1.x container. The 503s appear to correlate with some traffic
> spikes at the same time.
>
> The connector is configured as follows:
>
> <Connector protocol="org.apache.coyote.http11.Http11NioProtocol"
> port="8443"
> maxThreads="2500"
> maxConnections="50000"
> maxPostSize="100000"
> maxParameterCount="1000"
> scheme="https" secure="true"
> SSLEnabled="true"
> >
>
> Can Tomcat log info such as when the maxThreads or maxConnections limits
> are reached? I'm basically trying to see if there is a good way to
> more definitively determine what may have caused the 503s and what may be
> feasible to mitigate them.
Are you thinking of a debugging feature or something to be left-on for a
live production system?
Such logging might be considered a DOS vector for a live system: you can
fill the logs by asking lots of trivial requests.
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org