You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Reindl Harald <h....@thelounge.net> on 2015/10/01 13:37:54 UTC
Re: SSLUseStapling: ssl handshake fails until httpd restart
Am 30.09.2015 um 08:42 schrieb Kaspar Brand:
> On 29.09.2015 18:24, Reindl Harald wrote:
>> i just restarted the servers and disabled stapling since all our
>> servcies where unreachable (before i write the second mail 5 different
>> hosts with several sites where affected)
>>
>> in fact the error caching does more harm than benefits - IHMO a better
>> "could not reach OCSP server or received a error from it" caching would
>> be just temporary disable stapling for 10 minutes instead lead in
>> connections fail even from clients which have disabled OCSP completly
>>
>>>> firefox refused to open our adminpanel with the error below until i
>>>> restarted httpd
>
> The default for SSLStaplingReturnResponderErrors is relatively odd.
> Not sure why it has always defaulted to "on" (r829619), but setting it
> to off should save you further troubles with Firefox clients.
not really, i had the error message just now again in FF, the difference
was that now a "try again" loaded the page but with
"SSLStaplingReturnResponderErrors" i would expect it invisible to
clients in general - GoDaddy seems to have massive problems with their
responders the last days and the defaults with stapling enabled make
them to a perfect DOS target
[Thu Oct 01 13:33:01.179365 2015] [ssl:error] [pid 19312] [client
10.0.0.99:37860] AH01980: bad response from OCSP server: (none)
[Thu Oct 01 13:33:01.179393 2015] [ssl:error] [pid 19312] AH01941:
stapling_renew_response: responder error
SSLStaplingCache shmcb:/var/cache/mod_ssl/ocsp_cache(1048576)
SSLStaplingStandardCacheTimeout 86400
SSLStaplingErrorCacheTimeout 300
SSLStaplingReturnResponderErrors Off
Re: SSLUseStapling: ssl handshake fails until httpd restart
Posted by Tim Bannister <is...@c8h10n4o2.org.uk>.
On 4 Oct 2015, at 11:38, Kaspar Brand wrote:
>
> As far as the mod_ssl side is related, it seems to me that for the "SSLStaplingReturnResponderErrors off" case, we should make sure that we only staple responses with status "good" (i.e. OCSP_RESPONSE_STATUS_SUCCESSFUL and V_OCSP_CERTSTATUS_GOOD for the cert).
If the OCSP response is successful but the status isn't V_OCSP_CERTSTATUS_GOOD, I'd want httpd to at least log a warning (as well as not stapling the OCSP information). Maybe even add a Warning: header for any client that's interested.
I can attempt a patch for this if other people think it'd be useful.
--
Tim Bannister – isoma@c8h10n4o2.org.uk
Re: SSLUseStapling: ssl handshake fails until httpd restart
Posted by Kaspar Brand <ht...@velox.ch>.
On 03.10.2015 12:07, Reindl Harald wrote:
> Am 03.10.2015 um 11:16 schrieb Kaspar Brand:
>> What do you have security.OCSP.require set to? If it's "true" (a setting
>> no longer configurable through the UI, BTW, see
>> https://bugzilla.mozilla.org/show_bug.cgi?id=1034360), then Firefox
>> shows a fairly peculiar behavior: even when OCSP checking is disabled
>> (by setting security.OCSP.enabled to "0", through the "Query OCSP
>> responder servers to confirm the current validity of certificates"
>> preference in the UI under Advanced -> Certificates), it still includes
>> a status_request extension in the TLS handshake, and will subsequently
>> fail when it receives a stapled tryLater OCSP response, as long as
>> security.OCSP.require=true
>
> security.OCSP.require=false is the default
Oh, looks like Firefox does sort of a "hard fail" when receiving a
stapled tryLater response - not what I would expect when
security.OCSP.require=false (and also security.OCSP.enabled=0), but
apparently, the security.ssl.enable_ocsp_stapling pref has precedence
over those.
As far as the mod_ssl side is related, it seems to me that for the
"SSLStaplingReturnResponderErrors off" case, we should make sure
that we only staple responses with status "good" (i.e.
OCSP_RESPONSE_STATUS_SUCCESSFUL and V_OCSP_CERTSTATUS_GOOD for the cert).
> but it's not only my client with random failed connections
This is really due to a very weird effect with ocsp.godaddy.com, as
further examination shows. GoDaddy is making use of Akamai's Global
Traffic Management service for "load balancing", i.e., depending on
where the client is located, you're getting one of the following
addresses back when querying for ocsp.godaddy.com.akadns.net (the CNAME
for ocsp.godaddy.com):
72.167.18.239
50.63.243.230
72.167.239.239
188.121.36.239
(the list might be incomplete, but it illustrates Akamai's GTM mechanism)
For Central Europe, you get back 188.121.36.239 (from GoDaddy's
Amsterdam data center), and from what I can tell, this site has a pretty
quirky behavior when handling new connections on port 80: the first
connection is instantaneously torn down with a TCP reset... but if you
immediately try again, the OCSP request is properly handled (for your
*.thelounge.net cert e.g., try [1]). Then, after some idle timeout, the
same effect kicks in again. I haven't been able to determine how long
the timeout is exactly, but judging from your log extracts, it seems to
be less than 30 minutes). Strangely enough, the other (US-based) sites
don't seem to exhibit this effect.
Kaspar
[1] http://188.121.36.239/MEgwRjBEMEIwQDAJBgUrDgMCGgUABBS2CA1fbGt26xPkOKX4ZguoUjM0TgQUQMK9J47MNIMwojPX+2yz8LQsgM4CByr+BC4coss=
Re: SSLUseStapling: ssl handshake fails until httpd restart
Posted by Reindl Harald <h....@thelounge.net>.
Am 03.10.2015 um 11:16 schrieb Kaspar Brand:
> On 01.10.2015 16:32, Reindl Harald wrote:
>> Am 01.10.2015 um 16:29 schrieb Plüm, Rüdiger, Vodafone Group:
>>> The question is: What happens on Firefox side. Of course it still tries to get to the OCSP server, but it should not cause an error on Firefox side if this does not work.
>>
>> no, it does not because "security.OCSP.enabled = 0" and i saw at least
>> two requests to different servers failing with my Firefox with the
>> responder error from the webserver
>
> What do you have security.OCSP.require set to? If it's "true" (a setting
> no longer configurable through the UI, BTW, see
> https://bugzilla.mozilla.org/show_bug.cgi?id=1034360), then Firefox
> shows a fairly peculiar behavior: even when OCSP checking is disabled
> (by setting security.OCSP.enabled to "0", through the "Query OCSP
> responder servers to confirm the current validity of certificates"
> preference in the UI under Advanced -> Certificates), it still includes
> a status_request extension in the TLS handshake, and will subsequently
> fail when it receives a stapled tryLater OCSP response, as long as
> security.OCSP.require=true
security.OCSP.require=false is the default
but it's not only my client with random failed connections
[Sat Oct 03 00:15:01.478741 2015] [ssl:error] [pid 27458]
(104)Connection reset by peer: [client 81.223.20.5:59844] AH01977:
failed reading line from OCSP server
[Sat Oct 03 00:45:01.618792 2015] [ssl:error] [pid 4965] (104)Connection
reset by peer: [client 81.223.20.5:33566] AH01977: failed reading line
from OCSP server
[Sat Oct 03 01:15:01.589643 2015] [ssl:error] [pid 5599] (104)Connection
reset by peer: [client 81.223.20.5:36218] AH01977: failed reading line
from OCSP server
[Sat Oct 03 01:45:01.816132 2015] [ssl:error] [pid 4965] (104)Connection
reset by peer: [client 81.223.20.5:38762] AH01977: failed reading line
from OCSP server
[Sat Oct 03 02:15:01.187187 2015] [ssl:error] [pid 14361]
(104)Connection reset by peer: [client 81.223.20.5:41043] AH01977:
failed reading line from OCSP server
[Sat Oct 03 02:45:01.292205 2015] [ssl:error] [pid 14366]
(104)Connection reset by peer: [client 81.223.20.5:42999] AH01977:
failed reading line from OCSP server
[Sat Oct 03 03:15:01.353077 2015] [ssl:error] [pid 14364]
(104)Connection reset by peer: [client 81.223.20.5:45180] AH01977:
failed reading line from OCSP server
[Sat Oct 03 03:45:01.428090 2015] [ssl:error] [pid 14364]
(104)Connection reset by peer: [client 81.223.20.5:46857] AH01977:
failed reading line from OCSP server
[Sat Oct 03 04:15:02.019261 2015] [ssl:error] [pid 26399]
(104)Connection reset by peer: [client 81.223.20.5:49007] AH01977:
failed reading line from OCSP server
Re: SSLUseStapling: ssl handshake fails until httpd restart
Posted by Kaspar Brand <ht...@velox.ch>.
On 01.10.2015 16:32, Reindl Harald wrote:
> Am 01.10.2015 um 16:29 schrieb Plüm, Rüdiger, Vodafone Group:
>> The question is: What happens on Firefox side. Of course it still tries to get to the OCSP server, but it should not cause an error on Firefox side if this does not work.
>
> no, it does not because "security.OCSP.enabled = 0" and i saw at least
> two requests to different servers failing with my Firefox with the
> responder error from the webserver
What do you have security.OCSP.require set to? If it's "true" (a setting
no longer configurable through the UI, BTW, see
https://bugzilla.mozilla.org/show_bug.cgi?id=1034360), then Firefox
shows a fairly peculiar behavior: even when OCSP checking is disabled
(by setting security.OCSP.enabled to "0", through the "Query OCSP
responder servers to confirm the current validity of certificates"
preference in the UI under Advanced -> Certificates), it still includes
a status_request extension in the TLS handshake, and will subsequently
fail when it receives a stapled tryLater OCSP response, as long as
security.OCSP.require=true.
Kaspar
Re: SSLUseStapling: ssl handshake fails until httpd restart
Posted by Reindl Harald <h....@thelounge.net>.
Am 01.10.2015 um 16:29 schrieb Plüm, Rüdiger, Vodafone Group:
>
>
>> -----Ursprüngliche Nachricht-----
>> Von: Reindl Harald [mailto:h.reindl@thelounge.net]
>> Gesendet: Donnerstag, 1. Oktober 2015 15:18
>> An: dev@httpd.apache.org
>> Betreff: Re: SSLUseStapling: ssl handshake fails until httpd restart
>>
>>
>>
>> Am 01.10.2015 um 15:08 schrieb Reindl Harald:
>>> Am 01.10.2015 um 14:53 schrieb Plüm, Rüdiger, Vodafone Group:
>>>>> not really, i had the error message just now again in FF, the
>> difference
>>>>> was that now a "try again" loaded the page but with
>>>>> "SSLStaplingReturnResponderErrors" i would expect it invisible to
>>>>> clients in general - GoDaddy seems to have massive problems with
>> their
>>>>> responders the last days and the defaults with stapling enabled make
>>>>> them to a perfect DOS target
>>>>>
>>>>> [Thu Oct 01 13:33:01.179365 2015] [ssl:error] [pid 19312] [client
>>>>> 10.0.0.99:37860] AH01980: bad response from OCSP server: (none)
>>>>> [Thu Oct 01 13:33:01.179393 2015] [ssl:error] [pid 19312] AH01941:
>>>>> stapling_renew_response: responder error
>>>>>
>>>>> SSLStaplingCache shmcb:/var/cache/mod_ssl/ocsp_cache(1048576)
>>>>> SSLStaplingStandardCacheTimeout 86400
>>>>> SSLStaplingErrorCacheTimeout 300
>>>>> SSLStaplingReturnResponderErrors Off
>>>>
>>>> What happens if you set
>>>>
>>>> SSLStaplingFakeTryLater off
>>>>
>>>> in addition?
>>>
>>> i added that now and will have a look over the serverlogs, it's not
>>> happening all the time but very often and so if the logs are clear
>>> within 24 hours the problem is likely solved
>>
>> looks not that good - "Connection reset by peer" indicates a failed
>> client request, the other lines could be just internal
>>
>> [Thu Oct 01 15:15:01.495986 2015] [ssl:error] [pid 17468]
>> (104)Connection reset by peer: [client 81.223.20.5:55156] AH01977:
>> failed reading line from OCSP server
>> [Thu Oct 01 15:15:01.496037 2015] [ssl:error] [pid 17468] [client
>> 81.223.20.5:55156] AH01980: bad response from OCSP server: (none)
>> [Thu Oct 01 15:15:01.496057 2015] [ssl:error] [pid 17468] AH01941:
>> stapling_renew_response: responder error
>
> The question is: What happens on Firefox side. Of course it still tries to get to the OCSP server, but it should not cause an error on Firefox side if this does not work.
no, it does not because "security.OCSP.enabled = 0" and i saw at least
two requests to different servers failing with my Firefox with the
responder error from the webserver
AW: SSLUseStapling: ssl handshake fails until httpd restart
Posted by Plüm,
Rüdiger,
Vodafone Group <ru...@vodafone.com>.
> -----Ursprüngliche Nachricht-----
> Von: Reindl Harald [mailto:h.reindl@thelounge.net]
> Gesendet: Donnerstag, 1. Oktober 2015 15:18
> An: dev@httpd.apache.org
> Betreff: Re: SSLUseStapling: ssl handshake fails until httpd restart
>
>
>
> Am 01.10.2015 um 15:08 schrieb Reindl Harald:
> > Am 01.10.2015 um 14:53 schrieb Plüm, Rüdiger, Vodafone Group:
> >>> not really, i had the error message just now again in FF, the
> difference
> >>> was that now a "try again" loaded the page but with
> >>> "SSLStaplingReturnResponderErrors" i would expect it invisible to
> >>> clients in general - GoDaddy seems to have massive problems with
> their
> >>> responders the last days and the defaults with stapling enabled make
> >>> them to a perfect DOS target
> >>>
> >>> [Thu Oct 01 13:33:01.179365 2015] [ssl:error] [pid 19312] [client
> >>> 10.0.0.99:37860] AH01980: bad response from OCSP server: (none)
> >>> [Thu Oct 01 13:33:01.179393 2015] [ssl:error] [pid 19312] AH01941:
> >>> stapling_renew_response: responder error
> >>>
> >>> SSLStaplingCache shmcb:/var/cache/mod_ssl/ocsp_cache(1048576)
> >>> SSLStaplingStandardCacheTimeout 86400
> >>> SSLStaplingErrorCacheTimeout 300
> >>> SSLStaplingReturnResponderErrors Off
> >>
> >> What happens if you set
> >>
> >> SSLStaplingFakeTryLater off
> >>
> >> in addition?
> >
> > i added that now and will have a look over the serverlogs, it's not
> > happening all the time but very often and so if the logs are clear
> > within 24 hours the problem is likely solved
>
> looks not that good - "Connection reset by peer" indicates a failed
> client request, the other lines could be just internal
>
> [Thu Oct 01 15:15:01.495986 2015] [ssl:error] [pid 17468]
> (104)Connection reset by peer: [client 81.223.20.5:55156] AH01977:
> failed reading line from OCSP server
> [Thu Oct 01 15:15:01.496037 2015] [ssl:error] [pid 17468] [client
> 81.223.20.5:55156] AH01980: bad response from OCSP server: (none)
> [Thu Oct 01 15:15:01.496057 2015] [ssl:error] [pid 17468] AH01941:
> stapling_renew_response: responder error
The question is: What happens on Firefox side. Of course it still tries to get to the OCSP server, but it should not cause an error on Firefox side if this does not work.
Regards
Rüdiger
Re: SSLUseStapling: ssl handshake fails until httpd restart
Posted by Reindl Harald <h....@thelounge.net>.
Am 01.10.2015 um 15:08 schrieb Reindl Harald:
> Am 01.10.2015 um 14:53 schrieb Plüm, Rüdiger, Vodafone Group:
>>> not really, i had the error message just now again in FF, the difference
>>> was that now a "try again" loaded the page but with
>>> "SSLStaplingReturnResponderErrors" i would expect it invisible to
>>> clients in general - GoDaddy seems to have massive problems with their
>>> responders the last days and the defaults with stapling enabled make
>>> them to a perfect DOS target
>>>
>>> [Thu Oct 01 13:33:01.179365 2015] [ssl:error] [pid 19312] [client
>>> 10.0.0.99:37860] AH01980: bad response from OCSP server: (none)
>>> [Thu Oct 01 13:33:01.179393 2015] [ssl:error] [pid 19312] AH01941:
>>> stapling_renew_response: responder error
>>>
>>> SSLStaplingCache shmcb:/var/cache/mod_ssl/ocsp_cache(1048576)
>>> SSLStaplingStandardCacheTimeout 86400
>>> SSLStaplingErrorCacheTimeout 300
>>> SSLStaplingReturnResponderErrors Off
>>
>> What happens if you set
>>
>> SSLStaplingFakeTryLater off
>>
>> in addition?
>
> i added that now and will have a look over the serverlogs, it's not
> happening all the time but very often and so if the logs are clear
> within 24 hours the problem is likely solved
looks not that good - "Connection reset by peer" indicates a failed
client request, the other lines could be just internal
[Thu Oct 01 15:15:01.495986 2015] [ssl:error] [pid 17468]
(104)Connection reset by peer: [client 81.223.20.5:55156] AH01977:
failed reading line from OCSP server
[Thu Oct 01 15:15:01.496037 2015] [ssl:error] [pid 17468] [client
81.223.20.5:55156] AH01980: bad response from OCSP server: (none)
[Thu Oct 01 15:15:01.496057 2015] [ssl:error] [pid 17468] AH01941:
stapling_renew_response: responder error
Re: SSLUseStapling: ssl handshake fails until httpd restart
Posted by Reindl Harald <h....@thelounge.net>.
Am 01.10.2015 um 14:53 schrieb Plüm, Rüdiger, Vodafone Group:
>> -----Ursprüngliche Nachricht-----
>> Von: Reindl Harald [mailto:h.reindl@thelounge.net]
>>> The default for SSLStaplingReturnResponderErrors is relatively odd.
>>> Not sure why it has always defaulted to "on" (r829619), but setting it
>>> to off should save you further troubles with Firefox clients.
>>
>> not really, i had the error message just now again in FF, the difference
>> was that now a "try again" loaded the page but with
>> "SSLStaplingReturnResponderErrors" i would expect it invisible to
>> clients in general - GoDaddy seems to have massive problems with their
>> responders the last days and the defaults with stapling enabled make
>> them to a perfect DOS target
>>
>> [Thu Oct 01 13:33:01.179365 2015] [ssl:error] [pid 19312] [client
>> 10.0.0.99:37860] AH01980: bad response from OCSP server: (none)
>> [Thu Oct 01 13:33:01.179393 2015] [ssl:error] [pid 19312] AH01941:
>> stapling_renew_response: responder error
>>
>> SSLStaplingCache shmcb:/var/cache/mod_ssl/ocsp_cache(1048576)
>> SSLStaplingStandardCacheTimeout 86400
>> SSLStaplingErrorCacheTimeout 300
>> SSLStaplingReturnResponderErrors Off
>
> What happens if you set
>
> SSLStaplingFakeTryLater off
>
> in addition?
i added that now and will have a look over the serverlogs, it's not
happening all the time but very often and so if the logs are clear
within 24 hours the problem is likely solved
thanks!
AW: SSLUseStapling: ssl handshake fails until httpd restart
Posted by Plüm,
Rüdiger,
Vodafone Group <ru...@vodafone.com>.
> -----Ursprüngliche Nachricht-----
> Von: Reindl Harald [mailto:h.reindl@thelounge.net]
> Gesendet: Donnerstag, 1. Oktober 2015 13:38
> An: dev@httpd.apache.org
> Betreff: Re: SSLUseStapling: ssl handshake fails until httpd restart
>
>
>
> Am 30.09.2015 um 08:42 schrieb Kaspar Brand:
> > On 29.09.2015 18:24, Reindl Harald wrote:
> >> i just restarted the servers and disabled stapling since all our
> >> servcies where unreachable (before i write the second mail 5
> different
> >> hosts with several sites where affected)
> >>
> >> in fact the error caching does more harm than benefits - IHMO a
> better
> >> "could not reach OCSP server or received a error from it" caching
> would
> >> be just temporary disable stapling for 10 minutes instead lead in
> >> connections fail even from clients which have disabled OCSP completly
> >>
> >>>> firefox refused to open our adminpanel with the error below until i
> >>>> restarted httpd
> >
> > The default for SSLStaplingReturnResponderErrors is relatively odd.
> > Not sure why it has always defaulted to "on" (r829619), but setting it
> > to off should save you further troubles with Firefox clients.
>
> not really, i had the error message just now again in FF, the difference
> was that now a "try again" loaded the page but with
> "SSLStaplingReturnResponderErrors" i would expect it invisible to
> clients in general - GoDaddy seems to have massive problems with their
> responders the last days and the defaults with stapling enabled make
> them to a perfect DOS target
>
> [Thu Oct 01 13:33:01.179365 2015] [ssl:error] [pid 19312] [client
> 10.0.0.99:37860] AH01980: bad response from OCSP server: (none)
> [Thu Oct 01 13:33:01.179393 2015] [ssl:error] [pid 19312] AH01941:
> stapling_renew_response: responder error
>
> SSLStaplingCache shmcb:/var/cache/mod_ssl/ocsp_cache(1048576)
> SSLStaplingStandardCacheTimeout 86400
> SSLStaplingErrorCacheTimeout 300
> SSLStaplingReturnResponderErrors Off
What happens if you set
SSLStaplingFakeTryLater off
in addition?
Regards
Rüdiger