You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by Rob Emery <ap...@codeweavers.net> on 2021/04/29 12:25:47 UTC
[users@httpd] Intermittently the TLS handshake results in plaintext 400 Bad Request
response
Hello,
We have a problem where intermittently users are getting a plaintext
400 Bad Request response in the middle of the TLS handshake (always
the 6th packet in the TCP stream); it happens about 1 in 40K requests
at current. As far as we can tell, there is no difference between a
successful connection from a client and these failures (we have
confirmed that all the options in the handshake are identical apart
from the session/random level components).
We have traffic captures of the problem occurring (see attached
screenshot with the end-user’s IP redacted) and it happens fairly
frequently for us in our production environment (about 10 per hour or
similar).
We've examined the user agents etc that those requests usually come
from and see a mixture of different types of clients (PHP + Curl,
Firefox, Chrome, Safari, Java, Python) and Operating Systems (iOS,
Linux, Windows 10, Android) etc, so there doesn't appear to be any
commonality between the clients.
There’s a firewall performing NAT between the client and the httpd
instance and the error is definitely coming from httpd as the traffic
captures were taken on the physical interface that httpd is listening
on. It is happening on multiple (> 5) servers that share nothing so we
don’t think it could be a physical issue.
They’re apache2 2.4.25-3+deb9u7 on Debian 9. This is 2 minor patches
behind the latest however we have reviewed the patches and there
doesn’t seem to be any way those changes could affect this behaviour.
We have also read through the changelog for Apache2, the only possible
related change that we can see is in 2.4.38:
*) mod_ssl: Fix the error code returned in an error path of
'ssl_io_filter_handshake()'. This messes-up error handling performed
in 'ssl_io_filter_error()' [Yann Ylavic]
However that change only resolves a situation where httpd returns a
502 when it should return a 400, so we don’t think that’s related. We
spent a good portion of yesterday reviewing the mod_ssl code, however
we weren't able to identify a situation where this would happen.
We have logging at “warn” everywhere, however these requests don’t
show in either the access or error log when we check for them.
We are currently trying to get this reproduced in a lab environment so
we can increase the log levels etc however any guidance as to where to
focus our efforts would be much appreciated.
Thanks
Rob
Other relevant information we can think of:
apache2 2.4.25-3+deb9u7
Linux 4.9.0-9-amd64 #1 SMP Debian 4.9.168-1 (2019-04-12) x86_64 GNU/Linux
openssl 1.1.0j-1~deb9u1
We’re using: mod_ssl and mpm_worker with:
StartServers 2
MinSpareThreads 25
MaxSpareThreads 75
ThreadLimit 64
ThreadsPerChild 25
MaxRequestWorkers 150
MaxConnectionsPerChild 0
Other modules we have enabled are:
access_compat.load
alias.load
auth_basic.load
authn_core.load
authn_file.load
authz_core.load
authz_host.load
authz_user.load
deflate.load
dir.load
env.load
filter.load
headers.load
lbmethod_byrequests.load
mime.load
negotiation.load
proxy_balancer.load
proxy_html.load
proxy_http.load
proxy.load
rewrite.load
setenvif.load
slotmem_shm.load
socache_shmcb.load
status.load
Xml2enc.load
Example of the site (edited for brevity):
<VirtualHost 10.1.17.209:443>
ServerName example.com
ErrorLog ${APACHE_LOG_DIR}/example.com.error.log
LogLevel warn
CustomLog ${APACHE_LOG_DIR}/example.com.access.log
vhost_combined_cw_tls env=!dontlog
#Enable mod-deflate for everything except images
SetOutputFilter DEFLATE
SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png)$ no-gzip
RewriteEngine On
RewriteCond %{REQUEST_METHOD}
!^(GET|HEAD|OPTIONS|POST|PUT|DELETE|PATCH)$ [NC]
RewriteRule .* "-" [F]
RequestHeader unset X-Forwarded-For
RequestHeader unset X-Forwarded-Host
RequestHeader unset X-Forwarded-Proto
RequestHeader unset Max-Forwards
SSLEngine On
SSLProtocol all -SSLv3 -TLSv1 -TLSv1.1
SSLProxyEngine on
SSLProxyVerify none
SSLHonorCipherOrder on
# MSIE 2-6
BrowserMatch "MSIE [2-6]" nokeepalive ssl-unclean-shutdown
downgrade-1.0 force-response-1.0
# MSIE 7 and newer should be able to use keepalive
BrowserMatch "MSIE [17-9]" ssl-unclean-shutdown
RequestHeader set X-ClientSSLProtocol "%{SSL_PROTOCOL}s"
RequestHeader set X-ClientSSLCipher "%{SSL_CIPHER}s"
RequestHeader set X_FORWARDED_PROTO "https"
RequestHeader set X-Forwarded-Proto "https"
SSLCertificateFile/etc/apache2/ssl/example.com/_.example.com.crt
SSLCertificateKeyFile/etc/apache2/ssl/example.com/_.example.com.key
SSLCertificateChainFile /etc/apache2/ssl/example.com/chain.crt
RewriteRule ^(.*)$http://upstreamserver/$1 [P,QSA]
</VirtualHost>
--
Re: [users@httpd] Intermittently the TLS handshake results in
plaintext 400 Bad Request response
Posted by Rob Emery <ap...@codeweavers.net>.
Hello,
I just wanted to provide a resolution to this problem for future
searches etc. So the behaviour we were seeing is totally normal for
httpd. If you do a HTTPS request to httpd on a socket that it is
listening on, but doesn't have a VirtualHost configured, it will return
a plaintext HTTP 400.
We ended up going round the houses on this issue, until we noticed that
the problem was that our load balancer (relayd in this case) would
'randomly' increment the IP address that it was directing the request to
by 1 (i.e. instead of handshaking with .144 it would handshake with
.145) which on some addresses, we didn't have a virtual host configured.
So, the long and short of it is: not a bug with httpd; as was predicted
by everyone, it's a problem outside of its control and misbehaviour
upstream.
The main thing I think that's useful information for other people
experiencing something similar is that the logs for this are only
available at debug, so we've changed our production httpd configuration
to be:
LogLevel warn core:debug
That way we get in the error log:
AH00566: request failed: malformed request line
This provided us with the visibility of the problem that then let us
track back exactly what was going on.
Thanks for the input everyone!
Rob
On 29/04/2021 14:36, Rob Emery wrote:
>
>> Assuming your site is public facing, give this evaluation a try and
>> see if anything interesting is mentioned.
>> https://www.ssllabs.com/ssltest/
>
> It is indeed public and I've just run that. Nothing strikes me as
> weird or unusual about it at all unfortunately:
>
> https://www.ssllabs.com/ssltest/analyze.html?d=services.codeweavers.net
>
> Thanks,
> Rob
>
--
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Intermittently the TLS handshake results in
plaintext 400 Bad Request response
Posted by Rob Emery <ap...@codeweavers.net>.
> Assuming your site is public facing, give this evaluation a try and
> see if anything interesting is mentioned.
> https://www.ssllabs.com/ssltest/
It is indeed public and I've just run that. Nothing strikes me as weird
or unusual about it at all unfortunately:
https://www.ssllabs.com/ssltest/analyze.html?d=services.codeweavers.net
Thanks,
Rob
--
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Intermittently the TLS handshake results in
plaintext 400 Bad Request response
Posted by Jim Albert <ji...@netrition.com>.
On 4/29/2021 9:06 AM, Rob Emery wrote:
> Hiya Jim
>
> Thanks for the reply.
>
>> If not already included, you could include %{SSL_PROTOCOL}x
> %{SSL_CIPHER}x in your request log and see if there is any commonality
> in requests assuming the communication is open long enough for the
> logging to occur or if the client's desired protocol and cipher might
> get listed.
>
> Yeah we actually already have that enabled in our access logs and we
> can see that the clients in question are using TLS1.2 when successful
> (i.e. on the next connection). However these connections that result
> in the plaintext response actually aren't logged in either the access
> or error log at all.
>
> However we can see from the packet captures that they are a TLS 1.2
> handshake and everything "looks fine" there when compared to a
> successful handshake.
>
Assuming your site is public facing, give this evaluation a try and see
if anything interesting is mentioned.
https://www.ssllabs.com/ssltest/
Jim
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Intermittently the TLS handshake results in
plaintext 400 Bad Request response
Posted by Rob Emery <ap...@codeweavers.net>.
> I actually thought your suggestion of a reverse proxy or load balancer
> presenting a problem had merit. I still think that's a good question
> so we know are we dealing with the error coming from a back end
> apache or something in front of it.
Indeed, I think (excluding the packet itself getting mangled upstream),
taking the capture on the httpd instance itself eliminates the error
coming from something in-front, as we wouldn't see it in that capture if
it was.
> Also, does your packet trace allow you the ability to see if a
> firewall is dropping packets and perhaps a misguided IPS blocking some
> traffic?
I do wish I could blame an IPS, however in this situation it's not
triggering anything and there's the same number of packets before the
server-response (5) between a successful handshake and an unsuccessful
one which has sort of lead me to exonerate it.
Thanks
Rob
--
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Intermittently the TLS handshake results in
plaintext 400 Bad Request response
Posted by Rob Emery <ap...@codeweavers.net>.
On 29/04/2021 17:51, Nick Folino wrote:
> Can you look at the Hello in the capture to see if it is OK?
>
Certainly, it looks perfect to me (I've attached a screenie of the
Wireshark decode in-case I've missed something).
Thanks,
Rob
--
Re: [users@httpd] Intermittently the TLS handshake results in
plaintext 400 Bad Request response
Posted by Nick Folino <ni...@folino.us>.
Can you look at the Hello in the capture to see if it is OK?
On Thu, Apr 29, 2021 at 11:43 AM Jim Albert <ji...@netrition.com> wrote:
> On 4/29/2021 11:11 AM, Liwei wrote:
> > On Thu, 29 Apr 2021 at 22:36, Liwei <xi...@gmail.com> wrote:
> >> On Thu, 29 Apr 2021 at 21:06, Rob Emery <ap...@codeweavers.net>
> wrote:
> >> ------ 8< Snip 8< ---------
> >>> Yeah we actually already have that enabled in our access logs and we
> can
> >>> see that the clients in question are using TLS1.2 when successful (i.e.
> >>> on the next connection). However these connections that result in the
> >>> plaintext response actually aren't logged in either the access or error
> >>> log at all.
> >> This seems to indicate something wrong in front of Apache. Likely some
> >> other machine trying to respond in http mode. A misconfigured load
> >> balancer perhaps?
> >>
> >> If you have some fancy multicast/round-robin DNS configuration, maybe
> >> a misconfigured endpoint? Seems like the domain is on Route 53, so
> >> that might be a possibility.
> >>
> >> Not as likely since you did report that a system integrator
> >> experienced the same problem, but do you have any local DNS overrides
> >> that might be interfering with things?
> >>
> >> Lih
> > Doh! Ignore my previous email, the capture and your first email
> > clearly stated that the response is coming from httpd, so what I said
> > doesn't make sense.
> >
> > Looking at common code paths that lead to a 400 error, I'd imagine two
> > possible scenarios:
> > 1. Something is mangling the initial TLS hello, can you verify that
> > the raw packet makes sense?
> > 2. Worker exhaustion, given that you seem to be proxying requests,
> > does this happen during particularly busy moments?
> >
> > There are too many variables to contend with here, especially with the
> > upstream firewall potentially mangling things and the proxy and
> > downstream server potentially killing a request early.
> >
> > You're trying to get this replicated in a lab environment, so I'd say
> > that would be the best way forward in eliminating other variables. I'd
> > probably try to replay the exact contents of the failing TLS hello you
> > captured to quickly eliminate the possibility of upstream mangling.
> > I'd also monitor or capture packets to the downstream server to see if
> > there's any correlation.
> >
> >
>
> I actually thought your suggestion of a reverse proxy or load balancer
> presenting a problem had merit. I still think that's a good question so
> we know are we dealing with the error coming from a back end apache or
> something in front of it.
>
> Also, does your packet trace allow you the ability to see if a firewall
> is dropping packets and perhaps a misguided IPS blocking some traffic?
>
> Jim
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
Re: [users@httpd] Intermittently the TLS handshake results in
plaintext 400 Bad Request response
Posted by Jim Albert <ji...@netrition.com>.
On 4/29/2021 11:11 AM, Liwei wrote:
> On Thu, 29 Apr 2021 at 22:36, Liwei <xi...@gmail.com> wrote:
>> On Thu, 29 Apr 2021 at 21:06, Rob Emery <ap...@codeweavers.net> wrote:
>> ------ 8< Snip 8< ---------
>>> Yeah we actually already have that enabled in our access logs and we can
>>> see that the clients in question are using TLS1.2 when successful (i.e.
>>> on the next connection). However these connections that result in the
>>> plaintext response actually aren't logged in either the access or error
>>> log at all.
>> This seems to indicate something wrong in front of Apache. Likely some
>> other machine trying to respond in http mode. A misconfigured load
>> balancer perhaps?
>>
>> If you have some fancy multicast/round-robin DNS configuration, maybe
>> a misconfigured endpoint? Seems like the domain is on Route 53, so
>> that might be a possibility.
>>
>> Not as likely since you did report that a system integrator
>> experienced the same problem, but do you have any local DNS overrides
>> that might be interfering with things?
>>
>> Lih
> Doh! Ignore my previous email, the capture and your first email
> clearly stated that the response is coming from httpd, so what I said
> doesn't make sense.
>
> Looking at common code paths that lead to a 400 error, I'd imagine two
> possible scenarios:
> 1. Something is mangling the initial TLS hello, can you verify that
> the raw packet makes sense?
> 2. Worker exhaustion, given that you seem to be proxying requests,
> does this happen during particularly busy moments?
>
> There are too many variables to contend with here, especially with the
> upstream firewall potentially mangling things and the proxy and
> downstream server potentially killing a request early.
>
> You're trying to get this replicated in a lab environment, so I'd say
> that would be the best way forward in eliminating other variables. I'd
> probably try to replay the exact contents of the failing TLS hello you
> captured to quickly eliminate the possibility of upstream mangling.
> I'd also monitor or capture packets to the downstream server to see if
> there's any correlation.
>
>
I actually thought your suggestion of a reverse proxy or load balancer
presenting a problem had merit. I still think that's a good question so
we know are we dealing with the error coming from a back end apache or
something in front of it.
Also, does your packet trace allow you the ability to see if a firewall
is dropping packets and perhaps a misguided IPS blocking some traffic?
Jim
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Intermittently the TLS handshake results in
plaintext 400 Bad Request response
Posted by Rob Emery <ap...@codeweavers.net>.
Apologies for the delay, been a crazy few days. Thanks for the reply.
> Looking at common code paths that lead to a 400 error, I'd imagine two
> possible scenarios:
> 1. Something is mangling the initial TLS hello, can you verify that
> the raw packet makes sense?
> 2. Worker exhaustion, given that you seem to be proxying requests,
> does this happen during particularly busy moments?
1 - That does seem like the most likely, however when we look at the pcaps
the ClientHello looks indistinguishable from a successful handshake with
the same client (excluding the random elements, ephemeral port etc)
2 - Interesting, I would expect worker exhaustion to log something along
the lines of the MaxClients in the error log. Is there a form of exhaustion
that wouldn't log?
> There are too many variables to contend with here, especially with the
> upstream firewall potentially mangling things and the proxy and
> downstream server potentially killing a request early.
Agreed, however as its within the TLS handshake I don't see how the proxied
server could have a bearing on it as the request physically hasn't provided
enough information to the httpd instance to be able to proxy it yet etc;
I'd
also expect that to log in the access log?
> We think we've got it reproduced in a more controlled environment however
we need to confirm with traffic captures etc.
Thanks,
Rob
--
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Intermittently the TLS handshake results in
plaintext 400 Bad Request response
Posted by Liwei <xi...@gmail.com>.
On Thu, 29 Apr 2021 at 22:36, Liwei <xi...@gmail.com> wrote:
>
> On Thu, 29 Apr 2021 at 21:06, Rob Emery <ap...@codeweavers.net> wrote:
> ------ 8< Snip 8< ---------
> >
> > Yeah we actually already have that enabled in our access logs and we can
> > see that the clients in question are using TLS1.2 when successful (i.e.
> > on the next connection). However these connections that result in the
> > plaintext response actually aren't logged in either the access or error
> > log at all.
>
> This seems to indicate something wrong in front of Apache. Likely some
> other machine trying to respond in http mode. A misconfigured load
> balancer perhaps?
>
> If you have some fancy multicast/round-robin DNS configuration, maybe
> a misconfigured endpoint? Seems like the domain is on Route 53, so
> that might be a possibility.
>
> Not as likely since you did report that a system integrator
> experienced the same problem, but do you have any local DNS overrides
> that might be interfering with things?
>
> Lih
Doh! Ignore my previous email, the capture and your first email
clearly stated that the response is coming from httpd, so what I said
doesn't make sense.
Looking at common code paths that lead to a 400 error, I'd imagine two
possible scenarios:
1. Something is mangling the initial TLS hello, can you verify that
the raw packet makes sense?
2. Worker exhaustion, given that you seem to be proxying requests,
does this happen during particularly busy moments?
There are too many variables to contend with here, especially with the
upstream firewall potentially mangling things and the proxy and
downstream server potentially killing a request early.
You're trying to get this replicated in a lab environment, so I'd say
that would be the best way forward in eliminating other variables. I'd
probably try to replay the exact contents of the failing TLS hello you
captured to quickly eliminate the possibility of upstream mangling.
I'd also monitor or capture packets to the downstream server to see if
there's any correlation.
Lih
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Intermittently the TLS handshake results in
plaintext 400 Bad Request response
Posted by Liwei <xi...@gmail.com>.
On Thu, 29 Apr 2021 at 21:06, Rob Emery <ap...@codeweavers.net> wrote:
------ 8< Snip 8< ---------
>
> Yeah we actually already have that enabled in our access logs and we can
> see that the clients in question are using TLS1.2 when successful (i.e.
> on the next connection). However these connections that result in the
> plaintext response actually aren't logged in either the access or error
> log at all.
This seems to indicate something wrong in front of Apache. Likely some
other machine trying to respond in http mode. A misconfigured load
balancer perhaps?
If you have some fancy multicast/round-robin DNS configuration, maybe
a misconfigured endpoint? Seems like the domain is on Route 53, so
that might be a possibility.
Not as likely since you did report that a system integrator
experienced the same problem, but do you have any local DNS overrides
that might be interfering with things?
Lih
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Intermittently the TLS handshake results in
plaintext 400 Bad Request response
Posted by Rob Emery <ap...@codeweavers.net>.
Hiya Jim
Thanks for the reply.
> If not already included, you could include %{SSL_PROTOCOL}x
%{SSL_CIPHER}x in your request log and see if there is any commonality
in requests assuming the communication is open long enough for the
logging to occur or if the client's desired protocol and cipher might
get listed.
Yeah we actually already have that enabled in our access logs and we can
see that the clients in question are using TLS1.2 when successful (i.e.
on the next connection). However these connections that result in the
plaintext response actually aren't logged in either the access or error
log at all.
However we can see from the packet captures that they are a TLS 1.2
handshake and everything "looks fine" there when compared to a
successful handshake.
Thanks
Rob
--
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Intermittently the TLS handshake results in
plaintext 400 Bad Request response
Posted by Jim Albert <ji...@netrition.com>.
On 4/29/2021 8:25 AM, Rob Emery wrote:
> Hello,
>
> We have a problem where intermittently users are getting a plaintext
> 400 Bad Request response in the middle of the TLS handshake (always
> the 6th packet in the TCP stream); it happens about 1 in 40K requests
> at current. As far as we can tell, there is no difference between a
> successful connection from a client and these failures (we have
> confirmed that all the options in the handshake are identical apart
> from the session/random level components).
>
> We have traffic captures of the problem occurring (see attached
> screenshot with the end-user’s IP redacted) and it happens fairly
> frequently for us in our production environment (about 10 per hour or
> similar).
>
> We've examined the user agents etc that those requests usually come
> from and see a mixture of different types of clients (PHP + Curl,
> Firefox, Chrome, Safari, Java, Python) and Operating Systems (iOS,
> Linux, Windows 10, Android) etc, so there doesn't appear to be any
> commonality between the clients.
>
> There’s a firewall performing NAT between the client and the httpd
> instance and the error is definitely coming from httpd as the traffic
> captures were taken on the physical interface that httpd is listening
> on. It is happening on multiple (> 5) servers that share nothing so we
> don’t think it could be a physical issue.
>
> They’re apache2 2.4.25-3+deb9u7 on Debian 9. This is 2 minor patches
> behind the latest however we have reviewed the patches and there
> doesn’t seem to be any way those changes could affect this behaviour.
> We have also read through the changelog for Apache2, the only possible
> related change that we can see is in 2.4.38:
>
> *) mod_ssl: Fix the error code returned in an error path of
> 'ssl_io_filter_handshake()'. This messes-up error handling performed
> in 'ssl_io_filter_error()' [Yann Ylavic]
>
> However that change only resolves a situation where httpd returns a
> 502 when it should return a 400, so we don’t think that’s related. We
> spent a good portion of yesterday reviewing the mod_ssl code, however
> we weren't able to identify a situation where this would happen.
>
> We have logging at “warn” everywhere, however these requests don’t
> show in either the access or error log when we check for them.
>
> We are currently trying to get this reproduced in a lab environment so
> we can increase the log levels etc however any guidance as to where to
> focus our efforts would be much appreciated.
>
> Thanks
> Rob
>
> Other relevant information we can think of:
>
> apache2 2.4.25-3+deb9u7
> Linux 4.9.0-9-amd64 #1 SMP Debian 4.9.168-1 (2019-04-12) x86_64 GNU/Linux
> openssl 1.1.0j-1~deb9u1
>
> We’re using: mod_ssl and mpm_worker with:
>
> StartServers 2
> MinSpareThreads 25
> MaxSpareThreads 75
> ThreadLimit 64
> ThreadsPerChild 25
> MaxRequestWorkers 150
> MaxConnectionsPerChild 0
>
> Other modules we have enabled are:
>
> access_compat.load
> alias.load
> auth_basic.load
> authn_core.load
> authn_file.load
> authz_core.load
> authz_host.load
> authz_user.load
> deflate.load
> dir.load
> env.load
> filter.load
> headers.load
> lbmethod_byrequests.load
> mime.load
> negotiation.load
> proxy_balancer.load
> proxy_html.load
> proxy_http.load
> proxy.load
> rewrite.load
> setenvif.load
> slotmem_shm.load
> socache_shmcb.load
> status.load
> Xml2enc.load
>
> Example of the site (edited for brevity):
>
> <VirtualHost 10.1.17.209:443>
> ServerName example.com
>
> ErrorLog ${APACHE_LOG_DIR}/example.com.error.log
> LogLevel warn
> CustomLog ${APACHE_LOG_DIR}/example.com.access.log
> vhost_combined_cw_tls env=!dontlog
>
> #Enable mod-deflate for everything except images
> SetOutputFilter DEFLATE
> SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png)$ no-gzip
>
> RewriteEngine On
>
> RewriteCond %{REQUEST_METHOD}
> !^(GET|HEAD|OPTIONS|POST|PUT|DELETE|PATCH)$ [NC]
> RewriteRule .* "-" [F]
>
> RequestHeader unset X-Forwarded-For
> RequestHeader unset X-Forwarded-Host
> RequestHeader unset X-Forwarded-Proto
> RequestHeader unset Max-Forwards
>
> SSLEngine On
> SSLProtocol all -SSLv3 -TLSv1 -TLSv1.1
> SSLProxyEngine on
> SSLProxyVerify none
> SSLHonorCipherOrder on
>
> # MSIE 2-6
> BrowserMatch "MSIE [2-6]" nokeepalive ssl-unclean-shutdown
> downgrade-1.0 force-response-1.0
> # MSIE 7 and newer should be able to use keepalive
> BrowserMatch "MSIE [17-9]" ssl-unclean-shutdown
>
> RequestHeader set X-ClientSSLProtocol "%{SSL_PROTOCOL}s"
> RequestHeader set X-ClientSSLCipher "%{SSL_CIPHER}s"
> RequestHeader set X_FORWARDED_PROTO "https"
> RequestHeader set X-Forwarded-Proto "https"
>
> SSLCertificateFile/etc/apache2/ssl/example.com/_.example.com.crt
> SSLCertificateKeyFile/etc/apache2/ssl/example.com/_.example.com.key
> SSLCertificateChainFile /etc/apache2/ssl/example.com/chain.crt
>
> RewriteRule ^(.*)$http://upstreamserver/$1 [P,QSA]
>
> </VirtualHost>
>
>
Can you tell from the user agents if they indicate old clients?
I see you disable TLSv1.1.
You could try temporarily enabling TLSv1.1 and see if those failures
stop and then you would know it's probably old clients that can't talk
TLSv1.2. Although the percentage of failures likely doesn't fit that.
If not already included, you could include %{SSL_PROTOCOL}x
%{SSL_CIPHER}x in your request log and see if there is any commonality
in requests assuming the communication is open long enough for the
logging to occur or if the client's desired protocol and cipher might
get listed.
Jim
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Intermittently the TLS handshake results in
plaintext 400 Bad Request response
Posted by Rob Emery <ap...@codeweavers.net>.
Hiya Nick,
Thanks for the reply.
> Do the clients that fail always fail?
No, they work most of the time with the same settings (as confirmed with
a friendly integrator).
> Do they support the protocols you require?
Yes, with the exact same configuration on the clients they work on the
next connection.
> Or are they trying to use a disabled protocol?
No, they are using supported protocols and again, a retry with the same
settings works.
Thanks,
Rob
--
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Intermittently the TLS handshake results in
plaintext 400 Bad Request response
Posted by Nick Folino <ni...@folino.us>.
Do the clients that fail always fail? Do they support the protocols you
require? Or are they trying to use a disabled protocol?
Nick
On Thu, Apr 29, 2021 at 8:26 AM Rob Emery <ap...@codeweavers.net>
wrote:
> Hello,
>
> We have a problem where intermittently users are getting a plaintext
> 400 Bad Request response in the middle of the TLS handshake (always
> the 6th packet in the TCP stream); it happens about 1 in 40K requests
> at current. As far as we can tell, there is no difference between a
> successful connection from a client and these failures (we have
> confirmed that all the options in the handshake are identical apart
> from the session/random level components).
>
> We have traffic captures of the problem occurring (see attached
> screenshot with the end-user’s IP redacted) and it happens fairly
> frequently for us in our production environment (about 10 per hour or
> similar).
>
> We've examined the user agents etc that those requests usually come
> from and see a mixture of different types of clients (PHP + Curl,
> Firefox, Chrome, Safari, Java, Python) and Operating Systems (iOS,
> Linux, Windows 10, Android) etc, so there doesn't appear to be any
> commonality between the clients.
>
> There’s a firewall performing NAT between the client and the httpd
> instance and the error is definitely coming from httpd as the traffic
> captures were taken on the physical interface that httpd is listening
> on. It is happening on multiple (> 5) servers that share nothing so we
> don’t think it could be a physical issue.
>
> They’re apache2 2.4.25-3+deb9u7 on Debian 9. This is 2 minor patches
> behind the latest however we have reviewed the patches and there
> doesn’t seem to be any way those changes could affect this behaviour.
> We have also read through the changelog for Apache2, the only possible
> related change that we can see is in 2.4.38:
>
> *) mod_ssl: Fix the error code returned in an error path of
> 'ssl_io_filter_handshake()'. This messes-up error handling performed
> in 'ssl_io_filter_error()' [Yann Ylavic]
>
> However that change only resolves a situation where httpd returns a
> 502 when it should return a 400, so we don’t think that’s related. We
> spent a good portion of yesterday reviewing the mod_ssl code, however
> we weren't able to identify a situation where this would happen.
>
> We have logging at “warn” everywhere, however these requests don’t
> show in either the access or error log when we check for them.
>
> We are currently trying to get this reproduced in a lab environment so
> we can increase the log levels etc however any guidance as to where to
> focus our efforts would be much appreciated.
>
> Thanks
> Rob
>
> Other relevant information we can think of:
>
> apache2 2.4.25-3+deb9u7
> Linux 4.9.0-9-amd64 #1 SMP Debian 4.9.168-1 (2019-04-12) x86_64 GNU/Linux
> openssl 1.1.0j-1~deb9u1
>
> We’re using: mod_ssl and mpm_worker with:
>
> StartServers 2
> MinSpareThreads 25
> MaxSpareThreads 75
> ThreadLimit 64
> ThreadsPerChild 25
> MaxRequestWorkers 150
> MaxConnectionsPerChild 0
>
> Other modules we have enabled are:
>
> access_compat.load
> alias.load
> auth_basic.load
> authn_core.load
> authn_file.load
> authz_core.load
> authz_host.load
> authz_user.load
> deflate.load
> dir.load
> env.load
> filter.load
> headers.load
> lbmethod_byrequests.load
> mime.load
> negotiation.load
> proxy_balancer.load
> proxy_html.load
> proxy_http.load
> proxy.load
> rewrite.load
> setenvif.load
> slotmem_shm.load
> socache_shmcb.load
> status.load
> Xml2enc.load
>
> Example of the site (edited for brevity):
>
> <VirtualHost 10.1.17.209:443>
> ServerName example.com
>
> ErrorLog ${APACHE_LOG_DIR}/example.com.error.log
> LogLevel warn
> CustomLog ${APACHE_LOG_DIR}/example.com.access.log
> vhost_combined_cw_tls env=!dontlog
>
> #Enable mod-deflate for everything except images
> SetOutputFilter DEFLATE
> SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png)$ no-gzip
>
> RewriteEngine On
>
> RewriteCond %{REQUEST_METHOD}
> !^(GET|HEAD|OPTIONS|POST|PUT|DELETE|PATCH)$ [NC]
> RewriteRule .* "-" [F]
>
> RequestHeader unset X-Forwarded-For
> RequestHeader unset X-Forwarded-Host
> RequestHeader unset X-Forwarded-Proto
> RequestHeader unset Max-Forwards
>
> SSLEngine On
> SSLProtocol all -SSLv3 -TLSv1 -TLSv1.1
> SSLProxyEngine on
> SSLProxyVerify none
> SSLHonorCipherOrder on
>
> # MSIE 2-6
> BrowserMatch "MSIE [2-6]" nokeepalive ssl-unclean-shutdown
> downgrade-1.0 force-response-1.0
> # MSIE 7 and newer should be able to use keepalive
> BrowserMatch "MSIE [17-9]" ssl-unclean-shutdown
>
> RequestHeader set X-ClientSSLProtocol "%{SSL_PROTOCOL}s"
> RequestHeader set X-ClientSSLCipher "%{SSL_CIPHER}s"
> RequestHeader set X_FORWARDED_PROTO "https"
> RequestHeader set X-Forwarded-Proto "https"
>
> SSLCertificateFile/etc/apache2/ssl/example.com/_.example.com.crt
> SSLCertificateKeyFile/etc/apache2/ssl/example.com/_.example.com.key
> SSLCertificateChainFile /etc/apache2/ssl/example.com/chain.crt
>
> RewriteRule ^(.*)$http://upstreamserver/$1 [P,QSA]
>
> </VirtualHost>
>
>
> --
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org