You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Folker Schamel <sc...@spinor.com> on 2018/07/31 13:56:11 UTC
Bug report: Regression SVN Client, SSL, Serf 1.3.9-3, SSLVerifyClient
require
Hello everyone,
After upgrading, Subversion SSL connections with "SSLVerifyClient
require" seem to be broken.
Broken: SVN Client 1.9.5, Serf 1.3.9-3, Server "SSLVerifyClient require"
Works: SVN Client 1.9.5, Serf 1.3.9-3, Server "SSLVerifyClient off"
Works: SVN Client 1.9.5, Serf 1.3.8-1, Server "SSLVerifyClient require"
For the broken setup, the client reports:
svn: E120171: Error running context: An error occurred during SSL
communication
And the server Apache log reports:
ssl_engine_io.c(1308): (70014)End of file found: [client xxxxx:xxxxx]
AH02007: SSL handshake interrupted by system [Hint: Stop button pressed
in browser?!]
Using the latest TortoiseSVN client reports the same problem, presumably
the same cause.
Additional details below.
Can I help with additional information?
Btw, thanks a lot to all Subversion developers and contributors for the
awesome work!!!
Cheers,
Folker
***** Client-side recipt (latest Debian stretch):
root@xxxxx:/# apt-get install libserf-1-1=1.3.8-1
.....
root@xxxxx:/# svn --version
svn, version 1.9.5 (r1770682)
compiled Jun 30 2018, 13:44:22 on x86_64-pc-linux-gnu
Copyright (C) 2016 The Apache Software Foundation.
This software consists of contributions made by many people;
see the NOTICE file for more information.
Subversion is open source software, see http://subversion.apache.org/
The following repository access (RA) modules are available:
* ra_svn : Module for accessing a repository using the svn network protocol.
- with Cyrus SASL authentication
- handles 'svn' scheme
* ra_local : Module for accessing a repository on local disk.
- handles 'file' scheme
* ra_serf : Module for accessing a repository via WebDAV protocol using
serf.
- using serf 1.3.8 (compiled with 1.3.9)
- handles 'http' scheme
- handles 'https' scheme
The following authentication credential caches are available:
* Plaintext cache in /root/.subversion
* Gnome Keyring
* GPG-Agent
* KWallet (KDE)
root@xxxxx:/# svn update
Updating '.':
At revision 828.
root@xxxxx:/# apt-get install libserf-1-1=1.3.9-3
.....
root@xxxxx:/# svn --version
svn, version 1.9.5 (r1770682)
compiled Jun 30 2018, 13:44:22 on x86_64-pc-linux-gnu
Copyright (C) 2016 The Apache Software Foundation.
This software consists of contributions made by many people;
see the NOTICE file for more information.
Subversion is open source software, see http://subversion.apache.org/
The following repository access (RA) modules are available:
* ra_svn : Module for accessing a repository using the svn network protocol.
- with Cyrus SASL authentication
- handles 'svn' scheme
* ra_local : Module for accessing a repository on local disk.
- handles 'file' scheme
* ra_serf : Module for accessing a repository via WebDAV protocol using
serf.
- using serf 1.3.9 (compiled with 1.3.9)
- handles 'http' scheme
- handles 'https' scheme
The following authentication credential caches are available:
* Plaintext cache in /root/.subversion
* Gnome Keyring
* GPG-Agent
* KWallet (KDE)
root@xxxxx:/# svn update
Updating '.':
svn: E170013: Unable to connect to a repository at URL
'https://xxxxx/xxxxx/xxxxx'
svn: E120171: Error running context: An error occurred during SSL
communication
root@xxxxx:/#
***** Client-side recipt continuation after SSLVerifyClient require -> off
root@xxxxx:/# svn update
Updating '.':
At revision 828.
root@xxxxx:/#
***** Server-side ssl-error.log:
...
[Tue Jul 31 15:30:43.885515 2018] [ssl:info] [pid xxxxx:tid xxxxx]
[client xxxxx:xxxxx] AH01964: Connection to child 68 established (server
localhost:443)
[Tue Jul 31 15:30:43.885795 2018] [ssl:trace2] [pid xxxxx:tid xxxxx]
ssl_engine_rand.c(126): Seeding PRNG with 656 bytes of entropy
[Tue Jul 31 15:30:43.885983 2018] [ssl:trace3] [pid xxxxx:tid xxxxx]
ssl_engine_kernel.c(1989): [client xxxxx:xxxxx] OpenSSL: Handshake: start
[Tue Jul 31 15:30:43.886064 2018] [ssl:trace3] [pid xxxxx:tid xxxxx]
ssl_engine_kernel.c(1998): [client xxxxx:xxxxx] OpenSSL: Loop:
before/accept initialization
[Tue Jul 31 15:30:43.886114 2018] [ssl:trace4] [pid xxxxx:tid xxxxx]
ssl_engine_io.c(2135): [client xxxxx:xxxxx] OpenSSL: read 5/5 bytes from
BIO#7fcef0001580 [mem: 7fcef0006dc3] (BIO dump follows)
[Tue Jul 31 15:30:43.886134 2018] [ssl:trace4] [pid xxxxx:tid xxxxx]
ssl_engine_io.c(2135): [client xxxxx:xxxxx] OpenSSL: read 191/191 bytes
from BIO#7fcef0001580 [mem: 7fcef0006dc8] (BIO dump follows)
[Tue Jul 31 15:30:43.886183 2018] [ssl:debug] [pid xxxxx:tid xxxxx]
ssl_engine_kernel.c(2122): [client xxxxx:xxxxx] AH02044: No matching SSL
virtual host for servername xxxxx found (using default/first virtual host)
[Tue Jul 31 15:30:43.886258 2018] [ssl:trace3] [pid xxxxx:tid xxxxx]
ssl_engine_kernel.c(1998): [client xxxxx:xxxxx] OpenSSL: Loop: unknown state
[Tue Jul 31 15:30:43.886294 2018] [ssl:trace3] [pid xxxxx:tid xxxxx]
ssl_engine_kernel.c(1998): [client xxxxx:xxxxx] OpenSSL: Loop: unknown state
[Tue Jul 31 15:30:43.886419 2018] [ssl:trace3] [pid xxxxx:tid xxxxx]
ssl_engine_kernel.c(1998): [client xxxxx:xxxxx] OpenSSL: Loop: unknown state
[Tue Jul 31 15:30:43.908313 2018] [ssl:trace3] [pid xxxxx:tid xxxxx]
ssl_engine_kernel.c(1998): [client xxxxx:xxxxx] OpenSSL: Loop: unknown state
[Tue Jul 31 15:30:43.908537 2018] [ssl:trace3] [pid xxxxx:tid xxxxx]
ssl_engine_kernel.c(1998): [client xxxxx:xxxxx] OpenSSL: Loop: unknown state
[Tue Jul 31 15:30:43.908769 2018] [ssl:trace4] [pid xxxxx:tid xxxxx]
ssl_engine_io.c(2135): [client xxxxx:xxxxx] OpenSSL: write 2173/2173
bytes to BIO#7fcef0001500 [mem: 7fcef0014030] (BIO dump follows)
[Tue Jul 31 15:30:43.909055 2018] [core:trace6] [pid xxxxx:tid xxxxx]
core_filters.c(525): [client xxxxx:xxxxx] core_output_filter: flushing
because of FLUSH bucket
[Tue Jul 31 15:30:43.909342 2018] [ssl:trace3] [pid xxxxx:tid xxxxx]
ssl_engine_kernel.c(1998): [client xxxxx:xxxxx] OpenSSL: Loop: unknown state
[Tue Jul 31 15:30:43.918838 2018] [ssl:trace4] [pid xxxxx:tid xxxxx]
ssl_engine_io.c(2144): [client xxxxx:xxxxx] OpenSSL: I/O error, 5 bytes
expected to read on BIO#7fcef0001580 [mem: 7fcef00150e3]
[Tue Jul 31 15:30:43.919121 2018] [ssl:trace3] [pid xxxxx:tid xxxxx]
ssl_engine_kernel.c(2027): [client xxxxx:xxxxx] OpenSSL: Exit: error in
unknown state
[Tue Jul 31 15:30:43.919427 2018] [ssl:debug] [pid xxxxx:tid xxxxx]
ssl_engine_io.c(1308): (70014)End of file found: [client xxxxx:xxxxx]
AH02007: SSL handshake interrupted by system [Hint: Stop button pressed
in browser?!]
[Tue Jul 31 15:30:43.919615 2018] [ssl:info] [pid xxxxx:tid xxxxx]
[client xxxxx:xxxxx] AH01998: Connection closed to child 68 with
abortive shutdown (server localhost:443)
...
***** Server-side Apache configuration (latest Debian stretch):
<VirtualHost>
.....
SSLEngine On
SSLCertificateFile /etc/apache2/ssl/apache.pem
SSLVerifyClient require
SSLVerifyDepth 1
SSLCACertificateFile /xxxxx/xxxxx.pem
</VirtualHost>
<Location /svn>
SetOutputFilter DEFLATE
SetInputFilter DEFLATE
Header append Vary User-Agent env=!dont-vary
</Location>
<Location /svn/xxxxx>
DAV svn
SVNPath /xxxxx
SVNAutoversioning On
SVNPathAuthz On
AuthType Basic
AuthName "xxxxx"
AuthUserFile /xxxxx/xxxxx
AuthzSVNAccessFile /xxxxx/xxxxx
Require valid-user
.....
</Location>
*****
Re: [PATCH] Was: Bug report: Regression SVN Client, SSL, Serf 1.3.9-3,
SSLVerifyClient require
Posted by Julian Foad <ju...@apache.org>.
Folker Schamel wrote:
> Daniel Shahaf wrote:
>> Folker Schamel wrote:
>>> * site/staging/faq.html:
>>> Add entry for "An error occurred during SSL communication" error.
>>> * site/staging/docs/release-notes/1.10.html:
>>> Add entry for an OpenSSL upgrade causing "An error occurred during
>>> SSL communication" error.
>>
>> Could we link to the OpenSSL upstream documentation --- their list of
>> incompatible changes, or upgrade guide, or something along these lines?
>
> In our case, I'm not aware of any such documentation which would have helped us.
[...]
I committed your patches in http://svn.apache.org/r1837321 (straight to the 'publish' tree, as I don't think there is anything there that could cause harm). We can adjust it further there if necessary.
Thanks for contributing back the results from what you learnt the hard way.
- Julian
Re: [PATCH] Was: Bug report: Regression SVN Client, SSL, Serf
1.3.9-3, SSLVerifyClient require
Posted by Folker Schamel <sc...@spinor.com>.
On 2018-08-03 12:46, Daniel Shahaf wrote:
> Folker Schamel wrote on Thu, 02 Aug 2018 15:34 +0200:
>> On 2018-08-01 19:19, Daniel Shahaf wrote:
>>> Folker Schamel wrote on Wed, 01 Aug 2018 17:51 +0200:
>>>> Hi Julian,
>>>>
>>>> Draft which may save you some time:
>>>>
>>>> First patch against trunk:
>>>> [[[
>>>> * site/staging/faq.html:
>>>> Add entry for "An error occurred during SSL communication" error.
>>>> ]]]
>>>>
>>>> Second patch against trunk:
>>>> [[[
>>>> * site/staging/docs/release-notes/1.10.html:
>>>> Add entry for an OpenSSL upgrade causing "An error occurred during
>>>> SSL communication" error.
>>>> ]]]
>>> Thanks for the patch, Folker.
>>>
>>> Could we link to the OpenSSL upstream documentation --- their list of
>>> incompatible changes, or upgrade guide, or something along these lines?
>> In our case, I'm not aware of any such documentation which would have
>> helped us.
> Then why not ask the OpenSSL project to create such documentation?
> Again, this problem and its solution are entirely in the domain of
> OpenSSL. The Serf changes discussed are simply about having Serf
> marshal to its caller all the information it receives from its
> dependency.
In our case, I don't see which additional OpenSSL documentation would
have helped us.
People never read any documentation, until they get an error.
Then they google for that error.
So getting a useful google hit would be great.
This specific error "An error occurred during SSL communication" is from
Subversion.
So for all practical purposes it must be handled in the Subversion
documentation, not in the OpenSSL documentation.
>
>>> I agree that there's room for some Subversion-specific documentation
>>> here, but I think it should focus on Subversion-specific parts (such as
>>> the ssl-authority-files knob, or failure modes that for whatever reason
>>> are more common with Subversion than with other SSL-using software)
>>> and refer to OpenSSL's docs for the lion's share of the content.
>> For us the key issue was:
>> What can I do if Subversion fails with "An error occurred during SSL
>> communication"?
>> The corresponding patches based on Philip's answer address this question.
>> We lost a lot of time by wrongly suspecting problems between serf,
>> mod_dav_svn and mod_ssl, for example
>> https://code.google.com/archive/p/serf/issues/135.
>>
> I appreciate that the system architecture involves several components
> (svn, serf, apr, openssl) and that it might not be obvious which
> component was responsible for which error message. That is knowledge
> I would love to see better transferred by our own documentation (the
> website or the svnbook).
>
> Documenting new releases of _our_ upstreams also makes sense, since we can
> read the upstream changelogs and summarize to our users what the impact
> of the upgrade on their Subversion deployments would be.
>
> What I disagree with is starting to document specific OpenSSL failure
> mode in our own documentation, when the failure modes are not Subversion-
> specific in any way. I hope you will spare me listing all the reasons why Project X
> shouldn't be documenting failure modes in Project Y [happy to do so but I think
> we are all professionals here and know these reasons].
I tried to write the patches in such way that they talk mainly about the
Subversion specific aspects of the generic OpenSLL case, and mention the
specific OpenSSL failure mode only as one particular sample.
> I will also do a commit review of your patch, in case it stays, but
> again, I think it should have been committed to OpenSSL's docs rather
> than ours.
>
> Hope you don't feel this as obstructions. I am merely trying to improve
> both our own docs and OpenSSL's, but part of improving our docs is to
> know what they should incorporate by value and what by reference.
Very valid discussion, just different views ;-)
Cheers,
Folker
> Thanks for writing the docs patch!
>
> Daniel
>
>> Cheers,
>> Folker
Re: [PATCH] Was: Bug report: Regression SVN Client, SSL, Serf 1.3.9-3,
SSLVerifyClient require
Posted by Daniel Shahaf <d....@daniel.shahaf.name>.
Folker Schamel wrote on Thu, 02 Aug 2018 15:34 +0200:
> On 2018-08-01 19:19, Daniel Shahaf wrote:
> > Folker Schamel wrote on Wed, 01 Aug 2018 17:51 +0200:
> >> Hi Julian,
> >>
> >> Draft which may save you some time:
> >>
> >> First patch against trunk:
> >> [[[
> >> * site/staging/faq.html:
> >> Add entry for "An error occurred during SSL communication" error.
> >> ]]]
> >>
> >> Second patch against trunk:
> >> [[[
> >> * site/staging/docs/release-notes/1.10.html:
> >> Add entry for an OpenSSL upgrade causing "An error occurred during
> >> SSL communication" error.
> >> ]]]
> > Thanks for the patch, Folker.
> >
> > Could we link to the OpenSSL upstream documentation --- their list of
> > incompatible changes, or upgrade guide, or something along these lines?
> In our case, I'm not aware of any such documentation which would have
> helped us.
Then why not ask the OpenSSL project to create such documentation?
Again, this problem and its solution are entirely in the domain of
OpenSSL. The Serf changes discussed are simply about having Serf
marshal to its caller all the information it receives from its
dependency.
> > I agree that there's room for some Subversion-specific documentation
> > here, but I think it should focus on Subversion-specific parts (such as
> > the ssl-authority-files knob, or failure modes that for whatever reason
> > are more common with Subversion than with other SSL-using software)
> > and refer to OpenSSL's docs for the lion's share of the content.
> For us the key issue was:
> What can I do if Subversion fails with "An error occurred during SSL
> communication"?
> The corresponding patches based on Philip's answer address this question.
> We lost a lot of time by wrongly suspecting problems between serf,
> mod_dav_svn and mod_ssl, for example
> https://code.google.com/archive/p/serf/issues/135.
>
I appreciate that the system architecture involves several components
(svn, serf, apr, openssl) and that it might not be obvious which
component was responsible for which error message. That is knowledge
I would love to see better transferred by our own documentation (the
website or the svnbook).
Documenting new releases of _our_ upstreams also makes sense, since we can
read the upstream changelogs and summarize to our users what the impact
of the upgrade on their Subversion deployments would be.
What I disagree with is starting to document specific OpenSSL failure
mode in our own documentation, when the failure modes are not Subversion-
specific in any way. I hope you will spare me listing all the reasons why Project X
shouldn't be documenting failure modes in Project Y [happy to do so but I think
we are all professionals here and know these reasons].
I will also do a commit review of your patch, in case it stays, but
again, I think it should have been committed to OpenSSL's docs rather
than ours.
Hope you don't feel this as obstructions. I am merely trying to improve
both our own docs and OpenSSL's, but part of improving our docs is to
know what they should incorporate by value and what by reference.
Thanks for writing the docs patch!
Daniel
> Cheers,
> Folker
Re: [PATCH] Was: Bug report: Regression SVN Client, SSL, Serf
1.3.9-3, SSLVerifyClient require
Posted by Folker Schamel <sc...@spinor.com>.
On 2018-08-01 19:19, Daniel Shahaf wrote:
> Folker Schamel wrote on Wed, 01 Aug 2018 17:51 +0200:
>> Hi Julian,
>>
>> Draft which may save you some time:
>>
>> First patch against trunk:
>> [[[
>> * site/staging/faq.html:
>> Add entry for "An error occurred during SSL communication" error.
>> ]]]
>>
>> Second patch against trunk:
>> [[[
>> * site/staging/docs/release-notes/1.10.html:
>> Add entry for an OpenSSL upgrade causing "An error occurred during
>> SSL communication" error.
>> ]]]
> Thanks for the patch, Folker.
>
> Could we link to the OpenSSL upstream documentation --- their list of
> incompatible changes, or upgrade guide, or something along these lines?
In our case, I'm not aware of any such documentation which would have
helped us.
> I agree that there's room for some Subversion-specific documentation
> here, but I think it should focus on Subversion-specific parts (such as
> the ssl-authority-files knob, or failure modes that for whatever reason
> are more common with Subversion than with other SSL-using software)
> and refer to OpenSSL's docs for the lion's share of the content.
For us the key issue was:
What can I do if Subversion fails with "An error occurred during SSL
communication"?
The corresponding patches based on Philip's answer address this question.
We lost a lot of time by wrongly suspecting problems between serf,
mod_dav_svn and mod_ssl, for example
https://code.google.com/archive/p/serf/issues/135.
Cheers,
Folker
Re: [PATCH] Was: Bug report: Regression SVN Client, SSL, Serf 1.3.9-3,
SSLVerifyClient require
Posted by Daniel Shahaf <d....@daniel.shahaf.name>.
Folker Schamel wrote on Wed, 01 Aug 2018 17:51 +0200:
> Hi Julian,
>
> Draft which may save you some time:
>
> First patch against trunk:
> [[[
> * site/staging/faq.html:
> Add entry for "An error occurred during SSL communication" error.
> ]]]
>
> Second patch against trunk:
> [[[
> * site/staging/docs/release-notes/1.10.html:
> Add entry for an OpenSSL upgrade causing "An error occurred during
> SSL communication" error.
> ]]]
Thanks for the patch, Folker.
Could we link to the OpenSSL upstream documentation --- their list of
incompatible changes, or upgrade guide, or something along these lines?
I agree that there's room for some Subversion-specific documentation
here, but I think it should focus on Subversion-specific parts (such as
the ssl-authority-files knob, or failure modes that for whatever reason
are more common with Subversion than with other SSL-using software)
and refer to OpenSSL's docs for the lion's share of the content.
HTH!
Daniel
[PATCH] Was: Bug report: Regression SVN Client, SSL, Serf 1.3.9-3,
SSLVerifyClient require
Posted by Folker Schamel <sc...@spinor.com>.
Hi Julian,
Draft which may save you some time:
First patch against trunk:
[[[
* site/staging/faq.html:
Add entry for "An error occurred during SSL communication" error.
]]]
Second patch against trunk:
[[[
* site/staging/docs/release-notes/1.10.html:
Add entry for an OpenSSL upgrade causing "An error occurred during SSL communication" error.
]]]
Cheers,
Folker
On 2018-08-01 15:03, Julian Foad wrote:
> Folker Schamel wrote:
>> Hi Stefan,
>>> That's the catch here. Subversion does not ship with OpenSSL by
>>> itself. From Subversion's point of view this is a 3rd-party
>>> dependency. [...] It could be something worthwhile adding to the FAQ
>>> however, though then in a more general manner like: Troubleshooting
>>> Subversion SSL connection.
>> Good point.
>> The FAQ seems to be a good place.
>>
>> Nevertheless, in such situations we are probably not the only ones
>> looking primarily into the Subversion release notes, not so much into
>> the Debian documentation or Subversion FAQ, because the problem
>> seemingly was caused - in simple terms - by the Subversion update.
>> Also note that new releases of distributions of Subversion are
>> usually strongly correlated with new Subversion releases.
>>
>> So I still suggest to also put a warning in the Subversion release
>> notes, for example:
>> "Your distribution may also upgrade OpenSSL along with the Subversion
>> upgrade, which may cause trouble, see xxxx in the FAQ."
>> At least us it would have spared a lot of time ;-)
>>
>> Even if you may insist that this logically the "wrong" place,
>> sometimes a note in such a "wrong" place can be very helpful for
>> users who are looking in that "wrong" place, ;-)
>
> I agree -- when something like this hits users in real life, we should
> add it to the release notes, either in total or as a pointer to a FAQ.
> And I want to say thank you for writing a helpful description for
> other users to diagnose the problem after being troubled by it yourself.
>
> If no-one else volunteers, I will try to do it in the next day or two
> (but I haven't much time).
>
> - Julian
>
Re: Bug report: Regression SVN Client, SSL, Serf 1.3.9-3,
SSLVerifyClient require
Posted by Julian Foad <ju...@apache.org>.
Folker Schamel wrote:
> Hi Stefan,
>> That's the catch here. Subversion does not ship with OpenSSL by
>> itself. From Subversion's point of view this is a 3rd-party
>> dependency. [...] It could be something worthwhile adding to the FAQ
>> however, though then in a more general manner like: Troubleshooting
>> Subversion SSL connection.> Good point. The FAQ seems to be a good place.
>
> Nevertheless, in such situations we are probably not the only ones
> looking primarily into the Subversion release notes, not so much into
> the Debian documentation or Subversion FAQ, because the problem
> seemingly was caused - in simple terms - by the Subversion update.
> Also note that new releases of distributions of Subversion are
> usually strongly correlated with new Subversion releases.
>
> So I still suggest to also put a warning in the Subversion release
> notes, for example: "Your distribution may also upgrade OpenSSL along
> with the Subversion upgrade, which may cause trouble, see xxxx in the
> FAQ." At least us it would have spared a lot of time ;-)
>
> Even if you may insist that this logically the "wrong" place,
> sometimes a note in such a "wrong" place can be very helpful for
> users who are looking in that "wrong" place, ;-)
I agree -- when something like this hits users in real life, we should
add it to the release notes, either in total or as a pointer to a FAQ.
And I want to say thank you for writing a helpful description for other
users to diagnose the problem after being troubled by it yourself.
If no-one else volunteers, I will try to do it in the next day or two
(but I haven't much time).
- Julian
Re: Bug report: Regression SVN Client, SSL, Serf 1.3.9-3,
SSLVerifyClient require
Posted by Folker Schamel <sc...@spinor.com>.
Hi Stefan,
> That's the catch here. Subversion does not ship with OpenSSL by
> itself. From Subversion's point of view this is a 3rd-party
> dependency. You can easily build Subversion 1.9.x/1.10.x with OpenSSL
> 1.0.x. Whether or not you run into this issue therefore is outside the
> scope of Subversion IMO. It's something the distribution of Subversion
> (in your case the Debian Subversion distribution) should document.
> Note that in principle you could very well run into the same situation
> with Subversion 1.8 or even 1.7, if you build one version with OpenSSL
> <= 1.0 and the other with OpenSSL >= 1.1 (or set certain OpenSSL
> configs which also would flag md5 digests as being too weak with older
> OpenSSL versions).
>
> It could be something worthwhile adding to the FAQ however, though
> then in a more general manner like:
> Troubleshooting Subversion SSL connection.
>
Good point.
The FAQ seems to be a good place.
Nevertheless, in such situations we are probably not the only ones
looking primarily into the Subversion release notes, not so much into
the Debian documentation or Subversion FAQ, because the problem
seemingly was caused - in simple terms - by the Subversion update.
Also note that new releases of distributions of Subversion are usually
strongly correlated with new Subversion releases.
So I still suggest to also put a warning in the Subversion release
notes, for example:
"Your distribution may also upgrade OpenSSL along with the Subversion
upgrade, which may cause trouble, see xxxx in the FAQ."
At least us it would have spared a lot of time ;-)
Even if you may insist that this logically the "wrong" place, sometimes
a note in such a "wrong" place can be very helpful for users who are
looking in that "wrong" place, ;-)
Cheers,
Folker
Re: Bug report: Regression SVN Client, SSL, Serf 1.3.9-3,
SSLVerifyClient require
Posted by Stefan Hett <st...@egosoft.com>.
On 8/1/2018 9:25 AM, Folker Schamel wrote:
> On 2018-07-31 21:09, Philip Martin wrote:
>> Daniel Shahaf<d....@daniel.shahaf.name> writes:
>>
>>> Subversion uses Serf, which uses OpenSSL, which talks to an SSL implementation
>>> on the server. The root cause of the error is known to the SSL implementation
>>> on the server (that's why you see it in the error log). It's not obvious that
>>> OpenSSL on the client side even knows what the root cause is.
>> In this case the client knows exactly what is wrong, it's the one
>> closing the connection because:
>>
>> 140258270184704:error:140AB18E:SSL routines:SSL_CTX_use_certificate:ca md too weak:../ssl/ssl_rsa.c:303:
>>
>> Could we get our client to show that error? We would need a new serf
>> API to marshal the error message back to Subversion.
> This sounds like the best solution to me.
>
> Until then, some short note in the Subversion release notes would have
> saved us a lot of time, for example:
>
> Section title: Potential need for creating new CA keys and new client
> certificates
> Section text:
> This Subversion release upgrades OpenSSL from 1.0 to 1.1, which
> doesn't allow md5 hashes for CA keys anymore.
That's the catch here. Subversion does not ship with OpenSSL by itself.
From Subversion's point of view this is a 3rd-party dependency. You can
easily build Subversion 1.9.x/1.10.x with OpenSSL 1.0.x. Whether or not
you run into this issue therefore is outside the scope of Subversion
IMO. It's something the distribution of Subversion (in your case the
Debian Subversion distribution) should document. Note that in principle
you could very well run into the same situation with Subversion 1.8 or
even 1.7, if you build one version with OpenSSL <= 1.0 and the other
with OpenSSL >= 1.1 (or set certain OpenSSL configs which also would
flag md5 digests as being too weak with older OpenSSL versions).
> When using client certificates signed by such a CA, the new Subversion
> client now fails with "An error occurred during SSL communication".
> You can analyze the underlying cause by converting the client
> certificate from p12 to pem by"openssl pkcs12 -in path/to/svn/cert.p12
> -out cert.pem" and then test the SSL connection by "openssl s_client
> -connect example.com:443 -servername example.com -cert cert.pem".
> If this test connection fails with "ca md too weak", then creating new
> CA keys using sha256 instead of md5 and corresponding new client
> certificates should solve the problem.
> See also
> https://lists.apache.org/thread.html/66b9bfa0a83693c3ccef34b29056c7e73a0d21cd4b70cd7f7519fa57@%3Cdev.subversion.apache.org%3E.
>
> Cheers,
> Folker
It could be something worthwhile adding to the FAQ however, though then
in a more general manner like:
Troubleshooting Subversion SSL connection.
--
Regards,
Stefan Hett
Re: Bug report: Regression SVN Client, SSL, Serf 1.3.9-3,
SSLVerifyClient require
Posted by Folker Schamel <sc...@spinor.com>.
On 2018-07-31 21:09, Philip Martin wrote:
> Daniel Shahaf <d....@daniel.shahaf.name> writes:
>
>> Subversion uses Serf, which uses OpenSSL, which talks to an SSL implementation
>> on the server. The root cause of the error is known to the SSL implementation
>> on the server (that's why you see it in the error log). It's not obvious that
>> OpenSSL on the client side even knows what the root cause is.
> In this case the client knows exactly what is wrong, it's the one
> closing the connection because:
>
> 140258270184704:error:140AB18E:SSL routines:SSL_CTX_use_certificate:ca md too weak:../ssl/ssl_rsa.c:303:
>
> Could we get our client to show that error? We would need a new serf
> API to marshal the error message back to Subversion.
This sounds like the best solution to me.
Until then, some short note in the Subversion release notes would have
saved us a lot of time, for example:
Section title: Potential need for creating new CA keys and new client
certificates
Section text:
This Subversion release upgrades OpenSSL from 1.0 to 1.1, which doesn't
allow md5 hashes for CA keys anymore.
When using client certificates signed by such a CA, the new Subversion
client now fails with "An error occurred during SSL communication".
You can analyze the underlying cause by converting the client
certificate from p12 to pem by"openssl pkcs12 -in path/to/svn/cert.p12
-out cert.pem" and then test the SSL connection by "openssl s_client
-connect example.com:443 -servername example.com -cert cert.pem".
If this test connection fails with "ca md too weak", then creating new
CA keys using sha256 instead of md5 and corresponding new client
certificates should solve the problem.
See also
https://lists.apache.org/thread.html/66b9bfa0a83693c3ccef34b29056c7e73a0d21cd4b70cd7f7519fa57@%3Cdev.subversion.apache.org%3E.
Cheers,
Folker
Re: Bug report: Regression SVN Client, SSL, Serf 1.3.9-3, SSLVerifyClient require
Posted by Philip Martin <ph...@codematters.co.uk>.
Daniel Shahaf <d....@daniel.shahaf.name> writes:
> Subversion uses Serf, which uses OpenSSL, which talks to an SSL implementation
> on the server. The root cause of the error is known to the SSL implementation
> on the server (that's why you see it in the error log). It's not obvious that
> OpenSSL on the client side even knows what the root cause is.
In this case the client knows exactly what is wrong, it's the one
closing the connection because:
140258270184704:error:140AB18E:SSL routines:SSL_CTX_use_certificate:ca md too weak:../ssl/ssl_rsa.c:303:
Could we get our client to show that error? We would need a new serf
API to marshal the error message back to Subversion.
--
Philip
Re: Bug report: Regression SVN Client, SSL, Serf 1.3.9-3,
SSLVerifyClient require
Posted by Daniel Shahaf <d....@daniel.shahaf.name>.
Folker Schamel wrote on Tue, Jul 31, 2018 at 17:42:10 +0200:
> On 2018-07-31 17:04, Philip Martin wrote:
> > Folker Schamel <sc...@spinor.com> writes:
> > > For the broken setup, the client reports:
> > > svn: E120171: Error running context: An error occurred during SSL communication
> > > And the server Apache log reports:
> > > ssl_engine_io.c(1308): (70014)End of file found: [client xxxxx:xxxxx]
> > > AH02007: SSL handshake interrupted by system [Hint: Stop button
> > > pressed in browser?!]
>
> Maybe a hint in the svn release notes could be useful, since the svn error
> messages are not very useful.
Subversion uses Serf, which uses OpenSSL, which talks to an SSL implementation
on the server. The root cause of the error is known to the SSL implementation
on the server (that's why you see it in the error log). It's not obvious that
OpenSSL on the client side even knows what the root cause is. It could be that
only the server-side SSL implementation knows the root cause; it could be that
the client-side SSL implementation also knows the root cause, but the ball of
relaying that information up the stack (openssl->serf->libsvn->stderr) was
dropped.
The error code in question (E120171, SERF_ERROR_SSL_COMM_FAILED) does appear to
be somewhat of a catchall, i.e., a code to which several openssl errors are
mapped; but nevertheless, I wouldn't be surprised if the openssl client-side
error message were less detailed than the server-side one.
For what it's worth, the only part of the quoted error message that Subversion
controls is the text "Error running context". The remainder, both the number
prefixed andthe error suffixed, is generated by the serf library, based on an
error returned by the openssl library.
That said, I do agree that "Error running context" isn't the best phrasing.
"Context", here, is the name of an internal API. Something like "HTTP request
failed" would be better, wouldn't it?
> Thanks a lot!
> And sorry for bothering on the dev list. I should have posted to user.
> Also thanks for your other SNI and DEFLATE tips!
Cheers,
Daniel
Re: Bug report: Regression SVN Client, SSL, Serf 1.3.9-3,
SSLVerifyClient require
Posted by Folker Schamel <sc...@spinor.com>.
Hi Philip,
this solved it!
Using "openssl s_client" as you described it reported:
error setting certificate
140258270184704:error:140AB18E:SSL routines:SSL_CTX_use_certificate:ca
md too weak:../ssl/ssl_rsa.c:303:
So we created new certificates with sha256 (default in openssl 1.1)
instead of md5.
This solved the problem.
Maybe a hint in the svn release notes could be useful, since the svn
error messages are not very useful.
Thanks a lot!
And sorry for bothering on the dev list. I should have posted to user.
Also thanks for your other SNI and DEFLATE tips!
Cheers,
Folker
On 2018-07-31 17:04, Philip Martin wrote:
> Folker Schamel <sc...@spinor.com> writes:
>
>> After upgrading, Subversion SSL connections with "SSLVerifyClient
>> require" seem to be broken.
>>
>> Broken: SVN Client 1.9.5, Serf 1.3.9-3, Server "SSLVerifyClient require"
>> Works: SVN Client 1.9.5, Serf 1.3.9-3, Server "SSLVerifyClient off"
>> Works: SVN Client 1.9.5, Serf 1.3.8-1, Server "SSLVerifyClient require"
> I can use client certificates on my Debian machine using 1.3.9-3 with
> "SSLVerifyClient require" so it does work in some configurations.
>
> I think that one of the changes between 1.3.8-1 and 1.3.9-3 on Debian is
> that openssl was upgraded from 1.0 to 1.1. That's a major upgrade and
> some sort of openssl incompatibility may be the problem.
>
> You could use the openssl binary to debug the ssl connection. First you
> need to convert Subversion's client certificate from pkcs12 to pem:
>
> openssl pkcs12 -in path/to/svn/cert.p12 -out cert.pem
>
> then you can use:
>
> openssl s_client -connect example.com:443 -servername example.com -cert cert.pem
>
> If you are using ssl-authority-files in .subversion/servers to verify
> the server cert you can get s_client to do the same with the additional
> parameter:
>
> openssl s_client ... -CAfile path/to/authority.pem
>
> The s_client output may indicate what problem is occurring.
>
>> For the broken setup, the client reports:
>> svn: E120171: Error running context: An error occurred during SSL communication
>> And the server Apache log reports:
>> ssl_engine_io.c(1308): (70014)End of file found: [client xxxxx:xxxxx]
>> AH02007: SSL handshake interrupted by system [Hint: Stop button
>> pressed in browser?!]
>>
>> Using the latest TortoiseSVN client reports the same problem,
>> presumably the same cause.
> TortoiseSVN is probably built with openssl 1.1 as well.
>
>> [Tue Jul 31 15:30:43.886183 2018] [ssl:debug] [pid xxxxx:tid xxxxx]
>> ssl_engine_kernel.c(2122): [client xxxxx:xxxxx] AH02044: No matching
>> SSL virtual host for servername xxxxx found (using default/first
>> virtual host)
> That looks like SNI isn't working but I don't know if that is relevant
> for your problem. Some sort of vhost/servername problem in the apache
> config?
>
>> <Location /svn>
>> SetOutputFilter DEFLATE
>> SetInputFilter DEFLATE
>> Header append Vary User-Agent env=!dont-vary
>> </Location>
> DEFLATE combined with mod_dav_svn had problems in the past but I think
> they are all fixed. Note that when mod_dav_svn and Subversion clients
> communicate they do so using compressed deltas, so adding DEFLATE
> doesn't result in very much additional compression. The combination has
> a more significant compression effect if users are using non-Subversion
> tools: web browsers, curl, etc. This is probably not relevant to your
> problem.
>
Re: Bug report: Regression SVN Client, SSL, Serf 1.3.9-3, SSLVerifyClient require
Posted by Philip Martin <ph...@codematters.co.uk>.
Folker Schamel <sc...@spinor.com> writes:
> After upgrading, Subversion SSL connections with "SSLVerifyClient
> require" seem to be broken.
>
> Broken: SVN Client 1.9.5, Serf 1.3.9-3, Server "SSLVerifyClient require"
> Works: SVN Client 1.9.5, Serf 1.3.9-3, Server "SSLVerifyClient off"
> Works: SVN Client 1.9.5, Serf 1.3.8-1, Server "SSLVerifyClient require"
I can use client certificates on my Debian machine using 1.3.9-3 with
"SSLVerifyClient require" so it does work in some configurations.
I think that one of the changes between 1.3.8-1 and 1.3.9-3 on Debian is
that openssl was upgraded from 1.0 to 1.1. That's a major upgrade and
some sort of openssl incompatibility may be the problem.
You could use the openssl binary to debug the ssl connection. First you
need to convert Subversion's client certificate from pkcs12 to pem:
openssl pkcs12 -in path/to/svn/cert.p12 -out cert.pem
then you can use:
openssl s_client -connect example.com:443 -servername example.com -cert cert.pem
If you are using ssl-authority-files in .subversion/servers to verify
the server cert you can get s_client to do the same with the additional
parameter:
openssl s_client ... -CAfile path/to/authority.pem
The s_client output may indicate what problem is occurring.
> For the broken setup, the client reports:
> svn: E120171: Error running context: An error occurred during SSL communication
> And the server Apache log reports:
> ssl_engine_io.c(1308): (70014)End of file found: [client xxxxx:xxxxx]
> AH02007: SSL handshake interrupted by system [Hint: Stop button
> pressed in browser?!]
>
> Using the latest TortoiseSVN client reports the same problem,
> presumably the same cause.
TortoiseSVN is probably built with openssl 1.1 as well.
> [Tue Jul 31 15:30:43.886183 2018] [ssl:debug] [pid xxxxx:tid xxxxx]
> ssl_engine_kernel.c(2122): [client xxxxx:xxxxx] AH02044: No matching
> SSL virtual host for servername xxxxx found (using default/first
> virtual host)
That looks like SNI isn't working but I don't know if that is relevant
for your problem. Some sort of vhost/servername problem in the apache
config?
> <Location /svn>
> SetOutputFilter DEFLATE
> SetInputFilter DEFLATE
> Header append Vary User-Agent env=!dont-vary
> </Location>
DEFLATE combined with mod_dav_svn had problems in the past but I think
they are all fixed. Note that when mod_dav_svn and Subversion clients
communicate they do so using compressed deltas, so adding DEFLATE
doesn't result in very much additional compression. The combination has
a more significant compression effect if users are using non-Subversion
tools: web browsers, curl, etc. This is probably not relevant to your
problem.
--
Philip