You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@openoffice.apache.org by Arrigo Marchiori <ar...@yahoo.it.INVALID> on 2022/08/29 20:14:24 UTC
Re: WebDAV module ported from serf to curl; curl using openssl and zlib
Hello Damjan, All,
I had a hard time restoring all the proper line breaks... this thread
is getting long! :-)
On Tue, Aug 23, 2022 at 03:22:07AM +0200, Damjan Jovanovic wrote:
> On Sat, May 28, 2022 at 6:54 PM Arrigo Marchiori <ar...@yahoo.it.invalid>
> wrote:
>
> > Hello Damjan, All,
> >
> > On Sat, May 28, 2022 at 03:22:53PM +0200, Damjan Jovanovic wrote:
> >
> > > On Sat, May 28, 2022 at 11:48 AM Arrigo Marchiori
> > <ar...@yahoo.it.invalid>
> > > wrote:
> > >
> > > > Hello Damjan,
> > > >
> > > > On Sat, May 28, 2022 at 11:12:45AM +0200, Damjan Jovanovic wrote:
> > > >
> > > > > On Sat, May 28, 2022 at 9:20 AM Arrigo Marchiori <ar...@yahoo.it.invalid> wrote:
> > > > >
> > > > > > Hello Damjan, all,
> > > > > >
> > > > > > On Fri, May 27, 2022 at 11:27:11PM +0200, Arrigo Marchiori wrote:
> > > > > >
> > > > > > > Hello,
> > > > > > >
> > > > > > > On Fri, May 27, 2022 at 09:46:51PM +0200, Arrigo Marchiori wrote:
> > > > > > >
> > > > > > > > Hello Damjan,
> > > > > > > >
> > > > > > > > On Sun, May 22, 2022 at 06:10:46PM +0200, Damjan Jovanovic wrote:
> > > > > > > >
> > > > > > > > > On Sun, May 22, 2022 at 2:43 PM Arrigo Marchiori <ar...@yahoo.it.invalid> wrote:
> > > > > > > > >
> > > > > > > > > > Hello Damjan, all,
> > > > > > > > > >
> > > > > > > > > > On Tue, Apr 26, 2022 at 07:56:22PM +0200, Damjan Jovanovic wrote:
> > > > > > > > > >
> > > > > > > > > > > On Mon, Nov 15, 2021 at 9:57 PM Jim Jagielski <ji...@jagunet.com> wrote:
> > > > > > > > > > >
> > > > > > > > > > > > I'm gonna look into the serf->(lib)curl option... Since we don't use any
> > > > > > > > > > > > of the fancy features of serf, I'm thinking that the easy option might be
> > > > > > > > > > > > best
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Hi
> > > > > > > > > > >
> > > > > > > > > > > I've ported our WebDAV content provider module from Serf to Curl.
> > > > > > > > > >
> > > > > > > > > > I just enhanced the error reporting a bit; I am finding a problem
> > > > > > > > > > under Linux and I do not really know how to assess it.
> > > > > > > > > >
> > > > > > > > > > The problem: if we build AOO on CentOS (that is our reference
> > > > > > > > > > platform) then Curl will look for CA certificates in
> > > > > > > > > > /etc/pki/tls/certs/ca-bundle.crt
> > > > > > > > > >
> > > > > > > > > > This will fail on openSUSE and probably on Ubuntu as well.
> > > > > > > > > >
> > > > > > > > > > It seems that the above path is set at configure time and embedded
> > > > > > > > > > into Curl's code as #define macros.
> > > > > > > > > >
> > > > > > > > > > Is there an ``official'' way to assess this? Like, can we depend on
> > > > > > > > > > NSS' certificate store as you wrote (quoted below)?
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Curl/OpenSSL have an enormous number of options and I am pretty sure it can
> > > > > > > > > be fixed, but first I need to understand where and how it's failing.
> > > > > > > > >
> > > > > > > > > We currently allow it to run with the default CA certificate path, do
> > > > > > > > > pre-verification on the server's certificate using those CA certificates,
> > > > > > > > > then call our SSL_VERIFY_PEER function where we override the verification
> > > > > > > > > result with the certificates from NSS.
> > > > > > > >
> > > > > > > > Apparently, it is failing before calling our SSL_VERIFY_PEER function.
> > > > > > > >
> > > > > > > > > If it's failing before reaching our SSL_VERIFY_PEER function, we should be
> > > > > > > > > able to use Curl's CURLOPT_CAINFO or CURLOPT_CAINFO_BLOB functions to set a
> > > > > > > > > custom CA certificate path (or in-memory buffer), maybe even an empty
> > > > > > > > > buffer, so that it proceeds further. ("man CURLOPT_CAINFO", "man
> > > > > > > > > CURLOPT_CAINFO_BLOB", or "man curl_easy_setopt" and read under the "SSL and
> > > > > > > > > SECURITY OPTIONS" section.)
> > > > > > > >
> > > > > > > > So we would need to hard-code and try all possible paths to the CA
> > > > > > > > bundle on Unix systems?
> > > > > > > >
> > > > > > > > > With the CURLOPT_CAINFO_BLOB option it might even be possible to skip the
> > > > > > > > > custom certificate verification we do later, and pre-populate Curl/OpenSSL
> > > > > > > > > with NSS certificates from the beginning, I just don't know enough about
> > > > > > > > > NSS to rely on that (eg. if you are using a cryptographic device or smart
> > > > > > > > > card in NSS, how does that work?). If that option is ok, then we might not
> > > > > > > > > even need the NSS libraries: recent versions of NSS store all the
> > > > > > > > > certificates in an SQLite database, which can be accessed with SQLite APIs
> > > > > > > > > directly, no need to build with or ship the NSS libraries at all.
> > > > > > > >
> > > > > > > > If I understood correctly [1], a NSS-linked Curl would query NSS by
> > > > > > > > itself... are we not in this condition?
> > > > > > [...]
> > > > > > > 1: https://curl.se/libcurl/c/CURLOPT_CAINFO.html
> > > > > >
> > > > > > Apparently, we are not. If we force CURLOPT_CAINFO and CURLOPT_CAPATH
> > > > > > to be NULL, we get a bit further but eventually Curl aborts because
> > > > > > validateServerX509Certificate fails.
> > > > > >
> > > > > > Here is the log for checking AOO updates (please excuse the long
> > > > > > lines). I added an extra bit of logging when entering and leaving
> > > > > > OPENSSL_ValidateServerCertificate, that is ``our SSL_VERIFY_PEER
> > > > > > function''.
> > > > > >
> > > > > > ----8<--------8<--------8<--------8<--------8<--------8<--------8<---------
> > > > > >
> > > > > > 1 9 2022-05-28 07:13:52.61 CurlSession::CurlSession with URL https://ooo-updates.apache.org/aoonext/check.Update?pkgfmt=installed
> > > > > > 2 9 2022-05-28 07:13:52.61 Not using a proxy server
> > > > > > 3 9 2022-05-28 07:13:52.61 GET line 1093
> > > > > > 4 9 2022-05-28 07:13:52.61 [CurlINFO ] Trying 151.101.2.132:443...
> > > > > > 5 9 2022-05-28 07:13:52.63 [CurlINFO ] Connected to ooo-updates.apache.org (151.101.2.132) port 443 (#0)
> > > > > > 6 9 2022-05-28 07:13:52.63 [CurlINFO ] ALPN, offering http/1.1
> > > > > > 7 9 2022-05-28 07:13:52.63 [CurlINFO ] Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
> > > > > > 8 9 2022-05-28 07:13:52.63 [CurlINFO ] TLSv1.2 (OUT), TLS handshake, Client hello (1):
> > > > > > 9 9 2022-05-28 07:13:52.64 [CurlINFO ] TLSv1.2 (IN), TLS handshake, Server hello (2):
> > > > > > 10 9 2022-05-28 07:13:52.64 [CurlINFO ] TLSv1.2 (IN), TLS handshake, Certificate (11):
> > > > > > --here we are inside OPENSSL_ValidateServerCertificate
> > > > > > 11 9 2022-05-28 07:13:52.64 validateServerX509Certificate() returning 0 at depth 2
> > > > > > --here we are leaving OPENSSL_ValidateServerCertificate
> > > > > > 12 9 2022-05-28 07:13:52.64 [CurlINFO ] TLSv1.2 (OUT), TLS alert, unknown CA (560):
> > > > > > 13 9 2022-05-28 07:13:52.64 [CurlINFO ] SSL certificate problem: unable to get local issuer certificate
> > > > > > 14 9 2022-05-28 07:13:52.64 [CurlINFO ] Closing connection 0
> > > > > > 15 9 2022-05-28 07:13:52.64 Curl request failed with CURLcode 60
> > > > > > 16 9 2022-05-28 07:13:52.64 CurlSession::~CurlSession: closed curl session
> > > > > >
> > > > > > ----8<--------8<--------8<--------8<--------8<--------8<--------8<---------
> > > > > >
> > > > > > I hope this helps.
> > > > > >
> > > > > >
> > > > > I think the problem there is NSS's validation.
> > > > >
> > > > > In CurlSession::validateServerX509Certificate(), try change the "if (depth == 0 )" to "if (1)".
> > > >
> > > > After this edit, the connection is established. The log is below.
> > > >
> > >
> > > That's good :).
> > >
> > >
> > > >
> > > > But NSS was also used before, with Serf... it stopped working when the
> > > > bundled CA certificates started to be ``too obsolete''. After we
> > > > updated the bundled NSS, then https connections, such as the check for
> > > > updates, started to work again.
> > > >
> > >
> > > We don't need NSS specifically. From Curl, we need OpenSSL, because without
> > > it, we are unable to ask the user to confirm self-signed certificates
> > > during TLS negotiations [1]. Upstream OpenSSL however doesn't ship any CA
> > > certificates, so the WebDAV provider gets them from our
> > > com.sun.star.security.CertificateContainer, which comes from WinCrypt on
> > > Windows (also called "mscrypt" by main/xmlsecurity/source/xmlsec/mscrypt)
> > > and NSS elsewhere.
> > >
> > > Therefore that update check should not have been failing on Windows. Only
> > > if your Windows is out of date, should WinCrypt CA certificates fail.
> >
> > Honestly, I do not really recall what happened on Windows.
> > I do recall that Linux builds were unable to check for update.
> > I thought that the reason was an outdated Serf library, and I proposed
> > an update, with the addition of SCons etc.
> > In parallel, we updated the version of the bundled NSS. When we did
> > _this_, the updates started to work again.
> > We never merged the serf update, so I think it makes sense to consider
> > the NSS obsolescence related.
> > We also tried to upgrade OpenSSL, but then stepped back. So NSS
> > remains the most probable cause.
> >
> > > It is also particularly strange for NSS certificates to be out of date. The
> > > NSS certificate database that our
> > > com.sun.star.security.CertificateContainer uses comes from a profile
> > > (getMozillaCurrentProfile() in
> > > main/xmlsecurity/source/nss/nssinitializer.cxx) which (AFAIK) is in the
> > > user's home directory, and shared with and updated by Firefox or
> > > Thunderbird. Only if you don't update AOO, and don't run Firefox, and don't
> > > run Thunderbird, can your NSS certificates get out of date.
> >
> > I do not know how to reply here. I can just share my findings from
> > above...
> >
> > > > Please also bear in mind that I am now testing a build from a _CentOS
> > > > Docker container_. My computer is not running CentOS, but rather
> > > > OpenSUSE. The binary is run on ``native'' OpenSUSE. If I recall
> > > > correctly, when I build on the same native OpenSUSE, the resulting AOO
> > > > binary connects properly to https web sites.
> > > >
> > > > For this reason, I was wondering if this problem is not just about
> > > > hardcoded paths to CA certificates.
> > > >
> > >
> > > Since Curl's OpenSSL CA path is hardcoded to a non-portable value at
> > > compile time, my proposed approach here disables OpenSSL's certificates and
> > > only uses the ones from our com.sun.star.security.CertificateContainer
> > > instead. (And, who knows, it may be helpful to make another
> > > com.sun.star.security.CertificateContainer that gets certificates from
> > > guessed system paths for all of AOO, not just WebDAV.)
> > >
> > > Is there something else you propose instead?
> >
> > I am not sure my emails are communicating the proper ``tone'', so
> > please let me state it explicitly: I am neither a cryptography, SSL or
> > NSS expert at all. You seem to be much more knowledgeable than me and
> > I am thankful to see your contributions. I am just doing my best to
> > validate them in all possible cases (mostly under different Linux
> > distributions).
> >
> Sorry if my tone sounded negative, I didn't mean that.
Your tone sounded perfect; I just wanted to point out that I am a bit
struggling to follow this thread as it is getting deep into code I do
not know well.
> My knowledge of cryptography/SSL/NSS could also be better.
>
> I've also been trying to get our cryptography working on many Linux
> distributions.
Praiseworthy, IMHO!
> > I am trying to give useful advice, such as the above comments about
> > NSS. I might just be wrong, in such case I apologize for the noise.
> >
> > Having said this... what do you mean exactly for your ``proposed
> > approach''? Is it the patch from "if (depth == 0)" to "if (1)"?
> >
> >
> That any-depth patch is probably needed no matter what else we do, because
> multiple certificates in the chain may need user confirmation. I've now
> committed it as c464040a4409a7ab63c22a7b2358ce0134c09c10.
>
> No, my "proposed approach" was: instead of OpenSSL using the certificate
> path hardcoded at compile-time, get it to use our certificate store, which
> is populated from NSS's certificates at present but could be populated by
> the Linux distro's certificates.
This is also IMHO the way we should go.
> > > [1] But is Curl's inability to use custom certificate verification
> > > functions (which can ask the user to confirm problematic certificates) with
> > > NSS a limitation of Curl, or a limitation of NSS? Firefox uses NSS, and it
> > > certainly asks the user to confirm expired and self-signed certificates.
> > > How does Firefox do that, by disconnecting, asking the user, and
> > > reconnecting if confirmed, or the Curl/OpenSSL way, by keeping the TCP
> > > connection open during the user interaction and then resuming TLS
> > > negotiations if confirmed?
> >
> > For what it's worth, I tried opening https://developer.berlios.de ,
> > which has an invalid SSL certificate. Firefox 91.9.1esr on Linux
> > x86_64 closes the TCP connection immediately and remains waiting for
> > user input.
> >
> > I hope this helps.
> >
> >
> Thank you. My Firefox does the same on sites with bad certificates. It's
> possible to patch our WebDAV module to do that too (disconnect, add the
> user-approved certificate to our certificate store, then reconnect), and
> would simplify some of the code and fix the hardcoded path problem. But I
> wonder whether that would fix expired certificates and other problems, or
> just allow self-signed but otherwise valid certificates. I'll give that
> some further thought.
Once we give users the option to access one kind of invalid
certificates, we should allow them to accept all kinds, IMHO.
> As for accessing Linux distro certificates, the p11-kit project existed
> since 2011, and comes with a CLI tool, "trust", which can examine system
> certificates on any Linux distribution (well, any that ships it, and it's a
> dependency for the widely used gcr package used by the gvfs package used by
> at least GNOME and XFCE, not sure about KDE). If you run "trust extract
> --format=openssl-bundle --purpose=server-auth filename", then that
> "filename" gets an OpenSSL-compatible certificate bundle, that can be
> imported into OpenSSL and used - portably. This could be done by AOO as
> needed. Because Linux distributions supply and maintain both p11-kit and
> the certificates, we shouldn't need NSS if we use it,
Now _this_ seems to be interesting! That's the cross-distribution tool
we need!
> although NSS might still be required for client certificates for
> signing documents (not sure how that works yet).
The xmlsecurity has nss and mscrypt alternative implementation of some
of its interfaces. It _seems_ that the nss implementation only uses
the NSS library for managing certificates (storage etc.) and for the
actual encryption/decryption/signature it calls libxmlsec functions.
But libxmlsec is compiled against nss... this looks complicated.
But did we already mention p11-glue?
https://p11-glue.github.io/p11-glue/p11-kit/manual/trust-nss.html
This could allow us to adopt p11-kit without changing (too much) our
current interface to NSS... what do you think?
Best regards,
--
Arrigo
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org