You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@openoffice.apache.org by Damjan Jovanovic <da...@apache.org> on 2022/04/26 17:56:22 UTC
WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)
On Mon, Nov 15, 2021 at 9:57 PM Jim Jagielski <ji...@jagunet.com> wrote:
> I'm gonna look into the serf->(lib)curl option... Since we don't use any
> of the fancy features of serf, I'm thinking that the easy option might be
> best
Hi
I've ported our WebDAV content provider module from Serf to Curl.
While it ended well, and several other bugs were found and fixed, it
definitely wasn't the "easy option" Jim ;). Starting with conservative
changes, it ended up needing total restructuring, and became more of a
rewrite. The crashes were frequent and hung connections many, and I had to
read up on the HTTP protocol, and read Curl and Serf's source code, but
eventually I prevailed, and a clean elegant stable Curl WebDAV module
emerged.
The huge patch is attached for anyone wishing to review and test. Unless
there are major objections, I'll push it in a couple of days.
STATUS
It builds and works well on FreeBSD and Windows.
Most of the code was reused, and all the operations and semantics
previously present with Serf, should have been preserved.
Browsing WebDAV files and directories, loading files, overwriting them
("Save"), creating them ("Save As"), renaming and deleting them, all works.
HTTP and HTTPS proxies work. Unlike Serf, Curl could also support SOCKS
proxies (with minimal further changes), but AOO lacks that setting in the
proxy UI and configuration.
Authentication works, both to the destination server and to the proxy
server. I've successfully tested Basic and Digest authentication. Curl
supports every authentication method Serf does and more.
HTTPS works, with a custom certificate verification function, using our own
certificate store from NSS and its API (like the Serf code used). A bug was
discovered (which is in the Serf implementation too) where self-signed
certificates were being unconditionally rejected; apparently NSS wants to
see that a copy of that certificate in its certificate chain parameter
too. Now they work, and the user gets prompted to allow access.
HTTPS and authentication can be used together on the same connection and
work well, both bringing up their UI dialogs as needed.
A bug was fixed where when username and password were both present in the
URL (dav://user:pass@host/path), the code was trying to split them at the
"@" instead of ":".
Unnecessary base64 encoding and decoding was removed, when importing the
TLS connection's certificates into our XSecurityEnvironment. They now come
in directly as ASN.1 DER, and still work.
The code was greatly restructured and cleaned up, as Curl's API is
synchronous and blocking, with parameters set in advance instead of through
many callbacks, which has allowed using short clear methods, and clean
separation between the session and request classes. The WebDAV content
provider has shrunk from 35 to 21 C++ files, 43 to 29 header files, and
19129 to 15991 lines of code. With WebDAV methods centralized and
refactored into only 10-20 lines of code each, instead of scattered across
4 files, it is much more understandable and maintainable now.
Curl is vastly more portable than Serf. We should build easily now even on
OS/2. We can remain with existing build tools instead of needing scons or
cmake just to build Serf.
3 now unused dependencies were removed: apr, apr-util, and serf. Serf isn't
so bad. APR's pool idea is an ingenious way of doing resource management in
C. However Curl has excellent documentation, guides, examples, and detailed
explanations and even example code for each setting, while Serf has no
documentation. Serf might be worth it in a project that already uses APR a
lot, but we don't.
Instead of the historical, crippled forms of logging like OSL_TRACE(),
which don't appear in release builds, I've made it use the newer
com.sun.star.logging UNO component (wrapped in comphelper::EventLogger),
which was inspired by java.util.logging, with configurable verbosity
levels, handlers (file and console) and output formats (plain, csv), and
importantly, which produces output in release builds too. I've also made it
so that on LogLevel::FINEST, Curl's verbose mode is enabled and Curl's
debug output is also logged through us, with descriptions of what Curl is
doing, and logs of all HTTP traffic including headers and bodies, before
encryption and after decryption in the case of HTTPS, giving us tremendous
detail that can be used for troubleshooting problems.
CURL CHANGED TO USE OPENSSL AND ZLIB
Curl only supports the custom TLS certificate verification function we use
(the CURLOPT_SSL_CTX_FUNCTION option) when built with OpenSSL, wolfSSL or
mbedTLS providers. We currently use schannel on Windows instead, which had
to be changed. I also made it use zlib, which generally helps, and WebDAV
uses XML which is very verbose and benefits from compression. On other OSes
with system curl, it is now checked for its SSL provider, and configure
fails if it isn't OpenSSL.
The new WebDAV module successfully builds and runs with both OpenSSL 1.0.2
or 1.1.1. However 1 function was renamed between those versions, so the
OpenSSL version at runtime probably has to match the one used at compile
time (although building with 1.0.2 headers might allow running with 1.1.1 -
not tested).
(After completing development and testing, it dawned on me there is a
completely different way to do the certification verification, which should
allow other SSL providers to be used and might be better in various ways.
See later.)
We currently build zlib as a static library only, and on Windows its C
runtime library is linked statically (_MT), which can't be mixed with
curl's dynamic linking of the runtime library (_MD). Thus curl was made to
link it statically too. Most if not all of our modules link the runtime
library statically too (which begs the question, why do we ship the msvcr*
redistributables to users at all then?).
ISSUES
The file open dialog (Ctrl+O) can hang for several minutes when first
connecting to a server. This is not new - it happens with Serf as well.
This appears to be caused by autocompletion in the file dialog. When typing
in a URL like "davs://127.0.0.1", the WebDAV content provider is first
called with a partial "https://1", before the rest of the URL is entered.
That "1" is (somehow) treated as IP address "0.0.0.1", and a TCP connection
to 0.0.0.1 is started. Only after several minutes, when that connection
times out and fails, does the content provider get another request with the
complete URL, which succeeds. The distinctly unpleasant wait, is luckily
only present that the first time that server is used, as caching remembers
the URL even across AOO restarts, so the "1" is automatically expanded to
the full URL and the content provider never sees "https://1" again.
You can only enter credentials for the HTTP(S) proxy or the destination,
never both, as the credential manager caches credentials per URL, not per
host/realm, so while you are prompted for credentials for both, and Curl is
told about both, the destination credentials overwrite the proxy
credentials in the cache, so one Curl request works but future Curl
requests use the wrong credentials to the proxy. This doesn't matter, as
AOO doesn't support passwords for proxy servers anyway, and Serf has the
same issue.
Damjan
--
P.S. APACHE 2.4 SETUP FOR TESTING
Put some files in /var/tmp/dav/files, then configure Apache HTTPD's
httpd.conf with something along these lines:
Listen 127.0.0.1:8080
LoadModule dav_module libexec/apache24/mod_dav.so
LoadModule dav_fs_module libexec/apache24/mod_dav_fs.so
<VirtualHost 127.0.0.1:8080>
DocumentRoot "/var/tmp/dav"
ErrorLog "/var/tmp/dav/errors.txt"
TransferLog "/var/tmp/dav/access.txt"
DavLockDB "/var/tmp/dav/DavLock"
# If using TLS, generate these self-signed certificates too:
# SSLEngine on
# SSLCertificateFile "/var/tmp/dav/cert.pem"
# SSLCertificateKeyFile "/var/tmp/dav/key.pem"
<Directory "/var/tmp/dav/files">
AllowOverride none
Require valid-user
# Require all granted
AuthType Basic
AuthName "WebDAV"
AuthBasicProvider file
AuthUserFile "/var/tmp/dav/passwords"
Dav on
</Directory>
</VirtualHost>
Enter your username and password with:
htpasswd -c /var/tmp/dav/passwords username
Then in the file open dialog, enter "dav://127.0.0.1:8080/files/" (or use
scheme davs:// instead, if using HTTPS).
Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)
Posted by Don Lewis <tr...@apache.org>.
On 26 Apr, Damjan Jovanovic wrote:
> On Mon, Nov 15, 2021 at 9:57 PM Jim Jagielski <ji...@jagunet.com> wrote:
>
>> I'm gonna look into the serf->(lib)curl option... Since we don't use any
>> of the fancy features of serf, I'm thinking that the easy option might be
>> best
>
>
>
> Hi
>
> I've ported our WebDAV content provider module from Serf to Curl.
>
> While it ended well, and several other bugs were found and fixed, it
> definitely wasn't the "easy option" Jim ;). Starting with conservative
> changes, it ended up needing total restructuring, and became more of a
> rewrite. The crashes were frequent and hung connections many, and I had to
> read up on the HTTP protocol, and read Curl and Serf's source code, but
> eventually I prevailed, and a clean elegant stable Curl WebDAV module
> emerged.
>
> The huge patch is attached for anyone wishing to review and test. Unless
> there are major objections, I'll push it in a couple of days.
>
> STATUS
>
> It builds and works well on FreeBSD and Windows.
>
> Most of the code was reused, and all the operations and semantics
> previously present with Serf, should have been preserved.
>
> Browsing WebDAV files and directories, loading files, overwriting them
> ("Save"), creating them ("Save As"), renaming and deleting them, all works.
>
> HTTP and HTTPS proxies work. Unlike Serf, Curl could also support SOCKS
> proxies (with minimal further changes), but AOO lacks that setting in the
> proxy UI and configuration.
>
> Authentication works, both to the destination server and to the proxy
> server. I've successfully tested Basic and Digest authentication. Curl
> supports every authentication method Serf does and more.
>
> HTTPS works, with a custom certificate verification function, using our own
> certificate store from NSS and its API (like the Serf code used). A bug was
> discovered (which is in the Serf implementation too) where self-signed
> certificates were being unconditionally rejected; apparently NSS wants to
> see that a copy of that certificate in its certificate chain parameter
> too. Now they work, and the user gets prompted to allow access.
>
> HTTPS and authentication can be used together on the same connection and
> work well, both bringing up their UI dialogs as needed.
>
> A bug was fixed where when username and password were both present in the
> URL (dav://user:pass@host/path), the code was trying to split them at the
> "@" instead of ":".
>
> Unnecessary base64 encoding and decoding was removed, when importing the
> TLS connection's certificates into our XSecurityEnvironment. They now come
> in directly as ASN.1 DER, and still work.
>
> The code was greatly restructured and cleaned up, as Curl's API is
> synchronous and blocking, with parameters set in advance instead of through
> many callbacks, which has allowed using short clear methods, and clean
> separation between the session and request classes. The WebDAV content
> provider has shrunk from 35 to 21 C++ files, 43 to 29 header files, and
> 19129 to 15991 lines of code. With WebDAV methods centralized and
> refactored into only 10-20 lines of code each, instead of scattered across
> 4 files, it is much more understandable and maintainable now.
>
> Curl is vastly more portable than Serf. We should build easily now even on
> OS/2. We can remain with existing build tools instead of needing scons or
> cmake just to build Serf.
>
> 3 now unused dependencies were removed: apr, apr-util, and serf. Serf isn't
> so bad. APR's pool idea is an ingenious way of doing resource management in
> C. However Curl has excellent documentation, guides, examples, and detailed
> explanations and even example code for each setting, while Serf has no
> documentation. Serf might be worth it in a project that already uses APR a
> lot, but we don't.
>
> Instead of the historical, crippled forms of logging like OSL_TRACE(),
> which don't appear in release builds, I've made it use the newer
> com.sun.star.logging UNO component (wrapped in comphelper::EventLogger),
> which was inspired by java.util.logging, with configurable verbosity
> levels, handlers (file and console) and output formats (plain, csv), and
> importantly, which produces output in release builds too. I've also made it
> so that on LogLevel::FINEST, Curl's verbose mode is enabled and Curl's
> debug output is also logged through us, with descriptions of what Curl is
> doing, and logs of all HTTP traffic including headers and bodies, before
> encryption and after decryption in the case of HTTPS, giving us tremendous
> detail that can be used for troubleshooting problems.
>
> CURL CHANGED TO USE OPENSSL AND ZLIB
>
> Curl only supports the custom TLS certificate verification function we use
> (the CURLOPT_SSL_CTX_FUNCTION option) when built with OpenSSL, wolfSSL or
> mbedTLS providers. We currently use schannel on Windows instead, which had
> to be changed. I also made it use zlib, which generally helps, and WebDAV
> uses XML which is very verbose and benefits from compression. On other OSes
> with system curl, it is now checked for its SSL provider, and configure
> fails if it isn't OpenSSL.
>
> The new WebDAV module successfully builds and runs with both OpenSSL 1.0.2
> or 1.1.1. However 1 function was renamed between those versions, so the
> OpenSSL version at runtime probably has to match the one used at compile
> time (although building with 1.0.2 headers might allow running with 1.1.1 -
> not tested).
>
> (After completing development and testing, it dawned on me there is a
> completely different way to do the certification verification, which should
> allow other SSL providers to be used and might be better in various ways.
> See later.)
>
> We currently build zlib as a static library only, and on Windows its C
> runtime library is linked statically (_MT), which can't be mixed with
> curl's dynamic linking of the runtime library (_MD). Thus curl was made to
> link it statically too. Most if not all of our modules link the runtime
> library statically too (which begs the question, why do we ship the msvcr*
> redistributables to users at all then?).
>
> ISSUES
>
> The file open dialog (Ctrl+O) can hang for several minutes when first
> connecting to a server. This is not new - it happens with Serf as well.
> This appears to be caused by autocompletion in the file dialog. When typing
> in a URL like "davs://127.0.0.1", the WebDAV content provider is first
> called with a partial "https://1", before the rest of the URL is entered.
> That "1" is (somehow) treated as IP address "0.0.0.1", and a TCP connection
> to 0.0.0.1 is started. Only after several minutes, when that connection
> times out and fails, does the content provider get another request with the
> complete URL, which succeeds. The distinctly unpleasant wait, is luckily
> only present that the first time that server is used, as caching remembers
> the URL even across AOO restarts, so the "1" is automatically expanded to
> the full URL and the content provider never sees "https://1" again.
>
> You can only enter credentials for the HTTP(S) proxy or the destination,
> never both, as the credential manager caches credentials per URL, not per
> host/realm, so while you are prompted for credentials for both, and Curl is
> told about both, the destination credentials overwrite the proxy
> credentials in the cache, so one Curl request works but future Curl
> requests use the wrong credentials to the proxy. This doesn't matter, as
> AOO doesn't support passwords for proxy servers anyway, and Serf has the
> same issue.
Some of the CVEs mentioned by https://curl.se/docs/vuln-7.82.0.html may
be now relavent for this new usage of curl.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)
Posted by Marcus <ma...@wtnet.de>.
Am 27.04.22 um 17:44 schrieb Matthias Seidel:
> Sounds great!
I also think that it's a great progress.
As none-developer I cannot really judge how good and valuable the
changes are. But all in all I'm happy to see this refactored to have
less problems.
Well done. :-)
Marcus
> Am 26.04.22 um 19:56 schrieb Damjan Jovanovic:
>> On Mon, Nov 15, 2021 at 9:57 PM Jim Jagielski <jim@jagunet.com
>> <ma...@jagunet.com>> wrote:
>>
>> I'm gonna look into the serf->(lib)curl option... Since we don't
>> use any of the fancy features of serf, I'm thinking that the easy
>> option might be best
>>
>>
>>
>> Hi
>>
>> I've ported our WebDAV content provider module from Serf to Curl.
>>
>> While it ended well, and several other bugs were found and fixed, it
>> definitely wasn't the "easy option" Jim ;). Starting with conservative
>> changes, it ended up needing total restructuring, and became more of a
>> rewrite. The crashes were frequent and hung connections many, and I
>> had to read up on the HTTP protocol, and read Curl and Serf's source
>> code, but eventually I prevailed, and a clean elegant stable Curl
>> WebDAV module emerged.
>>
>> The huge patch is attached for anyone wishing to review and test.
>> Unless there are major objections, I'll push it in a couple of days.
>>
>> STATUS
>>
>> It builds and works well on FreeBSD and Windows.
>>
>> Most of the code was reused, and all the operations and semantics
>> previously present with Serf, should have been preserved.
>>
>> Browsing WebDAV files and directories, loading files, overwriting them
>> ("Save"), creating them ("Save As"), renaming and deleting them, all
>> works.
>>
>> HTTP and HTTPS proxies work. Unlike Serf, Curl could also support
>> SOCKS proxies (with minimal further changes), but AOO lacks that
>> setting in the proxy UI and configuration.
>>
>> Authentication works, both to the destination server and to the proxy
>> server. I've successfully tested Basic and Digest authentication. Curl
>> supports every authentication method Serf does and more.
>>
>> HTTPS works, with a custom certificate verification function, using
>> our own certificate store from NSS and its API (like the Serf code
>> used). A bug was discovered (which is in the Serf implementation too)
>> where self-signed certificates were being unconditionally rejected;
>> apparently NSS wants to see that a copy of that certificate in its
>> certificate chain parameter too. Now they work, and the user gets
>> prompted to allow access.
>>
>> HTTPS and authentication can be used together on the same connection
>> and work well, both bringing up their UI dialogs as needed.
>>
>> A bug was fixed where when username and password were both present in
>> the URL (dav://user:pass@host/path), the code was trying to split them
>> at the "@" instead of ":".
>>
>> Unnecessary base64 encoding and decoding was removed, when importing
>> the TLS connection's certificates into our XSecurityEnvironment. They
>> now come in directly as ASN.1 DER, and still work.
>>
>> The code was greatly restructured and cleaned up, as Curl's API is
>> synchronous and blocking, with parameters set in advance instead of
>> through many callbacks, which has allowed using short clear methods,
>> and clean separation between the session and request classes. The
>> WebDAV content provider has shrunk from 35 to 21 C++ files, 43 to 29
>> header files, and 19129 to 15991 lines of code. With WebDAV methods
>> centralized and refactored into only 10-20 lines of code each, instead
>> of scattered across 4 files, it is much more understandable and
>> maintainable now.
>>
>> Curl is vastly more portable than Serf. We should build easily now
>> even on OS/2. We can remain with existing build tools instead of
>> needing scons or cmake just to build Serf.
>>
>> 3 now unused dependencies were removed: apr, apr-util, and serf. Serf
>> isn't so bad. APR's pool idea is an ingenious way of doing resource
>> management in C. However Curl has excellent documentation, guides,
>> examples, and detailed explanations and even example code for each
>> setting, while Serf has no documentation. Serf might be worth it in a
>> project that already uses APR a lot, but we don't.
>>
>> Instead of the historical, crippled forms of logging like OSL_TRACE(),
>> which don't appear in release builds, I've made it use the newer
>> com.sun.star.logging UNO component (wrapped in
>> comphelper::EventLogger), which was inspired by java.util.logging,
>> with configurable verbosity levels, handlers (file and console) and
>> output formats (plain, csv), and importantly, which produces output in
>> release builds too. I've also made it so that on LogLevel::FINEST,
>> Curl's verbose mode is enabled and Curl's debug output is also logged
>> through us, with descriptions of what Curl is doing, and logs of all
>> HTTP traffic including headers and bodies, before encryption and after
>> decryption in the case of HTTPS, giving us tremendous detail that can
>> be used for troubleshooting problems.
>>
>> CURL CHANGED TO USE OPENSSL AND ZLIB
>>
>> Curl only supports the custom TLS certificate verification function we
>> use (the CURLOPT_SSL_CTX_FUNCTION option) when built with OpenSSL,
>> wolfSSL or mbedTLS providers. We currently use schannel on Windows
>> instead, which had to be changed. I also made it use zlib, which
>> generally helps, and WebDAV uses XML which is very verbose and
>> benefits from compression. On other OSes with system curl, it is now
>> checked for its SSL provider, and configure fails if it isn't OpenSSL.
>>
>> The new WebDAV module successfully builds and runs with both OpenSSL
>> 1.0.2 or 1.1.1. However 1 function was renamed between those versions,
>> so the OpenSSL version at runtime probably has to match the one used
>> at compile time (although building with 1.0.2 headers might allow
>> running with 1.1.1 - not tested).
>>
>> (After completing development and testing, it dawned on me there is a
>> completely different way to do the certification verification, which
>> should allow other SSL providers to be used and might be better in
>> various ways. See later.)
>>
>> We currently build zlib as a static library only, and on Windows its C
>> runtime library is linked statically (_MT), which can't be mixed with
>> curl's dynamic linking of the runtime library (_MD). Thus curl was
>> made to link it statically too. Most if not all of our modules link
>> the runtime library statically too (which begs the question, why do we
>> ship the msvcr* redistributables to users at all then?).
>>
>> ISSUES
>>
>> The file open dialog (Ctrl+O) can hang for several minutes when first
>> connecting to a server. This is not new - it happens with Serf as
>> well. This appears to be caused by autocompletion in the file dialog.
>> When typing in a URL like "davs://127.0.0.1 <http://127.0.0.1>", the
>> WebDAV content provider is first called with a partial "https://1
>> <https://1>", before the rest of the URL is entered. That "1" is
>> (somehow) treated as IP address "0.0.0.1", and a TCP connection to
>> 0.0.0.1 is started. Only after several minutes, when that connection
>> times out and fails, does the content provider get another request
>> with the complete URL, which succeeds. The distinctly unpleasant wait,
>> is luckily only present that the first time that server is used, as
>> caching remembers the URL even across AOO restarts, so the "1" is
>> automatically expanded to the full URL and the content provider never
>> sees "https://1 <https://1>" again.
>>
>> You can only enter credentials for the HTTP(S) proxy or the
>> destination, never both, as the credential manager caches credentials
>> per URL, not per host/realm, so while you are prompted for credentials
>> for both, and Curl is told about both, the destination credentials
>> overwrite the proxy credentials in the cache, so one Curl request
>> works but future Curl requests use the wrong credentials to the proxy.
>> This doesn't matter, as AOO doesn't support passwords for proxy
>> servers anyway, and Serf has the same issue.
>>
>> Damjan
>>
>> --
>>
>> P.S. APACHE 2.4 SETUP FOR TESTING
>>
>> Put some files in /var/tmp/dav/files, then configure Apache HTTPD's
>> httpd.conf with something along these lines:
>>
>> Listen 127.0.0.1:8080 <http://127.0.0.1:8080>
>> LoadModule dav_module libexec/apache24/mod_dav.so
>> LoadModule dav_fs_module libexec/apache24/mod_dav_fs.so
>>
>> <VirtualHost 127.0.0.1:8080 <http://127.0.0.1:8080>>
>> DocumentRoot "/var/tmp/dav"
>> ErrorLog "/var/tmp/dav/errors.txt"
>> TransferLog "/var/tmp/dav/access.txt"
>> DavLockDB "/var/tmp/dav/DavLock"
>>
>> # If using TLS, generate these self-signed certificates too:
>> # SSLEngine on
>> # SSLCertificateFile "/var/tmp/dav/cert.pem"
>> # SSLCertificateKeyFile "/var/tmp/dav/key.pem"
>>
>> <Directory "/var/tmp/dav/files">
>> AllowOverride none
>> Require valid-user
>> # Require all granted
>>
>> AuthType Basic
>> AuthName "WebDAV"
>> AuthBasicProvider file
>> AuthUserFile "/var/tmp/dav/passwords"
>>
>> Dav on
>> </Directory>
>> </VirtualHost>
>>
>> Enter your username and password with:
>> htpasswd -c /var/tmp/dav/passwords username
>>
>> Then in the file open dialog, enter "dav://127.0.0.1:8080/files/
>> <http://127.0.0.1:8080/files/>" (or use scheme davs:// instead, if
>> using HTTPS).
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)
Posted by Arrigo Marchiori <ar...@yahoo.it.INVALID>.
Dear All,
first of all thank you Damjan, for this huge work!
It looks... wonderful.
On Wed, Apr 27, 2022 at 07:11:14PM +0200, Damjan Jovanovic wrote:
> Thank you Matthias.
>
> Not too happy to use Github, but here is the pull request:
> https://github.com/apache/openoffice/pull/146
This is much helpful, thanks.
I see that my review was requested. I need a couple of days to check
it builds fine under Linux. Matthias, will you please check under
Windows?
Jim, could you please check MacOS? I cannot build trunk on that
platform yet.
I am not asking because I don't trust Damjan, but just to suggest we
honestly hunt for ``works on my machine'' effects ;-)
The commit is... big and IMHO we shall not hurry.
Best regards.
> > On Mon, Nov 15, 2021 at 9:57 PM Jim Jagielski <ji...@jagunet.com> wrote:
> >
> >> I'm gonna look into the serf->(lib)curl option... Since we don't use any
> >> of the fancy features of serf, I'm thinking that the easy option might be
> >> best
> >
> >
> >
> > Hi
> >
> > I've ported our WebDAV content provider module from Serf to Curl.
[...]
--
Arrigo
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)
Posted by Damjan Jovanovic <da...@apache.org>.
On Wed, Apr 27, 2022 at 9:22 PM Matthias Seidel <ma...@hamburg.de>
wrote:
> Hi,
>
> Am 27.04.22 um 21:09 schrieb Damjan Jovanovic:
> > On Wed, Apr 27, 2022 at 8:07 PM Matthias Seidel <
> matthias.seidel@hamburg.de>
> > wrote:
> >
> >> Hi Damjan,
> >>
> >> Am 27.04.22 um 19:11 schrieb Damjan Jovanovic:
> >>> Thank you Matthias.
> >>>
> >>> Not too happy to use Github, but here is the pull request:
> >>> https://github.com/apache/openoffice/pull/146
> >> Thanks, Git makes it a lot easier for me to build...
> >>
> >> BTW: Is it OK when I cherry-pick your other two commits to AOO42X?
> >>
> >>
> > Which ones? The Python 3.8 and the syslog commits?
> Yes
> >
>
>
Sure, cherry-pick them. The IDL file for the syslog patch needs to be
changed to say "since 4.2" though, instead of the current 4.5.
Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)
Posted by Matthias Seidel <ma...@hamburg.de>.
Hi,
Am 27.04.22 um 21:09 schrieb Damjan Jovanovic:
> On Wed, Apr 27, 2022 at 8:07 PM Matthias Seidel <ma...@hamburg.de>
> wrote:
>
>> Hi Damjan,
>>
>> Am 27.04.22 um 19:11 schrieb Damjan Jovanovic:
>>> Thank you Matthias.
>>>
>>> Not too happy to use Github, but here is the pull request:
>>> https://github.com/apache/openoffice/pull/146
>> Thanks, Git makes it a lot easier for me to build...
>>
>> BTW: Is it OK when I cherry-pick your other two commits to AOO42X?
>>
>>
> Which ones? The Python 3.8 and the syslog commits?
Yes
>
Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)
Posted by Damjan Jovanovic <da...@apache.org>.
On Wed, Apr 27, 2022 at 8:07 PM Matthias Seidel <ma...@hamburg.de>
wrote:
> Hi Damjan,
>
> Am 27.04.22 um 19:11 schrieb Damjan Jovanovic:
> > Thank you Matthias.
> >
> > Not too happy to use Github, but here is the pull request:
> > https://github.com/apache/openoffice/pull/146
>
> Thanks, Git makes it a lot easier for me to build...
>
> BTW: Is it OK when I cherry-pick your other two commits to AOO42X?
>
>
Which ones? The Python 3.8 and the syslog commits?
Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)
Posted by Matthias Seidel <ma...@hamburg.de>.
Hi Damjan,
Am 27.04.22 um 19:11 schrieb Damjan Jovanovic:
> Thank you Matthias.
>
> Not too happy to use Github, but here is the pull request:
> https://github.com/apache/openoffice/pull/146
Thanks, Git makes it a lot easier for me to build...
BTW: Is it OK when I cherry-pick your other two commits to AOO42X?
Regards,
Matthias
>
>
>
> On Wed, Apr 27, 2022 at 5:44 PM Matthias Seidel <ma...@hamburg.de>
> wrote:
>
>> Hi Damjan,
>>
>> Sounds great!
>>
>> Can we have that as a Pull Request?
>>
>> Regards,
>>
>> Matthias
>> Am 26.04.22 um 19:56 schrieb Damjan Jovanovic:
>>
>> On Mon, Nov 15, 2021 at 9:57 PM Jim Jagielski <ji...@jagunet.com> wrote:
>>
>>> I'm gonna look into the serf->(lib)curl option... Since we don't use any
>>> of the fancy features of serf, I'm thinking that the easy option might be
>>> best
>>
>>
>> Hi
>>
>> I've ported our WebDAV content provider module from Serf to Curl.
>>
>> While it ended well, and several other bugs were found and fixed, it
>> definitely wasn't the "easy option" Jim ;). Starting with conservative
>> changes, it ended up needing total restructuring, and became more of a
>> rewrite. The crashes were frequent and hung connections many, and I had to
>> read up on the HTTP protocol, and read Curl and Serf's source code, but
>> eventually I prevailed, and a clean elegant stable Curl WebDAV module
>> emerged.
>>
>> The huge patch is attached for anyone wishing to review and test. Unless
>> there are major objections, I'll push it in a couple of days.
>>
>> STATUS
>>
>> It builds and works well on FreeBSD and Windows.
>>
>> Most of the code was reused, and all the operations and semantics
>> previously present with Serf, should have been preserved.
>>
>> Browsing WebDAV files and directories, loading files, overwriting them
>> ("Save"), creating them ("Save As"), renaming and deleting them, all works.
>>
>> HTTP and HTTPS proxies work. Unlike Serf, Curl could also support SOCKS
>> proxies (with minimal further changes), but AOO lacks that setting in the
>> proxy UI and configuration.
>>
>> Authentication works, both to the destination server and to the proxy
>> server. I've successfully tested Basic and Digest authentication. Curl
>> supports every authentication method Serf does and more.
>>
>> HTTPS works, with a custom certificate verification function, using our
>> own certificate store from NSS and its API (like the Serf code used). A bug
>> was discovered (which is in the Serf implementation too) where self-signed
>> certificates were being unconditionally rejected; apparently NSS wants to
>> see that a copy of that certificate in its certificate chain parameter
>> too. Now they work, and the user gets prompted to allow access.
>>
>> HTTPS and authentication can be used together on the same connection and
>> work well, both bringing up their UI dialogs as needed.
>>
>> A bug was fixed where when username and password were both present in the
>> URL (dav://user:pass@host/path), the code was trying to split them at the
>> "@" instead of ":".
>>
>> Unnecessary base64 encoding and decoding was removed, when importing the
>> TLS connection's certificates into our XSecurityEnvironment. They now come
>> in directly as ASN.1 DER, and still work.
>>
>> The code was greatly restructured and cleaned up, as Curl's API is
>> synchronous and blocking, with parameters set in advance instead of through
>> many callbacks, which has allowed using short clear methods, and clean
>> separation between the session and request classes. The WebDAV content
>> provider has shrunk from 35 to 21 C++ files, 43 to 29 header files, and
>> 19129 to 15991 lines of code. With WebDAV methods centralized and
>> refactored into only 10-20 lines of code each, instead of scattered across
>> 4 files, it is much more understandable and maintainable now.
>>
>> Curl is vastly more portable than Serf. We should build easily now even on
>> OS/2. We can remain with existing build tools instead of needing scons or
>> cmake just to build Serf.
>>
>> 3 now unused dependencies were removed: apr, apr-util, and serf. Serf
>> isn't so bad. APR's pool idea is an ingenious way of doing resource
>> management in C. However Curl has excellent documentation, guides,
>> examples, and detailed explanations and even example code for each setting,
>> while Serf has no documentation. Serf might be worth it in a project that
>> already uses APR a lot, but we don't.
>>
>> Instead of the historical, crippled forms of logging like OSL_TRACE(),
>> which don't appear in release builds, I've made it use the newer
>> com.sun.star.logging UNO component (wrapped in comphelper::EventLogger),
>> which was inspired by java.util.logging, with configurable verbosity
>> levels, handlers (file and console) and output formats (plain, csv), and
>> importantly, which produces output in release builds too. I've also made it
>> so that on LogLevel::FINEST, Curl's verbose mode is enabled and Curl's
>> debug output is also logged through us, with descriptions of what Curl is
>> doing, and logs of all HTTP traffic including headers and bodies, before
>> encryption and after decryption in the case of HTTPS, giving us tremendous
>> detail that can be used for troubleshooting problems.
>>
>> CURL CHANGED TO USE OPENSSL AND ZLIB
>>
>> Curl only supports the custom TLS certificate verification function we use
>> (the CURLOPT_SSL_CTX_FUNCTION option) when built with OpenSSL, wolfSSL or
>> mbedTLS providers. We currently use schannel on Windows instead, which had
>> to be changed. I also made it use zlib, which generally helps, and WebDAV
>> uses XML which is very verbose and benefits from compression. On other OSes
>> with system curl, it is now checked for its SSL provider, and configure
>> fails if it isn't OpenSSL.
>>
>> The new WebDAV module successfully builds and runs with both OpenSSL 1.0.2
>> or 1.1.1. However 1 function was renamed between those versions, so the
>> OpenSSL version at runtime probably has to match the one used at compile
>> time (although building with 1.0.2 headers might allow running with 1.1.1 -
>> not tested).
>>
>> (After completing development and testing, it dawned on me there is a
>> completely different way to do the certification verification, which should
>> allow other SSL providers to be used and might be better in various ways.
>> See later.)
>>
>> We currently build zlib as a static library only, and on Windows its C
>> runtime library is linked statically (_MT), which can't be mixed with
>> curl's dynamic linking of the runtime library (_MD). Thus curl was made to
>> link it statically too. Most if not all of our modules link the runtime
>> library statically too (which begs the question, why do we ship the msvcr*
>> redistributables to users at all then?).
>>
>> ISSUES
>>
>> The file open dialog (Ctrl+O) can hang for several minutes when first
>> connecting to a server. This is not new - it happens with Serf as well.
>> This appears to be caused by autocompletion in the file dialog. When typing
>> in a URL like "davs://127.0.0.1", the WebDAV content provider is first
>> called with a partial "https://1", before the rest of the URL is entered.
>> That "1" is (somehow) treated as IP address "0.0.0.1", and a TCP connection
>> to 0.0.0.1 is started. Only after several minutes, when that connection
>> times out and fails, does the content provider get another request with the
>> complete URL, which succeeds. The distinctly unpleasant wait, is luckily
>> only present that the first time that server is used, as caching remembers
>> the URL even across AOO restarts, so the "1" is automatically expanded to
>> the full URL and the content provider never sees "https://1" again.
>>
>> You can only enter credentials for the HTTP(S) proxy or the destination,
>> never both, as the credential manager caches credentials per URL, not per
>> host/realm, so while you are prompted for credentials for both, and Curl is
>> told about both, the destination credentials overwrite the proxy
>> credentials in the cache, so one Curl request works but future Curl
>> requests use the wrong credentials to the proxy. This doesn't matter, as
>> AOO doesn't support passwords for proxy servers anyway, and Serf has the
>> same issue.
>>
>> Damjan
>>
>> --
>>
>> P.S. APACHE 2.4 SETUP FOR TESTING
>>
>> Put some files in /var/tmp/dav/files, then configure Apache HTTPD's
>> httpd.conf with something along these lines:
>>
>> Listen 127.0.0.1:8080
>> LoadModule dav_module libexec/apache24/mod_dav.so
>> LoadModule dav_fs_module libexec/apache24/mod_dav_fs.so
>>
>> <VirtualHost 127.0.0.1:8080>
>> DocumentRoot "/var/tmp/dav"
>> ErrorLog "/var/tmp/dav/errors.txt"
>> TransferLog "/var/tmp/dav/access.txt"
>> DavLockDB "/var/tmp/dav/DavLock"
>>
>> # If using TLS, generate these self-signed certificates too:
>> # SSLEngine on
>> # SSLCertificateFile "/var/tmp/dav/cert.pem"
>> # SSLCertificateKeyFile "/var/tmp/dav/key.pem"
>>
>> <Directory "/var/tmp/dav/files">
>> AllowOverride none
>> Require valid-user
>> # Require all granted
>>
>> AuthType Basic
>> AuthName "WebDAV"
>> AuthBasicProvider file
>> AuthUserFile "/var/tmp/dav/passwords"
>>
>> Dav on
>> </Directory>
>> </VirtualHost>
>>
>> Enter your username and password with:
>> htpasswd -c /var/tmp/dav/passwords username
>>
>> Then in the file open dialog, enter "dav://127.0.0.1:8080/files/" (or use
>> scheme davs:// instead, if using HTTPS).
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
>> For additional commands, e-mail: dev-help@openoffice.apache.org
>>
>>
Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)
Posted by Damjan Jovanovic <da...@apache.org>.
Thank you Matthias.
Not too happy to use Github, but here is the pull request:
https://github.com/apache/openoffice/pull/146
On Wed, Apr 27, 2022 at 5:44 PM Matthias Seidel <ma...@hamburg.de>
wrote:
> Hi Damjan,
>
> Sounds great!
>
> Can we have that as a Pull Request?
>
> Regards,
>
> Matthias
> Am 26.04.22 um 19:56 schrieb Damjan Jovanovic:
>
> On Mon, Nov 15, 2021 at 9:57 PM Jim Jagielski <ji...@jagunet.com> wrote:
>
>> I'm gonna look into the serf->(lib)curl option... Since we don't use any
>> of the fancy features of serf, I'm thinking that the easy option might be
>> best
>
>
>
> Hi
>
> I've ported our WebDAV content provider module from Serf to Curl.
>
> While it ended well, and several other bugs were found and fixed, it
> definitely wasn't the "easy option" Jim ;). Starting with conservative
> changes, it ended up needing total restructuring, and became more of a
> rewrite. The crashes were frequent and hung connections many, and I had to
> read up on the HTTP protocol, and read Curl and Serf's source code, but
> eventually I prevailed, and a clean elegant stable Curl WebDAV module
> emerged.
>
> The huge patch is attached for anyone wishing to review and test. Unless
> there are major objections, I'll push it in a couple of days.
>
> STATUS
>
> It builds and works well on FreeBSD and Windows.
>
> Most of the code was reused, and all the operations and semantics
> previously present with Serf, should have been preserved.
>
> Browsing WebDAV files and directories, loading files, overwriting them
> ("Save"), creating them ("Save As"), renaming and deleting them, all works.
>
> HTTP and HTTPS proxies work. Unlike Serf, Curl could also support SOCKS
> proxies (with minimal further changes), but AOO lacks that setting in the
> proxy UI and configuration.
>
> Authentication works, both to the destination server and to the proxy
> server. I've successfully tested Basic and Digest authentication. Curl
> supports every authentication method Serf does and more.
>
> HTTPS works, with a custom certificate verification function, using our
> own certificate store from NSS and its API (like the Serf code used). A bug
> was discovered (which is in the Serf implementation too) where self-signed
> certificates were being unconditionally rejected; apparently NSS wants to
> see that a copy of that certificate in its certificate chain parameter
> too. Now they work, and the user gets prompted to allow access.
>
> HTTPS and authentication can be used together on the same connection and
> work well, both bringing up their UI dialogs as needed.
>
> A bug was fixed where when username and password were both present in the
> URL (dav://user:pass@host/path), the code was trying to split them at the
> "@" instead of ":".
>
> Unnecessary base64 encoding and decoding was removed, when importing the
> TLS connection's certificates into our XSecurityEnvironment. They now come
> in directly as ASN.1 DER, and still work.
>
> The code was greatly restructured and cleaned up, as Curl's API is
> synchronous and blocking, with parameters set in advance instead of through
> many callbacks, which has allowed using short clear methods, and clean
> separation between the session and request classes. The WebDAV content
> provider has shrunk from 35 to 21 C++ files, 43 to 29 header files, and
> 19129 to 15991 lines of code. With WebDAV methods centralized and
> refactored into only 10-20 lines of code each, instead of scattered across
> 4 files, it is much more understandable and maintainable now.
>
> Curl is vastly more portable than Serf. We should build easily now even on
> OS/2. We can remain with existing build tools instead of needing scons or
> cmake just to build Serf.
>
> 3 now unused dependencies were removed: apr, apr-util, and serf. Serf
> isn't so bad. APR's pool idea is an ingenious way of doing resource
> management in C. However Curl has excellent documentation, guides,
> examples, and detailed explanations and even example code for each setting,
> while Serf has no documentation. Serf might be worth it in a project that
> already uses APR a lot, but we don't.
>
> Instead of the historical, crippled forms of logging like OSL_TRACE(),
> which don't appear in release builds, I've made it use the newer
> com.sun.star.logging UNO component (wrapped in comphelper::EventLogger),
> which was inspired by java.util.logging, with configurable verbosity
> levels, handlers (file and console) and output formats (plain, csv), and
> importantly, which produces output in release builds too. I've also made it
> so that on LogLevel::FINEST, Curl's verbose mode is enabled and Curl's
> debug output is also logged through us, with descriptions of what Curl is
> doing, and logs of all HTTP traffic including headers and bodies, before
> encryption and after decryption in the case of HTTPS, giving us tremendous
> detail that can be used for troubleshooting problems.
>
> CURL CHANGED TO USE OPENSSL AND ZLIB
>
> Curl only supports the custom TLS certificate verification function we use
> (the CURLOPT_SSL_CTX_FUNCTION option) when built with OpenSSL, wolfSSL or
> mbedTLS providers. We currently use schannel on Windows instead, which had
> to be changed. I also made it use zlib, which generally helps, and WebDAV
> uses XML which is very verbose and benefits from compression. On other OSes
> with system curl, it is now checked for its SSL provider, and configure
> fails if it isn't OpenSSL.
>
> The new WebDAV module successfully builds and runs with both OpenSSL 1.0.2
> or 1.1.1. However 1 function was renamed between those versions, so the
> OpenSSL version at runtime probably has to match the one used at compile
> time (although building with 1.0.2 headers might allow running with 1.1.1 -
> not tested).
>
> (After completing development and testing, it dawned on me there is a
> completely different way to do the certification verification, which should
> allow other SSL providers to be used and might be better in various ways.
> See later.)
>
> We currently build zlib as a static library only, and on Windows its C
> runtime library is linked statically (_MT), which can't be mixed with
> curl's dynamic linking of the runtime library (_MD). Thus curl was made to
> link it statically too. Most if not all of our modules link the runtime
> library statically too (which begs the question, why do we ship the msvcr*
> redistributables to users at all then?).
>
> ISSUES
>
> The file open dialog (Ctrl+O) can hang for several minutes when first
> connecting to a server. This is not new - it happens with Serf as well.
> This appears to be caused by autocompletion in the file dialog. When typing
> in a URL like "davs://127.0.0.1", the WebDAV content provider is first
> called with a partial "https://1", before the rest of the URL is entered.
> That "1" is (somehow) treated as IP address "0.0.0.1", and a TCP connection
> to 0.0.0.1 is started. Only after several minutes, when that connection
> times out and fails, does the content provider get another request with the
> complete URL, which succeeds. The distinctly unpleasant wait, is luckily
> only present that the first time that server is used, as caching remembers
> the URL even across AOO restarts, so the "1" is automatically expanded to
> the full URL and the content provider never sees "https://1" again.
>
> You can only enter credentials for the HTTP(S) proxy or the destination,
> never both, as the credential manager caches credentials per URL, not per
> host/realm, so while you are prompted for credentials for both, and Curl is
> told about both, the destination credentials overwrite the proxy
> credentials in the cache, so one Curl request works but future Curl
> requests use the wrong credentials to the proxy. This doesn't matter, as
> AOO doesn't support passwords for proxy servers anyway, and Serf has the
> same issue.
>
> Damjan
>
> --
>
> P.S. APACHE 2.4 SETUP FOR TESTING
>
> Put some files in /var/tmp/dav/files, then configure Apache HTTPD's
> httpd.conf with something along these lines:
>
> Listen 127.0.0.1:8080
> LoadModule dav_module libexec/apache24/mod_dav.so
> LoadModule dav_fs_module libexec/apache24/mod_dav_fs.so
>
> <VirtualHost 127.0.0.1:8080>
> DocumentRoot "/var/tmp/dav"
> ErrorLog "/var/tmp/dav/errors.txt"
> TransferLog "/var/tmp/dav/access.txt"
> DavLockDB "/var/tmp/dav/DavLock"
>
> # If using TLS, generate these self-signed certificates too:
> # SSLEngine on
> # SSLCertificateFile "/var/tmp/dav/cert.pem"
> # SSLCertificateKeyFile "/var/tmp/dav/key.pem"
>
> <Directory "/var/tmp/dav/files">
> AllowOverride none
> Require valid-user
> # Require all granted
>
> AuthType Basic
> AuthName "WebDAV"
> AuthBasicProvider file
> AuthUserFile "/var/tmp/dav/passwords"
>
> Dav on
> </Directory>
> </VirtualHost>
>
> Enter your username and password with:
> htpasswd -c /var/tmp/dav/passwords username
>
> Then in the file open dialog, enter "dav://127.0.0.1:8080/files/" (or use
> scheme davs:// instead, if using HTTPS).
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
> For additional commands, e-mail: dev-help@openoffice.apache.org
>
>
Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)
Posted by Matthias Seidel <ma...@hamburg.de>.
Hi Damjan,
Sounds great!
Can we have that as a Pull Request?
Regards,
Matthias
Am 26.04.22 um 19:56 schrieb Damjan Jovanovic:
> On Mon, Nov 15, 2021 at 9:57 PM Jim Jagielski <jim@jagunet.com
> <ma...@jagunet.com>> wrote:
>
> I'm gonna look into the serf->(lib)curl option... Since we don't
> use any of the fancy features of serf, I'm thinking that the easy
> option might be best
>
>
>
> Hi
>
> I've ported our WebDAV content provider module from Serf to Curl.
>
> While it ended well, and several other bugs were found and fixed, it
> definitely wasn't the "easy option" Jim ;). Starting with conservative
> changes, it ended up needing total restructuring, and became more of a
> rewrite. The crashes were frequent and hung connections many, and I
> had to read up on the HTTP protocol, and read Curl and Serf's source
> code, but eventually I prevailed, and a clean elegant stable Curl
> WebDAV module emerged.
>
> The huge patch is attached for anyone wishing to review and test.
> Unless there are major objections, I'll push it in a couple of days.
>
> STATUS
>
> It builds and works well on FreeBSD and Windows.
>
> Most of the code was reused, and all the operations and semantics
> previously present with Serf, should have been preserved.
>
> Browsing WebDAV files and directories, loading files, overwriting them
> ("Save"), creating them ("Save As"), renaming and deleting them, all
> works.
>
> HTTP and HTTPS proxies work. Unlike Serf, Curl could also support
> SOCKS proxies (with minimal further changes), but AOO lacks that
> setting in the proxy UI and configuration.
>
> Authentication works, both to the destination server and to the proxy
> server. I've successfully tested Basic and Digest authentication. Curl
> supports every authentication method Serf does and more.
>
> HTTPS works, with a custom certificate verification function, using
> our own certificate store from NSS and its API (like the Serf code
> used). A bug was discovered (which is in the Serf implementation too)
> where self-signed certificates were being unconditionally rejected;
> apparently NSS wants to see that a copy of that certificate in its
> certificate chain parameter too. Now they work, and the user gets
> prompted to allow access.
>
> HTTPS and authentication can be used together on the same connection
> and work well, both bringing up their UI dialogs as needed.
>
> A bug was fixed where when username and password were both present in
> the URL (dav://user:pass@host/path), the code was trying to split them
> at the "@" instead of ":".
>
> Unnecessary base64 encoding and decoding was removed, when importing
> the TLS connection's certificates into our XSecurityEnvironment. They
> now come in directly as ASN.1 DER, and still work.
>
> The code was greatly restructured and cleaned up, as Curl's API is
> synchronous and blocking, with parameters set in advance instead of
> through many callbacks, which has allowed using short clear methods,
> and clean separation between the session and request classes. The
> WebDAV content provider has shrunk from 35 to 21 C++ files, 43 to 29
> header files, and 19129 to 15991 lines of code. With WebDAV methods
> centralized and refactored into only 10-20 lines of code each, instead
> of scattered across 4 files, it is much more understandable and
> maintainable now.
>
> Curl is vastly more portable than Serf. We should build easily now
> even on OS/2. We can remain with existing build tools instead of
> needing scons or cmake just to build Serf.
>
> 3 now unused dependencies were removed: apr, apr-util, and serf. Serf
> isn't so bad. APR's pool idea is an ingenious way of doing resource
> management in C. However Curl has excellent documentation, guides,
> examples, and detailed explanations and even example code for each
> setting, while Serf has no documentation. Serf might be worth it in a
> project that already uses APR a lot, but we don't.
>
> Instead of the historical, crippled forms of logging like OSL_TRACE(),
> which don't appear in release builds, I've made it use the newer
> com.sun.star.logging UNO component (wrapped in
> comphelper::EventLogger), which was inspired by java.util.logging,
> with configurable verbosity levels, handlers (file and console) and
> output formats (plain, csv), and importantly, which produces output in
> release builds too. I've also made it so that on LogLevel::FINEST,
> Curl's verbose mode is enabled and Curl's debug output is also logged
> through us, with descriptions of what Curl is doing, and logs of all
> HTTP traffic including headers and bodies, before encryption and after
> decryption in the case of HTTPS, giving us tremendous detail that can
> be used for troubleshooting problems.
>
> CURL CHANGED TO USE OPENSSL AND ZLIB
>
> Curl only supports the custom TLS certificate verification function we
> use (the CURLOPT_SSL_CTX_FUNCTION option) when built with OpenSSL,
> wolfSSL or mbedTLS providers. We currently use schannel on Windows
> instead, which had to be changed. I also made it use zlib, which
> generally helps, and WebDAV uses XML which is very verbose and
> benefits from compression. On other OSes with system curl, it is now
> checked for its SSL provider, and configure fails if it isn't OpenSSL.
>
> The new WebDAV module successfully builds and runs with both OpenSSL
> 1.0.2 or 1.1.1. However 1 function was renamed between those versions,
> so the OpenSSL version at runtime probably has to match the one used
> at compile time (although building with 1.0.2 headers might allow
> running with 1.1.1 - not tested).
>
> (After completing development and testing, it dawned on me there is a
> completely different way to do the certification verification, which
> should allow other SSL providers to be used and might be better in
> various ways. See later.)
>
> We currently build zlib as a static library only, and on Windows its C
> runtime library is linked statically (_MT), which can't be mixed with
> curl's dynamic linking of the runtime library (_MD). Thus curl was
> made to link it statically too. Most if not all of our modules link
> the runtime library statically too (which begs the question, why do we
> ship the msvcr* redistributables to users at all then?).
>
> ISSUES
>
> The file open dialog (Ctrl+O) can hang for several minutes when first
> connecting to a server. This is not new - it happens with Serf as
> well. This appears to be caused by autocompletion in the file dialog.
> When typing in a URL like "davs://127.0.0.1 <http://127.0.0.1>", the
> WebDAV content provider is first called with a partial "https://1",
> before the rest of the URL is entered. That "1" is (somehow) treated
> as IP address "0.0.0.1", and a TCP connection to 0.0.0.1 is started.
> Only after several minutes, when that connection times out and fails,
> does the content provider get another request with the complete URL,
> which succeeds. The distinctly unpleasant wait, is luckily only
> present that the first time that server is used, as caching remembers
> the URL even across AOO restarts, so the "1" is automatically expanded
> to the full URL and the content provider never sees "https://1" again.
>
> You can only enter credentials for the HTTP(S) proxy or the
> destination, never both, as the credential manager caches credentials
> per URL, not per host/realm, so while you are prompted for credentials
> for both, and Curl is told about both, the destination credentials
> overwrite the proxy credentials in the cache, so one Curl request
> works but future Curl requests use the wrong credentials to the proxy.
> This doesn't matter, as AOO doesn't support passwords for proxy
> servers anyway, and Serf has the same issue.
>
> Damjan
>
> --
>
> P.S. APACHE 2.4 SETUP FOR TESTING
>
> Put some files in /var/tmp/dav/files, then configure Apache HTTPD's
> httpd.conf with something along these lines:
>
> Listen 127.0.0.1:8080 <http://127.0.0.1:8080>
> LoadModule dav_module libexec/apache24/mod_dav.so
> LoadModule dav_fs_module libexec/apache24/mod_dav_fs.so
>
> <VirtualHost 127.0.0.1:8080 <http://127.0.0.1:8080>>
> DocumentRoot "/var/tmp/dav"
> ErrorLog "/var/tmp/dav/errors.txt"
> TransferLog "/var/tmp/dav/access.txt"
> DavLockDB "/var/tmp/dav/DavLock"
>
> # If using TLS, generate these self-signed certificates too:
> # SSLEngine on
> # SSLCertificateFile "/var/tmp/dav/cert.pem"
> # SSLCertificateKeyFile "/var/tmp/dav/key.pem"
>
> <Directory "/var/tmp/dav/files">
> AllowOverride none
> Require valid-user
> # Require all granted
>
> AuthType Basic
> AuthName "WebDAV"
> AuthBasicProvider file
> AuthUserFile "/var/tmp/dav/passwords"
>
> Dav on
> </Directory>
> </VirtualHost>
>
> Enter your username and password with:
> htpasswd -c /var/tmp/dav/passwords username
>
> Then in the file open dialog, enter "dav://127.0.0.1:8080/files/
> <http://127.0.0.1:8080/files/>" (or use scheme davs:// instead, if
> using HTTPS).
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
> For additional commands, e-mail: dev-help@openoffice.apache.org
Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)
Posted by Damjan Jovanovic <da...@apache.org>.
Hi Pedro
That's great.
I think all you need to do to test on Windows is replace libcurl.dll and
ucpdav1.dll with the copies in the ZIP file you download from here:
https://drive.google.com/file/d/1YnpCr4woe8laah4nxtgzGfuEfGpVrf9L/view?usp=sharing
Regards
Damjan
On Tue, Apr 26, 2022 at 8:20 PM Pedro Lino <pe...@mailbox.org.invalid>
wrote:
>
> Hi Damian
>
>
>
> I use Webdav almost on a daily basis.
> Looking forward to testing a new build under Linux as soon as you push it
> to 42X
>
> Can you share your working binaries for Windows?
>
>
>
> Thanks!
>
> Pedro
> > On 04/26/2022 6:56 PM Damjan Jovanovic <da...@apache.org> wrote:
> >
> >
> >
> >
> >
> >
> > On Mon, Nov 15, 2021 at 9:57 PM Jim Jagielski <jim@jagunet.com <mailto:
> jim@jagunet.com>> wrote:
> > > I'm gonna look into the serf->(lib)curl option... Since we don't use
> any of the fancy features of serf, I'm thinking that the easy option might
> be best
> > >
> > >
> > > Hi
> > >
> > > I've ported our WebDAV content provider module from Serf to Curl.
> > >
> > > While it ended well, and several other bugs were found and fixed, it
> definitely wasn't the "easy option" Jim ;). Starting with conservative
> changes, it ended up needing total restructuring, and became more of a
> rewrite. The crashes were frequent and hung connections many, and I had to
> read up on the HTTP protocol, and read Curl and Serf's source code, but
> eventually I prevailed, and a clean elegant stable Curl WebDAV module
> emerged.
> > >
> > > The huge patch is attached for anyone wishing to review and test.
> Unless there are major objections, I'll push it in a couple of days.
> > >
> > > STATUS
> > >
> > > It builds and works well on FreeBSD and Windows.
> > >
> > > Most of the code was reused, and all the operations and semantics
> previously present with Serf, should have been preserved.
> > >
> > > Browsing WebDAV files and directories, loading files, overwriting them
> ("Save"), creating them ("Save As"), renaming and deleting them, all works.
> > >
> > > HTTP and HTTPS proxies work. Unlike Serf, Curl could also support
> SOCKS proxies (with minimal further changes), but AOO lacks that setting in
> the proxy UI and configuration.
> > >
> > > Authentication works, both to the destination server and to the proxy
> server. I've successfully tested Basic and Digest authentication. Curl
> supports every authentication method Serf does and more.
> > >
> > > HTTPS works, with a custom certificate verification function, using
> our own certificate store from NSS and its API (like the Serf code used). A
> bug was discovered (which is in the Serf implementation too) where
> self-signed certificates were being unconditionally rejected; apparently
> NSS wants to see that a copy of that certificate in its certificate chain
> parameter too. Now they work, and the user gets prompted to allow access.
> > >
> > >
> > > HTTPS and authentication can be used together on the same connection
> and work well, both bringing up their UI dialogs as needed.
> > >
> > > A bug was fixed where when username and password were both present in
> the URL (dav://user:pass@host/path), the code was trying to split them at
> the "@" instead of ":".
> > >
> > > Unnecessary base64 encoding and decoding was removed, when importing
> the TLS connection's certificates into our XSecurityEnvironment. They now
> come in directly as ASN.1 DER, and still work.
> > >
> > > The code was greatly restructured and cleaned up, as Curl's API is
> synchronous and blocking, with parameters set in advance instead of through
> many callbacks, which has allowed using short clear methods, and clean
> separation between the session and request classes. The WebDAV content
> provider has shrunk from 35 to 21 C++ files, 43 to 29 header files, and
> 19129 to 15991 lines of code. With WebDAV methods centralized and
> refactored into only 10-20 lines of code each, instead of scattered across
> 4 files, it is much more understandable and maintainable now.
> > >
> > > Curl is vastly more portable than Serf. We should build easily now
> even on OS/2. We can remain with existing build tools instead of needing
> scons or cmake just to build Serf.
> > >
> > > 3 now unused dependencies were removed: apr, apr-util, and serf. Serf
> isn't so bad. APR's pool idea is an ingenious way of doing resource
> management in C. However Curl has excellent documentation, guides,
> examples, and detailed explanations and even example code for each setting,
> while Serf has no documentation. Serf might be worth it in a project that
> already uses APR a lot, but we don't.
> > >
> > > Instead of the historical, crippled forms of logging like OSL_TRACE(),
> which don't appear in release builds, I've made it use the newer
> com.sun.star.logging UNO component (wrapped in comphelper::EventLogger),
> which was inspired by java.util.logging, with configurable verbosity
> levels, handlers (file and console) and output formats (plain, csv), and
> importantly, which produces output in release builds too. I've also made it
> so that on LogLevel::FINEST, Curl's verbose mode is enabled and Curl's
> debug output is also logged through us, with descriptions of what Curl is
> doing, and logs of all HTTP traffic including headers and bodies, before
> encryption and after decryption in the case of HTTPS, giving us tremendous
> detail that can be used for troubleshooting problems.
> > >
> > > CURL CHANGED TO USE OPENSSL AND ZLIB
> > >
> > > Curl only supports the custom TLS certificate verification function we
> use (the CURLOPT_SSL_CTX_FUNCTION option) when built with OpenSSL, wolfSSL
> or mbedTLS providers. We currently use schannel on Windows instead, which
> had to be changed. I also made it use zlib, which generally helps, and
> WebDAV uses XML which is very verbose and benefits from compression. On
> other OSes with system curl, it is now checked for its SSL provider, and
> configure fails if it isn't OpenSSL.
> > >
> > > The new WebDAV module successfully builds and runs with both OpenSSL
> 1.0.2 or 1.1.1. However 1 function was renamed between those versions, so
> the OpenSSL version at runtime probably has to match the one used at
> compile time (although building with 1.0.2 headers might allow running with
> 1.1.1 - not tested).
> > >
> > > (After completing development and testing, it dawned on me there is a
> completely different way to do the certification verification, which should
> allow other SSL providers to be used and might be better in various ways.
> See later.)
> > >
> > > We currently build zlib as a static library only, and on Windows its C
> runtime library is linked statically (_MT), which can't be mixed with
> curl's dynamic linking of the runtime library (_MD). Thus curl was made to
> link it statically too. Most if not all of our modules link the runtime
> library statically too (which begs the question, why do we ship the msvcr*
> redistributables to users at all then?).
> > >
> > > ISSUES
> > >
> > > The file open dialog (Ctrl+O) can hang for several minutes when first
> connecting to a server. This is not new - it happens with Serf as well.
> This appears to be caused by autocompletion in the file dialog. When typing
> in a URL like "davs://127.0.0.1 <http://127.0.0.1>", the WebDAV content
> provider is first called with a partial "https://1", before the rest of
> the URL is entered. That "1" is (somehow) treated as IP address "0.0.0.1",
> and a TCP connection to 0.0.0.1 is started. Only after several minutes,
> when that connection times out and fails, does the content provider get
> another request with the complete URL, which succeeds. The distinctly
> unpleasant wait, is luckily only present that the first time that server is
> used, as caching remembers the URL even across AOO restarts, so the "1" is
> automatically expanded to the full URL and the content provider never sees "
> https://1" again.
> > >
> > > You can only enter credentials for the HTTP(S) proxy or the
> destination, never both, as the credential manager caches credentials per
> URL, not per host/realm, so while you are prompted for credentials for
> both, and Curl is told about both, the destination credentials overwrite
> the proxy credentials in the cache, so one Curl request works but future
> Curl requests use the wrong credentials to the proxy. This doesn't matter,
> as AOO doesn't support passwords for proxy servers anyway, and Serf has the
> same issue.
> > >
> > > Damjan
> > >
> > > --
> > >
> > > P.S. APACHE 2.4 SETUP FOR TESTING
> > >
> > > Put some files in /var/tmp/dav/files, then configure Apache HTTPD's
> httpd.conf with something along these lines:
> > >
> > > Listen 127.0.0.1:8080 <http://127.0.0.1:8080>
> > > LoadModule dav_module libexec/apache24/mod_dav.so
> > > LoadModule dav_fs_module libexec/apache24/mod_dav_fs.so
> > >
> > > <VirtualHost 127.0.0.1:8080 <http://127.0.0.1:8080>>
> > > DocumentRoot "/var/tmp/dav"
> > > ErrorLog "/var/tmp/dav/errors.txt"
> > > TransferLog "/var/tmp/dav/access.txt"
> > > DavLockDB "/var/tmp/dav/DavLock"
> > >
> > > # If using TLS, generate these self-signed certificates too:
> > > # SSLEngine on
> > > # SSLCertificateFile "/var/tmp/dav/cert.pem"
> > > # SSLCertificateKeyFile "/var/tmp/dav/key.pem"
> > >
> > > <Directory "/var/tmp/dav/files">
> > > AllowOverride none
> > > Require valid-user
> > > # Require all granted
> > >
> > > AuthType Basic
> > > AuthName "WebDAV"
> > > AuthBasicProvider file
> > > AuthUserFile "/var/tmp/dav/passwords"
> > >
> > > Dav on
> > > </Directory>
> > > </VirtualHost>
> > >
> > > Enter your username and password with:
> > > htpasswd -c /var/tmp/dav/passwords username
> > >
> > > Then in the file open dialog, enter "dav://127.0.0.1:8080/files/ <
> http://127.0.0.1:8080/files/>" (or use scheme davs:// instead, if using
> HTTPS).
> > >
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
> > > For additional commands, e-mail: dev-help@openoffice.apache.org
Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)
Posted by Pedro Lino <pe...@mailbox.org.INVALID>.
Hi Damian
I use Webdav almost on a daily basis.
Looking forward to testing a new build under Linux as soon as you push it to 42X
Can you share your working binaries for Windows?
Thanks!
Pedro
> On 04/26/2022 6:56 PM Damjan Jovanovic <da...@apache.org> wrote:
>
>
>
>
>
>
> On Mon, Nov 15, 2021 at 9:57 PM Jim Jagielski <jim@jagunet.com <ma...@jagunet.com>> wrote:
> > I'm gonna look into the serf->(lib)curl option... Since we don't use any of the fancy features of serf, I'm thinking that the easy option might be best
> >
> >
> > Hi
> >
> > I've ported our WebDAV content provider module from Serf to Curl.
> >
> > While it ended well, and several other bugs were found and fixed, it definitely wasn't the "easy option" Jim ;). Starting with conservative changes, it ended up needing total restructuring, and became more of a rewrite. The crashes were frequent and hung connections many, and I had to read up on the HTTP protocol, and read Curl and Serf's source code, but eventually I prevailed, and a clean elegant stable Curl WebDAV module emerged.
> >
> > The huge patch is attached for anyone wishing to review and test. Unless there are major objections, I'll push it in a couple of days.
> >
> > STATUS
> >
> > It builds and works well on FreeBSD and Windows.
> >
> > Most of the code was reused, and all the operations and semantics previously present with Serf, should have been preserved.
> >
> > Browsing WebDAV files and directories, loading files, overwriting them ("Save"), creating them ("Save As"), renaming and deleting them, all works.
> >
> > HTTP and HTTPS proxies work. Unlike Serf, Curl could also support SOCKS proxies (with minimal further changes), but AOO lacks that setting in the proxy UI and configuration.
> >
> > Authentication works, both to the destination server and to the proxy server. I've successfully tested Basic and Digest authentication. Curl supports every authentication method Serf does and more.
> >
> > HTTPS works, with a custom certificate verification function, using our own certificate store from NSS and its API (like the Serf code used). A bug was discovered (which is in the Serf implementation too) where self-signed certificates were being unconditionally rejected; apparently NSS wants to see that a copy of that certificate in its certificate chain parameter too. Now they work, and the user gets prompted to allow access.
> >
> >
> > HTTPS and authentication can be used together on the same connection and work well, both bringing up their UI dialogs as needed.
> >
> > A bug was fixed where when username and password were both present in the URL (dav://user:pass@host/path), the code was trying to split them at the "@" instead of ":".
> >
> > Unnecessary base64 encoding and decoding was removed, when importing the TLS connection's certificates into our XSecurityEnvironment. They now come in directly as ASN.1 DER, and still work.
> >
> > The code was greatly restructured and cleaned up, as Curl's API is synchronous and blocking, with parameters set in advance instead of through many callbacks, which has allowed using short clear methods, and clean separation between the session and request classes. The WebDAV content provider has shrunk from 35 to 21 C++ files, 43 to 29 header files, and 19129 to 15991 lines of code. With WebDAV methods centralized and refactored into only 10-20 lines of code each, instead of scattered across 4 files, it is much more understandable and maintainable now.
> >
> > Curl is vastly more portable than Serf. We should build easily now even on OS/2. We can remain with existing build tools instead of needing scons or cmake just to build Serf.
> >
> > 3 now unused dependencies were removed: apr, apr-util, and serf. Serf isn't so bad. APR's pool idea is an ingenious way of doing resource management in C. However Curl has excellent documentation, guides, examples, and detailed explanations and even example code for each setting, while Serf has no documentation. Serf might be worth it in a project that already uses APR a lot, but we don't.
> >
> > Instead of the historical, crippled forms of logging like OSL_TRACE(), which don't appear in release builds, I've made it use the newer com.sun.star.logging UNO component (wrapped in comphelper::EventLogger), which was inspired by java.util.logging, with configurable verbosity levels, handlers (file and console) and output formats (plain, csv), and importantly, which produces output in release builds too. I've also made it so that on LogLevel::FINEST, Curl's verbose mode is enabled and Curl's debug output is also logged through us, with descriptions of what Curl is doing, and logs of all HTTP traffic including headers and bodies, before encryption and after decryption in the case of HTTPS, giving us tremendous detail that can be used for troubleshooting problems.
> >
> > CURL CHANGED TO USE OPENSSL AND ZLIB
> >
> > Curl only supports the custom TLS certificate verification function we use (the CURLOPT_SSL_CTX_FUNCTION option) when built with OpenSSL, wolfSSL or mbedTLS providers. We currently use schannel on Windows instead, which had to be changed. I also made it use zlib, which generally helps, and WebDAV uses XML which is very verbose and benefits from compression. On other OSes with system curl, it is now checked for its SSL provider, and configure fails if it isn't OpenSSL.
> >
> > The new WebDAV module successfully builds and runs with both OpenSSL 1.0.2 or 1.1.1. However 1 function was renamed between those versions, so the OpenSSL version at runtime probably has to match the one used at compile time (although building with 1.0.2 headers might allow running with 1.1.1 - not tested).
> >
> > (After completing development and testing, it dawned on me there is a completely different way to do the certification verification, which should allow other SSL providers to be used and might be better in various ways. See later.)
> >
> > We currently build zlib as a static library only, and on Windows its C runtime library is linked statically (_MT), which can't be mixed with curl's dynamic linking of the runtime library (_MD). Thus curl was made to link it statically too. Most if not all of our modules link the runtime library statically too (which begs the question, why do we ship the msvcr* redistributables to users at all then?).
> >
> > ISSUES
> >
> > The file open dialog (Ctrl+O) can hang for several minutes when first connecting to a server. This is not new - it happens with Serf as well. This appears to be caused by autocompletion in the file dialog. When typing in a URL like "davs://127.0.0.1 <http://127.0.0.1>", the WebDAV content provider is first called with a partial "https://1", before the rest of the URL is entered. That "1" is (somehow) treated as IP address "0.0.0.1", and a TCP connection to 0.0.0.1 is started. Only after several minutes, when that connection times out and fails, does the content provider get another request with the complete URL, which succeeds. The distinctly unpleasant wait, is luckily only present that the first time that server is used, as caching remembers the URL even across AOO restarts, so the "1" is automatically expanded to the full URL and the content provider never sees "https://1" again.
> >
> > You can only enter credentials for the HTTP(S) proxy or the destination, never both, as the credential manager caches credentials per URL, not per host/realm, so while you are prompted for credentials for both, and Curl is told about both, the destination credentials overwrite the proxy credentials in the cache, so one Curl request works but future Curl requests use the wrong credentials to the proxy. This doesn't matter, as AOO doesn't support passwords for proxy servers anyway, and Serf has the same issue.
> >
> > Damjan
> >
> > --
> >
> > P.S. APACHE 2.4 SETUP FOR TESTING
> >
> > Put some files in /var/tmp/dav/files, then configure Apache HTTPD's httpd.conf with something along these lines:
> >
> > Listen 127.0.0.1:8080 <http://127.0.0.1:8080>
> > LoadModule dav_module libexec/apache24/mod_dav.so
> > LoadModule dav_fs_module libexec/apache24/mod_dav_fs.so
> >
> > <VirtualHost 127.0.0.1:8080 <http://127.0.0.1:8080>>
> > DocumentRoot "/var/tmp/dav"
> > ErrorLog "/var/tmp/dav/errors.txt"
> > TransferLog "/var/tmp/dav/access.txt"
> > DavLockDB "/var/tmp/dav/DavLock"
> >
> > # If using TLS, generate these self-signed certificates too:
> > # SSLEngine on
> > # SSLCertificateFile "/var/tmp/dav/cert.pem"
> > # SSLCertificateKeyFile "/var/tmp/dav/key.pem"
> >
> > <Directory "/var/tmp/dav/files">
> > AllowOverride none
> > Require valid-user
> > # Require all granted
> >
> > AuthType Basic
> > AuthName "WebDAV"
> > AuthBasicProvider file
> > AuthUserFile "/var/tmp/dav/passwords"
> >
> > Dav on
> > </Directory>
> > </VirtualHost>
> >
> > Enter your username and password with:
> > htpasswd -c /var/tmp/dav/passwords username
> >
> > Then in the file open dialog, enter "dav://127.0.0.1:8080/files/ <http://127.0.0.1:8080/files/>" (or use scheme davs:// instead, if using HTTPS).
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
> > For additional commands, e-mail: dev-help@openoffice.apache.org
Re: WebDAV module ported from serf to curl; curl using openssl and zlib
Posted by Arrigo Marchiori <ar...@yahoo.it.INVALID>.
Hello Damjan, All,
I had a hard time restoring all the proper line breaks... this thread
is getting long! :-)
On Tue, Aug 23, 2022 at 03:22:07AM +0200, Damjan Jovanovic wrote:
> On Sat, May 28, 2022 at 6:54 PM Arrigo Marchiori <ar...@yahoo.it.invalid>
> wrote:
>
> > Hello Damjan, All,
> >
> > On Sat, May 28, 2022 at 03:22:53PM +0200, Damjan Jovanovic wrote:
> >
> > > On Sat, May 28, 2022 at 11:48 AM Arrigo Marchiori
> > <ar...@yahoo.it.invalid>
> > > wrote:
> > >
> > > > Hello Damjan,
> > > >
> > > > On Sat, May 28, 2022 at 11:12:45AM +0200, Damjan Jovanovic wrote:
> > > >
> > > > > On Sat, May 28, 2022 at 9:20 AM Arrigo Marchiori <ar...@yahoo.it.invalid> wrote:
> > > > >
> > > > > > Hello Damjan, all,
> > > > > >
> > > > > > On Fri, May 27, 2022 at 11:27:11PM +0200, Arrigo Marchiori wrote:
> > > > > >
> > > > > > > Hello,
> > > > > > >
> > > > > > > On Fri, May 27, 2022 at 09:46:51PM +0200, Arrigo Marchiori wrote:
> > > > > > >
> > > > > > > > Hello Damjan,
> > > > > > > >
> > > > > > > > On Sun, May 22, 2022 at 06:10:46PM +0200, Damjan Jovanovic wrote:
> > > > > > > >
> > > > > > > > > On Sun, May 22, 2022 at 2:43 PM Arrigo Marchiori <ar...@yahoo.it.invalid> wrote:
> > > > > > > > >
> > > > > > > > > > Hello Damjan, all,
> > > > > > > > > >
> > > > > > > > > > On Tue, Apr 26, 2022 at 07:56:22PM +0200, Damjan Jovanovic wrote:
> > > > > > > > > >
> > > > > > > > > > > On Mon, Nov 15, 2021 at 9:57 PM Jim Jagielski <ji...@jagunet.com> wrote:
> > > > > > > > > > >
> > > > > > > > > > > > I'm gonna look into the serf->(lib)curl option... Since we don't use any
> > > > > > > > > > > > of the fancy features of serf, I'm thinking that the easy option might be
> > > > > > > > > > > > best
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Hi
> > > > > > > > > > >
> > > > > > > > > > > I've ported our WebDAV content provider module from Serf to Curl.
> > > > > > > > > >
> > > > > > > > > > I just enhanced the error reporting a bit; I am finding a problem
> > > > > > > > > > under Linux and I do not really know how to assess it.
> > > > > > > > > >
> > > > > > > > > > The problem: if we build AOO on CentOS (that is our reference
> > > > > > > > > > platform) then Curl will look for CA certificates in
> > > > > > > > > > /etc/pki/tls/certs/ca-bundle.crt
> > > > > > > > > >
> > > > > > > > > > This will fail on openSUSE and probably on Ubuntu as well.
> > > > > > > > > >
> > > > > > > > > > It seems that the above path is set at configure time and embedded
> > > > > > > > > > into Curl's code as #define macros.
> > > > > > > > > >
> > > > > > > > > > Is there an ``official'' way to assess this? Like, can we depend on
> > > > > > > > > > NSS' certificate store as you wrote (quoted below)?
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Curl/OpenSSL have an enormous number of options and I am pretty sure it can
> > > > > > > > > be fixed, but first I need to understand where and how it's failing.
> > > > > > > > >
> > > > > > > > > We currently allow it to run with the default CA certificate path, do
> > > > > > > > > pre-verification on the server's certificate using those CA certificates,
> > > > > > > > > then call our SSL_VERIFY_PEER function where we override the verification
> > > > > > > > > result with the certificates from NSS.
> > > > > > > >
> > > > > > > > Apparently, it is failing before calling our SSL_VERIFY_PEER function.
> > > > > > > >
> > > > > > > > > If it's failing before reaching our SSL_VERIFY_PEER function, we should be
> > > > > > > > > able to use Curl's CURLOPT_CAINFO or CURLOPT_CAINFO_BLOB functions to set a
> > > > > > > > > custom CA certificate path (or in-memory buffer), maybe even an empty
> > > > > > > > > buffer, so that it proceeds further. ("man CURLOPT_CAINFO", "man
> > > > > > > > > CURLOPT_CAINFO_BLOB", or "man curl_easy_setopt" and read under the "SSL and
> > > > > > > > > SECURITY OPTIONS" section.)
> > > > > > > >
> > > > > > > > So we would need to hard-code and try all possible paths to the CA
> > > > > > > > bundle on Unix systems?
> > > > > > > >
> > > > > > > > > With the CURLOPT_CAINFO_BLOB option it might even be possible to skip the
> > > > > > > > > custom certificate verification we do later, and pre-populate Curl/OpenSSL
> > > > > > > > > with NSS certificates from the beginning, I just don't know enough about
> > > > > > > > > NSS to rely on that (eg. if you are using a cryptographic device or smart
> > > > > > > > > card in NSS, how does that work?). If that option is ok, then we might not
> > > > > > > > > even need the NSS libraries: recent versions of NSS store all the
> > > > > > > > > certificates in an SQLite database, which can be accessed with SQLite APIs
> > > > > > > > > directly, no need to build with or ship the NSS libraries at all.
> > > > > > > >
> > > > > > > > If I understood correctly [1], a NSS-linked Curl would query NSS by
> > > > > > > > itself... are we not in this condition?
> > > > > > [...]
> > > > > > > 1: https://curl.se/libcurl/c/CURLOPT_CAINFO.html
> > > > > >
> > > > > > Apparently, we are not. If we force CURLOPT_CAINFO and CURLOPT_CAPATH
> > > > > > to be NULL, we get a bit further but eventually Curl aborts because
> > > > > > validateServerX509Certificate fails.
> > > > > >
> > > > > > Here is the log for checking AOO updates (please excuse the long
> > > > > > lines). I added an extra bit of logging when entering and leaving
> > > > > > OPENSSL_ValidateServerCertificate, that is ``our SSL_VERIFY_PEER
> > > > > > function''.
> > > > > >
> > > > > > ----8<--------8<--------8<--------8<--------8<--------8<--------8<---------
> > > > > >
> > > > > > 1 9 2022-05-28 07:13:52.61 CurlSession::CurlSession with URL https://ooo-updates.apache.org/aoonext/check.Update?pkgfmt=installed
> > > > > > 2 9 2022-05-28 07:13:52.61 Not using a proxy server
> > > > > > 3 9 2022-05-28 07:13:52.61 GET line 1093
> > > > > > 4 9 2022-05-28 07:13:52.61 [CurlINFO ] Trying 151.101.2.132:443...
> > > > > > 5 9 2022-05-28 07:13:52.63 [CurlINFO ] Connected to ooo-updates.apache.org (151.101.2.132) port 443 (#0)
> > > > > > 6 9 2022-05-28 07:13:52.63 [CurlINFO ] ALPN, offering http/1.1
> > > > > > 7 9 2022-05-28 07:13:52.63 [CurlINFO ] Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
> > > > > > 8 9 2022-05-28 07:13:52.63 [CurlINFO ] TLSv1.2 (OUT), TLS handshake, Client hello (1):
> > > > > > 9 9 2022-05-28 07:13:52.64 [CurlINFO ] TLSv1.2 (IN), TLS handshake, Server hello (2):
> > > > > > 10 9 2022-05-28 07:13:52.64 [CurlINFO ] TLSv1.2 (IN), TLS handshake, Certificate (11):
> > > > > > --here we are inside OPENSSL_ValidateServerCertificate
> > > > > > 11 9 2022-05-28 07:13:52.64 validateServerX509Certificate() returning 0 at depth 2
> > > > > > --here we are leaving OPENSSL_ValidateServerCertificate
> > > > > > 12 9 2022-05-28 07:13:52.64 [CurlINFO ] TLSv1.2 (OUT), TLS alert, unknown CA (560):
> > > > > > 13 9 2022-05-28 07:13:52.64 [CurlINFO ] SSL certificate problem: unable to get local issuer certificate
> > > > > > 14 9 2022-05-28 07:13:52.64 [CurlINFO ] Closing connection 0
> > > > > > 15 9 2022-05-28 07:13:52.64 Curl request failed with CURLcode 60
> > > > > > 16 9 2022-05-28 07:13:52.64 CurlSession::~CurlSession: closed curl session
> > > > > >
> > > > > > ----8<--------8<--------8<--------8<--------8<--------8<--------8<---------
> > > > > >
> > > > > > I hope this helps.
> > > > > >
> > > > > >
> > > > > I think the problem there is NSS's validation.
> > > > >
> > > > > In CurlSession::validateServerX509Certificate(), try change the "if (depth == 0 )" to "if (1)".
> > > >
> > > > After this edit, the connection is established. The log is below.
> > > >
> > >
> > > That's good :).
> > >
> > >
> > > >
> > > > But NSS was also used before, with Serf... it stopped working when the
> > > > bundled CA certificates started to be ``too obsolete''. After we
> > > > updated the bundled NSS, then https connections, such as the check for
> > > > updates, started to work again.
> > > >
> > >
> > > We don't need NSS specifically. From Curl, we need OpenSSL, because without
> > > it, we are unable to ask the user to confirm self-signed certificates
> > > during TLS negotiations [1]. Upstream OpenSSL however doesn't ship any CA
> > > certificates, so the WebDAV provider gets them from our
> > > com.sun.star.security.CertificateContainer, which comes from WinCrypt on
> > > Windows (also called "mscrypt" by main/xmlsecurity/source/xmlsec/mscrypt)
> > > and NSS elsewhere.
> > >
> > > Therefore that update check should not have been failing on Windows. Only
> > > if your Windows is out of date, should WinCrypt CA certificates fail.
> >
> > Honestly, I do not really recall what happened on Windows.
> > I do recall that Linux builds were unable to check for update.
> > I thought that the reason was an outdated Serf library, and I proposed
> > an update, with the addition of SCons etc.
> > In parallel, we updated the version of the bundled NSS. When we did
> > _this_, the updates started to work again.
> > We never merged the serf update, so I think it makes sense to consider
> > the NSS obsolescence related.
> > We also tried to upgrade OpenSSL, but then stepped back. So NSS
> > remains the most probable cause.
> >
> > > It is also particularly strange for NSS certificates to be out of date. The
> > > NSS certificate database that our
> > > com.sun.star.security.CertificateContainer uses comes from a profile
> > > (getMozillaCurrentProfile() in
> > > main/xmlsecurity/source/nss/nssinitializer.cxx) which (AFAIK) is in the
> > > user's home directory, and shared with and updated by Firefox or
> > > Thunderbird. Only if you don't update AOO, and don't run Firefox, and don't
> > > run Thunderbird, can your NSS certificates get out of date.
> >
> > I do not know how to reply here. I can just share my findings from
> > above...
> >
> > > > Please also bear in mind that I am now testing a build from a _CentOS
> > > > Docker container_. My computer is not running CentOS, but rather
> > > > OpenSUSE. The binary is run on ``native'' OpenSUSE. If I recall
> > > > correctly, when I build on the same native OpenSUSE, the resulting AOO
> > > > binary connects properly to https web sites.
> > > >
> > > > For this reason, I was wondering if this problem is not just about
> > > > hardcoded paths to CA certificates.
> > > >
> > >
> > > Since Curl's OpenSSL CA path is hardcoded to a non-portable value at
> > > compile time, my proposed approach here disables OpenSSL's certificates and
> > > only uses the ones from our com.sun.star.security.CertificateContainer
> > > instead. (And, who knows, it may be helpful to make another
> > > com.sun.star.security.CertificateContainer that gets certificates from
> > > guessed system paths for all of AOO, not just WebDAV.)
> > >
> > > Is there something else you propose instead?
> >
> > I am not sure my emails are communicating the proper ``tone'', so
> > please let me state it explicitly: I am neither a cryptography, SSL or
> > NSS expert at all. You seem to be much more knowledgeable than me and
> > I am thankful to see your contributions. I am just doing my best to
> > validate them in all possible cases (mostly under different Linux
> > distributions).
> >
> Sorry if my tone sounded negative, I didn't mean that.
Your tone sounded perfect; I just wanted to point out that I am a bit
struggling to follow this thread as it is getting deep into code I do
not know well.
> My knowledge of cryptography/SSL/NSS could also be better.
>
> I've also been trying to get our cryptography working on many Linux
> distributions.
Praiseworthy, IMHO!
> > I am trying to give useful advice, such as the above comments about
> > NSS. I might just be wrong, in such case I apologize for the noise.
> >
> > Having said this... what do you mean exactly for your ``proposed
> > approach''? Is it the patch from "if (depth == 0)" to "if (1)"?
> >
> >
> That any-depth patch is probably needed no matter what else we do, because
> multiple certificates in the chain may need user confirmation. I've now
> committed it as c464040a4409a7ab63c22a7b2358ce0134c09c10.
>
> No, my "proposed approach" was: instead of OpenSSL using the certificate
> path hardcoded at compile-time, get it to use our certificate store, which
> is populated from NSS's certificates at present but could be populated by
> the Linux distro's certificates.
This is also IMHO the way we should go.
> > > [1] But is Curl's inability to use custom certificate verification
> > > functions (which can ask the user to confirm problematic certificates) with
> > > NSS a limitation of Curl, or a limitation of NSS? Firefox uses NSS, and it
> > > certainly asks the user to confirm expired and self-signed certificates.
> > > How does Firefox do that, by disconnecting, asking the user, and
> > > reconnecting if confirmed, or the Curl/OpenSSL way, by keeping the TCP
> > > connection open during the user interaction and then resuming TLS
> > > negotiations if confirmed?
> >
> > For what it's worth, I tried opening https://developer.berlios.de ,
> > which has an invalid SSL certificate. Firefox 91.9.1esr on Linux
> > x86_64 closes the TCP connection immediately and remains waiting for
> > user input.
> >
> > I hope this helps.
> >
> >
> Thank you. My Firefox does the same on sites with bad certificates. It's
> possible to patch our WebDAV module to do that too (disconnect, add the
> user-approved certificate to our certificate store, then reconnect), and
> would simplify some of the code and fix the hardcoded path problem. But I
> wonder whether that would fix expired certificates and other problems, or
> just allow self-signed but otherwise valid certificates. I'll give that
> some further thought.
Once we give users the option to access one kind of invalid
certificates, we should allow them to accept all kinds, IMHO.
> As for accessing Linux distro certificates, the p11-kit project existed
> since 2011, and comes with a CLI tool, "trust", which can examine system
> certificates on any Linux distribution (well, any that ships it, and it's a
> dependency for the widely used gcr package used by the gvfs package used by
> at least GNOME and XFCE, not sure about KDE). If you run "trust extract
> --format=openssl-bundle --purpose=server-auth filename", then that
> "filename" gets an OpenSSL-compatible certificate bundle, that can be
> imported into OpenSSL and used - portably. This could be done by AOO as
> needed. Because Linux distributions supply and maintain both p11-kit and
> the certificates, we shouldn't need NSS if we use it,
Now _this_ seems to be interesting! That's the cross-distribution tool
we need!
> although NSS might still be required for client certificates for
> signing documents (not sure how that works yet).
The xmlsecurity has nss and mscrypt alternative implementation of some
of its interfaces. It _seems_ that the nss implementation only uses
the NSS library for managing certificates (storage etc.) and for the
actual encryption/decryption/signature it calls libxmlsec functions.
But libxmlsec is compiled against nss... this looks complicated.
But did we already mention p11-glue?
https://p11-glue.github.io/p11-glue/p11-kit/manual/trust-nss.html
This could allow us to adopt p11-kit without changing (too much) our
current interface to NSS... what do you think?
Best regards,
--
Arrigo
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)
Posted by Damjan Jovanovic <da...@apache.org>.
On Sat, May 28, 2022 at 6:54 PM Arrigo Marchiori <ar...@yahoo.it.invalid>
wrote:
> Hello Damjan, All,
>
> On Sat, May 28, 2022 at 03:22:53PM +0200, Damjan Jovanovic wrote:
>
> > On Sat, May 28, 2022 at 11:48 AM Arrigo Marchiori
> <ar...@yahoo.it.invalid>
> > wrote:
> >
> > > Hello Damjan,
> > >
> > > On Sat, May 28, 2022 at 11:12:45AM +0200, Damjan Jovanovic wrote:
> > >
> > > > On Sat, May 28, 2022 at 9:20 AM Arrigo Marchiori
> <ardovm@yahoo.it.invalid
> > > >
> > > > wrote:
> > > >
> > > > > Hello Damjan, all,
> > > > >
> > > > > On Fri, May 27, 2022 at 11:27:11PM +0200, Arrigo Marchiori wrote:
> > > > >
> > > > > > Hello,
> > > > > >
> > > > > > On Fri, May 27, 2022 at 09:46:51PM +0200, Arrigo Marchiori wrote:
> > > > > >
> > > > > > > Hello Damjan,
> > > > > > >
> > > > > > > On Sun, May 22, 2022 at 06:10:46PM +0200, Damjan Jovanovic
> wrote:
> > > > > > >
> > > > > > > > On Sun, May 22, 2022 at 2:43 PM Arrigo Marchiori
> > > > > <ar...@yahoo.it.invalid>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hello Damjan, all,
> > > > > > > > >
> > > > > > > > > On Tue, Apr 26, 2022 at 07:56:22PM +0200, Damjan Jovanovic
> > > wrote:
> > > > > > > > >
> > > > > > > > > > On Mon, Nov 15, 2021 at 9:57 PM Jim Jagielski <
> > > jim@jagunet.com>
> > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > I'm gonna look into the serf->(lib)curl option...
> Since we
> > > > > don't use
> > > > > > > > > any
> > > > > > > > > > > of the fancy features of serf, I'm thinking that the
> easy
> > > > > option might
> > > > > > > > > be
> > > > > > > > > > > best
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Hi
> > > > > > > > > >
> > > > > > > > > > I've ported our WebDAV content provider module from Serf
> to
> > > Curl.
> > > > > > > > >
> > > > > > > > > I just enhanced the error reporting a bit; I am finding a
> > > problem
> > > > > > > > > under Linux and I do not really know how to assess it.
> > > > > > > > >
> > > > > > > > > The problem: if we build AOO on CentOS (that is our
> reference
> > > > > > > > > platform) then Curl will look for CA certificates in
> > > > > > > > > /etc/pki/tls/certs/ca-bundle.crt
> > > > > > > > >
> > > > > > > > > This will fail on openSUSE and probably on Ubuntu as well.
> > > > > > > > >
> > > > > > > > > It seems that the above path is set at configure time and
> > > embedded
> > > > > > > > > into Curl's code as #define macros.
> > > > > > > > >
> > > > > > > > > Is there an ``official'' way to assess this? Like, can we
> > > depend on
> > > > > > > > > NSS' certificate store as you wrote (quoted below)?
> > > > > > > > >
> > > > > > > >
> > > > > > > > Curl/OpenSSL have an enormous number of options and I am
> pretty
> > > sure
> > > > > it can
> > > > > > > > be fixed, but first I need to understand where and how it's
> > > failing.
> > > > > > > >
> > > > > > > > We currently allow it to run with the default CA certificate
> > > path, do
> > > > > > > > pre-verification on the server's certificate using those CA
> > > > > certificates,
> > > > > > > > then call our SSL_VERIFY_PEER function where we override the
> > > > > verification
> > > > > > > > result with the certificates from NSS.
> > > > > > >
> > > > > > > Apparently, it is failing before calling our SSL_VERIFY_PEER
> > > function.
> > > > > > >
> > > > > > > > If it's failing before reaching our SSL_VERIFY_PEER
> function, we
> > > > > should be
> > > > > > > > able to use Curl's CURLOPT_CAINFO or CURLOPT_CAINFO_BLOB
> > > functions
> > > > > to set a
> > > > > > > > custom CA certificate path (or in-memory buffer), maybe even
> an
> > > empty
> > > > > > > > buffer, so that it proceeds further. ("man CURLOPT_CAINFO",
> "man
> > > > > > > > CURLOPT_CAINFO_BLOB", or "man curl_easy_setopt" and read
> under
> > > the
> > > > > "SSL and
> > > > > > > > SECURITY OPTIONS" section.)
> > > > > > >
> > > > > > > So we would need to hard-code and try all possible paths to
> the CA
> > > > > > > bundle on Unix systems?
> > > > > > >
> > > > > > > > With the CURLOPT_CAINFO_BLOB option it might even be
> possible to
> > > > > skip the
> > > > > > > > custom certificate verification we do later, and pre-populate
> > > > > Curl/OpenSSL
> > > > > > > > with NSS certificates from the beginning, I just don't know
> > > enough
> > > > > about
> > > > > > > > NSS to rely on that (eg. if you are using a cryptographic
> device
> > > or
> > > > > smart
> > > > > > > > card in NSS, how does that work?). If that option is ok,
> then we
> > > > > might not
> > > > > > > > even need the NSS libraries: recent versions of NSS store
> all the
> > > > > > > > certificates in an SQLite database, which can be accessed
> with
> > > > > SQLite APIs
> > > > > > > > directly, no need to build with or ship the NSS libraries at
> all.
> > > > > > >
> > > > > > > If I understood correctly [1], a NSS-linked Curl would query
> NSS by
> > > > > > > itself... are we not in this condition?
> > > > > [...]
> > > > > > 1: https://curl.se/libcurl/c/CURLOPT_CAINFO.html
> > > > >
> > > > > Apparently, we are not. If we force CURLOPT_CAINFO and
> CURLOPT_CAPATH
> > > > > to be NULL, we get a bit further but eventually Curl aborts because
> > > > > validateServerX509Certificate fails.
> > > > >
> > > > > Here is the log for checking AOO updates (please excuse the long
> > > > > lines). I added an extra bit of logging when entering and leaving
> > > > > OPENSSL_ValidateServerCertificate, that is ``our SSL_VERIFY_PEER
> > > > > function''.
> > > > >
> > > > >
> > >
> ----8<--------8<--------8<--------8<--------8<--------8<--------8<---------
> > > > >
> > > > > 1 9 2022-05-28 07:13:52.61 CurlSession::CurlSession
> > > with
> > > > > URL
> > > https://ooo-updates.apache.org/aoonext/check.Update?pkgfmt=installed
> > > > > 2 9 2022-05-28 07:13:52.61 Not using a proxy server
> > > > > 3 9 2022-05-28 07:13:52.61 GET line 1093
> > > > > 4 9 2022-05-28 07:13:52.61 [CurlINFO ] Trying
> > > > > 151.101.2.132:443...
> > > > > 5 9 2022-05-28 07:13:52.63 [CurlINFO ] Connected
> to
> > > > > ooo-updates.apache.org (151.101.2.132) port 443 (#0)
> > > > > 6 9 2022-05-28 07:13:52.63 [CurlINFO ] ALPN,
> offering
> > > > > http/1.1
> > > > > 7 9 2022-05-28 07:13:52.63 [CurlINFO ] Cipher
> > > selection:
> > > > > ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
> > > > > 8 9 2022-05-28 07:13:52.63 [CurlINFO ] TLSv1.2
> (OUT),
> > > TLS
> > > > > handshake, Client hello (1):
> > > > > 9 9 2022-05-28 07:13:52.64 [CurlINFO ] TLSv1.2
> (IN),
> > > TLS
> > > > > handshake, Server hello (2):
> > > > > 10 9 2022-05-28 07:13:52.64 [CurlINFO ] TLSv1.2
> (IN),
> > > TLS
> > > > > handshake, Certificate (11):
> > > > > --here we are inside OPENSSL_ValidateServerCertificate
> > > > > 11 9 2022-05-28 07:13:52.64
> > > validateServerX509Certificate()
> > > > > returning 0 at depth 2
> > > > > --here we are leaving OPENSSL_ValidateServerCertificate
> > > > > 12 9 2022-05-28 07:13:52.64 [CurlINFO ] TLSv1.2
> (OUT),
> > > TLS
> > > > > alert, unknown CA (560):
> > > > > 13 9 2022-05-28 07:13:52.64 [CurlINFO ] SSL
> certificate
> > > > > problem: unable to get local issuer certificate
> > > > > 14 9 2022-05-28 07:13:52.64 [CurlINFO ] Closing
> > > connection
> > > > > 0
> > > > > 15 9 2022-05-28 07:13:52.64 Curl request failed with
> > > > > CURLcode 60
> > > > > 16 9 2022-05-28 07:13:52.64
> CurlSession::~CurlSession:
> > > > > closed curl session
> > > > >
> > > > >
> > >
> ----8<--------8<--------8<--------8<--------8<--------8<--------8<---------
> > > > >
> > > > > I hope this helps.
> > > > >
> > > > >
> > > > I think the problem there is NSS's validation.
> > > >
> > > > In CurlSession::validateServerX509Certificate(), try change the "if (
> > > depth
> > > > == 0 )" to "if (1)".
> > >
> > > After this edit, the connection is established. The log is below.
> > >
> >
> > That's good :).
> >
> >
> > >
> > > But NSS was also used before, with Serf... it stopped working when the
> > > bundled CA certificates started to be ``too obsolete''. After we
> > > updated the bundled NSS, then https connections, such as the check for
> > > updates, started to work again.
> > >
> >
> > We don't need NSS specifically. From Curl, we need OpenSSL, because
> without
> > it, we are unable to ask the user to confirm self-signed certificates
> > during TLS negotiations [1]. Upstream OpenSSL however doesn't ship any CA
> > certificates, so the WebDAV provider gets them from our
> > com.sun.star.security.CertificateContainer, which comes from WinCrypt on
> > Windows (also called "mscrypt" by main/xmlsecurity/source/xmlsec/mscrypt)
> > and NSS elsewhere.
> >
> > Therefore that update check should not have been failing on Windows. Only
> > if your Windows is out of date, should WinCrypt CA certificates fail.
>
> Honestly, I do not really recall what happened on Windows.
> I do recall that Linux builds were unable to check for update.
> I thought that the reason was an outdated Serf library, and I proposed
> an update, with the addition of SCons etc.
> In parallel, we updated the version of the bundled NSS. When we did
> _this_, the updates started to work again.
> We never merged the serf update, so I think it makes sense to consider
> the NSS obsolescence related.
> We also tried to upgrade OpenSSL, but then stepped back. So NSS
> remains the most probable cause.
>
> > It is also particularly strange for NSS certificates to be out of date.
> The
> > NSS certificate database that our
> > com.sun.star.security.CertificateContainer uses comes from a profile
> > (getMozillaCurrentProfile() in
> > main/xmlsecurity/source/nss/nssinitializer.cxx) which (AFAIK) is in the
> > user's home directory, and shared with and updated by Firefox or
> > Thunderbird. Only if you don't update AOO, and don't run Firefox, and
> don't
> > run Thunderbird, can your NSS certificates get out of date.
>
> I do not know how to reply here. I can just share my findings from
> above...
>
> > > Please also bear in mind that I am now testing a build from a _CentOS
> > > Docker container_. My computer is not running CentOS, but rather
> > > OpenSUSE. The binary is run on ``native'' OpenSUSE. If I recall
> > > correctly, when I build on the same native OpenSUSE, the resulting AOO
> > > binary connects properly to https web sites.
> > >
> > > For this reason, I was wondering if this problem is not just about
> > > hardcoded paths to CA certificates.
> > >
> >
> > Since Curl's OpenSSL CA path is hardcoded to a non-portable value at
> > compile time, my proposed approach here disables OpenSSL's certificates
> and
> > only uses the ones from our com.sun.star.security.CertificateContainer
> > instead. (And, who knows, it may be helpful to make another
> > com.sun.star.security.CertificateContainer that gets certificates from
> > guessed system paths for all of AOO, not just WebDAV.)
> >
> > Is there something else you propose instead?
>
> I am not sure my emails are communicating the proper ``tone'', so
> please let me state it explicitly: I am neither a cryptography, SSL or
> NSS expert at all. You seem to be much more knowledgeable than me and
> I am thankful to see your contributions. I am just doing my best to
> validate them in all possible cases (mostly under different Linux
> distributions).
>
>
Sorry if my tone sounded negative, I didn't mean that.
My knowledge of cryptography/SSL/NSS could also be better.
I've also been trying to get our cryptography working on many Linux
distributions.
> I am trying to give useful advice, such as the above comments about
> NSS. I might just be wrong, in such case I apologize for the noise.
>
> Having said this... what do you mean exactly for your ``proposed
> approach''? Is it the patch from "if (depth == 0)" to "if (1)"?
>
>
That any-depth patch is probably needed no matter what else we do, because
multiple certificates in the chain may need user confirmation. I've now
committed it as c464040a4409a7ab63c22a7b2358ce0134c09c10.
No, my "proposed approach" was: instead of OpenSSL using the certificate
path hardcoded at compile-time, get it to use our certificate store, which
is populated from NSS's certificates at present but could be populated by
the Linux distro's certificates.
> > [1] But is Curl's inability to use custom certificate verification
> > functions (which can ask the user to confirm problematic certificates)
> with
> > NSS a limitation of Curl, or a limitation of NSS? Firefox uses NSS, and
> it
> > certainly asks the user to confirm expired and self-signed certificates.
> > How does Firefox do that, by disconnecting, asking the user, and
> > reconnecting if confirmed, or the Curl/OpenSSL way, by keeping the TCP
> > connection open during the user interaction and then resuming TLS
> > negotiations if confirmed?
>
> For what it's worth, I tried opening https://developer.berlios.de ,
> which has an invalid SSL certificate. Firefox 91.9.1esr on Linux
> x86_64 closes the TCP connection immediately and remains waiting for
> user input.
>
> I hope this helps.
>
>
Thank you. My Firefox does the same on sites with bad certificates. It's
possible to patch our WebDAV module to do that too (disconnect, add the
user-approved certificate to our certificate store, then reconnect), and
would simplify some of the code and fix the hardcoded path problem. But I
wonder whether that would fix expired certificates and other problems, or
just allow self-signed but otherwise valid certificates. I'll give that
some further thought.
As for accessing Linux distro certificates, the p11-kit project existed
since 2011, and comes with a CLI tool, "trust", which can examine system
certificates on any Linux distribution (well, any that ships it, and it's a
dependency for the widely used gcr package used by the gvfs package used by
at least GNOME and XFCE, not sure about KDE). If you run "trust extract
--format=openssl-bundle --purpose=server-auth filename", then that
"filename" gets an OpenSSL-compatible certificate bundle, that can be
imported into OpenSSL and used - portably. This could be done by AOO as
needed. Because Linux distributions supply and maintain both p11-kit and
the certificates, we shouldn't need NSS if we use it, although NSS might
still be required for client certificates for signing documents (not sure
how that works yet).
> Best regards,
> --
> Arrigo
>
>
Regards
Damjan
Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)
Posted by Arrigo Marchiori <ar...@yahoo.it.INVALID>.
Hello Damjan, All,
On Sat, May 28, 2022 at 03:22:53PM +0200, Damjan Jovanovic wrote:
> On Sat, May 28, 2022 at 11:48 AM Arrigo Marchiori <ar...@yahoo.it.invalid>
> wrote:
>
> > Hello Damjan,
> >
> > On Sat, May 28, 2022 at 11:12:45AM +0200, Damjan Jovanovic wrote:
> >
> > > On Sat, May 28, 2022 at 9:20 AM Arrigo Marchiori <ardovm@yahoo.it.invalid
> > >
> > > wrote:
> > >
> > > > Hello Damjan, all,
> > > >
> > > > On Fri, May 27, 2022 at 11:27:11PM +0200, Arrigo Marchiori wrote:
> > > >
> > > > > Hello,
> > > > >
> > > > > On Fri, May 27, 2022 at 09:46:51PM +0200, Arrigo Marchiori wrote:
> > > > >
> > > > > > Hello Damjan,
> > > > > >
> > > > > > On Sun, May 22, 2022 at 06:10:46PM +0200, Damjan Jovanovic wrote:
> > > > > >
> > > > > > > On Sun, May 22, 2022 at 2:43 PM Arrigo Marchiori
> > > > <ar...@yahoo.it.invalid>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hello Damjan, all,
> > > > > > > >
> > > > > > > > On Tue, Apr 26, 2022 at 07:56:22PM +0200, Damjan Jovanovic
> > wrote:
> > > > > > > >
> > > > > > > > > On Mon, Nov 15, 2021 at 9:57 PM Jim Jagielski <
> > jim@jagunet.com>
> > > > wrote:
> > > > > > > > >
> > > > > > > > > > I'm gonna look into the serf->(lib)curl option... Since we
> > > > don't use
> > > > > > > > any
> > > > > > > > > > of the fancy features of serf, I'm thinking that the easy
> > > > option might
> > > > > > > > be
> > > > > > > > > > best
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Hi
> > > > > > > > >
> > > > > > > > > I've ported our WebDAV content provider module from Serf to
> > Curl.
> > > > > > > >
> > > > > > > > I just enhanced the error reporting a bit; I am finding a
> > problem
> > > > > > > > under Linux and I do not really know how to assess it.
> > > > > > > >
> > > > > > > > The problem: if we build AOO on CentOS (that is our reference
> > > > > > > > platform) then Curl will look for CA certificates in
> > > > > > > > /etc/pki/tls/certs/ca-bundle.crt
> > > > > > > >
> > > > > > > > This will fail on openSUSE and probably on Ubuntu as well.
> > > > > > > >
> > > > > > > > It seems that the above path is set at configure time and
> > embedded
> > > > > > > > into Curl's code as #define macros.
> > > > > > > >
> > > > > > > > Is there an ``official'' way to assess this? Like, can we
> > depend on
> > > > > > > > NSS' certificate store as you wrote (quoted below)?
> > > > > > > >
> > > > > > >
> > > > > > > Curl/OpenSSL have an enormous number of options and I am pretty
> > sure
> > > > it can
> > > > > > > be fixed, but first I need to understand where and how it's
> > failing.
> > > > > > >
> > > > > > > We currently allow it to run with the default CA certificate
> > path, do
> > > > > > > pre-verification on the server's certificate using those CA
> > > > certificates,
> > > > > > > then call our SSL_VERIFY_PEER function where we override the
> > > > verification
> > > > > > > result with the certificates from NSS.
> > > > > >
> > > > > > Apparently, it is failing before calling our SSL_VERIFY_PEER
> > function.
> > > > > >
> > > > > > > If it's failing before reaching our SSL_VERIFY_PEER function, we
> > > > should be
> > > > > > > able to use Curl's CURLOPT_CAINFO or CURLOPT_CAINFO_BLOB
> > functions
> > > > to set a
> > > > > > > custom CA certificate path (or in-memory buffer), maybe even an
> > empty
> > > > > > > buffer, so that it proceeds further. ("man CURLOPT_CAINFO", "man
> > > > > > > CURLOPT_CAINFO_BLOB", or "man curl_easy_setopt" and read under
> > the
> > > > "SSL and
> > > > > > > SECURITY OPTIONS" section.)
> > > > > >
> > > > > > So we would need to hard-code and try all possible paths to the CA
> > > > > > bundle on Unix systems?
> > > > > >
> > > > > > > With the CURLOPT_CAINFO_BLOB option it might even be possible to
> > > > skip the
> > > > > > > custom certificate verification we do later, and pre-populate
> > > > Curl/OpenSSL
> > > > > > > with NSS certificates from the beginning, I just don't know
> > enough
> > > > about
> > > > > > > NSS to rely on that (eg. if you are using a cryptographic device
> > or
> > > > smart
> > > > > > > card in NSS, how does that work?). If that option is ok, then we
> > > > might not
> > > > > > > even need the NSS libraries: recent versions of NSS store all the
> > > > > > > certificates in an SQLite database, which can be accessed with
> > > > SQLite APIs
> > > > > > > directly, no need to build with or ship the NSS libraries at all.
> > > > > >
> > > > > > If I understood correctly [1], a NSS-linked Curl would query NSS by
> > > > > > itself... are we not in this condition?
> > > > [...]
> > > > > 1: https://curl.se/libcurl/c/CURLOPT_CAINFO.html
> > > >
> > > > Apparently, we are not. If we force CURLOPT_CAINFO and CURLOPT_CAPATH
> > > > to be NULL, we get a bit further but eventually Curl aborts because
> > > > validateServerX509Certificate fails.
> > > >
> > > > Here is the log for checking AOO updates (please excuse the long
> > > > lines). I added an extra bit of logging when entering and leaving
> > > > OPENSSL_ValidateServerCertificate, that is ``our SSL_VERIFY_PEER
> > > > function''.
> > > >
> > > >
> > ----8<--------8<--------8<--------8<--------8<--------8<--------8<---------
> > > >
> > > > 1 9 2022-05-28 07:13:52.61 CurlSession::CurlSession
> > with
> > > > URL
> > https://ooo-updates.apache.org/aoonext/check.Update?pkgfmt=installed
> > > > 2 9 2022-05-28 07:13:52.61 Not using a proxy server
> > > > 3 9 2022-05-28 07:13:52.61 GET line 1093
> > > > 4 9 2022-05-28 07:13:52.61 [CurlINFO ] Trying
> > > > 151.101.2.132:443...
> > > > 5 9 2022-05-28 07:13:52.63 [CurlINFO ] Connected to
> > > > ooo-updates.apache.org (151.101.2.132) port 443 (#0)
> > > > 6 9 2022-05-28 07:13:52.63 [CurlINFO ] ALPN, offering
> > > > http/1.1
> > > > 7 9 2022-05-28 07:13:52.63 [CurlINFO ] Cipher
> > selection:
> > > > ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
> > > > 8 9 2022-05-28 07:13:52.63 [CurlINFO ] TLSv1.2 (OUT),
> > TLS
> > > > handshake, Client hello (1):
> > > > 9 9 2022-05-28 07:13:52.64 [CurlINFO ] TLSv1.2 (IN),
> > TLS
> > > > handshake, Server hello (2):
> > > > 10 9 2022-05-28 07:13:52.64 [CurlINFO ] TLSv1.2 (IN),
> > TLS
> > > > handshake, Certificate (11):
> > > > --here we are inside OPENSSL_ValidateServerCertificate
> > > > 11 9 2022-05-28 07:13:52.64
> > validateServerX509Certificate()
> > > > returning 0 at depth 2
> > > > --here we are leaving OPENSSL_ValidateServerCertificate
> > > > 12 9 2022-05-28 07:13:52.64 [CurlINFO ] TLSv1.2 (OUT),
> > TLS
> > > > alert, unknown CA (560):
> > > > 13 9 2022-05-28 07:13:52.64 [CurlINFO ] SSL certificate
> > > > problem: unable to get local issuer certificate
> > > > 14 9 2022-05-28 07:13:52.64 [CurlINFO ] Closing
> > connection
> > > > 0
> > > > 15 9 2022-05-28 07:13:52.64 Curl request failed with
> > > > CURLcode 60
> > > > 16 9 2022-05-28 07:13:52.64 CurlSession::~CurlSession:
> > > > closed curl session
> > > >
> > > >
> > ----8<--------8<--------8<--------8<--------8<--------8<--------8<---------
> > > >
> > > > I hope this helps.
> > > >
> > > >
> > > I think the problem there is NSS's validation.
> > >
> > > In CurlSession::validateServerX509Certificate(), try change the "if (
> > depth
> > > == 0 )" to "if (1)".
> >
> > After this edit, the connection is established. The log is below.
> >
>
> That's good :).
>
>
> >
> > But NSS was also used before, with Serf... it stopped working when the
> > bundled CA certificates started to be ``too obsolete''. After we
> > updated the bundled NSS, then https connections, such as the check for
> > updates, started to work again.
> >
>
> We don't need NSS specifically. From Curl, we need OpenSSL, because without
> it, we are unable to ask the user to confirm self-signed certificates
> during TLS negotiations [1]. Upstream OpenSSL however doesn't ship any CA
> certificates, so the WebDAV provider gets them from our
> com.sun.star.security.CertificateContainer, which comes from WinCrypt on
> Windows (also called "mscrypt" by main/xmlsecurity/source/xmlsec/mscrypt)
> and NSS elsewhere.
>
> Therefore that update check should not have been failing on Windows. Only
> if your Windows is out of date, should WinCrypt CA certificates fail.
Honestly, I do not really recall what happened on Windows.
I do recall that Linux builds were unable to check for update.
I thought that the reason was an outdated Serf library, and I proposed
an update, with the addition of SCons etc.
In parallel, we updated the version of the bundled NSS. When we did
_this_, the updates started to work again.
We never merged the serf update, so I think it makes sense to consider
the NSS obsolescence related.
We also tried to upgrade OpenSSL, but then stepped back. So NSS
remains the most probable cause.
> It is also particularly strange for NSS certificates to be out of date. The
> NSS certificate database that our
> com.sun.star.security.CertificateContainer uses comes from a profile
> (getMozillaCurrentProfile() in
> main/xmlsecurity/source/nss/nssinitializer.cxx) which (AFAIK) is in the
> user's home directory, and shared with and updated by Firefox or
> Thunderbird. Only if you don't update AOO, and don't run Firefox, and don't
> run Thunderbird, can your NSS certificates get out of date.
I do not know how to reply here. I can just share my findings from
above...
> > Please also bear in mind that I am now testing a build from a _CentOS
> > Docker container_. My computer is not running CentOS, but rather
> > OpenSUSE. The binary is run on ``native'' OpenSUSE. If I recall
> > correctly, when I build on the same native OpenSUSE, the resulting AOO
> > binary connects properly to https web sites.
> >
> > For this reason, I was wondering if this problem is not just about
> > hardcoded paths to CA certificates.
> >
>
> Since Curl's OpenSSL CA path is hardcoded to a non-portable value at
> compile time, my proposed approach here disables OpenSSL's certificates and
> only uses the ones from our com.sun.star.security.CertificateContainer
> instead. (And, who knows, it may be helpful to make another
> com.sun.star.security.CertificateContainer that gets certificates from
> guessed system paths for all of AOO, not just WebDAV.)
>
> Is there something else you propose instead?
I am not sure my emails are communicating the proper ``tone'', so
please let me state it explicitly: I am neither a cryptography, SSL or
NSS expert at all. You seem to be much more knowledgeable than me and
I am thankful to see your contributions. I am just doing my best to
validate them in all possible cases (mostly under different Linux
distributions).
I am trying to give useful advice, such as the above comments about
NSS. I might just be wrong, in such case I apologize for the noise.
Having said this... what do you mean exactly for your ``proposed
approach''? Is it the patch from "if (depth == 0)" to "if (1)"?
> [1] But is Curl's inability to use custom certificate verification
> functions (which can ask the user to confirm problematic certificates) with
> NSS a limitation of Curl, or a limitation of NSS? Firefox uses NSS, and it
> certainly asks the user to confirm expired and self-signed certificates.
> How does Firefox do that, by disconnecting, asking the user, and
> reconnecting if confirmed, or the Curl/OpenSSL way, by keeping the TCP
> connection open during the user interaction and then resuming TLS
> negotiations if confirmed?
For what it's worth, I tried opening https://developer.berlios.de ,
which has an invalid SSL certificate. Firefox 91.9.1esr on Linux
x86_64 closes the TCP connection immediately and remains waiting for
user input.
I hope this helps.
Best regards,
--
Arrigo
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)
Posted by Damjan Jovanovic <da...@apache.org>.
On Sat, May 28, 2022 at 11:48 AM Arrigo Marchiori <ar...@yahoo.it.invalid>
wrote:
> Hello Damjan,
>
> On Sat, May 28, 2022 at 11:12:45AM +0200, Damjan Jovanovic wrote:
>
> > On Sat, May 28, 2022 at 9:20 AM Arrigo Marchiori <ardovm@yahoo.it.invalid
> >
> > wrote:
> >
> > > Hello Damjan, all,
> > >
> > > On Fri, May 27, 2022 at 11:27:11PM +0200, Arrigo Marchiori wrote:
> > >
> > > > Hello,
> > > >
> > > > On Fri, May 27, 2022 at 09:46:51PM +0200, Arrigo Marchiori wrote:
> > > >
> > > > > Hello Damjan,
> > > > >
> > > > > On Sun, May 22, 2022 at 06:10:46PM +0200, Damjan Jovanovic wrote:
> > > > >
> > > > > > On Sun, May 22, 2022 at 2:43 PM Arrigo Marchiori
> > > <ar...@yahoo.it.invalid>
> > > > > > wrote:
> > > > > >
> > > > > > > Hello Damjan, all,
> > > > > > >
> > > > > > > On Tue, Apr 26, 2022 at 07:56:22PM +0200, Damjan Jovanovic
> wrote:
> > > > > > >
> > > > > > > > On Mon, Nov 15, 2021 at 9:57 PM Jim Jagielski <
> jim@jagunet.com>
> > > wrote:
> > > > > > > >
> > > > > > > > > I'm gonna look into the serf->(lib)curl option... Since we
> > > don't use
> > > > > > > any
> > > > > > > > > of the fancy features of serf, I'm thinking that the easy
> > > option might
> > > > > > > be
> > > > > > > > > best
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Hi
> > > > > > > >
> > > > > > > > I've ported our WebDAV content provider module from Serf to
> Curl.
> > > > > > >
> > > > > > > I just enhanced the error reporting a bit; I am finding a
> problem
> > > > > > > under Linux and I do not really know how to assess it.
> > > > > > >
> > > > > > > The problem: if we build AOO on CentOS (that is our reference
> > > > > > > platform) then Curl will look for CA certificates in
> > > > > > > /etc/pki/tls/certs/ca-bundle.crt
> > > > > > >
> > > > > > > This will fail on openSUSE and probably on Ubuntu as well.
> > > > > > >
> > > > > > > It seems that the above path is set at configure time and
> embedded
> > > > > > > into Curl's code as #define macros.
> > > > > > >
> > > > > > > Is there an ``official'' way to assess this? Like, can we
> depend on
> > > > > > > NSS' certificate store as you wrote (quoted below)?
> > > > > > >
> > > > > >
> > > > > > Curl/OpenSSL have an enormous number of options and I am pretty
> sure
> > > it can
> > > > > > be fixed, but first I need to understand where and how it's
> failing.
> > > > > >
> > > > > > We currently allow it to run with the default CA certificate
> path, do
> > > > > > pre-verification on the server's certificate using those CA
> > > certificates,
> > > > > > then call our SSL_VERIFY_PEER function where we override the
> > > verification
> > > > > > result with the certificates from NSS.
> > > > >
> > > > > Apparently, it is failing before calling our SSL_VERIFY_PEER
> function.
> > > > >
> > > > > > If it's failing before reaching our SSL_VERIFY_PEER function, we
> > > should be
> > > > > > able to use Curl's CURLOPT_CAINFO or CURLOPT_CAINFO_BLOB
> functions
> > > to set a
> > > > > > custom CA certificate path (or in-memory buffer), maybe even an
> empty
> > > > > > buffer, so that it proceeds further. ("man CURLOPT_CAINFO", "man
> > > > > > CURLOPT_CAINFO_BLOB", or "man curl_easy_setopt" and read under
> the
> > > "SSL and
> > > > > > SECURITY OPTIONS" section.)
> > > > >
> > > > > So we would need to hard-code and try all possible paths to the CA
> > > > > bundle on Unix systems?
> > > > >
> > > > > > With the CURLOPT_CAINFO_BLOB option it might even be possible to
> > > skip the
> > > > > > custom certificate verification we do later, and pre-populate
> > > Curl/OpenSSL
> > > > > > with NSS certificates from the beginning, I just don't know
> enough
> > > about
> > > > > > NSS to rely on that (eg. if you are using a cryptographic device
> or
> > > smart
> > > > > > card in NSS, how does that work?). If that option is ok, then we
> > > might not
> > > > > > even need the NSS libraries: recent versions of NSS store all the
> > > > > > certificates in an SQLite database, which can be accessed with
> > > SQLite APIs
> > > > > > directly, no need to build with or ship the NSS libraries at all.
> > > > >
> > > > > If I understood correctly [1], a NSS-linked Curl would query NSS by
> > > > > itself... are we not in this condition?
> > > [...]
> > > > 1: https://curl.se/libcurl/c/CURLOPT_CAINFO.html
> > >
> > > Apparently, we are not. If we force CURLOPT_CAINFO and CURLOPT_CAPATH
> > > to be NULL, we get a bit further but eventually Curl aborts because
> > > validateServerX509Certificate fails.
> > >
> > > Here is the log for checking AOO updates (please excuse the long
> > > lines). I added an extra bit of logging when entering and leaving
> > > OPENSSL_ValidateServerCertificate, that is ``our SSL_VERIFY_PEER
> > > function''.
> > >
> > >
> ----8<--------8<--------8<--------8<--------8<--------8<--------8<---------
> > >
> > > 1 9 2022-05-28 07:13:52.61 CurlSession::CurlSession
> with
> > > URL
> https://ooo-updates.apache.org/aoonext/check.Update?pkgfmt=installed
> > > 2 9 2022-05-28 07:13:52.61 Not using a proxy server
> > > 3 9 2022-05-28 07:13:52.61 GET line 1093
> > > 4 9 2022-05-28 07:13:52.61 [CurlINFO ] Trying
> > > 151.101.2.132:443...
> > > 5 9 2022-05-28 07:13:52.63 [CurlINFO ] Connected to
> > > ooo-updates.apache.org (151.101.2.132) port 443 (#0)
> > > 6 9 2022-05-28 07:13:52.63 [CurlINFO ] ALPN, offering
> > > http/1.1
> > > 7 9 2022-05-28 07:13:52.63 [CurlINFO ] Cipher
> selection:
> > > ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
> > > 8 9 2022-05-28 07:13:52.63 [CurlINFO ] TLSv1.2 (OUT),
> TLS
> > > handshake, Client hello (1):
> > > 9 9 2022-05-28 07:13:52.64 [CurlINFO ] TLSv1.2 (IN),
> TLS
> > > handshake, Server hello (2):
> > > 10 9 2022-05-28 07:13:52.64 [CurlINFO ] TLSv1.2 (IN),
> TLS
> > > handshake, Certificate (11):
> > > --here we are inside OPENSSL_ValidateServerCertificate
> > > 11 9 2022-05-28 07:13:52.64
> validateServerX509Certificate()
> > > returning 0 at depth 2
> > > --here we are leaving OPENSSL_ValidateServerCertificate
> > > 12 9 2022-05-28 07:13:52.64 [CurlINFO ] TLSv1.2 (OUT),
> TLS
> > > alert, unknown CA (560):
> > > 13 9 2022-05-28 07:13:52.64 [CurlINFO ] SSL certificate
> > > problem: unable to get local issuer certificate
> > > 14 9 2022-05-28 07:13:52.64 [CurlINFO ] Closing
> connection
> > > 0
> > > 15 9 2022-05-28 07:13:52.64 Curl request failed with
> > > CURLcode 60
> > > 16 9 2022-05-28 07:13:52.64 CurlSession::~CurlSession:
> > > closed curl session
> > >
> > >
> ----8<--------8<--------8<--------8<--------8<--------8<--------8<---------
> > >
> > > I hope this helps.
> > >
> > >
> > I think the problem there is NSS's validation.
> >
> > In CurlSession::validateServerX509Certificate(), try change the "if (
> depth
> > == 0 )" to "if (1)".
>
> After this edit, the connection is established. The log is below.
>
That's good :).
>
> But NSS was also used before, with Serf... it stopped working when the
> bundled CA certificates started to be ``too obsolete''. After we
> updated the bundled NSS, then https connections, such as the check for
> updates, started to work again.
>
We don't need NSS specifically. From Curl, we need OpenSSL, because without
it, we are unable to ask the user to confirm self-signed certificates
during TLS negotiations [1]. Upstream OpenSSL however doesn't ship any CA
certificates, so the WebDAV provider gets them from our
com.sun.star.security.CertificateContainer, which comes from WinCrypt on
Windows (also called "mscrypt" by main/xmlsecurity/source/xmlsec/mscrypt)
and NSS elsewhere.
Therefore that update check should not have been failing on Windows. Only
if your Windows is out of date, should WinCrypt CA certificates fail.
It is also particularly strange for NSS certificates to be out of date. The
NSS certificate database that our
com.sun.star.security.CertificateContainer uses comes from a profile
(getMozillaCurrentProfile() in
main/xmlsecurity/source/nss/nssinitializer.cxx) which (AFAIK) is in the
user's home directory, and shared with and updated by Firefox or
Thunderbird. Only if you don't update AOO, and don't run Firefox, and don't
run Thunderbird, can your NSS certificates get out of date.
>
> Please also bear in mind that I am now testing a build from a _CentOS
> Docker container_. My computer is not running CentOS, but rather
> OpenSUSE. The binary is run on ``native'' OpenSUSE. If I recall
> correctly, when I build on the same native OpenSUSE, the resulting AOO
> binary connects properly to https web sites.
>
> For this reason, I was wondering if this problem is not just about
> hardcoded paths to CA certificates.
>
Since Curl's OpenSSL CA path is hardcoded to a non-portable value at
compile time, my proposed approach here disables OpenSSL's certificates and
only uses the ones from our com.sun.star.security.CertificateContainer
instead. (And, who knows, it may be helpful to make another
com.sun.star.security.CertificateContainer that gets certificates from
guessed system paths for all of AOO, not just WebDAV.)
Is there something else you propose instead?
Regards
Damjan
[1] But is Curl's inability to use custom certificate verification
functions (which can ask the user to confirm problematic certificates) with
NSS a limitation of Curl, or a limitation of NSS? Firefox uses NSS, and it
certainly asks the user to confirm expired and self-signed certificates.
How does Firefox do that, by disconnecting, asking the user, and
reconnecting if confirmed, or the Curl/OpenSSL way, by keeping the TCP
connection open during the user interaction and then resuming TLS
negotiations if confirmed?
Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)
Posted by Arrigo Marchiori <ar...@yahoo.it.INVALID>.
Hello Damjan,
On Sat, May 28, 2022 at 11:12:45AM +0200, Damjan Jovanovic wrote:
> On Sat, May 28, 2022 at 9:20 AM Arrigo Marchiori <ar...@yahoo.it.invalid>
> wrote:
>
> > Hello Damjan, all,
> >
> > On Fri, May 27, 2022 at 11:27:11PM +0200, Arrigo Marchiori wrote:
> >
> > > Hello,
> > >
> > > On Fri, May 27, 2022 at 09:46:51PM +0200, Arrigo Marchiori wrote:
> > >
> > > > Hello Damjan,
> > > >
> > > > On Sun, May 22, 2022 at 06:10:46PM +0200, Damjan Jovanovic wrote:
> > > >
> > > > > On Sun, May 22, 2022 at 2:43 PM Arrigo Marchiori
> > <ar...@yahoo.it.invalid>
> > > > > wrote:
> > > > >
> > > > > > Hello Damjan, all,
> > > > > >
> > > > > > On Tue, Apr 26, 2022 at 07:56:22PM +0200, Damjan Jovanovic wrote:
> > > > > >
> > > > > > > On Mon, Nov 15, 2021 at 9:57 PM Jim Jagielski <ji...@jagunet.com>
> > wrote:
> > > > > > >
> > > > > > > > I'm gonna look into the serf->(lib)curl option... Since we
> > don't use
> > > > > > any
> > > > > > > > of the fancy features of serf, I'm thinking that the easy
> > option might
> > > > > > be
> > > > > > > > best
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Hi
> > > > > > >
> > > > > > > I've ported our WebDAV content provider module from Serf to Curl.
> > > > > >
> > > > > > I just enhanced the error reporting a bit; I am finding a problem
> > > > > > under Linux and I do not really know how to assess it.
> > > > > >
> > > > > > The problem: if we build AOO on CentOS (that is our reference
> > > > > > platform) then Curl will look for CA certificates in
> > > > > > /etc/pki/tls/certs/ca-bundle.crt
> > > > > >
> > > > > > This will fail on openSUSE and probably on Ubuntu as well.
> > > > > >
> > > > > > It seems that the above path is set at configure time and embedded
> > > > > > into Curl's code as #define macros.
> > > > > >
> > > > > > Is there an ``official'' way to assess this? Like, can we depend on
> > > > > > NSS' certificate store as you wrote (quoted below)?
> > > > > >
> > > > >
> > > > > Curl/OpenSSL have an enormous number of options and I am pretty sure
> > it can
> > > > > be fixed, but first I need to understand where and how it's failing.
> > > > >
> > > > > We currently allow it to run with the default CA certificate path, do
> > > > > pre-verification on the server's certificate using those CA
> > certificates,
> > > > > then call our SSL_VERIFY_PEER function where we override the
> > verification
> > > > > result with the certificates from NSS.
> > > >
> > > > Apparently, it is failing before calling our SSL_VERIFY_PEER function.
> > > >
> > > > > If it's failing before reaching our SSL_VERIFY_PEER function, we
> > should be
> > > > > able to use Curl's CURLOPT_CAINFO or CURLOPT_CAINFO_BLOB functions
> > to set a
> > > > > custom CA certificate path (or in-memory buffer), maybe even an empty
> > > > > buffer, so that it proceeds further. ("man CURLOPT_CAINFO", "man
> > > > > CURLOPT_CAINFO_BLOB", or "man curl_easy_setopt" and read under the
> > "SSL and
> > > > > SECURITY OPTIONS" section.)
> > > >
> > > > So we would need to hard-code and try all possible paths to the CA
> > > > bundle on Unix systems?
> > > >
> > > > > With the CURLOPT_CAINFO_BLOB option it might even be possible to
> > skip the
> > > > > custom certificate verification we do later, and pre-populate
> > Curl/OpenSSL
> > > > > with NSS certificates from the beginning, I just don't know enough
> > about
> > > > > NSS to rely on that (eg. if you are using a cryptographic device or
> > smart
> > > > > card in NSS, how does that work?). If that option is ok, then we
> > might not
> > > > > even need the NSS libraries: recent versions of NSS store all the
> > > > > certificates in an SQLite database, which can be accessed with
> > SQLite APIs
> > > > > directly, no need to build with or ship the NSS libraries at all.
> > > >
> > > > If I understood correctly [1], a NSS-linked Curl would query NSS by
> > > > itself... are we not in this condition?
> > [...]
> > > 1: https://curl.se/libcurl/c/CURLOPT_CAINFO.html
> >
> > Apparently, we are not. If we force CURLOPT_CAINFO and CURLOPT_CAPATH
> > to be NULL, we get a bit further but eventually Curl aborts because
> > validateServerX509Certificate fails.
> >
> > Here is the log for checking AOO updates (please excuse the long
> > lines). I added an extra bit of logging when entering and leaving
> > OPENSSL_ValidateServerCertificate, that is ``our SSL_VERIFY_PEER
> > function''.
> >
> > ----8<--------8<--------8<--------8<--------8<--------8<--------8<---------
> >
> > 1 9 2022-05-28 07:13:52.61 CurlSession::CurlSession with
> > URL https://ooo-updates.apache.org/aoonext/check.Update?pkgfmt=installed
> > 2 9 2022-05-28 07:13:52.61 Not using a proxy server
> > 3 9 2022-05-28 07:13:52.61 GET line 1093
> > 4 9 2022-05-28 07:13:52.61 [CurlINFO ] Trying
> > 151.101.2.132:443...
> > 5 9 2022-05-28 07:13:52.63 [CurlINFO ] Connected to
> > ooo-updates.apache.org (151.101.2.132) port 443 (#0)
> > 6 9 2022-05-28 07:13:52.63 [CurlINFO ] ALPN, offering
> > http/1.1
> > 7 9 2022-05-28 07:13:52.63 [CurlINFO ] Cipher selection:
> > ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
> > 8 9 2022-05-28 07:13:52.63 [CurlINFO ] TLSv1.2 (OUT), TLS
> > handshake, Client hello (1):
> > 9 9 2022-05-28 07:13:52.64 [CurlINFO ] TLSv1.2 (IN), TLS
> > handshake, Server hello (2):
> > 10 9 2022-05-28 07:13:52.64 [CurlINFO ] TLSv1.2 (IN), TLS
> > handshake, Certificate (11):
> > --here we are inside OPENSSL_ValidateServerCertificate
> > 11 9 2022-05-28 07:13:52.64 validateServerX509Certificate()
> > returning 0 at depth 2
> > --here we are leaving OPENSSL_ValidateServerCertificate
> > 12 9 2022-05-28 07:13:52.64 [CurlINFO ] TLSv1.2 (OUT), TLS
> > alert, unknown CA (560):
> > 13 9 2022-05-28 07:13:52.64 [CurlINFO ] SSL certificate
> > problem: unable to get local issuer certificate
> > 14 9 2022-05-28 07:13:52.64 [CurlINFO ] Closing connection
> > 0
> > 15 9 2022-05-28 07:13:52.64 Curl request failed with
> > CURLcode 60
> > 16 9 2022-05-28 07:13:52.64 CurlSession::~CurlSession:
> > closed curl session
> >
> > ----8<--------8<--------8<--------8<--------8<--------8<--------8<---------
> >
> > I hope this helps.
> >
> >
> I think the problem there is NSS's validation.
>
> In CurlSession::validateServerX509Certificate(), try change the "if ( depth
> == 0 )" to "if (1)".
After this edit, the connection is established. The log is below.
But NSS was also used before, with Serf... it stopped working when the
bundled CA certificates started to be ``too obsolete''. After we
updated the bundled NSS, then https connections, such as the check for
updates, started to work again.
Please also bear in mind that I am now testing a build from a _CentOS
Docker container_. My computer is not running CentOS, but rather
OpenSUSE. The binary is run on ``native'' OpenSUSE. If I recall
correctly, when I build on the same native OpenSUSE, the resulting AOO
binary connects properly to https web sites.
For this reason, I was wondering if this problem is not just about
hardcoded paths to CA certificates.
----8<--------8<--------8<--------8<--------8<--------8<--------8<---------
1 14 2022-05-28 09:38:47.66 CurlSession::CurlSession with URL https://ooo-updates.apache.org/aoonext/chec
k.Update?pkgfmt=installed 2 14 2022-05-28 09:38:47.66 Not using a proxy server
3 14 2022-05-28 09:38:47.66 GET line 1093 4 14 2022-05-28 09:38:47.66 [CurlINFO ] Trying 151.101.2.132:443...
5 14 2022-05-28 09:38:47.67 [CurlINFO ] Connected to ooo-updates.apache.org (151.101.2.132) port 443 (#0)
6 14 2022-05-28 09:38:47.67 [CurlINFO ] ALPN, offering http/1.1 7 14 2022-05-28 09:38:47.67 [CurlINFO ] Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!R
C4:@STRENGTH 8 14 2022-05-28 09:38:47.67 [CurlINFO ] TLSv1.2 (OUT), TLS handshake, Client hello (1):
9 14 2022-05-28 09:38:47.68 [CurlINFO ] TLSv1.2 (IN), TLS handshake, Server hello (2): 10 14 2022-05-28 09:38:47.68 [CurlINFO ] TLSv1.2 (IN), TLS handshake, Certificate (11):
FIXME inside OPENSSL_ValidateServerCertificate 11 14 2022-05-28 09:38:47.68 Cached certificate found with status=trusted
12 14 2022-05-28 09:38:47.68 validateServerX509Certificate() returning 1 at depth 2 FIXME end of OPENSSL_ValidateServerCertificate
FIXME inside OPENSSL_ValidateServerCertificate 13 14 2022-05-28 09:38:47.68 Cached certificate found with status=trusted
14 14 2022-05-28 09:38:47.68 validateServerX509Certificate() returning 1 at depth 2 FIXME end of OPENSSL_ValidateServerCertificate
FIXME inside OPENSSL_ValidateServerCertificate
15 14 2022-05-28 09:38:47.68 Cached certificate found with status=trusted
16 14 2022-05-28 09:38:47.68 validateServerX509Certificate() returning 1 at depth 1
FIXME end of OPENSSL_ValidateServerCertificate
FIXME inside OPENSSL_ValidateServerCertificate
17 14 2022-05-28 09:38:47.68 Cached certificate found with status=trusted
18 14 2022-05-28 09:38:47.68 validateServerX509Certificate() returning 1 at depth 0
FIXME end of OPENSSL_ValidateServerCertificate
19 14 2022-05-28 09:38:47.68 [CurlINFO ] TLSv1.2 (IN), TLS handshake, Server key exchange (12):
20 14 2022-05-28 09:38:47.68 [CurlINFO ] TLSv1.2 (IN), TLS handshake, Server finished (14):
21 14 2022-05-28 09:38:47.68 [CurlINFO ] TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
22 14 2022-05-28 09:38:47.68 [CurlINFO ] TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
23 14 2022-05-28 09:38:47.68 [CurlINFO ] TLSv1.2 (OUT), TLS handshake, Finished (20):
24 14 2022-05-28 09:38:47.69 [CurlINFO ] TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
25 14 2022-05-28 09:38:47.69 [CurlINFO ] TLSv1.2 (IN), TLS handshake, Finished (20):
26 14 2022-05-28 09:38:47.69 [CurlINFO ] SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
27 14 2022-05-28 09:38:47.69 [CurlINFO ] ALPN, server accepted to use http/1.1
28 14 2022-05-28 09:38:47.69 [CurlINFO ] Server certificate:
29 14 2022-05-28 09:38:47.69 [CurlINFO ] subject: CN=*.apache.org
30 14 2022-05-28 09:38:47.69 [CurlINFO ] start date: Apr 18 19:43:41 2022 GMT
31 14 2022-05-28 09:38:47.69 [CurlINFO ] expire date: Jul 17 19:43:40 2022 GMT
32 14 2022-05-28 09:38:47.69 [CurlINFO ] subjectAltName: host "ooo-updates.apache.org" matched cert's "*
.apache.org"
33 14 2022-05-28 09:38:47.69 [CurlINFO ] issuer: C=US; O=Let's Encrypt; CN=R3
34 14 2022-05-28 09:38:47.69 [CurlINFO ] SSL certificate verify ok.
[35 and following messages are about the actual transfer ]
----8<--------8<--------8<--------8<--------8<--------8<--------8<---------
I hope this helps.
Best regards,
--
Arrigo
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)
Posted by Damjan Jovanovic <da...@apache.org>.
On Sat, May 28, 2022 at 9:20 AM Arrigo Marchiori <ar...@yahoo.it.invalid>
wrote:
> Hello Damjan, all,
>
> On Fri, May 27, 2022 at 11:27:11PM +0200, Arrigo Marchiori wrote:
>
> > Hello,
> >
> > On Fri, May 27, 2022 at 09:46:51PM +0200, Arrigo Marchiori wrote:
> >
> > > Hello Damjan,
> > >
> > > On Sun, May 22, 2022 at 06:10:46PM +0200, Damjan Jovanovic wrote:
> > >
> > > > On Sun, May 22, 2022 at 2:43 PM Arrigo Marchiori
> <ar...@yahoo.it.invalid>
> > > > wrote:
> > > >
> > > > > Hello Damjan, all,
> > > > >
> > > > > On Tue, Apr 26, 2022 at 07:56:22PM +0200, Damjan Jovanovic wrote:
> > > > >
> > > > > > On Mon, Nov 15, 2021 at 9:57 PM Jim Jagielski <ji...@jagunet.com>
> wrote:
> > > > > >
> > > > > > > I'm gonna look into the serf->(lib)curl option... Since we
> don't use
> > > > > any
> > > > > > > of the fancy features of serf, I'm thinking that the easy
> option might
> > > > > be
> > > > > > > best
> > > > > >
> > > > > >
> > > > > >
> > > > > > Hi
> > > > > >
> > > > > > I've ported our WebDAV content provider module from Serf to Curl.
> > > > >
> > > > > I just enhanced the error reporting a bit; I am finding a problem
> > > > > under Linux and I do not really know how to assess it.
> > > > >
> > > > > The problem: if we build AOO on CentOS (that is our reference
> > > > > platform) then Curl will look for CA certificates in
> > > > > /etc/pki/tls/certs/ca-bundle.crt
> > > > >
> > > > > This will fail on openSUSE and probably on Ubuntu as well.
> > > > >
> > > > > It seems that the above path is set at configure time and embedded
> > > > > into Curl's code as #define macros.
> > > > >
> > > > > Is there an ``official'' way to assess this? Like, can we depend on
> > > > > NSS' certificate store as you wrote (quoted below)?
> > > > >
> > > >
> > > > Curl/OpenSSL have an enormous number of options and I am pretty sure
> it can
> > > > be fixed, but first I need to understand where and how it's failing.
> > > >
> > > > We currently allow it to run with the default CA certificate path, do
> > > > pre-verification on the server's certificate using those CA
> certificates,
> > > > then call our SSL_VERIFY_PEER function where we override the
> verification
> > > > result with the certificates from NSS.
> > >
> > > Apparently, it is failing before calling our SSL_VERIFY_PEER function.
> > >
> > > > If it's failing before reaching our SSL_VERIFY_PEER function, we
> should be
> > > > able to use Curl's CURLOPT_CAINFO or CURLOPT_CAINFO_BLOB functions
> to set a
> > > > custom CA certificate path (or in-memory buffer), maybe even an empty
> > > > buffer, so that it proceeds further. ("man CURLOPT_CAINFO", "man
> > > > CURLOPT_CAINFO_BLOB", or "man curl_easy_setopt" and read under the
> "SSL and
> > > > SECURITY OPTIONS" section.)
> > >
> > > So we would need to hard-code and try all possible paths to the CA
> > > bundle on Unix systems?
> > >
> > > > With the CURLOPT_CAINFO_BLOB option it might even be possible to
> skip the
> > > > custom certificate verification we do later, and pre-populate
> Curl/OpenSSL
> > > > with NSS certificates from the beginning, I just don't know enough
> about
> > > > NSS to rely on that (eg. if you are using a cryptographic device or
> smart
> > > > card in NSS, how does that work?). If that option is ok, then we
> might not
> > > > even need the NSS libraries: recent versions of NSS store all the
> > > > certificates in an SQLite database, which can be accessed with
> SQLite APIs
> > > > directly, no need to build with or ship the NSS libraries at all.
> > >
> > > If I understood correctly [1], a NSS-linked Curl would query NSS by
> > > itself... are we not in this condition?
> [...]
> > 1: https://curl.se/libcurl/c/CURLOPT_CAINFO.html
>
> Apparently, we are not. If we force CURLOPT_CAINFO and CURLOPT_CAPATH
> to be NULL, we get a bit further but eventually Curl aborts because
> validateServerX509Certificate fails.
>
> Here is the log for checking AOO updates (please excuse the long
> lines). I added an extra bit of logging when entering and leaving
> OPENSSL_ValidateServerCertificate, that is ``our SSL_VERIFY_PEER
> function''.
>
> ----8<--------8<--------8<--------8<--------8<--------8<--------8<---------
>
> 1 9 2022-05-28 07:13:52.61 CurlSession::CurlSession with
> URL https://ooo-updates.apache.org/aoonext/check.Update?pkgfmt=installed
> 2 9 2022-05-28 07:13:52.61 Not using a proxy server
> 3 9 2022-05-28 07:13:52.61 GET line 1093
> 4 9 2022-05-28 07:13:52.61 [CurlINFO ] Trying
> 151.101.2.132:443...
> 5 9 2022-05-28 07:13:52.63 [CurlINFO ] Connected to
> ooo-updates.apache.org (151.101.2.132) port 443 (#0)
> 6 9 2022-05-28 07:13:52.63 [CurlINFO ] ALPN, offering
> http/1.1
> 7 9 2022-05-28 07:13:52.63 [CurlINFO ] Cipher selection:
> ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
> 8 9 2022-05-28 07:13:52.63 [CurlINFO ] TLSv1.2 (OUT), TLS
> handshake, Client hello (1):
> 9 9 2022-05-28 07:13:52.64 [CurlINFO ] TLSv1.2 (IN), TLS
> handshake, Server hello (2):
> 10 9 2022-05-28 07:13:52.64 [CurlINFO ] TLSv1.2 (IN), TLS
> handshake, Certificate (11):
> --here we are inside OPENSSL_ValidateServerCertificate
> 11 9 2022-05-28 07:13:52.64 validateServerX509Certificate()
> returning 0 at depth 2
> --here we are leaving OPENSSL_ValidateServerCertificate
> 12 9 2022-05-28 07:13:52.64 [CurlINFO ] TLSv1.2 (OUT), TLS
> alert, unknown CA (560):
> 13 9 2022-05-28 07:13:52.64 [CurlINFO ] SSL certificate
> problem: unable to get local issuer certificate
> 14 9 2022-05-28 07:13:52.64 [CurlINFO ] Closing connection
> 0
> 15 9 2022-05-28 07:13:52.64 Curl request failed with
> CURLcode 60
> 16 9 2022-05-28 07:13:52.64 CurlSession::~CurlSession:
> closed curl session
>
> ----8<--------8<--------8<--------8<--------8<--------8<--------8<---------
>
> I hope this helps.
>
>
I think the problem there is NSS's validation.
In CurlSession::validateServerX509Certificate(), try change the "if ( depth
== 0 )" to "if (1)".
Regards
Damjan
Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)
Posted by Damjan Jovanovic <da...@apache.org>.
On Sat, May 28, 2022 at 7:20 AM Arrigo Marchiori <ar...@yahoo.it.invalid>
wrote:
>
> Apparently, we are not. If we force CURLOPT_CAINFO and CURLOPT_CAPATH
> to be NULL, we get a bit further but eventually Curl aborts because
> validateServerX509Certificate fails.
>
>
Sorry to resurrect an old thread, but I think I now understand how this
crash happens.
So I always thought it was OpenSSL doing something wrong with the
certificates, but then in Curl's configure.ac, close to the end of the
file, there is the following:
---snip---
ca cert bundle: ${ca}${ca_warning}
ca cert path: ${capath}${capath_warning}
ca fallback: ${with_ca_fallback}
---snip---
which apparently comes from:
---snip---
dnl **********************************************************************
dnl Check for the CA bundle
dnl **********************************************************************
if test -n "$check_for_ca_bundle"; then
CURL_CHECK_CA_BUNDLE
fi
---snip---
and in Curl's acinclude.m4 we see that the CURL_CHECK_CA_BUNDLE function
uses a number of options, distro-specific search paths, etc.
This would also explain why system Curl always worked perfectly for me on
FreeBSD - the Port already built it with
--with-ca-bundle="${LOCALBASE}/share/certs/ca-root-nss.crt".
Now presumably things go wrong when Curl is using a bad path to the CA
bundle, because either it detected the path wrong during ./configure, or
it's running on a different distro.
So we should either build our Curl with the --without-ca-bundle
--without-ca-path --without-ca-fallback options, or disable use of those
settings at runtime. I prefer doing it at runtime, to be on the safe side.
I have a patch ready, but I am now unsure about other issues. If we are
going to use our own certificate validation instead of OpenSSL's, shouldn't
we use SSL_CTX_set_cert_verify_callback() which completely replaces
OpenSSL's verification, instead of SSL_CTX_set_verify() which just allows
us to override its result on each certificate? Also what steps does OpenSSL
follow? We have a page on our Wiki (
https://wiki.openoffice.org/wiki/Certificate_Path_Validation) with proposed
certificate validation requirements, and also need to confirm what OpenSSL
does, and whether it does the more elaborate checks like connecting to OCSP
and CRL servers to verify revocation status. If not, we'd have to develop
that ourselves.
Regards
Damjan
Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)
Posted by Arrigo Marchiori <ar...@yahoo.it.INVALID>.
Hello Damjan, all,
On Fri, May 27, 2022 at 11:27:11PM +0200, Arrigo Marchiori wrote:
> Hello,
>
> On Fri, May 27, 2022 at 09:46:51PM +0200, Arrigo Marchiori wrote:
>
> > Hello Damjan,
> >
> > On Sun, May 22, 2022 at 06:10:46PM +0200, Damjan Jovanovic wrote:
> >
> > > On Sun, May 22, 2022 at 2:43 PM Arrigo Marchiori <ar...@yahoo.it.invalid>
> > > wrote:
> > >
> > > > Hello Damjan, all,
> > > >
> > > > On Tue, Apr 26, 2022 at 07:56:22PM +0200, Damjan Jovanovic wrote:
> > > >
> > > > > On Mon, Nov 15, 2021 at 9:57 PM Jim Jagielski <ji...@jagunet.com> wrote:
> > > > >
> > > > > > I'm gonna look into the serf->(lib)curl option... Since we don't use
> > > > any
> > > > > > of the fancy features of serf, I'm thinking that the easy option might
> > > > be
> > > > > > best
> > > > >
> > > > >
> > > > >
> > > > > Hi
> > > > >
> > > > > I've ported our WebDAV content provider module from Serf to Curl.
> > > >
> > > > I just enhanced the error reporting a bit; I am finding a problem
> > > > under Linux and I do not really know how to assess it.
> > > >
> > > > The problem: if we build AOO on CentOS (that is our reference
> > > > platform) then Curl will look for CA certificates in
> > > > /etc/pki/tls/certs/ca-bundle.crt
> > > >
> > > > This will fail on openSUSE and probably on Ubuntu as well.
> > > >
> > > > It seems that the above path is set at configure time and embedded
> > > > into Curl's code as #define macros.
> > > >
> > > > Is there an ``official'' way to assess this? Like, can we depend on
> > > > NSS' certificate store as you wrote (quoted below)?
> > > >
> > >
> > > Curl/OpenSSL have an enormous number of options and I am pretty sure it can
> > > be fixed, but first I need to understand where and how it's failing.
> > >
> > > We currently allow it to run with the default CA certificate path, do
> > > pre-verification on the server's certificate using those CA certificates,
> > > then call our SSL_VERIFY_PEER function where we override the verification
> > > result with the certificates from NSS.
> >
> > Apparently, it is failing before calling our SSL_VERIFY_PEER function.
> >
> > > If it's failing before reaching our SSL_VERIFY_PEER function, we should be
> > > able to use Curl's CURLOPT_CAINFO or CURLOPT_CAINFO_BLOB functions to set a
> > > custom CA certificate path (or in-memory buffer), maybe even an empty
> > > buffer, so that it proceeds further. ("man CURLOPT_CAINFO", "man
> > > CURLOPT_CAINFO_BLOB", or "man curl_easy_setopt" and read under the "SSL and
> > > SECURITY OPTIONS" section.)
> >
> > So we would need to hard-code and try all possible paths to the CA
> > bundle on Unix systems?
> >
> > > With the CURLOPT_CAINFO_BLOB option it might even be possible to skip the
> > > custom certificate verification we do later, and pre-populate Curl/OpenSSL
> > > with NSS certificates from the beginning, I just don't know enough about
> > > NSS to rely on that (eg. if you are using a cryptographic device or smart
> > > card in NSS, how does that work?). If that option is ok, then we might not
> > > even need the NSS libraries: recent versions of NSS store all the
> > > certificates in an SQLite database, which can be accessed with SQLite APIs
> > > directly, no need to build with or ship the NSS libraries at all.
> >
> > If I understood correctly [1], a NSS-linked Curl would query NSS by
> > itself... are we not in this condition?
[...]
> 1: https://curl.se/libcurl/c/CURLOPT_CAINFO.html
Apparently, we are not. If we force CURLOPT_CAINFO and CURLOPT_CAPATH
to be NULL, we get a bit further but eventually Curl aborts because
validateServerX509Certificate fails.
Here is the log for checking AOO updates (please excuse the long
lines). I added an extra bit of logging when entering and leaving
OPENSSL_ValidateServerCertificate, that is ``our SSL_VERIFY_PEER
function''.
----8<--------8<--------8<--------8<--------8<--------8<--------8<---------
1 9 2022-05-28 07:13:52.61 CurlSession::CurlSession with URL https://ooo-updates.apache.org/aoonext/check.Update?pkgfmt=installed
2 9 2022-05-28 07:13:52.61 Not using a proxy server
3 9 2022-05-28 07:13:52.61 GET line 1093
4 9 2022-05-28 07:13:52.61 [CurlINFO ] Trying 151.101.2.132:443...
5 9 2022-05-28 07:13:52.63 [CurlINFO ] Connected to ooo-updates.apache.org (151.101.2.132) port 443 (#0)
6 9 2022-05-28 07:13:52.63 [CurlINFO ] ALPN, offering http/1.1
7 9 2022-05-28 07:13:52.63 [CurlINFO ] Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
8 9 2022-05-28 07:13:52.63 [CurlINFO ] TLSv1.2 (OUT), TLS handshake, Client hello (1):
9 9 2022-05-28 07:13:52.64 [CurlINFO ] TLSv1.2 (IN), TLS handshake, Server hello (2):
10 9 2022-05-28 07:13:52.64 [CurlINFO ] TLSv1.2 (IN), TLS handshake, Certificate (11):
--here we are inside OPENSSL_ValidateServerCertificate
11 9 2022-05-28 07:13:52.64 validateServerX509Certificate() returning 0 at depth 2
--here we are leaving OPENSSL_ValidateServerCertificate
12 9 2022-05-28 07:13:52.64 [CurlINFO ] TLSv1.2 (OUT), TLS alert, unknown CA (560):
13 9 2022-05-28 07:13:52.64 [CurlINFO ] SSL certificate problem: unable to get local issuer certificate
14 9 2022-05-28 07:13:52.64 [CurlINFO ] Closing connection 0
15 9 2022-05-28 07:13:52.64 Curl request failed with CURLcode 60
16 9 2022-05-28 07:13:52.64 CurlSession::~CurlSession: closed curl session
----8<--------8<--------8<--------8<--------8<--------8<--------8<---------
I hope this helps.
Best regards,
--
Arrigo
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)
Posted by Arrigo Marchiori <ar...@yahoo.it.INVALID>.
Hello,
On Fri, May 27, 2022 at 09:46:51PM +0200, Arrigo Marchiori wrote:
> Hello Damjan,
>
> On Sun, May 22, 2022 at 06:10:46PM +0200, Damjan Jovanovic wrote:
>
> > On Sun, May 22, 2022 at 2:43 PM Arrigo Marchiori <ar...@yahoo.it.invalid>
> > wrote:
> >
> > > Hello Damjan, all,
> > >
> > > On Tue, Apr 26, 2022 at 07:56:22PM +0200, Damjan Jovanovic wrote:
> > >
> > > > On Mon, Nov 15, 2021 at 9:57 PM Jim Jagielski <ji...@jagunet.com> wrote:
> > > >
> > > > > I'm gonna look into the serf->(lib)curl option... Since we don't use
> > > any
> > > > > of the fancy features of serf, I'm thinking that the easy option might
> > > be
> > > > > best
> > > >
> > > >
> > > >
> > > > Hi
> > > >
> > > > I've ported our WebDAV content provider module from Serf to Curl.
> > >
> > > I just enhanced the error reporting a bit; I am finding a problem
> > > under Linux and I do not really know how to assess it.
> > >
> > > The problem: if we build AOO on CentOS (that is our reference
> > > platform) then Curl will look for CA certificates in
> > > /etc/pki/tls/certs/ca-bundle.crt
> > >
> > > This will fail on openSUSE and probably on Ubuntu as well.
> > >
> > > It seems that the above path is set at configure time and embedded
> > > into Curl's code as #define macros.
> > >
> > > Is there an ``official'' way to assess this? Like, can we depend on
> > > NSS' certificate store as you wrote (quoted below)?
> > >
> >
> > Curl/OpenSSL have an enormous number of options and I am pretty sure it can
> > be fixed, but first I need to understand where and how it's failing.
> >
> > We currently allow it to run with the default CA certificate path, do
> > pre-verification on the server's certificate using those CA certificates,
> > then call our SSL_VERIFY_PEER function where we override the verification
> > result with the certificates from NSS.
>
> Apparently, it is failing before calling our SSL_VERIFY_PEER function.
>
> > If it's failing before reaching our SSL_VERIFY_PEER function, we should be
> > able to use Curl's CURLOPT_CAINFO or CURLOPT_CAINFO_BLOB functions to set a
> > custom CA certificate path (or in-memory buffer), maybe even an empty
> > buffer, so that it proceeds further. ("man CURLOPT_CAINFO", "man
> > CURLOPT_CAINFO_BLOB", or "man curl_easy_setopt" and read under the "SSL and
> > SECURITY OPTIONS" section.)
>
> So we would need to hard-code and try all possible paths to the CA
> bundle on Unix systems?
>
> > With the CURLOPT_CAINFO_BLOB option it might even be possible to skip the
> > custom certificate verification we do later, and pre-populate Curl/OpenSSL
> > with NSS certificates from the beginning, I just don't know enough about
> > NSS to rely on that (eg. if you are using a cryptographic device or smart
> > card in NSS, how does that work?). If that option is ok, then we might not
> > even need the NSS libraries: recent versions of NSS store all the
> > certificates in an SQLite database, which can be accessed with SQLite APIs
> > directly, no need to build with or ship the NSS libraries at all.
>
> If I understood correctly [1], a NSS-linked Curl would query NSS by
> itself... are we not in this condition?
Sorry, I forgot the link!
Here it is:
1: https://curl.se/libcurl/c/CURLOPT_CAINFO.html
> Thank you in advance and best regards,
--
Arrigo
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)
Posted by Damjan Jovanovic <da...@apache.org>.
On Fri, May 27, 2022 at 9:47 PM Arrigo Marchiori <ar...@yahoo.it.invalid>
wrote:
> Hello Damjan,
>
> On Sun, May 22, 2022 at 06:10:46PM +0200, Damjan Jovanovic wrote:
>
> > On Sun, May 22, 2022 at 2:43 PM Arrigo Marchiori <ardovm@yahoo.it.invalid
> >
> > wrote:
> >
> > > Hello Damjan, all,
> > >
> > > On Tue, Apr 26, 2022 at 07:56:22PM +0200, Damjan Jovanovic wrote:
> > >
> > > > On Mon, Nov 15, 2021 at 9:57 PM Jim Jagielski <ji...@jagunet.com>
> wrote:
> > > >
> > > > > I'm gonna look into the serf->(lib)curl option... Since we don't
> use
> > > any
> > > > > of the fancy features of serf, I'm thinking that the easy option
> might
> > > be
> > > > > best
> > > >
> > > >
> > > >
> > > > Hi
> > > >
> > > > I've ported our WebDAV content provider module from Serf to Curl.
> > >
> > > I just enhanced the error reporting a bit; I am finding a problem
> > > under Linux and I do not really know how to assess it.
> > >
> > > The problem: if we build AOO on CentOS (that is our reference
> > > platform) then Curl will look for CA certificates in
> > > /etc/pki/tls/certs/ca-bundle.crt
> > >
> > > This will fail on openSUSE and probably on Ubuntu as well.
> > >
> > > It seems that the above path is set at configure time and embedded
> > > into Curl's code as #define macros.
> > >
> > > Is there an ``official'' way to assess this? Like, can we depend on
> > > NSS' certificate store as you wrote (quoted below)?
> > >
> >
> > Curl/OpenSSL have an enormous number of options and I am pretty sure it
> can
> > be fixed, but first I need to understand where and how it's failing.
> >
> > We currently allow it to run with the default CA certificate path, do
> > pre-verification on the server's certificate using those CA certificates,
> > then call our SSL_VERIFY_PEER function where we override the verification
> > result with the certificates from NSS.
>
> Apparently, it is failing before calling our SSL_VERIFY_PEER function.
>
> > If it's failing before reaching our SSL_VERIFY_PEER function, we should
> be
> > able to use Curl's CURLOPT_CAINFO or CURLOPT_CAINFO_BLOB functions to
> set a
> > custom CA certificate path (or in-memory buffer), maybe even an empty
> > buffer, so that it proceeds further. ("man CURLOPT_CAINFO", "man
> > CURLOPT_CAINFO_BLOB", or "man curl_easy_setopt" and read under the "SSL
> and
> > SECURITY OPTIONS" section.)
>
> So we would need to hard-code and try all possible paths to the CA
> bundle on Unix systems?
>
> > With the CURLOPT_CAINFO_BLOB option it might even be possible to skip the
> > custom certificate verification we do later, and pre-populate
> Curl/OpenSSL
> > with NSS certificates from the beginning, I just don't know enough about
> > NSS to rely on that (eg. if you are using a cryptographic device or smart
> > card in NSS, how does that work?). If that option is ok, then we might
> not
> > even need the NSS libraries: recent versions of NSS store all the
> > certificates in an SQLite database, which can be accessed with SQLite
> APIs
> > directly, no need to build with or ship the NSS libraries at all.
>
> If I understood correctly [1], a NSS-linked Curl would query NSS by
> itself... are we not in this condition?
>
Yes but NSS is category B, shouldn't we be avoiding it?
Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)
Posted by Arrigo Marchiori <ar...@yahoo.it.INVALID>.
Hello Damjan,
On Sun, May 22, 2022 at 06:10:46PM +0200, Damjan Jovanovic wrote:
> On Sun, May 22, 2022 at 2:43 PM Arrigo Marchiori <ar...@yahoo.it.invalid>
> wrote:
>
> > Hello Damjan, all,
> >
> > On Tue, Apr 26, 2022 at 07:56:22PM +0200, Damjan Jovanovic wrote:
> >
> > > On Mon, Nov 15, 2021 at 9:57 PM Jim Jagielski <ji...@jagunet.com> wrote:
> > >
> > > > I'm gonna look into the serf->(lib)curl option... Since we don't use
> > any
> > > > of the fancy features of serf, I'm thinking that the easy option might
> > be
> > > > best
> > >
> > >
> > >
> > > Hi
> > >
> > > I've ported our WebDAV content provider module from Serf to Curl.
> >
> > I just enhanced the error reporting a bit; I am finding a problem
> > under Linux and I do not really know how to assess it.
> >
> > The problem: if we build AOO on CentOS (that is our reference
> > platform) then Curl will look for CA certificates in
> > /etc/pki/tls/certs/ca-bundle.crt
> >
> > This will fail on openSUSE and probably on Ubuntu as well.
> >
> > It seems that the above path is set at configure time and embedded
> > into Curl's code as #define macros.
> >
> > Is there an ``official'' way to assess this? Like, can we depend on
> > NSS' certificate store as you wrote (quoted below)?
> >
>
> Curl/OpenSSL have an enormous number of options and I am pretty sure it can
> be fixed, but first I need to understand where and how it's failing.
>
> We currently allow it to run with the default CA certificate path, do
> pre-verification on the server's certificate using those CA certificates,
> then call our SSL_VERIFY_PEER function where we override the verification
> result with the certificates from NSS.
Apparently, it is failing before calling our SSL_VERIFY_PEER function.
> If it's failing before reaching our SSL_VERIFY_PEER function, we should be
> able to use Curl's CURLOPT_CAINFO or CURLOPT_CAINFO_BLOB functions to set a
> custom CA certificate path (or in-memory buffer), maybe even an empty
> buffer, so that it proceeds further. ("man CURLOPT_CAINFO", "man
> CURLOPT_CAINFO_BLOB", or "man curl_easy_setopt" and read under the "SSL and
> SECURITY OPTIONS" section.)
So we would need to hard-code and try all possible paths to the CA
bundle on Unix systems?
> With the CURLOPT_CAINFO_BLOB option it might even be possible to skip the
> custom certificate verification we do later, and pre-populate Curl/OpenSSL
> with NSS certificates from the beginning, I just don't know enough about
> NSS to rely on that (eg. if you are using a cryptographic device or smart
> card in NSS, how does that work?). If that option is ok, then we might not
> even need the NSS libraries: recent versions of NSS store all the
> certificates in an SQLite database, which can be accessed with SQLite APIs
> directly, no need to build with or ship the NSS libraries at all.
If I understood correctly [1], a NSS-linked Curl would query NSS by
itself... are we not in this condition?
Thank you in advance and best regards,
--
Arrigo
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)
Posted by Damjan Jovanovic <da...@apache.org>.
On Sun, May 22, 2022 at 2:43 PM Arrigo Marchiori <ar...@yahoo.it.invalid>
wrote:
> Hello Damjan, all,
>
> On Tue, Apr 26, 2022 at 07:56:22PM +0200, Damjan Jovanovic wrote:
>
> > On Mon, Nov 15, 2021 at 9:57 PM Jim Jagielski <ji...@jagunet.com> wrote:
> >
> > > I'm gonna look into the serf->(lib)curl option... Since we don't use
> any
> > > of the fancy features of serf, I'm thinking that the easy option might
> be
> > > best
> >
> >
> >
> > Hi
> >
> > I've ported our WebDAV content provider module from Serf to Curl.
>
> I just enhanced the error reporting a bit; I am finding a problem
> under Linux and I do not really know how to assess it.
>
> The problem: if we build AOO on CentOS (that is our reference
> platform) then Curl will look for CA certificates in
> /etc/pki/tls/certs/ca-bundle.crt
>
> This will fail on openSUSE and probably on Ubuntu as well.
>
> It seems that the above path is set at configure time and embedded
> into Curl's code as #define macros.
>
> Is there an ``official'' way to assess this? Like, can we depend on
> NSS' certificate store as you wrote (quoted below)?
>
Curl/OpenSSL have an enormous number of options and I am pretty sure it can
be fixed, but first I need to understand where and how it's failing.
We currently allow it to run with the default CA certificate path, do
pre-verification on the server's certificate using those CA certificates,
then call our SSL_VERIFY_PEER function where we override the verification
result with the certificates from NSS.
If it's failing before reaching our SSL_VERIFY_PEER function, we should be
able to use Curl's CURLOPT_CAINFO or CURLOPT_CAINFO_BLOB functions to set a
custom CA certificate path (or in-memory buffer), maybe even an empty
buffer, so that it proceeds further. ("man CURLOPT_CAINFO", "man
CURLOPT_CAINFO_BLOB", or "man curl_easy_setopt" and read under the "SSL and
SECURITY OPTIONS" section.)
With the CURLOPT_CAINFO_BLOB option it might even be possible to skip the
custom certificate verification we do later, and pre-populate Curl/OpenSSL
with NSS certificates from the beginning, I just don't know enough about
NSS to rely on that (eg. if you are using a cryptographic device or smart
card in NSS, how does that work?). If that option is ok, then we might not
even need the NSS libraries: recent versions of NSS store all the
certificates in an SQLite database, which can be accessed with SQLite APIs
directly, no need to build with or ship the NSS libraries at all.
I am planning to write a separate email, when I get a chance, about the
cryptography libraries and certificates story.
Regards
Damjan
Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)
Posted by Arrigo Marchiori <ar...@yahoo.it.INVALID>.
Hello Damjan, all,
On Tue, Apr 26, 2022 at 07:56:22PM +0200, Damjan Jovanovic wrote:
> On Mon, Nov 15, 2021 at 9:57 PM Jim Jagielski <ji...@jagunet.com> wrote:
>
> > I'm gonna look into the serf->(lib)curl option... Since we don't use any
> > of the fancy features of serf, I'm thinking that the easy option might be
> > best
>
>
>
> Hi
>
> I've ported our WebDAV content provider module from Serf to Curl.
I just enhanced the error reporting a bit; I am finding a problem
under Linux and I do not really know how to assess it.
The problem: if we build AOO on CentOS (that is our reference
platform) then Curl will look for CA certificates in
/etc/pki/tls/certs/ca-bundle.crt
This will fail on openSUSE and probably on Ubuntu as well.
It seems that the above path is set at configure time and embedded
into Curl's code as #define macros.
Is there an ``official'' way to assess this? Like, can we depend on
NSS' certificate store as you wrote (quoted below)?
[...]
> HTTPS works, with a custom certificate verification function, using our own
> certificate store from NSS and its API (like the Serf code used). A bug was
> discovered (which is in the Serf implementation too) where self-signed
> certificates were being unconditionally rejected; apparently NSS wants to
> see that a copy of that certificate in its certificate chain parameter
> too. Now they work, and the user gets prompted to allow access.
Thank you in advance and best regards,
--
Arrigo
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)
Posted by Matthias Seidel <ma...@hamburg.de>.
Hi Pedro,
Am 28.04.22 um 18:49 schrieb Pedro Lino:
> Hi Damjan, Arrigo, all
>
>> On 04/28/2022 5:11 PM Arrigo Marchiori <ar...@yahoo.it.invalid> wrote:
>> Binary for Linux available here for download:
>> https://home.apache.org/~ardovm/openoffice/linux/openoffice4-serf2curl-2022-04-28-installed.tar.bz2
>> I tried "Open" and entered a https address, that I know is password
>> protected.
>>
>> The current trunk would ask for the password. I got an error message
>> instead:
>>
>>>> Nonexistent object.
>>>> Nonexistent file.
> Same problem under Linux. Maybe it only works on 42X?
>
> I could not test under Windows because there aren't any Windows Dev builds (42X or any other) available at
> https://nightlies.apache.org/openoffice/install/
> so I can't replace the DLLs
Yes, the Windows buildbot is broken for some quite time (looks like
Cygwin was updated without modifying awk/gawk).
But I didn't have the time to open a JIRA ticket and apparently nobody
else is interested in the Windows buildbots.
Regards,
Matthias
>
> Best,
> Pedro
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
> For additional commands, e-mail: dev-help@openoffice.apache.org
>
Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)
Posted by Pedro Lino <pe...@mailbox.org.INVALID>.
Hi Damjan, Arrigo, all
> On 04/28/2022 5:11 PM Arrigo Marchiori <ar...@yahoo.it.invalid> wrote:
> Binary for Linux available here for download:
> https://home.apache.org/~ardovm/openoffice/linux/openoffice4-serf2curl-2022-04-28-installed.tar.bz2
> I tried "Open" and entered a https address, that I know is password
> protected.
>
> The current trunk would ask for the password. I got an error message
> instead:
>
> > > Nonexistent object.
> > > Nonexistent file.
Same problem under Linux. Maybe it only works on 42X?
I could not test under Windows because there aren't any Windows Dev builds (42X or any other) available at
https://nightlies.apache.org/openoffice/install/
so I can't replace the DLLs
Best,
Pedro
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)
Posted by Arrigo Marchiori <ar...@yahoo.it.INVALID>.
Hello Damjan, all,
On Sun, May 15, 2022 at 04:12:14PM +0200, Damjan Jovanovic wrote:
> On Sun, May 15, 2022 at 3:57 PM Arrigo Marchiori <ar...@yahoo.it.invalid>
> wrote:
>
> > Dear All,
> >
> > On Thu, May 12, 2022 at 07:36:49PM +0200, Arrigo Marchiori wrote:
> >
> > > Dear all,
> > >
> > > if nobody objects, I will merge this PR during the week-end.
> > >
> > > Reference: https://github.com/apache/openoffice/pull/146
> >
> > Done.
> >
> > Thank you again, Damjan!
> >
>
> It's a pleasure :).
>
> The next library to update could be zlib, because this version of curl
> > does _not_ compile with the bundled zlib.
> >
>
> I distinctly recall that zlib and curl compiled for me on Windows. Where
> and how is it failing for you?
It's failing on Linux.
If we want to use the bundled zlib, we have to tell the configure
script --with-zlib=$(OUTDIR) .
The configure script would look for:
1- the header file in $(OUTDIR)/include
2- the library in $(OUTDIR)/lib/
But the bundled zlib module installs:
1a- the header file into $(OUTDIR)/inc
2a- the static library as file $(OUTDIR)/lib/libzlib.a
So we have to patch it in a similar way as we did for the openssl paths
(i.e. replicate bb956dfd85428dd08d991a01ee6a6c4e966bf431)
Even in that case, it is not easy to understand if we are using the
bundled zlib or the system-level one. The main cause, IMHO, is that
system-level zlib is surely installed (because of other build
dependencies) and is auto-detected by configure scripts and default
Makefile settings.
To a quick glance, also the "openssl" module seems to link to the
system zlib on Linux.
As the bundled zlib is linked statically, I wonder if invoking ldd on
the built executables and .so files is the only way to check for
proper linking.
I am sorry I do not have more time to get into details by now. I hope
the above makes sense.
Best regards,
--
Arrigo
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)
Posted by Damjan Jovanovic <da...@apache.org>.
On Sun, May 15, 2022 at 3:57 PM Arrigo Marchiori <ar...@yahoo.it.invalid>
wrote:
> Dear All,
>
> On Thu, May 12, 2022 at 07:36:49PM +0200, Arrigo Marchiori wrote:
>
> > Dear all,
> >
> > if nobody objects, I will merge this PR during the week-end.
> >
> > Reference: https://github.com/apache/openoffice/pull/146
>
> Done.
>
> Thank you again, Damjan!
>
It's a pleasure :).
The next library to update could be zlib, because this version of curl
> does _not_ compile with the bundled zlib.
>
I distinctly recall that zlib and curl compiled for me on Windows. Where
and how is it failing for you?
Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)
Posted by Arrigo Marchiori <ar...@yahoo.it.INVALID>.
Dear All,
On Thu, May 12, 2022 at 07:36:49PM +0200, Arrigo Marchiori wrote:
> Dear all,
>
> if nobody objects, I will merge this PR during the week-end.
>
> Reference: https://github.com/apache/openoffice/pull/146
Done.
Thank you again, Damjan!
The next library to update could be zlib, because this version of curl
does _not_ compile with the bundled zlib.
But I guess it is only one more... among many.
Best regards,
--
Arrigo
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)
Posted by Arrigo Marchiori <ar...@yahoo.it.INVALID>.
Dear all,
if nobody objects, I will merge this PR during the week-end.
Reference: https://github.com/apache/openoffice/pull/146
Best regards,
--
Arrigo
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)
Posted by Arrigo Marchiori <ar...@yahoo.it.INVALID>.
Hello All,
replying to myself now...
On Mon, May 02, 2022 at 10:28:51PM +0200, Arrigo Marchiori wrote:
> > > > On Fri, Apr 29, 2022 at 04:53:59AM +0200, Damjan Jovanovic wrote:
> > > [...]
> > > > > When I run your Linux binaries, I also get that error.
> > > > >
> > > > > One problem is libcurl -> openssl.
[...]
> I suggest:
[...]
> 2- we patch curl's configure script to use the bundled OpenSSL
> library, when requested.
PR here: https://github.com/DamjanJovanovic/openoffice/pull/1
This should fix the crash.
Best regards,
--
Arrigo
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)
Posted by Dave Fisher <wa...@apache.org>.
> On May 6, 2022, at 12:56 PM, Arrigo Marchiori <ar...@yahoo.it.INVALID> wrote:
>
> Hello Dave,
>
> On Mon, May 02, 2022 at 05:53:19PM -0700, Dave Fisher wrote:
>
>> Just so that you aren’t the only one here.
>
> I am sure I am not alone... but thank you anyway ;-)
>
>> It will be important to write good release notes about these changes.
>>
>> Users will need to understand clearly that updates to cURL and
>> OpenSSL for CVEs won’t happen immediately unless they build with 2
>> on their own.
>
> I am not sure I understand this. We are just updating/changing bundled
> libraries... Can you please explain better what you mean?
Some users might hear about a CVE on cURL and/or OpenSSL and update their systems thinking that they have also fixed OpenOffice when due to our now bundling of these they would need to wait for an OpenOffice release.
ATB,
Dave
>
> Best regards,
> --
> Arrigo
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
> For additional commands, e-mail: dev-help@openoffice.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)
Posted by Arrigo Marchiori <ar...@yahoo.it.INVALID>.
Hello Dave,
On Mon, May 02, 2022 at 05:53:19PM -0700, Dave Fisher wrote:
> Just so that you aren’t the only one here.
I am sure I am not alone... but thank you anyway ;-)
> It will be important to write good release notes about these changes.
>
> Users will need to understand clearly that updates to cURL and
> OpenSSL for CVEs won’t happen immediately unless they build with 2
> on their own.
I am not sure I understand this. We are just updating/changing bundled
libraries... Can you please explain better what you mean?
Best regards,
--
Arrigo
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)
Posted by Dave Fisher <wa...@apache.org>.
> On May 2, 2022, at 1:28 PM, Arrigo Marchiori <ar...@yahoo.it.INVALID> wrote:
>
> Hello All,
>
> On Sat, Apr 30, 2022 at 10:31:33PM +0200, Arrigo Marchiori wrote:
>
>> Replying to myself again...
>>
>> On Sat, Apr 30, 2022 at 10:03:54PM +0200, Arrigo Marchiori wrote:
>>
>>> Replying to myself...
>>>
>>> On Sat, Apr 30, 2022 at 09:41:04PM +0200, Arrigo Marchiori wrote:
>>>
>>>> Hello Damjan,
>>>>
>>>> On Fri, Apr 29, 2022 at 04:53:59AM +0200, Damjan Jovanovic wrote:
>>> [...]
>>>>> When I run your Linux binaries, I also get that error.
>>>>>
>>>>> One problem is libcurl -> openssl.
>>>>>
>>>>> "ldd ./libcurl.so" shows:
>>>>> libssl.so.10 => not found
>>>>> libcrypto.so.10 => not found
>>> [...]
>>>>>
>>>>> It connects and seems to begin the SSL negotiations, and then probably
>>>>> crashes while doing the verify_callback we set with SSL_CTX_set_verify().
>>>>>
>>>>> I suspect the Curl and/or OpenSSL headers used to build AOO, and their
>>>>> libraries available at runtime, are incompatible.
>>>>>
>>>>> What happens if you run AOO on the same machine it was built on?
>>> [...]
>>>> As I wrote above, according to your suspects, it may be worthwile to
>>>> bundle curl and openssl binaries. I will try to follow this path. It
>>>> should be a matter of parameters to the configure script.
>>>
>>> Curl and openssl are already bundled.
>>>
>>> But Curl is not using the openssl from AOO, but rather from the
>>> system. In other words, it does not honor the environment variable
>>> SYSTEM_OPENSSL=NO set by the AOO build system.
>>>
>>> Curl's configure script looks for the SSL headers and libraries either
>>> from pkgconfig or at a path provided with parameter --with-ssl
>>>
>>> We do not support pkgconfig inside the solver, so we have to rely on
>>> the path.
>>>
>>> But Curl's configure script expects to find a directory structure such as:
>>> - $PREFIX/include/openssl
>>> - $PREFIX/lib[64]
>>> whereas our $OUTDIR has openssl installed into different paths:
>>> - $OUTDIR/inc/external/openssl
>>> - $OUTDIR/lib
>>>
>>> I believe this means we have to patch Curl's configure script.
>>
>> Not only. We also have to actually bundle libssl.so and libcrypto.so
>> because they are not!
>>
>> This means we have to patch:
>>
>> - main/openssl/prj/d.lst to have them included into the solver;
>>
>> - scp2/source/ooo/file_library_ooo.scp to have their Unix names
>> included.
>>
>> This is a separate bug and will be worth its own pull request.
>
> Here it is:
> https://github.com/apache/openoffice/pull/147
>
> I suggest:
>
> 1- we merge the above PR (I will on Friday, if no one vetoes and no
> issues are raised);
>
> 2- we patch curl's configure script to use the bundled OpenSSL
> library, when requested.
>
> Comments are always welcome.
Just so that you aren’t the only one here.
It will be important to write good release notes about these changes.
Users will need to understand clearly that updates to cURL and OpenSSL for CVEs won’t happen immediately unless they build with 2 on their own.
Regards,
Dave
>
> Best regards,
> --
> Arrigo
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
> For additional commands, e-mail: dev-help@openoffice.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)
Posted by Arrigo Marchiori <ar...@yahoo.it.INVALID>.
Hello All,
On Sat, Apr 30, 2022 at 10:31:33PM +0200, Arrigo Marchiori wrote:
> Replying to myself again...
>
> On Sat, Apr 30, 2022 at 10:03:54PM +0200, Arrigo Marchiori wrote:
>
> > Replying to myself...
> >
> > On Sat, Apr 30, 2022 at 09:41:04PM +0200, Arrigo Marchiori wrote:
> >
> > > Hello Damjan,
> > >
> > > On Fri, Apr 29, 2022 at 04:53:59AM +0200, Damjan Jovanovic wrote:
> > [...]
> > > > When I run your Linux binaries, I also get that error.
> > > >
> > > > One problem is libcurl -> openssl.
> > > >
> > > > "ldd ./libcurl.so" shows:
> > > > libssl.so.10 => not found
> > > > libcrypto.so.10 => not found
> > [...]
> > > >
> > > > It connects and seems to begin the SSL negotiations, and then probably
> > > > crashes while doing the verify_callback we set with SSL_CTX_set_verify().
> > > >
> > > > I suspect the Curl and/or OpenSSL headers used to build AOO, and their
> > > > libraries available at runtime, are incompatible.
> > > >
> > > > What happens if you run AOO on the same machine it was built on?
> > [...]
> > > As I wrote above, according to your suspects, it may be worthwile to
> > > bundle curl and openssl binaries. I will try to follow this path. It
> > > should be a matter of parameters to the configure script.
> >
> > Curl and openssl are already bundled.
> >
> > But Curl is not using the openssl from AOO, but rather from the
> > system. In other words, it does not honor the environment variable
> > SYSTEM_OPENSSL=NO set by the AOO build system.
> >
> > Curl's configure script looks for the SSL headers and libraries either
> > from pkgconfig or at a path provided with parameter --with-ssl
> >
> > We do not support pkgconfig inside the solver, so we have to rely on
> > the path.
> >
> > But Curl's configure script expects to find a directory structure such as:
> > - $PREFIX/include/openssl
> > - $PREFIX/lib[64]
> > whereas our $OUTDIR has openssl installed into different paths:
> > - $OUTDIR/inc/external/openssl
> > - $OUTDIR/lib
> >
> > I believe this means we have to patch Curl's configure script.
>
> Not only. We also have to actually bundle libssl.so and libcrypto.so
> because they are not!
>
> This means we have to patch:
>
> - main/openssl/prj/d.lst to have them included into the solver;
>
> - scp2/source/ooo/file_library_ooo.scp to have their Unix names
> included.
>
> This is a separate bug and will be worth its own pull request.
Here it is:
https://github.com/apache/openoffice/pull/147
I suggest:
1- we merge the above PR (I will on Friday, if no one vetoes and no
issues are raised);
2- we patch curl's configure script to use the bundled OpenSSL
library, when requested.
Comments are always welcome.
Best regards,
--
Arrigo
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)
Posted by Arrigo Marchiori <ar...@yahoo.it.INVALID>.
Replying to myself again...
On Sat, Apr 30, 2022 at 10:03:54PM +0200, Arrigo Marchiori wrote:
> Replying to myself...
>
> On Sat, Apr 30, 2022 at 09:41:04PM +0200, Arrigo Marchiori wrote:
>
> > Hello Damjan,
> >
> > On Fri, Apr 29, 2022 at 04:53:59AM +0200, Damjan Jovanovic wrote:
> [...]
> > > When I run your Linux binaries, I also get that error.
> > >
> > > One problem is libcurl -> openssl.
> > >
> > > "ldd ./libcurl.so" shows:
> > > libssl.so.10 => not found
> > > libcrypto.so.10 => not found
> [...]
> > >
> > > It connects and seems to begin the SSL negotiations, and then probably
> > > crashes while doing the verify_callback we set with SSL_CTX_set_verify().
> > >
> > > I suspect the Curl and/or OpenSSL headers used to build AOO, and their
> > > libraries available at runtime, are incompatible.
> > >
> > > What happens if you run AOO on the same machine it was built on?
> [...]
> > As I wrote above, according to your suspects, it may be worthwile to
> > bundle curl and openssl binaries. I will try to follow this path. It
> > should be a matter of parameters to the configure script.
>
> Curl and openssl are already bundled.
>
> But Curl is not using the openssl from AOO, but rather from the
> system. In other words, it does not honor the environment variable
> SYSTEM_OPENSSL=NO set by the AOO build system.
>
> Curl's configure script looks for the SSL headers and libraries either
> from pkgconfig or at a path provided with parameter --with-ssl
>
> We do not support pkgconfig inside the solver, so we have to rely on
> the path.
>
> But Curl's configure script expects to find a directory structure such as:
> - $PREFIX/include/openssl
> - $PREFIX/lib[64]
> whereas our $OUTDIR has openssl installed into different paths:
> - $OUTDIR/inc/external/openssl
> - $OUTDIR/lib
>
> I believe this means we have to patch Curl's configure script.
Not only. We also have to actually bundle libssl.so and libcrypto.so
because they are not!
This means we have to patch:
- main/openssl/prj/d.lst to have them included into the solver;
- scp2/source/ooo/file_library_ooo.scp to have their Unix names
included.
This is a separate bug and will be worth its own pull request.
Comments are welcome.
Best regards,
--
Arrigo
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)
Posted by Arrigo Marchiori <ar...@yahoo.it.INVALID>.
Replying to myself...
On Sat, Apr 30, 2022 at 09:41:04PM +0200, Arrigo Marchiori wrote:
> Hello Damjan,
>
> On Fri, Apr 29, 2022 at 04:53:59AM +0200, Damjan Jovanovic wrote:
[...]
> > When I run your Linux binaries, I also get that error.
> >
> > One problem is libcurl -> openssl.
> >
> > "ldd ./libcurl.so" shows:
> > libssl.so.10 => not found
> > libcrypto.so.10 => not found
[...]
> >
> > It connects and seems to begin the SSL negotiations, and then probably
> > crashes while doing the verify_callback we set with SSL_CTX_set_verify().
> >
> > I suspect the Curl and/or OpenSSL headers used to build AOO, and their
> > libraries available at runtime, are incompatible.
> >
> > What happens if you run AOO on the same machine it was built on?
[...]
> As I wrote above, according to your suspects, it may be worthwile to
> bundle curl and openssl binaries. I will try to follow this path. It
> should be a matter of parameters to the configure script.
Curl and openssl are already bundled.
But Curl is not using the openssl from AOO, but rather from the
system. In other words, it does not honor the environment variable
SYSTEM_OPENSSL=NO set by the AOO build system.
Curl's configure script looks for the SSL headers and libraries either
from pkgconfig or at a path provided with parameter --with-ssl
We do not support pkgconfig inside the solver, so we have to rely on
the path.
But Curl's configure script expects to find a directory structure such as:
- $PREFIX/include/openssl
- $PREFIX/lib[64]
whereas our $OUTDIR has openssl installed into different paths:
- $OUTDIR/inc/external/openssl
- $OUTDIR/lib
I believe this means we have to patch Curl's configure script.
Best regards,
--
Arrigo
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)
Posted by Arrigo Marchiori <ar...@yahoo.it.INVALID>.
Hello Damjan,
On Fri, Apr 29, 2022 at 04:53:59AM +0200, Damjan Jovanovic wrote:
> On Thu, Apr 28, 2022 at 8:46 PM Arrigo Marchiori <ar...@yahoo.it.invalid>
> wrote:
>
> > Hello Damjan,
> >
> > On Thu, Apr 28, 2022 at 06:53:43PM +0200, Damjan Jovanovic wrote:
> >
> > > On Thu, Apr 28, 2022 at 6:12 PM Arrigo Marchiori <ardovm@yahoo.it.invalid
> > >
> > > wrote:
> > >
> > > > Dear Damjan, All,
> > > >
> > > > On Tue, Apr 26, 2022 at 07:56:22PM +0200, Damjan Jovanovic wrote:
> > > >
> > > > > On Mon, Nov 15, 2021 at 9:57 PM Jim Jagielski <ji...@jagunet.com>
> > wrote:
> > > > >
> > > > > > I'm gonna look into the serf->(lib)curl option... Since we don't
> > use
> > > > any
> > > > > > of the fancy features of serf, I'm thinking that the easy option
> > might
> > > > be
> > > > > > best
> > > > >
> > > > >
> > > > >
> > > > > Hi
> > > > >
> > > > > I've ported our WebDAV content provider module from Serf to Curl.
> > > >
> > > > Binary for Linux available here for download:
> > > >
> > > >
> > https://home.apache.org/~ardovm/openoffice/linux/openoffice4-serf2curl-2022-04-28-installed.tar.bz2
> > > >
> > > > I understand from your previous message that you don't really like
> > > > GitHub, so I am writing here.
> > > >
> > > > [...]
> > > > > STATUS
> > > > >
> > > > > It builds and works well on FreeBSD and Windows.
> > > > >
> > > > > Most of the code was reused, and all the operations and semantics
> > > > > previously present with Serf, should have been preserved.
> > > > >
> > > > > Browsing WebDAV files and directories, loading files, overwriting
> > them
> > > > > ("Save"), creating them ("Save As"), renaming and deleting them, all
> > > > works.
> > > >
> > > > I am testing the binary for Linux linked above.
> > > >
> > > > I tried "Open" and entered a https address, that I know is password
> > > > protected.
> > > >
> > > > The current trunk would ask for the password. I got an error message
> > > > instead:
> > > >
> > > > > > Nonexistent object.
> > > > > > Nonexistent file.
> > > >
> > > > The address I tried to open is in the form https://host.domain:port/
> > > >
> > > > I tried to substitute "https" with "davs" and I got the same error.
> > > >
> > > > Maybe something is going wrong in the Linux build?
> > > >
> > > > I will now begin recompiling with debugging symbols enabled. Please
> > > > let me know how I can help.
> > > >
> > >
> > > That's not good :(.
> > >
> > > Set your macro security to "Medium", open the spreadsheet I've attached,
> > > and run the "RunMe" Basic macro. That should enable logging to the
> > console
> > > at the finest level of detail. Then exit and re-run AOO like this:
> > > soffice 2>&1 | tee /tmp/log.txt
> > > and examine the console output interactively as you use AOO, and/or check
> > > the log file afterwards. It should have everything, our logging, HTTP
> > > request and response headers and bodies, Curl's internal logging.
> >
> > I can't get to that point.
> >
> > I fired gdb and it seems that I end up into the blind
> > catch (Exception const &)
> > block in file ucb/source/core/provprox.cxx:361
> >
> > Method UcbContentProviderProxy::getContentProvider() in fact is called
> > many times, but it only fails when I enter the https url in the "Open"
> > dialog box and press ENTER.
> >
> > Sorry this is all the debugging I can do for today. I hope it helps.
> >
> >
> When I run your Linux binaries, I also get that error.
>
> One problem is libcurl -> openssl.
>
> "ldd ./libcurl.so" shows:
> libssl.so.10 => not found
> libcrypto.so.10 => not found
This probably means that we have to ship our own binaries of openssl
on Linux.
> When I rename AOO's libcurl.so so that it's forced to use my system libcurl
> instead, it proceeds further, but runs into another problem:
>
> Thread 1 "soffice.bin" received signal SIGSEGV, Segmentation fault.
> 0x0000000000000001 in ?? ()
[...snip of stack trace with no debugging symbols...]
>
> It gets far enough to log and produces some output before the crash:
>
> 1 1 2022-04-29 02:24:24.03 CurlSession::CurlSession with
> URL https://10.0.2.2:82/files/
> 2 1 2022-04-29 02:24:24.03 Not using a proxy server
> 3 1 2022-04-29 02:24:24.03 PROPFIND line 914
> 4 1 2022-04-29 02:24:24.03 [CurlINFO ] Trying 10.0.2.2:82
> ...
> 5 1 2022-04-29 02:24:24.03 [CurlINFO ] TCP_NODELAY set
> 6 1 2022-04-29 02:24:24.03 [CurlINFO ] Connected to
> 10.0.2.2 (10.0.2.2) port 82 (#0)
> 7 1 2022-04-29 02:24:24.03 [CurlINFO ] ALPN, offering h2
> 8 1 2022-04-29 02:24:24.03 [CurlINFO ] ALPN, offering
> http/1.1
> 9 1 2022-04-29 02:24:24.04 [CurlINFO ] successfully set
> certificate verify locations:
> 10 1 2022-04-29 02:24:24.04 [CurlINFO ] CAfile:
> /etc/ssl/certs/ca-certificates.crt
> CApath: /etc/ssl/certs
>
> It connects and seems to begin the SSL negotiations, and then probably
> crashes while doing the verify_callback we set with SSL_CTX_set_verify().
>
> I suspect the Curl and/or OpenSSL headers used to build AOO, and their
> libraries available at runtime, are incompatible.
>
> What happens if you run AOO on the same machine it was built on?
I get more or less the same outcome as you.
To be clear: I rebuilt on ``bare metal'' (no Docker) and ran the
binary straight away.
The fastest way to crash is to check for updates.
Stack trace: [with debugging symbols only for AOO-provided libraries]
#0 0x00007fff00000001 in ()
#1 0x00007fffc3d96172 in () at /usr/lib64/libssl.so.1.1
#2 0x00007fffc3d96772 in () at /usr/lib64/libssl.so.1.1
#3 0x00007fffc3dc7ac4 in () at /usr/lib64/libssl.so.1.1
#4 0x00007fffc3dbed06 in () at /usr/lib64/libssl.so.1.1
#5 0x00007fffc3daaf44 in SSL_do_handshake () at /usr/lib64/libssl.so.1.1
#6 0x00007fffcc199eec in ossl_connect_step2 ()
at /workspace/AOO/main/instsetoo_native/unxlngx6.pro/Apache_OpenOffice/installed/install/en-US/openoffice4/program/../program/libcurl.so.4
#7 0x00007fffcc19c411 in ossl_connect_common ()
at /workspace/AOO/main/instsetoo_native/unxlngx6.pro/Apache_OpenOffice/installed/install/en-US/openoffice4/program/../program/libcurl.so.4
#8 0x00007fffcc19d2b7 in Curl_ssl_connect_nonblocking ()
at /workspace/AOO/main/instsetoo_native/unxlngx6.pro/Apache_OpenOffice/installed/install/en-US/openoffice4/program/../program/libcurl.so.4
#9 0x00007fffcc169302 in https_connecting ()
at /workspace/AOO/main/instsetoo_native/unxlngx6.pro/Apache_OpenOffice/installed/install/en-US/openoffice4/program/../program/libcurl.so.4
#10 0x00007fffcc16a7ab in Curl_http_connect ()
at /workspace/AOO/main/instsetoo_native/unxlngx6.pro/Apache_OpenOffice/installed/install/en-US/openoffice4/program/../program/libcurl.so.4
#11 0x00007fffcc1796f2 in multi_runsingle ()
at /workspace/AOO/main/instsetoo_native/unxlngx6.pro/Apache_OpenOffice/installed/install/en-US/openoffice4/program/../program/libcurl.so.4
#12 0x00007fffcc17a7db in curl_multi_perform ()
at /workspace/AOO/main/instsetoo_native/unxlngx6.pro/Apache_OpenOffice/installed/install/en-US/openoffice4/program/../program/libcurl.so.4
#13 0x00007fffcc159833 in curl_easy_perform ()
at /workspace/AOO/main/instsetoo_native/unxlngx6.pro/Apache_OpenOffice/installed/install/en-US/openoffice4/program/../program/libcurl.so.4
#14 0x00007fff9aff4d38 in http_dav_ucp::CurlRequest::perform() (this=this@entry=0x7fff9bdee500)
at /workspace/AOO/main/ucb/source/ucp/webdav/CurlRequest.cxx:266
#15 0x00007fff9aff4fc6 in http_dav_ucp::CurlRequest::get(http_dav_ucp::CurlUri, rtl::OUString)
(this=this@entry=0x7fff9bdee500, uri=..., path=...)
at /workspace/AOO/main/ucb/source/ucp/webdav/CurlRequest.cxx:189
#16 0x00007fff9aff0d57 in http_dav_ucp::CurlSession::GET(rtl::OUString const&, std::vector<rtl::OUString, std::allocator<rtl::OUString> > const&, http_dav_ucp::DAVResource&, http_dav_ucp::DAVRequestEnvironment const&)
(this=this@entry=0x7fffa066b8b0, inPath=..., inHeaderNames=..., ioResource=..., rEnv=...)
at /workspace/AOO/main/ucb/source/ucp/webdav/CurlSession.cxx:1098
#17 0x00007fff9afe1229 in http_dav_ucp::DAVResourceAccess::GET(std::vector<rtl::OUString, std::allocator<rtl::OUString> > const&, http_dav_ucp::DAVResource&, com::sun::star::uno::Reference<com::sun::star::ucb::XCommandEnvironment> const&)
(this=0x7fffa0677b50, rHeaderNames=..., rResource=..., xEnv=...)
at /workspace/AOO/main/ucb/source/ucp/webdav/DAVResourceAccess.cxx:443
#18 0x00007fff9afcd03b in http_dav_ucp::Content::open(com::sun::star::ucb::OpenCommandArgument2 const&, com::sun::star::uno::Reference<com::sun::star::ucb::XCommandEnvironment> const&) (this=this@entry=0x7fffa066b7b0, rArg=..., xEnv=...)
at /workspace/AOO/main/ucb/source/ucp/webdav/webdavcontent.cxx:2282
As I wrote above, according to your suspects, it may be worthwile to
bundle curl and openssl binaries. I will try to follow this path. It
should be a matter of parameters to the configure script.
Best regards,
--
Arrigo
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)
Posted by Damjan Jovanovic <da...@apache.org>.
On Thu, Apr 28, 2022 at 8:46 PM Arrigo Marchiori <ar...@yahoo.it.invalid>
wrote:
> Hello Damjan,
>
> On Thu, Apr 28, 2022 at 06:53:43PM +0200, Damjan Jovanovic wrote:
>
> > On Thu, Apr 28, 2022 at 6:12 PM Arrigo Marchiori <ardovm@yahoo.it.invalid
> >
> > wrote:
> >
> > > Dear Damjan, All,
> > >
> > > On Tue, Apr 26, 2022 at 07:56:22PM +0200, Damjan Jovanovic wrote:
> > >
> > > > On Mon, Nov 15, 2021 at 9:57 PM Jim Jagielski <ji...@jagunet.com>
> wrote:
> > > >
> > > > > I'm gonna look into the serf->(lib)curl option... Since we don't
> use
> > > any
> > > > > of the fancy features of serf, I'm thinking that the easy option
> might
> > > be
> > > > > best
> > > >
> > > >
> > > >
> > > > Hi
> > > >
> > > > I've ported our WebDAV content provider module from Serf to Curl.
> > >
> > > Binary for Linux available here for download:
> > >
> > >
> https://home.apache.org/~ardovm/openoffice/linux/openoffice4-serf2curl-2022-04-28-installed.tar.bz2
> > >
> > > I understand from your previous message that you don't really like
> > > GitHub, so I am writing here.
> > >
> > > [...]
> > > > STATUS
> > > >
> > > > It builds and works well on FreeBSD and Windows.
> > > >
> > > > Most of the code was reused, and all the operations and semantics
> > > > previously present with Serf, should have been preserved.
> > > >
> > > > Browsing WebDAV files and directories, loading files, overwriting
> them
> > > > ("Save"), creating them ("Save As"), renaming and deleting them, all
> > > works.
> > >
> > > I am testing the binary for Linux linked above.
> > >
> > > I tried "Open" and entered a https address, that I know is password
> > > protected.
> > >
> > > The current trunk would ask for the password. I got an error message
> > > instead:
> > >
> > > > > Nonexistent object.
> > > > > Nonexistent file.
> > >
> > > The address I tried to open is in the form https://host.domain:port/
> > >
> > > I tried to substitute "https" with "davs" and I got the same error.
> > >
> > > Maybe something is going wrong in the Linux build?
> > >
> > > I will now begin recompiling with debugging symbols enabled. Please
> > > let me know how I can help.
> > >
> >
> > That's not good :(.
> >
> > Set your macro security to "Medium", open the spreadsheet I've attached,
> > and run the "RunMe" Basic macro. That should enable logging to the
> console
> > at the finest level of detail. Then exit and re-run AOO like this:
> > soffice 2>&1 | tee /tmp/log.txt
> > and examine the console output interactively as you use AOO, and/or check
> > the log file afterwards. It should have everything, our logging, HTTP
> > request and response headers and bodies, Curl's internal logging.
>
> I can't get to that point.
>
> I fired gdb and it seems that I end up into the blind
> catch (Exception const &)
> block in file ucb/source/core/provprox.cxx:361
>
> Method UcbContentProviderProxy::getContentProvider() in fact is called
> many times, but it only fails when I enter the https url in the "Open"
> dialog box and press ENTER.
>
> Sorry this is all the debugging I can do for today. I hope it helps.
>
>
When I run your Linux binaries, I also get that error.
One problem is libcurl -> openssl.
"ldd ./libcurl.so" shows:
libssl.so.10 => not found
libcrypto.so.10 => not found
When I rename AOO's libcurl.so so that it's forced to use my system libcurl
instead, it proceeds further, but runs into another problem:
Thread 1 "soffice.bin" received signal SIGSEGV, Segmentation fault.
0x0000000000000001 in ?? ()
(gdb) bt
#0 0x0000000000000001 in ?? ()
#1 0x00007fffd6d52a76 in ?? () from /lib/x86_64-linux-gnu/libssl.so.1.1
#2 0x00007fffd6d52f75 in ?? () from /lib/x86_64-linux-gnu/libssl.so.1.1
#3 0x00007fffd6d8505d in ?? () from /lib/x86_64-linux-gnu/libssl.so.1.1
#4 0x00007fffd6d7c37e in ?? () from /lib/x86_64-linux-gnu/libssl.so.1.1
#5 0x00007fffd6d67d98 in SSL_do_handshake () from
/lib/x86_64-linux-gnu/libssl.so.1.1
#6 0x00007fffe4154e40 in ?? () from /lib/x86_64-linux-gnu/libcurl.so.4
#7 0x00007fffe41571a2 in ?? () from /lib/x86_64-linux-gnu/libcurl.so.4
#8 0x00007fffe415814f in ?? () from /lib/x86_64-linux-gnu/libcurl.so.4
#9 0x00007fffe4103296 in ?? () from /lib/x86_64-linux-gnu/libcurl.so.4
#10 0x00007fffe4104d13 in ?? () from /lib/x86_64-linux-gnu/libcurl.so.4
#11 0x00007fffe412597d in ?? () from /lib/x86_64-linux-gnu/libcurl.so.4
#12 0x00007fffe4126a11 in curl_multi_perform () from
/lib/x86_64-linux-gnu/libcurl.so.4
#13 0x00007fffe411ce8b in curl_easy_perform () from
/lib/x86_64-linux-gnu/libcurl.so.4
#14 0x00007fffc3c1c972 in ?? () from
/home/user/openoffice4/program/../program/libucpdav1.so
#15 0x00007fffc3c1d02a in ?? () from
/home/user/openoffice4/program/../program/libucpdav1.so
#16 0x00007fffc3c13e37 in ?? () from
/home/user/openoffice4/program/../program/libucpdav1.so
#17 0x00007fffc3c18c70 in ?? () from
/home/user/openoffice4/program/../program/libucpdav1.so
#18 0x00007fffc3c08586 in ?? () from
/home/user/openoffice4/program/../program/libucpdav1.so
#19 0x00007fffc3bf25c2 in ?? () from
/home/user/openoffice4/program/../program/libucpdav1.so
#20 0x00007fffc3bf2b34 in ?? () from
/home/user/openoffice4/program/../program/libucpdav1.so
#21 0x00007fffc3bf7681 in ?? () from
/home/user/openoffice4/program/../program/libucpdav1.so
#22 0x00007ffff48380e0 in ?? () from
/home/user/openoffice4/program/libucbhelpergcc3.so
#23 0x00007ffff483835a in
ucbhelper::Content::getPropertyValuesInterface(com::sun::star::uno::Sequence<rtl::OUString>
const&) () from /home/user/openoffice4/program/libucbhelpergcc3.so
#24 0x00007ffff48383db in
ucbhelper::Content::getPropertyValues(com::sun::star::uno::Sequence<rtl::OUString>
const&) () from /home/user/openoffice4/program/libucbhelpergcc3.so
#25 0x00007ffff483851b in
ucbhelper::Content::getPropertyValue(rtl::OUString const&) () from
/home/user/openoffice4/program/libucbhelpergcc3.so
#26 0x00007ffff48386bd in ucbhelper::Content::isFolder() () from
/home/user/openoffice4/program/libucbhelpergcc3.so
It gets far enough to log and produces some output before the crash:
1 1 2022-04-29 02:24:24.03 CurlSession::CurlSession with
URL https://10.0.2.2:82/files/
2 1 2022-04-29 02:24:24.03 Not using a proxy server
3 1 2022-04-29 02:24:24.03 PROPFIND line 914
4 1 2022-04-29 02:24:24.03 [CurlINFO ] Trying 10.0.2.2:82
...
5 1 2022-04-29 02:24:24.03 [CurlINFO ] TCP_NODELAY set
6 1 2022-04-29 02:24:24.03 [CurlINFO ] Connected to
10.0.2.2 (10.0.2.2) port 82 (#0)
7 1 2022-04-29 02:24:24.03 [CurlINFO ] ALPN, offering h2
8 1 2022-04-29 02:24:24.03 [CurlINFO ] ALPN, offering
http/1.1
9 1 2022-04-29 02:24:24.04 [CurlINFO ] successfully set
certificate verify locations:
10 1 2022-04-29 02:24:24.04 [CurlINFO ] CAfile:
/etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
It connects and seems to begin the SSL negotiations, and then probably
crashes while doing the verify_callback we set with SSL_CTX_set_verify().
I suspect the Curl and/or OpenSSL headers used to build AOO, and their
libraries available at runtime, are incompatible.
What happens if you run AOO on the same machine it was built on?
Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)
Posted by Arrigo Marchiori <ar...@yahoo.it.INVALID>.
Hello Damjan,
On Thu, Apr 28, 2022 at 06:53:43PM +0200, Damjan Jovanovic wrote:
> On Thu, Apr 28, 2022 at 6:12 PM Arrigo Marchiori <ar...@yahoo.it.invalid>
> wrote:
>
> > Dear Damjan, All,
> >
> > On Tue, Apr 26, 2022 at 07:56:22PM +0200, Damjan Jovanovic wrote:
> >
> > > On Mon, Nov 15, 2021 at 9:57 PM Jim Jagielski <ji...@jagunet.com> wrote:
> > >
> > > > I'm gonna look into the serf->(lib)curl option... Since we don't use
> > any
> > > > of the fancy features of serf, I'm thinking that the easy option might
> > be
> > > > best
> > >
> > >
> > >
> > > Hi
> > >
> > > I've ported our WebDAV content provider module from Serf to Curl.
> >
> > Binary for Linux available here for download:
> >
> > https://home.apache.org/~ardovm/openoffice/linux/openoffice4-serf2curl-2022-04-28-installed.tar.bz2
> >
> > I understand from your previous message that you don't really like
> > GitHub, so I am writing here.
> >
> > [...]
> > > STATUS
> > >
> > > It builds and works well on FreeBSD and Windows.
> > >
> > > Most of the code was reused, and all the operations and semantics
> > > previously present with Serf, should have been preserved.
> > >
> > > Browsing WebDAV files and directories, loading files, overwriting them
> > > ("Save"), creating them ("Save As"), renaming and deleting them, all
> > works.
> >
> > I am testing the binary for Linux linked above.
> >
> > I tried "Open" and entered a https address, that I know is password
> > protected.
> >
> > The current trunk would ask for the password. I got an error message
> > instead:
> >
> > > > Nonexistent object.
> > > > Nonexistent file.
> >
> > The address I tried to open is in the form https://host.domain:port/
> >
> > I tried to substitute "https" with "davs" and I got the same error.
> >
> > Maybe something is going wrong in the Linux build?
> >
> > I will now begin recompiling with debugging symbols enabled. Please
> > let me know how I can help.
> >
>
> That's not good :(.
>
> Set your macro security to "Medium", open the spreadsheet I've attached,
> and run the "RunMe" Basic macro. That should enable logging to the console
> at the finest level of detail. Then exit and re-run AOO like this:
> soffice 2>&1 | tee /tmp/log.txt
> and examine the console output interactively as you use AOO, and/or check
> the log file afterwards. It should have everything, our logging, HTTP
> request and response headers and bodies, Curl's internal logging.
I can't get to that point.
I fired gdb and it seems that I end up into the blind
catch (Exception const &)
block in file ucb/source/core/provprox.cxx:361
Method UcbContentProviderProxy::getContentProvider() in fact is called
many times, but it only fails when I enter the https url in the "Open"
dialog box and press ENTER.
Sorry this is all the debugging I can do for today. I hope it helps.
Best regards,
--
Arrigo
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)
Posted by Damjan Jovanovic <da...@apache.org>.
On Thu, Apr 28, 2022 at 6:12 PM Arrigo Marchiori <ar...@yahoo.it.invalid>
wrote:
> Dear Damjan, All,
>
> On Tue, Apr 26, 2022 at 07:56:22PM +0200, Damjan Jovanovic wrote:
>
> > On Mon, Nov 15, 2021 at 9:57 PM Jim Jagielski <ji...@jagunet.com> wrote:
> >
> > > I'm gonna look into the serf->(lib)curl option... Since we don't use
> any
> > > of the fancy features of serf, I'm thinking that the easy option might
> be
> > > best
> >
> >
> >
> > Hi
> >
> > I've ported our WebDAV content provider module from Serf to Curl.
>
> Binary for Linux available here for download:
>
> https://home.apache.org/~ardovm/openoffice/linux/openoffice4-serf2curl-2022-04-28-installed.tar.bz2
>
> I understand from your previous message that you don't really like
> GitHub, so I am writing here.
>
> [...]
> > STATUS
> >
> > It builds and works well on FreeBSD and Windows.
> >
> > Most of the code was reused, and all the operations and semantics
> > previously present with Serf, should have been preserved.
> >
> > Browsing WebDAV files and directories, loading files, overwriting them
> > ("Save"), creating them ("Save As"), renaming and deleting them, all
> works.
>
> I am testing the binary for Linux linked above.
>
> I tried "Open" and entered a https address, that I know is password
> protected.
>
> The current trunk would ask for the password. I got an error message
> instead:
>
> > > Nonexistent object.
> > > Nonexistent file.
>
> The address I tried to open is in the form https://host.domain:port/
>
> I tried to substitute "https" with "davs" and I got the same error.
>
> Maybe something is going wrong in the Linux build?
>
> I will now begin recompiling with debugging symbols enabled. Please
> let me know how I can help.
>
That's not good :(.
Set your macro security to "Medium", open the spreadsheet I've attached,
and run the "RunMe" Basic macro. That should enable logging to the console
at the finest level of detail. Then exit and re-run AOO like this:
soffice 2>&1 | tee /tmp/log.txt
and examine the console output interactively as you use AOO, and/or check
the log file afterwards. It should have everything, our logging, HTTP
request and response headers and bodies, Curl's internal logging.
If you can't see anything obviously wrong, send me that log file, but audit
it carefully first, it will probably contain your password in plaintext!
>
> [...]
> > P.S. APACHE 2.4 SETUP FOR TESTING
> [...]
>
> I still have to try this. Thank you for this tutorial!!
>
>
Pleasure :).
Re: WebDAV module ported from serf to curl; curl using openssl and zlib (was: Re: Openssl, serf and curl)
Posted by Arrigo Marchiori <ar...@yahoo.it.INVALID>.
Dear Damjan, All,
On Tue, Apr 26, 2022 at 07:56:22PM +0200, Damjan Jovanovic wrote:
> On Mon, Nov 15, 2021 at 9:57 PM Jim Jagielski <ji...@jagunet.com> wrote:
>
> > I'm gonna look into the serf->(lib)curl option... Since we don't use any
> > of the fancy features of serf, I'm thinking that the easy option might be
> > best
>
>
>
> Hi
>
> I've ported our WebDAV content provider module from Serf to Curl.
Binary for Linux available here for download:
https://home.apache.org/~ardovm/openoffice/linux/openoffice4-serf2curl-2022-04-28-installed.tar.bz2
I understand from your previous message that you don't really like
GitHub, so I am writing here.
[...]
> STATUS
>
> It builds and works well on FreeBSD and Windows.
>
> Most of the code was reused, and all the operations and semantics
> previously present with Serf, should have been preserved.
>
> Browsing WebDAV files and directories, loading files, overwriting them
> ("Save"), creating them ("Save As"), renaming and deleting them, all works.
I am testing the binary for Linux linked above.
I tried "Open" and entered a https address, that I know is password
protected.
The current trunk would ask for the password. I got an error message
instead:
> > Nonexistent object.
> > Nonexistent file.
The address I tried to open is in the form https://host.domain:port/
I tried to substitute "https" with "davs" and I got the same error.
Maybe something is going wrong in the Linux build?
I will now begin recompiling with debugging symbols enabled. Please
let me know how I can help.
[...]
> P.S. APACHE 2.4 SETUP FOR TESTING
[...]
I still have to try this. Thank you for this tutorial!!
Best regards,
--
Arrigo
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org