You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by GitBox <gi...@apache.org> on 2020/07/03 08:44:46 UTC
[GitHub] [cloudstack] VincentHermes opened a new issue #4199: HTTPS does not initiate
VincentHermes opened a new issue #4199:
URL: https://github.com/apache/cloudstack/issues/4199
#### ISSUE TYPE
* Other / HTTPS
#### COMPONENT NAME
* Webserver
#### CLOUDSTACK VERSION
* 4.14
#### CONFIGURATION
* server.properties:
`https.enable=true`
`https.port=8443`
`https.keystore=/etc/cloudstack/management/thecurrentkeystore.pkcs12`
`https.keystore.password=currentkeystorepassword`
* Firewall disabled
#### OS / ENVIRONMENT
* CentOS7
* MGMT Server on VMWare
* Cloudstack 4.14
* Java 11
* OpenSSL 1.0.2k-fips
#### SUMMARY
We are not able to access the 4.14 Webserver over HTTPS after upgrading from a functioning 4.13. The Webserver seems to not send anything back. If we curl the https it just loads infinitely:
`# curl -v https://localhost:8443/client`
`* About to connect() to localhost port 8443 (#0)`
`* Trying ::1...`
`* Connected to localhost (::1) port 8443 (#0)`
`* Initializing NSS with certpath: sql:/etc/pki/nssdb`
`* CAfile: /etc/pki/tls/certs/ca-bundle.crt`
` CApath: none`
`....(waits)`
Previous investigations regarding networking etc can be seen here:
https://lists.apache.org/thread.html/r50fa6f94dae308a598eb2eeb738f4325e29814d8da83de8558bccfb2%40%3Cusers.cloudstack.apache.org%3E
_Using a self-signed certificate actually works, only when using an adequate certificate (wildcard in our case) it stops functioning_
#### Commands used:
_Combine Files_
`cat key.key servercert.crt intermediate.crt root.crt > combined.crt`
_Create Keystore_
`openssl pkcs12 -in combined.crt -export -out combined.pkcs12`
_Import Keystore_
`keytool -importkeystore -srckeystore combined.pkcs12 -srcstoretype PKCS12 -destkeystore /etc/cloudstack/management/combined.pkcs12 -deststoretype pkcs12`
_Then change https.keystore= and https.keystore.password= accordingly and restart cloudstack-management_
#### Logs found:
`2020-06-29 12:01:02,052 INFO [o.e.j.s.h.ContextHandler] (main:null) (logid:) Started o.e.j.w.WebAppContext@311bf055{/client,file:///usr/share/cloudstack-management/webapp/,AVAILABLE}{/usr/share/cloudstack-management/webapp}`
`2020-06-29 12:01:02,053 INFO [o.e.j.s.h.ContextHandler] (main:null) (logid:) Started o.e.j.s.h.MovedContextHandler@451001e5{/,null,AVAILABLE}`
`2020-06-29 12:01:02,076 INFO [o.e.j.s.AbstractConnector] (main:null) (logid:) Started ServerConnector@6f46426d{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}`
`2020-06-29 12:01:02,090 INFO [o.e.j.u.s.SslContextFactory] (main:null) (logid:) x509=X509@25c6abfa(1,h=[our acual domain name],w=[our domain name again]) for SslContextFactory@4991c0f7[provider=null,keyStore=file:///etc/cloudstack/management/combined.pkcs12,trustStore=null]`
#### Things tried:
* Tested Firefox, Chrome, Edge, IE
* Cache cleared / Private Mode
* Multiple Client Systems
* Numerous combinations of key-cert-intermediate-root when generating the pkcs12
* Keystore only with key and server certificate without any CA's
* Different Certificate Vendors (2 different Wildcard Certificates, Sectigo and Digicert/RapidSSL)
* Generated Certificate via internal Domains Certificate Authority
* Checked all certificate combinations via certutil in Windows Powershell
* Changed keystore password to minimal ones without special characters (e.g. 123456)
* Changed the `https.port` to any other Port in `server.properties`
* Switched back to java-1.8.0 - of course management server failed to start
* Uploaded certificate chain via 8080 Web GUI - Found working combination of cert-intermediate-root to work with by opening console proxy in separate window and checking certificate validity
* Definitely working combination (for firefox) gets the same outcome when being used as pkcs12
* Updating to a newer Openssl Version fails because its the newest for CentOS7
* Just in case it has an impact - Outcommented `jdk.tls.disabledAlgorithms` in `java.security.ciphers`
#### Further Investigations:
* If we change the keystore password to nonsense it makes no difference
* If we change the keystore name or path to nonsense it almost instantly says that the Website is unreachable and does not load infinitely and curl says Connection Refused
* Fresh Install shows the same behaviour
#### STEPS TO REPRODUCE
* Install Cloudstack 4.14 on CentOS7 (No need to configure Infrastructure)
* Create Keystore and configure HTTPS
* Try to get https://mymgmtserver.mydomain.de:8443/client to respond
#### EXPECTED RESULTS
* A response from 8443 as it has been before the upgrade or at least an SSL error regarding a wrong certificate
#### ACTUAL RESULTS
* Webserver not responding / loading forever no matter what certificate is used
* Actually it isn't really "not responding" as it is in fact trying to communicate but nothing happens
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd closed issue #4199: HTTPS does not initiate
Posted by GitBox <gi...@apache.org>.
rhtyd closed issue #4199:
URL: https://github.com/apache/cloudstack/issues/4199
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] dhslove commented on issue #4199: HTTPS does not initiate
Posted by GitBox <gi...@apache.org>.
dhslove commented on issue #4199:
URL: https://github.com/apache/cloudstack/issues/4199#issuecomment-676829617
@andrijapanicsb
I tested on my cloudstack 4.14 environments.
As @VincentHermes reported, if the SSL certificate was a single certificate, then SSL is set. However, in the case of multiple certificates, SSL configuration fails.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rvalle commented on issue #4199: HTTPS does not initiate
Posted by GitBox <gi...@apache.org>.
rvalle commented on issue #4199:
URL: https://github.com/apache/cloudstack/issues/4199#issuecomment-681911432
thanks @rhtyd
Unfortunately, it does not look like an easy workaround is possible on 4.14 , is it?
I can see that you marked the PR for 4.14.1
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rvalle commented on issue #4199: HTTPS does not initiate
Posted by GitBox <gi...@apache.org>.
rvalle commented on issue #4199:
URL: https://github.com/apache/cloudstack/issues/4199#issuecomment-682373161
@rhtyd yes, I am using haproxy while waiting for 4.14.1.0, also experimenting with single entrypoint for svms/manager.
can wait :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rvalle commented on issue #4199: HTTPS does not initiate
Posted by GitBox <gi...@apache.org>.
rvalle commented on issue #4199:
URL: https://github.com/apache/cloudstack/issues/4199#issuecomment-681713147
I am also seeing this issue on a clean 4.14
furthermore I am also testing my keystore on a vanilla distribution of jetty 9.4 and it does work.
SSL on system VMs does work OK too.
It is something specific to the Jetty SSL connector in the manager.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rvalle commented on issue #4199: HTTPS does not initiate
Posted by GitBox <gi...@apache.org>.
rvalle commented on issue #4199:
URL: https://github.com/apache/cloudstack/issues/4199#issuecomment-681727982
@rhtyd I downloaded a vanilla jetty 9 and it works
With certificate, key, intermediate certificate and root ca.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rvalle edited a comment on issue #4199: HTTPS does not initiate
Posted by GitBox <gi...@apache.org>.
rvalle edited a comment on issue #4199:
URL: https://github.com/apache/cloudstack/issues/4199#issuecomment-681727982
@rhtyd I downloaded a vanilla jetty 9 and jdk 11 and it works
With certificate, key, intermediate certificate and root ca.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] VincentHermes commented on issue #4199: HTTPS does not initiate
Posted by GitBox <gi...@apache.org>.
VincentHermes commented on issue #4199:
URL: https://github.com/apache/cloudstack/issues/4199#issuecomment-654885550
@rhtyd We tried these certificate combinations:
* Key - Server Cert - Intermediate Cert - RootCA
* Key - Server Cert - Intermediate Cert
* Key - Server Cert - RootCA
* Key - Server Cert
Only the server cert doesn't work of course because generating the keystore requires a fitting key. None of these worked however.
The creation of the Self Signed happened like that:
`openssl genrsa -out selfsigned.key 2048`
`openssl req -new -key selfsigned.key -out selfsigned.csr`
`openssl x509 -req -days 3650 -in selfsigned.csr -signkey selfsigned.key -out selfsigned.crt`
`cat selfsigned.key selfsigned.crt > selfsignedcombined.crt`
`openssl pkcs12 -in selfsignedcombined.crt -export -out selfsignedcombined.pkcs12`
`keytool -importkeystore -srckeystore selfsignedcombined.pkcs12 -srcstoretype PKCS12 -destkeystore /etc/cloudstack/management/selfsignedcombined.pkcs12 -deststoretype pkcs12`
And after editing `server.properties` it launched HTTPS with that self signed cert.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on issue #4199: HTTPS does not initiate
Posted by GitBox <gi...@apache.org>.
rhtyd commented on issue #4199:
URL: https://github.com/apache/cloudstack/issues/4199#issuecomment-677603194
I'll take it up, I think on master I also found that SSL did not work for one of our use-cases (wildcard cert).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rvalle commented on issue #4199: HTTPS does not initiate
Posted by GitBox <gi...@apache.org>.
rvalle commented on issue #4199:
URL: https://github.com/apache/cloudstack/issues/4199#issuecomment-681740102
Here is the working configuration, in case you find it useful:
- jetty-9.4.31.v20200723
- openjdk version "11.0.8" 2020-07-14
the keystore created like this:
```
- name: Generate PKCS#12 file
openssl_pkcs12:
action: export
path: jetty.pkcs12
friendly_name: jetty
privatekey_path: Test_Server_Key.pem
certificate_path: VDC_Test_Server.crt
other_certificates:
- VDC_Test_Intermediate_CA.crt
- VDC_Test_CA.crt
state: present
passphrase: Secret
- name: Generate Java Key Store
shell: keytool -importkeystore -srckeystore jetty.pkcs12 -srcstoretype PKCS12 -destkeystore keystore
```
Which is basically the same thing that jetty documentation says but in ansible commands
I got stuck in the vanilla jetty configuration becuase I had to set the keystore password at 3 different places and I have to enable 2 different modules:
- java -jar start.jar --add-to-start=ssl,https to enable the modules
- jetty.sslContext.keyStorePassword
- jetty.sslContext.keyManagerPassword
- jetty.sslContext.trustStorePassword
but after that it works and the cert chain reported to the browser is correct.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] VincentHermes commented on issue #4199: HTTPS does not initiate
Posted by GitBox <gi...@apache.org>.
VincentHermes commented on issue #4199:
URL: https://github.com/apache/cloudstack/issues/4199#issuecomment-654841256
After additional investigation with our very capable colleague we seem to have found the error, but are not able to solve it. There is an exception on jetty regarding keystores:
`Exception in thread "main" java.lang.IllegalStateException: KeyStores with multiple certificates are not supported on the`
`base class org.eclipse.jetty.util.ssl.SslContextFactory. (Use org.eclipse.jetty.util.ssl.SslContextFactory$Server or` `org.eclipse.jetty.util.ssl.SslContextFactory$Client instead)`
` at org.eclipse.jetty.util.ssl.SslContextFactory.newSniX509ExtendedKeyManager(SslContextFactory.java:1275)`
` .... `
`KeyStores with multiple certificates are not supported on the base class org.eclipse.jetty.util.ssl.SslContextFactory`
Starting with `jetty-9.4.23.v20191118` the usage of `SslContextFactory` is deprecated and `SslContextFactory.Server` must be used instead. The embedded Version in CS 4.14 is `jetty-9.4.26.v20200117`.
I think this is the root of our HTTPS Issue. We are not able to solve this but we are using an Apache Proxy now, which however does not allow to upload from local but that's another story.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] andrijapanicsb commented on issue #4199: HTTPS does not initiate
Posted by GitBox <gi...@apache.org>.
andrijapanicsb commented on issue #4199:
URL: https://github.com/apache/cloudstack/issues/4199#issuecomment-669267851
Can you try a clean 4.14 install and ssl setup by following this https://www.shapeblue.com/securing-cloudstack-4-11-with-https-tls/ - just the later part (skip the securing system VMs)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on issue #4199: HTTPS does not initiate
Posted by GitBox <gi...@apache.org>.
rhtyd commented on issue #4199:
URL: https://github.com/apache/cloudstack/issues/4199#issuecomment-682357192
@rvalle unfortunately this requires a code specific fix (fixed in https://github.com/apache/cloudstack/commit/0501575efaf98f00727a0074dd8dcc6e2c545d30 towards 4.14/master branches). You may cherry-pick the commit and build it on top of 4.14.0.0; or wait when we've 4.14.1.0/4.15.0.0 releases. As a workaround you can setup a nginx SSL-offloading server and reverse proxy to your CloudStack mgmt server. For example see:
https://docs.nginx.com/nginx/admin-guide/security-controls/securing-http-traffic-upstream/
https://docs.nginx.com/nginx/admin-guide/security-controls/terminating-ssl-http/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rvalle commented on issue #4199: HTTPS does not initiate
Posted by GitBox <gi...@apache.org>.
rvalle commented on issue #4199:
URL: https://github.com/apache/cloudstack/issues/4199#issuecomment-681767539
yes, it does work with jetty-9.4.26.v20200117, the same packaged with the manager.
So, it must be an instantiation issue, hopefully a configuration issue that can be fixed with a system property...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rvalle commented on issue #4199: HTTPS does not initiate
Posted by GitBox <gi...@apache.org>.
rvalle commented on issue #4199:
URL: https://github.com/apache/cloudstack/issues/4199#issuecomment-681755978
@rhtyd while you fix this I am trying to find out if there is a workaround for 4.14
I am going to start by downloading vanilla jetty-9.4.26.v20200117 and testing the same configuration I have
Then check if this is due to any setting that can be added....
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on issue #4199: HTTPS does not initiate
Posted by GitBox <gi...@apache.org>.
rhtyd commented on issue #4199:
URL: https://github.com/apache/cloudstack/issues/4199#issuecomment-681719887
Yes @rvalle I've reproduced this with a bunch of keystore combinations with the latest jetty 9.x and JDK11, it does not happen with older releases that used JDK8 such as 4.13. I've tested the following combinations:
- with a single self-signed cert
- with a cert and chain of intermediate and root-ca cert
- keystore created in two formats (the jks and PKCS12, I'm exploring this now)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on issue #4199: HTTPS does not initiate
Posted by GitBox <gi...@apache.org>.
rhtyd commented on issue #4199:
URL: https://github.com/apache/cloudstack/issues/4199#issuecomment-654845984
@VincentHermes thanks for sharing, does this work for you if you use a single certificate?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org