You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by GitBox <gi...@apache.org> on 2020/11/09 20:38:27 UTC
[GitHub] [cloudstack] Lucasgranet opened a new issue #4455: Cannot start ASC 4.15-SNAPSHOT on CentOS 8.2
Lucasgranet opened a new issue #4455:
URL: https://github.com/apache/cloudstack/issues/4455
<!--
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release and master branch are affected too.
Always add information AFTER of these HTML comments, but no need to delete the comments.
-->
##### ISSUE TYPE
<!-- Pick one below and delete the rest -->
* Bug Report
##### COMPONENT NAME
<!--
Categorize the issue, e.g. API, VR, VPN, UI, etc.
-->
~~~
API
~~~
##### CLOUDSTACK VERSION
<!--
New line separated list of affected versions, commit ID for issues on master branch.
-->
~~~
4.15-SNAPSHOT 5f8289ffe90fd829493bf4e0d23c64ef50313627 (master - 11/09/2020)
~~~
##### CONFIGURATION
<!--
Information about the configuration if relevant, e.g. basic network, advanced networking, etc. N/A otherwise
-->
Built on a CentOS8.2 (up to date) - Deployed on a CentOS8.2 (up to date)
##### OS / ENVIRONMENT
<!--
Information about the environment if relevant, N/A otherwise
-->
##### SUMMARY
<!-- Explain the problem/feature briefly -->
I cannot start ASC 4.15-SNAPSHOT on a CentOS8.2.
- I have already start ASC 4.15 on C8. I wanted to fetch last updates to start a new cluster but a SSL connection cannot be established to the management server.
The service retry in loop and cause a very high CPU usage.
- All APIs are very very slow (and not sure they are working well)
- The issue seems to be caused by a bad certificate generation (bad usage - see below)
##### STEPS TO REPRODUCE
<!--
For bugs, show exactly how to reproduce the problem, using a minimal test-case. Use Screenshots if accurate.
For new features, show how the feature would be used.
-->
Build from master, deploy on a C8.
<!-- Paste example playbooks or commands between quotes below -->
<!-- You can also paste gist.github.com links for larger files -->
##### EXPECTED RESULTS
<!-- What did you expect to happen when running the steps above? -->
~~~
No major log issue
~~~
##### ACTUAL RESULTS
<!-- What actually happened? -->
<!-- Paste verbatim command output between quotes below -->
Log from the service
~~~
nov. 09 20:42:36 cs.iaasm.lgr.fr java[3568]: INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:null) (logid:) Starting CloudStack Components
nov. 09 20:42:36 cs.iaasm.lgr.fr java[3568]: INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:null) (logid:) Done Starting CloudStack Components
nov. 09 20:42:37 cs.iaasm.lgr.fr java[3568]: INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:null) (logid:) Configuring CloudStack Components
nov. 09 20:42:37 cs.iaasm.lgr.fr java[3568]: INFO [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:null) (logid:) Done Configuring CloudStack Components
nov. 09 20:42:38 cs.iaasm.lgr.fr java[3568]: INFO [c.c.u.LogUtils] (main:null) (logid:) log4j configuration found at /etc/cloudstack/management/log4j-cloud.xml
nov. 09 20:42:39 cs.iaasm.lgr.fr java[3568]: WARN [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-39:null) (logid:) This SSL engine was forced to close inbound due to end of stream.
nov. 09 20:42:39 cs.iaasm.lgr.fr java[3568]: javax.net.ssl.SSLException: closing inbound before receiving peer's close_notify
nov. 09 20:42:39 cs.iaasm.lgr.fr java[3568]: at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:133)
nov. 09 20:42:39 cs.iaasm.lgr.fr java[3568]: at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117)
nov. 09 20:42:39 cs.iaasm.lgr.fr java[3568]: at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:337)
nov. 09 20:42:39 cs.iaasm.lgr.fr java[3568]: at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:293)
nov. 09 20:42:39 cs.iaasm.lgr.fr java[3568]: at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:284)
nov. 09 20:42:39 cs.iaasm.lgr.fr java[3568]: at java.base/sun.security.ssl.SSLEngineImpl.closeInbound(SSLEngineImpl.java:733)
nov. 09 20:42:39 cs.iaasm.lgr.fr java[3568]: at com.cloud.utils.nio.Link.doHandshakeUnwrap(Link.java:490)
nov. 09 20:42:39 cs.iaasm.lgr.fr java[3568]: at com.cloud.utils.nio.Link.doHandshake(Link.java:618)
nov. 09 20:42:39 cs.iaasm.lgr.fr java[3568]: at com.cloud.utils.nio.NioConnection$1.run(NioConnection.java:216)
nov. 09 20:42:39 cs.iaasm.lgr.fr java[3568]: at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
nov. 09 20:42:39 cs.iaasm.lgr.fr java[3568]: at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
nov. 09 20:42:39 cs.iaasm.lgr.fr java[3568]: at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
nov. 09 20:42:39 cs.iaasm.lgr.fr java[3568]: at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
nov. 09 20:42:39 cs.iaasm.lgr.fr java[3568]: at java.base/java.lang.Thread.run(Thread.java:834)
nov. 09 20:42:40 cs.iaasm.lgr.fr java[3568]: WARN [c.c.a.AlertManagerImpl] (Cluster-Notification-1:ctx-f38888de) (logid:9d1cfe62) AlertType:: 14 | dataCenterId:: 0 | podId:: 0 | clusterId:: null | message:: Management server node 10.2.112.14 is up
nov. 09 20:42:41 cs.iaasm.lgr.fr java[3568]: INFO [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-2:null) (logid:) Connection from /10.2.112.177 closed but no cleanup was done.
nov. 09 20:42:43 cs.iaasm.lgr.fr java[3568]: ERROR [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-72:null) (logid:) SSL error caught during wrap data: Unexpected handshake message: client_hello, for local address=/10.2.112.14:8250, remote address=/10.2.112.177:50496.
nov. 09 20:42:45 cs.iaasm.lgr.fr java[3568]: ERROR [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-37:null) (logid:) SSL error caught during wrap data: Unexpected handshake message: client_hello, for local address=/10.2.112.14:8250, remote address=/10.2.112.177:50388.
nov. 09 20:42:46 cs.iaasm.lgr.fr java[3568]: INFO [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-5:null) (logid:) Connection from /10.2.112.177 closed but no cleanup was done.
nov. 09 20:42:48 cs.iaasm.lgr.fr java[3568]: INFO [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-6:null) (logid:) Connection from /10.2.112.177 closed but no cleanup was done.
nov. 09 20:42:59 cs.iaasm.lgr.fr java[3568]: ERROR [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-42:null) (logid:) SSL error caught during wrap data: Unexpected handshake message: client_hello, for local address=/10.2.112.14:8250, remote address=/10.2.112.177:50398.
nov. 09 20:43:03 cs.iaasm.lgr.fr java[3568]: WARN [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-3:null) (logid:) SSL Handshake has taken more than 30s to connect to: /10.2.112.177:50320. Please investigate this connection.
nov. 09 20:43:03 cs.iaasm.lgr.fr java[3568]: WARN [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-6:null) (logid:) SSL Handshake has taken more than 30s to connect to: /10.2.112.177:50326. Please investigate this connection.
nov. 09 20:43:05 cs.iaasm.lgr.fr java[3568]: WARN [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-11:null) (logid:) SSL Handshake has taken more than 30s to connect to: /10.2.112.177:50336. Please investigate this connection.
nov. 09 20:43:06 cs.iaasm.lgr.fr java[3568]: WARN [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-9:null) (logid:) SSL Handshake has taken more than 30s to connect to: /10.2.112.177:50332. Please investigate this connection.
nov. 09 20:43:06 cs.iaasm.lgr.fr java[3568]: WARN [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-4:null) (logid:) SSL Handshake has taken more than 30s to connect to: /10.2.112.177:50322. Please investigate this connection.
nov. 09 20:43:06 cs.iaasm.lgr.fr java[3568]: WARN [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-1:null) (logid:) SSL Handshake has taken more than 30s to connect to: /10.2.112.177:50316. Please investigate this connection.
nov. 09 20:43:06 cs.iaasm.lgr.fr java[3568]: WARN [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-7:null) (logid:) SSL Handshake has taken more than 30s to connect to: /10.2.112.177:50328. Please investigate this connection.
nov. 09 20:43:06 cs.iaasm.lgr.fr java[3568]: WARN [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-8:null) (logid:) SSL Handshake has taken more than 30s to connect to: /10.2.112.177:50330. Please investigate this connection.
nov. 09 20:43:06 cs.iaasm.lgr.fr java[3568]: WARN [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-10:null) (logid:) SSL Handshake has taken more than 30s to connect to: /10.2.112.177:50334. Please investigate this connection.
nov. 09 20:43:06 cs.iaasm.lgr.fr java[3568]: WARN [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-15:null) (logid:) SSL Handshake has taken more than 30s to connect to: /10.2.112.177:50344. Please investigate this connection.
nov. 09 20:43:06 cs.iaasm.lgr.fr java[3568]: WARN [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-13:null) (logid:) SSL Handshake has taken more than 30s to connect to: /10.2.112.177:50340. Please investigate this connection.
nov. 09 20:43:06 cs.iaasm.lgr.fr java[3568]: WARN [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-108:null) (logid:) This SSL engine was forced to close inbound due to end of stream.
nov. 09 20:43:06 cs.iaasm.lgr.fr java[3568]: javax.net.ssl.SSLException: closing inbound before receiving peer's close_notify
nov. 09 20:43:06 cs.iaasm.lgr.fr java[3568]: at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:133)
nov. 09 20:43:06 cs.iaasm.lgr.fr java[3568]: at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117)
nov. 09 20:43:06 cs.iaasm.lgr.fr java[3568]: at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:337)
nov. 09 20:43:06 cs.iaasm.lgr.fr java[3568]: at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:293)
nov. 09 20:43:06 cs.iaasm.lgr.fr java[3568]: at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:284)
nov. 09 20:43:06 cs.iaasm.lgr.fr java[3568]: at java.base/sun.security.ssl.SSLEngineImpl.closeInbound(SSLEngineImpl.java:733)
nov. 09 20:43:06 cs.iaasm.lgr.fr java[3568]: at com.cloud.utils.nio.Link.doHandshakeUnwrap(Link.java:490)
nov. 09 20:43:06 cs.iaasm.lgr.fr java[3568]: at com.cloud.utils.nio.Link.doHandshake(Link.java:618)
nov. 09 20:43:06 cs.iaasm.lgr.fr java[3568]: at com.cloud.utils.nio.NioConnection$1.run(NioConnection.java:216)
nov. 09 20:43:06 cs.iaasm.lgr.fr java[3568]: at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
nov. 09 20:43:06 cs.iaasm.lgr.fr java[3568]: at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
nov. 09 20:43:06 cs.iaasm.lgr.fr java[3568]: at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
nov. 09 20:43:06 cs.iaasm.lgr.fr java[3568]: at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
nov. 09 20:43:06 cs.iaasm.lgr.fr java[3568]: at java.base/java.lang.Thread.run(Thread.java:834)
nov. 09 20:43:06 cs.iaasm.lgr.fr java[3568]: WARN [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-12:null) (logid:) SSL Handshake has taken more than 30s to connect to: /10.2.112.177:50338. Please investigate this connection.
~~~
I performed an OpenSSL connection to the management service - TCP Link cannot be established, OpenSSL refused the certificate. Java must not like too.
~~~
-> # openssl s_client -connect 10.2.112.14:8250
CONNECTED(00000003)
Can't use SSL_get_servername
depth=0 CN = ca.cloudstack.apache.org
verify error:num=18:self signed certificate
verify return:1
depth=0 CN = ca.cloudstack.apache.org
verify error:num=26:unsupported certificate purpose
verify return:1
depth=0 CN = ca.cloudstack.apache.org
verify return:1
---
Certificate chain
0 s:CN = ca.cloudstack.apache.org
i:CN = ca.cloudstack.apache.org
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFDTCCAvWgAwIBAgIJAMtEGDhOFq8iMA0GCSqGSIb3DQEBCwUAMCMxITAfBgNV
BAMMGGNhLmNsb3Vkc3RhY2suYXBhY2hlLm9yZzAgFw0yMDExMDkwNzMzNTJaGA8y
MDUwMTEwMjE5MzM1MlowIzEhMB8GA1UEAwwYY2EuY2xvdWRzdGFjay5hcGFjaGUu
b3JnMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAq2Z0gJIKjv3lLIFQ
bV7ARjqQBwAQd4hl674mTknzTBMvq2rnw9wSlnIcngK+8OsSHXLHIoD9wWr5C/+h
OB43rRUoYBJ2KjFg9I/HF8tSiaRPMIZ0DVfyZSmttdo7kQBhKgxBPRpfegD8ry/p
tx8YZb75ADAZKVPwM8B4O1+cgYSYG92b54K/pa6W5YV2eXcplOYKlo4zXMGvEQ3z
CwjT74WBub+u284/B4FZHkwZY+Pq90gzFkeEbUzsnXVfRZr4sPuzLiza07DfExFA
GzVwMpKIeCu8YRHL+iTnzQKoKeaeXgIRQcviCf5N+zk2tnqG/2O5CcFTtVfYYq52
iUPRmAEpUYQl9LEWcOwmJiNhNPG/EBsIjoe8ZxsB3gH4sqe17ieOAWZvZCTmvPvH
dryGoSYafICHpMLrTD24zb8uC6mrpyqF7v1M/aoXxxoWE2Y3hD5UnThWU2w2ZIrx
62UeF9XZGsq6s2/oFC8Saeg+cJcW/an42VnVJ0Mp6SNEGlU04Gr0fmQIRcFkI79Y
uQvsUUnxaBGKivTf+KbySBZ7pa7Tn2v8tlwAXIqQqcWZ4qqpTu9R0PApmSkoZCKl
HKPQJpSHrIkE7BWcjViO9RS0xed1J9jyNsFMsynqItZT29gETQokFRdIOWti5ALw
hDpnC/NxOZpOOgClu0gVgUq4g3UCAwEAAaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAO
BgNVHQ8BAf8EBAMCAQYwHQYDVR0OBBYEFPb7eAs0X8PkriAswvCuW3Xfkhn2MA0G
CSqGSIb3DQEBCwUAA4ICAQAWWLimV3S/iB8tvbx56dn1u2HyG9Q49pSD/TFMB0jJ
agiX5hbOYn7o2SgHJQDOBuPF31fCLMTU3R+O3WwXjh15AWm03KLDnZs5wgSGzE0g
+XD47hhh4wWEx48NACToMqUHMcPIR+G9dV4w9npe8LSzO1HzuObAXwmD1JqbVQL0
63ZMGG0zD5Pqx2/ykWu5tA0oR3SAlgoKB8B+vuEmyyGxKtsBKT5U2f69diW/upH6
mzgRIHTnx965k9e4xt8NHZNa0jHBD/fV9hkne0wt7PSM2GotZmGcaQw5TqFmxC5p
uelakiUdMW2x9Ij2KGA1rTGaqDG8n9tWV7DaMDRTfm7/3Qg4ae6uSaQh07ELnzT0
Ixm9MLZk6AjpKUO0AcoL+vAuWuinNManAuuGoU9XZVUsUIyYdtUVDp1M4lPms4ut
8AboxSoPWSXXnQXaGnG6HcP3sn0KNGvB9j7OHLMFfE+EOtE2kp6Khm2zFUVu+IM7
CtuamA+1nb2koFh9BxD3WkiDf9wHBW/lyA1nn8vd5qK/Mn4MsUxKdW1mOfIISak/
Bvkm+lQH9LapjmNsT+iocD0a6ysH9nJN6/R94mxc1PrTo+T8sQqtffqEjfKZIS7J
qmL0FZQD8O0rRPi0TCudknt5TvTeEfZiB89I8/Doc3zJvN9o41QalhRQy0AKFO1N
fA==
-----END CERTIFICATE-----
subject=CN = ca.cloudstack.apache.org
issuer=CN = ca.cloudstack.apache.org
---
No client certificate CA names sent
Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA1:RSA+SHA1
Shared Requested Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 2387 bytes and written 846 bytes
Verification error: unsupported certificate purpose
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 4096 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 26 (unsupported certificate purpose)
---
140036465194816:error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate:ssl/record/rec_layer_s3.c:1543:SSL alert number 42
~~~
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Lucasgranet closed issue #4455: Cannot start ASC 4.15-SNAPSHOT on CentOS 8.2
Posted by GitBox <gi...@apache.org>.
Lucasgranet closed issue #4455:
URL: https://github.com/apache/cloudstack/issues/4455
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] DaanHoogland commented on issue #4455: Cannot start ASC 4.15-SNAPSHOT on CentOS 8.2
Posted by GitBox <gi...@apache.org>.
DaanHoogland commented on issue #4455:
URL: https://github.com/apache/cloudstack/issues/4455#issuecomment-724768520
@PaulAngus the type labels use dashes and not underscores!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] shwstppr commented on issue #4455: Cannot start ASC 4.15-SNAPSHOT on CentOS 8.2
Posted by GitBox <gi...@apache.org>.
shwstppr commented on issue #4455:
URL: https://github.com/apache/cloudstack/issues/4455#issuecomment-843802726
@Lucasgranet were you able check 4.15.0.0 release? Are you still facing the same issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Lucasgranet commented on issue #4455: Cannot start ASC 4.15-SNAPSHOT on CentOS 8.2
Posted by GitBox <gi...@apache.org>.
Lucasgranet commented on issue #4455:
URL: https://github.com/apache/cloudstack/issues/4455#issuecomment-844064508
Hi,
Since CentOS 8 is now dead, I think the support of CentOS8 is not needed anymore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on issue #4455: Cannot start ASC 4.15-SNAPSHOT on CentOS 8.2
Posted by GitBox <gi...@apache.org>.
rhtyd commented on issue #4455:
URL: https://github.com/apache/cloudstack/issues/4455#issuecomment-791907826
@Lucasgranet can you check against 4.15.0.0 release and CentOS8 now?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] DaanHoogland commented on issue #4455: Cannot start ASC 4.15-SNAPSHOT on CentOS 8.2
Posted by GitBox <gi...@apache.org>.
DaanHoogland commented on issue #4455:
URL: https://github.com/apache/cloudstack/issues/4455#issuecomment-724778388
ACS doesn't support centos 8.2 yet, @Lucasgranet . I think there is work underway in some pull request.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org