You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by GitBox <gi...@apache.org> on 2021/02/07 01:02:07 UTC
[GitHub] [cloudstack] abdelouahabb opened a new issue #4659: SSL Handshake failed while connecting to host: 192.168.56.85 port: 8250
abdelouahabb opened a new issue #4659:
URL: https://github.com/apache/cloudstack/issues/4659
##### ISSUE TYPE
* Bug Report
##### COMPONENT NAME
~~~
cloudstack-agent - kvm
~~~
##### CLOUDSTACK VERSION
~~~
4.15.0.0
~~~
##### SUMMARY
I cant add a `kvm`, it works with `xcp-ng/xen` but not with `kvm` and I get this error :
~~~
Feb 7 00:59:56 cloud java[3387]: INFO [cloud.agent.Agent] (main:) (logid:) Connecting to host:192.168.56.85
Feb 7 00:59:56 cloud java[3387]: INFO [utils.nio.NioClient] (main:) (logid:) Connecting to 192.168.56.85:8250
Feb 7 00:59:56 cloud java[3387]: INFO [utils.nio.Link] (main:) (logid:) Conf file found: /etc/cloudstack/agent/agent.properties
Feb 7 00:59:56 cloud java[3387]: WARN [utils.nio.Link] (main:) (logid:) Failed to load keystore, using trust all manager
Feb 7 00:59:56 cloud java[1689]: ERROR [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-1:null) (logid:) SSL error caught during wrap data: Empty server certificate chain, for local address=>Feb 7 00:59:56 cloud java[3387]: ERROR [utils.nio.Link] (main:) (logid:) SSL error caught during unwrap data: Received fatal alert: bad_certificate, for local address=/192.168.56.85:53164, r>Feb 7 00:59:56 cloud java[3387]: ERROR [utils.nio.NioClient] (main:) (logid:) SSL Handshake failed while connecting to host: 192.168.56.85 port: 8250
Feb 7 00:59:56 cloud java[3387]: ERROR [utils.nio.NioConnection] (main:) (logid:) Unable to initialize the threads.
Feb 7 00:59:56 cloud java[3387]: java.io.IOException: SSL Handshake failed while connecting to host: 192.168.56.85 port: 8250
Feb 7 00:59:56 cloud java[3387]: #011at com.cloud.utils.nio.NioClient.init(NioClient.java:67)
Feb 7 00:59:56 cloud java[3387]: #011at com.cloud.utils.nio.NioConnection.start(NioConnection.java:95)
Feb 7 00:59:56 cloud java[3387]: #011at com.cloud.agent.Agent.start(Agent.java:294)
Feb 7 00:59:56 cloud java[3387]: #011at com.cloud.agent.AgentShell.launchNewAgent(AgentShell.java:455)
Feb 7 00:59:56 cloud java[3387]: #011at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:422)
Feb 7 00:59:56 cloud java[3387]: #011at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:406)
Feb 7 00:59:56 cloud java[3387]: #011at com.cloud.agent.AgentShell.start(AgentShell.java:512)
Feb 7 00:59:56 cloud java[3387]: #011at com.cloud.agent.AgentShell.main(AgentShell.java:547)
Feb 7 00:59:56 cloud java[3387]: INFO [utils.nio.NioClient] (main:) (logid:) NioClient connection closed
Feb 7 00:59:56 cloud java[3387]: INFO [cloud.agent.Agent] (main:) (logid:) Attempted to connect to the server, but received an unexpected exception, trying again...
Feb 7 00:59:56 cloud java[3387]: com.cloud.utils.exception.NioConnectionException: SSL Handshake failed while connecting to host: 192.168.56.85 port: 8250
Feb 7 00:59:56 cloud java[3387]: #011at com.cloud.utils.nio.NioConnection.start(NioConnection.java:101)
Feb 7 00:59:56 cloud java[3387]: #011at com.cloud.agent.Agent.start(Agent.java:294)
Feb 7 00:59:56 cloud java[3387]: #011at com.cloud.agent.AgentShell.launchNewAgent(AgentShell.java:455)
Feb 7 00:59:56 cloud java[3387]: #011at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:422)
Feb 7 00:59:56 cloud java[3387]: #011at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:406)
Feb 7 00:59:56 cloud java[3387]: #011at com.cloud.agent.AgentShell.start(AgentShell.java:512)
Feb 7 00:59:56 cloud java[3387]: #011at com.cloud.agent.AgentShell.main(AgentShell.java:547)
Feb 7 00:59:56 cloud java[3387]: Caused by: java.io.IOException: SSL Handshake failed while connecting to host: 192.168.56.85 port: 8250
Feb 7 00:59:56 cloud java[3387]: #011at com.cloud.utils.nio.NioClient.init(NioClient.java:67)
Feb 7 00:59:56 cloud java[3387]: #011at com.cloud.utils.nio.NioConnection.start(NioConnection.java:95)
Feb 7 00:59:56 cloud java[3387]: #011... 6 more
Feb 7 00:59:56 cloud java[1689]: INFO [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-3:null) (logid:) Connection from /192.168.56.85 closed but no cleanup was done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] abdelouahabb commented on issue #4659: Agent : SSL Handshake failed while connecting to host on port: 8250
Posted by GitBox <gi...@apache.org>.
abdelouahabb commented on issue #4659:
URL: https://github.com/apache/cloudstack/issues/4659#issuecomment-776866417
> @abdelouahabb this is working for loads of people all over the world. I suggest you ask for help on [users@cloudstack.apache.org](mailto:users@cloudstack.apache.org)
> By the looks of your log, the problem is not with ssh but with the certificates or the jdk encryption policies available to your host or management server.
Checked the policy and it is enable :
` nano $JAVA_HOME/conf/security/java.security` and the value is `crypto.policy=unlimited`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] abdelouahabb commented on issue #4659: Agent : SSL Handshake failed while connecting to host on port: 8250
Posted by GitBox <gi...@apache.org>.
abdelouahabb commented on issue #4659:
URL: https://github.com/apache/cloudstack/issues/4659#issuecomment-778553357
> ok @abdelouahabb , I think you have a problem on your host (using the host certificate) I added to milestone 4.15.1, but I think it is a problem with the host installation. Please add some information from the agent if you can.
Thank you,
Here is the log, this keeps looping over and over
```
sudo tail -f /var/log/cloudstack/agent/agent.log
2021-02-13 03:10:24,613 INFO [cloud.agent.Agent] (main:null) (logid:) Connecting to host:192.168.56.85
2021-02-13 03:10:24,614 INFO [utils.nio.NioClient] (main:null) (logid:) Connecting to 192.168.56.85:8250
2021-02-13 03:10:24,615 INFO [utils.nio.Link] (main:null) (logid:) Conf file found: /etc/cloudstack/agent/agent.properties
2021-02-13 03:10:24,617 WARN [utils.nio.Link] (main:null) (logid:) Failed to load keystore, using trust all manager
2021-02-13 03:10:24,710 ERROR [utils.nio.Link] (main:null) (logid:) SSL error caught during unwrap data: Received fatal alert: bad_certificate, for local address=/192.168.56.85:57384, remote address=/192.168.56.85:8250. The client may have invalid ca-certificates.
2021-02-13 03:10:24,711 ERROR [utils.nio.NioClient] (main:null) (logid:) SSL Handshake failed while connecting to host: 192.168.56.85 port: 8250
2021-02-13 03:10:24,711 ERROR [utils.nio.NioConnection] (main:null) (logid:) Unable to initialize the threads.
java.io.IOException: SSL Handshake failed while connecting to host: 192.168.56.85 port: 8250
at com.cloud.utils.nio.NioClient.init(NioClient.java:67)
at com.cloud.utils.nio.NioConnection.start(NioConnection.java:95)
at com.cloud.agent.Agent.start(Agent.java:294)
at com.cloud.agent.AgentShell.launchNewAgent(AgentShell.java:455)
at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:422)
at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:406)
at com.cloud.agent.AgentShell.start(AgentShell.java:512)
at com.cloud.agent.AgentShell.main(AgentShell.java:547)
2021-02-13 03:10:24,711 INFO [utils.nio.NioClient] (main:null) (logid:) NioClient connection closed
2021-02-13 03:10:24,711 INFO [cloud.agent.Agent] (main:null) (logid:) Attempted to connect to the server, but received an unexpected exception, trying again...
com.cloud.utils.exception.NioConnectionException: SSL Handshake failed while connecting to host: 192.168.56.85 port: 8250
at com.cloud.utils.nio.NioConnection.start(NioConnection.java:101)
at com.cloud.agent.Agent.start(Agent.java:294)
at com.cloud.agent.AgentShell.launchNewAgent(AgentShell.java:455)
at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:422)
at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:406)
at com.cloud.agent.AgentShell.start(AgentShell.java:512)
at com.cloud.agent.AgentShell.main(AgentShell.java:547)
```
And I have even replaced the openjdk with the oracle java
```
>> sudo update-alternatives --config java
There are 2 choices for the alternative java (providing /usr/bin/java).
Selection Path Priority Status
------------------------------------------------------------
0 /usr/lib/jvm/java-11-openjdk-amd64/bin/java 1111 auto mode
1 /usr/lib/jvm/java-11-openjdk-amd64/bin/java 1111 manual mode
* 2 /usr/lib/jvm/java-15-oracle/bin/java 1091 manual mode
Press <enter> to keep the current choice[*], or type selection number:
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] abdelouahabb commented on issue #4659: Agent : SSL Handshake failed while connecting to host on port: 8250
Posted by GitBox <gi...@apache.org>.
abdelouahabb commented on issue #4659:
URL: https://github.com/apache/cloudstack/issues/4659#issuecomment-782516820
Update : Worked Ubuntu 18.04.5 by making `ca.plugin.root.auth.strictness false`, it seems Ubuntu-server (no GUI) 20.04 is messy
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] abdelouahabb commented on issue #4659: Agent : SSL Handshake failed while connecting to host on port: 8250
Posted by GitBox <gi...@apache.org>.
abdelouahabb commented on issue #4659:
URL: https://github.com/apache/cloudstack/issues/4659#issuecomment-778914316
I tried doing it manually (as i think understanding the security concept in CloudStack) :
`sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/management/server.properties ./cloud.jks dummyPassword007 365 ./cloud.csr`
`sudo /usr/share/cloudstack-common/scripts/util/keystore-cert-import /etc/cloudstack/management/server.properties ./cloud.jks mode:agent ./a "/C=DZ /ST=akham/L=akham/O=akham/OU=akham/CN=akham.alien.local" ./b "/C=DZ/ST=akham/L=akham/O=akham/OU=akham/CN=akham.alien.local" ./c
CACERT_FILE: ./b CERT_FILE: ./a PRIVKEY_FILE: ./c`
But still the same result,
After investigating in the content of the script, it seems that are some folders/files dont exist in the Ubuntu server installation :
```
abdelouahab@akham:/var/cache/cloudstack$
>> file /etc/libvirt/libvirtd.conf
/etc/libvirt/libvirtd.conf: ASCII text
abdelouahab@akham:/var/cache/cloudstack$
>> file /var/cache/cloud/cmdline
/var/cache/cloud/cmdline: cannot open `/var/cache/cloud/cmdline' (No such file or directory)
abdelouahab@akham:/var/cache/cloudstack$
>> pwd
/var/cache/cloudstack
abdelouahab@akham:/var/cache/cloudstack$
>> file /etc/pki/libvirt/private/serverkey.pem
/etc/pki/libvirt/private/serverkey.pem: broken symbolic link to /etc/cloudstack/agent/cloud.key
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] abdelouahabb commented on issue #4659: Agent : SSL Handshake failed while connecting to host on port: 8250
Posted by GitBox <gi...@apache.org>.
abdelouahabb commented on issue #4659:
URL: https://github.com/apache/cloudstack/issues/4659#issuecomment-782787080
> @abdelouahabb
> I have seen these logs in #4715
>
> ```
> 2021-02-20 00:42:47,784 DEBUG [c.c.u.s.SSHCmdHelper] (qtp182531396-22:ctx-6c2d2d39 ctx-e64efde7) (logid:980a971b) Executing cmd: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/
> 2021-02-20 00:42:48,844 DEBUG [c.c.u.s.SSHCmdHelper] (qtp182531396-22:ctx-6c2d2d39 ctx-e64efde7) (logid:980a971b) SSH command: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/
> SSH command output:
> sudo: no tty present and no askpass program specified
> ```
>
> it seems some misconfiguration with sudo.
That was from the same machine too, I use Virtualbox on Windows 7
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] abdelouahabb edited a comment on issue #4659: Agent : SSL Handshake failed while connecting to host on port: 8250
Posted by GitBox <gi...@apache.org>.
abdelouahabb edited a comment on issue #4659:
URL: https://github.com/apache/cloudstack/issues/4659#issuecomment-782516820
Update : Worked Lubuntu 18.04.5 by making `ca.plugin.root.auth.strictness false`, it seems Ubuntu-server (no GUI) 20.04 is messy
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] weizhouapache commented on issue #4659: Agent : SSL Handshake failed while connecting to host on port: 8250
Posted by GitBox <gi...@apache.org>.
weizhouapache commented on issue #4659:
URL: https://github.com/apache/cloudstack/issues/4659#issuecomment-781681733
###
> > > #key_file="/etc/pki/libvirt/private/serverkey.pem"
> > > #cert_file="/etc/pki/libvirt/servercert.pem"
> > > #ca_file="/etc/pki/CA/cacert.pem"
> >
> >
> > @abdelouahabb you use centos ? you need to use 'libvirt' instead of 'libvirtd.
> > The other lines should be same. the following lines should be uncommented.
> > ```
> > #tls_port="16514"
> > auth_tcp="none"
> > auth_tls="none"
> > #key_file="/etc/pki/libvirt/private/serverkey.pem"
> > #cert_file="/etc/pki/libvirt/servercert.pem"
> > #ca_file="/etc/pki/CA/cacert.pem"
> > ```
>
> I am on Ubuntu, I [disabled the security](https://libvirt.org/daemons.html) for the debug reason and getting the same issue
@abdelouahabb which ubuntu version ? 20.04 ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] abdelouahabb commented on issue #4659: Agent : SSL Handshake failed while connecting to host on port: 8250
Posted by GitBox <gi...@apache.org>.
abdelouahabb commented on issue #4659:
URL: https://github.com/apache/cloudstack/issues/4659#issuecomment-781608205
> I'm getting nearly the same issues as you. I was also experiencing the SSL problem, and disabled it in global configs. I then got the "unable to find private nics" and manually configured that in my agent.properties. I now am just getting
> "error: conection with libvirtd is broken : invalid connection pointer in virConnectGetVersion" as well as the UEFI.properties error.
> And on libvirtd status I am getting :
> "47271 error: virNetSocketReadWire:1806: end of file while reading data : Input/output error"
>
> Please let me know if you have found a solution.
@ThyLAW I am getting the same issue too with libvirt, what is your setup ? virtualizing it on bare metal or on top of another hypervisor ?
@DaanHoogland any update ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] abdelouahabb commented on issue #4659: Agent : SSL Handshake failed while connecting to host on port: 8250
Posted by GitBox <gi...@apache.org>.
abdelouahabb commented on issue #4659:
URL: https://github.com/apache/cloudstack/issues/4659#issuecomment-775556420
Still nothing, even modifying the `root ssh access`
```
PermitRootLogin yes
PermitRootLogin prohibit-password
PubkeyAuthentication yes
RSAAuthentication yes
AuthorizedKeysFile .ssh/authorized_keys
```
Same error
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd closed issue #4659: Agent : SSL Handshake failed while connecting to host on port: 8250
Posted by GitBox <gi...@apache.org>.
rhtyd closed issue #4659:
URL: https://github.com/apache/cloudstack/issues/4659
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] abdelouahabb commented on issue #4659: Agent : SSL Handshake failed while connecting to host on port: 8250
Posted by GitBox <gi...@apache.org>.
abdelouahabb commented on issue #4659:
URL: https://github.com/apache/cloudstack/issues/4659#issuecomment-781694306
> > > > > > #key_file="/etc/pki/libvirt/private/serverkey.pem"
> > > > > > #cert_file="/etc/pki/libvirt/servercert.pem"
> > > > > > #ca_file="/etc/pki/CA/cacert.pem"
> > > > >
> > > > >
> > > > > @abdelouahabb you use centos ? you need to use 'libvirt' instead of 'libvirtd.
> > > > > The other lines should be same. the following lines should be uncommented.
> > > > > ```
> > > > > #tls_port="16514"
> > > > > auth_tcp="none"
> > > > > auth_tls="none"
> > > > > #key_file="/etc/pki/libvirt/private/serverkey.pem"
> > > > > #cert_file="/etc/pki/libvirt/servercert.pem"
> > > > > #ca_file="/etc/pki/CA/cacert.pem"
> > > > > ```
> > > >
> > > >
> > > > I am on Ubuntu, I [disabled the security](https://libvirt.org/daemons.html) for the debug reason and getting the same issue
> > >
> > >
> > > @abdelouahabb which ubuntu version ? 20.04 ?
> >
> >
> > Yes, I think there is an issue with this release ?
> > ```
> > abdelouahab@akham:~$
> > >> uname -a
> > Linux akham.alien.local 5.4.0-65-generic #73-Ubuntu SMP Mon Jan 18 17:25:17 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
> > abdelouahab@akham:~$
> > >> cat /etc/os-release
> > NAME="Ubuntu"
> > VERSION="20.04.2 LTS (Focal Fossa)"
> > ID=ubuntu
> > ID_LIKE=debian
> > PRETTY_NAME="Ubuntu 20.04.2 LTS"
> > VERSION_ID="20.04"
> > HOME_URL="https://www.ubuntu.com/"
> > SUPPORT_URL="https://help.ubuntu.com/"
> > BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
> > PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
> > VERSION_CODENAME=focal
> > UBUNTU_CODENAME=focal
> > abdelouahab@akham:~$
> > >> cat /etc/apt/sources.list.d/cloudstack.list
> > deb http://download.cloudstack.org/ubuntu focal 4.15
> > ```
>
> @abdelouahabb no.
> I have a testing environment with ubuntu 20.04 as hypervisors. they work fine.
Maybe because I am using it ontop of VirtualBox (on Windows 7)
```
abdelouahab@akham:~$
>> cat /proc/cpuinfo | grep flags
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq vmx ssse3 cx16 sse4_1 sse4_2 x2apic popcnt aes xsave avx hypervisor lahf_lm pti tpr_shadow flexpriority
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq vmx ssse3 cx16 sse4_1 sse4_2 x2apic popcnt aes xsave avx hypervisor lahf_lm pti tpr_shadow flexpriority
abdelouahab@akham:~$
>> kvm-ok
INFO: /dev/kvm exists
KVM acceleration can be used
abdelouahab@akham:~$
>> dmesg | grep virtual
[ 0.002262] CPU MTRRs all blank - virtualized system.
[ 0.165718] Booting paravirtualized kernel on KVM
[ 7.798059] systemd[1]: Detected virtualization oracle.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on issue #4659: Agent : SSL Handshake failed while connecting to host on port: 8250
Posted by GitBox <gi...@apache.org>.
rhtyd commented on issue #4659:
URL: https://github.com/apache/cloudstack/issues/4659#issuecomment-791906236
@abdelouahabb since you've disabled auth strictness you may want to do the same on your libvirtd, set the listen_tls = 0 and listen_tcp = 1 and try again.
Since the original issue of agent SSL failure was fixed with the global setting workaround, I'll close the ticket. Please open new ticket for unrelated failures. I think you're using a nested env in VirtualBox and the env lack entropy, otherwise the SSL handshake would also work. (from experience VMs in some desktop hypervisors for ex. VirtualBox run slower compared to on baremetal or more modern/well-supported hypervisors such as KVM, VMware).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] abdelouahabb commented on issue #4659: Agent : SSL Handshake failed while connecting to host on port: 8250
Posted by GitBox <gi...@apache.org>.
abdelouahabb commented on issue #4659:
URL: https://github.com/apache/cloudstack/issues/4659#issuecomment-777645868
> @abdelouahabb is there error log on management server ?
The content of the folder belong to the user `cloud`
```
root@alien:/var/log/cloudstack/management# ls -l
total 22536
-rw-r--r-- 1 cloud cloud 1858062 Feb 11 03:39 access.log
-rw-r--r-- 1 cloud cloud 6751349 Feb 11 03:39 apilog.log
-rw-r--r-- 1 cloud cloud 872990 Jan 20 00:06 apilog.log.2021-01-19.gz
-rw-r--r-- 1 cloud cloud 6536434 Feb 9 00:01 apilog.log.2021-02-08.gz
-rw-r--r-- 1 cloud cloud 3216363 Feb 11 00:03 apilog.log.2021-02-10.gz
-rw-r--r-- 1 cloud cloud 2613436 Feb 11 16:26 management-server.log
-rw-r--r-- 1 cloud cloud 1138719 Feb 11 00:00 management-server.log.2021-02-10.gz
-rw-r--r-- 1 cloud cloud 54939 Feb 10 17:11 setupManagement.log
```
I made an operation so I can record the log : login, adding primary storage (error), secondary storage (success), adding new host (error)
` sudo tail -f /var/log/cloudstack/management/management-server.log > error.txt`
```
2021-02-11 16:58:13,845 ERROR [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-2:null) (logid:) SSL error caught during wrap data: Empty server certificate chain, for local address=/192.168.56.85:8250, remote address=/192.168.56.85:34012.
2021-02-11 16:58:13,848 INFO [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-11:null) (logid:) Connection from /192.168.56.85 closed but no cleanup was done.
2021-02-11 16:58:18,941 ERROR [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-2:null) (logid:) SSL error caught during wrap data: Empty server certificate chain, for local address=/192.168.56.85:8250, remote address=/192.168.56.85:34014.
2021-02-11 16:58:18,947 INFO [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-13:null) (logid:) Connection from /192.168.56.85 closed but no cleanup was done.
2021-02-11 16:58:21,229 DEBUG [c.c.s.StatsCollector] (StatsCollector-6:ctx-713e842d) (logid:459a0b20) AutoScaling Monitor is running...
2021-02-11 16:58:21,242 DEBUG [c.c.s.StatsCollector] (StatsCollector-6:ctx-10955270) (logid:0c1d2c6f) HostStatsCollector is running...
2021-02-11 16:58:21,444 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:ctx-044de3ce) (logid:cdb25f6b) StorageCollector is running...
2021-02-11 16:58:22,102 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-e404b003) (logid:1ecba277) Skip capacity scan as there is no Primary Storage in 'Up' state
2021-02-11 16:58:24,064 ERROR [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-2:null) (logid:) SSL error caught during wrap data: Empty server certificate chain, for local address=/192.168.56.85:8250, remote address=/192.168.56.85:34016.
2021-02-11 16:58:24,069 INFO [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-15:null) (logid:) Connection from /192.168.56.85 closed but no cleanup was done.
2021-02-11 16:58:29,180 ERROR [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-2:null) (logid:) SSL error caught during wrap data: Empty server certificate chain, for local address=/192.168.56.85:8250, remote address=/192.168.56.85:34018.
2021-02-11 16:58:29,184 INFO [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-2:null) (logid:) Connection from /192.168.56.85 closed but no cleanup was done.
2021-02-11 16:58:34,232 ERROR [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-2:null) (logid:) SSL error caught during wrap data: Empty server certificate chain, for local address=/192.168.56.85:8250, remote address=/192.168.56.85:34020.
2021-02-11 16:58:34,236 INFO [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-4:null) (logid:) Connection from /192.168.56.85 closed but no cleanup was done.
2021-02-11 16:58:35,146 DEBUG [c.c.a.ApiServlet] (qtp1766145591-17:ctx-85e017e4) (logid:4afb75f5) ===START=== 192.168.56.1 -- GET command=listIdps&response=json
2021-02-11 16:58:35,146 DEBUG [c.c.a.ApiServer] (qtp1766145591-17:ctx-85e017e4 ctx-9332f19e) (logid:4afb75f5) The given command listIdps either does not exist, is not available for user, or not available from ip address '192.168.56.1'.
2021-02-11 16:58:35,147 DEBUG [c.c.a.ApiServlet] (qtp1766145591-17:ctx-85e017e4 ctx-9332f19e) (logid:4afb75f5) ===END=== 192.168.56.1 -- GET command=listIdps&response=json
2021-02-11 16:58:36,102 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-7a3c3d1e) (logid:3fac441a) Found 0 routers to update status.
2021-02-11 16:58:36,106 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-7a3c3d1e) (logid:3fac441a) Found 0 VPC's to update Redundant State.
2021-02-11 16:58:36,111 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-7a3c3d1e) (logid:3fac441a) Found 0 networks to update RvR status.
2021-02-11 16:58:38,537 DEBUG [c.c.a.ApiServlet] (qtp1766145591-22:ctx-c1924809) (logid:6ac471ab) ===START=== 192.168.56.1 -- POST
2021-02-11 16:58:38,546 DEBUG [c.c.u.AccountManagerImpl] (qtp1766145591-22:ctx-c1924809) (logid:6ac471ab) Attempting to log in user: admin in domain 1
2021-02-11 16:58:38,548 DEBUG [o.a.c.s.a.PBKDF2UserAuthenticator] (qtp1766145591-22:ctx-c1924809) (logid:6ac471ab) Retrieving user: admin
2021-02-11 16:58:39,007 DEBUG [c.c.u.AccountManagerImpl] (qtp1766145591-22:ctx-c1924809) (logid:6ac471ab) CIDRs from which account 'Acct[e4b9dbf4-6a86-11eb-8c34-08002799a040-admin]' is allowed to perform API calls: 0.0.0.0/0,::/0
2021-02-11 16:58:39,008 DEBUG [c.c.u.AccountManagerImpl] (qtp1766145591-22:ctx-c1924809) (logid:6ac471ab) User: admin in domain 1 has successfully logged in
2021-02-11 16:58:39,018 INFO [c.c.a.ApiServer] (qtp1766145591-22:ctx-c1924809) (logid:6ac471ab) Current user logged in under UTC timezone
2021-02-11 16:58:39,019 INFO [c.c.a.ApiServer] (qtp1766145591-22:ctx-c1924809) (logid:6ac471ab) Timezone offset from UTC is: 0.0
2021-02-11 16:58:39,026 DEBUG [c.c.a.ApiServlet] (qtp1766145591-22:ctx-c1924809) (logid:6ac471ab) ===END=== 192.168.56.1 -- POST
2021-02-11 16:58:39,051 DEBUG [c.c.a.ApiServlet] (qtp1766145591-17:ctx-b459ae8c) (logid:80c15691) ===START=== 192.168.56.1 -- GET listall=true&command=listZones&response=json
2021-02-11 16:58:39,052 DEBUG [c.c.a.ApiServlet] (qtp1766145591-22:ctx-404e580f) (logid:f70657f3) ===START=== 192.168.56.1 -- GET command=listApis&response=json
2021-02-11 16:58:39,060 DEBUG [c.c.a.ApiServlet] (qtp1766145591-15:ctx-dbb776f8) (logid:4324c0f4) ===START=== 192.168.56.1 -- GET username=admin&command=listUsers&response=json
2021-02-11 16:58:39,061 DEBUG [c.c.a.ApiServlet] (qtp1766145591-21:ctx-6e25da50) (logid:2bcf881f) ===START=== 192.168.56.1 -- GET command=listCapabilities&response=json
2021-02-11 16:58:39,065 DEBUG [c.c.a.ApiServlet] (qtp1766145591-19:ctx-f49fae1c) (logid:5f53ea04) ===START=== 192.168.56.1 -- GET command=listLdapConfigurations&response=json
2021-02-11 16:58:39,072 DEBUG [c.c.a.ApiServer] (qtp1766145591-22:ctx-404e580f ctx-ac96fd95) (logid:f70657f3) CIDRs from which account 'Acct[e4b9dbf4-6a86-11eb-8c34-08002799a040-admin]' is allowed to perform API calls: 0.0.0.0/0,::/0
2021-02-11 16:58:39,077 DEBUG [c.c.a.ApiServer] (qtp1766145591-19:ctx-f49fae1c ctx-1b6ee335) (logid:5f53ea04) CIDRs from which account 'Acct[e4b9dbf4-6a86-11eb-8c34-08002799a040-admin]' is allowed to perform API calls: 0.0.0.0/0,::/0
2021-02-11 16:58:39,081 DEBUG [c.c.a.ApiServer] (qtp1766145591-17:ctx-b459ae8c ctx-472fadd9) (logid:80c15691) CIDRs from which account 'Acct[e4b9dbf4-6a86-11eb-8c34-08002799a040-admin]' is allowed to perform API calls: 0.0.0.0/0,::/0
2021-02-11 16:58:39,087 DEBUG [c.c.a.ApiServer] (qtp1766145591-21:ctx-6e25da50 ctx-c1cb6a8c) (logid:2bcf881f) CIDRs from which account 'Acct[e4b9dbf4-6a86-11eb-8c34-08002799a040-admin]' is allowed to perform API calls: 0.0.0.0/0,::/0
2021-02-11 16:58:39,082 DEBUG [c.c.a.ApiServlet] (qtp1766145591-16:ctx-a9b8d5fc) (logid:6acec26b) ===START=== 192.168.56.1 -- GET command=cloudianIsEnabled&response=json
2021-02-11 16:58:39,092 DEBUG [c.c.a.ApiServer] (qtp1766145591-15:ctx-dbb776f8 ctx-85fdd866) (logid:4324c0f4) CIDRs from which account 'Acct[e4b9dbf4-6a86-11eb-8c34-08002799a040-admin]' is allowed to perform API calls: 0.0.0.0/0,::/0
2021-02-11 16:58:39,099 DEBUG [c.c.a.ApiServlet] (qtp1766145591-19:ctx-f49fae1c ctx-1b6ee335) (logid:5f53ea04) ===END=== 192.168.56.1 -- GET command=listLdapConfigurations&response=json
2021-02-11 16:58:39,107 DEBUG [c.c.a.ApiServer] (qtp1766145591-16:ctx-a9b8d5fc ctx-ea23ada2) (logid:6acec26b) CIDRs from which account 'Acct[e4b9dbf4-6a86-11eb-8c34-08002799a040-admin]' is allowed to perform API calls: 0.0.0.0/0,::/0
2021-02-11 16:58:39,111 WARN [c.c.a.d.ParamGenericValidationWorker] (qtp1766145591-17:ctx-b459ae8c ctx-472fadd9) (logid:80c15691) Received unknown parameters for command listZones. Unknown parameters : listall
2021-02-11 16:58:39,116 DEBUG [c.c.a.ApiServlet] (qtp1766145591-15:ctx-dbb776f8 ctx-85fdd866) (logid:4324c0f4) ===END=== 192.168.56.1 -- GET username=admin&command=listUsers&response=json
2021-02-11 16:58:39,133 DEBUG [c.c.a.ApiServlet] (qtp1766145591-17:ctx-b459ae8c ctx-472fadd9) (logid:80c15691) ===END=== 192.168.56.1 -- GET listall=true&command=listZones&response=json
2021-02-11 16:58:39,134 DEBUG [c.c.a.ApiServlet] (qtp1766145591-16:ctx-a9b8d5fc ctx-ea23ada2) (logid:6acec26b) ===END=== 192.168.56.1 -- GET command=cloudianIsEnabled&response=json
2021-02-11 16:58:39,136 DEBUG [c.c.a.ApiServlet] (qtp1766145591-21:ctx-6e25da50 ctx-c1cb6a8c) (logid:2bcf881f) ===END=== 192.168.56.1 -- GET command=listCapabilities&response=json
2021-02-11 16:58:39,357 ERROR [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-2:null) (logid:) SSL error caught during wrap data: Empty server certificate chain, for local address=/192.168.56.85:8250, remote address=/192.168.56.85:34022.
2021-02-11 16:58:39,364 INFO [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-6:null) (logid:) Connection from /192.168.56.85 closed but no cleanup was done.
2021-02-11 16:58:40,808 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-077c3d23) (logid:7aeb1e26) Resetting hosts suitable for reconnect
2021-02-11 16:58:40,809 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-077c3d23) (logid:7aeb1e26) Completed resetting hosts suitable for reconnect
2021-02-11 16:58:40,809 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-077c3d23) (logid:7aeb1e26) Acquiring hosts for clusters already owned by this management server
2021-02-11 16:58:40,810 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-077c3d23) (logid:7aeb1e26) Completed acquiring hosts for clusters already owned by this management server
2021-02-11 16:58:40,810 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-077c3d23) (logid:7aeb1e26) Acquiring hosts for clusters not owned by any management server
2021-02-11 16:58:40,811 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-077c3d23) (logid:7aeb1e26) Completed acquiring hosts for clusters not owned by any management server
2021-02-11 16:58:42,169 DEBUG [c.c.a.ApiServlet] (qtp1766145591-22:ctx-404e580f ctx-ac96fd95) (logid:f70657f3) ===END=== 192.168.56.1 -- GET command=listApis&response=json
2021-02-11 16:58:42,650 DEBUG [c.c.a.ApiServlet] (qtp1766145591-19:ctx-b1d1408c) (logid:253ee22f) ===START=== 192.168.56.1 -- GET command=listZones&response=json
2021-02-11 16:58:42,653 DEBUG [c.c.a.ApiServlet] (qtp1766145591-18:ctx-281a2e2a) (logid:7e86594b) ===START=== 192.168.56.1 -- GET page=1&pagesize=6&listall=true&command=listEvents&response=json
2021-02-11 16:58:42,654 DEBUG [c.c.a.ApiServlet] (qtp1766145591-20:ctx-9dcfcbdb) (logid:aba30f6c) ===START=== 192.168.56.1 -- GET listAll=true&details=min&page=1&pageSize=500&command=listAndSwitchSamlAccount&response=json
2021-02-11 16:58:42,653 DEBUG [c.c.a.ApiServlet] (qtp1766145591-13:ctx-36942ae8) (logid:2d86e8c5) ===START=== 192.168.56.1 -- GET listAll=true&details=min&page=1&pageSize=500&command=listProjects&response=json
2021-02-11 16:58:42,677 DEBUG [c.c.a.ApiServer] (qtp1766145591-19:ctx-b1d1408c ctx-650d7135) (logid:253ee22f) CIDRs from which account 'Acct[e4b9dbf4-6a86-11eb-8c34-08002799a040-admin]' is allowed to perform API calls: 0.0.0.0/0,::/0
2021-02-11 16:58:42,679 DEBUG [c.c.a.ApiServer] (qtp1766145591-18:ctx-281a2e2a ctx-0d2835d6) (logid:7e86594b) CIDRs from which account 'Acct[e4b9dbf4-6a86-11eb-8c34-08002799a040-admin]' is allowed to perform API calls: 0.0.0.0/0,::/0
2021-02-11 16:58:42,680 DEBUG [c.c.a.ApiServer] (qtp1766145591-20:ctx-9dcfcbdb ctx-b3ced455) (logid:aba30f6c) CIDRs from which account 'Acct[e4b9dbf4-6a86-11eb-8c34-08002799a040-admin]' is allowed to perform API calls: 0.0.0.0/0,::/0
2021-02-11 16:58:42,680 DEBUG [c.c.a.ApiServer] (qtp1766145591-13:ctx-36942ae8 ctx-5337cb39) (logid:2d86e8c5) CIDRs from which account 'Acct[e4b9dbf4-6a86-11eb-8c34-08002799a040-admin]' is allowed to perform API calls: 0.0.0.0/0,::/0
2021-02-11 16:58:42,687 WARN [c.c.a.ApiServer] (qtp1766145591-20:ctx-9dcfcbdb ctx-b3ced455) (logid:aba30f6c) Unknown API command: listAndSwitchSamlAccount
2021-02-11 16:58:42,688 INFO [c.c.a.ApiServer] (qtp1766145591-20:ctx-9dcfcbdb ctx-b3ced455) (logid:aba30f6c) Unknown API command: listAndSwitchSamlAccount
2021-02-11 16:58:42,690 DEBUG [c.c.a.ApiServlet] (qtp1766145591-20:ctx-9dcfcbdb ctx-b3ced455) (logid:aba30f6c) ===END=== 192.168.56.1 -- GET listAll=true&details=min&page=1&pageSize=500&command=listAndSwitchSamlAccount&response=json
2021-02-11 16:58:42,724 DEBUG [c.c.a.ApiServlet] (qtp1766145591-18:ctx-281a2e2a ctx-0d2835d6) (logid:7e86594b) ===END=== 192.168.56.1 -- GET page=1&pagesize=6&listall=true&command=listEvents&response=json
2021-02-11 16:58:42,725 DEBUG [c.c.a.ApiServlet] (qtp1766145591-13:ctx-36942ae8 ctx-5337cb39) (logid:2d86e8c5) ===END=== 192.168.56.1 -- GET listAll=true&details=min&page=1&pageSize=500&command=listProjects&response=json
2021-02-11 16:58:42,725 DEBUG [c.c.a.ApiServlet] (qtp1766145591-19:ctx-b1d1408c ctx-650d7135) (logid:253ee22f) ===END=== 192.168.56.1 -- GET command=listZones&response=json
2021-02-11 16:58:42,732 DEBUG [c.c.a.ApiServlet] (qtp1766145591-15:ctx-d3d8574a) (logid:fdbe0337) ===START=== 192.168.56.1 -- GET command=listZones&response=json
2021-02-11 16:58:42,743 DEBUG [c.c.a.ApiServer] (qtp1766145591-15:ctx-d3d8574a ctx-eefd6224) (logid:fdbe0337) CIDRs from which account 'Acct[e4b9dbf4-6a86-11eb-8c34-08002799a040-admin]' is allowed to perform API calls: 0.0.0.0/0,::/0
2021-02-11 16:58:42,760 DEBUG [c.c.a.ApiServlet] (qtp1766145591-15:ctx-d3d8574a ctx-eefd6224) (logid:fdbe0337) ===END=== 192.168.56.1 -- GET command=listZones&response=json
2021-02-11 16:58:42,797 DEBUG [c.c.a.ApiServlet] (qtp1766145591-20:ctx-96d15575) (logid:2df28513) ===START=== 192.168.56.1 -- GET zoneid=49101c55-454f-4131-acb8-3fa6e81671eb&fetchlatest=false&command=listCapacity&response=json
2021-02-11 16:58:42,808 DEBUG [c.c.a.ApiServer] (qtp1766145591-20:ctx-96d15575 ctx-e125eef4) (logid:2df28513) CIDRs from which account 'Acct[e4b9dbf4-6a86-11eb-8c34-08002799a040-admin]' is allowed to perform API calls: 0.0.0.0/0,::/0
2021-02-11 16:58:42,825 DEBUG [c.c.a.ApiServlet] (qtp1766145591-20:ctx-96d15575 ctx-e125eef4) (logid:2df28513) ===END=== 192.168.56.1 -- GET zoneid=49101c55-454f-4131-acb8-3fa6e81671eb&fetchlatest=false&command=listCapacity&response=json
2021-02-11 16:58:44,448 ERROR [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-2:null) (logid:) SSL error caught during wrap data: Empty server certificate chain, for local address=/192.168.56.85:8250, remote address=/192.168.56.85:34024.
2021-02-11 16:58:44,453 INFO [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-8:null) (logid:) Connection from /192.168.56.85 closed but no cleanup was done.
2021-02-11 16:58:45,959 DEBUG [c.c.a.ApiServlet] (qtp1766145591-15:ctx-2db41c03) (logid:c89033b1) ===START=== 192.168.56.1 -- GET command=listInfrastructure&response=json
2021-02-11 16:58:45,969 DEBUG [c.c.a.ApiServer] (qtp1766145591-15:ctx-2db41c03 ctx-4e7e290d) (logid:c89033b1) CIDRs from which account 'Acct[e4b9dbf4-6a86-11eb-8c34-08002799a040-admin]' is allowed to perform API calls: 0.0.0.0/0,::/0
2021-02-11 16:58:46,007 DEBUG [c.c.a.ApiServlet] (qtp1766145591-15:ctx-2db41c03 ctx-4e7e290d) (logid:c89033b1) ===END=== 192.168.56.1 -- GET command=listInfrastructure&response=json
2021-02-11 16:58:48,736 DEBUG [c.c.a.ApiServlet] (qtp1766145591-20:ctx-802bbb6f) (logid:e44541a9) ===START=== 192.168.56.1 -- GET listall=true&page=1&pagesize=20&command=listStoragePoolsMetrics&response=json
2021-02-11 16:58:48,744 DEBUG [c.c.a.ApiServer] (qtp1766145591-20:ctx-802bbb6f ctx-fd1020ff) (logid:e44541a9) CIDRs from which account 'Acct[e4b9dbf4-6a86-11eb-8c34-08002799a040-admin]' is allowed to perform API calls: 0.0.0.0/0,::/0
2021-02-11 16:58:48,748 WARN [c.c.a.d.ParamGenericValidationWorker] (qtp1766145591-20:ctx-802bbb6f ctx-fd1020ff) (logid:e44541a9) Received unknown parameters for command listStoragePoolsMetrics. Unknown parameters : listall
2021-02-11 16:58:48,757 DEBUG [c.c.a.ApiServlet] (qtp1766145591-20:ctx-802bbb6f ctx-fd1020ff) (logid:e44541a9) ===END=== 192.168.56.1 -- GET listall=true&page=1&pagesize=20&command=listStoragePoolsMetrics&response=json
2021-02-11 16:58:49,542 ERROR [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-2:null) (logid:) SSL error caught during wrap data: Empty server certificate chain, for local address=/192.168.56.85:8250, remote address=/192.168.56.85:34026.
2021-02-11 16:58:49,547 INFO [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-10:null) (logid:) Connection from /192.168.56.85 closed but no cleanup was done.
2021-02-11 16:58:50,768 DEBUG [c.c.a.ApiServlet] (qtp1766145591-15:ctx-544ebca9) (logid:ba59e5ae) ===START=== 192.168.56.1 -- GET command=listZones&response=json
2021-02-11 16:58:50,769 DEBUG [c.c.a.ApiServlet] (qtp1766145591-17:ctx-9e5843fc) (logid:52456a77) ===START=== 192.168.56.1 -- GET type=primary&command=listStorageProviders&response=json
2021-02-11 16:58:50,770 DEBUG [c.c.a.ApiServlet] (qtp1766145591-20:ctx-fd7f54a8) (logid:b7fc8c93) ===START=== 192.168.56.1 -- GET command=listStorageTags&response=json
2021-02-11 16:58:50,781 DEBUG [c.c.a.ApiServer] (qtp1766145591-15:ctx-544ebca9 ctx-5a918db8) (logid:ba59e5ae) CIDRs from which account 'Acct[e4b9dbf4-6a86-11eb-8c34-08002799a040-admin]' is allowed to perform API calls: 0.0.0.0/0,::/0
2021-02-11 16:58:50,784 DEBUG [c.c.a.ApiServer] (qtp1766145591-17:ctx-9e5843fc ctx-6e1e57a2) (logid:52456a77) CIDRs from which account 'Acct[e4b9dbf4-6a86-11eb-8c34-08002799a040-admin]' is allowed to perform API calls: 0.0.0.0/0,::/0
2021-02-11 16:58:50,793 DEBUG [c.c.a.ApiServer] (qtp1766145591-20:ctx-fd7f54a8 ctx-8b64c32b) (logid:b7fc8c93) CIDRs from which account 'Acct[e4b9dbf4-6a86-11eb-8c34-08002799a040-admin]' is allowed to perform API calls: 0.0.0.0/0,::/0
2021-02-11 16:58:50,796 DEBUG [c.c.a.ApiServlet] (qtp1766145591-17:ctx-9e5843fc ctx-6e1e57a2) (logid:52456a77) ===END=== 192.168.56.1 -- GET type=primary&command=listStorageProviders&response=json
2021-02-11 16:58:50,803 DEBUG [c.c.a.ApiServlet] (qtp1766145591-15:ctx-544ebca9 ctx-5a918db8) (logid:ba59e5ae) ===END=== 192.168.56.1 -- GET command=listZones&response=json
2021-02-11 16:58:50,805 DEBUG [c.c.a.ApiServlet] (qtp1766145591-20:ctx-fd7f54a8 ctx-8b64c32b) (logid:b7fc8c93) ===END=== 192.168.56.1 -- GET command=listStorageTags&response=json
2021-02-11 16:58:50,997 DEBUG [c.c.a.ApiServlet] (qtp1766145591-16:ctx-4cba34b6) (logid:b3282c3d) ===START=== 192.168.56.1 -- GET zoneid=49101c55-454f-4131-acb8-3fa6e81671eb&command=listPods&response=json
2021-02-11 16:58:51,005 DEBUG [c.c.a.ApiServer] (qtp1766145591-16:ctx-4cba34b6 ctx-c1e1e3fd) (logid:b3282c3d) CIDRs from which account 'Acct[e4b9dbf4-6a86-11eb-8c34-08002799a040-admin]' is allowed to perform API calls: 0.0.0.0/0,::/0
2021-02-11 16:58:51,021 DEBUG [c.c.a.ApiServlet] (qtp1766145591-16:ctx-4cba34b6 ctx-c1e1e3fd) (logid:b3282c3d) ===END=== 192.168.56.1 -- GET zoneid=49101c55-454f-4131-acb8-3fa6e81671eb&command=listPods&response=json
2021-02-11 16:58:51,090 DEBUG [c.c.a.ApiServlet] (qtp1766145591-17:ctx-0a381b5d) (logid:daf10f2e) ===START=== 192.168.56.1 -- GET podid=d5274d8a-4878-44f3-89ab-ba4654299e1a&command=listClusters&response=json
2021-02-11 16:58:51,103 DEBUG [c.c.a.ApiServer] (qtp1766145591-17:ctx-0a381b5d ctx-42a8c25b) (logid:daf10f2e) CIDRs from which account 'Acct[e4b9dbf4-6a86-11eb-8c34-08002799a040-admin]' is allowed to perform API calls: 0.0.0.0/0,::/0
2021-02-11 16:58:51,123 DEBUG [c.c.a.ApiServlet] (qtp1766145591-17:ctx-0a381b5d ctx-42a8c25b) (logid:daf10f2e) ===END=== 192.168.56.1 -- GET podid=d5274d8a-4878-44f3-89ab-ba4654299e1a&command=listClusters&response=json
2021-02-11 16:58:51,176 DEBUG [c.c.a.ApiServlet] (qtp1766145591-16:ctx-de6a2ec6) (logid:baea16d4) ===START=== 192.168.56.1 -- GET clusterid=acd89d8e-d25b-4144-8b74-a8b08d544bf1&command=listHosts&response=json
2021-02-11 16:58:51,185 DEBUG [c.c.a.ApiServer] (qtp1766145591-16:ctx-de6a2ec6 ctx-9db9874c) (logid:baea16d4) CIDRs from which account 'Acct[e4b9dbf4-6a86-11eb-8c34-08002799a040-admin]' is allowed to perform API calls: 0.0.0.0/0,::/0
2021-02-11 16:58:51,190 DEBUG [c.c.a.q.QueryManagerImpl] (qtp1766145591-16:ctx-de6a2ec6 ctx-9db9874c) (logid:baea16d4) >>>Searching for hosts>>>
2021-02-11 16:58:51,195 DEBUG [c.c.a.q.QueryManagerImpl] (qtp1766145591-16:ctx-de6a2ec6 ctx-9db9874c) (logid:baea16d4) >>>Generating Response>>>
2021-02-11 16:58:51,195 DEBUG [c.c.a.ApiServlet] (qtp1766145591-16:ctx-de6a2ec6 ctx-9db9874c) (logid:baea16d4) ===END=== 192.168.56.1 -- GET clusterid=acd89d8e-d25b-4144-8b74-a8b08d544bf1&command=listHosts&response=json
2021-02-11 16:58:52,112 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-3e125cd5) (logid:8004a04b) Skip capacity scan as there is no Primary Storage in 'Up' state
2021-02-11 16:58:54,613 ERROR [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-2:null) (logid:) SSL error caught during wrap data: Empty server certificate chain, for local address=/192.168.56.85:8250, remote address=/192.168.56.85:34028.
2021-02-11 16:58:54,619 INFO [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-12:null) (logid:) Connection from /192.168.56.85 closed but no cleanup was done.
2021-02-11 16:58:59,702 ERROR [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-2:null) (logid:) SSL error caught during wrap data: Empty server certificate chain, for local address=/192.168.56.85:8250, remote address=/192.168.56.85:34030.
2021-02-11 16:58:59,708 INFO [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-14:null) (logid:) Connection from /192.168.56.85 closed but no cleanup was done.
2021-02-11 16:59:04,801 ERROR [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-2:null) (logid:) SSL error caught during wrap data: Empty server certificate chain, for local address=/192.168.56.85:8250, remote address=/192.168.56.85:34032.
2021-02-11 16:59:04,812 INFO [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-1:null) (logid:) Connection from /192.168.56.85 closed but no cleanup was done.
2021-02-11 16:59:06,092 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-124be51a) (logid:73ac7ab2) Found 0 routers to update status.
2021-02-11 16:59:06,097 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-124be51a) (logid:73ac7ab2) Found 0 VPC's to update Redundant State.
2021-02-11 16:59:06,102 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-124be51a) (logid:73ac7ab2) Found 0 networks to update RvR status.
2021-02-11 16:59:08,629 DEBUG [c.c.a.ApiServlet] (qtp1766145591-17:ctx-af578a13) (logid:e6a6d9a8) ===START=== 192.168.56.1 -- POST command=createStoragePool&response=json
2021-02-11 16:59:08,641 DEBUG [c.c.a.ApiServer] (qtp1766145591-17:ctx-af578a13 ctx-c967a7cc) (logid:e6a6d9a8) CIDRs from which account 'Acct[e4b9dbf4-6a86-11eb-8c34-08002799a040-admin]' is allowed to perform API calls: 0.0.0.0/0,::/0
2021-02-11 16:59:08,666 DEBUG [o.a.c.s.d.l.CloudStackPrimaryDataStoreLifeCycleImpl] (qtp1766145591-17:ctx-af578a13 ctx-c967a7cc) (logid:e6a6d9a8) createPool Params @ scheme - nfs storageHost - 192.168.56.85 hostPath - /export/primary port - -1
2021-02-11 16:59:08,807 DEBUG [c.c.s.StorageManagerImpl] (qtp1766145591-17:ctx-af578a13 ctx-c967a7cc) (logid:e6a6d9a8) Failed to add data store: No host up to associate a storage pool with in cluster 4
com.cloud.utils.exception.CloudRuntimeException: No host up to associate a storage pool with in cluster 4
at org.apache.cloudstack.storage.datastore.lifecycle.CloudStackPrimaryDataStoreLifeCycleImpl.attachCluster(CloudStackPrimaryDataStoreLifeCycleImpl.java:430)
at com.cloud.storage.StorageManagerImpl.createPool(StorageManagerImpl.java:741)
at com.cloud.storage.StorageManagerImpl.createPool(StorageManagerImpl.java:210)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy96.createPool(Unknown Source)
at org.apache.cloudstack.api.command.admin.storage.CreateStoragePoolCmd.execute(CreateStoragePoolCmd.java:163)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:156)
at com.cloud.api.ApiServer.queueCommand(ApiServer.java:764)
at com.cloud.api.ApiServer.handleRequest(ApiServer.java:588)
at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:321)
at com.cloud.api.ApiServlet$1.run(ApiServlet.java:134)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:131)
at com.cloud.api.ApiServlet.doPost(ApiServlet.java:98)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:665)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
at org.eclipse.jetty.servlet.ServletHolder$NotAsyncServlet.service(ServletHolder.java:1386)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:755)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:547)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1610)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1300)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1580)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1215)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:767)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:500)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
at java.base/java.lang.Thread.run(Thread.java:834)
2021-02-11 16:59:08,815 DEBUG [c.c.s.StorageManagerImpl] (qtp1766145591-17:ctx-af578a13 ctx-c967a7cc) (logid:e6a6d9a8) Failed to clean up storage pool: null
2021-02-11 16:59:08,815 INFO [c.c.a.ApiServer] (qtp1766145591-17:ctx-af578a13 ctx-c967a7cc) (logid:e6a6d9a8) Failed to add data store: No host up to associate a storage pool with in cluster 4
2021-02-11 16:59:08,816 DEBUG [c.c.a.ApiServlet] (qtp1766145591-17:ctx-af578a13 ctx-c967a7cc) (logid:e6a6d9a8) ===END=== 192.168.56.1 -- POST command=createStoragePool&response=json
2021-02-11 16:59:09,885 ERROR [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-2:null) (logid:) SSL error caught during wrap data: Empty server certificate chain, for local address=/192.168.56.85:8250, remote address=/192.168.56.85:34034.
2021-02-11 16:59:09,888 INFO [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-3:null) (logid:) Connection from /192.168.56.85 closed but no cleanup was done.
2021-02-11 16:59:11,388 DEBUG [c.c.a.ApiServlet] (qtp1766145591-21:ctx-5752d23f) (logid:679c0528) ===START=== 192.168.56.1 -- GET listall=true&page=1&pagesize=20&command=listImageStores&response=json
2021-02-11 16:59:11,406 DEBUG [c.c.a.ApiServer] (qtp1766145591-21:ctx-5752d23f ctx-d0c09938) (logid:679c0528) CIDRs from which account 'Acct[e4b9dbf4-6a86-11eb-8c34-08002799a040-admin]' is allowed to perform API calls: 0.0.0.0/0,::/0
2021-02-11 16:59:11,416 WARN [c.c.a.d.ParamGenericValidationWorker] (qtp1766145591-21:ctx-5752d23f ctx-d0c09938) (logid:679c0528) Received unknown parameters for command listImageStores. Unknown parameters : listall
2021-02-11 16:59:11,423 DEBUG [c.c.a.ApiServlet] (qtp1766145591-21:ctx-5752d23f ctx-d0c09938) (logid:679c0528) ===END=== 192.168.56.1 -- GET listall=true&page=1&pagesize=20&command=listImageStores&response=json
2021-02-11 16:59:12,956 DEBUG [c.c.a.ApiServlet] (qtp1766145591-22:ctx-6573050f) (logid:9b99717e) ===START=== 192.168.56.1 -- GET command=listZones&response=json
2021-02-11 16:59:12,965 DEBUG [c.c.a.ApiServer] (qtp1766145591-22:ctx-6573050f ctx-740cb711) (logid:9b99717e) CIDRs from which account 'Acct[e4b9dbf4-6a86-11eb-8c34-08002799a040-admin]' is allowed to perform API calls: 0.0.0.0/0,::/0
2021-02-11 16:59:12,982 DEBUG [c.c.a.ApiServlet] (qtp1766145591-22:ctx-6573050f ctx-740cb711) (logid:9b99717e) ===END=== 192.168.56.1 -- GET command=listZones&response=json
2021-02-11 16:59:14,981 ERROR [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-2:null) (logid:) SSL error caught during wrap data: Empty server certificate chain, for local address=/192.168.56.85:8250, remote address=/192.168.56.85:34036.
2021-02-11 16:59:14,995 INFO [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-5:null) (logid:) Connection from /192.168.56.85 closed but no cleanup was done.
2021-02-11 16:59:20,080 ERROR [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-2:null) (logid:) SSL error caught during wrap data: Empty server certificate chain, for local address=/192.168.56.85:8250, remote address=/192.168.56.85:34038.
2021-02-11 16:59:20,083 INFO [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-7:null) (logid:) Connection from /192.168.56.85 closed but no cleanup was done.
2021-02-11 16:59:21,240 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-7e0a37be) (logid:c7f23758) AutoScaling Monitor is running...
2021-02-11 16:59:21,252 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:ctx-26c1c289) (logid:2e30b588) HostStatsCollector is running...
2021-02-11 16:59:21,452 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:ctx-e78c0b57) (logid:718cfc59) StorageCollector is running...
2021-02-11 16:59:22,102 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-dd0caf32) (logid:83ad9e78) Skip capacity scan as there is no Primary Storage in 'Up' state
2021-02-11 16:59:25,146 ERROR [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-2:null) (logid:) SSL error caught during wrap data: Empty server certificate chain, for local address=/192.168.56.85:8250, remote address=/192.168.56.85:34040.
2021-02-11 16:59:25,154 INFO [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-9:null) (logid:) Connection from /192.168.56.85 closed but no cleanup was done.
2021-02-11 16:59:27,263 DEBUG [c.c.a.ApiServlet] (qtp1766145591-22:ctx-5a199363) (logid:327b3b04) ===START=== 192.168.56.1 -- GET name=secondaire&url=nfs:%2F%2F192.168.56.85%2Fexport%2Fsecondary&provider=NFS&zoneid=49101c55-454f-4131-acb8-3fa6e81671eb&command=addImageStore&response=json
2021-02-11 16:59:27,272 DEBUG [c.c.a.ApiServer] (qtp1766145591-22:ctx-5a199363 ctx-ecf15a4c) (logid:327b3b04) CIDRs from which account 'Acct[e4b9dbf4-6a86-11eb-8c34-08002799a040-admin]' is allowed to perform API calls: 0.0.0.0/0,::/0
2021-02-11 16:59:27,297 INFO [o.a.c.s.d.l.CloudStackImageStoreLifeCycleImpl] (qtp1766145591-22:ctx-5a199363 ctx-ecf15a4c) (logid:327b3b04) Trying to add a new data store at nfs://192.168.56.85/export/secondary to data center 1
2021-02-11 16:59:27,430 ERROR [o.a.c.s.i.TemplateServiceImpl] (qtp1766145591-22:ctx-5a199363 ctx-ecf15a4c) (logid:327b3b04) No remote endpoint to send command, check if host or ssvm is down?
2021-02-11 16:59:27,431 DEBUG [o.a.c.s.i.TemplateServiceImpl] (qtp1766145591-22:ctx-5a199363 ctx-ecf15a4c) (logid:327b3b04) can not list template for secondary storage host 3
2021-02-11 16:59:27,447 DEBUG [c.c.a.ApiServlet] (qtp1766145591-22:ctx-5a199363 ctx-ecf15a4c) (logid:327b3b04) ===END=== 192.168.56.1 -- GET name=secondaire&url=nfs:%2F%2F192.168.56.85%2Fexport%2Fsecondary&provider=NFS&zoneid=49101c55-454f-4131-acb8-3fa6e81671eb&command=addImageStore&response=json
2021-02-11 16:59:27,504 DEBUG [c.c.a.ApiServlet] (qtp1766145591-16:ctx-0d943231) (logid:50c82862) ===START=== 192.168.56.1 -- GET listall=true&page=1&pagesize=20&command=listImageStores&response=json
2021-02-11 16:59:27,512 DEBUG [c.c.a.ApiServer] (qtp1766145591-16:ctx-0d943231 ctx-f87c5e98) (logid:50c82862) CIDRs from which account 'Acct[e4b9dbf4-6a86-11eb-8c34-08002799a040-admin]' is allowed to perform API calls: 0.0.0.0/0,::/0
2021-02-11 16:59:27,526 WARN [c.c.a.d.ParamGenericValidationWorker] (qtp1766145591-16:ctx-0d943231 ctx-f87c5e98) (logid:50c82862) Received unknown parameters for command listImageStores. Unknown parameters : listall
2021-02-11 16:59:27,534 DEBUG [c.c.a.ApiServlet] (qtp1766145591-16:ctx-0d943231 ctx-f87c5e98) (logid:50c82862) ===END=== 192.168.56.1 -- GET listall=true&page=1&pagesize=20&command=listImageStores&response=json
2021-02-11 16:59:30,239 ERROR [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-2:null) (logid:) SSL error caught during wrap data: Empty server certificate chain, for local address=/192.168.56.85:8250, remote address=/192.168.56.85:34042.
2021-02-11 16:59:30,244 INFO [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-11:null) (logid:) Connection from /192.168.56.85 closed but no cleanup was done.
2021-02-11 16:59:30,355 DEBUG [c.c.a.ApiServlet] (qtp1766145591-22:ctx-68ebde37) (logid:339b3a73) ===START=== 192.168.56.1 -- GET listall=true&type=routing&page=1&pagesize=20&command=listHostsMetrics&response=json
2021-02-11 16:59:30,367 DEBUG [c.c.a.ApiServer] (qtp1766145591-22:ctx-68ebde37 ctx-f9fbe647) (logid:339b3a73) CIDRs from which account 'Acct[e4b9dbf4-6a86-11eb-8c34-08002799a040-admin]' is allowed to perform API calls: 0.0.0.0/0,::/0
2021-02-11 16:59:30,376 WARN [c.c.a.d.ParamGenericValidationWorker] (qtp1766145591-22:ctx-68ebde37 ctx-f9fbe647) (logid:339b3a73) Received unknown parameters for command listHostsMetrics. Unknown parameters : listall
2021-02-11 16:59:30,377 DEBUG [c.c.a.q.QueryManagerImpl] (qtp1766145591-22:ctx-68ebde37 ctx-f9fbe647) (logid:339b3a73) >>>Searching for hosts>>>
2021-02-11 16:59:30,384 DEBUG [c.c.a.q.QueryManagerImpl] (qtp1766145591-22:ctx-68ebde37 ctx-f9fbe647) (logid:339b3a73) >>>Generating Response>>>
2021-02-11 16:59:30,385 DEBUG [c.c.a.ApiServlet] (qtp1766145591-22:ctx-68ebde37 ctx-f9fbe647) (logid:339b3a73) ===END=== 192.168.56.1 -- GET listall=true&type=routing&page=1&pagesize=20&command=listHostsMetrics&response=json
2021-02-11 16:59:31,942 DEBUG [c.c.a.ApiServlet] (qtp1766145591-16:ctx-4cb12cca) (logid:8a8d7253) ===START=== 192.168.56.1 -- GET command=listZones&response=json
2021-02-11 16:59:31,944 DEBUG [c.c.a.ApiServlet] (qtp1766145591-22:ctx-bfe20a38) (logid:eb8f6f48) ===START=== 192.168.56.1 -- GET command=listHostTags&response=json
2021-02-11 16:59:31,958 DEBUG [c.c.a.ApiServer] (qtp1766145591-16:ctx-4cb12cca ctx-ffcf85dc) (logid:8a8d7253) CIDRs from which account 'Acct[e4b9dbf4-6a86-11eb-8c34-08002799a040-admin]' is allowed to perform API calls: 0.0.0.0/0,::/0
2021-02-11 16:59:31,959 DEBUG [c.c.a.ApiServer] (qtp1766145591-22:ctx-bfe20a38 ctx-d8b0f446) (logid:eb8f6f48) CIDRs from which account 'Acct[e4b9dbf4-6a86-11eb-8c34-08002799a040-admin]' is allowed to perform API calls: 0.0.0.0/0,::/0
2021-02-11 16:59:31,971 DEBUG [c.c.a.ApiServlet] (qtp1766145591-22:ctx-bfe20a38 ctx-d8b0f446) (logid:eb8f6f48) ===END=== 192.168.56.1 -- GET command=listHostTags&response=json
2021-02-11 16:59:31,972 DEBUG [c.c.a.ApiServlet] (qtp1766145591-16:ctx-4cb12cca ctx-ffcf85dc) (logid:8a8d7253) ===END=== 192.168.56.1 -- GET command=listZones&response=json
2021-02-11 16:59:32,008 DEBUG [c.c.a.ApiServlet] (qtp1766145591-19:ctx-f8544c2a) (logid:b6903fab) ===START=== 192.168.56.1 -- GET zoneid=49101c55-454f-4131-acb8-3fa6e81671eb&command=listPods&response=json
2021-02-11 16:59:32,018 DEBUG [c.c.a.ApiServer] (qtp1766145591-19:ctx-f8544c2a ctx-9caaa3f8) (logid:b6903fab) CIDRs from which account 'Acct[e4b9dbf4-6a86-11eb-8c34-08002799a040-admin]' is allowed to perform API calls: 0.0.0.0/0,::/0
2021-02-11 16:59:32,031 DEBUG [c.c.a.ApiServlet] (qtp1766145591-19:ctx-f8544c2a ctx-9caaa3f8) (logid:b6903fab) ===END=== 192.168.56.1 -- GET zoneid=49101c55-454f-4131-acb8-3fa6e81671eb&command=listPods&response=json
2021-02-11 16:59:32,057 DEBUG [c.c.a.ApiServlet] (qtp1766145591-16:ctx-11d8bd37) (logid:d783b1a4) ===START=== 192.168.56.1 -- GET podid=d5274d8a-4878-44f3-89ab-ba4654299e1a&command=listClusters&response=json
2021-02-11 16:59:32,066 DEBUG [c.c.a.ApiServer] (qtp1766145591-16:ctx-11d8bd37 ctx-c53bbcde) (logid:d783b1a4) CIDRs from which account 'Acct[e4b9dbf4-6a86-11eb-8c34-08002799a040-admin]' is allowed to perform API calls: 0.0.0.0/0,::/0
2021-02-11 16:59:32,082 DEBUG [c.c.a.ApiServlet] (qtp1766145591-16:ctx-11d8bd37 ctx-c53bbcde) (logid:d783b1a4) ===END=== 192.168.56.1 -- GET podid=d5274d8a-4878-44f3-89ab-ba4654299e1a&command=listClusters&response=json
2021-02-11 16:59:35,340 ERROR [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-2:null) (logid:) SSL error caught during wrap data: Empty server certificate chain, for local address=/192.168.56.85:8250, remote address=/192.168.56.85:34044.
2021-02-11 16:59:35,343 INFO [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-13:null) (logid:) Connection from /192.168.56.85 closed but no cleanup was done.
2021-02-11 16:59:36,090 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-900bd7de) (logid:fadcceb3) Found 0 routers to update status.
2021-02-11 16:59:36,094 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-900bd7de) (logid:fadcceb3) Found 0 VPC's to update Redundant State.
2021-02-11 16:59:36,098 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-900bd7de) (logid:fadcceb3) Found 0 networks to update RvR status.
2021-02-11 16:59:40,393 ERROR [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-2:null) (logid:) SSL error caught during wrap data: Empty server certificate chain, for local address=/192.168.56.85:8250, remote address=/192.168.56.85:34046.
2021-02-11 16:59:40,398 INFO [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-15:null) (logid:) Connection from /192.168.56.85 closed but no cleanup was done.
2021-02-11 16:59:42,398 DEBUG [c.c.a.ApiServlet] (qtp1766145591-19:ctx-814b5d53) (logid:88b770f8) ===START=== 192.168.56.1 -- POST command=addHost&response=json
2021-02-11 16:59:42,411 DEBUG [c.c.a.ApiServer] (qtp1766145591-19:ctx-814b5d53 ctx-64f6dac6) (logid:88b770f8) CIDRs from which account 'Acct[e4b9dbf4-6a86-11eb-8c34-08002799a040-admin]' is allowed to perform API calls: 0.0.0.0/0,::/0
2021-02-11 16:59:42,426 WARN [c.c.a.d.ParamGenericValidationWorker] (qtp1766145591-19:ctx-814b5d53 ctx-64f6dac6) (logid:88b770f8) Received unknown parameters for command addHost. Unknown parameters : clustertype
2021-02-11 16:59:42,433 INFO [c.c.r.ResourceManagerImpl] (qtp1766145591-19:ctx-814b5d53 ctx-64f6dac6) (logid:88b770f8) Trying to add a new host at http://192.168.56.85 in data center 1
2021-02-11 16:59:42,512 WARN [c.c.h.k.d.LibvirtServerDiscoverer] (qtp1766145591-19:ctx-814b5d53 ctx-64f6dac6) (logid:88b770f8) can't setup agent, due to java.io.IOException: There was a problem while connecting to 192.168.56.85:22 - There was a problem while connecting to 192.168.56.85:22
2021-02-11 16:59:42,513 WARN [c.c.r.ResourceManagerImpl] (qtp1766145591-19:ctx-814b5d53 ctx-64f6dac6) (logid:88b770f8) Unable to find the server resources at http://192.168.56.85
2021-02-11 16:59:42,513 INFO [c.c.u.e.CSExceptionErrorCode] (qtp1766145591-19:ctx-814b5d53 ctx-64f6dac6) (logid:88b770f8) Could not find exception: com.cloud.exception.DiscoveryException in error code list for exceptions
2021-02-11 16:59:42,513 WARN [o.a.c.a.c.a.h.AddHostCmd] (qtp1766145591-19:ctx-814b5d53 ctx-64f6dac6) (logid:88b770f8) Exception:
com.cloud.exception.DiscoveryException: Unable to add the host
at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:826)
at com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.java:612)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy188.discoverHosts(Unknown Source)
at org.apache.cloudstack.api.command.admin.host.AddHostCmd.execute(AddHostCmd.java:142)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:156)
at com.cloud.api.ApiServer.queueCommand(ApiServer.java:764)
at com.cloud.api.ApiServer.handleRequest(ApiServer.java:588)
at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:321)
at com.cloud.api.ApiServlet$1.run(ApiServlet.java:134)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:131)
at com.cloud.api.ApiServlet.doPost(ApiServlet.java:98)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:665)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
at org.eclipse.jetty.servlet.ServletHolder$NotAsyncServlet.service(ServletHolder.java:1386)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:755)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:547)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1610)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1300)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1580)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1215)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:767)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:500)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
at java.base/java.lang.Thread.run(Thread.java:834)
2021-02-11 16:59:42,516 INFO [c.c.a.ApiServer] (qtp1766145591-19:ctx-814b5d53 ctx-64f6dac6) (logid:88b770f8) Unable to add the host
2021-02-11 16:59:42,517 DEBUG [c.c.a.ApiServlet] (qtp1766145591-19:ctx-814b5d53 ctx-64f6dac6) (logid:88b770f8) ===END=== 192.168.56.1 -- POST command=addHost&response=json
2021-02-11 16:59:45,464 ERROR [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-2:null) (logid:) SSL error caught during wrap data: Empty server certificate chain, for local address=/192.168.56.85:8250, remote address=/192.168.56.85:34050.
2021-02-11 16:59:45,466 INFO [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-2:null) (logid:) Connection from /192.168.56.85 closed but no cleanup was done.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] weizhouapache commented on issue #4659: Agent : SSL Handshake failed while connecting to host on port: 8250
Posted by GitBox <gi...@apache.org>.
weizhouapache commented on issue #4659:
URL: https://github.com/apache/cloudstack/issues/4659#issuecomment-781675256
> #key_file="/etc/pki/libvirt/private/serverkey.pem"
> #cert_file="/etc/pki/libvirt/servercert.pem"
> #ca_file="/etc/pki/CA/cacert.pem"
@abdelouahabb you use centos ? you need to use 'libvirt' instead of 'libvirtd.
The other lines should be same. the following lines should be uncommented.
```
#tls_port="16514"
auth_tcp="none"
auth_tls="none"
#key_file="/etc/pki/libvirt/private/serverkey.pem"
#cert_file="/etc/pki/libvirt/servercert.pem"
#ca_file="/etc/pki/CA/cacert.pem"
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] abdelouahabb edited a comment on issue #4659: Agent : SSL Handshake failed while connecting to host on port: 8250
Posted by GitBox <gi...@apache.org>.
abdelouahabb edited a comment on issue #4659:
URL: https://github.com/apache/cloudstack/issues/4659#issuecomment-781669388
> > ontent of the configuration file of libvirtd:
> > ```
> > log_level=1
> > listen_tls=1
> > mdns_adv=0
> > listen_tcp=0
> > tcp_port="16509"
> > unix_sock_group = "libvirt"
> > unix_sock_ro_perms = "0777"
> > unix_sock_rw_perms = "0770"
> > #tls_port="16514"
> > auth_tcp="none"
> > auth_tls="none"
> > #key_file="/etc/pki/libvirt/private/serverkey.pem"
> > #cert_file="/etc/pki/libvirt/servercert.pem"
> > #ca_file="/etc/pki/CA/cacert.pem"
> > ```
>
> @abdelouahabb
> the libvirtd.conf above is incorrect.
>
> here is the content of libvirtd.conf on a server in my testing env.
>
> ```
> # cat /etc/libvirt/libvirtd.conf |egrep -v "^$|^#"
> unix_sock_group = "libvirtd"
> unix_sock_ro_perms = "0777"
> unix_sock_rw_perms = "0770"
> auth_unix_ro = "none"
> auth_unix_rw = "none"
> listen_tcp=0
> listen_tls=1
> key_file="/etc/pki/libvirt/private/serverkey.pem"
> cert_file="/etc/pki/libvirt/servercert.pem"
> ca_file="/etc/pki/CA/cacert.pem"
> tcp_port="16509"
> tls_port="16514"
> auth_tcp="none"
> auth_tls="none"
> ```
This is the same ? the group `libvirtd` dont exist in my machine, the group is called `libvirt` :
```
abdelouahab@akham:~$
>> cat /etc/group | grep libvirt
libvirt:x:121:abdelouahab
libvirt-qemu:x:64055:libvirt-qemu
libvirt-dnsmasq:x:122:
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] abdelouahabb commented on issue #4659: Agent : SSL Handshake failed while connecting to host on port: 8250
Posted by GitBox <gi...@apache.org>.
abdelouahabb commented on issue #4659:
URL: https://github.com/apache/cloudstack/issues/4659#issuecomment-781679868
> > #key_file="/etc/pki/libvirt/private/serverkey.pem"
> > #cert_file="/etc/pki/libvirt/servercert.pem"
> > #ca_file="/etc/pki/CA/cacert.pem"
>
> @abdelouahabb you use centos ? you need to use 'libvirt' instead of 'libvirtd.
> The other lines should be same. the following lines should be uncommented.
>
> ```
> #tls_port="16514"
> auth_tcp="none"
> auth_tls="none"
> #key_file="/etc/pki/libvirt/private/serverkey.pem"
> #cert_file="/etc/pki/libvirt/servercert.pem"
> #ca_file="/etc/pki/CA/cacert.pem"
> ```
I am on Ubuntu, I [disabled the security](https://libvirt.org/daemons.html) for the debug reason and getting the same issue
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] weizhouapache commented on issue #4659: Agent : SSL Handshake failed while connecting to host on port: 8250
Posted by GitBox <gi...@apache.org>.
weizhouapache commented on issue #4659:
URL: https://github.com/apache/cloudstack/issues/4659#issuecomment-782577728
@abdelouahabb
I have seen these logs in #4715
```
2021-02-20 00:42:47,784 DEBUG [c.c.u.s.SSHCmdHelper] (qtp182531396-22:ctx-6c2d2d39 ctx-e64efde7) (logid:980a971b) Executing cmd: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/
2021-02-20 00:42:48,844 DEBUG [c.c.u.s.SSHCmdHelper] (qtp182531396-22:ctx-6c2d2d39 ctx-e64efde7) (logid:980a971b) SSH command: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/
SSH command output:
sudo: no tty present and no askpass program specified
```
it seems some misconfiguration with sudo.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] ThyLAW commented on issue #4659: Agent : SSL Handshake failed while connecting to host on port: 8250
Posted by GitBox <gi...@apache.org>.
ThyLAW commented on issue #4659:
URL: https://github.com/apache/cloudstack/issues/4659#issuecomment-781645064
> > I'm getting nearly the same issues as you. I was also experiencing the SSL problem, and disabled it in global configs. I then got the "unable to find private nics" and manually configured that in my agent.properties. I now am just getting
> > "error: conection with libvirtd is broken : invalid connection pointer in virConnectGetVersion" as well as the UEFI.properties error.
> > And on libvirtd status I am getting :
> > "47271 error: virNetSocketReadWire:1806: end of file while reading data : Input/output error"
> > Please let me know if you have found a solution.
>
> @ThyLAW I am getting the same issue too with libvirt, what is your setup ? virtualizing it on bare metal or on top of another hypervisor ?
> @DaanHoogland any update ?
On top of a hypervisor .Hyper-v gen 2 This is how I set it up.
[New.Apache.Documentatino.docx](https://github.com/apache/cloudstack/files/6005970/New.Apache.Documentatino.docx)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] abdelouahabb edited a comment on issue #4659: Agent : SSL Handshake failed while connecting to host on port: 8250
Posted by GitBox <gi...@apache.org>.
abdelouahabb edited a comment on issue #4659:
URL: https://github.com/apache/cloudstack/issues/4659#issuecomment-776866417
> @abdelouahabb this is working for loads of people all over the world. I suggest you ask for help on [users@cloudstack.apache.org](mailto:users@cloudstack.apache.org)
> By the looks of your log, the problem is not with ssh but with the certificates or the jdk encryption policies available to your host or management server.
Checked the policy and it is enable :
` nano $JAVA_HOME/conf/security/java.security` and the value is `crypto.policy=unlimited`
```
java --version
openjdk 11.0.10 2021-01-19
OpenJDK Runtime Environment (build 11.0.10+9-Ubuntu-0ubuntu1.20.04)
OpenJDK 64-Bit Server VM (build 11.0.10+9-Ubuntu-0ubuntu1.20.04, mixed mode, sharing)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on issue #4659: Agent : SSL Handshake failed while connecting to host on port: 8250
Posted by GitBox <gi...@apache.org>.
rhtyd commented on issue #4659:
URL: https://github.com/apache/cloudstack/issues/4659#issuecomment-784108280
@ThyLAW @abdelouahabb is sudo and haveged installed on your system? Also note CloudStack required full virtualisation on KVM (PV may not work or at least not tested, usually you can use VMware workstation/fusion than VirtualBox on Windows/Mac and KVM on Linux, i.e. on your laptop/desktop). For Ubuntu-based quick install guide, you can refer to https://rohityadav.cloud/blog/cloudstack-kvm/ (change the version/repo accordingly) - see if there's any step you missed?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] DaanHoogland commented on issue #4659: Agent : SSL Handshake failed while connecting to host on port: 8250
Posted by GitBox <gi...@apache.org>.
DaanHoogland commented on issue #4659:
URL: https://github.com/apache/cloudstack/issues/4659#issuecomment-782877614
@ThyLAW check if there are any remnant certificates on the host and whether empty the certs table. hope that helps.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] abdelouahabb edited a comment on issue #4659: Agent : SSL Handshake failed while connecting to host on port: 8250
Posted by GitBox <gi...@apache.org>.
abdelouahabb edited a comment on issue #4659:
URL: https://github.com/apache/cloudstack/issues/4659#issuecomment-777132557
> @abdelouahabb
> can you try the following step on node ?
> (1) rm /etc/cloudstack/agent/cloud.*
> (2) systemctl restart cloudstack-agent
> if it does not work, remove host from cloudstack, and re-add it.
It dident work,
I even deleted the certificates and installed new ones :
```
sudo dpkg --purge --force-depends ca-certificates-java
sudo apt-get install ca-certificates-java
```
And here is my network configuration :
```
auto lo
iface lo inet loopback
auto enp0s8
iface enp0s8 inet dhcp
auto enp0s3
iface enp0s3 inet manual
auto cloudbr0
iface cloudbr0 inet static
address 192.168.56.85
netmask 255.255.255.0
gateway 192.168.56.1 metric 100
dns-nameservers 8.8.8.8
bridge_ports enp0s3
bridge_fd 5
bridge_stp off
bridge_maxwait 1
auto cloudbr1
iface cloudbr1 inet manual
bridge_ports none
bridge_fd 5
bridge_stp off
bridge_maxwait 1
```
Is it a bug with Ubuntu 20.04 ?
```
cat /etc/apt/sources.list.d/cloudstack.list
deb http://cloudstack.apt-get.eu/ubuntu focal 4.15
```
```
>> cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.2 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.2 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd edited a comment on issue #4659: Agent : SSL Handshake failed while connecting to host on port: 8250
Posted by GitBox <gi...@apache.org>.
rhtyd edited a comment on issue #4659:
URL: https://github.com/apache/cloudstack/issues/4659#issuecomment-791906236
@abdelouahabb since you've disabled auth strictness you may want to do the same on your libvirtd, set the listen_tls = 0 and listen_tcp = 1 in libvirtd conf, restart libvirtd and try again.
Since the original issue of agent SSL failure was fixed with the global setting workaround, I'll close the ticket. Please open new ticket for unrelated failures. I think you're using a nested env in VirtualBox and the env lack entropy, otherwise the SSL handshake would also work. (from experience VMs in some desktop hypervisors for ex. VirtualBox run slower compared to on baremetal or more modern/well-supported hypervisors such as KVM, VMware).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] ThyLAW commented on issue #4659: Agent : SSL Handshake failed while connecting to host on port: 8250
Posted by GitBox <gi...@apache.org>.
ThyLAW commented on issue #4659:
URL: https://github.com/apache/cloudstack/issues/4659#issuecomment-786687067
> @ThyLAW @abdelouahabb is sudo and haveged installed on your system? Also note CloudStack required full virtualisation on KVM (PV may not work or at least not tested, usually you can use VMware workstation/fusion than VirtualBox on Windows/Mac and KVM on Linux, i.e. on your laptop/desktop). For Ubuntu-based quick install guide, you can refer to https://rohityadav.cloud/blog/cloudstack-kvm/ (change the version/repo accordingly) - see if there's any step you missed?
Hello, virtualization works on Hyper-V by running a powershell script to enable it. I have tested it by creating a new virtual machine using virt-manager. I am going to go through your installation guide to see if that works. If not, I am going to try VMware though I don't think it is the hypervisor that is causing issues.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] abdelouahabb commented on issue #4659: Agent : SSL Handshake failed while connecting to host on port: 8250
Posted by GitBox <gi...@apache.org>.
abdelouahabb commented on issue #4659:
URL: https://github.com/apache/cloudstack/issues/4659#issuecomment-777132557
> @abdelouahabb
> can you try the following step on node ?
> (1) rm /etc/cloudstack/agent/cloud.*
> (2) systemctl restart cloudstack-agent
> if it does not work, remove host from cloudstack, and re-add it.
It dident work,
I even deleted the certificates and installed new ones :+1:
```
sudo dpkg --purge --force-depends ca-certificates-java
sudo apt-get install ca-certificates-java
```
And here is my network configuration :
```
auto lo
iface lo inet loopback
auto enp0s8
iface enp0s8 inet dhcp
auto enp0s3
iface enp0s3 inet manual
auto cloudbr0
iface cloudbr0 inet static
address 192.168.56.85
netmask 255.255.255.0
gateway 192.168.56.1 metric 100
dns-nameservers 8.8.8.8
bridge_ports enp0s3
bridge_fd 5
bridge_stp off
bridge_maxwait 1
auto cloudbr1
iface cloudbr1 inet manual
bridge_ports none
bridge_fd 5
bridge_stp off
bridge_maxwait 1
```
Is it a bug with Ubuntu 20.04 ?
```
cat /etc/apt/sources.list.d/cloudstack.list
deb http://cloudstack.apt-get.eu/ubuntu focal 4.15
```
```
>> cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.2 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.2 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] abdelouahabb commented on issue #4659: Agent : SSL Handshake failed while connecting to host on port: 8250
Posted by GitBox <gi...@apache.org>.
abdelouahabb commented on issue #4659:
URL: https://github.com/apache/cloudstack/issues/4659#issuecomment-781669388
> > ontent of the configuration file of libvirtd:
> > ```
> > log_level=1
> > listen_tls=1
> > mdns_adv=0
> > listen_tcp=0
> > tcp_port="16509"
> > unix_sock_group = "libvirt"
> > unix_sock_ro_perms = "0777"
> > unix_sock_rw_perms = "0770"
> > #tls_port="16514"
> > auth_tcp="none"
> > auth_tls="none"
> > #key_file="/etc/pki/libvirt/private/serverkey.pem"
> > #cert_file="/etc/pki/libvirt/servercert.pem"
> > #ca_file="/etc/pki/CA/cacert.pem"
> > ```
>
> @abdelouahabb
> the libvirtd.conf above is incorrect.
>
> here is the content of libvirtd.conf on a server in my testing env.
>
> ```
> # cat /etc/libvirt/libvirtd.conf |egrep -v "^$|^#"
> unix_sock_group = "libvirtd"
> unix_sock_ro_perms = "0777"
> unix_sock_rw_perms = "0770"
> auth_unix_ro = "none"
> auth_unix_rw = "none"
> listen_tcp=0
> listen_tls=1
> key_file="/etc/pki/libvirt/private/serverkey.pem"
> cert_file="/etc/pki/libvirt/servercert.pem"
> ca_file="/etc/pki/CA/cacert.pem"
> tcp_port="16509"
> tls_port="16514"
> auth_tcp="none"
> auth_tls="none"
> ```
This is the same ? the group `libvirtd` dont exist in my machine :
```
abdelouahab@akham:~$
>> cat /etc/group | grep libvir
libvirt:x:121:abdelouahab
libvirt-qemu:x:64055:libvirt-qemu
libvirt-dnsmasq:x:122:
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] weizhouapache commented on issue #4659: Agent : SSL Handshake failed while connecting to host on port: 8250
Posted by GitBox <gi...@apache.org>.
weizhouapache commented on issue #4659:
URL: https://github.com/apache/cloudstack/issues/4659#issuecomment-781685921
> > > > > #key_file="/etc/pki/libvirt/private/serverkey.pem"
> > > > > #cert_file="/etc/pki/libvirt/servercert.pem"
> > > > > #ca_file="/etc/pki/CA/cacert.pem"
> > > >
> > > >
> > > > @abdelouahabb you use centos ? you need to use 'libvirt' instead of 'libvirtd.
> > > > The other lines should be same. the following lines should be uncommented.
> > > > ```
> > > > #tls_port="16514"
> > > > auth_tcp="none"
> > > > auth_tls="none"
> > > > #key_file="/etc/pki/libvirt/private/serverkey.pem"
> > > > #cert_file="/etc/pki/libvirt/servercert.pem"
> > > > #ca_file="/etc/pki/CA/cacert.pem"
> > > > ```
> > >
> > >
> > > I am on Ubuntu, I [disabled the security](https://libvirt.org/daemons.html) for the debug reason and getting the same issue
> >
> >
> > @abdelouahabb which ubuntu version ? 20.04 ?
>
> Yes, I think there is an issue with this release ?
>
> ```
> abdelouahab@akham:~$
> >> uname -a
> Linux akham.alien.local 5.4.0-65-generic #73-Ubuntu SMP Mon Jan 18 17:25:17 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
> abdelouahab@akham:~$
> >> cat /etc/os-release
> NAME="Ubuntu"
> VERSION="20.04.2 LTS (Focal Fossa)"
> ID=ubuntu
> ID_LIKE=debian
> PRETTY_NAME="Ubuntu 20.04.2 LTS"
> VERSION_ID="20.04"
> HOME_URL="https://www.ubuntu.com/"
> SUPPORT_URL="https://help.ubuntu.com/"
> BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
> PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
> VERSION_CODENAME=focal
> UBUNTU_CODENAME=focal
> abdelouahab@akham:~$
> >> cat /etc/apt/sources.list.d/cloudstack.list
> deb http://download.cloudstack.org/ubuntu focal 4.15
> ```
@abdelouahabb no.
I have a testing environment with ubuntu 20.04 as hypervisors. they work fine.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] abdelouahabb commented on issue #4659: Agent : SSL Handshake failed while connecting to host on port: 8250
Posted by GitBox <gi...@apache.org>.
abdelouahabb commented on issue #4659:
URL: https://github.com/apache/cloudstack/issues/4659#issuecomment-779460083
Update : after disabling the security check `ca.plugin.root.auth.strictness false` in global settings, it seems there is a bug with `libvirt`
Here is the new updated issue from the agent :
```
2021-02-15 22:06:35,371 INFO [utils.nio.Link] (main:null) (logid:) Conf file found: /etc/cloudstack/agent/agent.properties
2021-02-15 22:06:35,420 WARN [utils.nio.Link] (main:null) (logid:) Failed to load keystore, using trust all manager
2021-02-15 22:06:35,857 INFO [utils.nio.NioClient] (main:null) (logid:) SSL: Handshake done
2021-02-15 22:06:35,858 INFO [utils.nio.NioClient] (main:null) (logid:) Connected to 192.168.56.85:8250
2021-02-15 22:06:35,867 INFO [utils.linux.KVMHostInfo] (Agent-Handler-1:null) (logid:) Could not read cpuinfo_max_freq, falling back on libvirt
2021-02-15 22:06:35,927 INFO [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:) Attempting to create storage pool 5b0c937c-471e-4ea3-92cb-6ba6cdcc6c30 (Filesystem) in libvirt
2021-02-15 22:06:35,932 ERROR [kvm.resource.LibvirtConnection] (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken: invalid connection pointer in virConnectGetVersion
2021-02-15 22:06:35,936 INFO [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:) Found existing defined storage pool 5b0c937c-471e-4ea3-92cb-6ba6cdcc6c30, using it.
2021-02-15 22:06:35,937 INFO [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:) Trying to fetch storage pool 5b0c937c-471e-4ea3-92cb-6ba6cdcc6c30 from libvirt
2021-02-15 22:06:36,015 INFO [cloud.serializer.GsonHelper] (Agent-Handler-1:null) (logid:) Default Builder inited.
2021-02-15 22:06:36,071 INFO [cloud.agent.Agent] (Agent-Handler-2:null) (logid:) Proccess agent startup answer, agent id = 0
2021-02-15 22:06:36,071 INFO [cloud.agent.Agent] (Agent-Handler-2:null) (logid:) Set agent id 0
2021-02-15 22:06:36,086 INFO [cloud.agent.Agent] (AgentShutdownThread:null) (logid:) Stopping the agent: Reason = sig.kill
2021-02-15 22:06:36,087 INFO [cloud.agent.Agent] (Agent-Handler-2:null) (logid:) Startup Response Received: agent id = 0
2021-02-15 22:06:47,969 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started
2021-02-15 22:06:47,973 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.15.0.0
2021-02-15 22:06:47,976 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
2021-02-15 22:06:47,994 INFO [cloud.agent.AgentShell] (main:null) (logid:) Defaulting to using properties file for storage
2021-02-15 22:06:47,996 INFO [cloud.agent.AgentShell] (main:null) (logid:) Defaulting to the constant time backoff algorithm
2021-02-15 22:06:47,999 INFO [cloud.utils.LogUtils] (main:null) (logid:) log4j configuration found at /etc/cloudstack/agent/log4j-cloud.xml
2021-02-15 22:06:48,015 INFO [cloud.agent.AgentShell] (main:null) (logid:) Using default Java settings for IPv6 preference for agent connection
2021-02-15 22:06:48,133 INFO [cloud.agent.Agent] (main:null) (logid:) id is 0
2021-02-15 22:06:48,140 WARN [cloud.resource.ServerResourceBase] (main:null) (logid:) Nics are not specified in properties file/db, will try to autodiscover
2021-02-15 22:06:48,145 INFO [cloud.resource.ServerResourceBase] (main:null) (logid:) Designating private to be nic cloudbr0
2021-02-15 22:06:48,148 ERROR [kvm.resource.LibvirtComputingResource] (main:null) (logid:) uefi properties file not found due to: Unable to find file uefi.properties.
2021-02-15 22:06:48,195 INFO [kvm.resource.LibvirtConnection] (main:null) (logid:) No existing libvirtd connection found. Opening a new one
2021-02-15 22:06:48,593 INFO [kvm.resource.LibvirtComputingResource] (main:null) (logid:) No libvirt.vif.driver specified. Defaults to BridgeVifDriver.
2021-02-15 22:06:48,813 INFO [kvm.resource.LibvirtComputingResource] (main:null) (logid:) iscsi session clean up is disabled
2021-02-15 22:06:48,827 INFO [cloud.agent.Agent] (main:null) (logid:) Agent [id = 0 : type = LibvirtComputingResource : zone = default : pod = default : workers = 5 : host = 192.168.56.85 : port = 8250
2021-02-15 22:06:48,839 INFO [utils.nio.NioClient] (main:null) (logid:) Connecting to 192.168.56.85:8250
2021-02-15 22:06:48,846 INFO [utils.nio.Link] (main:null) (logid:) Conf file found: /etc/cloudstack/agent/agent.properties
2021-02-15 22:06:48,875 WARN [utils.nio.Link] (main:null) (logid:) Failed to load keystore, using trust all manager
2021-02-15 22:06:49,466 INFO [utils.nio.NioClient] (main:null) (logid:) SSL: Handshake done
2021-02-15 22:06:49,469 INFO [utils.nio.NioClient] (main:null) (logid:) Connected to 192.168.56.85:8250
2021-02-15 22:06:49,481 INFO [utils.linux.KVMHostInfo] (Agent-Handler-1:null) (logid:) Could not read cpuinfo_max_freq, falling back on libvirt
2021-02-15 22:06:49,546 INFO [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:) Attempting to create storage pool 5b0c937c-471e-4ea3-92cb-6ba6cdcc6c30 (Filesystem) in libvirt
2021-02-15 22:06:49,551 ERROR [kvm.resource.LibvirtConnection] (Agent-Handler-1:null) (logid:) Connection with libvirtd is broken: invalid connection pointer in virConnectGetVersion
2021-02-15 22:06:49,554 INFO [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:) Found existing defined storage pool 5b0c937c-471e-4ea3-92cb-6ba6cdcc6c30, using it.
2021-02-15 22:06:49,555 INFO [kvm.storage.LibvirtStorageAdaptor] (Agent-Handler-1:null) (logid:) Trying to fetch storage pool 5b0c937c-471e-4ea3-92cb-6ba6cdcc6c30 from libvirt
2021-02-15 22:06:49,619 INFO [cloud.serializer.GsonHelper] (Agent-Handler-1:null) (logid:) Default Builder inited.
2021-02-15 22:06:49,659 INFO [cloud.agent.Agent] (Agent-Handler-2:null) (logid:) Proccess agent startup answer, agent id = 0
2021-02-15 22:06:49,661 INFO [cloud.agent.Agent] (Agent-Handler-2:null) (logid:) Set agent id 0
2021-02-15 22:06:49,676 INFO [cloud.agent.Agent] (Agent-Handler-2:null) (logid:) Startup Response Received: agent id = 0
2021-02-15 22:06:49,679 INFO [cloud.agent.Agent] (AgentShutdownThread:null) (logid:) Stopping the agent: Reason = sig.kill
```
```
>> virsh pool-list
Name State Autostart
------------------------------------------------------------
5b0c937c-471e-4ea3-92cb-6ba6cdcc6c30 active no
tmp active yes
```
Content of the configuration file of libvirtd:
```
log_level=1
listen_tls=1
mdns_adv=0
listen_tcp=0
tcp_port="16509"
unix_sock_group = "libvirt"
unix_sock_ro_perms = "0777"
unix_sock_rw_perms = "0770"
#tls_port="16514"
auth_tcp="none"
auth_tls="none"
#key_file="/etc/pki/libvirt/private/serverkey.pem"
#cert_file="/etc/pki/libvirt/servercert.pem"
#ca_file="/etc/pki/CA/cacert.pem"
```
```
>> sudo systemctl status libvirtd
● libvirtd.service - Virtualization daemon
Loaded: loaded (/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2021-02-15 22:29:19 CET; 9min ago
TriggeredBy: ● libvirtd-ro.socket
● libvirtd-admin.socket
Docs: man:libvirtd(8)
https://libvirt.org
Main PID: 33440 (libvirtd)
Tasks: 17 (limit: 32768)
Memory: 13.5M
CGroup: /system.slice/libvirtd.service
└─33440 /usr/sbin/libvirtd
Feb 15 22:38:26 akham.alien.local libvirtd[33440]: EVENT_POLL_DISPATCH_HANDLE: watch=10 events=1
Feb 15 22:38:26 akham.alien.local libvirtd[33440]: EVENT_POLL_RUN: nhandles=9 timeout=-1
Feb 15 22:38:26 akham.alien.local libvirtd[33440]: EVENT_POLL_DISPATCH_HANDLE: watch=10 events=1
Feb 15 22:38:26 akham.alien.local libvirtd[33440]: EVENT_POLL_RUN: nhandles=9 timeout=-1
Feb 15 22:38:26 akham.alien.local libvirtd[33440]: EVENT_POLL_DISPATCH_HANDLE: watch=10 events=1
Feb 15 22:38:26 akham.alien.local libvirtd[33440]: EVENT_POLL_RUN: nhandles=9 timeout=-1
Feb 15 22:38:26 akham.alien.local libvirtd[33440]: EVENT_POLL_DISPATCH_HANDLE: watch=10 events=1
Feb 15 22:38:26 akham.alien.local libvirtd[33440]: EVENT_POLL_RUN: nhandles=9 timeout=-1
Feb 15 22:38:26 akham.alien.local libvirtd[33440]: EVENT_POLL_DISPATCH_HANDLE: watch=10 events=1
Feb 15 22:38:26 akham.alien.local libvirtd[33440]: EVENT_POLL_RUN: nhandles=9 timeout=-1
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] abdelouahabb commented on issue #4659: Agent : SSL Handshake failed while connecting to host on port: 8250
Posted by GitBox <gi...@apache.org>.
abdelouahabb commented on issue #4659:
URL: https://github.com/apache/cloudstack/issues/4659#issuecomment-781682950
> > > > #key_file="/etc/pki/libvirt/private/serverkey.pem"
> > > > #cert_file="/etc/pki/libvirt/servercert.pem"
> > > > #ca_file="/etc/pki/CA/cacert.pem"
> > >
> > >
> > > @abdelouahabb you use centos ? you need to use 'libvirt' instead of 'libvirtd.
> > > The other lines should be same. the following lines should be uncommented.
> > > ```
> > > #tls_port="16514"
> > > auth_tcp="none"
> > > auth_tls="none"
> > > #key_file="/etc/pki/libvirt/private/serverkey.pem"
> > > #cert_file="/etc/pki/libvirt/servercert.pem"
> > > #ca_file="/etc/pki/CA/cacert.pem"
> > > ```
> >
> >
> > I am on Ubuntu, I [disabled the security](https://libvirt.org/daemons.html) for the debug reason and getting the same issue
>
> @abdelouahabb which ubuntu version ? 20.04 ?
Yes, I think there is an issue with this release ?
```
abdelouahab@akham:~$
>> uname -a
Linux akham.alien.local 5.4.0-65-generic #73-Ubuntu SMP Mon Jan 18 17:25:17 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
abdelouahab@akham:~$
>> cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.2 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.2 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
abdelouahab@akham:~$
>> cat /etc/apt/sources.list.d/cloudstack.list
deb http://download.cloudstack.org/ubuntu focal 4.15
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] abdelouahabb edited a comment on issue #4659: Agent : SSL Handshake failed while connecting to host on port: 8250
Posted by GitBox <gi...@apache.org>.
abdelouahabb edited a comment on issue #4659:
URL: https://github.com/apache/cloudstack/issues/4659#issuecomment-778914316
I tried doing it manually (as i think understanding the security concept in CloudStack) :
`sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/management/server.properties ./cloud.jks dummyPassword007 365 ./cloud.csr`
`sudo /usr/share/cloudstack-common/scripts/util/keystore-cert-import /etc/cloudstack/management/server.properties ./cloud.jks mode:agent ./a "/C=DZ /ST=akham/L=akham/O=akham/OU=akham/CN=akham.alien.local" ./b "/C=DZ/ST=akham/L=akham/O=akham/OU=akham/CN=akham.alien.local" ./c
CACERT_FILE: ./b CERT_FILE: ./a PRIVKEY_FILE: ./c`
But still the same result,
After investigating in the content of the script, it seems that are some folders/files dont exist in the Ubuntu server installation :
```
>> file /etc/libvirt/libvirtd.conf
/etc/libvirt/libvirtd.conf: ASCII text
>> file /var/cache/cloud/cmdline
/var/cache/cloud/cmdline: cannot open `/var/cache/cloud/cmdline' (No such file or directory)
>> pwd
/var/cache/cloudstack
>> file /var/cache/cloud/
/var/cache/cloud/: cannot open `/var/cache/cloud/' (No such file or directory)
>> file /etc/pki/libvirt/private/serverkey.pem
/etc/pki/libvirt/private/serverkey.pem: broken symbolic link to /etc/cloudstack/agent/cloud.key
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] abdelouahabb edited a comment on issue #4659: Agent : SSL Handshake failed while connecting to host on port: 8250
Posted by GitBox <gi...@apache.org>.
abdelouahabb edited a comment on issue #4659:
URL: https://github.com/apache/cloudstack/issues/4659#issuecomment-781669388
> > ontent of the configuration file of libvirtd:
> > ```
> > log_level=1
> > listen_tls=1
> > mdns_adv=0
> > listen_tcp=0
> > tcp_port="16509"
> > unix_sock_group = "libvirt"
> > unix_sock_ro_perms = "0777"
> > unix_sock_rw_perms = "0770"
> > #tls_port="16514"
> > auth_tcp="none"
> > auth_tls="none"
> > #key_file="/etc/pki/libvirt/private/serverkey.pem"
> > #cert_file="/etc/pki/libvirt/servercert.pem"
> > #ca_file="/etc/pki/CA/cacert.pem"
> > ```
>
> @abdelouahabb
> the libvirtd.conf above is incorrect.
>
> here is the content of libvirtd.conf on a server in my testing env.
>
> ```
> # cat /etc/libvirt/libvirtd.conf |egrep -v "^$|^#"
> unix_sock_group = "libvirtd"
> unix_sock_ro_perms = "0777"
> unix_sock_rw_perms = "0770"
> auth_unix_ro = "none"
> auth_unix_rw = "none"
> listen_tcp=0
> listen_tls=1
> key_file="/etc/pki/libvirt/private/serverkey.pem"
> cert_file="/etc/pki/libvirt/servercert.pem"
> ca_file="/etc/pki/CA/cacert.pem"
> tcp_port="16509"
> tls_port="16514"
> auth_tcp="none"
> auth_tls="none"
> ```
This is the same ? the group `libvirtd` dont exist in my machine, the group is called `libvirt` :
```
abdelouahab@akham:~$
>> cat /etc/group | grep libvir
libvirt:x:121:abdelouahab
libvirt-qemu:x:64055:libvirt-qemu
libvirt-dnsmasq:x:122:
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] DaanHoogland commented on issue #4659: Agent : SSL Handshake failed while connecting to host on port: 8250
Posted by GitBox <gi...@apache.org>.
DaanHoogland commented on issue #4659:
URL: https://github.com/apache/cloudstack/issues/4659#issuecomment-776525902
@abdelouahabb this is working for loads of people all over the world. I suggest you ask for help on users@cloudstack.apache.org
By the looks of your log, the problem is not with ssh but with the certificates or the jdk encryption policies available to your host or management server.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] abdelouahabb commented on issue #4659: Agent : SSL Handshake failed while connecting to host on port: 8250
Posted by GitBox <gi...@apache.org>.
abdelouahabb commented on issue #4659:
URL: https://github.com/apache/cloudstack/issues/4659#issuecomment-781650371
> > > I'm getting nearly the same issues as you. I was also experiencing the SSL problem, and disabled it in global configs. I then got the "unable to find private nics" and manually configured that in my agent.properties. I now am just getting
> > > "error: conection with libvirtd is broken : invalid connection pointer in virConnectGetVersion" as well as the UEFI.properties error.
> > > And on libvirtd status I am getting :
> > > "47271 error: virNetSocketReadWire:1806: end of file while reading data : Input/output error"
> > > Please let me know if you have found a solution.
> >
> >
> > @ThyLAW I am getting the same issue too with libvirt, what is your setup ? virtualizing it on bare metal or on top of another hypervisor ?
> > @DaanHoogland any update ?
>
> On top of a hypervisor .Hyper-v gen 2 This is how I set it up.
> [New.Apache.Documentatino.docx](https://github.com/apache/cloudstack/files/6005970/New.Apache.Documentatino.docx)
Everything seems right, if you `virsh pool-list` you should see the `id` you put for the disk
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] weizhouapache commented on issue #4659: Agent : SSL Handshake failed while connecting to host on port: 8250
Posted by GitBox <gi...@apache.org>.
weizhouapache commented on issue #4659:
URL: https://github.com/apache/cloudstack/issues/4659#issuecomment-777281477
@abdelouahabb is there error log on management server ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] weizhouapache commented on issue #4659: Agent : SSL Handshake failed while connecting to host on port: 8250
Posted by GitBox <gi...@apache.org>.
weizhouapache commented on issue #4659:
URL: https://github.com/apache/cloudstack/issues/4659#issuecomment-781664172
> ontent of the configuration file of libvirtd:
>
> ```
> log_level=1
> listen_tls=1
> mdns_adv=0
> listen_tcp=0
> tcp_port="16509"
> unix_sock_group = "libvirt"
> unix_sock_ro_perms = "0777"
> unix_sock_rw_perms = "0770"
> #tls_port="16514"
> auth_tcp="none"
> auth_tls="none"
> #key_file="/etc/pki/libvirt/private/serverkey.pem"
> #cert_file="/etc/pki/libvirt/servercert.pem"
> #ca_file="/etc/pki/CA/cacert.pem"
> ```
@abdelouahabb
the libvirtd.conf above is incorrect.
here is the content of libvirtd.conf on a server in my testing env.
```
# cat /etc/libvirt/libvirtd.conf |egrep -v "^$|^#"
unix_sock_group = "libvirtd"
unix_sock_ro_perms = "0777"
unix_sock_rw_perms = "0770"
auth_unix_ro = "none"
auth_unix_rw = "none"
listen_tcp=0
listen_tls=1
key_file="/etc/pki/libvirt/private/serverkey.pem"
cert_file="/etc/pki/libvirt/servercert.pem"
ca_file="/etc/pki/CA/cacert.pem"
tcp_port="16509"
tls_port="16514"
auth_tcp="none"
auth_tls="none"
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] ThyLAW commented on issue #4659: Agent : SSL Handshake failed while connecting to host on port: 8250
Posted by GitBox <gi...@apache.org>.
ThyLAW commented on issue #4659:
URL: https://github.com/apache/cloudstack/issues/4659#issuecomment-781571346
I'm getting nearly the same issues as you. I was also experiencing the SSL problem, and disabled it in global configs. I then got the "unable to find private nics" and manually configured that in my agent.properties. I now am just getting
"error: conection with libvirtd is broken : invalid connection pointer in virConnectGetVersion" as well as the UEFI.properties error.
And on libvirtd status I am getting :
"47271 error: virNetSocketReadWire:1806: end of file while reading data : Input/output error"
Please let me know if you have found a solution.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] DaanHoogland commented on issue #4659: Agent : SSL Handshake failed while connecting to host on port: 8250
Posted by GitBox <gi...@apache.org>.
DaanHoogland commented on issue #4659:
URL: https://github.com/apache/cloudstack/issues/4659#issuecomment-778090543
ok @abdelouahabb , I think you have a problem on your host (using the host certificate) I added to milestone 4.15.1, but I think it is a problem with the host installation. Please add some information from the agent if you can.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] ThyLAW commented on issue #4659: Agent : SSL Handshake failed while connecting to host on port: 8250
Posted by GitBox <gi...@apache.org>.
ThyLAW commented on issue #4659:
URL: https://github.com/apache/cloudstack/issues/4659#issuecomment-782793173
Unfortunately I still have the issues even when disabling ca.plugin.root.auth.strictness on Centos7. Any other advice?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] weizhouapache commented on issue #4659: Agent : SSL Handshake failed while connecting to host on port: 8250
Posted by GitBox <gi...@apache.org>.
weizhouapache commented on issue #4659:
URL: https://github.com/apache/cloudstack/issues/4659#issuecomment-776995025
@abdelouahabb
can you try the following step on node ?
(1) rm /etc/cloudstack/agent/cloud.*
(2) systemctl restart cloudstack-agent
if it does not work, remove host from cloudstack, and re-add it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] abdelouahabb edited a comment on issue #4659: Agent : SSL Handshake failed while connecting to host on port: 8250
Posted by GitBox <gi...@apache.org>.
abdelouahabb edited a comment on issue #4659:
URL: https://github.com/apache/cloudstack/issues/4659#issuecomment-778553357
> ok @abdelouahabb , I think you have a problem on your host (using the host certificate) I added to milestone 4.15.1, but I think it is a problem with the host installation. Please add some information from the agent if you can.
Thank you,
Here is the log, this keeps looping over and over
Maybe the process of configuring the certificate will be more helpful if it will be added to the first installation and bootstraping configuration ?
```
sudo tail -f /var/log/cloudstack/agent/agent.log
2021-02-13 03:10:24,613 INFO [cloud.agent.Agent] (main:null) (logid:) Connecting to host:192.168.56.85
2021-02-13 03:10:24,614 INFO [utils.nio.NioClient] (main:null) (logid:) Connecting to 192.168.56.85:8250
2021-02-13 03:10:24,615 INFO [utils.nio.Link] (main:null) (logid:) Conf file found: /etc/cloudstack/agent/agent.properties
2021-02-13 03:10:24,617 WARN [utils.nio.Link] (main:null) (logid:) Failed to load keystore, using trust all manager
2021-02-13 03:10:24,710 ERROR [utils.nio.Link] (main:null) (logid:) SSL error caught during unwrap data: Received fatal alert: bad_certificate, for local address=/192.168.56.85:57384, remote address=/192.168.56.85:8250. The client may have invalid ca-certificates.
2021-02-13 03:10:24,711 ERROR [utils.nio.NioClient] (main:null) (logid:) SSL Handshake failed while connecting to host: 192.168.56.85 port: 8250
2021-02-13 03:10:24,711 ERROR [utils.nio.NioConnection] (main:null) (logid:) Unable to initialize the threads.
java.io.IOException: SSL Handshake failed while connecting to host: 192.168.56.85 port: 8250
at com.cloud.utils.nio.NioClient.init(NioClient.java:67)
at com.cloud.utils.nio.NioConnection.start(NioConnection.java:95)
at com.cloud.agent.Agent.start(Agent.java:294)
at com.cloud.agent.AgentShell.launchNewAgent(AgentShell.java:455)
at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:422)
at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:406)
at com.cloud.agent.AgentShell.start(AgentShell.java:512)
at com.cloud.agent.AgentShell.main(AgentShell.java:547)
2021-02-13 03:10:24,711 INFO [utils.nio.NioClient] (main:null) (logid:) NioClient connection closed
2021-02-13 03:10:24,711 INFO [cloud.agent.Agent] (main:null) (logid:) Attempted to connect to the server, but received an unexpected exception, trying again...
com.cloud.utils.exception.NioConnectionException: SSL Handshake failed while connecting to host: 192.168.56.85 port: 8250
at com.cloud.utils.nio.NioConnection.start(NioConnection.java:101)
at com.cloud.agent.Agent.start(Agent.java:294)
at com.cloud.agent.AgentShell.launchNewAgent(AgentShell.java:455)
at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:422)
at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:406)
at com.cloud.agent.AgentShell.start(AgentShell.java:512)
at com.cloud.agent.AgentShell.main(AgentShell.java:547)
```
And I have even replaced the openjdk with the oracle java
```
>> sudo update-alternatives --config java
There are 2 choices for the alternative java (providing /usr/bin/java).
Selection Path Priority Status
------------------------------------------------------------
0 /usr/lib/jvm/java-11-openjdk-amd64/bin/java 1111 auto mode
1 /usr/lib/jvm/java-11-openjdk-amd64/bin/java 1111 manual mode
* 2 /usr/lib/jvm/java-15-oracle/bin/java 1091 manual mode
Press <enter> to keep the current choice[*], or type selection number:
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org