You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Jason Kinsella <ja...@cloudpeople.com.au> on 2017/05/23 12:11:59 UTC
SSVM NIO SSL Handshake error
Hi,
We recently upgraded from 4.5.0 to 4.9.2.0 and encountered a problem with the SSVM and Console Proxy. They cannot connect to the management server. The SSVM cloud.log repeats this error every couple of seconds.
2017-05-23 11:58:22,461 INFO [utils.nio.NioClient] (main:null) Connecting to 192.168.12.1:8250
2017-05-23 11:58:22,465 WARN [utils.nio.Link] (main:null) This SSL engine was forced to close inbound due to end of stream.
2017-05-23 11:58:22,465 ERROR [utils.nio.Link] (main:null) Failed to send server's CLOSE message due to socket channel's failure.
2017-05-23 11:58:22,466 ERROR [utils.nio.NioClient] (main:null) SSL Handshake failed while connecting to host: 192.168.12.1 port: 8250
2017-05-23 11:58:22,466 ERROR [utils.nio.NioConnection] (main:null) Unable to initialize the threads.
java.io.IOException: SSL Handshake failed while connecting to host: 192.168.12.1 port: 8250
at com.cloud.utils.nio.NioClient.init(NioClient.java:67)
at com.cloud.utils.nio.NioConnection.start(NioConnection.java:88)
at com.cloud.agent.Agent.start(Agent.java:237)
at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:399)
at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:367)
at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:351)
at com.cloud.agent.AgentShell.start(AgentShell.java:456)
at com.cloud.agent.AgentShell.main(AgentShell.java:491)
2017-05-23 11:58:22,468 INFO [utils.exception.CSExceptionErrorCode] (main:null) Could not find exception: com.cloud.utils.exception.NioConnectionException in error code list for exceptions
2017-05-23 11:58:22,468 WARN [cloud.agent.Agent] (main:null) NIO Connection Exception com.cloud.utils.exception.NioConnectionException: SSL Handshake failed while connecting to host: 192.168.12.1 port: 8250
The setup is very simple. Single management server and ports are open.
Things checked / tried:
· Destroyed SSVM multiple times – still same problem.
· SSH to SSVM from MS using ssh -i /var/cloudstack/management/.ssh/id_rsa -p 3922 root@IPADDRESS – PASS
· SSVM telnet on 8250 to MS – PASS
I’ve also tested a restore of the DB into our working development 4.9.2.0 server. It also exhibits the handshake errors, so most likely DB related.
I’ve used up all my skills. Please help
Regards,
Jason
Re: SSVM NIO SSL Handshake error
Posted by Yiping Zhang <yz...@marketo.com>.
Hi, Jason:
By any chance, did your “full update” happen to also update your java version to IBM java?
Yiping
From: Jason Kinsella <ja...@cloudpeople.com.au>
Reply-To: "users@cloudstack.apache.org" <us...@cloudstack.apache.org>
Date: Tuesday, May 23, 2017 at 7:19 AM
To: "users@cloudstack.apache.org" <us...@cloudstack.apache.org>
Subject: Re: SSVM NIO SSL Handshake error
Hi Vivek,
Version is centos-release-6-5.el6.centos.11.2.x86_64
Nss* packages before update are
nss-softokn-freebl-3.14.3-23.3.el6_8.x86_64
nss-tools-3.21.3-2.el6_8.x86_64
nss-util-3.21.3-1.el6_8.x86_64
nss-softokn-3.14.3-23.3.el6_8.x86_64
nss-3.21.3-2.el6_8.x86_64
nss-sysinit-3.21.3-2.el6_8.x86_64
nss-softokn-freebl-3.14.3-23.3.el6_8.i686
I’ve just updated all nss* packages, restart of MS, destroy & replace SSVM did not help.
I will try a full yum update tomorrow, just in case we missed something last time we tried.
Jason
From: Vivek Kumar <vi...@indiqus.com>
Reply-To: "users@cloudstack.apache.org" <us...@cloudstack.apache.org>
Date: Tuesday, 23 May 2017 at 11:55 pm
To: "users@cloudstack.apache.org" <us...@cloudstack.apache.org>
Subject: Re: SSVM NIO SSL Handshake error
Hello Jason,
I also faced this issue earlier with my centos 6, can you confirm the exact version of centos ?, and if possible please update all packages related to nss* and then restart the management services.
Vivek Kumar
Virtualization and Cloud Consultant
[cid:image001.jpg@01D2D423.5F7FA370]
IndiQus Technologies Pvt Ltd
A-98, LGF, C.R.Park, New Delhi - 110019
24x7 +91 11 4055 1409 | M +91 7503460090
www.indiqus.com<http://www.indiqus.com>
On 23-May-2017, at 6:35 PM, Jason Kinsella <ja...@cloudpeople.com.au>> wrote:
Hi Erik,
Yes - the box is a CentOS 6 box. It hadn’t been updated in a while, but we did a full update without fixing it. Also, the dev server is exactly the same CentOS 6 version and happily running 4.9.2.0
Thanks,
Jason
On 23/5/17, 10:56 pm, "Erik Weber" <te...@gmail.com>> wrote:
What's the OS? Think I saw something like this on some old CentOS 6 machines
--
Erik
On Tue, May 23, 2017 at 2:11 PM, Jason Kinsella
<ja...@cloudpeople.com.au>> wrote:
Hi,
We recently upgraded from 4.5.0 to 4.9.2.0 and encountered a problem with the SSVM and Console Proxy. They cannot connect to the management server. The SSVM cloud.log repeats this error every couple of seconds.
2017-05-23 11:58:22,461 INFO [utils.nio.NioClient] (main:null) Connecting to 192.168.12.1:8250
2017-05-23 11:58:22,465 WARN [utils.nio.Link] (main:null) This SSL engine was forced to close inbound due to end of stream.
2017-05-23 11:58:22,465 ERROR [utils.nio.Link] (main:null) Failed to send server's CLOSE message due to socket channel's failure.
2017-05-23 11:58:22,466 ERROR [utils.nio.NioClient] (main:null) SSL Handshake failed while connecting to host: 192.168.12.1 port: 8250
2017-05-23 11:58:22,466 ERROR [utils.nio.NioConnection] (main:null) Unable to initialize the threads.
java.io.IOException: SSL Handshake failed while connecting to host: 192.168.12.1 port: 8250
at com.cloud.utils.nio.NioClient.init(NioClient.java:67)
at com.cloud.utils.nio.NioConnection.start(NioConnection.java:88)
at com.cloud.agent.Agent.start(Agent.java:237)
at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:399)
at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:367)
at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:351)
at com.cloud.agent.AgentShell.start(AgentShell.java:456)
at com.cloud.agent.AgentShell.main(AgentShell.java:491)
2017-05-23 11:58:22,468 INFO [utils.exception.CSExceptionErrorCode] (main:null) Could not find exception: com.cloud.utils.exception.NioConnectionException in error code list for exceptions
2017-05-23 11:58:22,468 WARN [cloud.agent.Agent] (main:null) NIO Connection Exception com.cloud.utils.exception.NioConnectionException: SSL Handshake failed while connecting to host: 192.168.12.1 port: 8250
The setup is very simple. Single management server and ports are open.
Things checked / tried:
· Destroyed SSVM multiple times – still same problem.
· SSH to SSVM from MS using ssh -i /var/cloudstack/management/.ssh/id_rsa -p 3922 root@IPADDRESS – PASS
· SSVM telnet on 8250 to MS – PASS
I’ve also tested a restore of the DB into our working development 4.9.2.0 server. It also exhibits the handshake errors, so most likely DB related.
I’ve used up all my skills. Please help
Regards,
Jason
Re: SSVM NIO SSL Handshake error
Posted by Jason Kinsella <ja...@cloudpeople.com.au>.
Hi Vivek,
Version is centos-release-6-5.el6.centos.11.2.x86_64
Nss* packages before update are
nss-softokn-freebl-3.14.3-23.3.el6_8.x86_64
nss-tools-3.21.3-2.el6_8.x86_64
nss-util-3.21.3-1.el6_8.x86_64
nss-softokn-3.14.3-23.3.el6_8.x86_64
nss-3.21.3-2.el6_8.x86_64
nss-sysinit-3.21.3-2.el6_8.x86_64
nss-softokn-freebl-3.14.3-23.3.el6_8.i686
I’ve just updated all nss* packages, restart of MS, destroy & replace SSVM did not help.
I will try a full yum update tomorrow, just in case we missed something last time we tried.
Jason
From: Vivek Kumar <vi...@indiqus.com>
Reply-To: "users@cloudstack.apache.org" <us...@cloudstack.apache.org>
Date: Tuesday, 23 May 2017 at 11:55 pm
To: "users@cloudstack.apache.org" <us...@cloudstack.apache.org>
Subject: Re: SSVM NIO SSL Handshake error
Hello Jason,
I also faced this issue earlier with my centos 6, can you confirm the exact version of centos ?, and if possible please update all packages related to nss* and then restart the management services.
Vivek Kumar
Virtualization and Cloud Consultant
[cid:image001.jpg@01D2D423.5F7FA370]
IndiQus Technologies Pvt Ltd
A-98, LGF, C.R.Park, New Delhi - 110019
24x7 +91 11 4055 1409 | M +91 7503460090
www.indiqus.com<http://www.indiqus.com>
On 23-May-2017, at 6:35 PM, Jason Kinsella <ja...@cloudpeople.com.au>> wrote:
Hi Erik,
Yes - the box is a CentOS 6 box. It hadn’t been updated in a while, but we did a full update without fixing it. Also, the dev server is exactly the same CentOS 6 version and happily running 4.9.2.0
Thanks,
Jason
On 23/5/17, 10:56 pm, "Erik Weber" <te...@gmail.com>> wrote:
What's the OS? Think I saw something like this on some old CentOS 6 machines
--
Erik
On Tue, May 23, 2017 at 2:11 PM, Jason Kinsella
<ja...@cloudpeople.com.au>> wrote:
Hi,
We recently upgraded from 4.5.0 to 4.9.2.0 and encountered a problem with the SSVM and Console Proxy. They cannot connect to the management server. The SSVM cloud.log repeats this error every couple of seconds.
2017-05-23 11:58:22,461 INFO [utils.nio.NioClient] (main:null) Connecting to 192.168.12.1:8250
2017-05-23 11:58:22,465 WARN [utils.nio.Link] (main:null) This SSL engine was forced to close inbound due to end of stream.
2017-05-23 11:58:22,465 ERROR [utils.nio.Link] (main:null) Failed to send server's CLOSE message due to socket channel's failure.
2017-05-23 11:58:22,466 ERROR [utils.nio.NioClient] (main:null) SSL Handshake failed while connecting to host: 192.168.12.1 port: 8250
2017-05-23 11:58:22,466 ERROR [utils.nio.NioConnection] (main:null) Unable to initialize the threads.
java.io.IOException: SSL Handshake failed while connecting to host: 192.168.12.1 port: 8250
at com.cloud.utils.nio.NioClient.init(NioClient.java:67)
at com.cloud.utils.nio.NioConnection.start(NioConnection.java:88)
at com.cloud.agent.Agent.start(Agent.java:237)
at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:399)
at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:367)
at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:351)
at com.cloud.agent.AgentShell.start(AgentShell.java:456)
at com.cloud.agent.AgentShell.main(AgentShell.java:491)
2017-05-23 11:58:22,468 INFO [utils.exception.CSExceptionErrorCode] (main:null) Could not find exception: com.cloud.utils.exception.NioConnectionException in error code list for exceptions
2017-05-23 11:58:22,468 WARN [cloud.agent.Agent] (main:null) NIO Connection Exception com.cloud.utils.exception.NioConnectionException: SSL Handshake failed while connecting to host: 192.168.12.1 port: 8250
The setup is very simple. Single management server and ports are open.
Things checked / tried:
· Destroyed SSVM multiple times – still same problem.
· SSH to SSVM from MS using ssh -i /var/cloudstack/management/.ssh/id_rsa -p 3922 root@IPADDRESS – PASS
· SSVM telnet on 8250 to MS – PASS
I’ve also tested a restore of the DB into our working development 4.9.2.0 server. It also exhibits the handshake errors, so most likely DB related.
I’ve used up all my skills. Please help
Regards,
Jason
Re: SSVM NIO SSL Handshake error
Posted by Vivek Kumar <vi...@indiqus.com>.
Hello Jason,
I also faced this issue earlier with my centos 6, can you confirm the exact version of centos ?, and if possible please update all packages related to nss* and then restart the management services.
Vivek Kumar
Virtualization and Cloud Consultant
IndiQus Technologies Pvt Ltd
A-98, LGF, C.R.Park, New Delhi - 110019
24x7 +91 11 4055 1409 | M +91 7503460090
www.indiqus.com
> On 23-May-2017, at 6:35 PM, Jason Kinsella <ja...@cloudpeople.com.au> wrote:
>
> Hi Erik,
> Yes - the box is a CentOS 6 box. It hadn’t been updated in a while, but we did a full update without fixing it. Also, the dev server is exactly the same CentOS 6 version and happily running 4.9.2.0
> Thanks,
> Jason
>
>
>
> On 23/5/17, 10:56 pm, "Erik Weber" <te...@gmail.com> wrote:
>
> What's the OS? Think I saw something like this on some old CentOS 6 machines
>
> --
> Erik
>
> On Tue, May 23, 2017 at 2:11 PM, Jason Kinsella
> <ja...@cloudpeople.com.au> wrote:
>> Hi,
>> We recently upgraded from 4.5.0 to 4.9.2.0 and encountered a problem with the SSVM and Console Proxy. They cannot connect to the management server. The SSVM cloud.log repeats this error every couple of seconds.
>>
>> 2017-05-23 11:58:22,461 INFO [utils.nio.NioClient] (main:null) Connecting to 192.168.12.1:8250
>> 2017-05-23 11:58:22,465 WARN [utils.nio.Link] (main:null) This SSL engine was forced to close inbound due to end of stream.
>> 2017-05-23 11:58:22,465 ERROR [utils.nio.Link] (main:null) Failed to send server's CLOSE message due to socket channel's failure.
>> 2017-05-23 11:58:22,466 ERROR [utils.nio.NioClient] (main:null) SSL Handshake failed while connecting to host: 192.168.12.1 port: 8250
>> 2017-05-23 11:58:22,466 ERROR [utils.nio.NioConnection] (main:null) Unable to initialize the threads.
>> java.io.IOException: SSL Handshake failed while connecting to host: 192.168.12.1 port: 8250
>> at com.cloud.utils.nio.NioClient.init(NioClient.java:67)
>> at com.cloud.utils.nio.NioConnection.start(NioConnection.java:88)
>> at com.cloud.agent.Agent.start(Agent.java:237)
>> at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:399)
>> at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:367)
>> at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:351)
>> at com.cloud.agent.AgentShell.start(AgentShell.java:456)
>> at com.cloud.agent.AgentShell.main(AgentShell.java:491)
>> 2017-05-23 11:58:22,468 INFO [utils.exception.CSExceptionErrorCode] (main:null) Could not find exception: com.cloud.utils.exception.NioConnectionException in error code list for exceptions
>> 2017-05-23 11:58:22,468 WARN [cloud.agent.Agent] (main:null) NIO Connection Exception com.cloud.utils.exception.NioConnectionException: SSL Handshake failed while connecting to host: 192.168.12.1 port: 8250
>>
>> The setup is very simple. Single management server and ports are open.
>>
>> Things checked / tried:
>>
>> · Destroyed SSVM multiple times – still same problem.
>>
>> · SSH to SSVM from MS using ssh -i /var/cloudstack/management/.ssh/id_rsa -p 3922 root@IPADDRESS – PASS
>>
>> · SSVM telnet on 8250 to MS – PASS
>>
>> I’ve also tested a restore of the DB into our working development 4.9.2.0 server. It also exhibits the handshake errors, so most likely DB related.
>>
>> I’ve used up all my skills. Please help
>>
>> Regards,
>> Jason
>>
>
>
Re: SSVM NIO SSL Handshake error
Posted by Jason Kinsella <ja...@cloudpeople.com.au>.
Hi Erik,
Yes - the box is a CentOS 6 box. It hadn’t been updated in a while, but we did a full update without fixing it. Also, the dev server is exactly the same CentOS 6 version and happily running 4.9.2.0
Thanks,
Jason
On 23/5/17, 10:56 pm, "Erik Weber" <te...@gmail.com> wrote:
What's the OS? Think I saw something like this on some old CentOS 6 machines
--
Erik
On Tue, May 23, 2017 at 2:11 PM, Jason Kinsella
<ja...@cloudpeople.com.au> wrote:
> Hi,
> We recently upgraded from 4.5.0 to 4.9.2.0 and encountered a problem with the SSVM and Console Proxy. They cannot connect to the management server. The SSVM cloud.log repeats this error every couple of seconds.
>
> 2017-05-23 11:58:22,461 INFO [utils.nio.NioClient] (main:null) Connecting to 192.168.12.1:8250
> 2017-05-23 11:58:22,465 WARN [utils.nio.Link] (main:null) This SSL engine was forced to close inbound due to end of stream.
> 2017-05-23 11:58:22,465 ERROR [utils.nio.Link] (main:null) Failed to send server's CLOSE message due to socket channel's failure.
> 2017-05-23 11:58:22,466 ERROR [utils.nio.NioClient] (main:null) SSL Handshake failed while connecting to host: 192.168.12.1 port: 8250
> 2017-05-23 11:58:22,466 ERROR [utils.nio.NioConnection] (main:null) Unable to initialize the threads.
> java.io.IOException: SSL Handshake failed while connecting to host: 192.168.12.1 port: 8250
> at com.cloud.utils.nio.NioClient.init(NioClient.java:67)
> at com.cloud.utils.nio.NioConnection.start(NioConnection.java:88)
> at com.cloud.agent.Agent.start(Agent.java:237)
> at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:399)
> at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:367)
> at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:351)
> at com.cloud.agent.AgentShell.start(AgentShell.java:456)
> at com.cloud.agent.AgentShell.main(AgentShell.java:491)
> 2017-05-23 11:58:22,468 INFO [utils.exception.CSExceptionErrorCode] (main:null) Could not find exception: com.cloud.utils.exception.NioConnectionException in error code list for exceptions
> 2017-05-23 11:58:22,468 WARN [cloud.agent.Agent] (main:null) NIO Connection Exception com.cloud.utils.exception.NioConnectionException: SSL Handshake failed while connecting to host: 192.168.12.1 port: 8250
>
> The setup is very simple. Single management server and ports are open.
>
> Things checked / tried:
>
> · Destroyed SSVM multiple times – still same problem.
>
> · SSH to SSVM from MS using ssh -i /var/cloudstack/management/.ssh/id_rsa -p 3922 root@IPADDRESS – PASS
>
> · SSVM telnet on 8250 to MS – PASS
>
> I’ve also tested a restore of the DB into our working development 4.9.2.0 server. It also exhibits the handshake errors, so most likely DB related.
>
> I’ve used up all my skills. Please help
>
> Regards,
> Jason
>
Re: SSVM NIO SSL Handshake error
Posted by Erik Weber <te...@gmail.com>.
What's the OS? Think I saw something like this on some old CentOS 6 machines
--
Erik
On Tue, May 23, 2017 at 2:11 PM, Jason Kinsella
<ja...@cloudpeople.com.au> wrote:
> Hi,
> We recently upgraded from 4.5.0 to 4.9.2.0 and encountered a problem with the SSVM and Console Proxy. They cannot connect to the management server. The SSVM cloud.log repeats this error every couple of seconds.
>
> 2017-05-23 11:58:22,461 INFO [utils.nio.NioClient] (main:null) Connecting to 192.168.12.1:8250
> 2017-05-23 11:58:22,465 WARN [utils.nio.Link] (main:null) This SSL engine was forced to close inbound due to end of stream.
> 2017-05-23 11:58:22,465 ERROR [utils.nio.Link] (main:null) Failed to send server's CLOSE message due to socket channel's failure.
> 2017-05-23 11:58:22,466 ERROR [utils.nio.NioClient] (main:null) SSL Handshake failed while connecting to host: 192.168.12.1 port: 8250
> 2017-05-23 11:58:22,466 ERROR [utils.nio.NioConnection] (main:null) Unable to initialize the threads.
> java.io.IOException: SSL Handshake failed while connecting to host: 192.168.12.1 port: 8250
> at com.cloud.utils.nio.NioClient.init(NioClient.java:67)
> at com.cloud.utils.nio.NioConnection.start(NioConnection.java:88)
> at com.cloud.agent.Agent.start(Agent.java:237)
> at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:399)
> at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:367)
> at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:351)
> at com.cloud.agent.AgentShell.start(AgentShell.java:456)
> at com.cloud.agent.AgentShell.main(AgentShell.java:491)
> 2017-05-23 11:58:22,468 INFO [utils.exception.CSExceptionErrorCode] (main:null) Could not find exception: com.cloud.utils.exception.NioConnectionException in error code list for exceptions
> 2017-05-23 11:58:22,468 WARN [cloud.agent.Agent] (main:null) NIO Connection Exception com.cloud.utils.exception.NioConnectionException: SSL Handshake failed while connecting to host: 192.168.12.1 port: 8250
>
> The setup is very simple. Single management server and ports are open.
>
> Things checked / tried:
>
> · Destroyed SSVM multiple times – still same problem.
>
> · SSH to SSVM from MS using ssh -i /var/cloudstack/management/.ssh/id_rsa -p 3922 root@IPADDRESS – PASS
>
> · SSVM telnet on 8250 to MS – PASS
>
> I’ve also tested a restore of the DB into our working development 4.9.2.0 server. It also exhibits the handshake errors, so most likely DB related.
>
> I’ve used up all my skills. Please help
>
> Regards,
> Jason
>
Re: SSVM NIO SSL Handshake error
Posted by Jason Kinsella <ja...@cloudpeople.com.au>.
Hi Simon,
I tried this already and it didn’t make a difference.
jk
On 23/5/17, 10:45 pm, "Simon Weller" <sw...@ena.com.INVALID> wrote:
Jason,
When I've seen this happen, it has often been related to a corrupt keystore.
Trying backing up your db and following this procedure to regenerate your keystore:
mysql> delete from configuration where name = "ssl.keystore" ;
mv /etc/cloudstack/management/cloudmanagementserver.keystore /etc/cloudstack/management/cloudmanagementserver.keystore.old
service cloudstack-management restart
That should force the keystore to be regenerated.
- Si
________________________________
From: Jason Kinsella <ja...@cloudpeople.com.au>
Sent: Tuesday, May 23, 2017 7:11 AM
To: users@cloudstack.apache.org
Subject: SSVM NIO SSL Handshake error
Hi,
We recently upgraded from 4.5.0 to 4.9.2.0 and encountered a problem with the SSVM and Console Proxy. They cannot connect to the management server. The SSVM cloud.log repeats this error every couple of seconds.
2017-05-23 11:58:22,461 INFO [utils.nio.NioClient] (main:null) Connecting to 192.168.12.1:8250
2017-05-23 11:58:22,465 WARN [utils.nio.Link] (main:null) This SSL engine was forced to close inbound due to end of stream.
2017-05-23 11:58:22,465 ERROR [utils.nio.Link] (main:null) Failed to send server's CLOSE message due to socket channel's failure.
2017-05-23 11:58:22,466 ERROR [utils.nio.NioClient] (main:null) SSL Handshake failed while connecting to host: 192.168.12.1 port: 8250
2017-05-23 11:58:22,466 ERROR [utils.nio.NioConnection] (main:null) Unable to initialize the threads.
java.io.IOException: SSL Handshake failed while connecting to host: 192.168.12.1 port: 8250
Simon Weller
Director of Technology

Education Networks of America
618 Grassmere Park Dr, Ste 12
Nashville, TN 37211
Desk: 615-312-6068
Fax: 615-312-6099
website
blog
support
  
at com.cloud.utils.nio.NioClient.init(NioClient.java:67)
at com.cloud.utils.nio.NioConnection.start(NioConnection.java:88)
at com.cloud.agent.Agent.start(Agent.java:237)
at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:399)
at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:367)
at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:351)
at com.cloud.agent.AgentShell.start(AgentShell.java:456)
at com.cloud.agent.AgentShell.main(AgentShell.java:491)
2017-05-23 11:58:22,468 INFO [utils.exception.CSExceptionErrorCode] (main:null) Could not find exception: com.cloud.utils.exception.NioConnectionException in error code list for exceptions
2017-05-23 11:58:22,468 WARN [cloud.agent.Agent] (main:null) NIO Connection Exception com.cloud.utils.exception.NioConnectionException: SSL Handshake failed while connecting to host: 192.168.12.1 port: 8250
The setup is very simple. Single management server and ports are open.
Things checked / tried:
· Destroyed SSVM multiple times – still same problem.
· SSH to SSVM from MS using ssh -i /var/cloudstack/management/.ssh/id_rsa -p 3922 root@IPADDRESS – PASS
· SSVM telnet on 8250 to MS – PASS
I’ve also tested a restore of the DB into our working development 4.9.2.0 server. It also exhibits the handshake errors, so most likely DB related.
I’ve used up all my skills. Please help
Regards,
Jason
Re: SSVM NIO SSL Handshake error
Posted by Simon Weller <sw...@ena.com.INVALID>.
Jason,
When I've seen this happen, it has often been related to a corrupt keystore.
Trying backing up your db and following this procedure to regenerate your keystore:
mysql> delete from configuration where name = "ssl.keystore" ;
mv /etc/cloudstack/management/cloudmanagementserver.keystore /etc/cloudstack/management/cloudmanagementserver.keystore.old
service cloudstack-management restart
That should force the keystore to be regenerated.
- Si
________________________________
From: Jason Kinsella <ja...@cloudpeople.com.au>
Sent: Tuesday, May 23, 2017 7:11 AM
To: users@cloudstack.apache.org
Subject: SSVM NIO SSL Handshake error
Hi,
We recently upgraded from 4.5.0 to 4.9.2.0 and encountered a problem with the SSVM and Console Proxy. They cannot connect to the management server. The SSVM cloud.log repeats this error every couple of seconds.
2017-05-23 11:58:22,461 INFO [utils.nio.NioClient] (main:null) Connecting to 192.168.12.1:8250
2017-05-23 11:58:22,465 WARN [utils.nio.Link] (main:null) This SSL engine was forced to close inbound due to end of stream.
2017-05-23 11:58:22,465 ERROR [utils.nio.Link] (main:null) Failed to send server's CLOSE message due to socket channel's failure.
2017-05-23 11:58:22,466 ERROR [utils.nio.NioClient] (main:null) SSL Handshake failed while connecting to host: 192.168.12.1 port: 8250
2017-05-23 11:58:22,466 ERROR [utils.nio.NioConnection] (main:null) Unable to initialize the threads.
java.io.IOException: SSL Handshake failed while connecting to host: 192.168.12.1 port: 8250
Simon Weller
Director of Technology

Education Networks of America
618 Grassmere Park Dr, Ste 12
Nashville, TN 37211
Desk: 615-312-6068
Fax: 615-312-6099
website
blog
support
  
at com.cloud.utils.nio.NioClient.init(NioClient.java:67)
at com.cloud.utils.nio.NioConnection.start(NioConnection.java:88)
at com.cloud.agent.Agent.start(Agent.java:237)
at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:399)
at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:367)
at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:351)
at com.cloud.agent.AgentShell.start(AgentShell.java:456)
at com.cloud.agent.AgentShell.main(AgentShell.java:491)
2017-05-23 11:58:22,468 INFO [utils.exception.CSExceptionErrorCode] (main:null) Could not find exception: com.cloud.utils.exception.NioConnectionException in error code list for exceptions
2017-05-23 11:58:22,468 WARN [cloud.agent.Agent] (main:null) NIO Connection Exception com.cloud.utils.exception.NioConnectionException: SSL Handshake failed while connecting to host: 192.168.12.1 port: 8250
The setup is very simple. Single management server and ports are open.
Things checked / tried:
· Destroyed SSVM multiple times – still same problem.
· SSH to SSVM from MS using ssh -i /var/cloudstack/management/.ssh/id_rsa -p 3922 root@IPADDRESS – PASS
· SSVM telnet on 8250 to MS – PASS
I’ve also tested a restore of the DB into our working development 4.9.2.0 server. It also exhibits the handshake errors, so most likely DB related.
I’ve used up all my skills. Please help
Regards,
Jason
Re: SSVM NIO SSL Handshake error
Posted by Jason Kinsella <ja...@cloudpeople.com.au>.
Hi ilya,
Java version is 1.7 on management server
192.168.12.1 is the management server
Permissions on keystore are open to cloud user.
Regards,
jason
On 27/5/17, 4:22 am, "ilya" <il...@gmail.com> wrote:
Just wanted to make sure you are
1) running java 1.7 on management server
2) 192.168.12.1 is actual cloudstack management server and not gateway
Check the permission on your keystore and make sure cloud user can
access it.
Regards
ilya
On 5/23/17 5:11 AM, Jason Kinsella wrote:
> 2017-05-23 11:58:22,461 INFO [utils.nio.NioClient] (main:null) Connecting to 192.168.12.1:8250
Re: SSVM NIO SSL Handshake error
Posted by ilya <il...@gmail.com>.
Just wanted to make sure you are
1) running java 1.7 on management server
2) 192.168.12.1 is actual cloudstack management server and not gateway
Check the permission on your keystore and make sure cloud user can
access it.
Regards
ilya
On 5/23/17 5:11 AM, Jason Kinsella wrote:
> 2017-05-23 11:58:22,461 INFO [utils.nio.NioClient] (main:null) Connecting to 192.168.12.1:8250
Re: SSVM NIO SSL Handshake error
Posted by Jason Kinsella <ja...@cloudpeople.com.au>.
Hi Rajani,
Yes that fixed it. Thanks!
On 24/5/17, 11:50 pm, "Rajani Karuturi" <ra...@apache.org> wrote:
That might due to the new dynamic role based access checker.
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Dynamic+Role+Based+API+Access+Checker+for+CloudStack
~Rajani
Sent from phone.
On 24 May 2017 7:10 p.m., "Jason Kinsella" <ja...@cloudpeople.com.au> wrote:
> Hi All,
> Based on the feedback it seems like the issue is related to CentOS
> version, so I’ve built a new CentOS7 Management server using Blueshape
> noredist. I’ve restored the 4.9.2.0 DB into this server and
> management-server.logs look clean on boot. The only problem is that I can’t
> log into the webUI.
>
> The logs show a successful login (user = kinsja), but the the API command
> either is not allowed or doesn’t exist for the user. This means the UI
> doesn’t load.
>
> Anyone seen this with a restored DB?
>
> 2017-05-24 09:26:08,239 DEBUG [c.c.u.AccountManagerImpl]
> (catalina-exec-17:ctx-ee2c5e26) (logid:a8ca5ee5) User: kinsja in domain 1
> has successfully logged in
> 2017-05-24 09:26:08,246 INFO [c.c.a.ApiServer] (catalina-exec-17:ctx-ee2c5e26)
> (logid:a8ca5ee5) Current user logged in under timezone
> 2017-05-24 09:26:08,246 INFO [c.c.a.ApiServer] (catalina-exec-17:ctx-ee2c5e26)
> (logid:a8ca5ee5) Timezone offset from UTC is: 0.0
> 2017-05-24 09:26:08,251 DEBUG [c.c.a.ApiServlet] (catalina-exec-17:ctx-ee2c5e26)
> (logid:a8ca5ee5) ===END=== 192.168.10.38 -- POST
> 2017-05-24 09:26:08,320 DEBUG [c.c.a.ApiServlet] (catalina-exec-13:ctx-a1d38347)
> (logid:3404c663) ===START=== 192.168.10.38 -- GET
> command=listCapabilities&response=json&_=1495632368256
> 2017-05-24 09:26:08,325 DEBUG [c.c.a.ApiServer]
> (catalina-exec-13:ctx-a1d38347 ctx-960796a5) (logid:3404c663) The user with
> id:31 is not allowed to request the API command or the API command does not
> exist: listCapabilities
>
> Thanks
> Jason
>
> From: Jason Kinsella <ja...@cloudpeople.com.au>
> Date: Tuesday, 23 May 2017 at 10:11 pm
> To: "users@cloudstack.apache.org" <us...@cloudstack.apache.org>
> Subject: SSVM NIO SSL Handshake error
>
> Hi,
> We recently upgraded from 4.5.0 to 4.9.2.0 and encountered a problem with
> the SSVM and Console Proxy. They cannot connect to the management server.
> The SSVM cloud.log repeats this error every couple of seconds.
>
> 2017-05-23 11:58:22,461 INFO [utils.nio.NioClient] (main:null) Connecting
> to 192.168.12.1:8250
> 2017-05-23 11:58:22,465 WARN [utils.nio.Link] (main:null) This SSL engine
> was forced to close inbound due to end of stream.
> 2017-05-23 11:58:22,465 ERROR [utils.nio.Link] (main:null) Failed to send
> server's CLOSE message due to socket channel's failure.
> 2017-05-23 11:58:22,466 ERROR [utils.nio.NioClient] (main:null) SSL
> Handshake failed while connecting to host: 192.168.12.1 port: 8250
> 2017-05-23 11:58:22,466 ERROR [utils.nio.NioConnection] (main:null) Unable
> to initialize the threads.
> java.io.IOException: SSL Handshake failed while connecting to host:
> 192.168.12.1 port: 8250
> at com.cloud.utils.nio.NioClient.init(NioClient.java:67)
> at com.cloud.utils.nio.NioConnection.start(
> NioConnection.java:88)
> at com.cloud.agent.Agent.start(Agent.java:237)
> at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:
> 399)
> at com.cloud.agent.AgentShell.launchAgentFromClassInfo(
> AgentShell.java:367)
> at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:
> 351)
> at com.cloud.agent.AgentShell.start(AgentShell.java:456)
> at com.cloud.agent.AgentShell.main(AgentShell.java:491)
> 2017-05-23 11:58:22,468 INFO [utils.exception.CSExceptionErrorCode]
> (main:null) Could not find exception: com.cloud.utils.exception.NioConnectionException
> in error code list for exceptions
> 2017-05-23 11:58:22,468 WARN [cloud.agent.Agent] (main:null) NIO
> Connection Exception com.cloud.utils.exception.NioConnectionException:
> SSL Handshake failed while connecting to host: 192.168.12.1 port: 8250
>
> The setup is very simple. Single management server and ports are open.
>
> Things checked / tried:
>
> · Destroyed SSVM multiple times – still same problem.
>
> · SSH to SSVM from MS using ssh -i /var/cloudstack/management/.ssh/id_rsa
> -p 3922 root@IPADDRESS – PASS
>
> · SSVM telnet on 8250 to MS – PASS
>
> I’ve also tested a restore of the DB into our working development 4.9.2.0
> server. It also exhibits the handshake errors, so most likely DB related.
>
> I’ve used up all my skills. Please help
>
> Regards,
> Jason
>
>
Re: SSVM NIO SSL Handshake error
Posted by Rajani Karuturi <ra...@apache.org>.
That might due to the new dynamic role based access checker.
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Dynamic+Role+Based+API+Access+Checker+for+CloudStack
~Rajani
Sent from phone.
On 24 May 2017 7:10 p.m., "Jason Kinsella" <ja...@cloudpeople.com.au> wrote:
> Hi All,
> Based on the feedback it seems like the issue is related to CentOS
> version, so I’ve built a new CentOS7 Management server using Blueshape
> noredist. I’ve restored the 4.9.2.0 DB into this server and
> management-server.logs look clean on boot. The only problem is that I can’t
> log into the webUI.
>
> The logs show a successful login (user = kinsja), but the the API command
> either is not allowed or doesn’t exist for the user. This means the UI
> doesn’t load.
>
> Anyone seen this with a restored DB?
>
> 2017-05-24 09:26:08,239 DEBUG [c.c.u.AccountManagerImpl]
> (catalina-exec-17:ctx-ee2c5e26) (logid:a8ca5ee5) User: kinsja in domain 1
> has successfully logged in
> 2017-05-24 09:26:08,246 INFO [c.c.a.ApiServer] (catalina-exec-17:ctx-ee2c5e26)
> (logid:a8ca5ee5) Current user logged in under timezone
> 2017-05-24 09:26:08,246 INFO [c.c.a.ApiServer] (catalina-exec-17:ctx-ee2c5e26)
> (logid:a8ca5ee5) Timezone offset from UTC is: 0.0
> 2017-05-24 09:26:08,251 DEBUG [c.c.a.ApiServlet] (catalina-exec-17:ctx-ee2c5e26)
> (logid:a8ca5ee5) ===END=== 192.168.10.38 -- POST
> 2017-05-24 09:26:08,320 DEBUG [c.c.a.ApiServlet] (catalina-exec-13:ctx-a1d38347)
> (logid:3404c663) ===START=== 192.168.10.38 -- GET
> command=listCapabilities&response=json&_=1495632368256
> 2017-05-24 09:26:08,325 DEBUG [c.c.a.ApiServer]
> (catalina-exec-13:ctx-a1d38347 ctx-960796a5) (logid:3404c663) The user with
> id:31 is not allowed to request the API command or the API command does not
> exist: listCapabilities
>
> Thanks
> Jason
>
> From: Jason Kinsella <ja...@cloudpeople.com.au>
> Date: Tuesday, 23 May 2017 at 10:11 pm
> To: "users@cloudstack.apache.org" <us...@cloudstack.apache.org>
> Subject: SSVM NIO SSL Handshake error
>
> Hi,
> We recently upgraded from 4.5.0 to 4.9.2.0 and encountered a problem with
> the SSVM and Console Proxy. They cannot connect to the management server.
> The SSVM cloud.log repeats this error every couple of seconds.
>
> 2017-05-23 11:58:22,461 INFO [utils.nio.NioClient] (main:null) Connecting
> to 192.168.12.1:8250
> 2017-05-23 11:58:22,465 WARN [utils.nio.Link] (main:null) This SSL engine
> was forced to close inbound due to end of stream.
> 2017-05-23 11:58:22,465 ERROR [utils.nio.Link] (main:null) Failed to send
> server's CLOSE message due to socket channel's failure.
> 2017-05-23 11:58:22,466 ERROR [utils.nio.NioClient] (main:null) SSL
> Handshake failed while connecting to host: 192.168.12.1 port: 8250
> 2017-05-23 11:58:22,466 ERROR [utils.nio.NioConnection] (main:null) Unable
> to initialize the threads.
> java.io.IOException: SSL Handshake failed while connecting to host:
> 192.168.12.1 port: 8250
> at com.cloud.utils.nio.NioClient.init(NioClient.java:67)
> at com.cloud.utils.nio.NioConnection.start(
> NioConnection.java:88)
> at com.cloud.agent.Agent.start(Agent.java:237)
> at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:
> 399)
> at com.cloud.agent.AgentShell.launchAgentFromClassInfo(
> AgentShell.java:367)
> at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:
> 351)
> at com.cloud.agent.AgentShell.start(AgentShell.java:456)
> at com.cloud.agent.AgentShell.main(AgentShell.java:491)
> 2017-05-23 11:58:22,468 INFO [utils.exception.CSExceptionErrorCode]
> (main:null) Could not find exception: com.cloud.utils.exception.NioConnectionException
> in error code list for exceptions
> 2017-05-23 11:58:22,468 WARN [cloud.agent.Agent] (main:null) NIO
> Connection Exception com.cloud.utils.exception.NioConnectionException:
> SSL Handshake failed while connecting to host: 192.168.12.1 port: 8250
>
> The setup is very simple. Single management server and ports are open.
>
> Things checked / tried:
>
> · Destroyed SSVM multiple times – still same problem.
>
> · SSH to SSVM from MS using ssh -i /var/cloudstack/management/.ssh/id_rsa
> -p 3922 root@IPADDRESS – PASS
>
> · SSVM telnet on 8250 to MS – PASS
>
> I’ve also tested a restore of the DB into our working development 4.9.2.0
> server. It also exhibits the handshake errors, so most likely DB related.
>
> I’ve used up all my skills. Please help
>
> Regards,
> Jason
>
>
Re: SSVM NIO SSL Handshake error
Posted by Erik Weber <te...@gmail.com>.
could you try this command from the ssvm?
openssl s_client -connect <mgmt ip>:8250
--
Erik
On Mon, May 29, 2017 at 12:26 PM, Jason Kinsella
<ja...@cloudpeople.com.au> wrote:
> UPDATE: The mgmt. logs show that the cloudmanagementserver.keystore is being generated with password “vmops.com”. When we test it, the password is correct.
>
> keytool -list -keystore cloudmanagementserver.keystore -storepass vmops.com
>
> Keystore type: JKS
> Keystore provider: SUN
>
> Your keystore contains 1 entry
>
> mykey, May 29, 2017, PrivateKeyEntry,
> Certificate fingerprint (SHA1): 9C:91:BA:06:BB:4B:D1:B4:24:3F:8A:13:03:FD:6B:76:4E:9F:EA:29
>
> LOGS
>
> ```1181097 2017-05-29 19:56:20,258 INFO [c.c.s.ConfigurationServerImpl] (main:null) Processing updateSSLKeyStore
> 1181104 2017-05-29 19:56:20,261 INFO [c.c.s.ConfigurationServerImpl] (main:null) SSL keystore located at /etc/cloudstack/management/cloudmanagementserver.keystore
> 1181105 2017-05-29 19:56:20,268 DEBUG [c.c.u.s.Script] (main:null) Executing: sudo keytool -genkey -keystore /etc/cloudstack/management/cloudmanagementserver.keystore -storepass vmops.com -keypass vmops.com -keyalg RSA -validity 3650 -dn1181105 ame cn="Cloudstack User",ou="localhost",o="localhost",c="Unknown"
> 1181106 2017-05-29 19:56:21,511 DEBUG [c.c.u.s.Script] (main:null) Execution is successful.
> 1181107 2017-05-29 19:56:21,512 INFO [c.c.s.ConfigurationServerImpl] (main:null) Generated SSL keystore.
> 1181111 2017-05-29 19:56:21,552 TRACE [c.c.u.d.T.Statement] (main:null) Preparing: INSERT INTO configuration (configuration.instance, configuration.component, configuration.name, configuration.value, configuration.default_value, configur1181111 ation.description, configuration.category, configuration.is_dynamic, configuration.scope, configuration.updated) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
> 1181112 2017-05-29 19:56:21,556 TRACE [c.c.u.d.T.Transaction] (main:null) txn: DB Changes committed. Time = 6
> 1181113 2017-05-29 19:56:21,557 TRACE [c.c.u.d.T.Statement] (main:null) Closing: com.mysql.jdbc.JDBC4PreparedStatement@29f7a353: INSERT INTO configuration (configuration.instance, configuration.component, configuration.name, configuratio1181113 n.value, configuration.default_value, configuration.description, configuration.category, configuration.is_dynamic, configuration.scope, configuration.updated) VALUES (_binary'DEFAULT', _binary'management-server', _binary'ssl.keys1181113 tore', _binary'WKcZHRCDXhtqcLDK3ZSTyUYoZ3N81oiYbuool9WU2glPddGLrDZ4FyfQuThHXS4fH1fWPbKsjo9LgbvYUfFeAi4mdrfwOHXbCZPozlxPnBPp6gQh2eAKwB2eGytwZITxp0x9fqTv1bk0wWkOoEAEH37e6h+lWPvmv6eYvL85XGBcAJFVxRRHJwXl6u3r9dUobWF25lb6LLvi0P9kBpRFtb1181113 iv/p4DnCZgDaWrEwQCdtETwE6fMCVPxrUm/uEV4t0rbYApLU3ttpQolrR2USsBFlxFIxK4vYI9NeQK5dHK252s7pJriuQzwA14AyI36HLJZRkl/qVTu/ttIMqT9UcWvC5UZFC6iuv8E+eH5KFQhlx3HY/nvyQ8ADDgopmv0svmCBEHXP7mTN+K7JuPN8qBq9gaTNRNMAR33ma0r6lWD4K1+c7luKQS3hAMDMD1181113 KlkN09QTReNjydySuLnF6tlQQ1I1FPfa+eFE1VSj75ktd0nKgbxVnps79DceCFx428w6Jh9M7GRMvXykSHxI4SxrqXUeo4h1pX618r1kGLoG6SYPfuawRQqiC7ZouScyBK/z1egxR+xajXhMWxBtjfPxRx6fgwG4ouFT9RH6YhtFA+ppug+fhJvL2x9EZ78o6zDYpVmOOehLffHRdZHDtwtFg7srdmsNBHHg31181113 ukIYlF1MWvhnyUE0+nDu6HAgl4p+lt8lohxEFUoMa4bWaHVYOLMvgtqecdCU6CKwzGy3TiL/FI0/e1jUX5n9Y9FY5w7cjMnl2T95VKGXlnrBAyjo08Xt3MPM7xQduHM2kRYJsDXpJtokv+zWPmLUlPjTvnfLbobsBcNdrpU1ZPDPFXjPto5Ud3oPHzD1nu1DsK/Mfae6Lo8C0rl+I2kfCi0BZEKCJ0um8MzqZ1181113 0u32cbSXS/T7YU4jJNaiAujS3BDCb9L+u7irvhFTqe2PzDizV/YJa4y2YpQOpLv3m9JCsG0A1gvdZfOwDVUmg580TpomjPcmeH9bsOP8ESXa3d4Sm8tPHsetZrhnEtss0+OERTtNabFohAcjSKABIvcodq5HEqBzQtpyQN5MJA3YfarQMnPnAfIbjXszvqI04euBbQ2gyvDWoRIrcPjCjsUh9jRSUOMHaLVDg1181113 Cr4gK02+LhnGQeCXWjFo2i4FXCEajDgVpMHwYFhD1Uhdf3ttHBQNTaZ1ObTmP7E27H1VRpDvCP4omdFpxGV250ls5mj8uwMxrPchX7S7BlRbUT8XMyOm9RZiavRenOJp0xGWh+/YKKQFhDWTUyV3MiKah4iEPJ02fSpZ8pB+y4IYDO9ya+apLv8E4Bi90AZYOpWSYQr6t0wGcXpmY0JMyDoUxidh24HqO+/xJ1181113 6npD7SvEkoyFuXFAUaoamZ0vn/thxF5ZlTpA3Gfpny1j/4SLyV+GZWTK7wRhLjbZOonghXm2N6jpBYX/2dfdvoogTkEaxJHNitNyiZYwJm8fC7iaVmSB/OsJPs3Nd1ZmeljUCX6xsmQuiI47qDRmp1ZseQQM2HdGQqrK1X3EEcPa4BpgmVeOVRZ+uZ6gxAFYqQFAeKz6tUO6vQsOO/GAXwNwFSQDtBPUzC0ic1181113 P01cVkeUVlj56BaAp0Ia/tUbSDe5LhEdWo/C5innVeltJ//4Evhk3zHuyT1aiMJgWEwzZLgYuF1jHqK//ybSquXZJOyjrun+LVZs1vpp/qtRg3fk9qMjglnP0WIN44Kh/vnc7dDCrsyr/Ezu5Bh9oM+miNXVv5KH+2fMFYVzElB7Q0+kVmKkeucjtaBIlgtRjHFgK+uQzb+OZJSHByGx4Ci0jkDAxBNP4OsLf1181113 SC4pv8i8IppmcTwZGUbOye4x7QtvCo4JVzD5eZydzTD+GYFXgoeeFLveSm5DU44jeVjc3tKfcrhPeBc8pM3oGuSyvalYs7LGbf/9ozeg2vJWmQwEamPqHtq6ceg4qVRs2cIzaelc+yGT9BBOcUnHzCgr+gOC/F3lCJHbjUmyH52j4SLHZ3sZ6LB6CCVnkJ2Z7QYpCXe7lEIx3iX13+xBMSGwE/p4jKwJYBW0O1181113 k5dgTUG/QOEWhQVtdn33+uOH8X2/MMFMofJzaF7XC/mOCWtzSucTq1Obbng8ySAvp5ESIYfBtGeG3jgN+Pda7SzyDv8kQbyHFulyK+wzhYq/3lqOLnXP4P4nShHn6td7MunMJ6yPUX/swx5Pa35vpH/6IGxWHFMjJtzH/y8sliyy9SE3o4z6R3kVK+AWG7GvHRCdS9BNIJJyrjuEUuBcRlFpWPjEMw4n4ZSy81181113 +JJ0wlDpk8mlHflOLcuB72RAXyuHgsqi/8gIc5eM8Jf2968Av5WOeW0V4UHop0/wXOyXZf0bQisNzU0iBcRbszemlHPS3wCNmgkDIUgq2fkRZaLQO2hyurDzsEARQUE90XIKg7JWzkPXOMXZ6XR57INVL8kB3dPLkwzNOtC62YjbS8670SfWFjochnJSjlhAqFtVkuY8etyZ5mJRzrLl9b3MYlsmxHmdCCP5b1181113 JfC/Nq7ugKLq5RO/ACaz9Y934Q68K1lJYY2mHJHVnbeK9PewVfTjRG8A5NDhDBIduABoZi0DIAsz3lmEZGWG2V2GkIfGfha/fyvz9sasgO/Wrx4+Xpkt2aT+IU9f289ak9w1Bo/5DzdxmAZIHdonOUlb5TET5Eel114iW8mVO42snhKukWPMSayZVFPH9aCvdv3+5v7q5z8HPsnvSLpEiQ04LhVcGRHGV9+Qn1181113 EjcfFBTTIFjTFXb5d93+LPjoM6061kyqe60TqaTp1T0nxg4czhxTU1lpRkAFl88dyhJC4ZlJBsOp7tltzjyEa19Dw2nCySnyJ0dGVbZQ4vrIWkoQS4XxxbcWtP7gGLjmIL+bm3hEKSpotuiflAZlfbmqYILqHFzjkloYxSYjjJCoYzGaKCNCBT/ChGpDZamChdOVd2usCcSOqWZGa2dlQKp9kDkXQAMpefa8c1181113 YW0DRsgdqHLgXujEtHYbTrKxGhQGb4/yamDbZ+V4XvErxfAinlCUotrYUm7B71MSGgndXBTcq+/+oETKbaRdkx2qgbEqOZXUtv6uMrXIEvL3l+msb3VG1gz7d8cqWXs9VY/6HHCxNDKeldGMrGqTQPYznMzRiiCXyp+u+nOTf+gOzTfSl2BUm94jv/Hcjpj4zXnL1TV3KSjwBNRJFCIhh1sdqhvt0Xi9QJn2R1181113 QaLJ9y2TrBDGYEAmMgmXDVTa5CXBau0BwdiNiwXFvJkt7Vsvn7dRXTFyJ1YTO4c0H3n5B69QiWYaD3V7iRVmRzuQg8nSXTGh35Ol5Qh5lNI8byroC1gDMbxCjdJwR0fpUbn5fq4QFBgHDr9agFFf1kR2T4if+m4WFEGCP8bTwek5FL/K8CQLJ1JgCccWZy+2JZh2WQbVUvL7Bq5zcADUlM3ikHg4CrRZR8A3p1181113 4PM95HN/fu9+f2ZRj6ZF5TYDxk0weLJBVHbnph8mOLB0li5iPdUytNuKTCFJc1G1QUjB7fuV9g6l5mTG46wg0xU1mYJ9LB1oFix1lysnXBBaSYmnEzfiJbYqKqcvXJ+ycMPPAfmNDEyrzo0LTQtvyat7M2UGiAQ5cn0ahXEcZzSqmog/VE8xGJkBi1coP7Unp0lscuLgRxD7ev5Yru9iv6at0WSuvpZskVajq1181113 +B+R2XVEBMPFTjVynSr53fI46lhqr/YxzCEjVSuz0JDs5IBaahzT1U/wVKHbY6it2ZFNAwsvAG46CoYtJF06rYsRNu7Y+WbJC97xJciY2rpEq+Y2ZMo7Z20/hTSnBCg0IqUNxO+G/p9GxGRqRPA440Mcs5wz9ye7sYz0Z2uyZ5UUgLiNqL5lyw+dd9IxcPq5qaxJYRscsiL2T7IlYAJLrBn/9iz2gHVFEvZQF1181113 1QRDJWQ3ApqmaQrT2ZyZINY4pDOkNshRnBOjyEFosp3DEvdQ==', null, _binary'SSL Keystore for the management servers', _binary'Hidden', 0, null, null)
> 1181114 2017-05-29 19:56:21,557 TRACE [c.c.u.d.T.Connection] (main:null) Closing DB connection: dbconn2026396576
> 1181115 2017-05-29 19:56:21,558 TRACE [c.c.u.d.T.Connection] (main:null) Creating a DB connection with no txn: for 0: dbconn1496793545. Stack: -TransactionLegacy.prepareStatement:467-TransactionLegacy.prepareAutoCloseStatement:460-Gene1181115 ricDaoBase.findById:1022-GenericDaoBase.findByIdIncludingRemoved:987-GenericDaoBase.persist:1435-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:57-DelegatingMethodAccessorImpl.invoke:43-Method.invoke:606-AopU1181115 tils.invokeJoinpointUsingReflection:317-ReflectiveMethodInvocation.invokeJoinpoint:183-ReflectiveMethodInvocation.proceed:150
> 1181116 2017-05-29 19:56:21,559 TRACE [c.c.u.d.T.Statement] (main:null) Preparing: SELECT configuration.instance, configuration.component, configuration.name, configuration.value, configuration.default_value, configuration.description, c1181116 onfiguration.category, configuration.is_dynamic, configuration.scope, configuration.updated FROM configuration WHERE configuration.name = ?
> 1181117 2017-05-29 19:56:21,562 TRACE [c.c.u.d.T.Statement] (main:null) Closing: com.mysql.jdbc.JDBC4PreparedStatement@3ac020e1: SELECT configuration.instance, configuration.component, configuration.name, configuration.value, configurati1181117 on.default_value, configuration.description, configuration.category, configuration.is_dynamic, configuration.scope, configuration.updated FROM configuration WHERE configuration.name = _binary'ssl.keystore'
> 1181118 2017-05-29 19:56:21,562 TRACE [c.c.u.d.T.Connection] (main:null) Closing DB connection: dbconn1496793545
> 1181119 2017-05-29 19:56:21,562 TRACE [c.c.u.d.T.Transaction] (main:null) Transaction is done
> 1181120 2017-05-29 19:56:21,562 INFO [c.c.s.ConfigurationServerImpl] (main:null) Stored SSL keystore to database.
>
> On 29/5/17, 8:12 pm, "Jason Kinsella" <ja...@cloudpeople.com.au> wrote:
>
> Hi Rohit,
> Here’s the telnet responses for all scenarios are the same: The connection is closed immediately.
>
> We found these in the trace logs
>
> 1187417 2017-05-29 19:56:41,471 TRACE [c.c.u.n.NioConnection] (pool-3-thread-1:null) Keys Processing: 1
> 1187419 2017-05-29 19:56:41,476 TRACE [c.c.u.n.NioConnection] (pool-3-thread-1:null) Connection accepted for Socket[addr=/192.168.12.63,port=35673,localport=8250]
> 1187426 2017-05-29 19:56:41,498 TRACE [c.c.u.n.NioConnection] (pool-3-thread-1:null) Connection closed due to failure: Keystore was tampered with, or password was incorrect
> 1187427 2017-05-29 19:56:41,499 TRACE [c.c.u.n.NioConnection] (pool-3-thread-1:null) Keys Done Processing.
> 1187428 2017-05-29 19:56:41,499 TRACE [c.c.u.n.NioConnection] (pool-3-thread-1:null) Keys Processing: 0
> 1187429 2017-05-29 19:56:41,499 TRACE [c.c.u.n.NioConnection] (pool-3-thread-1:null) Keys Done Processing.
>
> Is this referring to the cloudmanagementserver.keystore file and mysql.cloud.configuration ssl.keystore entry? These have been recreated many times and permissions appear correct.
>
> Hopefully this helps.
>
> Regards, Jason
>
>
>
>
> On 29/5/17, 5:35 pm, "Rohit Yadav" <ro...@shapeblue.com> wrote:
>
> Hi Jason,
>
>
> - Ensure that the 'host' global setting is the IP of the mgmt server or the VIP/LB in front of the mgmt server(s)
>
> - If you need to change the host global setting, restart mgmt server and destroy old SSVMs
>
> - If you try telnet on mgmt server (host) IP, on port 8250, does telnet immediately stop:
>
> telnet localhost 8250
> Trying 127.0.0.1...
> Connected to localhost.
> Escape character is '^]'.
>
> Try telnetting both from the mgmt server host (i.e. use localhost) and from the ssvm.
>
> - If telnet exists immediately, that means mgmt server host is unable to accept client connections on port 8250 because of which SSL handshake fails leading to the exceptions in the logs. The error message "Failed to send server's CLOSE message due to socket channel's failure." suggests that mgmt server was not able to accept the connection. The possible reasons for such a behaviour can be: (a) there is an issue with keystore setup in the mgmt server (the server component NioServer handles accept(), during which SSL handshake is performed and it fails), i.e. the SSLContext is not properly initialized so any client ssl handshake fails; (b) iptables or network/firewall rules blocking connections to port 8250 on the mgmt server host, either flush the 'filter' table or add ALLOW rules for 0.0.0.0/0 on port 8250.
>
> - In case the issue is with keystore setup, delete any *.keystore file from the mgmt server's classpath, i.e. search and remove any .keystore file from /etc/cloudstack/management and from /usr/share/cloudstack-management/, and restart the management server.
>
> Regards.
>
>
> ________________________________
> From: Jason Kinsella <ja...@cloudpeople.com.au>
> Sent: 29 May 2017 05:23:09
> To: users@cloudstack.apache.org
> Subject: Re: SSVM NIO SSL Handshake error
>
> UPDATE: I manually propagated a new systemvm.iso to secstorage and I can now ssh to the ssvm.
>
> Unfortunately same handshake errors in logs.
>
> On 28/5/17, 9:44 pm, "Jason Kinsella" <ja...@cloudpeople.com.au> wrote:
>
> Hi Rohit,
> I completed the test. The results are as follows:
>
> The ssh.publickey and ssh.privatekey deletions in DB triggered the keys to be recreated, but not in the /var/lib/cloudstack/management/.ssh folder. They were recreated in /var/cloudstack/management/.ssh folder.
>
> Here’s the log entry:
>
> 2017-05-28 20:53:28,508 DEBUG [c.c.u.s.Script] (localhost-startStop-1:null) (logid:) Executing: /bin/bash -c if [ -f /var/cloudstack/management/.ssh/id_rsa ]; then rm -f /var/cloudstack/management/.ssh/id_rsa; fi; ssh-keygen -t rsa -N '' -f /var/cloudstack/management/.ssh/id_rsa –q
>
> I then destroyed the ssvm and after recreation could not ssh to it from management server due to key publickey error. I have always previously been able to ssh to the ssvm using the var/cloudstack/management/.ssh keys.
>
> I checked the management server logs and found entry about id_rsa files differ – injecting into systemvm.iso
>
> Hopefully this is helpful.
>
> Regards,
> Jason
>
>
> On 28/5/17, 5:29 pm, "Rohit Yadav" <ro...@shapeblue.com> wrote:
>
> Hi Jason,
>
>
> In your test environment that uses the same db, can you try to do a workaround-experiment from [1]:
>
>
> 0) chmod +r and chown cloud:cloud relevant file and locations.
>
>
> 1) Stop Management Server
>
> 2) Delete SSH Keys in mysql Database: delete from configuration where name = "ssh.publickey" ; delete from configuration where name = "ssh.privatekey" ;
>
> 3) Delete the SSH Keys rm /var/lib/cloudstack/management/.ssh/id_rsa.pub rm /var/lib/cloudstack/management/.ssh/id_rsa
>
> 4) Start the Management Server - SSH Keys are generated and mysql entries inserted
>
>
>
> [1] http://markmail.org/message/zfjyd7s22itg7t7q
>
>
> Regards.
>
> ________________________________
> From: Jason Kinsella <ja...@cloudpeople.com.au>
> Sent: 27 May 2017 05:33:43
> To: users@cloudstack.apache.org
> Subject: Re: SSVM NIO SSL Handshake error
>
> Files are linked here.
>
> https://dl.dropboxusercontent.com/u/10588206/acs492/managmenet-server-logs.tar.gz
> https://dl.dropboxusercontent.com/u/10588206/acs492/systemvm.tar.gz
>
> Today we did a couple of additional tests that proved interesting. We’ve got a prod and a dev server. Both were upgraded last month. The prod has the error, but the dev is working. Everything was the same including CentOS 6.5.
>
> We restored the dev DB into the fresh CentOS7 box and it displayed the same problem. This would suggest an OS issue. Therefore, the converse should work. We restored the prod DB into the dev server and it continues to exhibit the problem.
>
> This suggests that we may have missed something in the migration between servers. Here’s steps:
>
> Stop cloudstack-man service on broken box
> Dump DB
> Copy to new and restore
> Copy db.properties & key files and update IP entry in db.properties
> Update DB entry host to new IP
> Delete DB ssl.keystore and keystore file
> Destroy systemVMs in Vmware
> Start cloudstack-man on new box
>
> The /var/cloudstack/management/.ssh/ files are referenced when we ssh to ssvm from MS so they are correct. What about ssh.public and ssh.private in db.cloud.configuration table?
>
> Regards,
> Jason
>
> On 25/5/17, 7:51 pm, "Rohit Yadav" <ro...@shapeblue.com> wrote:
>
> Hi Jason,
>
>
> Thanks for sharing the details. Yes, with the new setup please share with us the mgmt server logs and ssvm logs with TRACE enabled in the log4j configuration.
>
>
> Regards.
>
> ________________________________
> From: Jason Kinsella <ja...@cloudpeople.com.au>
> Sent: 25 May 2017 12:49:50
> To: users@cloudstack.apache.org
> Subject: Re: SSVM NIO SSL Handshake error
>
> Hi Rohit,
> API login – fixed.
>
> Latest systemvmtemplate (shapeblue new) in place – no improvement
>
> No loadbalancer or known service on MS port 8250
>
> I am doing my testing now on a fresh install of CentOS7 using shapeblue noredist with DB restored.
>
> Hypervisor = vmware vsphere 6.5 with ESX 6.5
>
> Systemvm.iso is dated today
>
> All systemvms are exhibiting same behaviour.
>
> Would any other logs help?
>
> Regards,
> Jason
>
> On 25/5/17, 4:55 pm, "Rohit Yadav" <ro...@shapeblue.com> wrote:
>
> Hi Jason,
>
>
> The API login issue can be fixed by following this, which I believe you have already fixed: http://docs.cloudstack.apache.org/projects/cloudstack-administration/en/4.9/accounts.html#using-dynamic-roles
>
>
> If not already in-use, can you try using the latest systemvmtemplate (for 4.6-4.9) from http://packages.shapeblue.com/systemvmtemplate/4.6/new.
>
>
> Do you have a load-balancer on port 8250 on the management server(s), or any script/service that may be trying to perform a tcp-connect on mgmt server's port 8250?
>
>
> When you upgrade can you make sure that both cloudstack-common and cloudstack-management packages are upgraded to 4.9.2.0? Also, what hypervisor(s) are you using?
>
>
> The following error may hint that the jars on systemvms may not be updated, as one of the exception classes are missing:
>
>
> 2017-05-23 11:58:22,468 INFO [utils.exception.CSExceptionErrorCode] (main:null) Could not find exception: com.cloud.utils.exception.NioConnectionException in error code list for exceptions
>
>
> Can you check that systemvm.iso are synced across hosts: (1) make sure cloudstack-common package is upgraded/updated to the same version as cloudstack-management (4.9.2.0), (2) if you're using vmware, delete this from the secondary storage, (3) for xenserver force reconnect on the host (from ui/api) or manually copy the scripts to xenserver host(s), (4) for kvm upgrade the cloudstack-common package.
>
>
> Destroy all other systemvms and see if you can reproduce the issue?
>
>
> Regards.
>
> ________________________________
> From: Jason Kinsella <ja...@cloudpeople.com.au>
> Sent: 25 May 2017 09:32:25
> To: users@cloudstack.apache.org
> Subject: Re: SSVM NIO SSL Handshake error
>
> Also, just wanted to mention that the symptoms we have with systemvms not connecting is described in the mail-list
>
> CS 4.9 NIO Selector wait time PR-1601 - https://www.mail-archive.com/dev@cloudstack.apache.org/msg69154.html
>
> The only difference is that this thread refers to KVM hosts not connecting.
>
> I’ve tried most suggestions in this thread.
>
> On 25/5/17, 1:51 pm, "Jason Kinsella" <ja...@cloudpeople.com.au> wrote:
>
> Java versions are as follows:
>
> MS: 1.7.0_141
> SSVM: 1.7.0_85
>
> Deleted keystore files (again) and restarted MS, then recreated the SSVM.
>
> Errors from SSVM:/var/log/cloud.log
>
> 2017-05-25 03:01:28,757 INFO [utils.nio.NioClient] (main:null) Connecting to 192.168.12.5:8250
> 2017-05-25 03:01:29,293 WARN [utils.nio.Link] (main:null) This SSL engine was forced to close inbound due to end of stream.
> 2017-05-25 03:01:29,293 ERROR [utils.nio.Link] (main:null) Failed to send server's CLOSE message due to socket channel's failure.
> 2017-05-25 03:01:29,294 ERROR [utils.nio.NioClient] (main:null) SSL Handshake failed while connecting to host: 192.168.12.5 port: 8250
> 2017-05-25 03:01:29,294 ERROR [utils.nio.NioConnection] (main:null) Unable to initialize the threads.
> java.io.IOException: SSL Handshake failed while connecting to host: 192.168.12.5 port: 8250
> at com.cloud.utils.nio.NioClient.init(NioClient.java:67)
> at com.cloud.utils.nio.NioConnection.start(NioConnection.java:88)
> at com.cloud.agent.Agent.start(Agent.java:228)
> at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:399)
> at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:367)
> at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:351)
> at com.cloud.agent.AgentShell.start(AgentShell.java:456)
> at com.cloud.agent.AgentShell.main(AgentShell.java:491)
>
> Same SSL engine forced to close inbound due to end of stream
>
>
>
>
>
>
>
> On 25/5/17, 1:51 am, "Rajani Karuturi" <ra...@apache.org> wrote:
>
> Can you check java version? Set the default java to 1.7 and delete keystore
> files and restart MS
>
> ~Rajani
>
> Sent from phone.
>
> On 24 May 2017 9:15 p.m., "Jason Kinsella" <ja...@cloudpeople.com.au> wrote:
>
> > I have now moved management server to a fresh CentOS7 server. But
> > unfortunately I’m getting the exact same SSL handshake error. Back to
> > square one.
> >
> > On 24/5/17, 11:40 pm, "Jason Kinsella" <ja...@cloudpeople.com.au> wrote:
> >
> > Hi All,
> > Based on the feedback it seems like the issue is related to CentOS
> > version, so I’ve built a new CentOS7 Management server using Blueshape
> > noredist. I’ve restored the 4.9.2.0 DB into this server and
> > management-server.logs look clean on boot. The only problem is that I can’t
> > log into the webUI.
> >
> > The logs show a successful login (user = kinsja), but the the API
> > command either is not allowed or doesn’t exist for the user. This means the
> > UI doesn’t load.
> >
> > Anyone seen this with a restored DB?
> >
> > 2017-05-24 09:26:08,239 DEBUG [c.c.u.AccountManagerImpl]
> > (catalina-exec-17:ctx-ee2c5e26) (logid:a8ca5ee5) User: kinsja in domain 1
> > has successfully logged in
> > 2017-05-24 09:26:08,246 INFO [c.c.a.ApiServer] (catalina-exec-17:ctx-ee2c5e26)
> > (logid:a8ca5ee5) Current user logged in under timezone
> > 2017-05-24 09:26:08,246 INFO [c.c.a.ApiServer] (catalina-exec-17:ctx-ee2c5e26)
> > (logid:a8ca5ee5) Timezone offset from UTC is: 0.0
> > 2017-05-24 09:26:08,251 DEBUG [c.c.a.ApiServlet] (catalina-exec-17:ctx-ee2c5e26)
> > (logid:a8ca5ee5) ===END=== 192.168.10.38 -- POST
> > 2017-05-24 09:26:08,320 DEBUG [c.c.a.ApiServlet] (catalina-exec-13:ctx-a1d38347)
> > (logid:3404c663) ===START=== 192.168.10.38 -- GET
> > command=listCapabilities&response=json&_=1495632368256
> > 2017-05-24 09:26:08,325 DEBUG [c.c.a.ApiServer]
> > (catalina-exec-13:ctx-a1d38347 ctx-960796a5) (logid:3404c663) The user with
> > id:31 is not allowed to request the API command or the API command does not
> > exist: listCapabilities
> >
> > Thanks
> > Jason
> >
> > From: Jason Kinsella <ja...@cloudpeople.com.au>
> > Date: Tuesday, 23 May 2017 at 10:11 pm
> > To: "users@cloudstack.apache.org" <us...@cloudstack.apache.org>
> > Subject: SSVM NIO SSL Handshake error
> >
> > Hi,
> > We recently upgraded from 4.5.0 to 4.9.2.0 and encountered a problem
> > with the SSVM and Console Proxy. They cannot connect to the management
> > server. The SSVM cloud.log repeats this error every couple of seconds.
> >
> > 2017-05-23 11:58:22,461 INFO [utils.nio.NioClient] (main:null)
> > Connecting to 192.168.12.1:8250
> > 2017-05-23 11:58:22,465 WARN [utils.nio.Link] (main:null) This SSL
> > engine was forced to close inbound due to end of stream.
> > 2017-05-23 11:58:22,465 ERROR [utils.nio.Link] (main:null) Failed to
> > send server's CLOSE message due to socket channel's failure.
> > 2017-05-23 11:58:22,466 ERROR [utils.nio.NioClient] (main:null) SSL
> > Handshake failed while connecting to host: 192.168.12.1 port: 8250
> > 2017-05-23 11:58:22,466 ERROR [utils.nio.NioConnection] (main:null)
> > Unable to initialize the threads.
> > java.io.IOException: SSL Handshake failed while connecting to host:
> > 192.168.12.1 port: 8250
> > at com.cloud.utils.nio.NioClient.
> > init(NioClient.java:67)
> > at com.cloud.utils.nio.NioConnection.start(
> > NioConnection.java:88)
> > at com.cloud.agent.Agent.start(Agent.java:237)
> > at com.cloud.agent.AgentShell.
> > launchAgent(AgentShell.java:399)
> > at com.cloud.agent.AgentShell.
> > launchAgentFromClassInfo(AgentShell.java:367)
> > at com.cloud.agent.AgentShell.
> > launchAgent(AgentShell.java:351)
> > at com.cloud.agent.AgentShell.
> > start(AgentShell.java:456)
> > at com.cloud.agent.AgentShell.
> > main(AgentShell.java:491)
> > 2017-05-23 11:58:22,468 INFO [utils.exception.CSExceptionErrorCode]
> > (main:null) Could not find exception: com.cloud.utils.exception.NioConnectionException
> > in error code list for exceptions
> > 2017-05-23 11:58:22,468 WARN [cloud.agent.Agent] (main:null) NIO
> > Connection Exception com.cloud.utils.exception.NioConnectionException:
> > SSL Handshake failed while connecting to host: 192.168.12.1 port: 8250
> >
> > The setup is very simple. Single management server and ports are open.
> >
> > Things checked / tried:
> >
> > · Destroyed SSVM multiple times – still same problem.
> >
> > · SSH to SSVM from MS using ssh -i /var/cloudstack/management/.ssh/id_rsa
> > -p 3922 root@IPADDRESS – PASS
> >
> > · SSVM telnet on 8250 to MS – PASS
> >
> > I’ve also tested a restore of the DB into our working development
> > 4.9.2.0 server. It also exhibits the handshake errors, so most likely DB
> > related.
> >
> > I’ve used up all my skills. Please help
> >
> > Regards,
> > Jason
> >
> >
> >
> >
>
>
>
>
>
> rohit.yadav@shapeblue.com
> www.shapeblue.com<http://www.shapeblue.com>
> 53 Chandos Place, Covent Garden, London WC2N 4HSUK
> @shapeblue
>
>
>
>
>
>
> rohit.yadav@shapeblue.com
> www.shapeblue.com<http://www.shapeblue.com>
> 53 Chandos Place, Covent Garden, London WC2N 4HSUK
> @shapeblue
>
>
>
>
>
>
>
>
> rohit.yadav@shapeblue.com
> www.shapeblue.com<http://www.shapeblue.com>
> 53 Chandos Place, Covent Garden, London WC2N 4HSUK
> @shapeblue
>
>
>
>
>
>
>
>
> rohit.yadav@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London WC2N 4HSUK
> @shapeblue
>
>
>
>
>
>
>
Re: SSVM NIO SSL Handshake error
Posted by Jason Kinsella <ja...@cloudpeople.com.au>.
SOLVED!!!!!!!!!!!!!!
Line 417 of this - https://github.com/apache/cloudstack/blob/87ef8137534fa798101f65c6691fcf71513ac978/utils/src/main/java/com/cloud/utils/nio/Link.java
File confFile = PropertiesUtil.findConfigFile("db.properties");
if (null != confFile && !isClient) {
final String pass = DbProperties.getDbProperties().getProperty("db.cloud.keyStorePassphrase");
char[] passphrase = "vmops.com".toCharArray();
if (pass != null) {
passphrase = pass.toCharArray();
}
We checked the db.properties file and db.cloud.keyStorePassphrase was present but did not have a value. In our dev system db.properties did not have this entry, so must have automatically reverted to vmops.com
I have no idea how this happened, but both systems had the same upgrade process.
Thanks everyone for their suggestions and assistance.
On 29/5/17, 8:26 pm, "Jason Kinsella" <ja...@cloudpeople.com.au> wrote:
UPDATE: The mgmt. logs show that the cloudmanagementserver.keystore is being generated with password “vmops.com”. When we test it, the password is correct.
keytool -list -keystore cloudmanagementserver.keystore -storepass vmops.com
Keystore type: JKS
Keystore provider: SUN
Your keystore contains 1 entry
mykey, May 29, 2017, PrivateKeyEntry,
Certificate fingerprint (SHA1): 9C:91:BA:06:BB:4B:D1:B4:24:3F:8A:13:03:FD:6B:76:4E:9F:EA:29
LOGS
```1181097 2017-05-29 19:56:20,258 INFO [c.c.s.ConfigurationServerImpl] (main:null) Processing updateSSLKeyStore
1181104 2017-05-29 19:56:20,261 INFO [c.c.s.ConfigurationServerImpl] (main:null) SSL keystore located at /etc/cloudstack/management/cloudmanagementserver.keystore
1181105 2017-05-29 19:56:20,268 DEBUG [c.c.u.s.Script] (main:null) Executing: sudo keytool -genkey -keystore /etc/cloudstack/management/cloudmanagementserver.keystore -storepass vmops.com -keypass vmops.com -keyalg RSA -validity 3650 -dn1181105 ame cn="Cloudstack User",ou="localhost",o="localhost",c="Unknown"
1181106 2017-05-29 19:56:21,511 DEBUG [c.c.u.s.Script] (main:null) Execution is successful.
1181107 2017-05-29 19:56:21,512 INFO [c.c.s.ConfigurationServerImpl] (main:null) Generated SSL keystore.
1181111 2017-05-29 19:56:21,552 TRACE [c.c.u.d.T.Statement] (main:null) Preparing: INSERT INTO configuration (configuration.instance, configuration.component, configuration.name, configuration.value, configuration.default_value, configur1181111 ation.description, configuration.category, configuration.is_dynamic, configuration.scope, configuration.updated) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
1181112 2017-05-29 19:56:21,556 TRACE [c.c.u.d.T.Transaction] (main:null) txn: DB Changes committed. Time = 6
1181113 2017-05-29 19:56:21,557 TRACE [c.c.u.d.T.Statement] (main:null) Closing: com.mysql.jdbc.JDBC4PreparedStatement@29f7a353: INSERT INTO configuration (configuration.instance, configuration.component, configuration.name, configuratio1181113 n.value, configuration.default_value, configuration.description, configuration.category, configuration.is_dynamic, configuration.scope, configuration.updated) VALUES (_binary'DEFAULT', _binary'management-server', _binary'ssl.keys1181113 tore', _binary'WKcZHRCDXhtqcLDK3ZSTyUYoZ3N81oiYbuool9WU2glPddGLrDZ4FyfQuThHXS4fH1fWPbKsjo9LgbvYUfFeAi4mdrfwOHXbCZPozlxPnBPp6gQh2eAKwB2eGytwZITxp0x9fqTv1bk0wWkOoEAEH37e6h+lWPvmv6eYvL85XGBcAJFVxRRHJwXl6u3r9dUobWF25lb6LLvi0P9kBpRFtb1181113 iv/p4DnCZgDaWrEwQCdtETwE6fMCVPxrUm/uEV4t0rbYApLU3ttpQolrR2USsBFlxFIxK4vYI9NeQK5dHK252s7pJriuQzwA14AyI36HLJZRkl/qVTu/ttIMqT9UcWvC5UZFC6iuv8E+eH5KFQhlx3HY/nvyQ8ADDgopmv0svmCBEHXP7mTN+K7JuPN8qBq9gaTNRNMAR33ma0r6lWD4K1+c7luKQS3hAMDMD1181113 KlkN09QTReNjydySuLnF6tlQQ1I1FPfa+eFE1VSj75ktd0nKgbxVnps79DceCFx428w6Jh9M7GRMvXykSHxI4SxrqXUeo4h1pX618r1kGLoG6SYPfuawRQqiC7ZouScyBK/z1egxR+xajXhMWxBtjfPxRx6fgwG4ouFT9RH6YhtFA+ppug+fhJvL2x9EZ78o6zDYpVmOOehLffHRdZHDtwtFg7srdmsNBHHg31181113 ukIYlF1MWvhnyUE0+nDu6HAgl4p+lt8lohxEFUoMa4bWaHVYOLMvgtqecdCU6CKwzGy3TiL/FI0/e1jUX5n9Y9FY5w7cjMnl2T95VKGXlnrBAyjo08Xt3MPM7xQduHM2kRYJsDXpJtokv+zWPmLUlPjTvnfLbobsBcNdrpU1ZPDPFXjPto5Ud3oPHzD1nu1DsK/Mfae6Lo8C0rl+I2kfCi0BZEKCJ0um8MzqZ1181113 0u32cbSXS/T7YU4jJNaiAujS3BDCb9L+u7irvhFTqe2PzDizV/YJa4y2YpQOpLv3m9JCsG0A1gvdZfOwDVUmg580TpomjPcmeH9bsOP8ESXa3d4Sm8tPHsetZrhnEtss0+OERTtNabFohAcjSKABIvcodq5HEqBzQtpyQN5MJA3YfarQMnPnAfIbjXszvqI04euBbQ2gyvDWoRIrcPjCjsUh9jRSUOMHaLVDg1181113 Cr4gK02+LhnGQeCXWjFo2i4FXCEajDgVpMHwYFhD1Uhdf3ttHBQNTaZ1ObTmP7E27H1VRpDvCP4omdFpxGV250ls5mj8uwMxrPchX7S7BlRbUT8XMyOm9RZiavRenOJp0xGWh+/YKKQFhDWTUyV3MiKah4iEPJ02fSpZ8pB+y4IYDO9ya+apLv8E4Bi90AZYOpWSYQr6t0wGcXpmY0JMyDoUxidh24HqO+/xJ1181113 6npD7SvEkoyFuXFAUaoamZ0vn/thxF5ZlTpA3Gfpny1j/4SLyV+GZWTK7wRhLjbZOonghXm2N6jpBYX/2dfdvoogTkEaxJHNitNyiZYwJm8fC7iaVmSB/OsJPs3Nd1ZmeljUCX6xsmQuiI47qDRmp1ZseQQM2HdGQqrK1X3EEcPa4BpgmVeOVRZ+uZ6gxAFYqQFAeKz6tUO6vQsOO/GAXwNwFSQDtBPUzC0ic1181113 P01cVkeUVlj56BaAp0Ia/tUbSDe5LhEdWo/C5innVeltJ//4Evhk3zHuyT1aiMJgWEwzZLgYuF1jHqK//ybSquXZJOyjrun+LVZs1vpp/qtRg3fk9qMjglnP0WIN44Kh/vnc7dDCrsyr/Ezu5Bh9oM+miNXVv5KH+2fMFYVzElB7Q0+kVmKkeucjtaBIlgtRjHFgK+uQzb+OZJSHByGx4Ci0jkDAxBNP4OsLf1181113 SC4pv8i8IppmcTwZGUbOye4x7QtvCo4JVzD5eZydzTD+GYFXgoeeFLveSm5DU44jeVjc3tKfcrhPeBc8pM3oGuSyvalYs7LGbf/9ozeg2vJWmQwEamPqHtq6ceg4qVRs2cIzaelc+yGT9BBOcUnHzCgr+gOC/F3lCJHbjUmyH52j4SLHZ3sZ6LB6CCVnkJ2Z7QYpCXe7lEIx3iX13+xBMSGwE/p4jKwJYBW0O1181113 k5dgTUG/QOEWhQVtdn33+uOH8X2/MMFMofJzaF7XC/mOCWtzSucTq1Obbng8ySAvp5ESIYfBtGeG3jgN+Pda7SzyDv8kQbyHFulyK+wzhYq/3lqOLnXP4P4nShHn6td7MunMJ6yPUX/swx5Pa35vpH/6IGxWHFMjJtzH/y8sliyy9SE3o4z6R3kVK+AWG7GvHRCdS9BNIJJyrjuEUuBcRlFpWPjEMw4n4ZSy81181113 +JJ0wlDpk8mlHflOLcuB72RAXyuHgsqi/8gIc5eM8Jf2968Av5WOeW0V4UHop0/wXOyXZf0bQisNzU0iBcRbszemlHPS3wCNmgkDIUgq2fkRZaLQO2hyurDzsEARQUE90XIKg7JWzkPXOMXZ6XR57INVL8kB3dPLkwzNOtC62YjbS8670SfWFjochnJSjlhAqFtVkuY8etyZ5mJRzrLl9b3MYlsmxHmdCCP5b1181113 JfC/Nq7ugKLq5RO/ACaz9Y934Q68K1lJYY2mHJHVnbeK9PewVfTjRG8A5NDhDBIduABoZi0DIAsz3lmEZGWG2V2GkIfGfha/fyvz9sasgO/Wrx4+Xpkt2aT+IU9f289ak9w1Bo/5DzdxmAZIHdonOUlb5TET5Eel114iW8mVO42snhKukWPMSayZVFPH9aCvdv3+5v7q5z8HPsnvSLpEiQ04LhVcGRHGV9+Qn1181113 EjcfFBTTIFjTFXb5d93+LPjoM6061kyqe60TqaTp1T0nxg4czhxTU1lpRkAFl88dyhJC4ZlJBsOp7tltzjyEa19Dw2nCySnyJ0dGVbZQ4vrIWkoQS4XxxbcWtP7gGLjmIL+bm3hEKSpotuiflAZlfbmqYILqHFzjkloYxSYjjJCoYzGaKCNCBT/ChGpDZamChdOVd2usCcSOqWZGa2dlQKp9kDkXQAMpefa8c1181113 YW0DRsgdqHLgXujEtHYbTrKxGhQGb4/yamDbZ+V4XvErxfAinlCUotrYUm7B71MSGgndXBTcq+/+oETKbaRdkx2qgbEqOZXUtv6uMrXIEvL3l+msb3VG1gz7d8cqWXs9VY/6HHCxNDKeldGMrGqTQPYznMzRiiCXyp+u+nOTf+gOzTfSl2BUm94jv/Hcjpj4zXnL1TV3KSjwBNRJFCIhh1sdqhvt0Xi9QJn2R1181113 QaLJ9y2TrBDGYEAmMgmXDVTa5CXBau0BwdiNiwXFvJkt7Vsvn7dRXTFyJ1YTO4c0H3n5B69QiWYaD3V7iRVmRzuQg8nSXTGh35Ol5Qh5lNI8byroC1gDMbxCjdJwR0fpUbn5fq4QFBgHDr9agFFf1kR2T4if+m4WFEGCP8bTwek5FL/K8CQLJ1JgCccWZy+2JZh2WQbVUvL7Bq5zcADUlM3ikHg4CrRZR8A3p1181113 4PM95HN/fu9+f2ZRj6ZF5TYDxk0weLJBVHbnph8mOLB0li5iPdUytNuKTCFJc1G1QUjB7fuV9g6l5mTG46wg0xU1mYJ9LB1oFix1lysnXBBaSYmnEzfiJbYqKqcvXJ+ycMPPAfmNDEyrzo0LTQtvyat7M2UGiAQ5cn0ahXEcZzSqmog/VE8xGJkBi1coP7Unp0lscuLgRxD7ev5Yru9iv6at0WSuvpZskVajq1181113 +B+R2XVEBMPFTjVynSr53fI46lhqr/YxzCEjVSuz0JDs5IBaahzT1U/wVKHbY6it2ZFNAwsvAG46CoYtJF06rYsRNu7Y+WbJC97xJciY2rpEq+Y2ZMo7Z20/hTSnBCg0IqUNxO+G/p9GxGRqRPA440Mcs5wz9ye7sYz0Z2uyZ5UUgLiNqL5lyw+dd9IxcPq5qaxJYRscsiL2T7IlYAJLrBn/9iz2gHVFEvZQF1181113 1QRDJWQ3ApqmaQrT2ZyZINY4pDOkNshRnBOjyEFosp3DEvdQ==', null, _binary'SSL Keystore for the management servers', _binary'Hidden', 0, null, null)
1181114 2017-05-29 19:56:21,557 TRACE [c.c.u.d.T.Connection] (main:null) Closing DB connection: dbconn2026396576
1181115 2017-05-29 19:56:21,558 TRACE [c.c.u.d.T.Connection] (main:null) Creating a DB connection with no txn: for 0: dbconn1496793545. Stack: -TransactionLegacy.prepareStatement:467-TransactionLegacy.prepareAutoCloseStatement:460-Gene1181115 ricDaoBase.findById:1022-GenericDaoBase.findByIdIncludingRemoved:987-GenericDaoBase.persist:1435-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:57-DelegatingMethodAccessorImpl.invoke:43-Method.invoke:606-AopU1181115 tils.invokeJoinpointUsingReflection:317-ReflectiveMethodInvocation.invokeJoinpoint:183-ReflectiveMethodInvocation.proceed:150
1181116 2017-05-29 19:56:21,559 TRACE [c.c.u.d.T.Statement] (main:null) Preparing: SELECT configuration.instance, configuration.component, configuration.name, configuration.value, configuration.default_value, configuration.description, c1181116 onfiguration.category, configuration.is_dynamic, configuration.scope, configuration.updated FROM configuration WHERE configuration.name = ?
1181117 2017-05-29 19:56:21,562 TRACE [c.c.u.d.T.Statement] (main:null) Closing: com.mysql.jdbc.JDBC4PreparedStatement@3ac020e1: SELECT configuration.instance, configuration.component, configuration.name, configuration.value, configurati1181117 on.default_value, configuration.description, configuration.category, configuration.is_dynamic, configuration.scope, configuration.updated FROM configuration WHERE configuration.name = _binary'ssl.keystore'
1181118 2017-05-29 19:56:21,562 TRACE [c.c.u.d.T.Connection] (main:null) Closing DB connection: dbconn1496793545
1181119 2017-05-29 19:56:21,562 TRACE [c.c.u.d.T.Transaction] (main:null) Transaction is done
1181120 2017-05-29 19:56:21,562 INFO [c.c.s.ConfigurationServerImpl] (main:null) Stored SSL keystore to database.
On 29/5/17, 8:12 pm, "Jason Kinsella" <ja...@cloudpeople.com.au> wrote:
Hi Rohit,
Here’s the telnet responses for all scenarios are the same: The connection is closed immediately.
We found these in the trace logs
1187417 2017-05-29 19:56:41,471 TRACE [c.c.u.n.NioConnection] (pool-3-thread-1:null) Keys Processing: 1
1187419 2017-05-29 19:56:41,476 TRACE [c.c.u.n.NioConnection] (pool-3-thread-1:null) Connection accepted for Socket[addr=/192.168.12.63,port=35673,localport=8250]
1187426 2017-05-29 19:56:41,498 TRACE [c.c.u.n.NioConnection] (pool-3-thread-1:null) Connection closed due to failure: Keystore was tampered with, or password was incorrect
1187427 2017-05-29 19:56:41,499 TRACE [c.c.u.n.NioConnection] (pool-3-thread-1:null) Keys Done Processing.
1187428 2017-05-29 19:56:41,499 TRACE [c.c.u.n.NioConnection] (pool-3-thread-1:null) Keys Processing: 0
1187429 2017-05-29 19:56:41,499 TRACE [c.c.u.n.NioConnection] (pool-3-thread-1:null) Keys Done Processing.
Is this referring to the cloudmanagementserver.keystore file and mysql.cloud.configuration ssl.keystore entry? These have been recreated many times and permissions appear correct.
Hopefully this helps.
Regards, Jason
On 29/5/17, 5:35 pm, "Rohit Yadav" <ro...@shapeblue.com> wrote:
Hi Jason,
- Ensure that the 'host' global setting is the IP of the mgmt server or the VIP/LB in front of the mgmt server(s)
- If you need to change the host global setting, restart mgmt server and destroy old SSVMs
- If you try telnet on mgmt server (host) IP, on port 8250, does telnet immediately stop:
telnet localhost 8250
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Try telnetting both from the mgmt server host (i.e. use localhost) and from the ssvm.
- If telnet exists immediately, that means mgmt server host is unable to accept client connections on port 8250 because of which SSL handshake fails leading to the exceptions in the logs. The error message "Failed to send server's CLOSE message due to socket channel's failure." suggests that mgmt server was not able to accept the connection. The possible reasons for such a behaviour can be: (a) there is an issue with keystore setup in the mgmt server (the server component NioServer handles accept(), during which SSL handshake is performed and it fails), i.e. the SSLContext is not properly initialized so any client ssl handshake fails; (b) iptables or network/firewall rules blocking connections to port 8250 on the mgmt server host, either flush the 'filter' table or add ALLOW rules for 0.0.0.0/0 on port 8250.
- In case the issue is with keystore setup, delete any *.keystore file from the mgmt server's classpath, i.e. search and remove any .keystore file from /etc/cloudstack/management and from /usr/share/cloudstack-management/, and restart the management server.
Regards.
________________________________
From: Jason Kinsella <ja...@cloudpeople.com.au>
Sent: 29 May 2017 05:23:09
To: users@cloudstack.apache.org
Subject: Re: SSVM NIO SSL Handshake error
UPDATE: I manually propagated a new systemvm.iso to secstorage and I can now ssh to the ssvm.
Unfortunately same handshake errors in logs.
On 28/5/17, 9:44 pm, "Jason Kinsella" <ja...@cloudpeople.com.au> wrote:
Hi Rohit,
I completed the test. The results are as follows:
The ssh.publickey and ssh.privatekey deletions in DB triggered the keys to be recreated, but not in the /var/lib/cloudstack/management/.ssh folder. They were recreated in /var/cloudstack/management/.ssh folder.
Here’s the log entry:
2017-05-28 20:53:28,508 DEBUG [c.c.u.s.Script] (localhost-startStop-1:null) (logid:) Executing: /bin/bash -c if [ -f /var/cloudstack/management/.ssh/id_rsa ]; then rm -f /var/cloudstack/management/.ssh/id_rsa; fi; ssh-keygen -t rsa -N '' -f /var/cloudstack/management/.ssh/id_rsa –q
I then destroyed the ssvm and after recreation could not ssh to it from management server due to key publickey error. I have always previously been able to ssh to the ssvm using the var/cloudstack/management/.ssh keys.
I checked the management server logs and found entry about id_rsa files differ – injecting into systemvm.iso
Hopefully this is helpful.
Regards,
Jason
On 28/5/17, 5:29 pm, "Rohit Yadav" <ro...@shapeblue.com> wrote:
Hi Jason,
In your test environment that uses the same db, can you try to do a workaround-experiment from [1]:
0) chmod +r and chown cloud:cloud relevant file and locations.
1) Stop Management Server
2) Delete SSH Keys in mysql Database: delete from configuration where name = "ssh.publickey" ; delete from configuration where name = "ssh.privatekey" ;
3) Delete the SSH Keys rm /var/lib/cloudstack/management/.ssh/id_rsa.pub rm /var/lib/cloudstack/management/.ssh/id_rsa
4) Start the Management Server - SSH Keys are generated and mysql entries inserted
[1] http://markmail.org/message/zfjyd7s22itg7t7q
Regards.
________________________________
From: Jason Kinsella <ja...@cloudpeople.com.au>
Sent: 27 May 2017 05:33:43
To: users@cloudstack.apache.org
Subject: Re: SSVM NIO SSL Handshake error
Files are linked here.
https://dl.dropboxusercontent.com/u/10588206/acs492/managmenet-server-logs.tar.gz
https://dl.dropboxusercontent.com/u/10588206/acs492/systemvm.tar.gz
Today we did a couple of additional tests that proved interesting. We’ve got a prod and a dev server. Both were upgraded last month. The prod has the error, but the dev is working. Everything was the same including CentOS 6.5.
We restored the dev DB into the fresh CentOS7 box and it displayed the same problem. This would suggest an OS issue. Therefore, the converse should work. We restored the prod DB into the dev server and it continues to exhibit the problem.
This suggests that we may have missed something in the migration between servers. Here’s steps:
Stop cloudstack-man service on broken box
Dump DB
Copy to new and restore
Copy db.properties & key files and update IP entry in db.properties
Update DB entry host to new IP
Delete DB ssl.keystore and keystore file
Destroy systemVMs in Vmware
Start cloudstack-man on new box
The /var/cloudstack/management/.ssh/ files are referenced when we ssh to ssvm from MS so they are correct. What about ssh.public and ssh.private in db.cloud.configuration table?
Regards,
Jason
On 25/5/17, 7:51 pm, "Rohit Yadav" <ro...@shapeblue.com> wrote:
Hi Jason,
Thanks for sharing the details. Yes, with the new setup please share with us the mgmt server logs and ssvm logs with TRACE enabled in the log4j configuration.
Regards.
________________________________
From: Jason Kinsella <ja...@cloudpeople.com.au>
Sent: 25 May 2017 12:49:50
To: users@cloudstack.apache.org
Subject: Re: SSVM NIO SSL Handshake error
Hi Rohit,
API login – fixed.
Latest systemvmtemplate (shapeblue new) in place – no improvement
No loadbalancer or known service on MS port 8250
I am doing my testing now on a fresh install of CentOS7 using shapeblue noredist with DB restored.
Hypervisor = vmware vsphere 6.5 with ESX 6.5
Systemvm.iso is dated today
All systemvms are exhibiting same behaviour.
Would any other logs help?
Regards,
Jason
On 25/5/17, 4:55 pm, "Rohit Yadav" <ro...@shapeblue.com> wrote:
Hi Jason,
The API login issue can be fixed by following this, which I believe you have already fixed: http://docs.cloudstack.apache.org/projects/cloudstack-administration/en/4.9/accounts.html#using-dynamic-roles
If not already in-use, can you try using the latest systemvmtemplate (for 4.6-4.9) from http://packages.shapeblue.com/systemvmtemplate/4.6/new.
Do you have a load-balancer on port 8250 on the management server(s), or any script/service that may be trying to perform a tcp-connect on mgmt server's port 8250?
When you upgrade can you make sure that both cloudstack-common and cloudstack-management packages are upgraded to 4.9.2.0? Also, what hypervisor(s) are you using?
The following error may hint that the jars on systemvms may not be updated, as one of the exception classes are missing:
2017-05-23 11:58:22,468 INFO [utils.exception.CSExceptionErrorCode] (main:null) Could not find exception: com.cloud.utils.exception.NioConnectionException in error code list for exceptions
Can you check that systemvm.iso are synced across hosts: (1) make sure cloudstack-common package is upgraded/updated to the same version as cloudstack-management (4.9.2.0), (2) if you're using vmware, delete this from the secondary storage, (3) for xenserver force reconnect on the host (from ui/api) or manually copy the scripts to xenserver host(s), (4) for kvm upgrade the cloudstack-common package.
Destroy all other systemvms and see if you can reproduce the issue?
Regards.
________________________________
From: Jason Kinsella <ja...@cloudpeople.com.au>
Sent: 25 May 2017 09:32:25
To: users@cloudstack.apache.org
Subject: Re: SSVM NIO SSL Handshake error
Also, just wanted to mention that the symptoms we have with systemvms not connecting is described in the mail-list
CS 4.9 NIO Selector wait time PR-1601 - https://www.mail-archive.com/dev@cloudstack.apache.org/msg69154.html
The only difference is that this thread refers to KVM hosts not connecting.
I’ve tried most suggestions in this thread.
On 25/5/17, 1:51 pm, "Jason Kinsella" <ja...@cloudpeople.com.au> wrote:
Java versions are as follows:
MS: 1.7.0_141
SSVM: 1.7.0_85
Deleted keystore files (again) and restarted MS, then recreated the SSVM.
Errors from SSVM:/var/log/cloud.log
2017-05-25 03:01:28,757 INFO [utils.nio.NioClient] (main:null) Connecting to 192.168.12.5:8250
2017-05-25 03:01:29,293 WARN [utils.nio.Link] (main:null) This SSL engine was forced to close inbound due to end of stream.
2017-05-25 03:01:29,293 ERROR [utils.nio.Link] (main:null) Failed to send server's CLOSE message due to socket channel's failure.
2017-05-25 03:01:29,294 ERROR [utils.nio.NioClient] (main:null) SSL Handshake failed while connecting to host: 192.168.12.5 port: 8250
2017-05-25 03:01:29,294 ERROR [utils.nio.NioConnection] (main:null) Unable to initialize the threads.
java.io.IOException: SSL Handshake failed while connecting to host: 192.168.12.5 port: 8250
at com.cloud.utils.nio.NioClient.init(NioClient.java:67)
at com.cloud.utils.nio.NioConnection.start(NioConnection.java:88)
at com.cloud.agent.Agent.start(Agent.java:228)
at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:399)
at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:367)
at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:351)
at com.cloud.agent.AgentShell.start(AgentShell.java:456)
at com.cloud.agent.AgentShell.main(AgentShell.java:491)
Same SSL engine forced to close inbound due to end of stream
On 25/5/17, 1:51 am, "Rajani Karuturi" <ra...@apache.org> wrote:
Can you check java version? Set the default java to 1.7 and delete keystore
files and restart MS
~Rajani
Sent from phone.
On 24 May 2017 9:15 p.m., "Jason Kinsella" <ja...@cloudpeople.com.au> wrote:
> I have now moved management server to a fresh CentOS7 server. But
> unfortunately I’m getting the exact same SSL handshake error. Back to
> square one.
>
> On 24/5/17, 11:40 pm, "Jason Kinsella" <ja...@cloudpeople.com.au> wrote:
>
> Hi All,
> Based on the feedback it seems like the issue is related to CentOS
> version, so I’ve built a new CentOS7 Management server using Blueshape
> noredist. I’ve restored the 4.9.2.0 DB into this server and
> management-server.logs look clean on boot. The only problem is that I can’t
> log into the webUI.
>
> The logs show a successful login (user = kinsja), but the the API
> command either is not allowed or doesn’t exist for the user. This means the
> UI doesn’t load.
>
> Anyone seen this with a restored DB?
>
> 2017-05-24 09:26:08,239 DEBUG [c.c.u.AccountManagerImpl]
> (catalina-exec-17:ctx-ee2c5e26) (logid:a8ca5ee5) User: kinsja in domain 1
> has successfully logged in
> 2017-05-24 09:26:08,246 INFO [c.c.a.ApiServer] (catalina-exec-17:ctx-ee2c5e26)
> (logid:a8ca5ee5) Current user logged in under timezone
> 2017-05-24 09:26:08,246 INFO [c.c.a.ApiServer] (catalina-exec-17:ctx-ee2c5e26)
> (logid:a8ca5ee5) Timezone offset from UTC is: 0.0
> 2017-05-24 09:26:08,251 DEBUG [c.c.a.ApiServlet] (catalina-exec-17:ctx-ee2c5e26)
> (logid:a8ca5ee5) ===END=== 192.168.10.38 -- POST
> 2017-05-24 09:26:08,320 DEBUG [c.c.a.ApiServlet] (catalina-exec-13:ctx-a1d38347)
> (logid:3404c663) ===START=== 192.168.10.38 -- GET
> command=listCapabilities&response=json&_=1495632368256
> 2017-05-24 09:26:08,325 DEBUG [c.c.a.ApiServer]
> (catalina-exec-13:ctx-a1d38347 ctx-960796a5) (logid:3404c663) The user with
> id:31 is not allowed to request the API command or the API command does not
> exist: listCapabilities
>
> Thanks
> Jason
>
> From: Jason Kinsella <ja...@cloudpeople.com.au>
> Date: Tuesday, 23 May 2017 at 10:11 pm
> To: "users@cloudstack.apache.org" <us...@cloudstack.apache.org>
> Subject: SSVM NIO SSL Handshake error
>
> Hi,
> We recently upgraded from 4.5.0 to 4.9.2.0 and encountered a problem
> with the SSVM and Console Proxy. They cannot connect to the management
> server. The SSVM cloud.log repeats this error every couple of seconds.
>
> 2017-05-23 11:58:22,461 INFO [utils.nio.NioClient] (main:null)
> Connecting to 192.168.12.1:8250
> 2017-05-23 11:58:22,465 WARN [utils.nio.Link] (main:null) This SSL
> engine was forced to close inbound due to end of stream.
> 2017-05-23 11:58:22,465 ERROR [utils.nio.Link] (main:null) Failed to
> send server's CLOSE message due to socket channel's failure.
> 2017-05-23 11:58:22,466 ERROR [utils.nio.NioClient] (main:null) SSL
> Handshake failed while connecting to host: 192.168.12.1 port: 8250
> 2017-05-23 11:58:22,466 ERROR [utils.nio.NioConnection] (main:null)
> Unable to initialize the threads.
> java.io.IOException: SSL Handshake failed while connecting to host:
> 192.168.12.1 port: 8250
> at com.cloud.utils.nio.NioClient.
> init(NioClient.java:67)
> at com.cloud.utils.nio.NioConnection.start(
> NioConnection.java:88)
> at com.cloud.agent.Agent.start(Agent.java:237)
> at com.cloud.agent.AgentShell.
> launchAgent(AgentShell.java:399)
> at com.cloud.agent.AgentShell.
> launchAgentFromClassInfo(AgentShell.java:367)
> at com.cloud.agent.AgentShell.
> launchAgent(AgentShell.java:351)
> at com.cloud.agent.AgentShell.
> start(AgentShell.java:456)
> at com.cloud.agent.AgentShell.
> main(AgentShell.java:491)
> 2017-05-23 11:58:22,468 INFO [utils.exception.CSExceptionErrorCode]
> (main:null) Could not find exception: com.cloud.utils.exception.NioConnectionException
> in error code list for exceptions
> 2017-05-23 11:58:22,468 WARN [cloud.agent.Agent] (main:null) NIO
> Connection Exception com.cloud.utils.exception.NioConnectionException:
> SSL Handshake failed while connecting to host: 192.168.12.1 port: 8250
>
> The setup is very simple. Single management server and ports are open.
>
> Things checked / tried:
>
> · Destroyed SSVM multiple times – still same problem.
>
> · SSH to SSVM from MS using ssh -i /var/cloudstack/management/.ssh/id_rsa
> -p 3922 root@IPADDRESS – PASS
>
> · SSVM telnet on 8250 to MS – PASS
>
> I’ve also tested a restore of the DB into our working development
> 4.9.2.0 server. It also exhibits the handshake errors, so most likely DB
> related.
>
> I’ve used up all my skills. Please help
>
> Regards,
> Jason
>
>
>
>
rohit.yadav@shapeblue.com
www.shapeblue.com<http://www.shapeblue.com>
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
rohit.yadav@shapeblue.com
www.shapeblue.com<http://www.shapeblue.com>
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
rohit.yadav@shapeblue.com
www.shapeblue.com<http://www.shapeblue.com>
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
rohit.yadav@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
Re: SSVM NIO SSL Handshake error
Posted by Jason Kinsella <ja...@cloudpeople.com.au>.
UPDATE: The mgmt. logs show that the cloudmanagementserver.keystore is being generated with password “vmops.com”. When we test it, the password is correct.
keytool -list -keystore cloudmanagementserver.keystore -storepass vmops.com
Keystore type: JKS
Keystore provider: SUN
Your keystore contains 1 entry
mykey, May 29, 2017, PrivateKeyEntry,
Certificate fingerprint (SHA1): 9C:91:BA:06:BB:4B:D1:B4:24:3F:8A:13:03:FD:6B:76:4E:9F:EA:29
LOGS
```1181097 2017-05-29 19:56:20,258 INFO [c.c.s.ConfigurationServerImpl] (main:null) Processing updateSSLKeyStore
1181104 2017-05-29 19:56:20,261 INFO [c.c.s.ConfigurationServerImpl] (main:null) SSL keystore located at /etc/cloudstack/management/cloudmanagementserver.keystore
1181105 2017-05-29 19:56:20,268 DEBUG [c.c.u.s.Script] (main:null) Executing: sudo keytool -genkey -keystore /etc/cloudstack/management/cloudmanagementserver.keystore -storepass vmops.com -keypass vmops.com -keyalg RSA -validity 3650 -dn1181105 ame cn="Cloudstack User",ou="localhost",o="localhost",c="Unknown"
1181106 2017-05-29 19:56:21,511 DEBUG [c.c.u.s.Script] (main:null) Execution is successful.
1181107 2017-05-29 19:56:21,512 INFO [c.c.s.ConfigurationServerImpl] (main:null) Generated SSL keystore.
1181111 2017-05-29 19:56:21,552 TRACE [c.c.u.d.T.Statement] (main:null) Preparing: INSERT INTO configuration (configuration.instance, configuration.component, configuration.name, configuration.value, configuration.default_value, configur1181111 ation.description, configuration.category, configuration.is_dynamic, configuration.scope, configuration.updated) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
1181112 2017-05-29 19:56:21,556 TRACE [c.c.u.d.T.Transaction] (main:null) txn: DB Changes committed. Time = 6
1181113 2017-05-29 19:56:21,557 TRACE [c.c.u.d.T.Statement] (main:null) Closing: com.mysql.jdbc.JDBC4PreparedStatement@29f7a353: INSERT INTO configuration (configuration.instance, configuration.component, configuration.name, configuratio1181113 n.value, configuration.default_value, configuration.description, configuration.category, configuration.is_dynamic, configuration.scope, configuration.updated) VALUES (_binary'DEFAULT', _binary'management-server', _binary'ssl.keys1181113 tore', _binary'WKcZHRCDXhtqcLDK3ZSTyUYoZ3N81oiYbuool9WU2glPddGLrDZ4FyfQuThHXS4fH1fWPbKsjo9LgbvYUfFeAi4mdrfwOHXbCZPozlxPnBPp6gQh2eAKwB2eGytwZITxp0x9fqTv1bk0wWkOoEAEH37e6h+lWPvmv6eYvL85XGBcAJFVxRRHJwXl6u3r9dUobWF25lb6LLvi0P9kBpRFtb1181113 iv/p4DnCZgDaWrEwQCdtETwE6fMCVPxrUm/uEV4t0rbYApLU3ttpQolrR2USsBFlxFIxK4vYI9NeQK5dHK252s7pJriuQzwA14AyI36HLJZRkl/qVTu/ttIMqT9UcWvC5UZFC6iuv8E+eH5KFQhlx3HY/nvyQ8ADDgopmv0svmCBEHXP7mTN+K7JuPN8qBq9gaTNRNMAR33ma0r6lWD4K1+c7luKQS3hAMDMD1181113 KlkN09QTReNjydySuLnF6tlQQ1I1FPfa+eFE1VSj75ktd0nKgbxVnps79DceCFx428w6Jh9M7GRMvXykSHxI4SxrqXUeo4h1pX618r1kGLoG6SYPfuawRQqiC7ZouScyBK/z1egxR+xajXhMWxBtjfPxRx6fgwG4ouFT9RH6YhtFA+ppug+fhJvL2x9EZ78o6zDYpVmOOehLffHRdZHDtwtFg7srdmsNBHHg31181113 ukIYlF1MWvhnyUE0+nDu6HAgl4p+lt8lohxEFUoMa4bWaHVYOLMvgtqecdCU6CKwzGy3TiL/FI0/e1jUX5n9Y9FY5w7cjMnl2T95VKGXlnrBAyjo08Xt3MPM7xQduHM2kRYJsDXpJtokv+zWPmLUlPjTvnfLbobsBcNdrpU1ZPDPFXjPto5Ud3oPHzD1nu1DsK/Mfae6Lo8C0rl+I2kfCi0BZEKCJ0um8MzqZ1181113 0u32cbSXS/T7YU4jJNaiAujS3BDCb9L+u7irvhFTqe2PzDizV/YJa4y2YpQOpLv3m9JCsG0A1gvdZfOwDVUmg580TpomjPcmeH9bsOP8ESXa3d4Sm8tPHsetZrhnEtss0+OERTtNabFohAcjSKABIvcodq5HEqBzQtpyQN5MJA3YfarQMnPnAfIbjXszvqI04euBbQ2gyvDWoRIrcPjCjsUh9jRSUOMHaLVDg1181113 Cr4gK02+LhnGQeCXWjFo2i4FXCEajDgVpMHwYFhD1Uhdf3ttHBQNTaZ1ObTmP7E27H1VRpDvCP4omdFpxGV250ls5mj8uwMxrPchX7S7BlRbUT8XMyOm9RZiavRenOJp0xGWh+/YKKQFhDWTUyV3MiKah4iEPJ02fSpZ8pB+y4IYDO9ya+apLv8E4Bi90AZYOpWSYQr6t0wGcXpmY0JMyDoUxidh24HqO+/xJ1181113 6npD7SvEkoyFuXFAUaoamZ0vn/thxF5ZlTpA3Gfpny1j/4SLyV+GZWTK7wRhLjbZOonghXm2N6jpBYX/2dfdvoogTkEaxJHNitNyiZYwJm8fC7iaVmSB/OsJPs3Nd1ZmeljUCX6xsmQuiI47qDRmp1ZseQQM2HdGQqrK1X3EEcPa4BpgmVeOVRZ+uZ6gxAFYqQFAeKz6tUO6vQsOO/GAXwNwFSQDtBPUzC0ic1181113 P01cVkeUVlj56BaAp0Ia/tUbSDe5LhEdWo/C5innVeltJ//4Evhk3zHuyT1aiMJgWEwzZLgYuF1jHqK//ybSquXZJOyjrun+LVZs1vpp/qtRg3fk9qMjglnP0WIN44Kh/vnc7dDCrsyr/Ezu5Bh9oM+miNXVv5KH+2fMFYVzElB7Q0+kVmKkeucjtaBIlgtRjHFgK+uQzb+OZJSHByGx4Ci0jkDAxBNP4OsLf1181113 SC4pv8i8IppmcTwZGUbOye4x7QtvCo4JVzD5eZydzTD+GYFXgoeeFLveSm5DU44jeVjc3tKfcrhPeBc8pM3oGuSyvalYs7LGbf/9ozeg2vJWmQwEamPqHtq6ceg4qVRs2cIzaelc+yGT9BBOcUnHzCgr+gOC/F3lCJHbjUmyH52j4SLHZ3sZ6LB6CCVnkJ2Z7QYpCXe7lEIx3iX13+xBMSGwE/p4jKwJYBW0O1181113 k5dgTUG/QOEWhQVtdn33+uOH8X2/MMFMofJzaF7XC/mOCWtzSucTq1Obbng8ySAvp5ESIYfBtGeG3jgN+Pda7SzyDv8kQbyHFulyK+wzhYq/3lqOLnXP4P4nShHn6td7MunMJ6yPUX/swx5Pa35vpH/6IGxWHFMjJtzH/y8sliyy9SE3o4z6R3kVK+AWG7GvHRCdS9BNIJJyrjuEUuBcRlFpWPjEMw4n4ZSy81181113 +JJ0wlDpk8mlHflOLcuB72RAXyuHgsqi/8gIc5eM8Jf2968Av5WOeW0V4UHop0/wXOyXZf0bQisNzU0iBcRbszemlHPS3wCNmgkDIUgq2fkRZaLQO2hyurDzsEARQUE90XIKg7JWzkPXOMXZ6XR57INVL8kB3dPLkwzNOtC62YjbS8670SfWFjochnJSjlhAqFtVkuY8etyZ5mJRzrLl9b3MYlsmxHmdCCP5b1181113 JfC/Nq7ugKLq5RO/ACaz9Y934Q68K1lJYY2mHJHVnbeK9PewVfTjRG8A5NDhDBIduABoZi0DIAsz3lmEZGWG2V2GkIfGfha/fyvz9sasgO/Wrx4+Xpkt2aT+IU9f289ak9w1Bo/5DzdxmAZIHdonOUlb5TET5Eel114iW8mVO42snhKukWPMSayZVFPH9aCvdv3+5v7q5z8HPsnvSLpEiQ04LhVcGRHGV9+Qn1181113 EjcfFBTTIFjTFXb5d93+LPjoM6061kyqe60TqaTp1T0nxg4czhxTU1lpRkAFl88dyhJC4ZlJBsOp7tltzjyEa19Dw2nCySnyJ0dGVbZQ4vrIWkoQS4XxxbcWtP7gGLjmIL+bm3hEKSpotuiflAZlfbmqYILqHFzjkloYxSYjjJCoYzGaKCNCBT/ChGpDZamChdOVd2usCcSOqWZGa2dlQKp9kDkXQAMpefa8c1181113 YW0DRsgdqHLgXujEtHYbTrKxGhQGb4/yamDbZ+V4XvErxfAinlCUotrYUm7B71MSGgndXBTcq+/+oETKbaRdkx2qgbEqOZXUtv6uMrXIEvL3l+msb3VG1gz7d8cqWXs9VY/6HHCxNDKeldGMrGqTQPYznMzRiiCXyp+u+nOTf+gOzTfSl2BUm94jv/Hcjpj4zXnL1TV3KSjwBNRJFCIhh1sdqhvt0Xi9QJn2R1181113 QaLJ9y2TrBDGYEAmMgmXDVTa5CXBau0BwdiNiwXFvJkt7Vsvn7dRXTFyJ1YTO4c0H3n5B69QiWYaD3V7iRVmRzuQg8nSXTGh35Ol5Qh5lNI8byroC1gDMbxCjdJwR0fpUbn5fq4QFBgHDr9agFFf1kR2T4if+m4WFEGCP8bTwek5FL/K8CQLJ1JgCccWZy+2JZh2WQbVUvL7Bq5zcADUlM3ikHg4CrRZR8A3p1181113 4PM95HN/fu9+f2ZRj6ZF5TYDxk0weLJBVHbnph8mOLB0li5iPdUytNuKTCFJc1G1QUjB7fuV9g6l5mTG46wg0xU1mYJ9LB1oFix1lysnXBBaSYmnEzfiJbYqKqcvXJ+ycMPPAfmNDEyrzo0LTQtvyat7M2UGiAQ5cn0ahXEcZzSqmog/VE8xGJkBi1coP7Unp0lscuLgRxD7ev5Yru9iv6at0WSuvpZskVajq1181113 +B+R2XVEBMPFTjVynSr53fI46lhqr/YxzCEjVSuz0JDs5IBaahzT1U/wVKHbY6it2ZFNAwsvAG46CoYtJF06rYsRNu7Y+WbJC97xJciY2rpEq+Y2ZMo7Z20/hTSnBCg0IqUNxO+G/p9GxGRqRPA440Mcs5wz9ye7sYz0Z2uyZ5UUgLiNqL5lyw+dd9IxcPq5qaxJYRscsiL2T7IlYAJLrBn/9iz2gHVFEvZQF1181113 1QRDJWQ3ApqmaQrT2ZyZINY4pDOkNshRnBOjyEFosp3DEvdQ==', null, _binary'SSL Keystore for the management servers', _binary'Hidden', 0, null, null)
1181114 2017-05-29 19:56:21,557 TRACE [c.c.u.d.T.Connection] (main:null) Closing DB connection: dbconn2026396576
1181115 2017-05-29 19:56:21,558 TRACE [c.c.u.d.T.Connection] (main:null) Creating a DB connection with no txn: for 0: dbconn1496793545. Stack: -TransactionLegacy.prepareStatement:467-TransactionLegacy.prepareAutoCloseStatement:460-Gene1181115 ricDaoBase.findById:1022-GenericDaoBase.findByIdIncludingRemoved:987-GenericDaoBase.persist:1435-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:57-DelegatingMethodAccessorImpl.invoke:43-Method.invoke:606-AopU1181115 tils.invokeJoinpointUsingReflection:317-ReflectiveMethodInvocation.invokeJoinpoint:183-ReflectiveMethodInvocation.proceed:150
1181116 2017-05-29 19:56:21,559 TRACE [c.c.u.d.T.Statement] (main:null) Preparing: SELECT configuration.instance, configuration.component, configuration.name, configuration.value, configuration.default_value, configuration.description, c1181116 onfiguration.category, configuration.is_dynamic, configuration.scope, configuration.updated FROM configuration WHERE configuration.name = ?
1181117 2017-05-29 19:56:21,562 TRACE [c.c.u.d.T.Statement] (main:null) Closing: com.mysql.jdbc.JDBC4PreparedStatement@3ac020e1: SELECT configuration.instance, configuration.component, configuration.name, configuration.value, configurati1181117 on.default_value, configuration.description, configuration.category, configuration.is_dynamic, configuration.scope, configuration.updated FROM configuration WHERE configuration.name = _binary'ssl.keystore'
1181118 2017-05-29 19:56:21,562 TRACE [c.c.u.d.T.Connection] (main:null) Closing DB connection: dbconn1496793545
1181119 2017-05-29 19:56:21,562 TRACE [c.c.u.d.T.Transaction] (main:null) Transaction is done
1181120 2017-05-29 19:56:21,562 INFO [c.c.s.ConfigurationServerImpl] (main:null) Stored SSL keystore to database.
On 29/5/17, 8:12 pm, "Jason Kinsella" <ja...@cloudpeople.com.au> wrote:
Hi Rohit,
Here’s the telnet responses for all scenarios are the same: The connection is closed immediately.
We found these in the trace logs
1187417 2017-05-29 19:56:41,471 TRACE [c.c.u.n.NioConnection] (pool-3-thread-1:null) Keys Processing: 1
1187419 2017-05-29 19:56:41,476 TRACE [c.c.u.n.NioConnection] (pool-3-thread-1:null) Connection accepted for Socket[addr=/192.168.12.63,port=35673,localport=8250]
1187426 2017-05-29 19:56:41,498 TRACE [c.c.u.n.NioConnection] (pool-3-thread-1:null) Connection closed due to failure: Keystore was tampered with, or password was incorrect
1187427 2017-05-29 19:56:41,499 TRACE [c.c.u.n.NioConnection] (pool-3-thread-1:null) Keys Done Processing.
1187428 2017-05-29 19:56:41,499 TRACE [c.c.u.n.NioConnection] (pool-3-thread-1:null) Keys Processing: 0
1187429 2017-05-29 19:56:41,499 TRACE [c.c.u.n.NioConnection] (pool-3-thread-1:null) Keys Done Processing.
Is this referring to the cloudmanagementserver.keystore file and mysql.cloud.configuration ssl.keystore entry? These have been recreated many times and permissions appear correct.
Hopefully this helps.
Regards, Jason
On 29/5/17, 5:35 pm, "Rohit Yadav" <ro...@shapeblue.com> wrote:
Hi Jason,
- Ensure that the 'host' global setting is the IP of the mgmt server or the VIP/LB in front of the mgmt server(s)
- If you need to change the host global setting, restart mgmt server and destroy old SSVMs
- If you try telnet on mgmt server (host) IP, on port 8250, does telnet immediately stop:
telnet localhost 8250
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Try telnetting both from the mgmt server host (i.e. use localhost) and from the ssvm.
- If telnet exists immediately, that means mgmt server host is unable to accept client connections on port 8250 because of which SSL handshake fails leading to the exceptions in the logs. The error message "Failed to send server's CLOSE message due to socket channel's failure." suggests that mgmt server was not able to accept the connection. The possible reasons for such a behaviour can be: (a) there is an issue with keystore setup in the mgmt server (the server component NioServer handles accept(), during which SSL handshake is performed and it fails), i.e. the SSLContext is not properly initialized so any client ssl handshake fails; (b) iptables or network/firewall rules blocking connections to port 8250 on the mgmt server host, either flush the 'filter' table or add ALLOW rules for 0.0.0.0/0 on port 8250.
- In case the issue is with keystore setup, delete any *.keystore file from the mgmt server's classpath, i.e. search and remove any .keystore file from /etc/cloudstack/management and from /usr/share/cloudstack-management/, and restart the management server.
Regards.
________________________________
From: Jason Kinsella <ja...@cloudpeople.com.au>
Sent: 29 May 2017 05:23:09
To: users@cloudstack.apache.org
Subject: Re: SSVM NIO SSL Handshake error
UPDATE: I manually propagated a new systemvm.iso to secstorage and I can now ssh to the ssvm.
Unfortunately same handshake errors in logs.
On 28/5/17, 9:44 pm, "Jason Kinsella" <ja...@cloudpeople.com.au> wrote:
Hi Rohit,
I completed the test. The results are as follows:
The ssh.publickey and ssh.privatekey deletions in DB triggered the keys to be recreated, but not in the /var/lib/cloudstack/management/.ssh folder. They were recreated in /var/cloudstack/management/.ssh folder.
Here’s the log entry:
2017-05-28 20:53:28,508 DEBUG [c.c.u.s.Script] (localhost-startStop-1:null) (logid:) Executing: /bin/bash -c if [ -f /var/cloudstack/management/.ssh/id_rsa ]; then rm -f /var/cloudstack/management/.ssh/id_rsa; fi; ssh-keygen -t rsa -N '' -f /var/cloudstack/management/.ssh/id_rsa –q
I then destroyed the ssvm and after recreation could not ssh to it from management server due to key publickey error. I have always previously been able to ssh to the ssvm using the var/cloudstack/management/.ssh keys.
I checked the management server logs and found entry about id_rsa files differ – injecting into systemvm.iso
Hopefully this is helpful.
Regards,
Jason
On 28/5/17, 5:29 pm, "Rohit Yadav" <ro...@shapeblue.com> wrote:
Hi Jason,
In your test environment that uses the same db, can you try to do a workaround-experiment from [1]:
0) chmod +r and chown cloud:cloud relevant file and locations.
1) Stop Management Server
2) Delete SSH Keys in mysql Database: delete from configuration where name = "ssh.publickey" ; delete from configuration where name = "ssh.privatekey" ;
3) Delete the SSH Keys rm /var/lib/cloudstack/management/.ssh/id_rsa.pub rm /var/lib/cloudstack/management/.ssh/id_rsa
4) Start the Management Server - SSH Keys are generated and mysql entries inserted
[1] http://markmail.org/message/zfjyd7s22itg7t7q
Regards.
________________________________
From: Jason Kinsella <ja...@cloudpeople.com.au>
Sent: 27 May 2017 05:33:43
To: users@cloudstack.apache.org
Subject: Re: SSVM NIO SSL Handshake error
Files are linked here.
https://dl.dropboxusercontent.com/u/10588206/acs492/managmenet-server-logs.tar.gz
https://dl.dropboxusercontent.com/u/10588206/acs492/systemvm.tar.gz
Today we did a couple of additional tests that proved interesting. We’ve got a prod and a dev server. Both were upgraded last month. The prod has the error, but the dev is working. Everything was the same including CentOS 6.5.
We restored the dev DB into the fresh CentOS7 box and it displayed the same problem. This would suggest an OS issue. Therefore, the converse should work. We restored the prod DB into the dev server and it continues to exhibit the problem.
This suggests that we may have missed something in the migration between servers. Here’s steps:
Stop cloudstack-man service on broken box
Dump DB
Copy to new and restore
Copy db.properties & key files and update IP entry in db.properties
Update DB entry host to new IP
Delete DB ssl.keystore and keystore file
Destroy systemVMs in Vmware
Start cloudstack-man on new box
The /var/cloudstack/management/.ssh/ files are referenced when we ssh to ssvm from MS so they are correct. What about ssh.public and ssh.private in db.cloud.configuration table?
Regards,
Jason
On 25/5/17, 7:51 pm, "Rohit Yadav" <ro...@shapeblue.com> wrote:
Hi Jason,
Thanks for sharing the details. Yes, with the new setup please share with us the mgmt server logs and ssvm logs with TRACE enabled in the log4j configuration.
Regards.
________________________________
From: Jason Kinsella <ja...@cloudpeople.com.au>
Sent: 25 May 2017 12:49:50
To: users@cloudstack.apache.org
Subject: Re: SSVM NIO SSL Handshake error
Hi Rohit,
API login – fixed.
Latest systemvmtemplate (shapeblue new) in place – no improvement
No loadbalancer or known service on MS port 8250
I am doing my testing now on a fresh install of CentOS7 using shapeblue noredist with DB restored.
Hypervisor = vmware vsphere 6.5 with ESX 6.5
Systemvm.iso is dated today
All systemvms are exhibiting same behaviour.
Would any other logs help?
Regards,
Jason
On 25/5/17, 4:55 pm, "Rohit Yadav" <ro...@shapeblue.com> wrote:
Hi Jason,
The API login issue can be fixed by following this, which I believe you have already fixed: http://docs.cloudstack.apache.org/projects/cloudstack-administration/en/4.9/accounts.html#using-dynamic-roles
If not already in-use, can you try using the latest systemvmtemplate (for 4.6-4.9) from http://packages.shapeblue.com/systemvmtemplate/4.6/new.
Do you have a load-balancer on port 8250 on the management server(s), or any script/service that may be trying to perform a tcp-connect on mgmt server's port 8250?
When you upgrade can you make sure that both cloudstack-common and cloudstack-management packages are upgraded to 4.9.2.0? Also, what hypervisor(s) are you using?
The following error may hint that the jars on systemvms may not be updated, as one of the exception classes are missing:
2017-05-23 11:58:22,468 INFO [utils.exception.CSExceptionErrorCode] (main:null) Could not find exception: com.cloud.utils.exception.NioConnectionException in error code list for exceptions
Can you check that systemvm.iso are synced across hosts: (1) make sure cloudstack-common package is upgraded/updated to the same version as cloudstack-management (4.9.2.0), (2) if you're using vmware, delete this from the secondary storage, (3) for xenserver force reconnect on the host (from ui/api) or manually copy the scripts to xenserver host(s), (4) for kvm upgrade the cloudstack-common package.
Destroy all other systemvms and see if you can reproduce the issue?
Regards.
________________________________
From: Jason Kinsella <ja...@cloudpeople.com.au>
Sent: 25 May 2017 09:32:25
To: users@cloudstack.apache.org
Subject: Re: SSVM NIO SSL Handshake error
Also, just wanted to mention that the symptoms we have with systemvms not connecting is described in the mail-list
CS 4.9 NIO Selector wait time PR-1601 - https://www.mail-archive.com/dev@cloudstack.apache.org/msg69154.html
The only difference is that this thread refers to KVM hosts not connecting.
I’ve tried most suggestions in this thread.
On 25/5/17, 1:51 pm, "Jason Kinsella" <ja...@cloudpeople.com.au> wrote:
Java versions are as follows:
MS: 1.7.0_141
SSVM: 1.7.0_85
Deleted keystore files (again) and restarted MS, then recreated the SSVM.
Errors from SSVM:/var/log/cloud.log
2017-05-25 03:01:28,757 INFO [utils.nio.NioClient] (main:null) Connecting to 192.168.12.5:8250
2017-05-25 03:01:29,293 WARN [utils.nio.Link] (main:null) This SSL engine was forced to close inbound due to end of stream.
2017-05-25 03:01:29,293 ERROR [utils.nio.Link] (main:null) Failed to send server's CLOSE message due to socket channel's failure.
2017-05-25 03:01:29,294 ERROR [utils.nio.NioClient] (main:null) SSL Handshake failed while connecting to host: 192.168.12.5 port: 8250
2017-05-25 03:01:29,294 ERROR [utils.nio.NioConnection] (main:null) Unable to initialize the threads.
java.io.IOException: SSL Handshake failed while connecting to host: 192.168.12.5 port: 8250
at com.cloud.utils.nio.NioClient.init(NioClient.java:67)
at com.cloud.utils.nio.NioConnection.start(NioConnection.java:88)
at com.cloud.agent.Agent.start(Agent.java:228)
at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:399)
at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:367)
at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:351)
at com.cloud.agent.AgentShell.start(AgentShell.java:456)
at com.cloud.agent.AgentShell.main(AgentShell.java:491)
Same SSL engine forced to close inbound due to end of stream
On 25/5/17, 1:51 am, "Rajani Karuturi" <ra...@apache.org> wrote:
Can you check java version? Set the default java to 1.7 and delete keystore
files and restart MS
~Rajani
Sent from phone.
On 24 May 2017 9:15 p.m., "Jason Kinsella" <ja...@cloudpeople.com.au> wrote:
> I have now moved management server to a fresh CentOS7 server. But
> unfortunately I’m getting the exact same SSL handshake error. Back to
> square one.
>
> On 24/5/17, 11:40 pm, "Jason Kinsella" <ja...@cloudpeople.com.au> wrote:
>
> Hi All,
> Based on the feedback it seems like the issue is related to CentOS
> version, so I’ve built a new CentOS7 Management server using Blueshape
> noredist. I’ve restored the 4.9.2.0 DB into this server and
> management-server.logs look clean on boot. The only problem is that I can’t
> log into the webUI.
>
> The logs show a successful login (user = kinsja), but the the API
> command either is not allowed or doesn’t exist for the user. This means the
> UI doesn’t load.
>
> Anyone seen this with a restored DB?
>
> 2017-05-24 09:26:08,239 DEBUG [c.c.u.AccountManagerImpl]
> (catalina-exec-17:ctx-ee2c5e26) (logid:a8ca5ee5) User: kinsja in domain 1
> has successfully logged in
> 2017-05-24 09:26:08,246 INFO [c.c.a.ApiServer] (catalina-exec-17:ctx-ee2c5e26)
> (logid:a8ca5ee5) Current user logged in under timezone
> 2017-05-24 09:26:08,246 INFO [c.c.a.ApiServer] (catalina-exec-17:ctx-ee2c5e26)
> (logid:a8ca5ee5) Timezone offset from UTC is: 0.0
> 2017-05-24 09:26:08,251 DEBUG [c.c.a.ApiServlet] (catalina-exec-17:ctx-ee2c5e26)
> (logid:a8ca5ee5) ===END=== 192.168.10.38 -- POST
> 2017-05-24 09:26:08,320 DEBUG [c.c.a.ApiServlet] (catalina-exec-13:ctx-a1d38347)
> (logid:3404c663) ===START=== 192.168.10.38 -- GET
> command=listCapabilities&response=json&_=1495632368256
> 2017-05-24 09:26:08,325 DEBUG [c.c.a.ApiServer]
> (catalina-exec-13:ctx-a1d38347 ctx-960796a5) (logid:3404c663) The user with
> id:31 is not allowed to request the API command or the API command does not
> exist: listCapabilities
>
> Thanks
> Jason
>
> From: Jason Kinsella <ja...@cloudpeople.com.au>
> Date: Tuesday, 23 May 2017 at 10:11 pm
> To: "users@cloudstack.apache.org" <us...@cloudstack.apache.org>
> Subject: SSVM NIO SSL Handshake error
>
> Hi,
> We recently upgraded from 4.5.0 to 4.9.2.0 and encountered a problem
> with the SSVM and Console Proxy. They cannot connect to the management
> server. The SSVM cloud.log repeats this error every couple of seconds.
>
> 2017-05-23 11:58:22,461 INFO [utils.nio.NioClient] (main:null)
> Connecting to 192.168.12.1:8250
> 2017-05-23 11:58:22,465 WARN [utils.nio.Link] (main:null) This SSL
> engine was forced to close inbound due to end of stream.
> 2017-05-23 11:58:22,465 ERROR [utils.nio.Link] (main:null) Failed to
> send server's CLOSE message due to socket channel's failure.
> 2017-05-23 11:58:22,466 ERROR [utils.nio.NioClient] (main:null) SSL
> Handshake failed while connecting to host: 192.168.12.1 port: 8250
> 2017-05-23 11:58:22,466 ERROR [utils.nio.NioConnection] (main:null)
> Unable to initialize the threads.
> java.io.IOException: SSL Handshake failed while connecting to host:
> 192.168.12.1 port: 8250
> at com.cloud.utils.nio.NioClient.
> init(NioClient.java:67)
> at com.cloud.utils.nio.NioConnection.start(
> NioConnection.java:88)
> at com.cloud.agent.Agent.start(Agent.java:237)
> at com.cloud.agent.AgentShell.
> launchAgent(AgentShell.java:399)
> at com.cloud.agent.AgentShell.
> launchAgentFromClassInfo(AgentShell.java:367)
> at com.cloud.agent.AgentShell.
> launchAgent(AgentShell.java:351)
> at com.cloud.agent.AgentShell.
> start(AgentShell.java:456)
> at com.cloud.agent.AgentShell.
> main(AgentShell.java:491)
> 2017-05-23 11:58:22,468 INFO [utils.exception.CSExceptionErrorCode]
> (main:null) Could not find exception: com.cloud.utils.exception.NioConnectionException
> in error code list for exceptions
> 2017-05-23 11:58:22,468 WARN [cloud.agent.Agent] (main:null) NIO
> Connection Exception com.cloud.utils.exception.NioConnectionException:
> SSL Handshake failed while connecting to host: 192.168.12.1 port: 8250
>
> The setup is very simple. Single management server and ports are open.
>
> Things checked / tried:
>
> · Destroyed SSVM multiple times – still same problem.
>
> · SSH to SSVM from MS using ssh -i /var/cloudstack/management/.ssh/id_rsa
> -p 3922 root@IPADDRESS – PASS
>
> · SSVM telnet on 8250 to MS – PASS
>
> I’ve also tested a restore of the DB into our working development
> 4.9.2.0 server. It also exhibits the handshake errors, so most likely DB
> related.
>
> I’ve used up all my skills. Please help
>
> Regards,
> Jason
>
>
>
>
rohit.yadav@shapeblue.com
www.shapeblue.com<http://www.shapeblue.com>
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
rohit.yadav@shapeblue.com
www.shapeblue.com<http://www.shapeblue.com>
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
rohit.yadav@shapeblue.com
www.shapeblue.com<http://www.shapeblue.com>
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
rohit.yadav@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
Re: SSVM NIO SSL Handshake error
Posted by Rohit Yadav <ro...@shapeblue.com>.
Jason,
Can you delete the keystore file and manually create it? Also check the file ownership and permissions.
This specific trace log suggests the issue is due to keystore file issue.
Regards.
________________________________
From: Jason Kinsella <ja...@cloudpeople.com.au>
Sent: 29 May 2017 15:42:59
To: users@cloudstack.apache.org
Subject: Re: SSVM NIO SSL Handshake error
Hi Rohit,
Here’s the telnet responses for all scenarios are the same: The connection is closed immediately.
We found these in the trace logs
1187417 2017-05-29 19:56:41,471 TRACE [c.c.u.n.NioConnection] (pool-3-thread-1:null) Keys Processing: 1
1187419 2017-05-29 19:56:41,476 TRACE [c.c.u.n.NioConnection] (pool-3-thread-1:null) Connection accepted for Socket[addr=/192.168.12.63,port=35673,localport=8250]
1187426 2017-05-29 19:56:41,498 TRACE [c.c.u.n.NioConnection] (pool-3-thread-1:null) Connection closed due to failure: Keystore was tampered with, or password was incorrect
1187427 2017-05-29 19:56:41,499 TRACE [c.c.u.n.NioConnection] (pool-3-thread-1:null) Keys Done Processing.
1187428 2017-05-29 19:56:41,499 TRACE [c.c.u.n.NioConnection] (pool-3-thread-1:null) Keys Processing: 0
1187429 2017-05-29 19:56:41,499 TRACE [c.c.u.n.NioConnection] (pool-3-thread-1:null) Keys Done Processing.
Is this referring to the cloudmanagementserver.keystore file and mysql.cloud.configuration ssl.keystore entry? These have been recreated many times and permissions appear correct.
Hopefully this helps.
Regards, Jason
On 29/5/17, 5:35 pm, "Rohit Yadav" <ro...@shapeblue.com> wrote:
Hi Jason,
- Ensure that the 'host' global setting is the IP of the mgmt server or the VIP/LB in front of the mgmt server(s)
- If you need to change the host global setting, restart mgmt server and destroy old SSVMs
- If you try telnet on mgmt server (host) IP, on port 8250, does telnet immediately stop:
telnet localhost 8250
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Try telnetting both from the mgmt server host (i.e. use localhost) and from the ssvm.
- If telnet exists immediately, that means mgmt server host is unable to accept client connections on port 8250 because of which SSL handshake fails leading to the exceptions in the logs. The error message "Failed to send server's CLOSE message due to socket channel's failure." suggests that mgmt server was not able to accept the connection. The possible reasons for such a behaviour can be: (a) there is an issue with keystore setup in the mgmt server (the server component NioServer handles accept(), during which SSL handshake is performed and it fails), i.e. the SSLContext is not properly initialized so any client ssl handshake fails; (b) iptables or network/firewall rules blocking connections to port 8250 on the mgmt server host, either flush the 'filter' table or add ALLOW rules for 0.0.0.0/0 on port 8250.
- In case the issue is with keystore setup, delete any *.keystore file from the mgmt server's classpath, i.e. search and remove any .keystore file from /etc/cloudstack/management and from /usr/share/cloudstack-management/, and restart the management server.
Regards.
________________________________
From: Jason Kinsella <ja...@cloudpeople.com.au>
Sent: 29 May 2017 05:23:09
To: users@cloudstack.apache.org
Subject: Re: SSVM NIO SSL Handshake error
UPDATE: I manually propagated a new systemvm.iso to secstorage and I can now ssh to the ssvm.
Unfortunately same handshake errors in logs.
On 28/5/17, 9:44 pm, "Jason Kinsella" <ja...@cloudpeople.com.au> wrote:
Hi Rohit,
I completed the test. The results are as follows:
The ssh.publickey and ssh.privatekey deletions in DB triggered the keys to be recreated, but not in the /var/lib/cloudstack/management/.ssh folder. They were recreated in /var/cloudstack/management/.ssh folder.
Here’s the log entry:
2017-05-28 20:53:28,508 DEBUG [c.c.u.s.Script] (localhost-startStop-1:null) (logid:) Executing: /bin/bash -c if [ -f /var/cloudstack/management/.ssh/id_rsa ]; then rm -f /var/cloudstack/management/.ssh/id_rsa; fi; ssh-keygen -t rsa -N '' -f /var/cloudstack/management/.ssh/id_rsa –q
I then destroyed the ssvm and after recreation could not ssh to it from management server due to key publickey error. I have always previously been able to ssh to the ssvm using the var/cloudstack/management/.ssh keys.
I checked the management server logs and found entry about id_rsa files differ – injecting into systemvm.iso
Hopefully this is helpful.
Regards,
Jason
On 28/5/17, 5:29 pm, "Rohit Yadav" <ro...@shapeblue.com> wrote:
Hi Jason,
In your test environment that uses the same db, can you try to do a workaround-experiment from [1]:
0) chmod +r and chown cloud:cloud relevant file and locations.
1) Stop Management Server
2) Delete SSH Keys in mysql Database: delete from configuration where name = "ssh.publickey" ; delete from configuration where name = "ssh.privatekey" ;
3) Delete the SSH Keys rm /var/lib/cloudstack/management/.ssh/id_rsa.pub rm /var/lib/cloudstack/management/.ssh/id_rsa
4) Start the Management Server - SSH Keys are generated and mysql entries inserted
[1] http://markmail.org/message/zfjyd7s22itg7t7q
Regards.
________________________________
From: Jason Kinsella <ja...@cloudpeople.com.au>
Sent: 27 May 2017 05:33:43
To: users@cloudstack.apache.org
Subject: Re: SSVM NIO SSL Handshake error
Files are linked here.
https://dl.dropboxusercontent.com/u/10588206/acs492/managmenet-server-logs.tar.gz
https://dl.dropboxusercontent.com/u/10588206/acs492/systemvm.tar.gz
Today we did a couple of additional tests that proved interesting. We’ve got a prod and a dev server. Both were upgraded last month. The prod has the error, but the dev is working. Everything was the same including CentOS 6.5.
We restored the dev DB into the fresh CentOS7 box and it displayed the same problem. This would suggest an OS issue. Therefore, the converse should work. We restored the prod DB into the dev server and it continues to exhibit the problem.
This suggests that we may have missed something in the migration between servers. Here’s steps:
Stop cloudstack-man service on broken box
Dump DB
Copy to new and restore
Copy db.properties & key files and update IP entry in db.properties
Update DB entry host to new IP
Delete DB ssl.keystore and keystore file
Destroy systemVMs in Vmware
Start cloudstack-man on new box
The /var/cloudstack/management/.ssh/ files are referenced when we ssh to ssvm from MS so they are correct. What about ssh.public and ssh.private in db.cloud.configuration table?
Regards,
Jason
On 25/5/17, 7:51 pm, "Rohit Yadav" <ro...@shapeblue.com> wrote:
Hi Jason,
Thanks for sharing the details. Yes, with the new setup please share with us the mgmt server logs and ssvm logs with TRACE enabled in the log4j configuration.
Regards.
________________________________
From: Jason Kinsella <ja...@cloudpeople.com.au>
Sent: 25 May 2017 12:49:50
To: users@cloudstack.apache.org
Subject: Re: SSVM NIO SSL Handshake error
Hi Rohit,
API login – fixed.
Latest systemvmtemplate (shapeblue new) in place – no improvement
No loadbalancer or known service on MS port 8250
I am doing my testing now on a fresh install of CentOS7 using shapeblue noredist with DB restored.
Hypervisor = vmware vsphere 6.5 with ESX 6.5
Systemvm.iso is dated today
All systemvms are exhibiting same behaviour.
Would any other logs help?
Regards,
Jason
On 25/5/17, 4:55 pm, "Rohit Yadav" <ro...@shapeblue.com> wrote:
Hi Jason,
The API login issue can be fixed by following this, which I believe you have already fixed: http://docs.cloudstack.apache.org/projects/cloudstack-administration/en/4.9/accounts.html#using-dynamic-roles
If not already in-use, can you try using the latest systemvmtemplate (for 4.6-4.9) from http://packages.shapeblue.com/systemvmtemplate/4.6/new.
Do you have a load-balancer on port 8250 on the management server(s), or any script/service that may be trying to perform a tcp-connect on mgmt server's port 8250?
When you upgrade can you make sure that both cloudstack-common and cloudstack-management packages are upgraded to 4.9.2.0? Also, what hypervisor(s) are you using?
The following error may hint that the jars on systemvms may not be updated, as one of the exception classes are missing:
2017-05-23 11:58:22,468 INFO [utils.exception.CSExceptionErrorCode] (main:null) Could not find exception: com.cloud.utils.exception.NioConnectionException in error code list for exceptions
Can you check that systemvm.iso are synced across hosts: (1) make sure cloudstack-common package is upgraded/updated to the same version as cloudstack-management (4.9.2.0), (2) if you're using vmware, delete this from the secondary storage, (3) for xenserver force reconnect on the host (from ui/api) or manually copy the scripts to xenserver host(s), (4) for kvm upgrade the cloudstack-common package.
Destroy all other systemvms and see if you can reproduce the issue?
Regards.
________________________________
From: Jason Kinsella <ja...@cloudpeople.com.au>
Sent: 25 May 2017 09:32:25
To: users@cloudstack.apache.org
Subject: Re: SSVM NIO SSL Handshake error
Also, just wanted to mention that the symptoms we have with systemvms not connecting is described in the mail-list
CS 4.9 NIO Selector wait time PR-1601 - https://www.mail-archive.com/dev@cloudstack.apache.org/msg69154.html
The only difference is that this thread refers to KVM hosts not connecting.
I’ve tried most suggestions in this thread.
On 25/5/17, 1:51 pm, "Jason Kinsella" <ja...@cloudpeople.com.au> wrote:
Java versions are as follows:
MS: 1.7.0_141
SSVM: 1.7.0_85
Deleted keystore files (again) and restarted MS, then recreated the SSVM.
Errors from SSVM:/var/log/cloud.log
2017-05-25 03:01:28,757 INFO [utils.nio.NioClient] (main:null) Connecting to 192.168.12.5:8250
2017-05-25 03:01:29,293 WARN [utils.nio.Link] (main:null) This SSL engine was forced to close inbound due to end of stream.
2017-05-25 03:01:29,293 ERROR [utils.nio.Link] (main:null) Failed to send server's CLOSE message due to socket channel's failure.
2017-05-25 03:01:29,294 ERROR [utils.nio.NioClient] (main:null) SSL Handshake failed while connecting to host: 192.168.12.5 port: 8250
2017-05-25 03:01:29,294 ERROR [utils.nio.NioConnection] (main:null) Unable to initialize the threads.
java.io.IOException: SSL Handshake failed while connecting to host: 192.168.12.5 port: 8250
at com.cloud.utils.nio.NioClient.init(NioClient.java:67)
at com.cloud.utils.nio.NioConnection.start(NioConnection.java:88)
at com.cloud.agent.Agent.start(Agent.java:228)
at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:399)
at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:367)
at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:351)
at com.cloud.agent.AgentShell.start(AgentShell.java:456)
at com.cloud.agent.AgentShell.main(AgentShell.java:491)
Same SSL engine forced to close inbound due to end of stream
On 25/5/17, 1:51 am, "Rajani Karuturi" <ra...@apache.org> wrote:
Can you check java version? Set the default java to 1.7 and delete keystore
files and restart MS
~Rajani
Sent from phone.
On 24 May 2017 9:15 p.m., "Jason Kinsella" <ja...@cloudpeople.com.au> wrote:
> I have now moved management server to a fresh CentOS7 server. But
> unfortunately I’m getting the exact same SSL handshake error. Back to
> square one.
>
> On 24/5/17, 11:40 pm, "Jason Kinsella" <ja...@cloudpeople.com.au> wrote:
>
> Hi All,
> Based on the feedback it seems like the issue is related to CentOS
> version, so I’ve built a new CentOS7 Management server using Blueshape
> noredist. I’ve restored the 4.9.2.0 DB into this server and
> management-server.logs look clean on boot. The only problem is that I can’t
> log into the webUI.
>
> The logs show a successful login (user = kinsja), but the the API
> command either is not allowed or doesn’t exist for the user. This means the
> UI doesn’t load.
>
> Anyone seen this with a restored DB?
>
> 2017-05-24 09:26:08,239 DEBUG [c.c.u.AccountManagerImpl]
> (catalina-exec-17:ctx-ee2c5e26) (logid:a8ca5ee5) User: kinsja in domain 1
> has successfully logged in
> 2017-05-24 09:26:08,246 INFO [c.c.a.ApiServer] (catalina-exec-17:ctx-ee2c5e26)
> (logid:a8ca5ee5) Current user logged in under timezone
> 2017-05-24 09:26:08,246 INFO [c.c.a.ApiServer] (catalina-exec-17:ctx-ee2c5e26)
> (logid:a8ca5ee5) Timezone offset from UTC is: 0.0
> 2017-05-24 09:26:08,251 DEBUG [c.c.a.ApiServlet] (catalina-exec-17:ctx-ee2c5e26)
> (logid:a8ca5ee5) ===END=== 192.168.10.38 -- POST
> 2017-05-24 09:26:08,320 DEBUG [c.c.a.ApiServlet] (catalina-exec-13:ctx-a1d38347)
> (logid:3404c663) ===START=== 192.168.10.38 -- GET
> command=listCapabilities&response=json&_=1495632368256
> 2017-05-24 09:26:08,325 DEBUG [c.c.a.ApiServer]
> (catalina-exec-13:ctx-a1d38347 ctx-960796a5) (logid:3404c663) The user with
> id:31 is not allowed to request the API command or the API command does not
> exist: listCapabilities
>
> Thanks
> Jason
>
> From: Jason Kinsella <ja...@cloudpeople.com.au>
> Date: Tuesday, 23 May 2017 at 10:11 pm
> To: "users@cloudstack.apache.org" <us...@cloudstack.apache.org>
> Subject: SSVM NIO SSL Handshake error
>
> Hi,
> We recently upgraded from 4.5.0 to 4.9.2.0 and encountered a problem
> with the SSVM and Console Proxy. They cannot connect to the management
> server. The SSVM cloud.log repeats this error every couple of seconds.
>
> 2017-05-23 11:58:22,461 INFO [utils.nio.NioClient] (main:null)
> Connecting to 192.168.12.1:8250
> 2017-05-23 11:58:22,465 WARN [utils.nio.Link] (main:null) This SSL
> engine was forced to close inbound due to end of stream.
> 2017-05-23 11:58:22,465 ERROR [utils.nio.Link] (main:null) Failed to
> send server's CLOSE message due to socket channel's failure.
> 2017-05-23 11:58:22,466 ERROR [utils.nio.NioClient] (main:null) SSL
> Handshake failed while connecting to host: 192.168.12.1 port: 8250
> 2017-05-23 11:58:22,466 ERROR [utils.nio.NioConnection] (main:null)
> Unable to initialize the threads.
> java.io.IOException: SSL Handshake failed while connecting to host:
> 192.168.12.1 port: 8250
> at com.cloud.utils.nio.NioClient.
> init(NioClient.java:67)
> at com.cloud.utils.nio.NioConnection.start(
> NioConnection.java:88)
> at com.cloud.agent.Agent.start(Agent.java:237)
> at com.cloud.agent.AgentShell.
> launchAgent(AgentShell.java:399)
> at com.cloud.agent.AgentShell.
> launchAgentFromClassInfo(AgentShell.java:367)
> at com.cloud.agent.AgentShell.
> launchAgent(AgentShell.java:351)
> at com.cloud.agent.AgentShell.
> start(AgentShell.java:456)
> at com.cloud.agent.AgentShell.
> main(AgentShell.java:491)
> 2017-05-23 11:58:22,468 INFO [utils.exception.CSExceptionErrorCode]
> (main:null) Could not find exception: com.cloud.utils.exception.NioConnectionException
> in error code list for exceptions
> 2017-05-23 11:58:22,468 WARN [cloud.agent.Agent] (main:null) NIO
> Connection Exception com.cloud.utils.exception.NioConnectionException:
> SSL Handshake failed while connecting to host: 192.168.12.1 port: 8250
>
> The setup is very simple. Single management server and ports are open.
>
> Things checked / tried:
>
> · Destroyed SSVM multiple times – still same problem.
>
> · SSH to SSVM from MS using ssh -i /var/cloudstack/management/.ssh/id_rsa
> -p 3922 root@IPADDRESS – PASS
>
> · SSVM telnet on 8250 to MS – PASS
>
> I’ve also tested a restore of the DB into our working development
> 4.9.2.0 server. It also exhibits the handshake errors, so most likely DB
> related.
>
> I’ve used up all my skills. Please help
>
> Regards,
> Jason
>
>
>
>
rohit.yadav@shapeblue.com
www.shapeblue.com<http://www.shapeblue.com>
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
rohit.yadav@shapeblue.com
www.shapeblue.com<http://www.shapeblue.com>
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
rohit.yadav@shapeblue.com
www.shapeblue.com<http://www.shapeblue.com>
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
rohit.yadav@shapeblue.com
www.shapeblue.com<http://www.shapeblue.com>
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
rohit.yadav@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
Re: SSVM NIO SSL Handshake error
Posted by Jason Kinsella <ja...@cloudpeople.com.au>.
Hi Rohit,
Here’s the telnet responses for all scenarios are the same: The connection is closed immediately.
We found these in the trace logs
1187417 2017-05-29 19:56:41,471 TRACE [c.c.u.n.NioConnection] (pool-3-thread-1:null) Keys Processing: 1
1187419 2017-05-29 19:56:41,476 TRACE [c.c.u.n.NioConnection] (pool-3-thread-1:null) Connection accepted for Socket[addr=/192.168.12.63,port=35673,localport=8250]
1187426 2017-05-29 19:56:41,498 TRACE [c.c.u.n.NioConnection] (pool-3-thread-1:null) Connection closed due to failure: Keystore was tampered with, or password was incorrect
1187427 2017-05-29 19:56:41,499 TRACE [c.c.u.n.NioConnection] (pool-3-thread-1:null) Keys Done Processing.
1187428 2017-05-29 19:56:41,499 TRACE [c.c.u.n.NioConnection] (pool-3-thread-1:null) Keys Processing: 0
1187429 2017-05-29 19:56:41,499 TRACE [c.c.u.n.NioConnection] (pool-3-thread-1:null) Keys Done Processing.
Is this referring to the cloudmanagementserver.keystore file and mysql.cloud.configuration ssl.keystore entry? These have been recreated many times and permissions appear correct.
Hopefully this helps.
Regards, Jason
On 29/5/17, 5:35 pm, "Rohit Yadav" <ro...@shapeblue.com> wrote:
Hi Jason,
- Ensure that the 'host' global setting is the IP of the mgmt server or the VIP/LB in front of the mgmt server(s)
- If you need to change the host global setting, restart mgmt server and destroy old SSVMs
- If you try telnet on mgmt server (host) IP, on port 8250, does telnet immediately stop:
telnet localhost 8250
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Try telnetting both from the mgmt server host (i.e. use localhost) and from the ssvm.
- If telnet exists immediately, that means mgmt server host is unable to accept client connections on port 8250 because of which SSL handshake fails leading to the exceptions in the logs. The error message "Failed to send server's CLOSE message due to socket channel's failure." suggests that mgmt server was not able to accept the connection. The possible reasons for such a behaviour can be: (a) there is an issue with keystore setup in the mgmt server (the server component NioServer handles accept(), during which SSL handshake is performed and it fails), i.e. the SSLContext is not properly initialized so any client ssl handshake fails; (b) iptables or network/firewall rules blocking connections to port 8250 on the mgmt server host, either flush the 'filter' table or add ALLOW rules for 0.0.0.0/0 on port 8250.
- In case the issue is with keystore setup, delete any *.keystore file from the mgmt server's classpath, i.e. search and remove any .keystore file from /etc/cloudstack/management and from /usr/share/cloudstack-management/, and restart the management server.
Regards.
________________________________
From: Jason Kinsella <ja...@cloudpeople.com.au>
Sent: 29 May 2017 05:23:09
To: users@cloudstack.apache.org
Subject: Re: SSVM NIO SSL Handshake error
UPDATE: I manually propagated a new systemvm.iso to secstorage and I can now ssh to the ssvm.
Unfortunately same handshake errors in logs.
On 28/5/17, 9:44 pm, "Jason Kinsella" <ja...@cloudpeople.com.au> wrote:
Hi Rohit,
I completed the test. The results are as follows:
The ssh.publickey and ssh.privatekey deletions in DB triggered the keys to be recreated, but not in the /var/lib/cloudstack/management/.ssh folder. They were recreated in /var/cloudstack/management/.ssh folder.
Here’s the log entry:
2017-05-28 20:53:28,508 DEBUG [c.c.u.s.Script] (localhost-startStop-1:null) (logid:) Executing: /bin/bash -c if [ -f /var/cloudstack/management/.ssh/id_rsa ]; then rm -f /var/cloudstack/management/.ssh/id_rsa; fi; ssh-keygen -t rsa -N '' -f /var/cloudstack/management/.ssh/id_rsa –q
I then destroyed the ssvm and after recreation could not ssh to it from management server due to key publickey error. I have always previously been able to ssh to the ssvm using the var/cloudstack/management/.ssh keys.
I checked the management server logs and found entry about id_rsa files differ – injecting into systemvm.iso
Hopefully this is helpful.
Regards,
Jason
On 28/5/17, 5:29 pm, "Rohit Yadav" <ro...@shapeblue.com> wrote:
Hi Jason,
In your test environment that uses the same db, can you try to do a workaround-experiment from [1]:
0) chmod +r and chown cloud:cloud relevant file and locations.
1) Stop Management Server
2) Delete SSH Keys in mysql Database: delete from configuration where name = "ssh.publickey" ; delete from configuration where name = "ssh.privatekey" ;
3) Delete the SSH Keys rm /var/lib/cloudstack/management/.ssh/id_rsa.pub rm /var/lib/cloudstack/management/.ssh/id_rsa
4) Start the Management Server - SSH Keys are generated and mysql entries inserted
[1] http://markmail.org/message/zfjyd7s22itg7t7q
Regards.
________________________________
From: Jason Kinsella <ja...@cloudpeople.com.au>
Sent: 27 May 2017 05:33:43
To: users@cloudstack.apache.org
Subject: Re: SSVM NIO SSL Handshake error
Files are linked here.
https://dl.dropboxusercontent.com/u/10588206/acs492/managmenet-server-logs.tar.gz
https://dl.dropboxusercontent.com/u/10588206/acs492/systemvm.tar.gz
Today we did a couple of additional tests that proved interesting. We’ve got a prod and a dev server. Both were upgraded last month. The prod has the error, but the dev is working. Everything was the same including CentOS 6.5.
We restored the dev DB into the fresh CentOS7 box and it displayed the same problem. This would suggest an OS issue. Therefore, the converse should work. We restored the prod DB into the dev server and it continues to exhibit the problem.
This suggests that we may have missed something in the migration between servers. Here’s steps:
Stop cloudstack-man service on broken box
Dump DB
Copy to new and restore
Copy db.properties & key files and update IP entry in db.properties
Update DB entry host to new IP
Delete DB ssl.keystore and keystore file
Destroy systemVMs in Vmware
Start cloudstack-man on new box
The /var/cloudstack/management/.ssh/ files are referenced when we ssh to ssvm from MS so they are correct. What about ssh.public and ssh.private in db.cloud.configuration table?
Regards,
Jason
On 25/5/17, 7:51 pm, "Rohit Yadav" <ro...@shapeblue.com> wrote:
Hi Jason,
Thanks for sharing the details. Yes, with the new setup please share with us the mgmt server logs and ssvm logs with TRACE enabled in the log4j configuration.
Regards.
________________________________
From: Jason Kinsella <ja...@cloudpeople.com.au>
Sent: 25 May 2017 12:49:50
To: users@cloudstack.apache.org
Subject: Re: SSVM NIO SSL Handshake error
Hi Rohit,
API login – fixed.
Latest systemvmtemplate (shapeblue new) in place – no improvement
No loadbalancer or known service on MS port 8250
I am doing my testing now on a fresh install of CentOS7 using shapeblue noredist with DB restored.
Hypervisor = vmware vsphere 6.5 with ESX 6.5
Systemvm.iso is dated today
All systemvms are exhibiting same behaviour.
Would any other logs help?
Regards,
Jason
On 25/5/17, 4:55 pm, "Rohit Yadav" <ro...@shapeblue.com> wrote:
Hi Jason,
The API login issue can be fixed by following this, which I believe you have already fixed: http://docs.cloudstack.apache.org/projects/cloudstack-administration/en/4.9/accounts.html#using-dynamic-roles
If not already in-use, can you try using the latest systemvmtemplate (for 4.6-4.9) from http://packages.shapeblue.com/systemvmtemplate/4.6/new.
Do you have a load-balancer on port 8250 on the management server(s), or any script/service that may be trying to perform a tcp-connect on mgmt server's port 8250?
When you upgrade can you make sure that both cloudstack-common and cloudstack-management packages are upgraded to 4.9.2.0? Also, what hypervisor(s) are you using?
The following error may hint that the jars on systemvms may not be updated, as one of the exception classes are missing:
2017-05-23 11:58:22,468 INFO [utils.exception.CSExceptionErrorCode] (main:null) Could not find exception: com.cloud.utils.exception.NioConnectionException in error code list for exceptions
Can you check that systemvm.iso are synced across hosts: (1) make sure cloudstack-common package is upgraded/updated to the same version as cloudstack-management (4.9.2.0), (2) if you're using vmware, delete this from the secondary storage, (3) for xenserver force reconnect on the host (from ui/api) or manually copy the scripts to xenserver host(s), (4) for kvm upgrade the cloudstack-common package.
Destroy all other systemvms and see if you can reproduce the issue?
Regards.
________________________________
From: Jason Kinsella <ja...@cloudpeople.com.au>
Sent: 25 May 2017 09:32:25
To: users@cloudstack.apache.org
Subject: Re: SSVM NIO SSL Handshake error
Also, just wanted to mention that the symptoms we have with systemvms not connecting is described in the mail-list
CS 4.9 NIO Selector wait time PR-1601 - https://www.mail-archive.com/dev@cloudstack.apache.org/msg69154.html
The only difference is that this thread refers to KVM hosts not connecting.
I’ve tried most suggestions in this thread.
On 25/5/17, 1:51 pm, "Jason Kinsella" <ja...@cloudpeople.com.au> wrote:
Java versions are as follows:
MS: 1.7.0_141
SSVM: 1.7.0_85
Deleted keystore files (again) and restarted MS, then recreated the SSVM.
Errors from SSVM:/var/log/cloud.log
2017-05-25 03:01:28,757 INFO [utils.nio.NioClient] (main:null) Connecting to 192.168.12.5:8250
2017-05-25 03:01:29,293 WARN [utils.nio.Link] (main:null) This SSL engine was forced to close inbound due to end of stream.
2017-05-25 03:01:29,293 ERROR [utils.nio.Link] (main:null) Failed to send server's CLOSE message due to socket channel's failure.
2017-05-25 03:01:29,294 ERROR [utils.nio.NioClient] (main:null) SSL Handshake failed while connecting to host: 192.168.12.5 port: 8250
2017-05-25 03:01:29,294 ERROR [utils.nio.NioConnection] (main:null) Unable to initialize the threads.
java.io.IOException: SSL Handshake failed while connecting to host: 192.168.12.5 port: 8250
at com.cloud.utils.nio.NioClient.init(NioClient.java:67)
at com.cloud.utils.nio.NioConnection.start(NioConnection.java:88)
at com.cloud.agent.Agent.start(Agent.java:228)
at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:399)
at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:367)
at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:351)
at com.cloud.agent.AgentShell.start(AgentShell.java:456)
at com.cloud.agent.AgentShell.main(AgentShell.java:491)
Same SSL engine forced to close inbound due to end of stream
On 25/5/17, 1:51 am, "Rajani Karuturi" <ra...@apache.org> wrote:
Can you check java version? Set the default java to 1.7 and delete keystore
files and restart MS
~Rajani
Sent from phone.
On 24 May 2017 9:15 p.m., "Jason Kinsella" <ja...@cloudpeople.com.au> wrote:
> I have now moved management server to a fresh CentOS7 server. But
> unfortunately I’m getting the exact same SSL handshake error. Back to
> square one.
>
> On 24/5/17, 11:40 pm, "Jason Kinsella" <ja...@cloudpeople.com.au> wrote:
>
> Hi All,
> Based on the feedback it seems like the issue is related to CentOS
> version, so I’ve built a new CentOS7 Management server using Blueshape
> noredist. I’ve restored the 4.9.2.0 DB into this server and
> management-server.logs look clean on boot. The only problem is that I can’t
> log into the webUI.
>
> The logs show a successful login (user = kinsja), but the the API
> command either is not allowed or doesn’t exist for the user. This means the
> UI doesn’t load.
>
> Anyone seen this with a restored DB?
>
> 2017-05-24 09:26:08,239 DEBUG [c.c.u.AccountManagerImpl]
> (catalina-exec-17:ctx-ee2c5e26) (logid:a8ca5ee5) User: kinsja in domain 1
> has successfully logged in
> 2017-05-24 09:26:08,246 INFO [c.c.a.ApiServer] (catalina-exec-17:ctx-ee2c5e26)
> (logid:a8ca5ee5) Current user logged in under timezone
> 2017-05-24 09:26:08,246 INFO [c.c.a.ApiServer] (catalina-exec-17:ctx-ee2c5e26)
> (logid:a8ca5ee5) Timezone offset from UTC is: 0.0
> 2017-05-24 09:26:08,251 DEBUG [c.c.a.ApiServlet] (catalina-exec-17:ctx-ee2c5e26)
> (logid:a8ca5ee5) ===END=== 192.168.10.38 -- POST
> 2017-05-24 09:26:08,320 DEBUG [c.c.a.ApiServlet] (catalina-exec-13:ctx-a1d38347)
> (logid:3404c663) ===START=== 192.168.10.38 -- GET
> command=listCapabilities&response=json&_=1495632368256
> 2017-05-24 09:26:08,325 DEBUG [c.c.a.ApiServer]
> (catalina-exec-13:ctx-a1d38347 ctx-960796a5) (logid:3404c663) The user with
> id:31 is not allowed to request the API command or the API command does not
> exist: listCapabilities
>
> Thanks
> Jason
>
> From: Jason Kinsella <ja...@cloudpeople.com.au>
> Date: Tuesday, 23 May 2017 at 10:11 pm
> To: "users@cloudstack.apache.org" <us...@cloudstack.apache.org>
> Subject: SSVM NIO SSL Handshake error
>
> Hi,
> We recently upgraded from 4.5.0 to 4.9.2.0 and encountered a problem
> with the SSVM and Console Proxy. They cannot connect to the management
> server. The SSVM cloud.log repeats this error every couple of seconds.
>
> 2017-05-23 11:58:22,461 INFO [utils.nio.NioClient] (main:null)
> Connecting to 192.168.12.1:8250
> 2017-05-23 11:58:22,465 WARN [utils.nio.Link] (main:null) This SSL
> engine was forced to close inbound due to end of stream.
> 2017-05-23 11:58:22,465 ERROR [utils.nio.Link] (main:null) Failed to
> send server's CLOSE message due to socket channel's failure.
> 2017-05-23 11:58:22,466 ERROR [utils.nio.NioClient] (main:null) SSL
> Handshake failed while connecting to host: 192.168.12.1 port: 8250
> 2017-05-23 11:58:22,466 ERROR [utils.nio.NioConnection] (main:null)
> Unable to initialize the threads.
> java.io.IOException: SSL Handshake failed while connecting to host:
> 192.168.12.1 port: 8250
> at com.cloud.utils.nio.NioClient.
> init(NioClient.java:67)
> at com.cloud.utils.nio.NioConnection.start(
> NioConnection.java:88)
> at com.cloud.agent.Agent.start(Agent.java:237)
> at com.cloud.agent.AgentShell.
> launchAgent(AgentShell.java:399)
> at com.cloud.agent.AgentShell.
> launchAgentFromClassInfo(AgentShell.java:367)
> at com.cloud.agent.AgentShell.
> launchAgent(AgentShell.java:351)
> at com.cloud.agent.AgentShell.
> start(AgentShell.java:456)
> at com.cloud.agent.AgentShell.
> main(AgentShell.java:491)
> 2017-05-23 11:58:22,468 INFO [utils.exception.CSExceptionErrorCode]
> (main:null) Could not find exception: com.cloud.utils.exception.NioConnectionException
> in error code list for exceptions
> 2017-05-23 11:58:22,468 WARN [cloud.agent.Agent] (main:null) NIO
> Connection Exception com.cloud.utils.exception.NioConnectionException:
> SSL Handshake failed while connecting to host: 192.168.12.1 port: 8250
>
> The setup is very simple. Single management server and ports are open.
>
> Things checked / tried:
>
> · Destroyed SSVM multiple times – still same problem.
>
> · SSH to SSVM from MS using ssh -i /var/cloudstack/management/.ssh/id_rsa
> -p 3922 root@IPADDRESS – PASS
>
> · SSVM telnet on 8250 to MS – PASS
>
> I’ve also tested a restore of the DB into our working development
> 4.9.2.0 server. It also exhibits the handshake errors, so most likely DB
> related.
>
> I’ve used up all my skills. Please help
>
> Regards,
> Jason
>
>
>
>
rohit.yadav@shapeblue.com
www.shapeblue.com<http://www.shapeblue.com>
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
rohit.yadav@shapeblue.com
www.shapeblue.com<http://www.shapeblue.com>
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
rohit.yadav@shapeblue.com
www.shapeblue.com<http://www.shapeblue.com>
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
rohit.yadav@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
Re: SSVM NIO SSL Handshake error
Posted by Rohit Yadav <ro...@shapeblue.com>.
Hi Jason,
- Ensure that the 'host' global setting is the IP of the mgmt server or the VIP/LB in front of the mgmt server(s)
- If you need to change the host global setting, restart mgmt server and destroy old SSVMs
- If you try telnet on mgmt server (host) IP, on port 8250, does telnet immediately stop:
telnet localhost 8250
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Try telnetting both from the mgmt server host (i.e. use localhost) and from the ssvm.
- If telnet exists immediately, that means mgmt server host is unable to accept client connections on port 8250 because of which SSL handshake fails leading to the exceptions in the logs. The error message "Failed to send server's CLOSE message due to socket channel's failure." suggests that mgmt server was not able to accept the connection. The possible reasons for such a behaviour can be: (a) there is an issue with keystore setup in the mgmt server (the server component NioServer handles accept(), during which SSL handshake is performed and it fails), i.e. the SSLContext is not properly initialized so any client ssl handshake fails; (b) iptables or network/firewall rules blocking connections to port 8250 on the mgmt server host, either flush the 'filter' table or add ALLOW rules for 0.0.0.0/0 on port 8250.
- In case the issue is with keystore setup, delete any *.keystore file from the mgmt server's classpath, i.e. search and remove any .keystore file from /etc/cloudstack/management and from /usr/share/cloudstack-management/, and restart the management server.
Regards.
________________________________
From: Jason Kinsella <ja...@cloudpeople.com.au>
Sent: 29 May 2017 05:23:09
To: users@cloudstack.apache.org
Subject: Re: SSVM NIO SSL Handshake error
UPDATE: I manually propagated a new systemvm.iso to secstorage and I can now ssh to the ssvm.
Unfortunately same handshake errors in logs.
On 28/5/17, 9:44 pm, "Jason Kinsella" <ja...@cloudpeople.com.au> wrote:
Hi Rohit,
I completed the test. The results are as follows:
The ssh.publickey and ssh.privatekey deletions in DB triggered the keys to be recreated, but not in the /var/lib/cloudstack/management/.ssh folder. They were recreated in /var/cloudstack/management/.ssh folder.
Here’s the log entry:
2017-05-28 20:53:28,508 DEBUG [c.c.u.s.Script] (localhost-startStop-1:null) (logid:) Executing: /bin/bash -c if [ -f /var/cloudstack/management/.ssh/id_rsa ]; then rm -f /var/cloudstack/management/.ssh/id_rsa; fi; ssh-keygen -t rsa -N '' -f /var/cloudstack/management/.ssh/id_rsa –q
I then destroyed the ssvm and after recreation could not ssh to it from management server due to key publickey error. I have always previously been able to ssh to the ssvm using the var/cloudstack/management/.ssh keys.
I checked the management server logs and found entry about id_rsa files differ – injecting into systemvm.iso
Hopefully this is helpful.
Regards,
Jason
On 28/5/17, 5:29 pm, "Rohit Yadav" <ro...@shapeblue.com> wrote:
Hi Jason,
In your test environment that uses the same db, can you try to do a workaround-experiment from [1]:
0) chmod +r and chown cloud:cloud relevant file and locations.
1) Stop Management Server
2) Delete SSH Keys in mysql Database: delete from configuration where name = "ssh.publickey" ; delete from configuration where name = "ssh.privatekey" ;
3) Delete the SSH Keys rm /var/lib/cloudstack/management/.ssh/id_rsa.pub rm /var/lib/cloudstack/management/.ssh/id_rsa
4) Start the Management Server - SSH Keys are generated and mysql entries inserted
[1] http://markmail.org/message/zfjyd7s22itg7t7q
Regards.
________________________________
From: Jason Kinsella <ja...@cloudpeople.com.au>
Sent: 27 May 2017 05:33:43
To: users@cloudstack.apache.org
Subject: Re: SSVM NIO SSL Handshake error
Files are linked here.
https://dl.dropboxusercontent.com/u/10588206/acs492/managmenet-server-logs.tar.gz
https://dl.dropboxusercontent.com/u/10588206/acs492/systemvm.tar.gz
Today we did a couple of additional tests that proved interesting. We’ve got a prod and a dev server. Both were upgraded last month. The prod has the error, but the dev is working. Everything was the same including CentOS 6.5.
We restored the dev DB into the fresh CentOS7 box and it displayed the same problem. This would suggest an OS issue. Therefore, the converse should work. We restored the prod DB into the dev server and it continues to exhibit the problem.
This suggests that we may have missed something in the migration between servers. Here’s steps:
Stop cloudstack-man service on broken box
Dump DB
Copy to new and restore
Copy db.properties & key files and update IP entry in db.properties
Update DB entry host to new IP
Delete DB ssl.keystore and keystore file
Destroy systemVMs in Vmware
Start cloudstack-man on new box
The /var/cloudstack/management/.ssh/ files are referenced when we ssh to ssvm from MS so they are correct. What about ssh.public and ssh.private in db.cloud.configuration table?
Regards,
Jason
On 25/5/17, 7:51 pm, "Rohit Yadav" <ro...@shapeblue.com> wrote:
Hi Jason,
Thanks for sharing the details. Yes, with the new setup please share with us the mgmt server logs and ssvm logs with TRACE enabled in the log4j configuration.
Regards.
________________________________
From: Jason Kinsella <ja...@cloudpeople.com.au>
Sent: 25 May 2017 12:49:50
To: users@cloudstack.apache.org
Subject: Re: SSVM NIO SSL Handshake error
Hi Rohit,
API login – fixed.
Latest systemvmtemplate (shapeblue new) in place – no improvement
No loadbalancer or known service on MS port 8250
I am doing my testing now on a fresh install of CentOS7 using shapeblue noredist with DB restored.
Hypervisor = vmware vsphere 6.5 with ESX 6.5
Systemvm.iso is dated today
All systemvms are exhibiting same behaviour.
Would any other logs help?
Regards,
Jason
On 25/5/17, 4:55 pm, "Rohit Yadav" <ro...@shapeblue.com> wrote:
Hi Jason,
The API login issue can be fixed by following this, which I believe you have already fixed: http://docs.cloudstack.apache.org/projects/cloudstack-administration/en/4.9/accounts.html#using-dynamic-roles
If not already in-use, can you try using the latest systemvmtemplate (for 4.6-4.9) from http://packages.shapeblue.com/systemvmtemplate/4.6/new.
Do you have a load-balancer on port 8250 on the management server(s), or any script/service that may be trying to perform a tcp-connect on mgmt server's port 8250?
When you upgrade can you make sure that both cloudstack-common and cloudstack-management packages are upgraded to 4.9.2.0? Also, what hypervisor(s) are you using?
The following error may hint that the jars on systemvms may not be updated, as one of the exception classes are missing:
2017-05-23 11:58:22,468 INFO [utils.exception.CSExceptionErrorCode] (main:null) Could not find exception: com.cloud.utils.exception.NioConnectionException in error code list for exceptions
Can you check that systemvm.iso are synced across hosts: (1) make sure cloudstack-common package is upgraded/updated to the same version as cloudstack-management (4.9.2.0), (2) if you're using vmware, delete this from the secondary storage, (3) for xenserver force reconnect on the host (from ui/api) or manually copy the scripts to xenserver host(s), (4) for kvm upgrade the cloudstack-common package.
Destroy all other systemvms and see if you can reproduce the issue?
Regards.
________________________________
From: Jason Kinsella <ja...@cloudpeople.com.au>
Sent: 25 May 2017 09:32:25
To: users@cloudstack.apache.org
Subject: Re: SSVM NIO SSL Handshake error
Also, just wanted to mention that the symptoms we have with systemvms not connecting is described in the mail-list
CS 4.9 NIO Selector wait time PR-1601 - https://www.mail-archive.com/dev@cloudstack.apache.org/msg69154.html
The only difference is that this thread refers to KVM hosts not connecting.
I’ve tried most suggestions in this thread.
On 25/5/17, 1:51 pm, "Jason Kinsella" <ja...@cloudpeople.com.au> wrote:
Java versions are as follows:
MS: 1.7.0_141
SSVM: 1.7.0_85
Deleted keystore files (again) and restarted MS, then recreated the SSVM.
Errors from SSVM:/var/log/cloud.log
2017-05-25 03:01:28,757 INFO [utils.nio.NioClient] (main:null) Connecting to 192.168.12.5:8250
2017-05-25 03:01:29,293 WARN [utils.nio.Link] (main:null) This SSL engine was forced to close inbound due to end of stream.
2017-05-25 03:01:29,293 ERROR [utils.nio.Link] (main:null) Failed to send server's CLOSE message due to socket channel's failure.
2017-05-25 03:01:29,294 ERROR [utils.nio.NioClient] (main:null) SSL Handshake failed while connecting to host: 192.168.12.5 port: 8250
2017-05-25 03:01:29,294 ERROR [utils.nio.NioConnection] (main:null) Unable to initialize the threads.
java.io.IOException: SSL Handshake failed while connecting to host: 192.168.12.5 port: 8250
at com.cloud.utils.nio.NioClient.init(NioClient.java:67)
at com.cloud.utils.nio.NioConnection.start(NioConnection.java:88)
at com.cloud.agent.Agent.start(Agent.java:228)
at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:399)
at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:367)
at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:351)
at com.cloud.agent.AgentShell.start(AgentShell.java:456)
at com.cloud.agent.AgentShell.main(AgentShell.java:491)
Same SSL engine forced to close inbound due to end of stream
On 25/5/17, 1:51 am, "Rajani Karuturi" <ra...@apache.org> wrote:
Can you check java version? Set the default java to 1.7 and delete keystore
files and restart MS
~Rajani
Sent from phone.
On 24 May 2017 9:15 p.m., "Jason Kinsella" <ja...@cloudpeople.com.au> wrote:
> I have now moved management server to a fresh CentOS7 server. But
> unfortunately I’m getting the exact same SSL handshake error. Back to
> square one.
>
> On 24/5/17, 11:40 pm, "Jason Kinsella" <ja...@cloudpeople.com.au> wrote:
>
> Hi All,
> Based on the feedback it seems like the issue is related to CentOS
> version, so I’ve built a new CentOS7 Management server using Blueshape
> noredist. I’ve restored the 4.9.2.0 DB into this server and
> management-server.logs look clean on boot. The only problem is that I can’t
> log into the webUI.
>
> The logs show a successful login (user = kinsja), but the the API
> command either is not allowed or doesn’t exist for the user. This means the
> UI doesn’t load.
>
> Anyone seen this with a restored DB?
>
> 2017-05-24 09:26:08,239 DEBUG [c.c.u.AccountManagerImpl]
> (catalina-exec-17:ctx-ee2c5e26) (logid:a8ca5ee5) User: kinsja in domain 1
> has successfully logged in
> 2017-05-24 09:26:08,246 INFO [c.c.a.ApiServer] (catalina-exec-17:ctx-ee2c5e26)
> (logid:a8ca5ee5) Current user logged in under timezone
> 2017-05-24 09:26:08,246 INFO [c.c.a.ApiServer] (catalina-exec-17:ctx-ee2c5e26)
> (logid:a8ca5ee5) Timezone offset from UTC is: 0.0
> 2017-05-24 09:26:08,251 DEBUG [c.c.a.ApiServlet] (catalina-exec-17:ctx-ee2c5e26)
> (logid:a8ca5ee5) ===END=== 192.168.10.38 -- POST
> 2017-05-24 09:26:08,320 DEBUG [c.c.a.ApiServlet] (catalina-exec-13:ctx-a1d38347)
> (logid:3404c663) ===START=== 192.168.10.38 -- GET
> command=listCapabilities&response=json&_=1495632368256
> 2017-05-24 09:26:08,325 DEBUG [c.c.a.ApiServer]
> (catalina-exec-13:ctx-a1d38347 ctx-960796a5) (logid:3404c663) The user with
> id:31 is not allowed to request the API command or the API command does not
> exist: listCapabilities
>
> Thanks
> Jason
>
> From: Jason Kinsella <ja...@cloudpeople.com.au>
> Date: Tuesday, 23 May 2017 at 10:11 pm
> To: "users@cloudstack.apache.org" <us...@cloudstack.apache.org>
> Subject: SSVM NIO SSL Handshake error
>
> Hi,
> We recently upgraded from 4.5.0 to 4.9.2.0 and encountered a problem
> with the SSVM and Console Proxy. They cannot connect to the management
> server. The SSVM cloud.log repeats this error every couple of seconds.
>
> 2017-05-23 11:58:22,461 INFO [utils.nio.NioClient] (main:null)
> Connecting to 192.168.12.1:8250
> 2017-05-23 11:58:22,465 WARN [utils.nio.Link] (main:null) This SSL
> engine was forced to close inbound due to end of stream.
> 2017-05-23 11:58:22,465 ERROR [utils.nio.Link] (main:null) Failed to
> send server's CLOSE message due to socket channel's failure.
> 2017-05-23 11:58:22,466 ERROR [utils.nio.NioClient] (main:null) SSL
> Handshake failed while connecting to host: 192.168.12.1 port: 8250
> 2017-05-23 11:58:22,466 ERROR [utils.nio.NioConnection] (main:null)
> Unable to initialize the threads.
> java.io.IOException: SSL Handshake failed while connecting to host:
> 192.168.12.1 port: 8250
> at com.cloud.utils.nio.NioClient.
> init(NioClient.java:67)
> at com.cloud.utils.nio.NioConnection.start(
> NioConnection.java:88)
> at com.cloud.agent.Agent.start(Agent.java:237)
> at com.cloud.agent.AgentShell.
> launchAgent(AgentShell.java:399)
> at com.cloud.agent.AgentShell.
> launchAgentFromClassInfo(AgentShell.java:367)
> at com.cloud.agent.AgentShell.
> launchAgent(AgentShell.java:351)
> at com.cloud.agent.AgentShell.
> start(AgentShell.java:456)
> at com.cloud.agent.AgentShell.
> main(AgentShell.java:491)
> 2017-05-23 11:58:22,468 INFO [utils.exception.CSExceptionErrorCode]
> (main:null) Could not find exception: com.cloud.utils.exception.NioConnectionException
> in error code list for exceptions
> 2017-05-23 11:58:22,468 WARN [cloud.agent.Agent] (main:null) NIO
> Connection Exception com.cloud.utils.exception.NioConnectionException:
> SSL Handshake failed while connecting to host: 192.168.12.1 port: 8250
>
> The setup is very simple. Single management server and ports are open.
>
> Things checked / tried:
>
> · Destroyed SSVM multiple times – still same problem.
>
> · SSH to SSVM from MS using ssh -i /var/cloudstack/management/.ssh/id_rsa
> -p 3922 root@IPADDRESS – PASS
>
> · SSVM telnet on 8250 to MS – PASS
>
> I’ve also tested a restore of the DB into our working development
> 4.9.2.0 server. It also exhibits the handshake errors, so most likely DB
> related.
>
> I’ve used up all my skills. Please help
>
> Regards,
> Jason
>
>
>
>
rohit.yadav@shapeblue.com
www.shapeblue.com<http://www.shapeblue.com>
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
rohit.yadav@shapeblue.com
www.shapeblue.com<http://www.shapeblue.com>
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
rohit.yadav@shapeblue.com
www.shapeblue.com<http://www.shapeblue.com>
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
rohit.yadav@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
Re: SSVM NIO SSL Handshake error
Posted by Jason Kinsella <ja...@cloudpeople.com.au>.
UPDATE: I manually propagated a new systemvm.iso to secstorage and I can now ssh to the ssvm.
Unfortunately same handshake errors in logs.
On 28/5/17, 9:44 pm, "Jason Kinsella" <ja...@cloudpeople.com.au> wrote:
Hi Rohit,
I completed the test. The results are as follows:
The ssh.publickey and ssh.privatekey deletions in DB triggered the keys to be recreated, but not in the /var/lib/cloudstack/management/.ssh folder. They were recreated in /var/cloudstack/management/.ssh folder.
Here’s the log entry:
2017-05-28 20:53:28,508 DEBUG [c.c.u.s.Script] (localhost-startStop-1:null) (logid:) Executing: /bin/bash -c if [ -f /var/cloudstack/management/.ssh/id_rsa ]; then rm -f /var/cloudstack/management/.ssh/id_rsa; fi; ssh-keygen -t rsa -N '' -f /var/cloudstack/management/.ssh/id_rsa –q
I then destroyed the ssvm and after recreation could not ssh to it from management server due to key publickey error. I have always previously been able to ssh to the ssvm using the var/cloudstack/management/.ssh keys.
I checked the management server logs and found entry about id_rsa files differ – injecting into systemvm.iso
Hopefully this is helpful.
Regards,
Jason
On 28/5/17, 5:29 pm, "Rohit Yadav" <ro...@shapeblue.com> wrote:
Hi Jason,
In your test environment that uses the same db, can you try to do a workaround-experiment from [1]:
0) chmod +r and chown cloud:cloud relevant file and locations.
1) Stop Management Server
2) Delete SSH Keys in mysql Database: delete from configuration where name = "ssh.publickey" ; delete from configuration where name = "ssh.privatekey" ;
3) Delete the SSH Keys rm /var/lib/cloudstack/management/.ssh/id_rsa.pub rm /var/lib/cloudstack/management/.ssh/id_rsa
4) Start the Management Server - SSH Keys are generated and mysql entries inserted
[1] http://markmail.org/message/zfjyd7s22itg7t7q
Regards.
________________________________
From: Jason Kinsella <ja...@cloudpeople.com.au>
Sent: 27 May 2017 05:33:43
To: users@cloudstack.apache.org
Subject: Re: SSVM NIO SSL Handshake error
Files are linked here.
https://dl.dropboxusercontent.com/u/10588206/acs492/managmenet-server-logs.tar.gz
https://dl.dropboxusercontent.com/u/10588206/acs492/systemvm.tar.gz
Today we did a couple of additional tests that proved interesting. We’ve got a prod and a dev server. Both were upgraded last month. The prod has the error, but the dev is working. Everything was the same including CentOS 6.5.
We restored the dev DB into the fresh CentOS7 box and it displayed the same problem. This would suggest an OS issue. Therefore, the converse should work. We restored the prod DB into the dev server and it continues to exhibit the problem.
This suggests that we may have missed something in the migration between servers. Here’s steps:
Stop cloudstack-man service on broken box
Dump DB
Copy to new and restore
Copy db.properties & key files and update IP entry in db.properties
Update DB entry host to new IP
Delete DB ssl.keystore and keystore file
Destroy systemVMs in Vmware
Start cloudstack-man on new box
The /var/cloudstack/management/.ssh/ files are referenced when we ssh to ssvm from MS so they are correct. What about ssh.public and ssh.private in db.cloud.configuration table?
Regards,
Jason
On 25/5/17, 7:51 pm, "Rohit Yadav" <ro...@shapeblue.com> wrote:
Hi Jason,
Thanks for sharing the details. Yes, with the new setup please share with us the mgmt server logs and ssvm logs with TRACE enabled in the log4j configuration.
Regards.
________________________________
From: Jason Kinsella <ja...@cloudpeople.com.au>
Sent: 25 May 2017 12:49:50
To: users@cloudstack.apache.org
Subject: Re: SSVM NIO SSL Handshake error
Hi Rohit,
API login – fixed.
Latest systemvmtemplate (shapeblue new) in place – no improvement
No loadbalancer or known service on MS port 8250
I am doing my testing now on a fresh install of CentOS7 using shapeblue noredist with DB restored.
Hypervisor = vmware vsphere 6.5 with ESX 6.5
Systemvm.iso is dated today
All systemvms are exhibiting same behaviour.
Would any other logs help?
Regards,
Jason
On 25/5/17, 4:55 pm, "Rohit Yadav" <ro...@shapeblue.com> wrote:
Hi Jason,
The API login issue can be fixed by following this, which I believe you have already fixed: http://docs.cloudstack.apache.org/projects/cloudstack-administration/en/4.9/accounts.html#using-dynamic-roles
If not already in-use, can you try using the latest systemvmtemplate (for 4.6-4.9) from http://packages.shapeblue.com/systemvmtemplate/4.6/new.
Do you have a load-balancer on port 8250 on the management server(s), or any script/service that may be trying to perform a tcp-connect on mgmt server's port 8250?
When you upgrade can you make sure that both cloudstack-common and cloudstack-management packages are upgraded to 4.9.2.0? Also, what hypervisor(s) are you using?
The following error may hint that the jars on systemvms may not be updated, as one of the exception classes are missing:
2017-05-23 11:58:22,468 INFO [utils.exception.CSExceptionErrorCode] (main:null) Could not find exception: com.cloud.utils.exception.NioConnectionException in error code list for exceptions
Can you check that systemvm.iso are synced across hosts: (1) make sure cloudstack-common package is upgraded/updated to the same version as cloudstack-management (4.9.2.0), (2) if you're using vmware, delete this from the secondary storage, (3) for xenserver force reconnect on the host (from ui/api) or manually copy the scripts to xenserver host(s), (4) for kvm upgrade the cloudstack-common package.
Destroy all other systemvms and see if you can reproduce the issue?
Regards.
________________________________
From: Jason Kinsella <ja...@cloudpeople.com.au>
Sent: 25 May 2017 09:32:25
To: users@cloudstack.apache.org
Subject: Re: SSVM NIO SSL Handshake error
Also, just wanted to mention that the symptoms we have with systemvms not connecting is described in the mail-list
CS 4.9 NIO Selector wait time PR-1601 - https://www.mail-archive.com/dev@cloudstack.apache.org/msg69154.html
The only difference is that this thread refers to KVM hosts not connecting.
I’ve tried most suggestions in this thread.
On 25/5/17, 1:51 pm, "Jason Kinsella" <ja...@cloudpeople.com.au> wrote:
Java versions are as follows:
MS: 1.7.0_141
SSVM: 1.7.0_85
Deleted keystore files (again) and restarted MS, then recreated the SSVM.
Errors from SSVM:/var/log/cloud.log
2017-05-25 03:01:28,757 INFO [utils.nio.NioClient] (main:null) Connecting to 192.168.12.5:8250
2017-05-25 03:01:29,293 WARN [utils.nio.Link] (main:null) This SSL engine was forced to close inbound due to end of stream.
2017-05-25 03:01:29,293 ERROR [utils.nio.Link] (main:null) Failed to send server's CLOSE message due to socket channel's failure.
2017-05-25 03:01:29,294 ERROR [utils.nio.NioClient] (main:null) SSL Handshake failed while connecting to host: 192.168.12.5 port: 8250
2017-05-25 03:01:29,294 ERROR [utils.nio.NioConnection] (main:null) Unable to initialize the threads.
java.io.IOException: SSL Handshake failed while connecting to host: 192.168.12.5 port: 8250
at com.cloud.utils.nio.NioClient.init(NioClient.java:67)
at com.cloud.utils.nio.NioConnection.start(NioConnection.java:88)
at com.cloud.agent.Agent.start(Agent.java:228)
at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:399)
at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:367)
at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:351)
at com.cloud.agent.AgentShell.start(AgentShell.java:456)
at com.cloud.agent.AgentShell.main(AgentShell.java:491)
Same SSL engine forced to close inbound due to end of stream
On 25/5/17, 1:51 am, "Rajani Karuturi" <ra...@apache.org> wrote:
Can you check java version? Set the default java to 1.7 and delete keystore
files and restart MS
~Rajani
Sent from phone.
On 24 May 2017 9:15 p.m., "Jason Kinsella" <ja...@cloudpeople.com.au> wrote:
> I have now moved management server to a fresh CentOS7 server. But
> unfortunately I’m getting the exact same SSL handshake error. Back to
> square one.
>
> On 24/5/17, 11:40 pm, "Jason Kinsella" <ja...@cloudpeople.com.au> wrote:
>
> Hi All,
> Based on the feedback it seems like the issue is related to CentOS
> version, so I’ve built a new CentOS7 Management server using Blueshape
> noredist. I’ve restored the 4.9.2.0 DB into this server and
> management-server.logs look clean on boot. The only problem is that I can’t
> log into the webUI.
>
> The logs show a successful login (user = kinsja), but the the API
> command either is not allowed or doesn’t exist for the user. This means the
> UI doesn’t load.
>
> Anyone seen this with a restored DB?
>
> 2017-05-24 09:26:08,239 DEBUG [c.c.u.AccountManagerImpl]
> (catalina-exec-17:ctx-ee2c5e26) (logid:a8ca5ee5) User: kinsja in domain 1
> has successfully logged in
> 2017-05-24 09:26:08,246 INFO [c.c.a.ApiServer] (catalina-exec-17:ctx-ee2c5e26)
> (logid:a8ca5ee5) Current user logged in under timezone
> 2017-05-24 09:26:08,246 INFO [c.c.a.ApiServer] (catalina-exec-17:ctx-ee2c5e26)
> (logid:a8ca5ee5) Timezone offset from UTC is: 0.0
> 2017-05-24 09:26:08,251 DEBUG [c.c.a.ApiServlet] (catalina-exec-17:ctx-ee2c5e26)
> (logid:a8ca5ee5) ===END=== 192.168.10.38 -- POST
> 2017-05-24 09:26:08,320 DEBUG [c.c.a.ApiServlet] (catalina-exec-13:ctx-a1d38347)
> (logid:3404c663) ===START=== 192.168.10.38 -- GET
> command=listCapabilities&response=json&_=1495632368256
> 2017-05-24 09:26:08,325 DEBUG [c.c.a.ApiServer]
> (catalina-exec-13:ctx-a1d38347 ctx-960796a5) (logid:3404c663) The user with
> id:31 is not allowed to request the API command or the API command does not
> exist: listCapabilities
>
> Thanks
> Jason
>
> From: Jason Kinsella <ja...@cloudpeople.com.au>
> Date: Tuesday, 23 May 2017 at 10:11 pm
> To: "users@cloudstack.apache.org" <us...@cloudstack.apache.org>
> Subject: SSVM NIO SSL Handshake error
>
> Hi,
> We recently upgraded from 4.5.0 to 4.9.2.0 and encountered a problem
> with the SSVM and Console Proxy. They cannot connect to the management
> server. The SSVM cloud.log repeats this error every couple of seconds.
>
> 2017-05-23 11:58:22,461 INFO [utils.nio.NioClient] (main:null)
> Connecting to 192.168.12.1:8250
> 2017-05-23 11:58:22,465 WARN [utils.nio.Link] (main:null) This SSL
> engine was forced to close inbound due to end of stream.
> 2017-05-23 11:58:22,465 ERROR [utils.nio.Link] (main:null) Failed to
> send server's CLOSE message due to socket channel's failure.
> 2017-05-23 11:58:22,466 ERROR [utils.nio.NioClient] (main:null) SSL
> Handshake failed while connecting to host: 192.168.12.1 port: 8250
> 2017-05-23 11:58:22,466 ERROR [utils.nio.NioConnection] (main:null)
> Unable to initialize the threads.
> java.io.IOException: SSL Handshake failed while connecting to host:
> 192.168.12.1 port: 8250
> at com.cloud.utils.nio.NioClient.
> init(NioClient.java:67)
> at com.cloud.utils.nio.NioConnection.start(
> NioConnection.java:88)
> at com.cloud.agent.Agent.start(Agent.java:237)
> at com.cloud.agent.AgentShell.
> launchAgent(AgentShell.java:399)
> at com.cloud.agent.AgentShell.
> launchAgentFromClassInfo(AgentShell.java:367)
> at com.cloud.agent.AgentShell.
> launchAgent(AgentShell.java:351)
> at com.cloud.agent.AgentShell.
> start(AgentShell.java:456)
> at com.cloud.agent.AgentShell.
> main(AgentShell.java:491)
> 2017-05-23 11:58:22,468 INFO [utils.exception.CSExceptionErrorCode]
> (main:null) Could not find exception: com.cloud.utils.exception.NioConnectionException
> in error code list for exceptions
> 2017-05-23 11:58:22,468 WARN [cloud.agent.Agent] (main:null) NIO
> Connection Exception com.cloud.utils.exception.NioConnectionException:
> SSL Handshake failed while connecting to host: 192.168.12.1 port: 8250
>
> The setup is very simple. Single management server and ports are open.
>
> Things checked / tried:
>
> · Destroyed SSVM multiple times – still same problem.
>
> · SSH to SSVM from MS using ssh -i /var/cloudstack/management/.ssh/id_rsa
> -p 3922 root@IPADDRESS – PASS
>
> · SSVM telnet on 8250 to MS – PASS
>
> I’ve also tested a restore of the DB into our working development
> 4.9.2.0 server. It also exhibits the handshake errors, so most likely DB
> related.
>
> I’ve used up all my skills. Please help
>
> Regards,
> Jason
>
>
>
>
rohit.yadav@shapeblue.com
www.shapeblue.com<http://www.shapeblue.com>
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
rohit.yadav@shapeblue.com
www.shapeblue.com<http://www.shapeblue.com>
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
rohit.yadav@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
Re: SSVM NIO SSL Handshake error
Posted by Jason Kinsella <ja...@cloudpeople.com.au>.
Hi Rohit,
I completed the test. The results are as follows:
The ssh.publickey and ssh.privatekey deletions in DB triggered the keys to be recreated, but not in the /var/lib/cloudstack/management/.ssh folder. They were recreated in /var/cloudstack/management/.ssh folder.
Here’s the log entry:
2017-05-28 20:53:28,508 DEBUG [c.c.u.s.Script] (localhost-startStop-1:null) (logid:) Executing: /bin/bash -c if [ -f /var/cloudstack/management/.ssh/id_rsa ]; then rm -f /var/cloudstack/management/.ssh/id_rsa; fi; ssh-keygen -t rsa -N '' -f /var/cloudstack/management/.ssh/id_rsa –q
I then destroyed the ssvm and after recreation could not ssh to it from management server due to key publickey error. I have always previously been able to ssh to the ssvm using the var/cloudstack/management/.ssh keys.
I checked the management server logs and found entry about id_rsa files differ – injecting into systemvm.iso
Hopefully this is helpful.
Regards,
Jason
On 28/5/17, 5:29 pm, "Rohit Yadav" <ro...@shapeblue.com> wrote:
Hi Jason,
In your test environment that uses the same db, can you try to do a workaround-experiment from [1]:
0) chmod +r and chown cloud:cloud relevant file and locations.
1) Stop Management Server
2) Delete SSH Keys in mysql Database: delete from configuration where name = "ssh.publickey" ; delete from configuration where name = "ssh.privatekey" ;
3) Delete the SSH Keys rm /var/lib/cloudstack/management/.ssh/id_rsa.pub rm /var/lib/cloudstack/management/.ssh/id_rsa
4) Start the Management Server - SSH Keys are generated and mysql entries inserted
[1] http://markmail.org/message/zfjyd7s22itg7t7q
Regards.
________________________________
From: Jason Kinsella <ja...@cloudpeople.com.au>
Sent: 27 May 2017 05:33:43
To: users@cloudstack.apache.org
Subject: Re: SSVM NIO SSL Handshake error
Files are linked here.
https://dl.dropboxusercontent.com/u/10588206/acs492/managmenet-server-logs.tar.gz
https://dl.dropboxusercontent.com/u/10588206/acs492/systemvm.tar.gz
Today we did a couple of additional tests that proved interesting. We’ve got a prod and a dev server. Both were upgraded last month. The prod has the error, but the dev is working. Everything was the same including CentOS 6.5.
We restored the dev DB into the fresh CentOS7 box and it displayed the same problem. This would suggest an OS issue. Therefore, the converse should work. We restored the prod DB into the dev server and it continues to exhibit the problem.
This suggests that we may have missed something in the migration between servers. Here’s steps:
Stop cloudstack-man service on broken box
Dump DB
Copy to new and restore
Copy db.properties & key files and update IP entry in db.properties
Update DB entry host to new IP
Delete DB ssl.keystore and keystore file
Destroy systemVMs in Vmware
Start cloudstack-man on new box
The /var/cloudstack/management/.ssh/ files are referenced when we ssh to ssvm from MS so they are correct. What about ssh.public and ssh.private in db.cloud.configuration table?
Regards,
Jason
On 25/5/17, 7:51 pm, "Rohit Yadav" <ro...@shapeblue.com> wrote:
Hi Jason,
Thanks for sharing the details. Yes, with the new setup please share with us the mgmt server logs and ssvm logs with TRACE enabled in the log4j configuration.
Regards.
________________________________
From: Jason Kinsella <ja...@cloudpeople.com.au>
Sent: 25 May 2017 12:49:50
To: users@cloudstack.apache.org
Subject: Re: SSVM NIO SSL Handshake error
Hi Rohit,
API login – fixed.
Latest systemvmtemplate (shapeblue new) in place – no improvement
No loadbalancer or known service on MS port 8250
I am doing my testing now on a fresh install of CentOS7 using shapeblue noredist with DB restored.
Hypervisor = vmware vsphere 6.5 with ESX 6.5
Systemvm.iso is dated today
All systemvms are exhibiting same behaviour.
Would any other logs help?
Regards,
Jason
On 25/5/17, 4:55 pm, "Rohit Yadav" <ro...@shapeblue.com> wrote:
Hi Jason,
The API login issue can be fixed by following this, which I believe you have already fixed: http://docs.cloudstack.apache.org/projects/cloudstack-administration/en/4.9/accounts.html#using-dynamic-roles
If not already in-use, can you try using the latest systemvmtemplate (for 4.6-4.9) from http://packages.shapeblue.com/systemvmtemplate/4.6/new.
Do you have a load-balancer on port 8250 on the management server(s), or any script/service that may be trying to perform a tcp-connect on mgmt server's port 8250?
When you upgrade can you make sure that both cloudstack-common and cloudstack-management packages are upgraded to 4.9.2.0? Also, what hypervisor(s) are you using?
The following error may hint that the jars on systemvms may not be updated, as one of the exception classes are missing:
2017-05-23 11:58:22,468 INFO [utils.exception.CSExceptionErrorCode] (main:null) Could not find exception: com.cloud.utils.exception.NioConnectionException in error code list for exceptions
Can you check that systemvm.iso are synced across hosts: (1) make sure cloudstack-common package is upgraded/updated to the same version as cloudstack-management (4.9.2.0), (2) if you're using vmware, delete this from the secondary storage, (3) for xenserver force reconnect on the host (from ui/api) or manually copy the scripts to xenserver host(s), (4) for kvm upgrade the cloudstack-common package.
Destroy all other systemvms and see if you can reproduce the issue?
Regards.
________________________________
From: Jason Kinsella <ja...@cloudpeople.com.au>
Sent: 25 May 2017 09:32:25
To: users@cloudstack.apache.org
Subject: Re: SSVM NIO SSL Handshake error
Also, just wanted to mention that the symptoms we have with systemvms not connecting is described in the mail-list
CS 4.9 NIO Selector wait time PR-1601 - https://www.mail-archive.com/dev@cloudstack.apache.org/msg69154.html
The only difference is that this thread refers to KVM hosts not connecting.
I’ve tried most suggestions in this thread.
On 25/5/17, 1:51 pm, "Jason Kinsella" <ja...@cloudpeople.com.au> wrote:
Java versions are as follows:
MS: 1.7.0_141
SSVM: 1.7.0_85
Deleted keystore files (again) and restarted MS, then recreated the SSVM.
Errors from SSVM:/var/log/cloud.log
2017-05-25 03:01:28,757 INFO [utils.nio.NioClient] (main:null) Connecting to 192.168.12.5:8250
2017-05-25 03:01:29,293 WARN [utils.nio.Link] (main:null) This SSL engine was forced to close inbound due to end of stream.
2017-05-25 03:01:29,293 ERROR [utils.nio.Link] (main:null) Failed to send server's CLOSE message due to socket channel's failure.
2017-05-25 03:01:29,294 ERROR [utils.nio.NioClient] (main:null) SSL Handshake failed while connecting to host: 192.168.12.5 port: 8250
2017-05-25 03:01:29,294 ERROR [utils.nio.NioConnection] (main:null) Unable to initialize the threads.
java.io.IOException: SSL Handshake failed while connecting to host: 192.168.12.5 port: 8250
at com.cloud.utils.nio.NioClient.init(NioClient.java:67)
at com.cloud.utils.nio.NioConnection.start(NioConnection.java:88)
at com.cloud.agent.Agent.start(Agent.java:228)
at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:399)
at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:367)
at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:351)
at com.cloud.agent.AgentShell.start(AgentShell.java:456)
at com.cloud.agent.AgentShell.main(AgentShell.java:491)
Same SSL engine forced to close inbound due to end of stream
On 25/5/17, 1:51 am, "Rajani Karuturi" <ra...@apache.org> wrote:
Can you check java version? Set the default java to 1.7 and delete keystore
files and restart MS
~Rajani
Sent from phone.
On 24 May 2017 9:15 p.m., "Jason Kinsella" <ja...@cloudpeople.com.au> wrote:
> I have now moved management server to a fresh CentOS7 server. But
> unfortunately I’m getting the exact same SSL handshake error. Back to
> square one.
>
> On 24/5/17, 11:40 pm, "Jason Kinsella" <ja...@cloudpeople.com.au> wrote:
>
> Hi All,
> Based on the feedback it seems like the issue is related to CentOS
> version, so I’ve built a new CentOS7 Management server using Blueshape
> noredist. I’ve restored the 4.9.2.0 DB into this server and
> management-server.logs look clean on boot. The only problem is that I can’t
> log into the webUI.
>
> The logs show a successful login (user = kinsja), but the the API
> command either is not allowed or doesn’t exist for the user. This means the
> UI doesn’t load.
>
> Anyone seen this with a restored DB?
>
> 2017-05-24 09:26:08,239 DEBUG [c.c.u.AccountManagerImpl]
> (catalina-exec-17:ctx-ee2c5e26) (logid:a8ca5ee5) User: kinsja in domain 1
> has successfully logged in
> 2017-05-24 09:26:08,246 INFO [c.c.a.ApiServer] (catalina-exec-17:ctx-ee2c5e26)
> (logid:a8ca5ee5) Current user logged in under timezone
> 2017-05-24 09:26:08,246 INFO [c.c.a.ApiServer] (catalina-exec-17:ctx-ee2c5e26)
> (logid:a8ca5ee5) Timezone offset from UTC is: 0.0
> 2017-05-24 09:26:08,251 DEBUG [c.c.a.ApiServlet] (catalina-exec-17:ctx-ee2c5e26)
> (logid:a8ca5ee5) ===END=== 192.168.10.38 -- POST
> 2017-05-24 09:26:08,320 DEBUG [c.c.a.ApiServlet] (catalina-exec-13:ctx-a1d38347)
> (logid:3404c663) ===START=== 192.168.10.38 -- GET
> command=listCapabilities&response=json&_=1495632368256
> 2017-05-24 09:26:08,325 DEBUG [c.c.a.ApiServer]
> (catalina-exec-13:ctx-a1d38347 ctx-960796a5) (logid:3404c663) The user with
> id:31 is not allowed to request the API command or the API command does not
> exist: listCapabilities
>
> Thanks
> Jason
>
> From: Jason Kinsella <ja...@cloudpeople.com.au>
> Date: Tuesday, 23 May 2017 at 10:11 pm
> To: "users@cloudstack.apache.org" <us...@cloudstack.apache.org>
> Subject: SSVM NIO SSL Handshake error
>
> Hi,
> We recently upgraded from 4.5.0 to 4.9.2.0 and encountered a problem
> with the SSVM and Console Proxy. They cannot connect to the management
> server. The SSVM cloud.log repeats this error every couple of seconds.
>
> 2017-05-23 11:58:22,461 INFO [utils.nio.NioClient] (main:null)
> Connecting to 192.168.12.1:8250
> 2017-05-23 11:58:22,465 WARN [utils.nio.Link] (main:null) This SSL
> engine was forced to close inbound due to end of stream.
> 2017-05-23 11:58:22,465 ERROR [utils.nio.Link] (main:null) Failed to
> send server's CLOSE message due to socket channel's failure.
> 2017-05-23 11:58:22,466 ERROR [utils.nio.NioClient] (main:null) SSL
> Handshake failed while connecting to host: 192.168.12.1 port: 8250
> 2017-05-23 11:58:22,466 ERROR [utils.nio.NioConnection] (main:null)
> Unable to initialize the threads.
> java.io.IOException: SSL Handshake failed while connecting to host:
> 192.168.12.1 port: 8250
> at com.cloud.utils.nio.NioClient.
> init(NioClient.java:67)
> at com.cloud.utils.nio.NioConnection.start(
> NioConnection.java:88)
> at com.cloud.agent.Agent.start(Agent.java:237)
> at com.cloud.agent.AgentShell.
> launchAgent(AgentShell.java:399)
> at com.cloud.agent.AgentShell.
> launchAgentFromClassInfo(AgentShell.java:367)
> at com.cloud.agent.AgentShell.
> launchAgent(AgentShell.java:351)
> at com.cloud.agent.AgentShell.
> start(AgentShell.java:456)
> at com.cloud.agent.AgentShell.
> main(AgentShell.java:491)
> 2017-05-23 11:58:22,468 INFO [utils.exception.CSExceptionErrorCode]
> (main:null) Could not find exception: com.cloud.utils.exception.NioConnectionException
> in error code list for exceptions
> 2017-05-23 11:58:22,468 WARN [cloud.agent.Agent] (main:null) NIO
> Connection Exception com.cloud.utils.exception.NioConnectionException:
> SSL Handshake failed while connecting to host: 192.168.12.1 port: 8250
>
> The setup is very simple. Single management server and ports are open.
>
> Things checked / tried:
>
> · Destroyed SSVM multiple times – still same problem.
>
> · SSH to SSVM from MS using ssh -i /var/cloudstack/management/.ssh/id_rsa
> -p 3922 root@IPADDRESS – PASS
>
> · SSVM telnet on 8250 to MS – PASS
>
> I’ve also tested a restore of the DB into our working development
> 4.9.2.0 server. It also exhibits the handshake errors, so most likely DB
> related.
>
> I’ve used up all my skills. Please help
>
> Regards,
> Jason
>
>
>
>
rohit.yadav@shapeblue.com
www.shapeblue.com<http://www.shapeblue.com>
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
rohit.yadav@shapeblue.com
www.shapeblue.com<http://www.shapeblue.com>
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
rohit.yadav@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
Re: SSVM NIO SSL Handshake error
Posted by Rohit Yadav <ro...@shapeblue.com>.
Hi Jason,
In your test environment that uses the same db, can you try to do a workaround-experiment from [1]:
0) chmod +r and chown cloud:cloud relevant file and locations.
1) Stop Management Server
2) Delete SSH Keys in mysql Database: delete from configuration where name = "ssh.publickey" ; delete from configuration where name = "ssh.privatekey" ;
3) Delete the SSH Keys rm /var/lib/cloudstack/management/.ssh/id_rsa.pub rm /var/lib/cloudstack/management/.ssh/id_rsa
4) Start the Management Server - SSH Keys are generated and mysql entries inserted
[1] http://markmail.org/message/zfjyd7s22itg7t7q
Regards.
________________________________
From: Jason Kinsella <ja...@cloudpeople.com.au>
Sent: 27 May 2017 05:33:43
To: users@cloudstack.apache.org
Subject: Re: SSVM NIO SSL Handshake error
Files are linked here.
https://dl.dropboxusercontent.com/u/10588206/acs492/managmenet-server-logs.tar.gz
https://dl.dropboxusercontent.com/u/10588206/acs492/systemvm.tar.gz
Today we did a couple of additional tests that proved interesting. We’ve got a prod and a dev server. Both were upgraded last month. The prod has the error, but the dev is working. Everything was the same including CentOS 6.5.
We restored the dev DB into the fresh CentOS7 box and it displayed the same problem. This would suggest an OS issue. Therefore, the converse should work. We restored the prod DB into the dev server and it continues to exhibit the problem.
This suggests that we may have missed something in the migration between servers. Here’s steps:
Stop cloudstack-man service on broken box
Dump DB
Copy to new and restore
Copy db.properties & key files and update IP entry in db.properties
Update DB entry host to new IP
Delete DB ssl.keystore and keystore file
Destroy systemVMs in Vmware
Start cloudstack-man on new box
The /var/cloudstack/management/.ssh/ files are referenced when we ssh to ssvm from MS so they are correct. What about ssh.public and ssh.private in db.cloud.configuration table?
Regards,
Jason
On 25/5/17, 7:51 pm, "Rohit Yadav" <ro...@shapeblue.com> wrote:
Hi Jason,
Thanks for sharing the details. Yes, with the new setup please share with us the mgmt server logs and ssvm logs with TRACE enabled in the log4j configuration.
Regards.
________________________________
From: Jason Kinsella <ja...@cloudpeople.com.au>
Sent: 25 May 2017 12:49:50
To: users@cloudstack.apache.org
Subject: Re: SSVM NIO SSL Handshake error
Hi Rohit,
API login – fixed.
Latest systemvmtemplate (shapeblue new) in place – no improvement
No loadbalancer or known service on MS port 8250
I am doing my testing now on a fresh install of CentOS7 using shapeblue noredist with DB restored.
Hypervisor = vmware vsphere 6.5 with ESX 6.5
Systemvm.iso is dated today
All systemvms are exhibiting same behaviour.
Would any other logs help?
Regards,
Jason
On 25/5/17, 4:55 pm, "Rohit Yadav" <ro...@shapeblue.com> wrote:
Hi Jason,
The API login issue can be fixed by following this, which I believe you have already fixed: http://docs.cloudstack.apache.org/projects/cloudstack-administration/en/4.9/accounts.html#using-dynamic-roles
If not already in-use, can you try using the latest systemvmtemplate (for 4.6-4.9) from http://packages.shapeblue.com/systemvmtemplate/4.6/new.
Do you have a load-balancer on port 8250 on the management server(s), or any script/service that may be trying to perform a tcp-connect on mgmt server's port 8250?
When you upgrade can you make sure that both cloudstack-common and cloudstack-management packages are upgraded to 4.9.2.0? Also, what hypervisor(s) are you using?
The following error may hint that the jars on systemvms may not be updated, as one of the exception classes are missing:
2017-05-23 11:58:22,468 INFO [utils.exception.CSExceptionErrorCode] (main:null) Could not find exception: com.cloud.utils.exception.NioConnectionException in error code list for exceptions
Can you check that systemvm.iso are synced across hosts: (1) make sure cloudstack-common package is upgraded/updated to the same version as cloudstack-management (4.9.2.0), (2) if you're using vmware, delete this from the secondary storage, (3) for xenserver force reconnect on the host (from ui/api) or manually copy the scripts to xenserver host(s), (4) for kvm upgrade the cloudstack-common package.
Destroy all other systemvms and see if you can reproduce the issue?
Regards.
________________________________
From: Jason Kinsella <ja...@cloudpeople.com.au>
Sent: 25 May 2017 09:32:25
To: users@cloudstack.apache.org
Subject: Re: SSVM NIO SSL Handshake error
Also, just wanted to mention that the symptoms we have with systemvms not connecting is described in the mail-list
CS 4.9 NIO Selector wait time PR-1601 - https://www.mail-archive.com/dev@cloudstack.apache.org/msg69154.html
The only difference is that this thread refers to KVM hosts not connecting.
I’ve tried most suggestions in this thread.
On 25/5/17, 1:51 pm, "Jason Kinsella" <ja...@cloudpeople.com.au> wrote:
Java versions are as follows:
MS: 1.7.0_141
SSVM: 1.7.0_85
Deleted keystore files (again) and restarted MS, then recreated the SSVM.
Errors from SSVM:/var/log/cloud.log
2017-05-25 03:01:28,757 INFO [utils.nio.NioClient] (main:null) Connecting to 192.168.12.5:8250
2017-05-25 03:01:29,293 WARN [utils.nio.Link] (main:null) This SSL engine was forced to close inbound due to end of stream.
2017-05-25 03:01:29,293 ERROR [utils.nio.Link] (main:null) Failed to send server's CLOSE message due to socket channel's failure.
2017-05-25 03:01:29,294 ERROR [utils.nio.NioClient] (main:null) SSL Handshake failed while connecting to host: 192.168.12.5 port: 8250
2017-05-25 03:01:29,294 ERROR [utils.nio.NioConnection] (main:null) Unable to initialize the threads.
java.io.IOException: SSL Handshake failed while connecting to host: 192.168.12.5 port: 8250
at com.cloud.utils.nio.NioClient.init(NioClient.java:67)
at com.cloud.utils.nio.NioConnection.start(NioConnection.java:88)
at com.cloud.agent.Agent.start(Agent.java:228)
at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:399)
at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:367)
at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:351)
at com.cloud.agent.AgentShell.start(AgentShell.java:456)
at com.cloud.agent.AgentShell.main(AgentShell.java:491)
Same SSL engine forced to close inbound due to end of stream
On 25/5/17, 1:51 am, "Rajani Karuturi" <ra...@apache.org> wrote:
Can you check java version? Set the default java to 1.7 and delete keystore
files and restart MS
~Rajani
Sent from phone.
On 24 May 2017 9:15 p.m., "Jason Kinsella" <ja...@cloudpeople.com.au> wrote:
> I have now moved management server to a fresh CentOS7 server. But
> unfortunately I’m getting the exact same SSL handshake error. Back to
> square one.
>
> On 24/5/17, 11:40 pm, "Jason Kinsella" <ja...@cloudpeople.com.au> wrote:
>
> Hi All,
> Based on the feedback it seems like the issue is related to CentOS
> version, so I’ve built a new CentOS7 Management server using Blueshape
> noredist. I’ve restored the 4.9.2.0 DB into this server and
> management-server.logs look clean on boot. The only problem is that I can’t
> log into the webUI.
>
> The logs show a successful login (user = kinsja), but the the API
> command either is not allowed or doesn’t exist for the user. This means the
> UI doesn’t load.
>
> Anyone seen this with a restored DB?
>
> 2017-05-24 09:26:08,239 DEBUG [c.c.u.AccountManagerImpl]
> (catalina-exec-17:ctx-ee2c5e26) (logid:a8ca5ee5) User: kinsja in domain 1
> has successfully logged in
> 2017-05-24 09:26:08,246 INFO [c.c.a.ApiServer] (catalina-exec-17:ctx-ee2c5e26)
> (logid:a8ca5ee5) Current user logged in under timezone
> 2017-05-24 09:26:08,246 INFO [c.c.a.ApiServer] (catalina-exec-17:ctx-ee2c5e26)
> (logid:a8ca5ee5) Timezone offset from UTC is: 0.0
> 2017-05-24 09:26:08,251 DEBUG [c.c.a.ApiServlet] (catalina-exec-17:ctx-ee2c5e26)
> (logid:a8ca5ee5) ===END=== 192.168.10.38 -- POST
> 2017-05-24 09:26:08,320 DEBUG [c.c.a.ApiServlet] (catalina-exec-13:ctx-a1d38347)
> (logid:3404c663) ===START=== 192.168.10.38 -- GET
> command=listCapabilities&response=json&_=1495632368256
> 2017-05-24 09:26:08,325 DEBUG [c.c.a.ApiServer]
> (catalina-exec-13:ctx-a1d38347 ctx-960796a5) (logid:3404c663) The user with
> id:31 is not allowed to request the API command or the API command does not
> exist: listCapabilities
>
> Thanks
> Jason
>
> From: Jason Kinsella <ja...@cloudpeople.com.au>
> Date: Tuesday, 23 May 2017 at 10:11 pm
> To: "users@cloudstack.apache.org" <us...@cloudstack.apache.org>
> Subject: SSVM NIO SSL Handshake error
>
> Hi,
> We recently upgraded from 4.5.0 to 4.9.2.0 and encountered a problem
> with the SSVM and Console Proxy. They cannot connect to the management
> server. The SSVM cloud.log repeats this error every couple of seconds.
>
> 2017-05-23 11:58:22,461 INFO [utils.nio.NioClient] (main:null)
> Connecting to 192.168.12.1:8250
> 2017-05-23 11:58:22,465 WARN [utils.nio.Link] (main:null) This SSL
> engine was forced to close inbound due to end of stream.
> 2017-05-23 11:58:22,465 ERROR [utils.nio.Link] (main:null) Failed to
> send server's CLOSE message due to socket channel's failure.
> 2017-05-23 11:58:22,466 ERROR [utils.nio.NioClient] (main:null) SSL
> Handshake failed while connecting to host: 192.168.12.1 port: 8250
> 2017-05-23 11:58:22,466 ERROR [utils.nio.NioConnection] (main:null)
> Unable to initialize the threads.
> java.io.IOException: SSL Handshake failed while connecting to host:
> 192.168.12.1 port: 8250
> at com.cloud.utils.nio.NioClient.
> init(NioClient.java:67)
> at com.cloud.utils.nio.NioConnection.start(
> NioConnection.java:88)
> at com.cloud.agent.Agent.start(Agent.java:237)
> at com.cloud.agent.AgentShell.
> launchAgent(AgentShell.java:399)
> at com.cloud.agent.AgentShell.
> launchAgentFromClassInfo(AgentShell.java:367)
> at com.cloud.agent.AgentShell.
> launchAgent(AgentShell.java:351)
> at com.cloud.agent.AgentShell.
> start(AgentShell.java:456)
> at com.cloud.agent.AgentShell.
> main(AgentShell.java:491)
> 2017-05-23 11:58:22,468 INFO [utils.exception.CSExceptionErrorCode]
> (main:null) Could not find exception: com.cloud.utils.exception.NioConnectionException
> in error code list for exceptions
> 2017-05-23 11:58:22,468 WARN [cloud.agent.Agent] (main:null) NIO
> Connection Exception com.cloud.utils.exception.NioConnectionException:
> SSL Handshake failed while connecting to host: 192.168.12.1 port: 8250
>
> The setup is very simple. Single management server and ports are open.
>
> Things checked / tried:
>
> · Destroyed SSVM multiple times – still same problem.
>
> · SSH to SSVM from MS using ssh -i /var/cloudstack/management/.ssh/id_rsa
> -p 3922 root@IPADDRESS – PASS
>
> · SSVM telnet on 8250 to MS – PASS
>
> I’ve also tested a restore of the DB into our working development
> 4.9.2.0 server. It also exhibits the handshake errors, so most likely DB
> related.
>
> I’ve used up all my skills. Please help
>
> Regards,
> Jason
>
>
>
>
rohit.yadav@shapeblue.com
www.shapeblue.com<http://www.shapeblue.com>
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
rohit.yadav@shapeblue.com
www.shapeblue.com<http://www.shapeblue.com>
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
rohit.yadav@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
Re: SSVM NIO SSL Handshake error
Posted by Jason Kinsella <ja...@cloudpeople.com.au>.
Files are linked here.
https://dl.dropboxusercontent.com/u/10588206/acs492/managmenet-server-logs.tar.gz
https://dl.dropboxusercontent.com/u/10588206/acs492/systemvm.tar.gz
Today we did a couple of additional tests that proved interesting. We’ve got a prod and a dev server. Both were upgraded last month. The prod has the error, but the dev is working. Everything was the same including CentOS 6.5.
We restored the dev DB into the fresh CentOS7 box and it displayed the same problem. This would suggest an OS issue. Therefore, the converse should work. We restored the prod DB into the dev server and it continues to exhibit the problem.
This suggests that we may have missed something in the migration between servers. Here’s steps:
Stop cloudstack-man service on broken box
Dump DB
Copy to new and restore
Copy db.properties & key files and update IP entry in db.properties
Update DB entry host to new IP
Delete DB ssl.keystore and keystore file
Destroy systemVMs in Vmware
Start cloudstack-man on new box
The /var/cloudstack/management/.ssh/ files are referenced when we ssh to ssvm from MS so they are correct. What about ssh.public and ssh.private in db.cloud.configuration table?
Regards,
Jason
On 25/5/17, 7:51 pm, "Rohit Yadav" <ro...@shapeblue.com> wrote:
Hi Jason,
Thanks for sharing the details. Yes, with the new setup please share with us the mgmt server logs and ssvm logs with TRACE enabled in the log4j configuration.
Regards.
________________________________
From: Jason Kinsella <ja...@cloudpeople.com.au>
Sent: 25 May 2017 12:49:50
To: users@cloudstack.apache.org
Subject: Re: SSVM NIO SSL Handshake error
Hi Rohit,
API login – fixed.
Latest systemvmtemplate (shapeblue new) in place – no improvement
No loadbalancer or known service on MS port 8250
I am doing my testing now on a fresh install of CentOS7 using shapeblue noredist with DB restored.
Hypervisor = vmware vsphere 6.5 with ESX 6.5
Systemvm.iso is dated today
All systemvms are exhibiting same behaviour.
Would any other logs help?
Regards,
Jason
On 25/5/17, 4:55 pm, "Rohit Yadav" <ro...@shapeblue.com> wrote:
Hi Jason,
The API login issue can be fixed by following this, which I believe you have already fixed: http://docs.cloudstack.apache.org/projects/cloudstack-administration/en/4.9/accounts.html#using-dynamic-roles
If not already in-use, can you try using the latest systemvmtemplate (for 4.6-4.9) from http://packages.shapeblue.com/systemvmtemplate/4.6/new.
Do you have a load-balancer on port 8250 on the management server(s), or any script/service that may be trying to perform a tcp-connect on mgmt server's port 8250?
When you upgrade can you make sure that both cloudstack-common and cloudstack-management packages are upgraded to 4.9.2.0? Also, what hypervisor(s) are you using?
The following error may hint that the jars on systemvms may not be updated, as one of the exception classes are missing:
2017-05-23 11:58:22,468 INFO [utils.exception.CSExceptionErrorCode] (main:null) Could not find exception: com.cloud.utils.exception.NioConnectionException in error code list for exceptions
Can you check that systemvm.iso are synced across hosts: (1) make sure cloudstack-common package is upgraded/updated to the same version as cloudstack-management (4.9.2.0), (2) if you're using vmware, delete this from the secondary storage, (3) for xenserver force reconnect on the host (from ui/api) or manually copy the scripts to xenserver host(s), (4) for kvm upgrade the cloudstack-common package.
Destroy all other systemvms and see if you can reproduce the issue?
Regards.
________________________________
From: Jason Kinsella <ja...@cloudpeople.com.au>
Sent: 25 May 2017 09:32:25
To: users@cloudstack.apache.org
Subject: Re: SSVM NIO SSL Handshake error
Also, just wanted to mention that the symptoms we have with systemvms not connecting is described in the mail-list
CS 4.9 NIO Selector wait time PR-1601 - https://www.mail-archive.com/dev@cloudstack.apache.org/msg69154.html
The only difference is that this thread refers to KVM hosts not connecting.
I’ve tried most suggestions in this thread.
On 25/5/17, 1:51 pm, "Jason Kinsella" <ja...@cloudpeople.com.au> wrote:
Java versions are as follows:
MS: 1.7.0_141
SSVM: 1.7.0_85
Deleted keystore files (again) and restarted MS, then recreated the SSVM.
Errors from SSVM:/var/log/cloud.log
2017-05-25 03:01:28,757 INFO [utils.nio.NioClient] (main:null) Connecting to 192.168.12.5:8250
2017-05-25 03:01:29,293 WARN [utils.nio.Link] (main:null) This SSL engine was forced to close inbound due to end of stream.
2017-05-25 03:01:29,293 ERROR [utils.nio.Link] (main:null) Failed to send server's CLOSE message due to socket channel's failure.
2017-05-25 03:01:29,294 ERROR [utils.nio.NioClient] (main:null) SSL Handshake failed while connecting to host: 192.168.12.5 port: 8250
2017-05-25 03:01:29,294 ERROR [utils.nio.NioConnection] (main:null) Unable to initialize the threads.
java.io.IOException: SSL Handshake failed while connecting to host: 192.168.12.5 port: 8250
at com.cloud.utils.nio.NioClient.init(NioClient.java:67)
at com.cloud.utils.nio.NioConnection.start(NioConnection.java:88)
at com.cloud.agent.Agent.start(Agent.java:228)
at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:399)
at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:367)
at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:351)
at com.cloud.agent.AgentShell.start(AgentShell.java:456)
at com.cloud.agent.AgentShell.main(AgentShell.java:491)
Same SSL engine forced to close inbound due to end of stream
On 25/5/17, 1:51 am, "Rajani Karuturi" <ra...@apache.org> wrote:
Can you check java version? Set the default java to 1.7 and delete keystore
files and restart MS
~Rajani
Sent from phone.
On 24 May 2017 9:15 p.m., "Jason Kinsella" <ja...@cloudpeople.com.au> wrote:
> I have now moved management server to a fresh CentOS7 server. But
> unfortunately I’m getting the exact same SSL handshake error. Back to
> square one.
>
> On 24/5/17, 11:40 pm, "Jason Kinsella" <ja...@cloudpeople.com.au> wrote:
>
> Hi All,
> Based on the feedback it seems like the issue is related to CentOS
> version, so I’ve built a new CentOS7 Management server using Blueshape
> noredist. I’ve restored the 4.9.2.0 DB into this server and
> management-server.logs look clean on boot. The only problem is that I can’t
> log into the webUI.
>
> The logs show a successful login (user = kinsja), but the the API
> command either is not allowed or doesn’t exist for the user. This means the
> UI doesn’t load.
>
> Anyone seen this with a restored DB?
>
> 2017-05-24 09:26:08,239 DEBUG [c.c.u.AccountManagerImpl]
> (catalina-exec-17:ctx-ee2c5e26) (logid:a8ca5ee5) User: kinsja in domain 1
> has successfully logged in
> 2017-05-24 09:26:08,246 INFO [c.c.a.ApiServer] (catalina-exec-17:ctx-ee2c5e26)
> (logid:a8ca5ee5) Current user logged in under timezone
> 2017-05-24 09:26:08,246 INFO [c.c.a.ApiServer] (catalina-exec-17:ctx-ee2c5e26)
> (logid:a8ca5ee5) Timezone offset from UTC is: 0.0
> 2017-05-24 09:26:08,251 DEBUG [c.c.a.ApiServlet] (catalina-exec-17:ctx-ee2c5e26)
> (logid:a8ca5ee5) ===END=== 192.168.10.38 -- POST
> 2017-05-24 09:26:08,320 DEBUG [c.c.a.ApiServlet] (catalina-exec-13:ctx-a1d38347)
> (logid:3404c663) ===START=== 192.168.10.38 -- GET
> command=listCapabilities&response=json&_=1495632368256
> 2017-05-24 09:26:08,325 DEBUG [c.c.a.ApiServer]
> (catalina-exec-13:ctx-a1d38347 ctx-960796a5) (logid:3404c663) The user with
> id:31 is not allowed to request the API command or the API command does not
> exist: listCapabilities
>
> Thanks
> Jason
>
> From: Jason Kinsella <ja...@cloudpeople.com.au>
> Date: Tuesday, 23 May 2017 at 10:11 pm
> To: "users@cloudstack.apache.org" <us...@cloudstack.apache.org>
> Subject: SSVM NIO SSL Handshake error
>
> Hi,
> We recently upgraded from 4.5.0 to 4.9.2.0 and encountered a problem
> with the SSVM and Console Proxy. They cannot connect to the management
> server. The SSVM cloud.log repeats this error every couple of seconds.
>
> 2017-05-23 11:58:22,461 INFO [utils.nio.NioClient] (main:null)
> Connecting to 192.168.12.1:8250
> 2017-05-23 11:58:22,465 WARN [utils.nio.Link] (main:null) This SSL
> engine was forced to close inbound due to end of stream.
> 2017-05-23 11:58:22,465 ERROR [utils.nio.Link] (main:null) Failed to
> send server's CLOSE message due to socket channel's failure.
> 2017-05-23 11:58:22,466 ERROR [utils.nio.NioClient] (main:null) SSL
> Handshake failed while connecting to host: 192.168.12.1 port: 8250
> 2017-05-23 11:58:22,466 ERROR [utils.nio.NioConnection] (main:null)
> Unable to initialize the threads.
> java.io.IOException: SSL Handshake failed while connecting to host:
> 192.168.12.1 port: 8250
> at com.cloud.utils.nio.NioClient.
> init(NioClient.java:67)
> at com.cloud.utils.nio.NioConnection.start(
> NioConnection.java:88)
> at com.cloud.agent.Agent.start(Agent.java:237)
> at com.cloud.agent.AgentShell.
> launchAgent(AgentShell.java:399)
> at com.cloud.agent.AgentShell.
> launchAgentFromClassInfo(AgentShell.java:367)
> at com.cloud.agent.AgentShell.
> launchAgent(AgentShell.java:351)
> at com.cloud.agent.AgentShell.
> start(AgentShell.java:456)
> at com.cloud.agent.AgentShell.
> main(AgentShell.java:491)
> 2017-05-23 11:58:22,468 INFO [utils.exception.CSExceptionErrorCode]
> (main:null) Could not find exception: com.cloud.utils.exception.NioConnectionException
> in error code list for exceptions
> 2017-05-23 11:58:22,468 WARN [cloud.agent.Agent] (main:null) NIO
> Connection Exception com.cloud.utils.exception.NioConnectionException:
> SSL Handshake failed while connecting to host: 192.168.12.1 port: 8250
>
> The setup is very simple. Single management server and ports are open.
>
> Things checked / tried:
>
> · Destroyed SSVM multiple times – still same problem.
>
> · SSH to SSVM from MS using ssh -i /var/cloudstack/management/.ssh/id_rsa
> -p 3922 root@IPADDRESS – PASS
>
> · SSVM telnet on 8250 to MS – PASS
>
> I’ve also tested a restore of the DB into our working development
> 4.9.2.0 server. It also exhibits the handshake errors, so most likely DB
> related.
>
> I’ve used up all my skills. Please help
>
> Regards,
> Jason
>
>
>
>
rohit.yadav@shapeblue.com
www.shapeblue.com<http://www.shapeblue.com>
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
rohit.yadav@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
Re: SSVM NIO SSL Handshake error
Posted by Rohit Yadav <ro...@shapeblue.com>.
Hi Jason,
Thanks for sharing the details. Yes, with the new setup please share with us the mgmt server logs and ssvm logs with TRACE enabled in the log4j configuration.
Regards.
________________________________
From: Jason Kinsella <ja...@cloudpeople.com.au>
Sent: 25 May 2017 12:49:50
To: users@cloudstack.apache.org
Subject: Re: SSVM NIO SSL Handshake error
Hi Rohit,
API login – fixed.
Latest systemvmtemplate (shapeblue new) in place – no improvement
No loadbalancer or known service on MS port 8250
I am doing my testing now on a fresh install of CentOS7 using shapeblue noredist with DB restored.
Hypervisor = vmware vsphere 6.5 with ESX 6.5
Systemvm.iso is dated today
All systemvms are exhibiting same behaviour.
Would any other logs help?
Regards,
Jason
On 25/5/17, 4:55 pm, "Rohit Yadav" <ro...@shapeblue.com> wrote:
Hi Jason,
The API login issue can be fixed by following this, which I believe you have already fixed: http://docs.cloudstack.apache.org/projects/cloudstack-administration/en/4.9/accounts.html#using-dynamic-roles
If not already in-use, can you try using the latest systemvmtemplate (for 4.6-4.9) from http://packages.shapeblue.com/systemvmtemplate/4.6/new.
Do you have a load-balancer on port 8250 on the management server(s), or any script/service that may be trying to perform a tcp-connect on mgmt server's port 8250?
When you upgrade can you make sure that both cloudstack-common and cloudstack-management packages are upgraded to 4.9.2.0? Also, what hypervisor(s) are you using?
The following error may hint that the jars on systemvms may not be updated, as one of the exception classes are missing:
2017-05-23 11:58:22,468 INFO [utils.exception.CSExceptionErrorCode] (main:null) Could not find exception: com.cloud.utils.exception.NioConnectionException in error code list for exceptions
Can you check that systemvm.iso are synced across hosts: (1) make sure cloudstack-common package is upgraded/updated to the same version as cloudstack-management (4.9.2.0), (2) if you're using vmware, delete this from the secondary storage, (3) for xenserver force reconnect on the host (from ui/api) or manually copy the scripts to xenserver host(s), (4) for kvm upgrade the cloudstack-common package.
Destroy all other systemvms and see if you can reproduce the issue?
Regards.
________________________________
From: Jason Kinsella <ja...@cloudpeople.com.au>
Sent: 25 May 2017 09:32:25
To: users@cloudstack.apache.org
Subject: Re: SSVM NIO SSL Handshake error
Also, just wanted to mention that the symptoms we have with systemvms not connecting is described in the mail-list
CS 4.9 NIO Selector wait time PR-1601 - https://www.mail-archive.com/dev@cloudstack.apache.org/msg69154.html
The only difference is that this thread refers to KVM hosts not connecting.
I’ve tried most suggestions in this thread.
On 25/5/17, 1:51 pm, "Jason Kinsella" <ja...@cloudpeople.com.au> wrote:
Java versions are as follows:
MS: 1.7.0_141
SSVM: 1.7.0_85
Deleted keystore files (again) and restarted MS, then recreated the SSVM.
Errors from SSVM:/var/log/cloud.log
2017-05-25 03:01:28,757 INFO [utils.nio.NioClient] (main:null) Connecting to 192.168.12.5:8250
2017-05-25 03:01:29,293 WARN [utils.nio.Link] (main:null) This SSL engine was forced to close inbound due to end of stream.
2017-05-25 03:01:29,293 ERROR [utils.nio.Link] (main:null) Failed to send server's CLOSE message due to socket channel's failure.
2017-05-25 03:01:29,294 ERROR [utils.nio.NioClient] (main:null) SSL Handshake failed while connecting to host: 192.168.12.5 port: 8250
2017-05-25 03:01:29,294 ERROR [utils.nio.NioConnection] (main:null) Unable to initialize the threads.
java.io.IOException: SSL Handshake failed while connecting to host: 192.168.12.5 port: 8250
at com.cloud.utils.nio.NioClient.init(NioClient.java:67)
at com.cloud.utils.nio.NioConnection.start(NioConnection.java:88)
at com.cloud.agent.Agent.start(Agent.java:228)
at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:399)
at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:367)
at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:351)
at com.cloud.agent.AgentShell.start(AgentShell.java:456)
at com.cloud.agent.AgentShell.main(AgentShell.java:491)
Same SSL engine forced to close inbound due to end of stream
On 25/5/17, 1:51 am, "Rajani Karuturi" <ra...@apache.org> wrote:
Can you check java version? Set the default java to 1.7 and delete keystore
files and restart MS
~Rajani
Sent from phone.
On 24 May 2017 9:15 p.m., "Jason Kinsella" <ja...@cloudpeople.com.au> wrote:
> I have now moved management server to a fresh CentOS7 server. But
> unfortunately I’m getting the exact same SSL handshake error. Back to
> square one.
>
> On 24/5/17, 11:40 pm, "Jason Kinsella" <ja...@cloudpeople.com.au> wrote:
>
> Hi All,
> Based on the feedback it seems like the issue is related to CentOS
> version, so I’ve built a new CentOS7 Management server using Blueshape
> noredist. I’ve restored the 4.9.2.0 DB into this server and
> management-server.logs look clean on boot. The only problem is that I can’t
> log into the webUI.
>
> The logs show a successful login (user = kinsja), but the the API
> command either is not allowed or doesn’t exist for the user. This means the
> UI doesn’t load.
>
> Anyone seen this with a restored DB?
>
> 2017-05-24 09:26:08,239 DEBUG [c.c.u.AccountManagerImpl]
> (catalina-exec-17:ctx-ee2c5e26) (logid:a8ca5ee5) User: kinsja in domain 1
> has successfully logged in
> 2017-05-24 09:26:08,246 INFO [c.c.a.ApiServer] (catalina-exec-17:ctx-ee2c5e26)
> (logid:a8ca5ee5) Current user logged in under timezone
> 2017-05-24 09:26:08,246 INFO [c.c.a.ApiServer] (catalina-exec-17:ctx-ee2c5e26)
> (logid:a8ca5ee5) Timezone offset from UTC is: 0.0
> 2017-05-24 09:26:08,251 DEBUG [c.c.a.ApiServlet] (catalina-exec-17:ctx-ee2c5e26)
> (logid:a8ca5ee5) ===END=== 192.168.10.38 -- POST
> 2017-05-24 09:26:08,320 DEBUG [c.c.a.ApiServlet] (catalina-exec-13:ctx-a1d38347)
> (logid:3404c663) ===START=== 192.168.10.38 -- GET
> command=listCapabilities&response=json&_=1495632368256
> 2017-05-24 09:26:08,325 DEBUG [c.c.a.ApiServer]
> (catalina-exec-13:ctx-a1d38347 ctx-960796a5) (logid:3404c663) The user with
> id:31 is not allowed to request the API command or the API command does not
> exist: listCapabilities
>
> Thanks
> Jason
>
> From: Jason Kinsella <ja...@cloudpeople.com.au>
> Date: Tuesday, 23 May 2017 at 10:11 pm
> To: "users@cloudstack.apache.org" <us...@cloudstack.apache.org>
> Subject: SSVM NIO SSL Handshake error
>
> Hi,
> We recently upgraded from 4.5.0 to 4.9.2.0 and encountered a problem
> with the SSVM and Console Proxy. They cannot connect to the management
> server. The SSVM cloud.log repeats this error every couple of seconds.
>
> 2017-05-23 11:58:22,461 INFO [utils.nio.NioClient] (main:null)
> Connecting to 192.168.12.1:8250
> 2017-05-23 11:58:22,465 WARN [utils.nio.Link] (main:null) This SSL
> engine was forced to close inbound due to end of stream.
> 2017-05-23 11:58:22,465 ERROR [utils.nio.Link] (main:null) Failed to
> send server's CLOSE message due to socket channel's failure.
> 2017-05-23 11:58:22,466 ERROR [utils.nio.NioClient] (main:null) SSL
> Handshake failed while connecting to host: 192.168.12.1 port: 8250
> 2017-05-23 11:58:22,466 ERROR [utils.nio.NioConnection] (main:null)
> Unable to initialize the threads.
> java.io.IOException: SSL Handshake failed while connecting to host:
> 192.168.12.1 port: 8250
> at com.cloud.utils.nio.NioClient.
> init(NioClient.java:67)
> at com.cloud.utils.nio.NioConnection.start(
> NioConnection.java:88)
> at com.cloud.agent.Agent.start(Agent.java:237)
> at com.cloud.agent.AgentShell.
> launchAgent(AgentShell.java:399)
> at com.cloud.agent.AgentShell.
> launchAgentFromClassInfo(AgentShell.java:367)
> at com.cloud.agent.AgentShell.
> launchAgent(AgentShell.java:351)
> at com.cloud.agent.AgentShell.
> start(AgentShell.java:456)
> at com.cloud.agent.AgentShell.
> main(AgentShell.java:491)
> 2017-05-23 11:58:22,468 INFO [utils.exception.CSExceptionErrorCode]
> (main:null) Could not find exception: com.cloud.utils.exception.NioConnectionException
> in error code list for exceptions
> 2017-05-23 11:58:22,468 WARN [cloud.agent.Agent] (main:null) NIO
> Connection Exception com.cloud.utils.exception.NioConnectionException:
> SSL Handshake failed while connecting to host: 192.168.12.1 port: 8250
>
> The setup is very simple. Single management server and ports are open.
>
> Things checked / tried:
>
> · Destroyed SSVM multiple times – still same problem.
>
> · SSH to SSVM from MS using ssh -i /var/cloudstack/management/.ssh/id_rsa
> -p 3922 root@IPADDRESS – PASS
>
> · SSVM telnet on 8250 to MS – PASS
>
> I’ve also tested a restore of the DB into our working development
> 4.9.2.0 server. It also exhibits the handshake errors, so most likely DB
> related.
>
> I’ve used up all my skills. Please help
>
> Regards,
> Jason
>
>
>
>
rohit.yadav@shapeblue.com
www.shapeblue.com<http://www.shapeblue.com>
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
rohit.yadav@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
Re: SSVM NIO SSL Handshake error
Posted by Jason Kinsella <ja...@cloudpeople.com.au>.
Hi Rohit,
API login – fixed.
Latest systemvmtemplate (shapeblue new) in place – no improvement
No loadbalancer or known service on MS port 8250
I am doing my testing now on a fresh install of CentOS7 using shapeblue noredist with DB restored.
Hypervisor = vmware vsphere 6.5 with ESX 6.5
Systemvm.iso is dated today
All systemvms are exhibiting same behaviour.
Would any other logs help?
Regards,
Jason
On 25/5/17, 4:55 pm, "Rohit Yadav" <ro...@shapeblue.com> wrote:
Hi Jason,
The API login issue can be fixed by following this, which I believe you have already fixed: http://docs.cloudstack.apache.org/projects/cloudstack-administration/en/4.9/accounts.html#using-dynamic-roles
If not already in-use, can you try using the latest systemvmtemplate (for 4.6-4.9) from http://packages.shapeblue.com/systemvmtemplate/4.6/new.
Do you have a load-balancer on port 8250 on the management server(s), or any script/service that may be trying to perform a tcp-connect on mgmt server's port 8250?
When you upgrade can you make sure that both cloudstack-common and cloudstack-management packages are upgraded to 4.9.2.0? Also, what hypervisor(s) are you using?
The following error may hint that the jars on systemvms may not be updated, as one of the exception classes are missing:
2017-05-23 11:58:22,468 INFO [utils.exception.CSExceptionErrorCode] (main:null) Could not find exception: com.cloud.utils.exception.NioConnectionException in error code list for exceptions
Can you check that systemvm.iso are synced across hosts: (1) make sure cloudstack-common package is upgraded/updated to the same version as cloudstack-management (4.9.2.0), (2) if you're using vmware, delete this from the secondary storage, (3) for xenserver force reconnect on the host (from ui/api) or manually copy the scripts to xenserver host(s), (4) for kvm upgrade the cloudstack-common package.
Destroy all other systemvms and see if you can reproduce the issue?
Regards.
________________________________
From: Jason Kinsella <ja...@cloudpeople.com.au>
Sent: 25 May 2017 09:32:25
To: users@cloudstack.apache.org
Subject: Re: SSVM NIO SSL Handshake error
Also, just wanted to mention that the symptoms we have with systemvms not connecting is described in the mail-list
CS 4.9 NIO Selector wait time PR-1601 - https://www.mail-archive.com/dev@cloudstack.apache.org/msg69154.html
The only difference is that this thread refers to KVM hosts not connecting.
I’ve tried most suggestions in this thread.
On 25/5/17, 1:51 pm, "Jason Kinsella" <ja...@cloudpeople.com.au> wrote:
Java versions are as follows:
MS: 1.7.0_141
SSVM: 1.7.0_85
Deleted keystore files (again) and restarted MS, then recreated the SSVM.
Errors from SSVM:/var/log/cloud.log
2017-05-25 03:01:28,757 INFO [utils.nio.NioClient] (main:null) Connecting to 192.168.12.5:8250
2017-05-25 03:01:29,293 WARN [utils.nio.Link] (main:null) This SSL engine was forced to close inbound due to end of stream.
2017-05-25 03:01:29,293 ERROR [utils.nio.Link] (main:null) Failed to send server's CLOSE message due to socket channel's failure.
2017-05-25 03:01:29,294 ERROR [utils.nio.NioClient] (main:null) SSL Handshake failed while connecting to host: 192.168.12.5 port: 8250
2017-05-25 03:01:29,294 ERROR [utils.nio.NioConnection] (main:null) Unable to initialize the threads.
java.io.IOException: SSL Handshake failed while connecting to host: 192.168.12.5 port: 8250
at com.cloud.utils.nio.NioClient.init(NioClient.java:67)
at com.cloud.utils.nio.NioConnection.start(NioConnection.java:88)
at com.cloud.agent.Agent.start(Agent.java:228)
at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:399)
at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:367)
at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:351)
at com.cloud.agent.AgentShell.start(AgentShell.java:456)
at com.cloud.agent.AgentShell.main(AgentShell.java:491)
Same SSL engine forced to close inbound due to end of stream
On 25/5/17, 1:51 am, "Rajani Karuturi" <ra...@apache.org> wrote:
Can you check java version? Set the default java to 1.7 and delete keystore
files and restart MS
~Rajani
Sent from phone.
On 24 May 2017 9:15 p.m., "Jason Kinsella" <ja...@cloudpeople.com.au> wrote:
> I have now moved management server to a fresh CentOS7 server. But
> unfortunately I’m getting the exact same SSL handshake error. Back to
> square one.
>
> On 24/5/17, 11:40 pm, "Jason Kinsella" <ja...@cloudpeople.com.au> wrote:
>
> Hi All,
> Based on the feedback it seems like the issue is related to CentOS
> version, so I’ve built a new CentOS7 Management server using Blueshape
> noredist. I’ve restored the 4.9.2.0 DB into this server and
> management-server.logs look clean on boot. The only problem is that I can’t
> log into the webUI.
>
> The logs show a successful login (user = kinsja), but the the API
> command either is not allowed or doesn’t exist for the user. This means the
> UI doesn’t load.
>
> Anyone seen this with a restored DB?
>
> 2017-05-24 09:26:08,239 DEBUG [c.c.u.AccountManagerImpl]
> (catalina-exec-17:ctx-ee2c5e26) (logid:a8ca5ee5) User: kinsja in domain 1
> has successfully logged in
> 2017-05-24 09:26:08,246 INFO [c.c.a.ApiServer] (catalina-exec-17:ctx-ee2c5e26)
> (logid:a8ca5ee5) Current user logged in under timezone
> 2017-05-24 09:26:08,246 INFO [c.c.a.ApiServer] (catalina-exec-17:ctx-ee2c5e26)
> (logid:a8ca5ee5) Timezone offset from UTC is: 0.0
> 2017-05-24 09:26:08,251 DEBUG [c.c.a.ApiServlet] (catalina-exec-17:ctx-ee2c5e26)
> (logid:a8ca5ee5) ===END=== 192.168.10.38 -- POST
> 2017-05-24 09:26:08,320 DEBUG [c.c.a.ApiServlet] (catalina-exec-13:ctx-a1d38347)
> (logid:3404c663) ===START=== 192.168.10.38 -- GET
> command=listCapabilities&response=json&_=1495632368256
> 2017-05-24 09:26:08,325 DEBUG [c.c.a.ApiServer]
> (catalina-exec-13:ctx-a1d38347 ctx-960796a5) (logid:3404c663) The user with
> id:31 is not allowed to request the API command or the API command does not
> exist: listCapabilities
>
> Thanks
> Jason
>
> From: Jason Kinsella <ja...@cloudpeople.com.au>
> Date: Tuesday, 23 May 2017 at 10:11 pm
> To: "users@cloudstack.apache.org" <us...@cloudstack.apache.org>
> Subject: SSVM NIO SSL Handshake error
>
> Hi,
> We recently upgraded from 4.5.0 to 4.9.2.0 and encountered a problem
> with the SSVM and Console Proxy. They cannot connect to the management
> server. The SSVM cloud.log repeats this error every couple of seconds.
>
> 2017-05-23 11:58:22,461 INFO [utils.nio.NioClient] (main:null)
> Connecting to 192.168.12.1:8250
> 2017-05-23 11:58:22,465 WARN [utils.nio.Link] (main:null) This SSL
> engine was forced to close inbound due to end of stream.
> 2017-05-23 11:58:22,465 ERROR [utils.nio.Link] (main:null) Failed to
> send server's CLOSE message due to socket channel's failure.
> 2017-05-23 11:58:22,466 ERROR [utils.nio.NioClient] (main:null) SSL
> Handshake failed while connecting to host: 192.168.12.1 port: 8250
> 2017-05-23 11:58:22,466 ERROR [utils.nio.NioConnection] (main:null)
> Unable to initialize the threads.
> java.io.IOException: SSL Handshake failed while connecting to host:
> 192.168.12.1 port: 8250
> at com.cloud.utils.nio.NioClient.
> init(NioClient.java:67)
> at com.cloud.utils.nio.NioConnection.start(
> NioConnection.java:88)
> at com.cloud.agent.Agent.start(Agent.java:237)
> at com.cloud.agent.AgentShell.
> launchAgent(AgentShell.java:399)
> at com.cloud.agent.AgentShell.
> launchAgentFromClassInfo(AgentShell.java:367)
> at com.cloud.agent.AgentShell.
> launchAgent(AgentShell.java:351)
> at com.cloud.agent.AgentShell.
> start(AgentShell.java:456)
> at com.cloud.agent.AgentShell.
> main(AgentShell.java:491)
> 2017-05-23 11:58:22,468 INFO [utils.exception.CSExceptionErrorCode]
> (main:null) Could not find exception: com.cloud.utils.exception.NioConnectionException
> in error code list for exceptions
> 2017-05-23 11:58:22,468 WARN [cloud.agent.Agent] (main:null) NIO
> Connection Exception com.cloud.utils.exception.NioConnectionException:
> SSL Handshake failed while connecting to host: 192.168.12.1 port: 8250
>
> The setup is very simple. Single management server and ports are open.
>
> Things checked / tried:
>
> · Destroyed SSVM multiple times – still same problem.
>
> · SSH to SSVM from MS using ssh -i /var/cloudstack/management/.ssh/id_rsa
> -p 3922 root@IPADDRESS – PASS
>
> · SSVM telnet on 8250 to MS – PASS
>
> I’ve also tested a restore of the DB into our working development
> 4.9.2.0 server. It also exhibits the handshake errors, so most likely DB
> related.
>
> I’ve used up all my skills. Please help
>
> Regards,
> Jason
>
>
>
>
rohit.yadav@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
Re: SSVM NIO SSL Handshake error
Posted by Rohit Yadav <ro...@shapeblue.com>.
Hi Jason,
The API login issue can be fixed by following this, which I believe you have already fixed: http://docs.cloudstack.apache.org/projects/cloudstack-administration/en/4.9/accounts.html#using-dynamic-roles
If not already in-use, can you try using the latest systemvmtemplate (for 4.6-4.9) from http://packages.shapeblue.com/systemvmtemplate/4.6/new.
Do you have a load-balancer on port 8250 on the management server(s), or any script/service that may be trying to perform a tcp-connect on mgmt server's port 8250?
When you upgrade can you make sure that both cloudstack-common and cloudstack-management packages are upgraded to 4.9.2.0? Also, what hypervisor(s) are you using?
The following error may hint that the jars on systemvms may not be updated, as one of the exception classes are missing:
2017-05-23 11:58:22,468 INFO [utils.exception.CSExceptionErrorCode] (main:null) Could not find exception: com.cloud.utils.exception.NioConnectionException in error code list for exceptions
Can you check that systemvm.iso are synced across hosts: (1) make sure cloudstack-common package is upgraded/updated to the same version as cloudstack-management (4.9.2.0), (2) if you're using vmware, delete this from the secondary storage, (3) for xenserver force reconnect on the host (from ui/api) or manually copy the scripts to xenserver host(s), (4) for kvm upgrade the cloudstack-common package.
Destroy all other systemvms and see if you can reproduce the issue?
Regards.
________________________________
From: Jason Kinsella <ja...@cloudpeople.com.au>
Sent: 25 May 2017 09:32:25
To: users@cloudstack.apache.org
Subject: Re: SSVM NIO SSL Handshake error
Also, just wanted to mention that the symptoms we have with systemvms not connecting is described in the mail-list
CS 4.9 NIO Selector wait time PR-1601 - https://www.mail-archive.com/dev@cloudstack.apache.org/msg69154.html
The only difference is that this thread refers to KVM hosts not connecting.
I’ve tried most suggestions in this thread.
On 25/5/17, 1:51 pm, "Jason Kinsella" <ja...@cloudpeople.com.au> wrote:
Java versions are as follows:
MS: 1.7.0_141
SSVM: 1.7.0_85
Deleted keystore files (again) and restarted MS, then recreated the SSVM.
Errors from SSVM:/var/log/cloud.log
2017-05-25 03:01:28,757 INFO [utils.nio.NioClient] (main:null) Connecting to 192.168.12.5:8250
2017-05-25 03:01:29,293 WARN [utils.nio.Link] (main:null) This SSL engine was forced to close inbound due to end of stream.
2017-05-25 03:01:29,293 ERROR [utils.nio.Link] (main:null) Failed to send server's CLOSE message due to socket channel's failure.
2017-05-25 03:01:29,294 ERROR [utils.nio.NioClient] (main:null) SSL Handshake failed while connecting to host: 192.168.12.5 port: 8250
2017-05-25 03:01:29,294 ERROR [utils.nio.NioConnection] (main:null) Unable to initialize the threads.
java.io.IOException: SSL Handshake failed while connecting to host: 192.168.12.5 port: 8250
at com.cloud.utils.nio.NioClient.init(NioClient.java:67)
at com.cloud.utils.nio.NioConnection.start(NioConnection.java:88)
at com.cloud.agent.Agent.start(Agent.java:228)
at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:399)
at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:367)
at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:351)
at com.cloud.agent.AgentShell.start(AgentShell.java:456)
at com.cloud.agent.AgentShell.main(AgentShell.java:491)
Same SSL engine forced to close inbound due to end of stream
On 25/5/17, 1:51 am, "Rajani Karuturi" <ra...@apache.org> wrote:
Can you check java version? Set the default java to 1.7 and delete keystore
files and restart MS
~Rajani
Sent from phone.
On 24 May 2017 9:15 p.m., "Jason Kinsella" <ja...@cloudpeople.com.au> wrote:
> I have now moved management server to a fresh CentOS7 server. But
> unfortunately I’m getting the exact same SSL handshake error. Back to
> square one.
>
> On 24/5/17, 11:40 pm, "Jason Kinsella" <ja...@cloudpeople.com.au> wrote:
>
> Hi All,
> Based on the feedback it seems like the issue is related to CentOS
> version, so I’ve built a new CentOS7 Management server using Blueshape
> noredist. I’ve restored the 4.9.2.0 DB into this server and
> management-server.logs look clean on boot. The only problem is that I can’t
> log into the webUI.
>
> The logs show a successful login (user = kinsja), but the the API
> command either is not allowed or doesn’t exist for the user. This means the
> UI doesn’t load.
>
> Anyone seen this with a restored DB?
>
> 2017-05-24 09:26:08,239 DEBUG [c.c.u.AccountManagerImpl]
> (catalina-exec-17:ctx-ee2c5e26) (logid:a8ca5ee5) User: kinsja in domain 1
> has successfully logged in
> 2017-05-24 09:26:08,246 INFO [c.c.a.ApiServer] (catalina-exec-17:ctx-ee2c5e26)
> (logid:a8ca5ee5) Current user logged in under timezone
> 2017-05-24 09:26:08,246 INFO [c.c.a.ApiServer] (catalina-exec-17:ctx-ee2c5e26)
> (logid:a8ca5ee5) Timezone offset from UTC is: 0.0
> 2017-05-24 09:26:08,251 DEBUG [c.c.a.ApiServlet] (catalina-exec-17:ctx-ee2c5e26)
> (logid:a8ca5ee5) ===END=== 192.168.10.38 -- POST
> 2017-05-24 09:26:08,320 DEBUG [c.c.a.ApiServlet] (catalina-exec-13:ctx-a1d38347)
> (logid:3404c663) ===START=== 192.168.10.38 -- GET
> command=listCapabilities&response=json&_=1495632368256
> 2017-05-24 09:26:08,325 DEBUG [c.c.a.ApiServer]
> (catalina-exec-13:ctx-a1d38347 ctx-960796a5) (logid:3404c663) The user with
> id:31 is not allowed to request the API command or the API command does not
> exist: listCapabilities
>
> Thanks
> Jason
>
> From: Jason Kinsella <ja...@cloudpeople.com.au>
> Date: Tuesday, 23 May 2017 at 10:11 pm
> To: "users@cloudstack.apache.org" <us...@cloudstack.apache.org>
> Subject: SSVM NIO SSL Handshake error
>
> Hi,
> We recently upgraded from 4.5.0 to 4.9.2.0 and encountered a problem
> with the SSVM and Console Proxy. They cannot connect to the management
> server. The SSVM cloud.log repeats this error every couple of seconds.
>
> 2017-05-23 11:58:22,461 INFO [utils.nio.NioClient] (main:null)
> Connecting to 192.168.12.1:8250
> 2017-05-23 11:58:22,465 WARN [utils.nio.Link] (main:null) This SSL
> engine was forced to close inbound due to end of stream.
> 2017-05-23 11:58:22,465 ERROR [utils.nio.Link] (main:null) Failed to
> send server's CLOSE message due to socket channel's failure.
> 2017-05-23 11:58:22,466 ERROR [utils.nio.NioClient] (main:null) SSL
> Handshake failed while connecting to host: 192.168.12.1 port: 8250
> 2017-05-23 11:58:22,466 ERROR [utils.nio.NioConnection] (main:null)
> Unable to initialize the threads.
> java.io.IOException: SSL Handshake failed while connecting to host:
> 192.168.12.1 port: 8250
> at com.cloud.utils.nio.NioClient.
> init(NioClient.java:67)
> at com.cloud.utils.nio.NioConnection.start(
> NioConnection.java:88)
> at com.cloud.agent.Agent.start(Agent.java:237)
> at com.cloud.agent.AgentShell.
> launchAgent(AgentShell.java:399)
> at com.cloud.agent.AgentShell.
> launchAgentFromClassInfo(AgentShell.java:367)
> at com.cloud.agent.AgentShell.
> launchAgent(AgentShell.java:351)
> at com.cloud.agent.AgentShell.
> start(AgentShell.java:456)
> at com.cloud.agent.AgentShell.
> main(AgentShell.java:491)
> 2017-05-23 11:58:22,468 INFO [utils.exception.CSExceptionErrorCode]
> (main:null) Could not find exception: com.cloud.utils.exception.NioConnectionException
> in error code list for exceptions
> 2017-05-23 11:58:22,468 WARN [cloud.agent.Agent] (main:null) NIO
> Connection Exception com.cloud.utils.exception.NioConnectionException:
> SSL Handshake failed while connecting to host: 192.168.12.1 port: 8250
>
> The setup is very simple. Single management server and ports are open.
>
> Things checked / tried:
>
> · Destroyed SSVM multiple times – still same problem.
>
> · SSH to SSVM from MS using ssh -i /var/cloudstack/management/.ssh/id_rsa
> -p 3922 root@IPADDRESS – PASS
>
> · SSVM telnet on 8250 to MS – PASS
>
> I’ve also tested a restore of the DB into our working development
> 4.9.2.0 server. It also exhibits the handshake errors, so most likely DB
> related.
>
> I’ve used up all my skills. Please help
>
> Regards,
> Jason
>
>
>
>
rohit.yadav@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
Re: SSVM NIO SSL Handshake error
Posted by Jason Kinsella <ja...@cloudpeople.com.au>.
Also, just wanted to mention that the symptoms we have with systemvms not connecting is described in the mail-list
CS 4.9 NIO Selector wait time PR-1601 - https://www.mail-archive.com/dev@cloudstack.apache.org/msg69154.html
The only difference is that this thread refers to KVM hosts not connecting.
I’ve tried most suggestions in this thread.
On 25/5/17, 1:51 pm, "Jason Kinsella" <ja...@cloudpeople.com.au> wrote:
Java versions are as follows:
MS: 1.7.0_141
SSVM: 1.7.0_85
Deleted keystore files (again) and restarted MS, then recreated the SSVM.
Errors from SSVM:/var/log/cloud.log
2017-05-25 03:01:28,757 INFO [utils.nio.NioClient] (main:null) Connecting to 192.168.12.5:8250
2017-05-25 03:01:29,293 WARN [utils.nio.Link] (main:null) This SSL engine was forced to close inbound due to end of stream.
2017-05-25 03:01:29,293 ERROR [utils.nio.Link] (main:null) Failed to send server's CLOSE message due to socket channel's failure.
2017-05-25 03:01:29,294 ERROR [utils.nio.NioClient] (main:null) SSL Handshake failed while connecting to host: 192.168.12.5 port: 8250
2017-05-25 03:01:29,294 ERROR [utils.nio.NioConnection] (main:null) Unable to initialize the threads.
java.io.IOException: SSL Handshake failed while connecting to host: 192.168.12.5 port: 8250
at com.cloud.utils.nio.NioClient.init(NioClient.java:67)
at com.cloud.utils.nio.NioConnection.start(NioConnection.java:88)
at com.cloud.agent.Agent.start(Agent.java:228)
at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:399)
at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:367)
at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:351)
at com.cloud.agent.AgentShell.start(AgentShell.java:456)
at com.cloud.agent.AgentShell.main(AgentShell.java:491)
Same SSL engine forced to close inbound due to end of stream
On 25/5/17, 1:51 am, "Rajani Karuturi" <ra...@apache.org> wrote:
Can you check java version? Set the default java to 1.7 and delete keystore
files and restart MS
~Rajani
Sent from phone.
On 24 May 2017 9:15 p.m., "Jason Kinsella" <ja...@cloudpeople.com.au> wrote:
> I have now moved management server to a fresh CentOS7 server. But
> unfortunately I’m getting the exact same SSL handshake error. Back to
> square one.
>
> On 24/5/17, 11:40 pm, "Jason Kinsella" <ja...@cloudpeople.com.au> wrote:
>
> Hi All,
> Based on the feedback it seems like the issue is related to CentOS
> version, so I’ve built a new CentOS7 Management server using Blueshape
> noredist. I’ve restored the 4.9.2.0 DB into this server and
> management-server.logs look clean on boot. The only problem is that I can’t
> log into the webUI.
>
> The logs show a successful login (user = kinsja), but the the API
> command either is not allowed or doesn’t exist for the user. This means the
> UI doesn’t load.
>
> Anyone seen this with a restored DB?
>
> 2017-05-24 09:26:08,239 DEBUG [c.c.u.AccountManagerImpl]
> (catalina-exec-17:ctx-ee2c5e26) (logid:a8ca5ee5) User: kinsja in domain 1
> has successfully logged in
> 2017-05-24 09:26:08,246 INFO [c.c.a.ApiServer] (catalina-exec-17:ctx-ee2c5e26)
> (logid:a8ca5ee5) Current user logged in under timezone
> 2017-05-24 09:26:08,246 INFO [c.c.a.ApiServer] (catalina-exec-17:ctx-ee2c5e26)
> (logid:a8ca5ee5) Timezone offset from UTC is: 0.0
> 2017-05-24 09:26:08,251 DEBUG [c.c.a.ApiServlet] (catalina-exec-17:ctx-ee2c5e26)
> (logid:a8ca5ee5) ===END=== 192.168.10.38 -- POST
> 2017-05-24 09:26:08,320 DEBUG [c.c.a.ApiServlet] (catalina-exec-13:ctx-a1d38347)
> (logid:3404c663) ===START=== 192.168.10.38 -- GET
> command=listCapabilities&response=json&_=1495632368256
> 2017-05-24 09:26:08,325 DEBUG [c.c.a.ApiServer]
> (catalina-exec-13:ctx-a1d38347 ctx-960796a5) (logid:3404c663) The user with
> id:31 is not allowed to request the API command or the API command does not
> exist: listCapabilities
>
> Thanks
> Jason
>
> From: Jason Kinsella <ja...@cloudpeople.com.au>
> Date: Tuesday, 23 May 2017 at 10:11 pm
> To: "users@cloudstack.apache.org" <us...@cloudstack.apache.org>
> Subject: SSVM NIO SSL Handshake error
>
> Hi,
> We recently upgraded from 4.5.0 to 4.9.2.0 and encountered a problem
> with the SSVM and Console Proxy. They cannot connect to the management
> server. The SSVM cloud.log repeats this error every couple of seconds.
>
> 2017-05-23 11:58:22,461 INFO [utils.nio.NioClient] (main:null)
> Connecting to 192.168.12.1:8250
> 2017-05-23 11:58:22,465 WARN [utils.nio.Link] (main:null) This SSL
> engine was forced to close inbound due to end of stream.
> 2017-05-23 11:58:22,465 ERROR [utils.nio.Link] (main:null) Failed to
> send server's CLOSE message due to socket channel's failure.
> 2017-05-23 11:58:22,466 ERROR [utils.nio.NioClient] (main:null) SSL
> Handshake failed while connecting to host: 192.168.12.1 port: 8250
> 2017-05-23 11:58:22,466 ERROR [utils.nio.NioConnection] (main:null)
> Unable to initialize the threads.
> java.io.IOException: SSL Handshake failed while connecting to host:
> 192.168.12.1 port: 8250
> at com.cloud.utils.nio.NioClient.
> init(NioClient.java:67)
> at com.cloud.utils.nio.NioConnection.start(
> NioConnection.java:88)
> at com.cloud.agent.Agent.start(Agent.java:237)
> at com.cloud.agent.AgentShell.
> launchAgent(AgentShell.java:399)
> at com.cloud.agent.AgentShell.
> launchAgentFromClassInfo(AgentShell.java:367)
> at com.cloud.agent.AgentShell.
> launchAgent(AgentShell.java:351)
> at com.cloud.agent.AgentShell.
> start(AgentShell.java:456)
> at com.cloud.agent.AgentShell.
> main(AgentShell.java:491)
> 2017-05-23 11:58:22,468 INFO [utils.exception.CSExceptionErrorCode]
> (main:null) Could not find exception: com.cloud.utils.exception.NioConnectionException
> in error code list for exceptions
> 2017-05-23 11:58:22,468 WARN [cloud.agent.Agent] (main:null) NIO
> Connection Exception com.cloud.utils.exception.NioConnectionException:
> SSL Handshake failed while connecting to host: 192.168.12.1 port: 8250
>
> The setup is very simple. Single management server and ports are open.
>
> Things checked / tried:
>
> · Destroyed SSVM multiple times – still same problem.
>
> · SSH to SSVM from MS using ssh -i /var/cloudstack/management/.ssh/id_rsa
> -p 3922 root@IPADDRESS – PASS
>
> · SSVM telnet on 8250 to MS – PASS
>
> I’ve also tested a restore of the DB into our working development
> 4.9.2.0 server. It also exhibits the handshake errors, so most likely DB
> related.
>
> I’ve used up all my skills. Please help
>
> Regards,
> Jason
>
>
>
>
Re: SSVM NIO SSL Handshake error
Posted by Jason Kinsella <ja...@cloudpeople.com.au>.
Java versions are as follows:
MS: 1.7.0_141
SSVM: 1.7.0_85
Deleted keystore files (again) and restarted MS, then recreated the SSVM.
Errors from SSVM:/var/log/cloud.log
2017-05-25 03:01:28,757 INFO [utils.nio.NioClient] (main:null) Connecting to 192.168.12.5:8250
2017-05-25 03:01:29,293 WARN [utils.nio.Link] (main:null) This SSL engine was forced to close inbound due to end of stream.
2017-05-25 03:01:29,293 ERROR [utils.nio.Link] (main:null) Failed to send server's CLOSE message due to socket channel's failure.
2017-05-25 03:01:29,294 ERROR [utils.nio.NioClient] (main:null) SSL Handshake failed while connecting to host: 192.168.12.5 port: 8250
2017-05-25 03:01:29,294 ERROR [utils.nio.NioConnection] (main:null) Unable to initialize the threads.
java.io.IOException: SSL Handshake failed while connecting to host: 192.168.12.5 port: 8250
at com.cloud.utils.nio.NioClient.init(NioClient.java:67)
at com.cloud.utils.nio.NioConnection.start(NioConnection.java:88)
at com.cloud.agent.Agent.start(Agent.java:228)
at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:399)
at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:367)
at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:351)
at com.cloud.agent.AgentShell.start(AgentShell.java:456)
at com.cloud.agent.AgentShell.main(AgentShell.java:491)
Same SSL engine forced to close inbound due to end of stream
On 25/5/17, 1:51 am, "Rajani Karuturi" <ra...@apache.org> wrote:
Can you check java version? Set the default java to 1.7 and delete keystore
files and restart MS
~Rajani
Sent from phone.
On 24 May 2017 9:15 p.m., "Jason Kinsella" <ja...@cloudpeople.com.au> wrote:
> I have now moved management server to a fresh CentOS7 server. But
> unfortunately I’m getting the exact same SSL handshake error. Back to
> square one.
>
> On 24/5/17, 11:40 pm, "Jason Kinsella" <ja...@cloudpeople.com.au> wrote:
>
> Hi All,
> Based on the feedback it seems like the issue is related to CentOS
> version, so I’ve built a new CentOS7 Management server using Blueshape
> noredist. I’ve restored the 4.9.2.0 DB into this server and
> management-server.logs look clean on boot. The only problem is that I can’t
> log into the webUI.
>
> The logs show a successful login (user = kinsja), but the the API
> command either is not allowed or doesn’t exist for the user. This means the
> UI doesn’t load.
>
> Anyone seen this with a restored DB?
>
> 2017-05-24 09:26:08,239 DEBUG [c.c.u.AccountManagerImpl]
> (catalina-exec-17:ctx-ee2c5e26) (logid:a8ca5ee5) User: kinsja in domain 1
> has successfully logged in
> 2017-05-24 09:26:08,246 INFO [c.c.a.ApiServer] (catalina-exec-17:ctx-ee2c5e26)
> (logid:a8ca5ee5) Current user logged in under timezone
> 2017-05-24 09:26:08,246 INFO [c.c.a.ApiServer] (catalina-exec-17:ctx-ee2c5e26)
> (logid:a8ca5ee5) Timezone offset from UTC is: 0.0
> 2017-05-24 09:26:08,251 DEBUG [c.c.a.ApiServlet] (catalina-exec-17:ctx-ee2c5e26)
> (logid:a8ca5ee5) ===END=== 192.168.10.38 -- POST
> 2017-05-24 09:26:08,320 DEBUG [c.c.a.ApiServlet] (catalina-exec-13:ctx-a1d38347)
> (logid:3404c663) ===START=== 192.168.10.38 -- GET
> command=listCapabilities&response=json&_=1495632368256
> 2017-05-24 09:26:08,325 DEBUG [c.c.a.ApiServer]
> (catalina-exec-13:ctx-a1d38347 ctx-960796a5) (logid:3404c663) The user with
> id:31 is not allowed to request the API command or the API command does not
> exist: listCapabilities
>
> Thanks
> Jason
>
> From: Jason Kinsella <ja...@cloudpeople.com.au>
> Date: Tuesday, 23 May 2017 at 10:11 pm
> To: "users@cloudstack.apache.org" <us...@cloudstack.apache.org>
> Subject: SSVM NIO SSL Handshake error
>
> Hi,
> We recently upgraded from 4.5.0 to 4.9.2.0 and encountered a problem
> with the SSVM and Console Proxy. They cannot connect to the management
> server. The SSVM cloud.log repeats this error every couple of seconds.
>
> 2017-05-23 11:58:22,461 INFO [utils.nio.NioClient] (main:null)
> Connecting to 192.168.12.1:8250
> 2017-05-23 11:58:22,465 WARN [utils.nio.Link] (main:null) This SSL
> engine was forced to close inbound due to end of stream.
> 2017-05-23 11:58:22,465 ERROR [utils.nio.Link] (main:null) Failed to
> send server's CLOSE message due to socket channel's failure.
> 2017-05-23 11:58:22,466 ERROR [utils.nio.NioClient] (main:null) SSL
> Handshake failed while connecting to host: 192.168.12.1 port: 8250
> 2017-05-23 11:58:22,466 ERROR [utils.nio.NioConnection] (main:null)
> Unable to initialize the threads.
> java.io.IOException: SSL Handshake failed while connecting to host:
> 192.168.12.1 port: 8250
> at com.cloud.utils.nio.NioClient.
> init(NioClient.java:67)
> at com.cloud.utils.nio.NioConnection.start(
> NioConnection.java:88)
> at com.cloud.agent.Agent.start(Agent.java:237)
> at com.cloud.agent.AgentShell.
> launchAgent(AgentShell.java:399)
> at com.cloud.agent.AgentShell.
> launchAgentFromClassInfo(AgentShell.java:367)
> at com.cloud.agent.AgentShell.
> launchAgent(AgentShell.java:351)
> at com.cloud.agent.AgentShell.
> start(AgentShell.java:456)
> at com.cloud.agent.AgentShell.
> main(AgentShell.java:491)
> 2017-05-23 11:58:22,468 INFO [utils.exception.CSExceptionErrorCode]
> (main:null) Could not find exception: com.cloud.utils.exception.NioConnectionException
> in error code list for exceptions
> 2017-05-23 11:58:22,468 WARN [cloud.agent.Agent] (main:null) NIO
> Connection Exception com.cloud.utils.exception.NioConnectionException:
> SSL Handshake failed while connecting to host: 192.168.12.1 port: 8250
>
> The setup is very simple. Single management server and ports are open.
>
> Things checked / tried:
>
> · Destroyed SSVM multiple times – still same problem.
>
> · SSH to SSVM from MS using ssh -i /var/cloudstack/management/.ssh/id_rsa
> -p 3922 root@IPADDRESS – PASS
>
> · SSVM telnet on 8250 to MS – PASS
>
> I’ve also tested a restore of the DB into our working development
> 4.9.2.0 server. It also exhibits the handshake errors, so most likely DB
> related.
>
> I’ve used up all my skills. Please help
>
> Regards,
> Jason
>
>
>
>
Re: SSVM NIO SSL Handshake error
Posted by Rajani Karuturi <ra...@apache.org>.
Can you check java version? Set the default java to 1.7 and delete keystore
files and restart MS
~Rajani
Sent from phone.
On 24 May 2017 9:15 p.m., "Jason Kinsella" <ja...@cloudpeople.com.au> wrote:
> I have now moved management server to a fresh CentOS7 server. But
> unfortunately I’m getting the exact same SSL handshake error. Back to
> square one.
>
> On 24/5/17, 11:40 pm, "Jason Kinsella" <ja...@cloudpeople.com.au> wrote:
>
> Hi All,
> Based on the feedback it seems like the issue is related to CentOS
> version, so I’ve built a new CentOS7 Management server using Blueshape
> noredist. I’ve restored the 4.9.2.0 DB into this server and
> management-server.logs look clean on boot. The only problem is that I can’t
> log into the webUI.
>
> The logs show a successful login (user = kinsja), but the the API
> command either is not allowed or doesn’t exist for the user. This means the
> UI doesn’t load.
>
> Anyone seen this with a restored DB?
>
> 2017-05-24 09:26:08,239 DEBUG [c.c.u.AccountManagerImpl]
> (catalina-exec-17:ctx-ee2c5e26) (logid:a8ca5ee5) User: kinsja in domain 1
> has successfully logged in
> 2017-05-24 09:26:08,246 INFO [c.c.a.ApiServer] (catalina-exec-17:ctx-ee2c5e26)
> (logid:a8ca5ee5) Current user logged in under timezone
> 2017-05-24 09:26:08,246 INFO [c.c.a.ApiServer] (catalina-exec-17:ctx-ee2c5e26)
> (logid:a8ca5ee5) Timezone offset from UTC is: 0.0
> 2017-05-24 09:26:08,251 DEBUG [c.c.a.ApiServlet] (catalina-exec-17:ctx-ee2c5e26)
> (logid:a8ca5ee5) ===END=== 192.168.10.38 -- POST
> 2017-05-24 09:26:08,320 DEBUG [c.c.a.ApiServlet] (catalina-exec-13:ctx-a1d38347)
> (logid:3404c663) ===START=== 192.168.10.38 -- GET
> command=listCapabilities&response=json&_=1495632368256
> 2017-05-24 09:26:08,325 DEBUG [c.c.a.ApiServer]
> (catalina-exec-13:ctx-a1d38347 ctx-960796a5) (logid:3404c663) The user with
> id:31 is not allowed to request the API command or the API command does not
> exist: listCapabilities
>
> Thanks
> Jason
>
> From: Jason Kinsella <ja...@cloudpeople.com.au>
> Date: Tuesday, 23 May 2017 at 10:11 pm
> To: "users@cloudstack.apache.org" <us...@cloudstack.apache.org>
> Subject: SSVM NIO SSL Handshake error
>
> Hi,
> We recently upgraded from 4.5.0 to 4.9.2.0 and encountered a problem
> with the SSVM and Console Proxy. They cannot connect to the management
> server. The SSVM cloud.log repeats this error every couple of seconds.
>
> 2017-05-23 11:58:22,461 INFO [utils.nio.NioClient] (main:null)
> Connecting to 192.168.12.1:8250
> 2017-05-23 11:58:22,465 WARN [utils.nio.Link] (main:null) This SSL
> engine was forced to close inbound due to end of stream.
> 2017-05-23 11:58:22,465 ERROR [utils.nio.Link] (main:null) Failed to
> send server's CLOSE message due to socket channel's failure.
> 2017-05-23 11:58:22,466 ERROR [utils.nio.NioClient] (main:null) SSL
> Handshake failed while connecting to host: 192.168.12.1 port: 8250
> 2017-05-23 11:58:22,466 ERROR [utils.nio.NioConnection] (main:null)
> Unable to initialize the threads.
> java.io.IOException: SSL Handshake failed while connecting to host:
> 192.168.12.1 port: 8250
> at com.cloud.utils.nio.NioClient.
> init(NioClient.java:67)
> at com.cloud.utils.nio.NioConnection.start(
> NioConnection.java:88)
> at com.cloud.agent.Agent.start(Agent.java:237)
> at com.cloud.agent.AgentShell.
> launchAgent(AgentShell.java:399)
> at com.cloud.agent.AgentShell.
> launchAgentFromClassInfo(AgentShell.java:367)
> at com.cloud.agent.AgentShell.
> launchAgent(AgentShell.java:351)
> at com.cloud.agent.AgentShell.
> start(AgentShell.java:456)
> at com.cloud.agent.AgentShell.
> main(AgentShell.java:491)
> 2017-05-23 11:58:22,468 INFO [utils.exception.CSExceptionErrorCode]
> (main:null) Could not find exception: com.cloud.utils.exception.NioConnectionException
> in error code list for exceptions
> 2017-05-23 11:58:22,468 WARN [cloud.agent.Agent] (main:null) NIO
> Connection Exception com.cloud.utils.exception.NioConnectionException:
> SSL Handshake failed while connecting to host: 192.168.12.1 port: 8250
>
> The setup is very simple. Single management server and ports are open.
>
> Things checked / tried:
>
> · Destroyed SSVM multiple times – still same problem.
>
> · SSH to SSVM from MS using ssh -i /var/cloudstack/management/.ssh/id_rsa
> -p 3922 root@IPADDRESS – PASS
>
> · SSVM telnet on 8250 to MS – PASS
>
> I’ve also tested a restore of the DB into our working development
> 4.9.2.0 server. It also exhibits the handshake errors, so most likely DB
> related.
>
> I’ve used up all my skills. Please help
>
> Regards,
> Jason
>
>
>
>
Re: SSVM NIO SSL Handshake error
Posted by Jason Kinsella <ja...@cloudpeople.com.au>.
I have now moved management server to a fresh CentOS7 server. But unfortunately I’m getting the exact same SSL handshake error. Back to square one.
On 24/5/17, 11:40 pm, "Jason Kinsella" <ja...@cloudpeople.com.au> wrote:
Hi All,
Based on the feedback it seems like the issue is related to CentOS version, so I’ve built a new CentOS7 Management server using Blueshape noredist. I’ve restored the 4.9.2.0 DB into this server and management-server.logs look clean on boot. The only problem is that I can’t log into the webUI.
The logs show a successful login (user = kinsja), but the the API command either is not allowed or doesn’t exist for the user. This means the UI doesn’t load.
Anyone seen this with a restored DB?
2017-05-24 09:26:08,239 DEBUG [c.c.u.AccountManagerImpl] (catalina-exec-17:ctx-ee2c5e26) (logid:a8ca5ee5) User: kinsja in domain 1 has successfully logged in
2017-05-24 09:26:08,246 INFO [c.c.a.ApiServer] (catalina-exec-17:ctx-ee2c5e26) (logid:a8ca5ee5) Current user logged in under timezone
2017-05-24 09:26:08,246 INFO [c.c.a.ApiServer] (catalina-exec-17:ctx-ee2c5e26) (logid:a8ca5ee5) Timezone offset from UTC is: 0.0
2017-05-24 09:26:08,251 DEBUG [c.c.a.ApiServlet] (catalina-exec-17:ctx-ee2c5e26) (logid:a8ca5ee5) ===END=== 192.168.10.38 -- POST
2017-05-24 09:26:08,320 DEBUG [c.c.a.ApiServlet] (catalina-exec-13:ctx-a1d38347) (logid:3404c663) ===START=== 192.168.10.38 -- GET command=listCapabilities&response=json&_=1495632368256
2017-05-24 09:26:08,325 DEBUG [c.c.a.ApiServer] (catalina-exec-13:ctx-a1d38347 ctx-960796a5) (logid:3404c663) The user with id:31 is not allowed to request the API command or the API command does not exist: listCapabilities
Thanks
Jason
From: Jason Kinsella <ja...@cloudpeople.com.au>
Date: Tuesday, 23 May 2017 at 10:11 pm
To: "users@cloudstack.apache.org" <us...@cloudstack.apache.org>
Subject: SSVM NIO SSL Handshake error
Hi,
We recently upgraded from 4.5.0 to 4.9.2.0 and encountered a problem with the SSVM and Console Proxy. They cannot connect to the management server. The SSVM cloud.log repeats this error every couple of seconds.
2017-05-23 11:58:22,461 INFO [utils.nio.NioClient] (main:null) Connecting to 192.168.12.1:8250
2017-05-23 11:58:22,465 WARN [utils.nio.Link] (main:null) This SSL engine was forced to close inbound due to end of stream.
2017-05-23 11:58:22,465 ERROR [utils.nio.Link] (main:null) Failed to send server's CLOSE message due to socket channel's failure.
2017-05-23 11:58:22,466 ERROR [utils.nio.NioClient] (main:null) SSL Handshake failed while connecting to host: 192.168.12.1 port: 8250
2017-05-23 11:58:22,466 ERROR [utils.nio.NioConnection] (main:null) Unable to initialize the threads.
java.io.IOException: SSL Handshake failed while connecting to host: 192.168.12.1 port: 8250
at com.cloud.utils.nio.NioClient.init(NioClient.java:67)
at com.cloud.utils.nio.NioConnection.start(NioConnection.java:88)
at com.cloud.agent.Agent.start(Agent.java:237)
at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:399)
at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:367)
at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:351)
at com.cloud.agent.AgentShell.start(AgentShell.java:456)
at com.cloud.agent.AgentShell.main(AgentShell.java:491)
2017-05-23 11:58:22,468 INFO [utils.exception.CSExceptionErrorCode] (main:null) Could not find exception: com.cloud.utils.exception.NioConnectionException in error code list for exceptions
2017-05-23 11:58:22,468 WARN [cloud.agent.Agent] (main:null) NIO Connection Exception com.cloud.utils.exception.NioConnectionException: SSL Handshake failed while connecting to host: 192.168.12.1 port: 8250
The setup is very simple. Single management server and ports are open.
Things checked / tried:
· Destroyed SSVM multiple times – still same problem.
· SSH to SSVM from MS using ssh -i /var/cloudstack/management/.ssh/id_rsa -p 3922 root@IPADDRESS – PASS
· SSVM telnet on 8250 to MS – PASS
I’ve also tested a restore of the DB into our working development 4.9.2.0 server. It also exhibits the handshake errors, so most likely DB related.
I’ve used up all my skills. Please help
Regards,
Jason
Re: SSVM NIO SSL Handshake error
Posted by Jason Kinsella <ja...@cloudpeople.com.au>.
Hi All,
Based on the feedback it seems like the issue is related to CentOS version, so I’ve built a new CentOS7 Management server using Blueshape noredist. I’ve restored the 4.9.2.0 DB into this server and management-server.logs look clean on boot. The only problem is that I can’t log into the webUI.
The logs show a successful login (user = kinsja), but the the API command either is not allowed or doesn’t exist for the user. This means the UI doesn’t load.
Anyone seen this with a restored DB?
2017-05-24 09:26:08,239 DEBUG [c.c.u.AccountManagerImpl] (catalina-exec-17:ctx-ee2c5e26) (logid:a8ca5ee5) User: kinsja in domain 1 has successfully logged in
2017-05-24 09:26:08,246 INFO [c.c.a.ApiServer] (catalina-exec-17:ctx-ee2c5e26) (logid:a8ca5ee5) Current user logged in under timezone
2017-05-24 09:26:08,246 INFO [c.c.a.ApiServer] (catalina-exec-17:ctx-ee2c5e26) (logid:a8ca5ee5) Timezone offset from UTC is: 0.0
2017-05-24 09:26:08,251 DEBUG [c.c.a.ApiServlet] (catalina-exec-17:ctx-ee2c5e26) (logid:a8ca5ee5) ===END=== 192.168.10.38 -- POST
2017-05-24 09:26:08,320 DEBUG [c.c.a.ApiServlet] (catalina-exec-13:ctx-a1d38347) (logid:3404c663) ===START=== 192.168.10.38 -- GET command=listCapabilities&response=json&_=1495632368256
2017-05-24 09:26:08,325 DEBUG [c.c.a.ApiServer] (catalina-exec-13:ctx-a1d38347 ctx-960796a5) (logid:3404c663) The user with id:31 is not allowed to request the API command or the API command does not exist: listCapabilities
Thanks
Jason
From: Jason Kinsella <ja...@cloudpeople.com.au>
Date: Tuesday, 23 May 2017 at 10:11 pm
To: "users@cloudstack.apache.org" <us...@cloudstack.apache.org>
Subject: SSVM NIO SSL Handshake error
Hi,
We recently upgraded from 4.5.0 to 4.9.2.0 and encountered a problem with the SSVM and Console Proxy. They cannot connect to the management server. The SSVM cloud.log repeats this error every couple of seconds.
2017-05-23 11:58:22,461 INFO [utils.nio.NioClient] (main:null) Connecting to 192.168.12.1:8250
2017-05-23 11:58:22,465 WARN [utils.nio.Link] (main:null) This SSL engine was forced to close inbound due to end of stream.
2017-05-23 11:58:22,465 ERROR [utils.nio.Link] (main:null) Failed to send server's CLOSE message due to socket channel's failure.
2017-05-23 11:58:22,466 ERROR [utils.nio.NioClient] (main:null) SSL Handshake failed while connecting to host: 192.168.12.1 port: 8250
2017-05-23 11:58:22,466 ERROR [utils.nio.NioConnection] (main:null) Unable to initialize the threads.
java.io.IOException: SSL Handshake failed while connecting to host: 192.168.12.1 port: 8250
at com.cloud.utils.nio.NioClient.init(NioClient.java:67)
at com.cloud.utils.nio.NioConnection.start(NioConnection.java:88)
at com.cloud.agent.Agent.start(Agent.java:237)
at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:399)
at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:367)
at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:351)
at com.cloud.agent.AgentShell.start(AgentShell.java:456)
at com.cloud.agent.AgentShell.main(AgentShell.java:491)
2017-05-23 11:58:22,468 INFO [utils.exception.CSExceptionErrorCode] (main:null) Could not find exception: com.cloud.utils.exception.NioConnectionException in error code list for exceptions
2017-05-23 11:58:22,468 WARN [cloud.agent.Agent] (main:null) NIO Connection Exception com.cloud.utils.exception.NioConnectionException: SSL Handshake failed while connecting to host: 192.168.12.1 port: 8250
The setup is very simple. Single management server and ports are open.
Things checked / tried:
· Destroyed SSVM multiple times – still same problem.
· SSH to SSVM from MS using ssh -i /var/cloudstack/management/.ssh/id_rsa -p 3922 root@IPADDRESS – PASS
· SSVM telnet on 8250 to MS – PASS
I’ve also tested a restore of the DB into our working development 4.9.2.0 server. It also exhibits the handshake errors, so most likely DB related.
I’ve used up all my skills. Please help
Regards,
Jason