You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by rv...@privaz.io.INVALID on 2020/08/13 08:06:29 UTC
Console Proxy keeps presenting wrong certificate (another IP)
Hi!
I am deploying my first ACS cluster, debugging the installation procedure step by step.
It is ACS 4.14, Ubuntu 18, KVM, Advanced Networking, Local Primary Storage.
I finally managed to get the Console Proxy working, managed to see the logon of another SVM
Of course, briefly after that I broke it.
The console proxy is in a state in which keeps presenting an invalid certificate (see below). The certificate is for the wrong IP.
I am not certain of what triggered this situation but I suspect restarting the host where this SVM is running.
How do I get the Console VM out of this state?
Rafael.
PS: certificate information
2020-08-13 04:01:23,695 DEBUG [o.a.c.c.p.RootCACustomTrustManager] (pool-489-thread-1:null) (logid:) A client/agent attempting connection from address=10.71.1.64 has presented these certificate(s):
Certificate [1] :
Serial: 9f9d03ab816b6d8d
Not Before:Tue Aug 11 15:20:02 EDT 2020
Not After:Thu Aug 12 03:20:02 EDT 2021
Signature Algorithm:SHA256withRSA
Version:3
Subject DN:C=cloudstack, O=cloudstack, OU=cloudstack, CN=v-2-VM
Issuer DN:CN=ca.cloudstack.apache.org
Alternative Names:[[7, 10.71.1.90], [7, 169.254.7.7], [7, 10.25.100.172], [2, v-2-VM]]
Certificate [2] :
Serial: c60329b2975855de
Not Before:Tue Aug 11 13:58:26 EDT 2020
Not After:Fri Aug 05 01:58:26 EDT 2050
Signature Algorithm:SHA256withRSA
Version:3
Subject DN:CN=ca.cloudstack.apache.org
Issuer DN:CN=ca.cloudstack.apache.org
Alternative Names:null
2020-08-13 04:01:23,721 ERROR [o.a.c.c.p.RootCACustomTrustManager] (pool-489-thread-1:null) (logid:) Certificate ownership verification failed for client: 10.71.1.64
Re: Console Proxy keeps presenting wrong certificate (another IP)
Posted by Rafael del Valle <rv...@privaz.io.INVALID>.
I have found out that the " Empty server certificate chain" is related to firewall rules.
I did, temporarily set IN, OUT and FWD default firewall policies to accept, destroyed the System VMs, and the newly created ones can connect, and report the agent UP.
Rafael.
On Thu, 2020-08-13 11:31 AM, Rafael del Valle <rv...@privaz.io.INVALID> wrote:
> I turns out to be IPs
> my error was to modify system ip reservation strictness without restarting the management server.
> system VMs would start (without a reserved IP) and later on (after management restarts) they would fail to get any IP.
>
> One issue less!
>
> but them, the certificate issue that I reported before is triggering on this cluster too, the good news is that it seems to be easy to reproduce, I am getting:
>
> 2020-08-13 05:25:10,389 ERROR [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-2:null) (logid:) SSL error caught during wrap data: Empty server certificate chain, for local address=/10.71.0.254:8250, remote address=/10.71.1.178:46930.
>
> Just like in the other/physical cluster.
>
> I am going to fiddle a bit with this and see if I find out something.
>
> Rafael
>
> On Thu, 2020-08-13 11:05 AM, Andrija Panic " target="_blank"><an...@gmail.com> wrote:
> > Insufficient capacity exception- can mean MANY things, and usually has
> > nothing to do with the capacity
> > you need to check mgmt logs and see BEFORE the exception happens, what are
> > the lines - they should explain that something is wrong.
> >
> > Best,
> >
> > On Thu, 13 Aug 2020 at 10:43, Rafael del Valle " target="_blank">" target="_blank"><rv...@privaz.io.invalid>
> > wrote:
> >
> > > After waiting for some time ACS finally presented an UI option to destroy
> > > the VM. I think this option is not presented in all states...
> > >
> > > I have destroyed the Proxy VM and it is attempting to create it again, I
> > > guess from the scratch, which seems good to me.
> > >
> > > However, now it feel into another failure loop: Insufficient capacity
> > > exception. Keep destroying and attempting to create the system VMs.
> > >
> > > Which is strange, because the VMs were running before. and the cluster is
> > > plenty of everything: memory, primary (local), ips, etc.
> > >
> > > Any idea what could be going wrong?
> > >
> > > Rafael
> > >
> > >
> > > On Thu, 2020-08-13 10:06 AM, rvalle@privaz.io.INVALID wrote:
> > > > Hi!
> > > >
> > > > I am deploying my first ACS cluster, debugging the installation
> > > procedure step by step.
> > > > It is ACS 4.14, Ubuntu 18, KVM, Advanced Networking, Local Primary
> > > Storage.
> > > >
> > > > I finally managed to get the Console Proxy working, managed to see the
> > > logon of another SVM
> > > >
> > > > Of course, briefly after that I broke it.
> > > >
> > > > The console proxy is in a state in which keeps presenting an invalid
> > > certificate (see below). The certificate is for the wrong IP.
> > > > I am not certain of what triggered this situation but I suspect
> > > restarting the host where this SVM is running.
> > > >
> > > > How do I get the Console VM out of this state?
> > > > Rafael.
> > > >
> > > > PS: certificate information
> > > >
> > > > 2020-08-13 04:01:23,695 DEBUG [o.a.c.c.p.RootCACustomTrustManager]
> > > (pool-489-thread-1:null) (logid:) A client/agent attempting connection from
> > > address=10.71.1.64 has presented these certificate(s):
> > > > Certificate [1] :
> > > > Serial: 9f9d03ab816b6d8d
> > > > Not Before:Tue Aug 11 15:20:02 EDT 2020
> > > > Not After:Thu Aug 12 03:20:02 EDT 2021
> > > > Signature Algorithm:SHA256withRSA
> > > > Version:3
> > > > Subject DN:C=cloudstack, O=cloudstack, OU=cloudstack, CN=v-2-VM
> > > > Issuer DN:CN=ca.cloudstack.apache.org
> > > > Alternative Names:[[7, 10.71.1.90], [7, 169.254.7.7], [7,
> > > 10.25.100.172], [2, v-2-VM]]
> > > > Certificate [2] :
> > > > Serial: c60329b2975855de
> > > > Not Before:Tue Aug 11 13:58:26 EDT 2020
> > > > Not After:Fri Aug 05 01:58:26 EDT 2050
> > > > Signature Algorithm:SHA256withRSA
> > > > Version:3
> > > > Subject DN:CN=ca.cloudstack.apache.org
> > > > Issuer DN:CN=ca.cloudstack.apache.org
> > > > Alternative Names:null
> > > > 2020-08-13 04:01:23,721 ERROR [o.a.c.c.p.RootCACustomTrustManager]
> > > (pool-489-thread-1:null) (logid:) Certificate ownership verification failed
> > > for client: 10.71.1.64
> > > >
> > > >
> > > >
> > > >
> >
> >
> >
> > --
> >
> > Andrija Panić
> >
Re: Console Proxy keeps presenting wrong certificate (another IP)
Posted by Rafael del Valle <rv...@privaz.io.INVALID>.
I turns out to be IPs
my error was to modify system ip reservation strictness without restarting the management server.
system VMs would start (without a reserved IP) and later on (after management restarts) they would fail to get any IP.
One issue less!
but them, the certificate issue that I reported before is triggering on this cluster too, the good news is that it seems to be easy to reproduce, I am getting:
2020-08-13 05:25:10,389 ERROR [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-2:null) (logid:) SSL error caught during wrap data: Empty server certificate chain, for local address=/10.71.0.254:8250, remote address=/10.71.1.178:46930.
Just like in the other/physical cluster.
I am going to fiddle a bit with this and see if I find out something.
Rafael
On Thu, 2020-08-13 11:05 AM, Andrija Panic <an...@gmail.com> wrote:
> Insufficient capacity exception- can mean MANY things, and usually has
> nothing to do with the capacity
> you need to check mgmt logs and see BEFORE the exception happens, what are
> the lines - they should explain that something is wrong.
>
> Best,
>
> On Thu, 13 Aug 2020 at 10:43, Rafael del Valle " target="_blank"><rv...@privaz.io.invalid>
> wrote:
>
> > After waiting for some time ACS finally presented an UI option to destroy
> > the VM. I think this option is not presented in all states...
> >
> > I have destroyed the Proxy VM and it is attempting to create it again, I
> > guess from the scratch, which seems good to me.
> >
> > However, now it feel into another failure loop: Insufficient capacity
> > exception. Keep destroying and attempting to create the system VMs.
> >
> > Which is strange, because the VMs were running before. and the cluster is
> > plenty of everything: memory, primary (local), ips, etc.
> >
> > Any idea what could be going wrong?
> >
> > Rafael
> >
> >
> > On Thu, 2020-08-13 10:06 AM, rvalle@privaz.io.INVALID wrote:
> > > Hi!
> > >
> > > I am deploying my first ACS cluster, debugging the installation
> > procedure step by step.
> > > It is ACS 4.14, Ubuntu 18, KVM, Advanced Networking, Local Primary
> > Storage.
> > >
> > > I finally managed to get the Console Proxy working, managed to see the
> > logon of another SVM
> > >
> > > Of course, briefly after that I broke it.
> > >
> > > The console proxy is in a state in which keeps presenting an invalid
> > certificate (see below). The certificate is for the wrong IP.
> > > I am not certain of what triggered this situation but I suspect
> > restarting the host where this SVM is running.
> > >
> > > How do I get the Console VM out of this state?
> > > Rafael.
> > >
> > > PS: certificate information
> > >
> > > 2020-08-13 04:01:23,695 DEBUG [o.a.c.c.p.RootCACustomTrustManager]
> > (pool-489-thread-1:null) (logid:) A client/agent attempting connection from
> > address=10.71.1.64 has presented these certificate(s):
> > > Certificate [1] :
> > > Serial: 9f9d03ab816b6d8d
> > > Not Before:Tue Aug 11 15:20:02 EDT 2020
> > > Not After:Thu Aug 12 03:20:02 EDT 2021
> > > Signature Algorithm:SHA256withRSA
> > > Version:3
> > > Subject DN:C=cloudstack, O=cloudstack, OU=cloudstack, CN=v-2-VM
> > > Issuer DN:CN=ca.cloudstack.apache.org
> > > Alternative Names:[[7, 10.71.1.90], [7, 169.254.7.7], [7,
> > 10.25.100.172], [2, v-2-VM]]
> > > Certificate [2] :
> > > Serial: c60329b2975855de
> > > Not Before:Tue Aug 11 13:58:26 EDT 2020
> > > Not After:Fri Aug 05 01:58:26 EDT 2050
> > > Signature Algorithm:SHA256withRSA
> > > Version:3
> > > Subject DN:CN=ca.cloudstack.apache.org
> > > Issuer DN:CN=ca.cloudstack.apache.org
> > > Alternative Names:null
> > > 2020-08-13 04:01:23,721 ERROR [o.a.c.c.p.RootCACustomTrustManager]
> > (pool-489-thread-1:null) (logid:) Certificate ownership verification failed
> > for client: 10.71.1.64
> > >
> > >
> > >
> > >
>
>
>
> --
>
> Andrija Panić
>
Re: Console Proxy keeps presenting wrong certificate (another IP)
Posted by Andrija Panic <an...@gmail.com>.
Insufficient capacity exception- can mean MANY things, and usually has
nothing to do with the capacity
you need to check mgmt logs and see BEFORE the exception happens, what are
the lines - they should explain that something is wrong.
Best,
On Thu, 13 Aug 2020 at 10:43, Rafael del Valle <rv...@privaz.io.invalid>
wrote:
> After waiting for some time ACS finally presented an UI option to destroy
> the VM. I think this option is not presented in all states...
>
> I have destroyed the Proxy VM and it is attempting to create it again, I
> guess from the scratch, which seems good to me.
>
> However, now it feel into another failure loop: Insufficient capacity
> exception. Keep destroying and attempting to create the system VMs.
>
> Which is strange, because the VMs were running before. and the cluster is
> plenty of everything: memory, primary (local), ips, etc.
>
> Any idea what could be going wrong?
>
> Rafael
>
>
> On Thu, 2020-08-13 10:06 AM, rvalle@privaz.io.INVALID wrote:
> > Hi!
> >
> > I am deploying my first ACS cluster, debugging the installation
> procedure step by step.
> > It is ACS 4.14, Ubuntu 18, KVM, Advanced Networking, Local Primary
> Storage.
> >
> > I finally managed to get the Console Proxy working, managed to see the
> logon of another SVM
> >
> > Of course, briefly after that I broke it.
> >
> > The console proxy is in a state in which keeps presenting an invalid
> certificate (see below). The certificate is for the wrong IP.
> > I am not certain of what triggered this situation but I suspect
> restarting the host where this SVM is running.
> >
> > How do I get the Console VM out of this state?
> > Rafael.
> >
> > PS: certificate information
> >
> > 2020-08-13 04:01:23,695 DEBUG [o.a.c.c.p.RootCACustomTrustManager]
> (pool-489-thread-1:null) (logid:) A client/agent attempting connection from
> address=10.71.1.64 has presented these certificate(s):
> > Certificate [1] :
> > Serial: 9f9d03ab816b6d8d
> > Not Before:Tue Aug 11 15:20:02 EDT 2020
> > Not After:Thu Aug 12 03:20:02 EDT 2021
> > Signature Algorithm:SHA256withRSA
> > Version:3
> > Subject DN:C=cloudstack, O=cloudstack, OU=cloudstack, CN=v-2-VM
> > Issuer DN:CN=ca.cloudstack.apache.org
> > Alternative Names:[[7, 10.71.1.90], [7, 169.254.7.7], [7,
> 10.25.100.172], [2, v-2-VM]]
> > Certificate [2] :
> > Serial: c60329b2975855de
> > Not Before:Tue Aug 11 13:58:26 EDT 2020
> > Not After:Fri Aug 05 01:58:26 EDT 2050
> > Signature Algorithm:SHA256withRSA
> > Version:3
> > Subject DN:CN=ca.cloudstack.apache.org
> > Issuer DN:CN=ca.cloudstack.apache.org
> > Alternative Names:null
> > 2020-08-13 04:01:23,721 ERROR [o.a.c.c.p.RootCACustomTrustManager]
> (pool-489-thread-1:null) (logid:) Certificate ownership verification failed
> for client: 10.71.1.64
> >
> >
> >
> >
--
Andrija Panić
Re: Console Proxy keeps presenting wrong certificate (another IP)
Posted by Rafael del Valle <rv...@privaz.io.INVALID>.
After waiting for some time ACS finally presented an UI option to destroy the VM. I think this option is not presented in all states...
I have destroyed the Proxy VM and it is attempting to create it again, I guess from the scratch, which seems good to me.
However, now it feel into another failure loop: Insufficient capacity exception. Keep destroying and attempting to create the system VMs.
Which is strange, because the VMs were running before. and the cluster is plenty of everything: memory, primary (local), ips, etc.
Any idea what could be going wrong?
Rafael
On Thu, 2020-08-13 10:06 AM, rvalle@privaz.io.INVALID wrote:
> Hi!
>
> I am deploying my first ACS cluster, debugging the installation procedure step by step.
> It is ACS 4.14, Ubuntu 18, KVM, Advanced Networking, Local Primary Storage.
>
> I finally managed to get the Console Proxy working, managed to see the logon of another SVM
>
> Of course, briefly after that I broke it.
>
> The console proxy is in a state in which keeps presenting an invalid certificate (see below). The certificate is for the wrong IP.
> I am not certain of what triggered this situation but I suspect restarting the host where this SVM is running.
>
> How do I get the Console VM out of this state?
> Rafael.
>
> PS: certificate information
>
> 2020-08-13 04:01:23,695 DEBUG [o.a.c.c.p.RootCACustomTrustManager] (pool-489-thread-1:null) (logid:) A client/agent attempting connection from address=10.71.1.64 has presented these certificate(s):
> Certificate [1] :
> Serial: 9f9d03ab816b6d8d
> Not Before:Tue Aug 11 15:20:02 EDT 2020
> Not After:Thu Aug 12 03:20:02 EDT 2021
> Signature Algorithm:SHA256withRSA
> Version:3
> Subject DN:C=cloudstack, O=cloudstack, OU=cloudstack, CN=v-2-VM
> Issuer DN:CN=ca.cloudstack.apache.org
> Alternative Names:[[7, 10.71.1.90], [7, 169.254.7.7], [7, 10.25.100.172], [2, v-2-VM]]
> Certificate [2] :
> Serial: c60329b2975855de
> Not Before:Tue Aug 11 13:58:26 EDT 2020
> Not After:Fri Aug 05 01:58:26 EDT 2050
> Signature Algorithm:SHA256withRSA
> Version:3
> Subject DN:CN=ca.cloudstack.apache.org
> Issuer DN:CN=ca.cloudstack.apache.org
> Alternative Names:null
> 2020-08-13 04:01:23,721 ERROR [o.a.c.c.p.RootCACustomTrustManager] (pool-489-thread-1:null) (logid:) Certificate ownership verification failed for client: 10.71.1.64
>
>
>
>