You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by cr...@istream.today on 2021/09/23 16:20:21 UTC
Failed to authentication SSH user root on host - repetitive error
Hello,
I was not able to fix this error "Failed to authentication SSH" and
looks like a bug for me, I will explain here why.
My setup before adding new ZONE:
1. Cloudstack 4.15.2 + VMware 6.5 with 1 Basic Network Zone. ( This is
an old setup, it was upgraded frequently, I think from 4.2 )
2. I add new VMware zone, but this one with Advanced Networking, I end
with this repetitive error and nothing else.
3. I decide to add a new zone, the same as I have at (2) with Advanced
Networking, everything works perfect, no issue at systemvm deploy, routers,
instances.
4. I delete the zone from point (2)
5. I add again the same zone, end with the same error.
I have tried multiple things, delete the template, add again, inject,
nothing works, I end with the same error, over and over, and I do not see
the why.
021-09-23 11:53:54,666 ERROR [c.c.h.v.r.VmwareResource]
(DirectAgent-228:ctx-083b9265 lnd-uk-002.test.host, job-35231/job-35263,
cmd: SetupKeyStoreCommand) (logid:df4131be) Command failed due to Exception:
java.lang.Exception
Message: Failed to authentication SSH user root on host 10.15.0.204
2021-09-23 11:53:54,674 ERROR [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-19:ctx-808054ab job-35231/job-35263 ctx-eb7c37bb)
(logid:df4131be) Retrying after catching exception while trying to secure
agent for systemvm id=2025
com.cloud.utils.exception.CloudRuntimeException: Unable to read/process CSR:
Command failed due to Exception: java.lang.Exception
Message: Failed to authentication SSH user root on host 10.15.0.204
Here is the full log : https://pastebin.com/raw/fFfmquVB
Maybe someone have a hint for me .
Regards,
Cristian
RE: Failed to authentication SSH user root on host - repetitive error
Posted by cr...@istream.today.
Hi Pearl,
I did the following multiple times:
1. remove systemvm.iso from secondary storage
2. remove vmdk template from ESXI host and from database spool ref.
3. I have deleted the old systemvm.iso.bak
4 restarted the MS
I didn't tried to change the minor version, how can I do this? In DB?
Is there a problem with this template?
/usr/share/cloudstack-common/scripts/storage/secondary/cloud-install-sys-tmp
lt -m /var/cloudstack/mnt/345049356158.7cfbc14b -u
http://packages.shapeblue.com.s3-eu-west-1.amazonaws.com/systemvmtemplate/4.
15/systemvmtemplate-4.15.1-vmware.ova -h vmware -F
Thank you,
Cristian
-----Original Message-----
From: Pearl d'Silva <pe...@shapeblue.com>
Sent: Friday, September 24, 2021 9:44 AM
To: users@cloudstack.apache.org
Subject: Re: Failed to authentication SSH user root on host - repetitive
error
By any chance, are there any old systemvm.isos in the secondary store, if
yes, can you please delete them. Based on the information you've provided,
it seems like, the key has been injected into systemvm.iso, but during boot
up, a script (cloud-early-config) that sets up the VM before bootstrapping /
patching isn't copying the auth key.
Can you try the steps mentioned under the VMWare section of the doc:
https://cwiki.apache.org/confluence/display/CLOUDSTACK/SystemVm.iso
and see if it helps.
SystemVm.iso - Apache Cloudstack - Apache Software
Foundation<https://cwiki.apache.org/confluence/display/CLOUDSTACK/SystemVm.i
so>
Mechanism of propagating systemvm.iso. Systemvm.iso gets propagated
differently for different hypervisors. It comes packaged as part of the
Cloudstack rpm or is built on dev environments with the command "mvn clean
install -P developer,systemvm" (More info on building cloudstack) .Once the
iso is at the appropriate location depending on the hypervisor it is
inserted as a cd drive and the ...
cwiki.apache.org
Thanks,
Pearl
________________________________
From: cristian.c@istream.today <cr...@istream.today>
Sent: Friday, September 24, 2021 11:45 AM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: RE: Failed to authentication SSH user root on host - repetitive
error
Hi Pearl,
I already did a check via console into the secondary storage vm, I saw
that there are not keys in authorized_keys.
If I try to inject, I see this : /bin/bash
/usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh
/var/cloudstack/management/.ssh/id_rsa.pub
/var/cloudstack/management/.ssh/id_rsa
/usr/share/cloudstack-common/vms/systemvm.iso
mount: /dev/loop0 is write-protected, mounting read-only New public key is
the same as the one in the systemvm.iso, not injecting it, not modifying
systemvm.iso
[root@cloud-emea systemvm]# cd /mnt/iso/ [root@cloud-emea iso]# ls agent.zip
authorized_keys cloud-scripts.tgz [root@cloud-emea iso]# cat
authorized_keys ssh-rsa
AAAAB3NzaC1yc2EAAAABIwAAAQEA0UEA0mRUzKbH1cHHY2GlsaQ18q1KvqfNSV/YqAScZhcPueIl
BbqwPUNznzfSiz/K/+DH8u5QkDIz+fC0Sx+xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx+Bd1Y2U1Tx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx+lYee
+zUh6vszEDwmiq5nTkuCJP1T3o8QL3gUekAFjW7CECsIzLkA41Q8lY0L3qaHBRUJntGGIMtZ
+26AP
IWMC7NHD0wFge3DEN5UhFODcB1f9U7oqa10XqgORjbd88JPfFv/0j92xaaerNpJKw==
cloud@cloud-emea.test.host[root@cloud-emea iso]#
is the same key as I have in : /var/cloudstack/management/.ssh/id_rsa.pub
(For security reason I have replaced characters from ssh pub key)
I'm stuck here, I do not understand the logic.
Regards,
Cristian
-----Original Message-----
From: Pearl d'Silva <pe...@shapeblue.com>
Sent: Friday, September 24, 2021 8:05 AM
To: users@cloudstack.apache.org
Subject: Re: Failed to authentication SSH user root on host - repetitive
error
Hi Cristian,
The exception you are seeing is most likely to happen if the systemvm.iso
hasn't been injected with the Management server's key. One way to validate
it, would be to go to your secondary store - I presume you are working on a
VMware setup - mount the systemvm-4.15.2.iso and verify the authorized_keys
data with what's present on the MS at
/var/cloudstack/management/.ssh/id_rsa.pub.
Thanks,
Pearl
________________________________
From: cristian.c@istream.today <cr...@istream.today>
Sent: Thursday, September 23, 2021 9:50 PM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: Failed to authentication SSH user root on host - repetitive error
Hello,
I was not able to fix this error "Failed to authentication SSH" and
looks like a bug for me, I will explain here why.
My setup before adding new ZONE:
1. Cloudstack 4.15.2 + VMware 6.5 with 1 Basic Network Zone. ( This is
an old setup, it was upgraded frequently, I think from 4.2 )
2. I add new VMware zone, but this one with Advanced Networking, I end
with this repetitive error and nothing else.
3. I decide to add a new zone, the same as I have at (2) with Advanced
Networking, everything works perfect, no issue at systemvm deploy, routers,
instances.
4. I delete the zone from point (2)
5. I add again the same zone, end with the same error.
I have tried multiple things, delete the template, add again, inject,
nothing works, I end with the same error, over and over, and I do not see
the why.
021-09-23 11:53:54,666 ERROR [c.c.h.v.r.VmwareResource]
(DirectAgent-228:ctx-083b9265 lnd-uk-002.test.host, job-35231/job-35263,
cmd: SetupKeyStoreCommand) (logid:df4131be) Command failed due to Exception:
java.lang.Exception
Message: Failed to authentication SSH user root on host 10.15.0.204
2021-09-23 11:53:54,674 ERROR [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-19:ctx-808054ab job-35231/job-35263 ctx-eb7c37bb)
(logid:df4131be) Retrying after catching exception while trying to secure
agent for systemvm id=2025
com.cloud.utils.exception.CloudRuntimeException: Unable to read/process CSR:
Command failed due to Exception: java.lang.Exception
Message: Failed to authentication SSH user root on host 10.15.0.204
Here is the full log : https://pastebin.com/raw/fFfmquVB
Maybe someone have a hint for me .
Regards,
Cristian
RE: Failed to authentication SSH user root on host - repetitive error
Posted by cr...@istream.today.
Hello,
I did this in 3 different new Setup (new Management install, NFS, etc),
no matter if I add as first zone or second zone.
" I have tested this with multiple Cloudstack versions, 4.15.0, 4.15.1,
4.15.2 and VMware, 6.5 and with 6.7 different patches and with the last
patches. I'm 100% that is related to these 2 servers, I do not understand
what is wrong with these servers, this is the problem.
We have 2 identical servers' hardware/ESXI, the only difference is the
location, the not working, are in UK, and any other location we have, DE,
NYC, FR, CA, etc. works fine. But we tested with the exact same servers
which are from DE(Germany) works perfect, with any ACS or VMware version
(also same CIDR range/size for public and private), when we test with UK, we
get same error no matter what."
I want to mention that we have these servers (UK, DE, FR, CA ) from OVH
in vRack and the UK, are not working. ( I manage these server for more
than 5 years )
Regards,
Cristian
-----Original Message-----
From: Harikrishna Patnala <Ha...@shapeblue.com>
Sent: Monday, September 27, 2021 1:29 PM
To: users@cloudstack.apache.org
Subject: Re: Failed to authentication SSH user root on host - repetitive
error
Hi Crisitian,
To my understanding, you are trying to add a new zone and getting issues
with the system VM having some SSH failures.
May I ask you to start everything fresh?
I mean to prepare new secondary storage, seed the system VM template into
new secondary storage and then deploy the zone from the beginning of the
zone creation wizard. Make sure your management server has the latest
packages and systemvm.iso is also as expected to the CloudStack version you
are trying.
Thanks,
Harikrishna
________________________________
From: Cristian Ciobanu <cr...@istream.today>
Sent: Friday, September 24, 2021 5:19 PM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: Re: Failed to authentication SSH user root on host - repetitive
error
Hi Pearl,
I did upgrades multiple times, I always add the template before upgrade.
If was something related to this, i was expecting not to work for all zones,
but right now is working for 2 zones from the same setup, environment.
Regards,
Cristian
On Fri, Sep 24, 2021, 14:40 Pearl d'Silva <pe...@shapeblue.com>
wrote:
> Hi Cristian,
>
> After going through your earlier mails again, and correct me if I am
> wrong, but I noticed in one of your mails you mentioned that the env
> that's causing issues was upgraded from a lower version, right? So,
> ideally when doing upgrades, prior to initiating the upgrade we first
> take a backup of the DB, register the template (new systemVM
> template) following this
> https://docs.cloudstack.apache.org/en/latest/upgrading/upgrade/upgrade
> -4.15.html#update-system-vm-templates
> and then upgrade the environment. However, you mentioned that you used
> the cloud-install-sys-tmplt script to seed and register the template.
> I am not familiar if using that script during upgrades works (it may,
> but I haven't ever used that approach).
> If it is possible, it would be advisable to rollback your env to the
> earlier version (disable your zone, destroy the systemVMs before that)
> and then follow the upgrade steps as mentioned in the documentation:
> https://docs.cloudstack.apache.org/en/latest/upgrading/index.html
>
> Thanks,
> Pearl
>
>
> ________________________________
> From: cristian.c@istream.today <cr...@istream.today>
> Sent: Friday, September 24, 2021 3:08 PM
> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> Subject: RE: Failed to authentication SSH user root on host -
> repetitive error
>
> Hi Pearl,
>
> I just did a SSVM redeploy in the same environment, but different zone
> and and I see the following differences :
>
> "isManaged":"false"}},"name":"ROOT-2055","size":"(2.44 GB)"
>
>
> Working zone - https://pastebin.com/raw/88N6Fg6z
>
> "isManaged":"false"}},"name":"ROOT-2055","size":"(2.44 GB)
>
>
> datastore-6588","managed":"false","storagePort":"0","volumeSize":"(2.4
> 4
> GB)
>
> 021-09-24 05:22:23,959 INFO [c.c.h.v.r.VmwareResource]
> (DirectAgent-437:ctx-2f1519ba fra-de-001.test.host,
> job-25099/job-35371,
> cmd: StartCommand) (logid:d442432d) Mapped disk datastore UUID is not
> the same as the cloned VM datastore UUID: fra-de-001 -
> 9d679070c7a6441592646bfbb82ef1b3
>
>
>
> Not working zone - https://pastebin.com/raw/Um2uHgWM
>
> "isManaged":"false"}},"name":"ROOT-2056","size":"(0 bytes)" here is 0
> bytes...
>
> datastore-6694","managed":"false","storagePort":"0","volumeSize":"(0
> bytes)
> here is 0 bytes...
>
> 2021-09-24 05:00:55,594 INFO [c.c.h.v.r.VmwareResource]
> (DirectAgent-30:ctx-5e49d288 lnd-uk-002.test.host,
> job-25099/job-35365,
> cmd:
> StartCommand) (logid:d442432d) Mapped disk size is not the same as the
> cloned VM disk size: 0 - 2621440000 ( here I see this line, with disk
> size 0 )
>
> 2021-09-24 05:00:55,594 INFO [c.c.h.v.r.VmwareResource]
> (DirectAgent-30:ctx-5e49d288 lnd-uk-002.test.host,
> job-25099/job-35365,
> cmd:
> StartCommand) (logid:d442432d) Mapped disk datastore UUID is not the
> same as the cloned VM datastore UUID: lnd-uk-002 -
> fb7b551a584e4b5192b8f371af56b303
>
>
>
> I do not understand why I have here the size "0"
>
>
> Regards,
> Cristian
>
>
>
>
> -----Original Message-----
> From: Pearl d'Silva <pe...@shapeblue.com>
> Sent: Friday, September 24, 2021 9:44 AM
> To: users@cloudstack.apache.org
> Subject: Re: Failed to authentication SSH user root on host -
> repetitive error
>
> By any chance, are there any old systemvm.isos in the secondary store,
> if yes, can you please delete them. Based on the information you've
> provided, it seems like, the key has been injected into systemvm.iso,
> but during boot up, a script (cloud-early-config) that sets up the VM
> before bootstrapping / patching isn't copying the auth key.
> Can you try the steps mentioned under the VMWare section of the doc:
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/SystemVm.iso
> and see if it helps.
> SystemVm.iso - Apache Cloudstack - Apache Software Foundation<
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/SystemVm.i
> so>
> Mechanism of propagating systemvm.iso. Systemvm.iso gets propagated
> differently for different hypervisors. It comes packaged as part of
> the Cloudstack rpm or is built on dev environments with the command
> "mvn clean install -P developer,systemvm" (More info on building
> cloudstack) .Once the iso is at the appropriate location depending on
> the hypervisor it is inserted as a cd drive and the ...
> cwiki.apache.org
> Thanks,
> Pearl
>
> ________________________________
> From: cristian.c@istream.today <cr...@istream.today>
> Sent: Friday, September 24, 2021 11:45 AM
> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> Subject: RE: Failed to authentication SSH user root on host -
> repetitive error
>
> Hi Pearl,
>
> I already did a check via console into the secondary storage vm,
> I saw that there are not keys in authorized_keys.
>
> If I try to inject, I see this : /bin/bash
> /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh
> /var/cloudstack/management/.ssh/id_rsa.pub
> /var/cloudstack/management/.ssh/id_rsa
> /usr/share/cloudstack-common/vms/systemvm.iso
> mount: /dev/loop0 is write-protected, mounting read-only New public
> key is the same as the one in the systemvm.iso, not injecting it, not
> modifying systemvm.iso
>
>
> [root@cloud-emea systemvm]# cd /mnt/iso/ [root@cloud-emea iso]# ls
> agent.zip authorized_keys cloud-scripts.tgz [root@cloud-emea iso]#
> cat authorized_keys ssh-rsa
>
> AAAAB3NzaC1yc2EAAAABIwAAAQEA0UEA0mRUzKbH1cHHY2GlsaQ18q1KvqfNSV/YqAScZh
> cPueIl
>
> BbqwPUNznzfSiz/K/+DH8u5QkDIz+fC0Sx+xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
> xxxxxx
> xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx+Bd1Y2U1
> xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx+Tx
> xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx+lYee
> +zUh6vszEDwmiq5nTkuCJP1T3o8QL3gUekAFjW7CECsIzLkA41Q8lY0L3qaHBRUJntGGIM
> +tZ
> +26AP
> IWMC7NHD0wFge3DEN5UhFODcB1f9U7oqa10XqgORjbd88JPfFv/0j92xaaerNpJKw==
> cloud@cloud-emea.test.host[root@cloud-emea iso]#
>
> is the same key as I have in :
> /var/cloudstack/management/.ssh/id_rsa.pub
>
> (For security reason I have replaced characters from ssh pub key)
>
>
> I'm stuck here, I do not understand the logic.
>
>
> Regards,
> Cristian
>
>
>
>
> -----Original Message-----
> From: Pearl d'Silva <pe...@shapeblue.com>
> Sent: Friday, September 24, 2021 8:05 AM
> To: users@cloudstack.apache.org
> Subject: Re: Failed to authentication SSH user root on host -
> repetitive error
>
> Hi Cristian,
>
> The exception you are seeing is most likely to happen if the
> systemvm.iso hasn't been injected with the Management server's key.
> One way to validate it, would be to go to your secondary store - I
> presume you are working on a VMware setup - mount the
> systemvm-4.15.2.iso and verify the authorized_keys data with what's
> present on the MS at /var/cloudstack/management/.ssh/id_rsa.pub.
>
> Thanks,
> Pearl
>
> ________________________________
> From: cristian.c@istream.today <cr...@istream.today>
> Sent: Thursday, September 23, 2021 9:50 PM
> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> Subject: Failed to authentication SSH user root on host - repetitive
> error
>
> Hello,
>
>
>
>
>
> I was not able to fix this error "Failed to authentication SSH"
> and looks like a bug for me, I will explain here why.
>
>
>
> My setup before adding new ZONE:
>
>
>
> 1. Cloudstack 4.15.2 + VMware 6.5 with 1 Basic Network Zone. ( This
> is
> an old setup, it was upgraded frequently, I think from 4.2 )
> 2. I add new VMware zone, but this one with Advanced Networking, I
end
> with this repetitive error and nothing else.
> 3. I decide to add a new zone, the same as I have at (2) with
Advanced
> Networking, everything works perfect, no issue at systemvm deploy,
> routers, instances.
> 4. I delete the zone from point (2)
> 5. I add again the same zone, end with the same error.
>
>
>
> I have tried multiple things, delete the template, add again,
> inject, nothing works, I end with the same error, over and over, and I
> do not see the why.
>
>
>
>
>
> 021-09-23 11:53:54,666 ERROR [c.c.h.v.r.VmwareResource]
> (DirectAgent-228:ctx-083b9265 lnd-uk-002.test.host,
> job-35231/job-35263,
> cmd: SetupKeyStoreCommand) (logid:df4131be) Command failed due to
> Exception:
> java.lang.Exception
>
> Message: Failed to authentication SSH user root on host 10.15.0.204
>
>
>
> 2021-09-23 11:53:54,674 ERROR [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-19:ctx-808054ab job-35231/job-35263 ctx-eb7c37bb)
> (logid:df4131be) Retrying after catching exception while trying to
> secure agent for systemvm id=2025
>
> com.cloud.utils.exception.CloudRuntimeException: Unable to
> read/process
> CSR:
> Command failed due to Exception: java.lang.Exception
>
> Message: Failed to authentication SSH user root on host 10.15.0.204
>
>
>
>
>
> Here is the full log : https://pastebin.com/raw/fFfmquVB
>
>
>
>
>
> Maybe someone have a hint for me .
>
>
>
> Regards,
>
> Cristian
>
>
>
>
>
>
>
>
>
>
>
Re: Failed to authentication SSH user root on host -
repetitive error
Posted by Harikrishna Patnala <Ha...@shapeblue.com>.
Hi Crisitian,
To my understanding, you are trying to add a new zone and getting issues with the system VM having some SSH failures.
May I ask you to start everything fresh?
I mean to prepare new secondary storage, seed the system VM template into new secondary storage and then deploy the zone from the beginning of the zone creation wizard. Make sure your management server has the latest packages and systemvm.iso is also as expected to the CloudStack version you are trying.
Thanks,
Harikrishna
________________________________
From: Cristian Ciobanu <cr...@istream.today>
Sent: Friday, September 24, 2021 5:19 PM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: Re: Failed to authentication SSH user root on host - repetitive error
Hi Pearl,
I did upgrades multiple times, I always add the template before
upgrade. If was something related to this, i was expecting not to work for
all zones, but right now is working for 2 zones from the same setup,
environment.
Regards,
Cristian
On Fri, Sep 24, 2021, 14:40 Pearl d'Silva <pe...@shapeblue.com>
wrote:
> Hi Cristian,
>
> After going through your earlier mails again, and correct me if I am
> wrong, but I noticed in one of your mails you mentioned that the env that's
> causing issues was upgraded from a lower version, right? So, ideally when
> doing upgrades, prior to initiating the upgrade we first take a backup of
> the DB, register the template (new systemVM template) following this
> https://docs.cloudstack.apache.org/en/latest/upgrading/upgrade/upgrade-4.15.html#update-system-vm-templates
> and then upgrade the environment. However, you mentioned that you used the
> cloud-install-sys-tmplt script to seed and register the template. I am not
> familiar if using that script during upgrades works (it may, but I haven't
> ever used that approach).
> If it is possible, it would be advisable to rollback your env to the
> earlier version (disable your zone, destroy the systemVMs before that) and
> then follow the upgrade steps as mentioned in the documentation:
> https://docs.cloudstack.apache.org/en/latest/upgrading/index.html
>
> Thanks,
> Pearl
>
>
> ________________________________
> From: cristian.c@istream.today <cr...@istream.today>
> Sent: Friday, September 24, 2021 3:08 PM
> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> Subject: RE: Failed to authentication SSH user root on host - repetitive
> error
>
> Hi Pearl,
>
> I just did a SSVM redeploy in the same environment, but different zone and
> and I see the following differences :
>
> "isManaged":"false"}},"name":"ROOT-2055","size":"(2.44 GB)"
>
>
> Working zone - https://pastebin.com/raw/88N6Fg6z
>
> "isManaged":"false"}},"name":"ROOT-2055","size":"(2.44 GB)
>
> datastore-6588","managed":"false","storagePort":"0","volumeSize":"(2.44
> GB)
>
> 021-09-24 05:22:23,959 INFO [c.c.h.v.r.VmwareResource]
> (DirectAgent-437:ctx-2f1519ba fra-de-001.test.host, job-25099/job-35371,
> cmd: StartCommand) (logid:d442432d) Mapped disk datastore UUID is not the
> same as the cloned VM datastore UUID: fra-de-001 -
> 9d679070c7a6441592646bfbb82ef1b3
>
>
>
> Not working zone - https://pastebin.com/raw/Um2uHgWM
>
> "isManaged":"false"}},"name":"ROOT-2056","size":"(0 bytes)" here is 0
> bytes...
>
> datastore-6694","managed":"false","storagePort":"0","volumeSize":"(0
> bytes)
> here is 0 bytes...
>
> 2021-09-24 05:00:55,594 INFO [c.c.h.v.r.VmwareResource]
> (DirectAgent-30:ctx-5e49d288 lnd-uk-002.test.host, job-25099/job-35365,
> cmd:
> StartCommand) (logid:d442432d) Mapped disk size is not the same as the
> cloned VM disk size: 0 - 2621440000 ( here I see this line, with disk
> size 0 )
>
> 2021-09-24 05:00:55,594 INFO [c.c.h.v.r.VmwareResource]
> (DirectAgent-30:ctx-5e49d288 lnd-uk-002.test.host, job-25099/job-35365,
> cmd:
> StartCommand) (logid:d442432d) Mapped disk datastore UUID is not the same
> as
> the cloned VM datastore UUID: lnd-uk-002 - fb7b551a584e4b5192b8f371af56b303
>
>
>
> I do not understand why I have here the size "0"
>
>
> Regards,
> Cristian
>
>
>
>
> -----Original Message-----
> From: Pearl d'Silva <pe...@shapeblue.com>
> Sent: Friday, September 24, 2021 9:44 AM
> To: users@cloudstack.apache.org
> Subject: Re: Failed to authentication SSH user root on host - repetitive
> error
>
> By any chance, are there any old systemvm.isos in the secondary store, if
> yes, can you please delete them. Based on the information you've provided,
> it seems like, the key has been injected into systemvm.iso, but during boot
> up, a script (cloud-early-config) that sets up the VM before bootstrapping
> /
> patching isn't copying the auth key.
> Can you try the steps mentioned under the VMWare section of the doc:
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/SystemVm.iso
> and see if it helps.
> SystemVm.iso - Apache Cloudstack - Apache Software
> Foundation<
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/SystemVm.i
> so>
> Mechanism of propagating systemvm.iso. Systemvm.iso gets propagated
> differently for different hypervisors. It comes packaged as part of the
> Cloudstack rpm or is built on dev environments with the command "mvn clean
> install -P developer,systemvm" (More info on building cloudstack) .Once the
> iso is at the appropriate location depending on the hypervisor it is
> inserted as a cd drive and the ...
> cwiki.apache.org
> Thanks,
> Pearl
>
> ________________________________
> From: cristian.c@istream.today <cr...@istream.today>
> Sent: Friday, September 24, 2021 11:45 AM
> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> Subject: RE: Failed to authentication SSH user root on host - repetitive
> error
>
> Hi Pearl,
>
> I already did a check via console into the secondary storage vm, I
> saw
> that there are not keys in authorized_keys.
>
> If I try to inject, I see this : /bin/bash
> /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh
> /var/cloudstack/management/.ssh/id_rsa.pub
> /var/cloudstack/management/.ssh/id_rsa
> /usr/share/cloudstack-common/vms/systemvm.iso
> mount: /dev/loop0 is write-protected, mounting read-only New public key is
> the same as the one in the systemvm.iso, not injecting it, not modifying
> systemvm.iso
>
>
> [root@cloud-emea systemvm]# cd /mnt/iso/ [root@cloud-emea iso]# ls
> agent.zip
> authorized_keys cloud-scripts.tgz [root@cloud-emea iso]# cat
> authorized_keys ssh-rsa
>
> AAAAB3NzaC1yc2EAAAABIwAAAQEA0UEA0mRUzKbH1cHHY2GlsaQ18q1KvqfNSV/YqAScZhcPueIl
>
> BbqwPUNznzfSiz/K/+DH8u5QkDIz+fC0Sx+xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
> xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx+Bd1Y2U1Tx
> xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx+lYee
> +zUh6vszEDwmiq5nTkuCJP1T3o8QL3gUekAFjW7CECsIzLkA41Q8lY0L3qaHBRUJntGGIMtZ
> +26AP
> IWMC7NHD0wFge3DEN5UhFODcB1f9U7oqa10XqgORjbd88JPfFv/0j92xaaerNpJKw==
> cloud@cloud-emea.test.host[root@cloud-emea iso]#
>
> is the same key as I have in : /var/cloudstack/management/.ssh/id_rsa.pub
>
> (For security reason I have replaced characters from ssh pub key)
>
>
> I'm stuck here, I do not understand the logic.
>
>
> Regards,
> Cristian
>
>
>
>
> -----Original Message-----
> From: Pearl d'Silva <pe...@shapeblue.com>
> Sent: Friday, September 24, 2021 8:05 AM
> To: users@cloudstack.apache.org
> Subject: Re: Failed to authentication SSH user root on host - repetitive
> error
>
> Hi Cristian,
>
> The exception you are seeing is most likely to happen if the systemvm.iso
> hasn't been injected with the Management server's key. One way to validate
> it, would be to go to your secondary store - I presume you are working on a
> VMware setup - mount the systemvm-4.15.2.iso and verify the authorized_keys
> data with what's present on the MS at
> /var/cloudstack/management/.ssh/id_rsa.pub.
>
> Thanks,
> Pearl
>
> ________________________________
> From: cristian.c@istream.today <cr...@istream.today>
> Sent: Thursday, September 23, 2021 9:50 PM
> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> Subject: Failed to authentication SSH user root on host - repetitive error
>
> Hello,
>
>
>
>
>
> I was not able to fix this error "Failed to authentication SSH" and
> looks like a bug for me, I will explain here why.
>
>
>
> My setup before adding new ZONE:
>
>
>
> 1. Cloudstack 4.15.2 + VMware 6.5 with 1 Basic Network Zone. ( This
> is
> an old setup, it was upgraded frequently, I think from 4.2 )
> 2. I add new VMware zone, but this one with Advanced Networking, I end
> with this repetitive error and nothing else.
> 3. I decide to add a new zone, the same as I have at (2) with Advanced
> Networking, everything works perfect, no issue at systemvm deploy, routers,
> instances.
> 4. I delete the zone from point (2)
> 5. I add again the same zone, end with the same error.
>
>
>
> I have tried multiple things, delete the template, add again, inject,
> nothing works, I end with the same error, over and over, and I do not see
> the why.
>
>
>
>
>
> 021-09-23 11:53:54,666 ERROR [c.c.h.v.r.VmwareResource]
> (DirectAgent-228:ctx-083b9265 lnd-uk-002.test.host, job-35231/job-35263,
> cmd: SetupKeyStoreCommand) (logid:df4131be) Command failed due to
> Exception:
> java.lang.Exception
>
> Message: Failed to authentication SSH user root on host 10.15.0.204
>
>
>
> 2021-09-23 11:53:54,674 ERROR [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-19:ctx-808054ab job-35231/job-35263 ctx-eb7c37bb)
> (logid:df4131be) Retrying after catching exception while trying to secure
> agent for systemvm id=2025
>
> com.cloud.utils.exception.CloudRuntimeException: Unable to read/process
> CSR:
> Command failed due to Exception: java.lang.Exception
>
> Message: Failed to authentication SSH user root on host 10.15.0.204
>
>
>
>
>
> Here is the full log : https://pastebin.com/raw/fFfmquVB
>
>
>
>
>
> Maybe someone have a hint for me .
>
>
>
> Regards,
>
> Cristian
>
>
>
>
>
>
>
>
>
>
>
Re: Failed to authentication SSH user root on host - repetitive error
Posted by Cristian Ciobanu <cr...@istream.today>.
Hi Pearl,
I did upgrades multiple times, I always add the template before
upgrade. If was something related to this, i was expecting not to work for
all zones, but right now is working for 2 zones from the same setup,
environment.
Regards,
Cristian
On Fri, Sep 24, 2021, 14:40 Pearl d'Silva <pe...@shapeblue.com>
wrote:
> Hi Cristian,
>
> After going through your earlier mails again, and correct me if I am
> wrong, but I noticed in one of your mails you mentioned that the env that's
> causing issues was upgraded from a lower version, right? So, ideally when
> doing upgrades, prior to initiating the upgrade we first take a backup of
> the DB, register the template (new systemVM template) following this
> https://docs.cloudstack.apache.org/en/latest/upgrading/upgrade/upgrade-4.15.html#update-system-vm-templates
> and then upgrade the environment. However, you mentioned that you used the
> cloud-install-sys-tmplt script to seed and register the template. I am not
> familiar if using that script during upgrades works (it may, but I haven't
> ever used that approach).
> If it is possible, it would be advisable to rollback your env to the
> earlier version (disable your zone, destroy the systemVMs before that) and
> then follow the upgrade steps as mentioned in the documentation:
> https://docs.cloudstack.apache.org/en/latest/upgrading/index.html
>
> Thanks,
> Pearl
>
>
> ________________________________
> From: cristian.c@istream.today <cr...@istream.today>
> Sent: Friday, September 24, 2021 3:08 PM
> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> Subject: RE: Failed to authentication SSH user root on host - repetitive
> error
>
> Hi Pearl,
>
> I just did a SSVM redeploy in the same environment, but different zone and
> and I see the following differences :
>
> "isManaged":"false"}},"name":"ROOT-2055","size":"(2.44 GB)"
>
>
> Working zone - https://pastebin.com/raw/88N6Fg6z
>
> "isManaged":"false"}},"name":"ROOT-2055","size":"(2.44 GB)
>
> datastore-6588","managed":"false","storagePort":"0","volumeSize":"(2.44
> GB)
>
> 021-09-24 05:22:23,959 INFO [c.c.h.v.r.VmwareResource]
> (DirectAgent-437:ctx-2f1519ba fra-de-001.test.host, job-25099/job-35371,
> cmd: StartCommand) (logid:d442432d) Mapped disk datastore UUID is not the
> same as the cloned VM datastore UUID: fra-de-001 -
> 9d679070c7a6441592646bfbb82ef1b3
>
>
>
> Not working zone - https://pastebin.com/raw/Um2uHgWM
>
> "isManaged":"false"}},"name":"ROOT-2056","size":"(0 bytes)" here is 0
> bytes...
>
> datastore-6694","managed":"false","storagePort":"0","volumeSize":"(0
> bytes)
> here is 0 bytes...
>
> 2021-09-24 05:00:55,594 INFO [c.c.h.v.r.VmwareResource]
> (DirectAgent-30:ctx-5e49d288 lnd-uk-002.test.host, job-25099/job-35365,
> cmd:
> StartCommand) (logid:d442432d) Mapped disk size is not the same as the
> cloned VM disk size: 0 - 2621440000 ( here I see this line, with disk
> size 0 )
>
> 2021-09-24 05:00:55,594 INFO [c.c.h.v.r.VmwareResource]
> (DirectAgent-30:ctx-5e49d288 lnd-uk-002.test.host, job-25099/job-35365,
> cmd:
> StartCommand) (logid:d442432d) Mapped disk datastore UUID is not the same
> as
> the cloned VM datastore UUID: lnd-uk-002 - fb7b551a584e4b5192b8f371af56b303
>
>
>
> I do not understand why I have here the size "0"
>
>
> Regards,
> Cristian
>
>
>
>
> -----Original Message-----
> From: Pearl d'Silva <pe...@shapeblue.com>
> Sent: Friday, September 24, 2021 9:44 AM
> To: users@cloudstack.apache.org
> Subject: Re: Failed to authentication SSH user root on host - repetitive
> error
>
> By any chance, are there any old systemvm.isos in the secondary store, if
> yes, can you please delete them. Based on the information you've provided,
> it seems like, the key has been injected into systemvm.iso, but during boot
> up, a script (cloud-early-config) that sets up the VM before bootstrapping
> /
> patching isn't copying the auth key.
> Can you try the steps mentioned under the VMWare section of the doc:
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/SystemVm.iso
> and see if it helps.
> SystemVm.iso - Apache Cloudstack - Apache Software
> Foundation<
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/SystemVm.i
> so>
> Mechanism of propagating systemvm.iso. Systemvm.iso gets propagated
> differently for different hypervisors. It comes packaged as part of the
> Cloudstack rpm or is built on dev environments with the command "mvn clean
> install -P developer,systemvm" (More info on building cloudstack) .Once the
> iso is at the appropriate location depending on the hypervisor it is
> inserted as a cd drive and the ...
> cwiki.apache.org
> Thanks,
> Pearl
>
> ________________________________
> From: cristian.c@istream.today <cr...@istream.today>
> Sent: Friday, September 24, 2021 11:45 AM
> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> Subject: RE: Failed to authentication SSH user root on host - repetitive
> error
>
> Hi Pearl,
>
> I already did a check via console into the secondary storage vm, I
> saw
> that there are not keys in authorized_keys.
>
> If I try to inject, I see this : /bin/bash
> /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh
> /var/cloudstack/management/.ssh/id_rsa.pub
> /var/cloudstack/management/.ssh/id_rsa
> /usr/share/cloudstack-common/vms/systemvm.iso
> mount: /dev/loop0 is write-protected, mounting read-only New public key is
> the same as the one in the systemvm.iso, not injecting it, not modifying
> systemvm.iso
>
>
> [root@cloud-emea systemvm]# cd /mnt/iso/ [root@cloud-emea iso]# ls
> agent.zip
> authorized_keys cloud-scripts.tgz [root@cloud-emea iso]# cat
> authorized_keys ssh-rsa
>
> AAAAB3NzaC1yc2EAAAABIwAAAQEA0UEA0mRUzKbH1cHHY2GlsaQ18q1KvqfNSV/YqAScZhcPueIl
>
> BbqwPUNznzfSiz/K/+DH8u5QkDIz+fC0Sx+xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
> xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx+Bd1Y2U1Tx
> xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx+lYee
> +zUh6vszEDwmiq5nTkuCJP1T3o8QL3gUekAFjW7CECsIzLkA41Q8lY0L3qaHBRUJntGGIMtZ
> +26AP
> IWMC7NHD0wFge3DEN5UhFODcB1f9U7oqa10XqgORjbd88JPfFv/0j92xaaerNpJKw==
> cloud@cloud-emea.test.host[root@cloud-emea iso]#
>
> is the same key as I have in : /var/cloudstack/management/.ssh/id_rsa.pub
>
> (For security reason I have replaced characters from ssh pub key)
>
>
> I'm stuck here, I do not understand the logic.
>
>
> Regards,
> Cristian
>
>
>
>
> -----Original Message-----
> From: Pearl d'Silva <pe...@shapeblue.com>
> Sent: Friday, September 24, 2021 8:05 AM
> To: users@cloudstack.apache.org
> Subject: Re: Failed to authentication SSH user root on host - repetitive
> error
>
> Hi Cristian,
>
> The exception you are seeing is most likely to happen if the systemvm.iso
> hasn't been injected with the Management server's key. One way to validate
> it, would be to go to your secondary store - I presume you are working on a
> VMware setup - mount the systemvm-4.15.2.iso and verify the authorized_keys
> data with what's present on the MS at
> /var/cloudstack/management/.ssh/id_rsa.pub.
>
> Thanks,
> Pearl
>
> ________________________________
> From: cristian.c@istream.today <cr...@istream.today>
> Sent: Thursday, September 23, 2021 9:50 PM
> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> Subject: Failed to authentication SSH user root on host - repetitive error
>
> Hello,
>
>
>
>
>
> I was not able to fix this error "Failed to authentication SSH" and
> looks like a bug for me, I will explain here why.
>
>
>
> My setup before adding new ZONE:
>
>
>
> 1. Cloudstack 4.15.2 + VMware 6.5 with 1 Basic Network Zone. ( This
> is
> an old setup, it was upgraded frequently, I think from 4.2 )
> 2. I add new VMware zone, but this one with Advanced Networking, I end
> with this repetitive error and nothing else.
> 3. I decide to add a new zone, the same as I have at (2) with Advanced
> Networking, everything works perfect, no issue at systemvm deploy, routers,
> instances.
> 4. I delete the zone from point (2)
> 5. I add again the same zone, end with the same error.
>
>
>
> I have tried multiple things, delete the template, add again, inject,
> nothing works, I end with the same error, over and over, and I do not see
> the why.
>
>
>
>
>
> 021-09-23 11:53:54,666 ERROR [c.c.h.v.r.VmwareResource]
> (DirectAgent-228:ctx-083b9265 lnd-uk-002.test.host, job-35231/job-35263,
> cmd: SetupKeyStoreCommand) (logid:df4131be) Command failed due to
> Exception:
> java.lang.Exception
>
> Message: Failed to authentication SSH user root on host 10.15.0.204
>
>
>
> 2021-09-23 11:53:54,674 ERROR [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-19:ctx-808054ab job-35231/job-35263 ctx-eb7c37bb)
> (logid:df4131be) Retrying after catching exception while trying to secure
> agent for systemvm id=2025
>
> com.cloud.utils.exception.CloudRuntimeException: Unable to read/process
> CSR:
> Command failed due to Exception: java.lang.Exception
>
> Message: Failed to authentication SSH user root on host 10.15.0.204
>
>
>
>
>
> Here is the full log : https://pastebin.com/raw/fFfmquVB
>
>
>
>
>
> Maybe someone have a hint for me .
>
>
>
> Regards,
>
> Cristian
>
>
>
>
>
>
>
>
>
>
>
Re: Failed to authentication SSH user root on host -
repetitive error
Posted by Pearl d'Silva <pe...@shapeblue.com>.
Hi Cristian,
After going through your earlier mails again, and correct me if I am wrong, but I noticed in one of your mails you mentioned that the env that's causing issues was upgraded from a lower version, right? So, ideally when doing upgrades, prior to initiating the upgrade we first take a backup of the DB, register the template (new systemVM template) following this https://docs.cloudstack.apache.org/en/latest/upgrading/upgrade/upgrade-4.15.html#update-system-vm-templates and then upgrade the environment. However, you mentioned that you used the cloud-install-sys-tmplt script to seed and register the template. I am not familiar if using that script during upgrades works (it may, but I haven't ever used that approach).
If it is possible, it would be advisable to rollback your env to the earlier version (disable your zone, destroy the systemVMs before that) and then follow the upgrade steps as mentioned in the documentation: https://docs.cloudstack.apache.org/en/latest/upgrading/index.html
Thanks,
Pearl
________________________________
From: cristian.c@istream.today <cr...@istream.today>
Sent: Friday, September 24, 2021 3:08 PM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: RE: Failed to authentication SSH user root on host - repetitive error
Hi Pearl,
I just did a SSVM redeploy in the same environment, but different zone and
and I see the following differences :
"isManaged":"false"}},"name":"ROOT-2055","size":"(2.44 GB)"
Working zone - https://pastebin.com/raw/88N6Fg6z
"isManaged":"false"}},"name":"ROOT-2055","size":"(2.44 GB)
datastore-6588","managed":"false","storagePort":"0","volumeSize":"(2.44 GB)
021-09-24 05:22:23,959 INFO [c.c.h.v.r.VmwareResource]
(DirectAgent-437:ctx-2f1519ba fra-de-001.test.host, job-25099/job-35371,
cmd: StartCommand) (logid:d442432d) Mapped disk datastore UUID is not the
same as the cloned VM datastore UUID: fra-de-001 -
9d679070c7a6441592646bfbb82ef1b3
Not working zone - https://pastebin.com/raw/Um2uHgWM
"isManaged":"false"}},"name":"ROOT-2056","size":"(0 bytes)" here is 0
bytes...
datastore-6694","managed":"false","storagePort":"0","volumeSize":"(0 bytes)
here is 0 bytes...
2021-09-24 05:00:55,594 INFO [c.c.h.v.r.VmwareResource]
(DirectAgent-30:ctx-5e49d288 lnd-uk-002.test.host, job-25099/job-35365, cmd:
StartCommand) (logid:d442432d) Mapped disk size is not the same as the
cloned VM disk size: 0 - 2621440000 ( here I see this line, with disk
size 0 )
2021-09-24 05:00:55,594 INFO [c.c.h.v.r.VmwareResource]
(DirectAgent-30:ctx-5e49d288 lnd-uk-002.test.host, job-25099/job-35365, cmd:
StartCommand) (logid:d442432d) Mapped disk datastore UUID is not the same as
the cloned VM datastore UUID: lnd-uk-002 - fb7b551a584e4b5192b8f371af56b303
I do not understand why I have here the size "0"
Regards,
Cristian
-----Original Message-----
From: Pearl d'Silva <pe...@shapeblue.com>
Sent: Friday, September 24, 2021 9:44 AM
To: users@cloudstack.apache.org
Subject: Re: Failed to authentication SSH user root on host - repetitive
error
By any chance, are there any old systemvm.isos in the secondary store, if
yes, can you please delete them. Based on the information you've provided,
it seems like, the key has been injected into systemvm.iso, but during boot
up, a script (cloud-early-config) that sets up the VM before bootstrapping /
patching isn't copying the auth key.
Can you try the steps mentioned under the VMWare section of the doc:
https://cwiki.apache.org/confluence/display/CLOUDSTACK/SystemVm.iso
and see if it helps.
SystemVm.iso - Apache Cloudstack - Apache Software
Foundation<https://cwiki.apache.org/confluence/display/CLOUDSTACK/SystemVm.i
so>
Mechanism of propagating systemvm.iso. Systemvm.iso gets propagated
differently for different hypervisors. It comes packaged as part of the
Cloudstack rpm or is built on dev environments with the command "mvn clean
install -P developer,systemvm" (More info on building cloudstack) .Once the
iso is at the appropriate location depending on the hypervisor it is
inserted as a cd drive and the ...
cwiki.apache.org
Thanks,
Pearl
________________________________
From: cristian.c@istream.today <cr...@istream.today>
Sent: Friday, September 24, 2021 11:45 AM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: RE: Failed to authentication SSH user root on host - repetitive
error
Hi Pearl,
I already did a check via console into the secondary storage vm, I saw
that there are not keys in authorized_keys.
If I try to inject, I see this : /bin/bash
/usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh
/var/cloudstack/management/.ssh/id_rsa.pub
/var/cloudstack/management/.ssh/id_rsa
/usr/share/cloudstack-common/vms/systemvm.iso
mount: /dev/loop0 is write-protected, mounting read-only New public key is
the same as the one in the systemvm.iso, not injecting it, not modifying
systemvm.iso
[root@cloud-emea systemvm]# cd /mnt/iso/ [root@cloud-emea iso]# ls agent.zip
authorized_keys cloud-scripts.tgz [root@cloud-emea iso]# cat
authorized_keys ssh-rsa
AAAAB3NzaC1yc2EAAAABIwAAAQEA0UEA0mRUzKbH1cHHY2GlsaQ18q1KvqfNSV/YqAScZhcPueIl
BbqwPUNznzfSiz/K/+DH8u5QkDIz+fC0Sx+xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx+Bd1Y2U1Tx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx+lYee
+zUh6vszEDwmiq5nTkuCJP1T3o8QL3gUekAFjW7CECsIzLkA41Q8lY0L3qaHBRUJntGGIMtZ
+26AP
IWMC7NHD0wFge3DEN5UhFODcB1f9U7oqa10XqgORjbd88JPfFv/0j92xaaerNpJKw==
cloud@cloud-emea.test.host[root@cloud-emea iso]#
is the same key as I have in : /var/cloudstack/management/.ssh/id_rsa.pub
(For security reason I have replaced characters from ssh pub key)
I'm stuck here, I do not understand the logic.
Regards,
Cristian
-----Original Message-----
From: Pearl d'Silva <pe...@shapeblue.com>
Sent: Friday, September 24, 2021 8:05 AM
To: users@cloudstack.apache.org
Subject: Re: Failed to authentication SSH user root on host - repetitive
error
Hi Cristian,
The exception you are seeing is most likely to happen if the systemvm.iso
hasn't been injected with the Management server's key. One way to validate
it, would be to go to your secondary store - I presume you are working on a
VMware setup - mount the systemvm-4.15.2.iso and verify the authorized_keys
data with what's present on the MS at
/var/cloudstack/management/.ssh/id_rsa.pub.
Thanks,
Pearl
________________________________
From: cristian.c@istream.today <cr...@istream.today>
Sent: Thursday, September 23, 2021 9:50 PM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: Failed to authentication SSH user root on host - repetitive error
Hello,
I was not able to fix this error "Failed to authentication SSH" and
looks like a bug for me, I will explain here why.
My setup before adding new ZONE:
1. Cloudstack 4.15.2 + VMware 6.5 with 1 Basic Network Zone. ( This is
an old setup, it was upgraded frequently, I think from 4.2 )
2. I add new VMware zone, but this one with Advanced Networking, I end
with this repetitive error and nothing else.
3. I decide to add a new zone, the same as I have at (2) with Advanced
Networking, everything works perfect, no issue at systemvm deploy, routers,
instances.
4. I delete the zone from point (2)
5. I add again the same zone, end with the same error.
I have tried multiple things, delete the template, add again, inject,
nothing works, I end with the same error, over and over, and I do not see
the why.
021-09-23 11:53:54,666 ERROR [c.c.h.v.r.VmwareResource]
(DirectAgent-228:ctx-083b9265 lnd-uk-002.test.host, job-35231/job-35263,
cmd: SetupKeyStoreCommand) (logid:df4131be) Command failed due to Exception:
java.lang.Exception
Message: Failed to authentication SSH user root on host 10.15.0.204
2021-09-23 11:53:54,674 ERROR [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-19:ctx-808054ab job-35231/job-35263 ctx-eb7c37bb)
(logid:df4131be) Retrying after catching exception while trying to secure
agent for systemvm id=2025
com.cloud.utils.exception.CloudRuntimeException: Unable to read/process CSR:
Command failed due to Exception: java.lang.Exception
Message: Failed to authentication SSH user root on host 10.15.0.204
Here is the full log : https://pastebin.com/raw/fFfmquVB
Maybe someone have a hint for me .
Regards,
Cristian
RE: Failed to authentication SSH user root on host - repetitive error
Posted by cr...@istream.today.
Hi Pearl,
I just did a SSVM redeploy in the same environment, but different zone and
and I see the following differences :
"isManaged":"false"}},"name":"ROOT-2055","size":"(2.44 GB)"
Working zone - https://pastebin.com/raw/88N6Fg6z
"isManaged":"false"}},"name":"ROOT-2055","size":"(2.44 GB)
datastore-6588","managed":"false","storagePort":"0","volumeSize":"(2.44 GB)
021-09-24 05:22:23,959 INFO [c.c.h.v.r.VmwareResource]
(DirectAgent-437:ctx-2f1519ba fra-de-001.test.host, job-25099/job-35371,
cmd: StartCommand) (logid:d442432d) Mapped disk datastore UUID is not the
same as the cloned VM datastore UUID: fra-de-001 -
9d679070c7a6441592646bfbb82ef1b3
Not working zone - https://pastebin.com/raw/Um2uHgWM
"isManaged":"false"}},"name":"ROOT-2056","size":"(0 bytes)" here is 0
bytes...
datastore-6694","managed":"false","storagePort":"0","volumeSize":"(0 bytes)
here is 0 bytes...
2021-09-24 05:00:55,594 INFO [c.c.h.v.r.VmwareResource]
(DirectAgent-30:ctx-5e49d288 lnd-uk-002.test.host, job-25099/job-35365, cmd:
StartCommand) (logid:d442432d) Mapped disk size is not the same as the
cloned VM disk size: 0 - 2621440000 ( here I see this line, with disk
size 0 )
2021-09-24 05:00:55,594 INFO [c.c.h.v.r.VmwareResource]
(DirectAgent-30:ctx-5e49d288 lnd-uk-002.test.host, job-25099/job-35365, cmd:
StartCommand) (logid:d442432d) Mapped disk datastore UUID is not the same as
the cloned VM datastore UUID: lnd-uk-002 - fb7b551a584e4b5192b8f371af56b303
I do not understand why I have here the size "0"
Regards,
Cristian
-----Original Message-----
From: Pearl d'Silva <pe...@shapeblue.com>
Sent: Friday, September 24, 2021 9:44 AM
To: users@cloudstack.apache.org
Subject: Re: Failed to authentication SSH user root on host - repetitive
error
By any chance, are there any old systemvm.isos in the secondary store, if
yes, can you please delete them. Based on the information you've provided,
it seems like, the key has been injected into systemvm.iso, but during boot
up, a script (cloud-early-config) that sets up the VM before bootstrapping /
patching isn't copying the auth key.
Can you try the steps mentioned under the VMWare section of the doc:
https://cwiki.apache.org/confluence/display/CLOUDSTACK/SystemVm.iso
and see if it helps.
SystemVm.iso - Apache Cloudstack - Apache Software
Foundation<https://cwiki.apache.org/confluence/display/CLOUDSTACK/SystemVm.i
so>
Mechanism of propagating systemvm.iso. Systemvm.iso gets propagated
differently for different hypervisors. It comes packaged as part of the
Cloudstack rpm or is built on dev environments with the command "mvn clean
install -P developer,systemvm" (More info on building cloudstack) .Once the
iso is at the appropriate location depending on the hypervisor it is
inserted as a cd drive and the ...
cwiki.apache.org
Thanks,
Pearl
________________________________
From: cristian.c@istream.today <cr...@istream.today>
Sent: Friday, September 24, 2021 11:45 AM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: RE: Failed to authentication SSH user root on host - repetitive
error
Hi Pearl,
I already did a check via console into the secondary storage vm, I saw
that there are not keys in authorized_keys.
If I try to inject, I see this : /bin/bash
/usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh
/var/cloudstack/management/.ssh/id_rsa.pub
/var/cloudstack/management/.ssh/id_rsa
/usr/share/cloudstack-common/vms/systemvm.iso
mount: /dev/loop0 is write-protected, mounting read-only New public key is
the same as the one in the systemvm.iso, not injecting it, not modifying
systemvm.iso
[root@cloud-emea systemvm]# cd /mnt/iso/ [root@cloud-emea iso]# ls agent.zip
authorized_keys cloud-scripts.tgz [root@cloud-emea iso]# cat
authorized_keys ssh-rsa
AAAAB3NzaC1yc2EAAAABIwAAAQEA0UEA0mRUzKbH1cHHY2GlsaQ18q1KvqfNSV/YqAScZhcPueIl
BbqwPUNznzfSiz/K/+DH8u5QkDIz+fC0Sx+xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx+Bd1Y2U1Tx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx+lYee
+zUh6vszEDwmiq5nTkuCJP1T3o8QL3gUekAFjW7CECsIzLkA41Q8lY0L3qaHBRUJntGGIMtZ
+26AP
IWMC7NHD0wFge3DEN5UhFODcB1f9U7oqa10XqgORjbd88JPfFv/0j92xaaerNpJKw==
cloud@cloud-emea.test.host[root@cloud-emea iso]#
is the same key as I have in : /var/cloudstack/management/.ssh/id_rsa.pub
(For security reason I have replaced characters from ssh pub key)
I'm stuck here, I do not understand the logic.
Regards,
Cristian
-----Original Message-----
From: Pearl d'Silva <pe...@shapeblue.com>
Sent: Friday, September 24, 2021 8:05 AM
To: users@cloudstack.apache.org
Subject: Re: Failed to authentication SSH user root on host - repetitive
error
Hi Cristian,
The exception you are seeing is most likely to happen if the systemvm.iso
hasn't been injected with the Management server's key. One way to validate
it, would be to go to your secondary store - I presume you are working on a
VMware setup - mount the systemvm-4.15.2.iso and verify the authorized_keys
data with what's present on the MS at
/var/cloudstack/management/.ssh/id_rsa.pub.
Thanks,
Pearl
________________________________
From: cristian.c@istream.today <cr...@istream.today>
Sent: Thursday, September 23, 2021 9:50 PM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: Failed to authentication SSH user root on host - repetitive error
Hello,
I was not able to fix this error "Failed to authentication SSH" and
looks like a bug for me, I will explain here why.
My setup before adding new ZONE:
1. Cloudstack 4.15.2 + VMware 6.5 with 1 Basic Network Zone. ( This is
an old setup, it was upgraded frequently, I think from 4.2 )
2. I add new VMware zone, but this one with Advanced Networking, I end
with this repetitive error and nothing else.
3. I decide to add a new zone, the same as I have at (2) with Advanced
Networking, everything works perfect, no issue at systemvm deploy, routers,
instances.
4. I delete the zone from point (2)
5. I add again the same zone, end with the same error.
I have tried multiple things, delete the template, add again, inject,
nothing works, I end with the same error, over and over, and I do not see
the why.
021-09-23 11:53:54,666 ERROR [c.c.h.v.r.VmwareResource]
(DirectAgent-228:ctx-083b9265 lnd-uk-002.test.host, job-35231/job-35263,
cmd: SetupKeyStoreCommand) (logid:df4131be) Command failed due to Exception:
java.lang.Exception
Message: Failed to authentication SSH user root on host 10.15.0.204
2021-09-23 11:53:54,674 ERROR [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-19:ctx-808054ab job-35231/job-35263 ctx-eb7c37bb)
(logid:df4131be) Retrying after catching exception while trying to secure
agent for systemvm id=2025
com.cloud.utils.exception.CloudRuntimeException: Unable to read/process CSR:
Command failed due to Exception: java.lang.Exception
Message: Failed to authentication SSH user root on host 10.15.0.204
Here is the full log : https://pastebin.com/raw/fFfmquVB
Maybe someone have a hint for me .
Regards,
Cristian
Re: Failed to authentication SSH user root on host -
repetitive error
Posted by Pearl d'Silva <pe...@shapeblue.com>.
By any chance, are there any old systemvm.isos in the secondary store, if yes, can you please delete them. Based on the information you've provided, it seems like, the key has been injected into systemvm.iso, but during boot up, a script (cloud-early-config) that sets up the VM before bootstrapping / patching isn't copying the auth key.
Can you try the steps mentioned under the VMWare section of the doc: https://cwiki.apache.org/confluence/display/CLOUDSTACK/SystemVm.iso
and see if it helps.
SystemVm.iso - Apache Cloudstack - Apache Software Foundation<https://cwiki.apache.org/confluence/display/CLOUDSTACK/SystemVm.iso>
Mechanism of propagating systemvm.iso. Systemvm.iso gets propagated differently for different hypervisors. It comes packaged as part of the Cloudstack rpm or is built on dev environments with the command "mvn clean install -P developer,systemvm" (More info on building cloudstack) .Once the iso is at the appropriate location depending on the hypervisor it is inserted as a cd drive and the ...
cwiki.apache.org
Thanks,
Pearl
________________________________
From: cristian.c@istream.today <cr...@istream.today>
Sent: Friday, September 24, 2021 11:45 AM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: RE: Failed to authentication SSH user root on host - repetitive error
Hi Pearl,
I already did a check via console into the secondary storage vm, I saw
that there are not keys in authorized_keys.
If I try to inject, I see this : /bin/bash
/usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh
/var/cloudstack/management/.ssh/id_rsa.pub
/var/cloudstack/management/.ssh/id_rsa
/usr/share/cloudstack-common/vms/systemvm.iso
mount: /dev/loop0 is write-protected, mounting read-only
New public key is the same as the one in the systemvm.iso, not injecting it,
not modifying systemvm.iso
[root@cloud-emea systemvm]# cd /mnt/iso/
[root@cloud-emea iso]# ls
agent.zip authorized_keys cloud-scripts.tgz
[root@cloud-emea iso]# cat authorized_keys
ssh-rsa
AAAAB3NzaC1yc2EAAAABIwAAAQEA0UEA0mRUzKbH1cHHY2GlsaQ18q1KvqfNSV/YqAScZhcPueIl
BbqwPUNznzfSiz/K/+DH8u5QkDIz+fC0Sx+xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx+Bd1Y2U1TxlYee
+zUh6vszEDwmiq5nTkuCJP1T3o8QL3gUekAFjW7CECsIzLkA41Q8lY0L3qaHBRUJntGGIMtZ26AP
IWMC7NHD0wFge3DEN5UhFODcB1f9U7oqa10XqgORjbd88JPfFv/0j92xaaerNpJKw==
cloud@cloud-emea.test.host[root@cloud-emea iso]#
is the same key as I have in : /var/cloudstack/management/.ssh/id_rsa.pub
(For security reason I have replaced characters from ssh pub key)
I'm stuck here, I do not understand the logic.
Regards,
Cristian
-----Original Message-----
From: Pearl d'Silva <pe...@shapeblue.com>
Sent: Friday, September 24, 2021 8:05 AM
To: users@cloudstack.apache.org
Subject: Re: Failed to authentication SSH user root on host - repetitive
error
Hi Cristian,
The exception you are seeing is most likely to happen if the systemvm.iso
hasn't been injected with the Management server's key. One way to validate
it, would be to go to your secondary store - I presume you are working on a
VMware setup - mount the systemvm-4.15.2.iso and verify the authorized_keys
data with what's present on the MS at
/var/cloudstack/management/.ssh/id_rsa.pub.
Thanks,
Pearl
________________________________
From: cristian.c@istream.today <cr...@istream.today>
Sent: Thursday, September 23, 2021 9:50 PM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: Failed to authentication SSH user root on host - repetitive error
Hello,
I was not able to fix this error "Failed to authentication SSH" and
looks like a bug for me, I will explain here why.
My setup before adding new ZONE:
1. Cloudstack 4.15.2 + VMware 6.5 with 1 Basic Network Zone. ( This is
an old setup, it was upgraded frequently, I think from 4.2 )
2. I add new VMware zone, but this one with Advanced Networking, I end
with this repetitive error and nothing else.
3. I decide to add a new zone, the same as I have at (2) with Advanced
Networking, everything works perfect, no issue at systemvm deploy, routers,
instances.
4. I delete the zone from point (2)
5. I add again the same zone, end with the same error.
I have tried multiple things, delete the template, add again, inject,
nothing works, I end with the same error, over and over, and I do not see
the why.
021-09-23 11:53:54,666 ERROR [c.c.h.v.r.VmwareResource]
(DirectAgent-228:ctx-083b9265 lnd-uk-002.test.host, job-35231/job-35263,
cmd: SetupKeyStoreCommand) (logid:df4131be) Command failed due to Exception:
java.lang.Exception
Message: Failed to authentication SSH user root on host 10.15.0.204
2021-09-23 11:53:54,674 ERROR [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-19:ctx-808054ab job-35231/job-35263 ctx-eb7c37bb)
(logid:df4131be) Retrying after catching exception while trying to secure
agent for systemvm id=2025
com.cloud.utils.exception.CloudRuntimeException: Unable to read/process CSR:
Command failed due to Exception: java.lang.Exception
Message: Failed to authentication SSH user root on host 10.15.0.204
Here is the full log : https://pastebin.com/raw/fFfmquVB
Maybe someone have a hint for me .
Regards,
Cristian
Re: Failed to authentication SSH user root on host - repetitive error
Posted by Wei ZHOU <us...@gmail.com>.
Hi Cristian,
Could you please check if the systemvm.iso on secondary storage has the
public key ?
# ls -l /var/cloudstack/mnt/32987294991135.77249d5c/systemvm/
total 92912
-rw-r--r--. 1 cloud cloud 95141888 Sep 15 15:44
systemvm-4.16.0.0-SNAPSHOT.iso
secondary storage is mounted on mgmt server
-Wei
On Fri, 24 Sept 2021 at 09:51, <cr...@istream.today> wrote:
> Hi Wei,
>
> I think here is the problem, when the key are injected, the SSVM is not
> running yet, is starting few seconds later.... So, the answer is that I do
> not have what to STOP, I can only delete the SSVM instance from VMware
> side..
>
> Thank you,
> Cristian
>
> -----Original Message-----
> From: Wei ZHOU <us...@gmail.com>
> Sent: Friday, September 24, 2021 9:43 AM
> To: users <us...@cloudstack.apache.org>
> Subject: Re: Failed to authentication SSH user root on host - repetitive
> error
>
> Hi Cristian,
>
> When the new public key has been injected into systemvm.iso, please
> stop/start or destroy SSVM.
>
> -Wei
>
> On Fri, 24 Sept 2021 at 08:16, <cr...@istream.today> wrote:
>
> > Hi Pearl,
> >
> > I already did a check via console into the secondary storage vm,
> > I saw that there are not keys in authorized_keys.
> >
> > If I try to inject, I see this : /bin/bash
> > /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh
> > /var/cloudstack/management/.ssh/id_rsa.pub
> > /var/cloudstack/management/.ssh/id_rsa
> > /usr/share/cloudstack-common/vms/systemvm.iso
> > mount: /dev/loop0 is write-protected, mounting read-only New public
> > key is the same as the one in the systemvm.iso, not injecting it, not
> > modifying systemvm.iso
> >
> >
> > [root@cloud-emea systemvm]# cd /mnt/iso/ [root@cloud-emea iso]# ls
> > agent.zip authorized_keys cloud-scripts.tgz [root@cloud-emea iso]#
> > cat authorized_keys ssh-rsa
> >
> > AAAAB3NzaC1yc2EAAAABIwAAAQEA0UEA0mRUzKbH1cHHY2GlsaQ18q1KvqfNSV/YqAScZh
> > cPueIl
> >
> > BbqwPUNznzfSiz/K/+DH8u5QkDIz+fC0Sx+xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
> > xxxxxx
> >
> > xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx+Bd1Y2U1
> > xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx+TxlYee
> >
> > +zUh6vszEDwmiq5nTkuCJP1T3o8QL3gUekAFjW7CECsIzLkA41Q8lY0L3qaHBRUJntGGIM
> > +tZ26AP
> > IWMC7NHD0wFge3DEN5UhFODcB1f9U7oqa10XqgORjbd88JPfFv/0j92xaaerNpJKw==
> > cloud@cloud-emea.test.host[root@cloud-emea iso]#
> >
> > is the same key as I have in :
> > /var/cloudstack/management/.ssh/id_rsa.pub
> >
> > (For security reason I have replaced characters from ssh pub key)
> >
> >
> > I'm stuck here, I do not understand the logic.
> >
> >
> > Regards,
> > Cristian
> >
> > -----Original Message-----
> > From: Pearl d'Silva <pe...@shapeblue.com>
> > Sent: Friday, September 24, 2021 8:05 AM
> > To: users@cloudstack.apache.org
> > Subject: Re: Failed to authentication SSH user root on host -
> > repetitive error
> >
> > Hi Cristian,
> >
> > The exception you are seeing is most likely to happen if the
> > systemvm.iso hasn't been injected with the Management server's key.
> > One way to validate it, would be to go to your secondary store - I
> > presume you are working on a VMware setup - mount the
> > systemvm-4.15.2.iso and verify the authorized_keys data with what's
> > present on the MS at /var/cloudstack/management/.ssh/id_rsa.pub.
> >
> > Thanks,
> > Pearl
> >
> > ________________________________
> > From: cristian.c@istream.today <cr...@istream.today>
> > Sent: Thursday, September 23, 2021 9:50 PM
> > To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> > Subject: Failed to authentication SSH user root on host - repetitive
> > error
> >
> > Hello,
> >
> >
> >
> >
> >
> > I was not able to fix this error "Failed to authentication SSH"
> > and looks like a bug for me, I will explain here why.
> >
> >
> >
> > My setup before adding new ZONE:
> >
> >
> >
> > 1. Cloudstack 4.15.2 + VMware 6.5 with 1 Basic Network Zone. ( This
> > is
> > an old setup, it was upgraded frequently, I think from 4.2 )
> > 2. I add new VMware zone, but this one with Advanced Networking, I
> end
> > with this repetitive error and nothing else.
> > 3. I decide to add a new zone, the same as I have at (2) with
> Advanced
> > Networking, everything works perfect, no issue at systemvm deploy,
> > routers, instances.
> > 4. I delete the zone from point (2)
> > 5. I add again the same zone, end with the same error.
> >
> >
> >
> > I have tried multiple things, delete the template, add again,
> > inject, nothing works, I end with the same error, over and over, and I
> > do not see the why.
> >
> >
> >
> >
> >
> > 021-09-23 11:53:54,666 ERROR [c.c.h.v.r.VmwareResource]
> > (DirectAgent-228:ctx-083b9265 lnd-uk-002.test.host,
> > job-35231/job-35263,
> > cmd: SetupKeyStoreCommand) (logid:df4131be) Command failed due to
> > Exception:
> > java.lang.Exception
> >
> > Message: Failed to authentication SSH user root on host 10.15.0.204
> >
> >
> >
> > 2021-09-23 11:53:54,674 ERROR [c.c.v.VirtualMachineManagerImpl]
> > (Work-Job-Executor-19:ctx-808054ab job-35231/job-35263 ctx-eb7c37bb)
> > (logid:df4131be) Retrying after catching exception while trying to
> > secure agent for systemvm id=2025
> >
> > com.cloud.utils.exception.CloudRuntimeException: Unable to
> > read/process
> > CSR:
> > Command failed due to Exception: java.lang.Exception
> >
> > Message: Failed to authentication SSH user root on host 10.15.0.204
> >
> >
> >
> >
> >
> > Here is the full log : https://pastebin.com/raw/fFfmquVB
> >
> >
> >
> >
> >
> > Maybe someone have a hint for me .
> >
> >
> >
> > Regards,
> >
> > Cristian
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
>
>
RE: Failed to authentication SSH user root on host - repetitive error
Posted by cr...@istream.today.
Hi Wei,
I think here is the problem, when the key are injected, the SSVM is not running yet, is starting few seconds later.... So, the answer is that I do not have what to STOP, I can only delete the SSVM instance from VMware side..
Thank you,
Cristian
-----Original Message-----
From: Wei ZHOU <us...@gmail.com>
Sent: Friday, September 24, 2021 9:43 AM
To: users <us...@cloudstack.apache.org>
Subject: Re: Failed to authentication SSH user root on host - repetitive error
Hi Cristian,
When the new public key has been injected into systemvm.iso, please stop/start or destroy SSVM.
-Wei
On Fri, 24 Sept 2021 at 08:16, <cr...@istream.today> wrote:
> Hi Pearl,
>
> I already did a check via console into the secondary storage vm,
> I saw that there are not keys in authorized_keys.
>
> If I try to inject, I see this : /bin/bash
> /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh
> /var/cloudstack/management/.ssh/id_rsa.pub
> /var/cloudstack/management/.ssh/id_rsa
> /usr/share/cloudstack-common/vms/systemvm.iso
> mount: /dev/loop0 is write-protected, mounting read-only New public
> key is the same as the one in the systemvm.iso, not injecting it, not
> modifying systemvm.iso
>
>
> [root@cloud-emea systemvm]# cd /mnt/iso/ [root@cloud-emea iso]# ls
> agent.zip authorized_keys cloud-scripts.tgz [root@cloud-emea iso]#
> cat authorized_keys ssh-rsa
>
> AAAAB3NzaC1yc2EAAAABIwAAAQEA0UEA0mRUzKbH1cHHY2GlsaQ18q1KvqfNSV/YqAScZh
> cPueIl
>
> BbqwPUNznzfSiz/K/+DH8u5QkDIz+fC0Sx+xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
> xxxxxx
>
> xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx+Bd1Y2U1
> xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx+TxlYee
>
> +zUh6vszEDwmiq5nTkuCJP1T3o8QL3gUekAFjW7CECsIzLkA41Q8lY0L3qaHBRUJntGGIM
> +tZ26AP
> IWMC7NHD0wFge3DEN5UhFODcB1f9U7oqa10XqgORjbd88JPfFv/0j92xaaerNpJKw==
> cloud@cloud-emea.test.host[root@cloud-emea iso]#
>
> is the same key as I have in :
> /var/cloudstack/management/.ssh/id_rsa.pub
>
> (For security reason I have replaced characters from ssh pub key)
>
>
> I'm stuck here, I do not understand the logic.
>
>
> Regards,
> Cristian
>
> -----Original Message-----
> From: Pearl d'Silva <pe...@shapeblue.com>
> Sent: Friday, September 24, 2021 8:05 AM
> To: users@cloudstack.apache.org
> Subject: Re: Failed to authentication SSH user root on host -
> repetitive error
>
> Hi Cristian,
>
> The exception you are seeing is most likely to happen if the
> systemvm.iso hasn't been injected with the Management server's key.
> One way to validate it, would be to go to your secondary store - I
> presume you are working on a VMware setup - mount the
> systemvm-4.15.2.iso and verify the authorized_keys data with what's
> present on the MS at /var/cloudstack/management/.ssh/id_rsa.pub.
>
> Thanks,
> Pearl
>
> ________________________________
> From: cristian.c@istream.today <cr...@istream.today>
> Sent: Thursday, September 23, 2021 9:50 PM
> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> Subject: Failed to authentication SSH user root on host - repetitive
> error
>
> Hello,
>
>
>
>
>
> I was not able to fix this error "Failed to authentication SSH"
> and looks like a bug for me, I will explain here why.
>
>
>
> My setup before adding new ZONE:
>
>
>
> 1. Cloudstack 4.15.2 + VMware 6.5 with 1 Basic Network Zone. ( This
> is
> an old setup, it was upgraded frequently, I think from 4.2 )
> 2. I add new VMware zone, but this one with Advanced Networking, I end
> with this repetitive error and nothing else.
> 3. I decide to add a new zone, the same as I have at (2) with Advanced
> Networking, everything works perfect, no issue at systemvm deploy,
> routers, instances.
> 4. I delete the zone from point (2)
> 5. I add again the same zone, end with the same error.
>
>
>
> I have tried multiple things, delete the template, add again,
> inject, nothing works, I end with the same error, over and over, and I
> do not see the why.
>
>
>
>
>
> 021-09-23 11:53:54,666 ERROR [c.c.h.v.r.VmwareResource]
> (DirectAgent-228:ctx-083b9265 lnd-uk-002.test.host,
> job-35231/job-35263,
> cmd: SetupKeyStoreCommand) (logid:df4131be) Command failed due to
> Exception:
> java.lang.Exception
>
> Message: Failed to authentication SSH user root on host 10.15.0.204
>
>
>
> 2021-09-23 11:53:54,674 ERROR [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-19:ctx-808054ab job-35231/job-35263 ctx-eb7c37bb)
> (logid:df4131be) Retrying after catching exception while trying to
> secure agent for systemvm id=2025
>
> com.cloud.utils.exception.CloudRuntimeException: Unable to
> read/process
> CSR:
> Command failed due to Exception: java.lang.Exception
>
> Message: Failed to authentication SSH user root on host 10.15.0.204
>
>
>
>
>
> Here is the full log : https://pastebin.com/raw/fFfmquVB
>
>
>
>
>
> Maybe someone have a hint for me .
>
>
>
> Regards,
>
> Cristian
>
>
>
>
>
>
>
>
>
>
Re: Failed to authentication SSH user root on host - repetitive error
Posted by Wei ZHOU <us...@gmail.com>.
Hi Cristian,
When the new public key has been injected into systemvm.iso, please
stop/start or destroy SSVM.
-Wei
On Fri, 24 Sept 2021 at 08:16, <cr...@istream.today> wrote:
> Hi Pearl,
>
> I already did a check via console into the secondary storage vm, I
> saw
> that there are not keys in authorized_keys.
>
> If I try to inject, I see this : /bin/bash
> /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh
> /var/cloudstack/management/.ssh/id_rsa.pub
> /var/cloudstack/management/.ssh/id_rsa
> /usr/share/cloudstack-common/vms/systemvm.iso
> mount: /dev/loop0 is write-protected, mounting read-only
> New public key is the same as the one in the systemvm.iso, not injecting
> it,
> not modifying systemvm.iso
>
>
> [root@cloud-emea systemvm]# cd /mnt/iso/
> [root@cloud-emea iso]# ls
> agent.zip authorized_keys cloud-scripts.tgz
> [root@cloud-emea iso]# cat authorized_keys
> ssh-rsa
>
> AAAAB3NzaC1yc2EAAAABIwAAAQEA0UEA0mRUzKbH1cHHY2GlsaQ18q1KvqfNSV/YqAScZhcPueIl
>
> BbqwPUNznzfSiz/K/+DH8u5QkDIz+fC0Sx+xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
>
> xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx+Bd1Y2U1TxlYee
>
> +zUh6vszEDwmiq5nTkuCJP1T3o8QL3gUekAFjW7CECsIzLkA41Q8lY0L3qaHBRUJntGGIMtZ26AP
> IWMC7NHD0wFge3DEN5UhFODcB1f9U7oqa10XqgORjbd88JPfFv/0j92xaaerNpJKw==
> cloud@cloud-emea.test.host[root@cloud-emea iso]#
>
> is the same key as I have in : /var/cloudstack/management/.ssh/id_rsa.pub
>
> (For security reason I have replaced characters from ssh pub key)
>
>
> I'm stuck here, I do not understand the logic.
>
>
> Regards,
> Cristian
>
> -----Original Message-----
> From: Pearl d'Silva <pe...@shapeblue.com>
> Sent: Friday, September 24, 2021 8:05 AM
> To: users@cloudstack.apache.org
> Subject: Re: Failed to authentication SSH user root on host - repetitive
> error
>
> Hi Cristian,
>
> The exception you are seeing is most likely to happen if the systemvm.iso
> hasn't been injected with the Management server's key. One way to validate
> it, would be to go to your secondary store - I presume you are working on a
> VMware setup - mount the systemvm-4.15.2.iso and verify the authorized_keys
> data with what's present on the MS at
> /var/cloudstack/management/.ssh/id_rsa.pub.
>
> Thanks,
> Pearl
>
> ________________________________
> From: cristian.c@istream.today <cr...@istream.today>
> Sent: Thursday, September 23, 2021 9:50 PM
> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> Subject: Failed to authentication SSH user root on host - repetitive error
>
> Hello,
>
>
>
>
>
> I was not able to fix this error "Failed to authentication SSH" and
> looks like a bug for me, I will explain here why.
>
>
>
> My setup before adding new ZONE:
>
>
>
> 1. Cloudstack 4.15.2 + VMware 6.5 with 1 Basic Network Zone. ( This
> is
> an old setup, it was upgraded frequently, I think from 4.2 )
> 2. I add new VMware zone, but this one with Advanced Networking, I end
> with this repetitive error and nothing else.
> 3. I decide to add a new zone, the same as I have at (2) with Advanced
> Networking, everything works perfect, no issue at systemvm deploy, routers,
> instances.
> 4. I delete the zone from point (2)
> 5. I add again the same zone, end with the same error.
>
>
>
> I have tried multiple things, delete the template, add again, inject,
> nothing works, I end with the same error, over and over, and I do not see
> the why.
>
>
>
>
>
> 021-09-23 11:53:54,666 ERROR [c.c.h.v.r.VmwareResource]
> (DirectAgent-228:ctx-083b9265 lnd-uk-002.test.host, job-35231/job-35263,
> cmd: SetupKeyStoreCommand) (logid:df4131be) Command failed due to
> Exception:
> java.lang.Exception
>
> Message: Failed to authentication SSH user root on host 10.15.0.204
>
>
>
> 2021-09-23 11:53:54,674 ERROR [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-19:ctx-808054ab job-35231/job-35263 ctx-eb7c37bb)
> (logid:df4131be) Retrying after catching exception while trying to secure
> agent for systemvm id=2025
>
> com.cloud.utils.exception.CloudRuntimeException: Unable to read/process
> CSR:
> Command failed due to Exception: java.lang.Exception
>
> Message: Failed to authentication SSH user root on host 10.15.0.204
>
>
>
>
>
> Here is the full log : https://pastebin.com/raw/fFfmquVB
>
>
>
>
>
> Maybe someone have a hint for me .
>
>
>
> Regards,
>
> Cristian
>
>
>
>
>
>
>
>
>
>
RE: Failed to authentication SSH user root on host - repetitive error
Posted by cr...@istream.today.
Hi Pearl,
I already did a check via console into the secondary storage vm, I saw
that there are not keys in authorized_keys.
If I try to inject, I see this : /bin/bash
/usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh
/var/cloudstack/management/.ssh/id_rsa.pub
/var/cloudstack/management/.ssh/id_rsa
/usr/share/cloudstack-common/vms/systemvm.iso
mount: /dev/loop0 is write-protected, mounting read-only
New public key is the same as the one in the systemvm.iso, not injecting it,
not modifying systemvm.iso
[root@cloud-emea systemvm]# cd /mnt/iso/
[root@cloud-emea iso]# ls
agent.zip authorized_keys cloud-scripts.tgz
[root@cloud-emea iso]# cat authorized_keys
ssh-rsa
AAAAB3NzaC1yc2EAAAABIwAAAQEA0UEA0mRUzKbH1cHHY2GlsaQ18q1KvqfNSV/YqAScZhcPueIl
BbqwPUNznzfSiz/K/+DH8u5QkDIz+fC0Sx+xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx+Bd1Y2U1TxlYee
+zUh6vszEDwmiq5nTkuCJP1T3o8QL3gUekAFjW7CECsIzLkA41Q8lY0L3qaHBRUJntGGIMtZ26AP
IWMC7NHD0wFge3DEN5UhFODcB1f9U7oqa10XqgORjbd88JPfFv/0j92xaaerNpJKw==
cloud@cloud-emea.test.host[root@cloud-emea iso]#
is the same key as I have in : /var/cloudstack/management/.ssh/id_rsa.pub
(For security reason I have replaced characters from ssh pub key)
I'm stuck here, I do not understand the logic.
Regards,
Cristian
-----Original Message-----
From: Pearl d'Silva <pe...@shapeblue.com>
Sent: Friday, September 24, 2021 8:05 AM
To: users@cloudstack.apache.org
Subject: Re: Failed to authentication SSH user root on host - repetitive
error
Hi Cristian,
The exception you are seeing is most likely to happen if the systemvm.iso
hasn't been injected with the Management server's key. One way to validate
it, would be to go to your secondary store - I presume you are working on a
VMware setup - mount the systemvm-4.15.2.iso and verify the authorized_keys
data with what's present on the MS at
/var/cloudstack/management/.ssh/id_rsa.pub.
Thanks,
Pearl
________________________________
From: cristian.c@istream.today <cr...@istream.today>
Sent: Thursday, September 23, 2021 9:50 PM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: Failed to authentication SSH user root on host - repetitive error
Hello,
I was not able to fix this error "Failed to authentication SSH" and
looks like a bug for me, I will explain here why.
My setup before adding new ZONE:
1. Cloudstack 4.15.2 + VMware 6.5 with 1 Basic Network Zone. ( This is
an old setup, it was upgraded frequently, I think from 4.2 )
2. I add new VMware zone, but this one with Advanced Networking, I end
with this repetitive error and nothing else.
3. I decide to add a new zone, the same as I have at (2) with Advanced
Networking, everything works perfect, no issue at systemvm deploy, routers,
instances.
4. I delete the zone from point (2)
5. I add again the same zone, end with the same error.
I have tried multiple things, delete the template, add again, inject,
nothing works, I end with the same error, over and over, and I do not see
the why.
021-09-23 11:53:54,666 ERROR [c.c.h.v.r.VmwareResource]
(DirectAgent-228:ctx-083b9265 lnd-uk-002.test.host, job-35231/job-35263,
cmd: SetupKeyStoreCommand) (logid:df4131be) Command failed due to Exception:
java.lang.Exception
Message: Failed to authentication SSH user root on host 10.15.0.204
2021-09-23 11:53:54,674 ERROR [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-19:ctx-808054ab job-35231/job-35263 ctx-eb7c37bb)
(logid:df4131be) Retrying after catching exception while trying to secure
agent for systemvm id=2025
com.cloud.utils.exception.CloudRuntimeException: Unable to read/process CSR:
Command failed due to Exception: java.lang.Exception
Message: Failed to authentication SSH user root on host 10.15.0.204
Here is the full log : https://pastebin.com/raw/fFfmquVB
Maybe someone have a hint for me .
Regards,
Cristian
Re: Failed to authentication SSH user root on host -
repetitive error
Posted by Pearl d'Silva <pe...@shapeblue.com>.
Hi Cristian,
The exception you are seeing is most likely to happen if the systemvm.iso hasn't been injected with the Management server's key. One way to validate it, would be to go to your secondary store - I presume you are working on a VMware setup - mount the systemvm-4.15.2.iso and verify the authorized_keys data with what's present on the MS at /var/cloudstack/management/.ssh/id_rsa.pub.
Thanks,
Pearl
________________________________
From: cristian.c@istream.today <cr...@istream.today>
Sent: Thursday, September 23, 2021 9:50 PM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: Failed to authentication SSH user root on host - repetitive error
Hello,
I was not able to fix this error "Failed to authentication SSH" and
looks like a bug for me, I will explain here why.
My setup before adding new ZONE:
1. Cloudstack 4.15.2 + VMware 6.5 with 1 Basic Network Zone. ( This is
an old setup, it was upgraded frequently, I think from 4.2 )
2. I add new VMware zone, but this one with Advanced Networking, I end
with this repetitive error and nothing else.
3. I decide to add a new zone, the same as I have at (2) with Advanced
Networking, everything works perfect, no issue at systemvm deploy, routers,
instances.
4. I delete the zone from point (2)
5. I add again the same zone, end with the same error.
I have tried multiple things, delete the template, add again, inject,
nothing works, I end with the same error, over and over, and I do not see
the why.
021-09-23 11:53:54,666 ERROR [c.c.h.v.r.VmwareResource]
(DirectAgent-228:ctx-083b9265 lnd-uk-002.test.host, job-35231/job-35263,
cmd: SetupKeyStoreCommand) (logid:df4131be) Command failed due to Exception:
java.lang.Exception
Message: Failed to authentication SSH user root on host 10.15.0.204
2021-09-23 11:53:54,674 ERROR [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-19:ctx-808054ab job-35231/job-35263 ctx-eb7c37bb)
(logid:df4131be) Retrying after catching exception while trying to secure
agent for systemvm id=2025
com.cloud.utils.exception.CloudRuntimeException: Unable to read/process CSR:
Command failed due to Exception: java.lang.Exception
Message: Failed to authentication SSH user root on host 10.15.0.204
Here is the full log : https://pastebin.com/raw/fFfmquVB
Maybe someone have a hint for me .
Regards,
Cristian