You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Ammad Syed <sy...@gmail.com> on 2020/09/02 15:42:59 UTC
Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error
Guys, please advise how to troubleshoot this further ? How can i enable trace level debugging ? any help in this will be highly appreciated?
Ammad
> On 27-Aug-2020, at 1:41 PM, Vivek Kumar <vi...@indiqus.com.invalid> wrote:
>
> Pardon, the whole mail chain was not visible so replied accordingly. It seems like you are doing SSH using root shell so it doesn’t matter.
>
> Vivek Kumar
> Manager - Cloud & DevOps
> IndiQus Technologies
> 24*7 O +91 11 4055 1411 | M +91 7503460090
> www.indiqus.com <http://indiqus.com/>
>
> This message is intended only for the use of the individual or entity to which it is addressed and may contain information that is confidential and/or privileged. If you are not the intended recipient please delete the original message and any copy of it from your computer system. You are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited unless proper authorization has been obtained for such action. If you have received this communication in error, please notify the sender immediately. Although IndiQus attempts to sweep e-mail and attachments for viruses, it does not guarantee that both are virus-free and accepts no liability for any damage sustained as a result of viruses.
>
>> On 27-Aug-2020, at 2:03 PM, Vivek Kumar <vi...@indiqus.com> wrote:
>>
>> Hello Ammad,
>>
>> You haven’t defined the user while logging to the system VM through the Link local IP that’s why you have got the error while logging to the system VM.
>>
>>>> ssh -i /root/.ssh/id_rsa.cloud 169.254.0.121 -p 3922
>>
>> Try instead below
>>
>>>> ssh -i /root/.ssh/id_rsa.cloud -p 3922 root@169.254.0.121 <ma...@169.254.0.121>
>>
>> Vivek Kumar
>> Manager - Cloud & DevOps
>> IndiQus Technologies
>> 24*7 O +91 11 4055 1411 | M +91 7503460090
>> www.indiqus.com <http://indiqus.com/>
>>
>> This message is intended only for the use of the individual or entity to which it is addressed and may contain information that is confidential and/or privileged. If you are not the intended recipient please delete the original message and any copy of it from your computer system. You are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited unless proper authorization has been obtained for such action. If you have received this communication in error, please notify the sender immediately. Although IndiQus attempts to sweep e-mail and attachments for viruses, it does not guarantee that both are virus-free and accepts no liability for any damage sustained as a result of viruses.
>>
>>>> On 27-Aug-2020, at 1:59 PM, Ammad Syed <syedammad83@gmail.com <ma...@gmail.com>> wrote:
>>>
>>> Guys, any help to troubleshoot this further would be highly appreciated.
>>>
>>>
>>> Ammad Ali
>>>> On 18-Aug-2020, at 1:05 PM, Ammad Syed <syedammad83@gmail.com <ma...@gmail.com>> wrote:
>>>>
>>>>
>>>> Hi,
>>>>
>>>> The systemVM is accessible with 169.x.x.x IP from xenserver host. But login with private key is denied. I am testing this on my test environment.
>>>>
>>>> [root@xenserver ~]# ping 169.254.0.121
>>>> PING 169.254.0.121 (169.254.0.121) 56(84) bytes of data.
>>>> 64 bytes from 169.254.0.121: icmp_seq=1 ttl=64 time=0.328 ms
>>>> 64 bytes from 169.254.0.121: icmp_seq=2 ttl=64 time=0.124 ms
>>>> 64 bytes from 169.254.0.121: icmp_seq=3 ttl=64 time=0.111 ms
>>>> 64 bytes from 169.254.0.121: icmp_seq=4 ttl=64 time=0.126 ms
>>>> ^C
>>>> --- 169.254.0.121 ping statistics ---
>>>> 4 packets transmitted, 4 received, 0% packet loss, time 2997ms
>>>> rtt min/avg/max/mdev = 0.111/0.172/0.328/0.090 ms
>>>> [root@xenserver ~]#
>>>> [root@xenserver ~]#
>>>> [root@xenserver ~]# ssh -i /root/.ssh/id_rsa.cloud 169.254.0.121 -p 3922
>>>> Permission denied (publickey).
>>>>
>>>> I have checked by logging into the systemVM, the public key is not found in /root/.ssh/authorized_keys.
>>>>
>>>> I have digged further, the systemvm.iso is same on the management node and xenserver host. I have checked by mounting the ISO as well, the public key is there in authorized_keys in ISO.
>>>>
>>>> Restarting / recreating systemVM isn't fixed the issue.
>>>>
>>>> The systemVM is already running on HVM mode on xenserver host as the template OS type is Debian 8 64bit is selected.
>>>>
>>>> On my PRD upgrade, I have tested on one zone only. Currently I am testing on my test ACS environment which have only one zone.
>>>>
>>>> Ammad Ali
>>>>
>>>>> On Mon, Aug 17, 2020 at 11:10 PM Andrija Panic <andrija.panic@gmail.com <ma...@gmail.com>> wrote:
>>>>> It was my tool (GUI based tool, that still doesn't return any line (???)
>>>>> but I see it.
>>>>>
>>>>> Problem is with 169.254.1.178 not being reachable over SSH (2020-07-25
>>>>> 01:45:52,097 DEBUG [c.c.h.x.r.CitrixResourceBase]
>>>>> (DirectAgent-57:ctx-ded7f7f9) (logid:a75971c2) Executing command in VR:
>>>>> /opt/cloud/bin/router_proxy.sh keystore-setup 169.254.1.178
>>>>> /usr/local/cloud/systemvm/conf/agent.properties
>>>>> /usr/local/cloud/systemvm/conf/cloud.jks Xe8zzqZmUp9wmABH 365
>>>>> /usr/local/cloud/systemvm/conf/cloud.csr)
>>>>>
>>>>> Can you try to ssh from that particular XenServer (where the SSVM is
>>>>> running) to the IP 169.xxxxx on port 3922 (ssh port) using the private key
>>>>> - if you log in to the VM via it's console (root/password) - do you see
>>>>> with ifconfig the interface being started with 169..xxxx IP address? I
>>>>> assume that destroying the SSVM and starting a new one doesn't fix the
>>>>> issue?
>>>>>
>>>>> You can also try changing the OS type of the systemVM template to Other
>>>>> Linux 64bit, so it runs in HVM mode (instead of PV) - you'll need to see ID
>>>>> for that OS type from the guest_os table, and set the guest_os_id field in
>>>>> the vm_template table for your systemVM template.
>>>>>
>>>>> Does the issue occur in other zones as well? (assuming same hypervisor type)
>>>>>
>>>>>
>>>>> On Mon, 17 Aug 2020 at 15:31, Ammad Syed <syedammad83@gmail.com <ma...@gmail.com>> wrote:
>>>>>
>>>>>> I have rechecked the logs are there. Please review below lines.
>>>>>>
>>>>>>
>>>>>>
>>>>>> C:\Users\ammad\Downloads\management-server.log-4.13\management-server.log-4.13.1-errors
>>>>>> (27 hits)
>>>>>> Line 409969: 2020-07-25 01:45:52,320 ERROR
>>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-58:ctx-9dc75334
>>>>>> job-1207969/job-1208104 ctx-09e752ec) (logid:a75971c2) Failed to setup
>>>>>> keystore and generate CSR for system vm: s-25023-VM
>>>>>> Line 437413: 2020-07-25 01:51:33,845 ERROR
>>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-70:ctx-457ce4ee
>>>>>> job-1207969/job-1208129 ctx-195da69e) (logid:a75971c2) Failed to setup
>>>>>> keystore and generate CSR for system vm: s-25024-VM
>>>>>> Line 549538: 2020-07-25 02:06:44,833 ERROR
>>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-3:ctx-4634f7e0
>>>>>> job-1208154/job-1208161 ctx-05d065dc) (logid:91fc037a) Failed to setup
>>>>>> keystore and generate CSR for system vm: v-25027-VM
>>>>>> Line 663111: 2020-07-25 02:23:43,783 ERROR
>>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-22:ctx-fe0b857b
>>>>>> job-1208154/job-1208222 ctx-ef088cd7) (logid:91fc037a) Failed to setup
>>>>>> keystore and generate CSR for system vm: v-20771-VM
>>>>>> Line 667060: 2020-07-25 02:24:30,221 ERROR
>>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-23:ctx-662cfe5d
>>>>>> job-1208154/job-1208229 ctx-e59878e9) (logid:91fc037a) Failed to setup
>>>>>> keystore and generate CSR for system vm: v-24763-VM
>>>>>> Line 674269: 2020-07-25 02:25:47,345 ERROR
>>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-24:ctx-b6529613
>>>>>> job-1208154/job-1208233 ctx-7da81cab) (logid:91fc037a) Failed to setup
>>>>>> keystore and generate CSR for system vm: v-20773-VM
>>>>>> Line 678169: 2020-07-25 02:26:20,355 ERROR
>>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-31:ctx-7a062a66
>>>>>> job-1208154/job-1208244 ctx-4f48d08a) (logid:91fc037a) Failed to setup
>>>>>> keystore and generate CSR for system vm: v-25027-VM
>>>>>> Line 697113: 2020-07-25 02:30:09,669 ERROR
>>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-40:ctx-3be9fddc
>>>>>> job-1208154/job-1208257 ctx-ed08fc43) (logid:91fc037a) Failed to setup
>>>>>> keystore and generate CSR for system vm: v-20771-VM
>>>>>> Line 701161: 2020-07-25 02:30:48,127 ERROR
>>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-41:ctx-4e3c666d
>>>>>> job-1208155/job-1208258 ctx-df740f75) (logid:9fa7dece) Failed to setup
>>>>>> keystore and generate CSR for system vm: s-24142-VM
>>>>>> Line 701838: 2020-07-25 02:30:56,759 ERROR
>>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-42:ctx-04ac8e71
>>>>>> job-1208154/job-1208259 ctx-36c67b5e) (logid:91fc037a) Failed to setup
>>>>>> keystore and generate CSR for system vm: v-24763-VM
>>>>>> Line 703865: 2020-07-25 02:31:25,123 ERROR
>>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-46:ctx-24bdbe8f
>>>>>> job-1208154/job-1208266 ctx-fa500d3c) (logid:91fc037a) Failed to setup
>>>>>> keystore and generate CSR for system vm: v-20773-VM
>>>>>> Line 712078: 2020-07-25 02:32:28,666 ERROR
>>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-48:ctx-4d5db5c2
>>>>>> job-1208155/job-1208269 ctx-b0f67c2e) (logid:9fa7dece) Failed to setup
>>>>>> keystore and generate CSR for system vm: s-25033-VM
>>>>>> Line 715204: 2020-07-25 02:33:02,875 ERROR
>>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-51:ctx-594da6d2
>>>>>> job-1208155/job-1208273 ctx-ca115800) (logid:9fa7dece) Failed to setup
>>>>>> keystore and generate CSR for system vm: s-25035-VM
>>>>>> Line 743140: 2020-07-25 02:38:52,257 ERROR
>>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-59:ctx-a3f032da
>>>>>> job-1208155/job-1208294 ctx-626a2716) (logid:9fa7dece) Failed to setup
>>>>>> keystore and generate CSR for system vm: s-25037-VM
>>>>>> Line 817646: 2020-07-25 02:47:07,127 ERROR
>>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-5:ctx-ca073bd1
>>>>>> job-1208297/job-1208305 ctx-6213af3b) (logid:1a2a782c) Failed to setup
>>>>>> keystore and generate CSR for system vm: s-25038-VM
>>>>>> Line 847150: 2020-07-25 02:51:46,293 ERROR
>>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-8:ctx-9db8c548
>>>>>> job-1208296/job-1208312 ctx-83c570ff) (logid:436d1800) Failed to setup
>>>>>> keystore and generate CSR for system vm: v-25040-VM
>>>>>> Line 903949: 2020-07-25 02:57:48,515 ERROR
>>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-4:ctx-f35fd59a
>>>>>> job-1208320/job-1208333 ctx-f81179ac) (logid:05eacf0d) Failed to setup
>>>>>> keystore and generate CSR for system vm: v-25041-VM
>>>>>> Line 957628: 2020-07-25 03:04:13,286 ERROR
>>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-24:ctx-dc18956a
>>>>>> job-1208320/job-1208381 ctx-b53f5d7c) (logid:05eacf0d) Failed to setup
>>>>>> keystore and generate CSR for system vm: v-20771-VM
>>>>>> Line 961606: 2020-07-25 03:05:00,522 ERROR
>>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-28:ctx-83e48421
>>>>>> job-1208320/job-1208392 ctx-e6767f04) (logid:05eacf0d) Failed to setup
>>>>>> keystore and generate CSR for system vm: v-24763-VM
>>>>>> Line 966633: 2020-07-25 03:05:29,767 ERROR
>>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-30:ctx-897e2a66
>>>>>> job-1208320/job-1208397 ctx-39c396e1) (logid:05eacf0d) Failed to setup
>>>>>> keystore and generate CSR for system vm: v-20773-VM
>>>>>> Line 968892: 2020-07-25 03:05:53,648 ERROR
>>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-36:ctx-68e45b80
>>>>>> job-1208320/job-1208405 ctx-1aaf96dd) (logid:05eacf0d) Failed to setup
>>>>>> keystore and generate CSR for system vm: v-25041-VM
>>>>>> Line 1003036: 2020-07-25 03:13:33,582 ERROR
>>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-45:ctx-0f552e0e
>>>>>> job-1208320/job-1208422 ctx-e8835fe9) (logid:05eacf0d) Failed to setup
>>>>>> keystore and generate CSR for system vm: v-20771-VM
>>>>>> Line 1006496: 2020-07-25 03:14:09,787 ERROR
>>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-46:ctx-08e5db24
>>>>>> job-1208320/job-1208423 ctx-2cb636c2) (logid:05eacf0d) Failed to setup
>>>>>> keystore and generate CSR for system vm: v-24763-VM
>>>>>> Line 1008408: 2020-07-25 03:14:39,584 ERROR
>>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-47:ctx-8b9288a9
>>>>>> job-1208320/job-1208424 ctx-c05d1544) (logid:05eacf0d) Failed to setup
>>>>>> keystore and generate CSR for system vm: v-20773-VM
>>>>>> Line 1023914: 2020-07-25 03:15:58,139 ERROR
>>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-48:ctx-4d54d717
>>>>>> job-1208320/job-1208425 ctx-27e5397d) (logid:05eacf0d) Failed to setup
>>>>>> keystore and generate CSR for system vm: v-25046-VM
>>>>>> Line 1089568: 2020-07-25 03:24:24,534 ERROR
>>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-59:ctx-62ff8844
>>>>>> job-1208320/job-1208449 ctx-d6a14f7f) (logid:05eacf0d) Failed to setup
>>>>>> keystore and generate CSR for system vm: v-25048-VM
>>>>>> Line 1097605: 2020-07-25 03:25:23,179 ERROR
>>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-62:ctx-24407a9e
>>>>>> job-1208321/job-1208452 ctx-84f80ffa) (logid:42edff3a) Failed to setup
>>>>>> keystore and generate CSR for system vm: s-25050-VM
>>>>>>
>>>>>> -Ammad Ali
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Aug 17, 2020 at 5:56 PM Andrija Panic <andrija.panic@gmail.com <ma...@gmail.com>>
>>>>>> wrote:
>>>>>>
>>>>>>> Ammad, no such log lines in the file you uploaded....?
>>>>>>>
>>>>>>> On Mon, 17 Aug 2020 at 09:02, Ammad Syed <syedammad83@gmail.com <ma...@gmail.com>> wrote:
>>>>>>>
>>>>>>>> Hi Andrija,
>>>>>>>>
>>>>>>>> The error occurred only one time. Can you please try to grep "Failed to
>>>>>>>> setup keystore and generate CSR" you will see all the systemVMs that I
>>>>>>>> tried to create failed because of certificate generation failed. OR you
>>>>>>> can
>>>>>>>> grep job-1208321/job-1208452 this job as well.
>>>>>>>>
>>>>>>>> Also I mentioned above that same thing is happening in my test
>>>>>>> environment
>>>>>>>> that I created 4.11.1 then upgraded 4.11.3 > 4.13.1.
>>>>>>>>
>>>>>>>> Ammad Ali
>>>>>>>>
>>>>>>>> On Mon, Aug 17, 2020 at 11:17 AM Andrija Panic <
>>>>>> andrija.panic@gmail.com <ma...@gmail.com>>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> 2020-07-25 03:23:51,093 WARN [c.c.v.SystemVmLoadScanner]
>>>>>>>>> (secstorage-1:ctx-50756e9b) (logid:3d7c25cb) Unexpected exception
>>>>>>> Failed
>>>>>>>> to
>>>>>>>>> fetch any free public IP address
>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Failed to fetch any
>>>>>>> free
>>>>>>>>> public IP address
>>>>>>>>>
>>>>>>>>> ?
>>>>>>>>>
>>>>>>>>> On Sat, 15 Aug 2020 at 18:05, Ammad Syed <syedammad83@gmail.com <ma...@gmail.com>>
>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> I have checked further, the key is successfully injected in
>>>>>>>> systemvm.iso
>>>>>>>>> on
>>>>>>>>>> the management node and copied successfully to xenserver. I have
>>>>>>>> checked
>>>>>>>>> on
>>>>>>>>>> systemVM there is no public key found in authorized_keys of root.
>>>>>>>>>>
>>>>>>>>>> How can I troubleshoot this further ? how can I enable trace
>>>>>> logging
>>>>>>>> for
>>>>>>>>>> management server to see if there is something problematic
>>>>>> happening.
>>>>>>>>>>
>>>>>>>>>> Ammad Ali
>>>>>>>>>>
>>>>>>>>>> On Sat, Aug 15, 2020 at 1:28 PM Ammad Syed <syedammad83@gmail.com <ma...@gmail.com>>
>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> I have setup my test environment, here is what I did:
>>>>>>>>>>>
>>>>>>>>>>> - Installed ACS 4.11.1 and added xenserver 7.0 host in it.
>>>>>>> SystemVMs
>>>>>>>>> are
>>>>>>>>>>> up in the zone and agents are up.
>>>>>>>>>>> - Then upgraded the system to 4.11.3. Recreated systemVMs, agent
>>>>>> is
>>>>>>>> up
>>>>>>>>>> and
>>>>>>>>>>> systemVM are up with 4.11.3 systemVM version.
>>>>>>>>>>> - Then upgraded the system to 4.13.1, recreated systemVMs are
>>>>>>> running
>>>>>>>>> but
>>>>>>>>>>> agents are not up.
>>>>>>>>>>>
>>>>>>>>>>> I have checked md5sum of systemvm.iso on xenserver and management
>>>>>>>>> server,
>>>>>>>>>>> both are same.
>>>>>>>>>>>
>>>>>>>>>>> [root@xenserver iso]# md5sum
>>>>>>>> /opt/xensource/packages/iso/systemvm.iso
>>>>>>>>>>> baba18f156395da3a5d8208727d8f421
>>>>>>>>>> /opt/xensource/packages/iso/systemvm.iso
>>>>>>>>>>>
>>>>>>>>>>> [root@cloudstack-upgrade vms]# md5sum
>>>>>>>>>>> /usr/share/cloudstack-common/vms/systemvm.iso
>>>>>>>>>>> baba18f156395da3a5d8208727d8f421
>>>>>>>>>>> /usr/share/cloudstack-common/vms/systemvm.iso
>>>>>>>>>>>
>>>>>>>>>>> Also the private key on xenserver root and management server are
>>>>>>>> same.
>>>>>>>>>>>
>>>>>>>>>>> management server
>>>>>>>>>>> /usr/share/cloudstack-common/scripts/vm/systemvm/id_rsa.cloud
>>>>>>>>>>>
>>>>>>>>>>> xenserver path
>>>>>>>>>>> /root/.ssh/id_rsa.cloud
>>>>>>>>>>>
>>>>>>>>>>> - Ammad Ali
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Aug 13, 2020 at 11:27 AM Ammad Syed <
>>>>>> syedammad83@gmail.com <ma...@gmail.com>
>>>>>>>>
>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Here is the link for download management logs.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>> https://drive.google.com/file/d/1l6HDPguGUNaOxsc7VSaj7eaP0F2UOFjA/view?usp=sharing
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Aug 13, 2020 at 11:22 AM Ammad Syed <
>>>>>>> syedammad83@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> I have reverted the version back to 4.11.3. But I have saved
>>>>>> logs
>>>>>>>>>>>>> starting from upgrade.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I think the key has been copied successfully in system vm iso.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2020-07-25 00:34:17,214 INFO [c.c.s.ConfigurationServerImpl]
>>>>>>>>>>>>> (main:null) (logid:) Going to update systemvm iso with
>>>>>> generated
>>>>>>>>>> keypairs
>>>>>>>>>>>>> if needed
>>>>>>>>>>>>> 2020-07-25 00:34:17,214 INFO [c.c.s.ConfigurationServerImpl]
>>>>>>>>>>>>> (main:null) (logid:) Trying to inject public and private keys
>>>>>>> into
>>>>>>>>>> systemvm
>>>>>>>>>>>>> iso
>>>>>>>>>>>>> 2020-07-25 00:34:17,217 DEBUG [c.c.u.s.Script] (main:null)
>>>>>>> (logid:)
>>>>>>>>>>>>> Looking for vms/systemvm.iso in the classpath
>>>>>>>>>>>>> 2020-07-25 00:34:17,217 DEBUG [c.c.u.s.Script] (main:null)
>>>>>>> (logid:)
>>>>>>>>>>>>> System resource:
>>>>>>> file:/usr/share/cloudstack-common/vms/systemvm.iso
>>>>>>>>>>>>> 2020-07-25 00:34:17,217 DEBUG [c.c.u.s.Script] (main:null)
>>>>>>> (logid:)
>>>>>>>>>>>>> Absolute path = /usr/share/cloudstack-common/vms/systemvm.iso
>>>>>>>>>>>>> 2020-07-25 00:34:17,218 DEBUG [c.c.s.ConfigurationServerImpl]
>>>>>>>>>>>>> (main:null) (logid:) Executing: /bin/bash
>>>>>>>>>>>>> /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh
>>>>>>>>>>>>> /var/cloudstack/management/.ssh/id_rsa.pub
>>>>>>>>>>>>> /var/cloudstack/management/.ssh/id_rsa
>>>>>>>>>>>>> /usr/share/cloudstack-common/vms/systemvm.iso
>>>>>>>>>>>>> 2020-07-25 00:34:17,636 INFO [c.c.s.ConfigurationServerImpl]
>>>>>>>>>>>>> (main:null) (logid:) Injected public and private keys into
>>>>>>> systemvm
>>>>>>>>> iso
>>>>>>>>>>>>> with result : null
>>>>>>>>>>>>> 2020-07-25 00:34:50,613 DEBUG [c.c.h.x.r.CitrixResourceBase]
>>>>>>>>>>>>> (DirectAgent-1:ctx-d3dc4cf2) (logid:1d22396d) Copying
>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/xenserver65/../../../../../vms/systemvm.iso
>>>>>>>>>>>>> to /opt/xensource/packages/iso on 172.16.2.22 with permission
>>>>>>> 0644
>>>>>>>>>>>>> 2020-07-25 00:34:52,566 DEBUG [c.c.h.x.r.CitrixResourceBase]
>>>>>>>>>>>>> (DirectAgent-2:ctx-2537b610) (logid:29c67b7a) Copying
>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/xenserver65/../../../../../vms/systemvm.iso
>>>>>>>>>>>>> to /opt/xensource/packages/iso on 172.16.2.5 with permission
>>>>>> 0644
>>>>>>>>>>>>> 2020-07-25 00:34:53,170 DEBUG [c.c.h.x.r.CitrixResourceBase]
>>>>>>>>>>>>> (DirectAgent-3:ctx-168ac27d) (logid:a7862c4b) Copying
>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/xenserver65/../../../../../vms/systemvm.iso
>>>>>>>>>>>>> to /opt/xensource/packages/iso on 172.16.2.9 with permission
>>>>>> 0644
>>>>>>>>>>>>> 2020-07-25 00:34:54,621 DEBUG [c.c.h.x.r.CitrixResourceBase]
>>>>>>>>>>>>> (DirectAgent-6:ctx-f640fe55) (logid:0a62d4cf) Copying
>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/xenserver65/../../../../../vms/systemvm.iso
>>>>>>>>>>>>> to /opt/xensource/packages/iso on 172.16.2.5 with permission
>>>>>> 0644
>>>>>>>>>>>>>
>>>>>>>>>>>>> I have attached complete management logs. The start of logs is
>>>>>>> the
>>>>>>>>>> start
>>>>>>>>>>>>> of management server after upgrade of management server from
>>>>>>> 4.11.3
>>>>>>>>> to
>>>>>>>>>>>>> 4.13.1.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Ammad Ali
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Aug 13, 2020 at 1:35 AM Andrija Panic <
>>>>>>>>> andrija.panic@gmail.com
>>>>>>>>>>>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Do you get an error while trying to inject ssh key into the
>>>>>>>>>> systemvm.iso
>>>>>>>>>>>>>> (mgmt logs) , or can you confirm that the systemvm.iso on XS
>>>>>>> host
>>>>>>>>> and
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> one on the mgmt server are identical, md5sum them (i.e. has
>>>>>> the
>>>>>>>> iso
>>>>>>>>>> been
>>>>>>>>>>>>>> copied over to the XS successfully) - this might explain not
>>>>>>> being
>>>>>>>>>> able
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> login via ssh with your private key.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, 12 Aug 2020, 09:08 Ammad Syed, <syedammad83@gmail.com
>>>>>>>
>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Yes this is exactly the same issue that i faced.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Sent from my iPhone
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 12-Aug-2020, at 8:35 AM, Eric Lee Green <
>>>>>>>>>>>>>> eric.lee.green@gmail.com>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Correct, 4.11.3 template is used for 4.11.3, 4.12, and
>>>>>>> 4.13.
>>>>>>>>> 4.14
>>>>>>>>>>>>>> moves
>>>>>>>>>>>>>>> to the 4.14.0 template.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> There seems to be something odd happening key-wise
>>>>>> sometimes
>>>>>>>>> with
>>>>>>>>>>>>>>> upgrades from 4.11.3 to 4.13.1 or 4.14.0. I managed an
>>>>>>> upgrade
>>>>>>>>>> from
>>>>>>>>>>>>>>> 4.11.3 to 4.13.1 that *almost* worked, but the secondary
>>>>>>> storage
>>>>>>>>> VM
>>>>>>>>>>>>>>> wouldn't work and thus I couldn't spawn new virtual
>>>>>> machines.
>>>>>>>> Same
>>>>>>>>>>>>>> symptom
>>>>>>>>>>>>>>> -- key error when the agent tried to ssh into it. And
>>>>>> deleting
>>>>>>>> it
>>>>>>>>>> and
>>>>>>>>>>>>>>> making it respawn didn't help. Then I tried 4.11.3 to 4.14.0
>>>>>>> and
>>>>>>>>>>>>>> *all* the
>>>>>>>>>>>>>>> VM's failed at that point (of course, that was with the new
>>>>>>>>>> template).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Right now I'm back at 4.11.3 until this can be figured
>>>>>> out.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On 8/11/2020 5:53 AM, Ammad Syed wrote:
>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I think 4.12 and 4.13 uses same systemVM template i.e
>>>>>>> 4.11.3
>>>>>>>>>>>>>> version,
>>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>>>> I already have registered. Currently I am running 4.11.3
>>>>>>>>> version
>>>>>>>>>>>>>> of ACS.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> MariaDB [cloud]> SELECT id,name,type,cross_zones,state
>>>>>> FROM
>>>>>>>>>>>>>>>>> cloud.vm_template WHERE name like '%systemvm-xenserver%'
>>>>>>> AND
>>>>>>>>>>>>>> removed IS
>>>>>>>>>>>>>>>>> NULL;
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>> +------+-----------------------------+--------+-------------+----------+
>>>>>>>>>>>>>>>>> | id | name | type |
>>>>>>> cross_zones |
>>>>>>>>>>>>>> state |
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>> +------+-----------------------------+--------+-------------+----------+
>>>>>>>>>>>>>>>>> | 337 | systemvm-xenserver-3.0.0 | SYSTEM |
>>>>>>> 0 |
>>>>>>>>>>>>>> Inactive |
>>>>>>>>>>>>>>>>> | 418 | systemvm-xenserver-4.2 | SYSTEM |
>>>>>>> 0 |
>>>>>>>>>>>>>> Active |
>>>>>>>>>>>>>>>>> | 472 | systemvm-xenserver-4.3 | USER |
>>>>>>> 1 |
>>>>>>>>>>>>>> Inactive |
>>>>>>>>>>>>>>>>> | 473 | systemvm-xenserver-4.3 | USER |
>>>>>>> 1 |
>>>>>>>>>>>>>> Inactive |
>>>>>>>>>>>>>>>>> | 474 | systemvm-xenserver-4.3 | USER |
>>>>>>> 1 |
>>>>>>>>>>>>>> Inactive |
>>>>>>>>>>>>>>>>> | 475 | systemvm-xenserver-4.3 | USER |
>>>>>>> 1 |
>>>>>>>>>>>>>> Inactive |
>>>>>>>>>>>>>>>>> | 476 | systemvm-xenserver-4.3 | USER |
>>>>>>> 0 |
>>>>>>>>>>>>>> Inactive |
>>>>>>>>>>>>>>>>> | 479 | systemvm-xenserver-4.3-2 | USER |
>>>>>>> 1 |
>>>>>>>>>>>>>> Inactive |
>>>>>>>>>>>>>>>>> | 480 | systemvm-xenserver-4.3 | SYSTEM |
>>>>>>> 0 |
>>>>>>>>>>>>>> Active |
>>>>>>>>>>>>>>>>> | 549 | systemvm-xenserver-4.5.1 | USER |
>>>>>>> 0 |
>>>>>>>>>>>>>> Active |
>>>>>>>>>>>>>>>>> | 550 | systemvm-xenserver-4.5.1 | SYSTEM |
>>>>>>> 0 |
>>>>>>>>>>>>>> Active |
>>>>>>>>>>>>>>>>> | 651 | systemvm-xenserver-4.7.0 | USER |
>>>>>>> 0 |
>>>>>>>>>>>>>> Inactive |
>>>>>>>>>>>>>>>>> | 652 | systemvm-xenserver-4.7.0 | USER |
>>>>>>> 0 |
>>>>>>>>>>>>>> Inactive |
>>>>>>>>>>>>>>>>> | 653 | systemvm-xenserver-4.7.0 | SYSTEM |
>>>>>>> 0 |
>>>>>>>>>>>>>> Inactive |
>>>>>>>>>>>>>>>>> | 737 | systemvm-xenserver-4.9.2 | SYSTEM |
>>>>>>> 1 |
>>>>>>>>>>>>>> Inactive |
>>>>>>>>>>>>>>>>> | 739 | systemvm-xenserver-4.9.2-sb | SYSTEM |
>>>>>>> 1 |
>>>>>>>>>>>>>> Active |
>>>>>>>>>>>>>>>>> | 1245 | systemvm-xenserver-4.11.1 | SYSTEM |
>>>>>>> 1 |
>>>>>>>>>>>>>> Active |
>>>>>>>>>>>>>>>>> | 1584 | systemvm-xenserver-4.11.2 | SYSTEM |
>>>>>>> 1 |
>>>>>>>>>>>>>> Active |
>>>>>>>>>>>>>>>>> | 1677 | systemvm-xenserver-4.11.3 | SYSTEM |
>>>>>>> 1 |
>>>>>>>>>>>>>> Active |
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>> +------+-----------------------------+--------+-------------+----------+
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> - Ammad
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Tue, Aug 11, 2020 at 5:17 PM Pierre-Luc Dion <
>>>>>>>>>>>>>> pdion891@apache.org>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>> db.
>>>>>>>>>>>>>>>>>> Hi Syed,
>>>>>>>>>>>>>>>>>> From 4.12, the systemvm template had to be upgraded
>>>>>>> because
>>>>>>>> of
>>>>>>>>>> OS
>>>>>>>>>>>>>>> change in
>>>>>>>>>>>>>>>>>> the template, moved to a latest version of Debian.
>>>>>> Because
>>>>>>>> of
>>>>>>>>>>>>>> that,
>>>>>>>>>>>>>>> some VR
>>>>>>>>>>>>>>>>>> scripts have changed and make obsolete older version of
>>>>>>> VRs,
>>>>>>>>> so
>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>> most likely have to register an updated systemvm
>>>>>> templates
>>>>>>>> and
>>>>>>>>>>>>>> upgrade
>>>>>>>>>>>>>>> your
>>>>>>>>>>>>>>>>>> system VMs and VRs.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Tue, Aug 11, 2020 at 6:24 AM Ammad Syed <
>>>>>>>>>>>>>> syedammad83@gmail.com>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi Guys,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I was previously on 4.9.3 cloudstack and upgraded to
>>>>>>> 4.11.1
>>>>>>>>>> then
>>>>>>>>>>>>>>> 4.11.3.
>>>>>>>>>>>>>>>>>>> The version 4.11.3 is working fine since six months.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Now I have tried to upgrade my system from 4.11.3 to
>>>>>>>> 4.13.1.
>>>>>>>>>> The
>>>>>>>>>>>>>>> upgrade
>>>>>>>>>>>>>>>>>>> goes successful. I didn't uploaded any system VM
>>>>>>> template.
>>>>>>>>>>>>>> However the
>>>>>>>>>>>>>>>>>>> problem occured when I recreated my systemVM of POD,
>>>>>> the
>>>>>>> VM
>>>>>>>>>>>>>> recreated
>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>> its state was running but agent state was not getting
>>>>>> up,
>>>>>>>> its
>>>>>>>>>>>>>> showing
>>>>>>>>>>>>>>>>>> blank
>>>>>>>>>>>>>>>>>>> in column.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Digging further via job logs, the job is failed with
>>>>>>> error
>>>>>>>>> that
>>>>>>>>>>>>>>> unable to
>>>>>>>>>>>>>>>>>>> execute command via ssh. Below are the logs.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 2020-07-25 02:30:48,126 ERROR [c.c.u.s.SshHelper]
>>>>>>>>>>>>>>>>>>> (DirectAgent-211:ctx-62f09b31) (logid:9fa7dece) SSH
>>>>>>>> execution
>>>>>>>>>> of
>>>>>>>>>>>>>>> command
>>>>>>>>>>>>>>>>>>> /opt/cloud/bin/router_proxy.sh keystore-s
>>>>>>>>>>>>>>>>>>> etup 169.254.2.199
>>>>>>>>>>>>>> /usr/local/cloud/systemvm/conf/agent.properties
>>>>>>>>>>>>>>>>>>> /usr/local/cloud/systemvm/conf/cloud.jks
>>>>>> TJaQYChYBwKh7Cx9
>>>>>>>> 365
>>>>>>>>>>>>>>>>>>> /usr/local/cloud/systemvm/conf/clou
>>>>>>>>>>>>>>>>>>> d.csr has an error status code in return. Result
>>>>>> output:
>>>>>>>>>>>>>>>>>>> 2020-07-25 02:30:48,127 DEBUG
>>>>>>> [c.c.a.m.DirectAgentAttache]
>>>>>>>>>>>>>>>>>>> (DirectAgent-211:ctx-62f09b31) (logid:9fa7dece) Seq
>>>>>>>>>>>>>>>>>>> 906-3195585410596077730: Response Received:
>>>>>>>>>>>>>>>>>>> 2020-07-25 02:30:48,127 DEBUG [c.c.a.t.Request]
>>>>>>>>>>>>>>>>>>> (DirectAgent-211:ctx-62f09b31) (logid:9fa7dece) Seq
>>>>>>>>>>>>>>>>>>> 906-3195585410596077730: Processing: { Ans: , MgmtId:
>>>>>>>>>> 779271079
>>>>>>>>>>>>>>>>>>> 43497, via: 906(xen-21-10-a3-khi02), Ver: v1, Flags:
>>>>>> 10,
>>>>>>>>>>>>>>>>>>> [{"org.apache.cloudstack.ca
>>>>>>>>>>>>>>>>>>> .SetupKeystoreAnswer":{"result":false,"wait":0}}]
>>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>> 2020-07-25 02:30:48,127 DEBUG [c.c.a.t.Request]
>>>>>>>>>>>>>>>>>>> (Work-Job-Executor-41:ctx-4e3c666d
>>>>>>> job-1208155/job-1208258
>>>>>>>>>>>>>>> ctx-df740f75)
>>>>>>>>>>>>>>>>>>> (logid:9fa7dece) Seq 906-319558541059607773
>>>>>>>>>>>>>>>>>>> 0: Received: { Ans: , MgmtId: 77927107943497, via:
>>>>>>>>>>>>>>>>>>> 906(xen-21-10-a3-khi02), Ver: v1, Flags: 10, {
>>>>>>>>>>>>>> SetupKeystoreAnswer } }
>>>>>>>>>>>>>>>>>>> 2020-07-25 02:30:48,127 ERROR
>>>>>>>>> [c.c.v.VirtualMachineManagerImpl]
>>>>>>>>>>>>>>>>>>> (Work-Job-Executor-41:ctx-4e3c666d
>>>>>>> job-1208155/job-1208258
>>>>>>>>>>>>>>> ctx-df740f75)
>>>>>>>>>>>>>>>>>>> (logid:9fa7dece) Failed to setup keystore and generate
>>>>>>> CSR
>>>>>>>>> for
>>>>>>>>>>>>>> system
>>>>>>>>>>>>>>> vm:
>>>>>>>>>>>>>>>>>>> s-24142-VM
>>>>>>>>>>>>>>>>>>> 2020-07-25 02:30:48,127 DEBUG
>>>>>>> [c.c.v.VmWorkJobHandlerProxy]
>>>>>>>>>>>>>>>>>>> (Work-Job-Executor-41:ctx-4e3c666d
>>>>>>> job-1208155/job-1208258
>>>>>>>>>>>>>>> ctx-df740f75)
>>>>>>>>>>>>>>>>>>> (logid:9fa7dece) Done executing VM work job:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":24142,"handlerName":"VirtualMachineManagerImpl"}
>>>>>>>>>>>>>>>>>>> 2020-07-25 02:30:48,128 DEBUG
>>>>>>>>> [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>>>>>>>>>>>>>> (Work-Job-Executor-41:ctx-4e3c666d
>>>>>>> job-1208155/job-1208258
>>>>>>>>>>>>>>> ctx-df740f75)
>>>>>>>>>>>>>>>>>>> (logid:9fa7dece) Complete async job-1208258, jobStatus:
>>>>>>>>>>>>>> SUCCEEDED,
>>>>>>>>>>>>>>>>>>> resultCode: 0, result: null
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I tried to dig it further, I was unable to login
>>>>>> systemVM
>>>>>>>> via
>>>>>>>>>>>>>> ssh from
>>>>>>>>>>>>>>>>>>> xenserver host with key /root/.ssh/id_rsa.cloud placed.
>>>>>>>> Look
>>>>>>>>>> like
>>>>>>>>>>>>>>> private
>>>>>>>>>>>>>>>>>>> key issue. However I am able to login on my old
>>>>>>> systemVMs (
>>>>>>>>> i.e
>>>>>>>>>>>>>>> created
>>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>>> ACS 4.11.3)
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Also I have SSL certificate enabled for console proxy
>>>>>> on
>>>>>>> my
>>>>>>>>> ACS
>>>>>>>>>>>>>> 4.11.3
>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>> I am using only xenserver 7.0 hosts.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I tried to disable SSL on secstorage and console proxy
>>>>>>> from
>>>>>>>>>>>>>> global
>>>>>>>>>>>>>>>>>>> settings, but still didn't worked.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I had a fresh installation of ACS 4.13.1 with xenserver
>>>>>>>> 7.0,
>>>>>>>>>>>>>> systemVMs
>>>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>>> working fine in it.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Please advise.
>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Syed Ammad Ali
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Syed Ammad Ali
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Regards,
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Syed Ammad Ali
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Regards,
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Syed Ammad Ali
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Regards,
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Syed Ammad Ali
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>>
>>>>>>>>> Andrija Panić
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Regards,
>>>>>>>>
>>>>>>>>
>>>>>>>> Syed Ammad Ali
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> Andrija Panić
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Regards,
>>>>>>
>>>>>>
>>>>>> Syed Ammad Ali
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Andrija Panić
>>>>
>>>>
>>>> --
>>>> Regards,
>>>>
>>>>
>>>> Syed Ammad Ali
>>
>
Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error
Posted by Ammad Syed <sy...@gmail.com>.
Hi,
I have performed an upgrade on one of my other production environments and
have three XS7.0 clusters with latest hotfixes. I have performed an upgrade
from 4.11.3 to 4.13.1. The systemVM deployment in one cluster is successful
but getting failed in another cluster.
https://drive.google.com/file/d/1eBD20a1WJfe_eYCaTxM5JneMBnnebkd1/view?usp=sharing
You can review logs from above link. Below are the systemVMs that have
failed to make agent up, the XS tools was installed successfully in it.
2020-09-15 12:37:52,415 ERROR [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-2:ctx-e478e44a job-13124/job-13131 ctx-7af48048)
(logid:f96bf120) Failed to setup keystore and generate CSR for system vm:
v-753-VM
2020-09-15 12:59:27,658 ERROR [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-15:ctx-5a06ea58 job-13124/job-13158 ctx-61790402)
(logid:f96bf120) Failed to setup keystore and generate CSR for system vm:
v-758-VM
2020-09-15 13:06:14,731 ERROR [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-24:ctx-4f858410 job-13124/job-13174 ctx-d127150a)
(logid:f96bf120) Failed to setup keystore and generate CSR for system vm:
v-760-VM
2020-09-15 13:21:37,907 ERROR [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-3:ctx-4c2ba231 job-13175/job-13184 ctx-6228d3cf)
(logid:6bd89e55) Failed to setup keystore and generate CSR for system vm:
v-761-VM
2020-09-15 13:25:43,696 ERROR [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-5:ctx-4ecf72a0 job-13175/job-13195 ctx-2aa50a59)
(logid:6bd89e55) Failed to setup keystore and generate CSR for system vm:
v-761-VM
You can check below systemVMs that goes up in cluster and have agent
connected. Its a weird problem, I didn't see any error XS logs.
v-754-VM
s-756-VM
Ammad Ali
On Fri, Oct 9, 2020 at 12:28 PM Andrija Panic <an...@gmail.com>
wrote:
> This error is a different zone, as I can see (what I shared above).
>
> For the VM which you asked us to check logs - can you see which zone this
> is in and then:
> - disable the zone,
> - destroy the SSVM (if it is showing as startING or stopING state - make
> sure to destroy the VM on XenCenter first, then edit the vm_instances table
> to set "state"=Stopped" for that s-25457-VM) - then destroy in ACS
> - enable the zone again, it should create a brand new SSVM (check that your
> CPUs are not heavily oversubscribed, as, in theory, this could also be an
> extremely slow CPU issue)
>
> If the issue continues, please post the log again (to pastebin or
> elsewhere) - and share the new SSVM name that failed to be configured.
> -- it's worth digging in XS logs to see if you have some other issues,
> which ACS is not aware of.
>
>
> Best,
>
> On Fri, 9 Oct 2020 at 09:09, Andrija Panic <an...@gmail.com>
> wrote:
>
> > Ammad,
> >
> > what's you current status with your issue?
> >
> > In logs I can see that there is some issue with SR:
> >
> > 2020-09-10 22:58:48,374 DEBUG
> [c.c.h.x.r.w.x.CitrixStartCommandWrapper]
> > (DirectAgent-270:ctx-3468556a) (logid:4cc5809d) 1. The VM s-25505-VM is
> in
> > Starting state.
> > 2020-09-10 22:58:48,375 DEBUG [c.c.h.x.r.CitrixResourceBase]
> > (DirectAgent-270:ctx-3468556a) (logid:4cc5809d) no guest OS type, start
> it
> > as HVM guest
> > 2020-09-10 22:58:48,390 DEBUG [c.c.h.x.r.CitrixResourceBase]
> > (DirectAgent-270:ctx-3468556a) (logid:4cc5809d) Created VM
> > 3fa168bd-9e29-03df-52bb-a0fe2a53d390 for s-25505-VM
> > 2020-09-10 22:58:48,394 DEBUG [c.c.h.x.r.CitrixResourceBase]
> > (DirectAgent-270:ctx-3468556a) (logid:4cc5809d) PV args are
> >
> %template=domP%type=secstorage%host=172.16.2.42%port=8250%name=s-25505-VM%zone=10%pod=10%guid=s-25505-VM%workers=5%resource=com.cloud.storage.resource.PremiumSecondaryStorageResource%instance=SecStorage%sslcopy=false%role=templateProcessor%mtu=1500%eth2ip=175.107.206.196%eth2mask=255.255.255.0%gateway=175.107.206.1%public.network.device=eth2%eth0ip=169.254.1.184%eth0mask=255.255.0.0%eth1ip=172.16.2.56%eth1mask=255.255.255.192%mgmtcidr=
> >
> 172.16.0.0/26%localgw=172.16.2.62%private.network.device=eth1%internaldns1=202.163.96.3%internaldns2=202.163.96.4%dns1=202.163.96.3%dns2=202.163.96.4%nfsVersion=null
> > 2020-09-10 22:58:48,396 DEBUG [c.c.h.x.r.CitrixResourceBase]
> > (DirectAgent-270:ctx-3468556a) (logid:4cc5809d) HVM args are
> template=domP
> > type=secstorage host=172.16.2.42 port=8250 name=s-25505-VM zone=10 pod=10
> > guid=s-25505-VM workers=5
> > resource=com.cloud.storage.resource.PremiumSecondaryStorageResource
> > instance=SecStorage sslcopy=false role=templateProcessor mtu=1500
> > eth2ip=175.107.206.196 eth2mask=255.255.255.0 gateway=175.107.206.1
> > public.network.device=eth2 eth0ip=169.254.1.184 eth0mask=255.255.0.0
> > eth1ip=172.16.2.56 eth1mask=255.255.255.192 mgmtcidr=172.16.0.0/26
> > localgw=172.16.2.62 private.network.device=eth1 internaldns1=202.163.96.3
> > internaldns2=202.163.96.4 dns1=202.163.96.3 dns2=202.163.96.4
> > nfsVersion=null
> > 2020-09-10 22:58:48,399 DEBUG [c.c.h.x.r.CitrixResourceBase]
> > (DirectAgent-270:ctx-3468556a) (logid:4cc5809d) Failed to find SR by name
> > 'XenServer Tools', will try to find 'XCP-ng Tools' SR
> > 2020-09-10 22:58:48,400 WARN
> [c.c.h.x.r.w.x.CitrixStartCommandWrapper]
> > (DirectAgent-270:ctx-3468556a) (logid:4cc5809d) Catch Exception: class
> > com.cloud.utils.exception.CloudRuntimeException due to
> > com.cloud.utils.exception.CloudRuntimeException: There are 0 SRs with
> name
> > XenServer Tools
> > com.cloud.utils.exception.CloudRuntimeException: There are 0 SRs with
> name
> > XenServer Tools
> > at
> >
> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.createPatchVbd(CitrixResourceBase.java:1061)
> >
> > Your SSVM can not even start, leave alone trying to access it etc.
> > Also, after the upgrade, I saw that the public SSH key was successfully
> > injected into the systemvm.iso, and that ISO is copied to all hosts, as
> > well as the current id_rsa private key - so assuming your SSVM starts
> > sucessfully, you should be able to access it, as a) you have the id_rsa
> on
> > each XS host, and b) systemvm.iso does contain a good public_key(inside
> the
> > authorized_keys file) - so assuming netwotk is UP (169.254...), you
> should
> > be able to access it via ssh on port 3922.
> >
> > Best,
> >
> >
> >
> > On Mon, 14 Sep 2020 at 11:39, Ammad Syed <sy...@gmail.com> wrote:
> >
> >> You guys can check job-1248274 logs or SSVM s-25457-VM that has
> >> successfully connected agent state.
> >>
> >> Ammad Ali
> >>
> >> On Mon, Sep 14, 2020 at 2:33 PM Ammad Syed <sy...@gmail.com>
> wrote:
> >>
> >> > Here are the upgrade management server logs.
> >> >
> >> >
> >> >
> >>
> https://drive.google.com/file/d/17cxh7f-24ibnXCKvUUPqt3p1YTzlMQN8/view?usp=sharing
> >> >
> >> > Ammad Ali
> >> >
> >> > On Mon, Sep 14, 2020 at 2:17 PM Ammad Syed <sy...@gmail.com>
> >> wrote:
> >> >
> >> >> In addition to previous email there is only one host in one zone
> where
> >> >> systemVM agent goes up and on all other hosts on that zone agent
> >> failed.
> >> >>
> >> >> If you guys need, I can provide management server logs as well.
> >> >>
> >> >> Also is there a way to enable debugging in ACS logs to specifically
> >> find
> >> >> out where the problem is?
> >> >>
> >> >> Ammad Ali
> >> >>
> >> >> On Mon, Sep 14, 2020 at 10:39 AM Ammad Syed <sy...@gmail.com>
> >> >> wrote:
> >> >>
> >> >>> Hi Perl,
> >> >>>
> >> >>> I have taken those steps and verified that systemvm.iso is copied to
> >> all
> >> >>> hosts in all zones.
> >> >>>
> >> >>> I have recreated the systemvm and ssh to that host and checked the
> >> >>> md5sum of iso there and on acs. Both were same.
> >> >>>
> >> >>> However the md5sum on which systemvm was working has also same
> md5sum
> >> of
> >> >>> systemvm iso. The iso is getting copied successfully. The problem
> >> looks
> >> >>> somewhere else.
> >> >>>
> >> >>> I have checked in xenserver logs as well but didn’t find any logs
> that
> >> >>> something has failed.
> >> >>>
> >> >>> Ammad
> >> >>> Sent from my iPhone
> >> >>>
> >> >>> > On 14-Sep-2020, at 9:15 AM, Pearl d'Silva <
> >> pearl.dsilva@shapeblue.com>
> >> >>> wrote:
> >> >>> >
> >> >>> > Hi Ammad,
> >> >>> >
> >> >>> > Is the understanding right that the steps as mentioned by you in
> the
> >> >>> previous mail has in-fact worked on one zone? If that's the case,
> >> could you
> >> >>> please ensure that all the hosts in all the other zones have the new
> >> >>> systemVM iso copied into them by checking the timestamps as well and
> >> >>> comparing the checksums against the iso on the management server, so
> >> that
> >> >>> when the system VM's are recreated, they pick up the new iso.
> >> >>> >
> >> >>> > Thanks,
> >> >>> > Pearl
> >> >>> >
> >> >>> > ________________________________
> >> >>> > From: Ammad Syed <sy...@gmail.com>
> >> >>> > Sent: Sunday, September 13, 2020 12:49 PM
> >> >>> > To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> >> >>> > Subject: Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error
> >> >>> >
> >> >>> > Hi Andrija,
> >> >>> >
> >> >>> > Here is the permission of mount point in SSVM.
> >> >>> >
> >> >>> > root@s-25437-VM:~# df -h
> >> >>> > Filesystem Size Used Avail Use% Mounted on
> >> >>> > udev 233M 0 233M 0% /dev
> >> >>> > tmpfs 98M 19M 80M 19% /run
> >> >>> > /dev/xvda5 1.1G 773M 282M 74% /
> >> >>> > tmpfs 244M 0 244M 0% /dev/shm
> >> >>> > tmpfs 5.0M 0 5.0M 0% /run/lock
> >> >>> > tmpfs 244M 0 244M 0% /sys/fs/cgroup
> >> >>> > /dev/xvda1 92M 35M 57M 39% /boot
> >> >>> > /dev/xvda6 435M 21M 410M 5% /var
> >> >>> > /dev/xvda7 75M 1.6M 72M 3% /tmp
> >> >>> > 172.16.10.35:/nfs/KHI02 12T 7.0T 5.1T 59%
> >> >>> > /mnt/SecStorage/8ea71ccb-e493-3c7e-8bb0-97871f5c2092
> >> >>> > tmpfs 49M 0 49M 0% /run/user/0
> >> >>> > root@s-25437-VM:~#
> >> >>> > root@s-25437-VM:~#
> >> >>> > root@s-25437-VM:~# cd
> >> >>> /mnt/SecStorage/8ea71ccb-e493-3c7e-8bb0-97871f5c2092
> >> >>> > root@s-25437-VM
> >> :/mnt/SecStorage/8ea71ccb-e493-3c7e-8bb0-97871f5c2092#
> >> >>> > root@s-25437-VM
> >> :/mnt/SecStorage/8ea71ccb-e493-3c7e-8bb0-97871f5c2092#
> >> >>> ls
> >> >>> > -lah
> >> >>> > total 12K
> >> >>> > drwxrwxrwx 5 root root 70 Dec 21 2018 .
> >> >>> > drwxrwxrwx 3 root root 4.0K Sep 10 23:12 ..
> >> >>> > drwxrwxrwx 52 root root 4.0K Aug 13 19:37 snapshots
> >> >>> > drwxrwxrwx 3 root root 26 Jun 7 2013 template
> >> >>> > drwxrwxrwx 98 root root 4.0K Sep 1 12:52 volumes
> >> >>> >
> >> >>> > -Ammad Ali
> >> >>> >
> >> >>> >> On Fri, Sep 11, 2020 at 6:09 PM Andrija Panic <
> >> >>> andrija.panic@gmail.com>
> >> >>> >> wrote:
> >> >>> >>
> >> >>> >> Can you share permissions of your secondary storage (mount it
> then
> >> ls
> >> >>> -lah
> >> >>> >> the mount point)
> >> >>> >>
> >> >>> >
> >> >>> > pearl.dsilva@shapeblue.com
> >> >>> > www.shapeblue.com
> >> >>> > 3 London Bridge Street, 3rd floor, News Building, London SE1
> 9SGUK
> >> >>> > @shapeblue
> >> >>> >
> >> >>> >
> >> >>> >
> >> >>> >>> On Fri, 11 Sep 2020 at 01:39, Ammad Syed <syedammad83@gmail.com
> >
> >> >>> wrote:
> >> >>> >>>
> >> >>> >>> Hi Andrija,
> >> >>> >>>
> >> >>> >>> I have performed an upgrade on my production system from 4.11.3
> to
> >> >>> >> 4.13.1.
> >> >>> >>> Even I cleared the tags but the issue is still there.
> >> >>> >>>
> >> >>> >>> I have four zones and only in one zone and on specific host, the
> >> >>> >>> systemVM's agent goes up but on all other zones, key injection
> to
> >> >>> >> systemVM
> >> >>> >>> is still failing on all zones and PODs. I have checked, the
> >> updated
> >> >>> ISO
> >> >>> >> is
> >> >>> >>> there on all hosts. The md5sum of systemvm.iso is same on xen
> >> hosts
> >> >>> and
> >> >>> >> acs
> >> >>> >>> host.
> >> >>> >>>
> >> >>> >>> Look like a weird problem. How can I troubleshoot this further ?
> >> Any
> >> >>> >> advise
> >> >>> >>> would be appreciated.
> >> >>> >>>
> >> >>> >>> -Ammad
> >> >>> >>>
> >> >>> >>
> >> >>> >>
> >> >>> >> --
> >> >>> >>
> >> >>> >> Andrija Panić
> >> >>> >>
> >> >>> >
> >> >>> >
> >> >>> > --
> >> >>> > Regards,
> >> >>> >
> >> >>> >
> >> >>> > Syed Ammad Ali
> >> >>>
> >> >>
> >> >>
> >> >> --
> >> >> Regards,
> >> >>
> >> >>
> >> >> Syed Ammad Ali
> >> >>
> >> >
> >> >
> >> > --
> >> > Regards,
> >> >
> >> >
> >> > Syed Ammad Ali
> >> >
> >>
> >>
> >> --
> >> Regards,
> >>
> >>
> >> Syed Ammad Ali
> >>
> >
> >
> > --
> >
> > Andrija Panić
> >
>
>
> --
>
> Andrija Panić
>
--
Regards,
Syed Ammad Ali
Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error
Posted by Andrija Panic <an...@gmail.com>.
This error is a different zone, as I can see (what I shared above).
For the VM which you asked us to check logs - can you see which zone this
is in and then:
- disable the zone,
- destroy the SSVM (if it is showing as startING or stopING state - make
sure to destroy the VM on XenCenter first, then edit the vm_instances table
to set "state"=Stopped" for that s-25457-VM) - then destroy in ACS
- enable the zone again, it should create a brand new SSVM (check that your
CPUs are not heavily oversubscribed, as, in theory, this could also be an
extremely slow CPU issue)
If the issue continues, please post the log again (to pastebin or
elsewhere) - and share the new SSVM name that failed to be configured.
-- it's worth digging in XS logs to see if you have some other issues,
which ACS is not aware of.
Best,
On Fri, 9 Oct 2020 at 09:09, Andrija Panic <an...@gmail.com> wrote:
> Ammad,
>
> what's you current status with your issue?
>
> In logs I can see that there is some issue with SR:
>
> 2020-09-10 22:58:48,374 DEBUG [c.c.h.x.r.w.x.CitrixStartCommandWrapper]
> (DirectAgent-270:ctx-3468556a) (logid:4cc5809d) 1. The VM s-25505-VM is in
> Starting state.
> 2020-09-10 22:58:48,375 DEBUG [c.c.h.x.r.CitrixResourceBase]
> (DirectAgent-270:ctx-3468556a) (logid:4cc5809d) no guest OS type, start it
> as HVM guest
> 2020-09-10 22:58:48,390 DEBUG [c.c.h.x.r.CitrixResourceBase]
> (DirectAgent-270:ctx-3468556a) (logid:4cc5809d) Created VM
> 3fa168bd-9e29-03df-52bb-a0fe2a53d390 for s-25505-VM
> 2020-09-10 22:58:48,394 DEBUG [c.c.h.x.r.CitrixResourceBase]
> (DirectAgent-270:ctx-3468556a) (logid:4cc5809d) PV args are
> %template=domP%type=secstorage%host=172.16.2.42%port=8250%name=s-25505-VM%zone=10%pod=10%guid=s-25505-VM%workers=5%resource=com.cloud.storage.resource.PremiumSecondaryStorageResource%instance=SecStorage%sslcopy=false%role=templateProcessor%mtu=1500%eth2ip=175.107.206.196%eth2mask=255.255.255.0%gateway=175.107.206.1%public.network.device=eth2%eth0ip=169.254.1.184%eth0mask=255.255.0.0%eth1ip=172.16.2.56%eth1mask=255.255.255.192%mgmtcidr=
> 172.16.0.0/26%localgw=172.16.2.62%private.network.device=eth1%internaldns1=202.163.96.3%internaldns2=202.163.96.4%dns1=202.163.96.3%dns2=202.163.96.4%nfsVersion=null
> 2020-09-10 22:58:48,396 DEBUG [c.c.h.x.r.CitrixResourceBase]
> (DirectAgent-270:ctx-3468556a) (logid:4cc5809d) HVM args are template=domP
> type=secstorage host=172.16.2.42 port=8250 name=s-25505-VM zone=10 pod=10
> guid=s-25505-VM workers=5
> resource=com.cloud.storage.resource.PremiumSecondaryStorageResource
> instance=SecStorage sslcopy=false role=templateProcessor mtu=1500
> eth2ip=175.107.206.196 eth2mask=255.255.255.0 gateway=175.107.206.1
> public.network.device=eth2 eth0ip=169.254.1.184 eth0mask=255.255.0.0
> eth1ip=172.16.2.56 eth1mask=255.255.255.192 mgmtcidr=172.16.0.0/26
> localgw=172.16.2.62 private.network.device=eth1 internaldns1=202.163.96.3
> internaldns2=202.163.96.4 dns1=202.163.96.3 dns2=202.163.96.4
> nfsVersion=null
> 2020-09-10 22:58:48,399 DEBUG [c.c.h.x.r.CitrixResourceBase]
> (DirectAgent-270:ctx-3468556a) (logid:4cc5809d) Failed to find SR by name
> 'XenServer Tools', will try to find 'XCP-ng Tools' SR
> 2020-09-10 22:58:48,400 WARN [c.c.h.x.r.w.x.CitrixStartCommandWrapper]
> (DirectAgent-270:ctx-3468556a) (logid:4cc5809d) Catch Exception: class
> com.cloud.utils.exception.CloudRuntimeException due to
> com.cloud.utils.exception.CloudRuntimeException: There are 0 SRs with name
> XenServer Tools
> com.cloud.utils.exception.CloudRuntimeException: There are 0 SRs with name
> XenServer Tools
> at
> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.createPatchVbd(CitrixResourceBase.java:1061)
>
> Your SSVM can not even start, leave alone trying to access it etc.
> Also, after the upgrade, I saw that the public SSH key was successfully
> injected into the systemvm.iso, and that ISO is copied to all hosts, as
> well as the current id_rsa private key - so assuming your SSVM starts
> sucessfully, you should be able to access it, as a) you have the id_rsa on
> each XS host, and b) systemvm.iso does contain a good public_key(inside the
> authorized_keys file) - so assuming netwotk is UP (169.254...), you should
> be able to access it via ssh on port 3922.
>
> Best,
>
>
>
> On Mon, 14 Sep 2020 at 11:39, Ammad Syed <sy...@gmail.com> wrote:
>
>> You guys can check job-1248274 logs or SSVM s-25457-VM that has
>> successfully connected agent state.
>>
>> Ammad Ali
>>
>> On Mon, Sep 14, 2020 at 2:33 PM Ammad Syed <sy...@gmail.com> wrote:
>>
>> > Here are the upgrade management server logs.
>> >
>> >
>> >
>> https://drive.google.com/file/d/17cxh7f-24ibnXCKvUUPqt3p1YTzlMQN8/view?usp=sharing
>> >
>> > Ammad Ali
>> >
>> > On Mon, Sep 14, 2020 at 2:17 PM Ammad Syed <sy...@gmail.com>
>> wrote:
>> >
>> >> In addition to previous email there is only one host in one zone where
>> >> systemVM agent goes up and on all other hosts on that zone agent
>> failed.
>> >>
>> >> If you guys need, I can provide management server logs as well.
>> >>
>> >> Also is there a way to enable debugging in ACS logs to specifically
>> find
>> >> out where the problem is?
>> >>
>> >> Ammad Ali
>> >>
>> >> On Mon, Sep 14, 2020 at 10:39 AM Ammad Syed <sy...@gmail.com>
>> >> wrote:
>> >>
>> >>> Hi Perl,
>> >>>
>> >>> I have taken those steps and verified that systemvm.iso is copied to
>> all
>> >>> hosts in all zones.
>> >>>
>> >>> I have recreated the systemvm and ssh to that host and checked the
>> >>> md5sum of iso there and on acs. Both were same.
>> >>>
>> >>> However the md5sum on which systemvm was working has also same md5sum
>> of
>> >>> systemvm iso. The iso is getting copied successfully. The problem
>> looks
>> >>> somewhere else.
>> >>>
>> >>> I have checked in xenserver logs as well but didn’t find any logs that
>> >>> something has failed.
>> >>>
>> >>> Ammad
>> >>> Sent from my iPhone
>> >>>
>> >>> > On 14-Sep-2020, at 9:15 AM, Pearl d'Silva <
>> pearl.dsilva@shapeblue.com>
>> >>> wrote:
>> >>> >
>> >>> > Hi Ammad,
>> >>> >
>> >>> > Is the understanding right that the steps as mentioned by you in the
>> >>> previous mail has in-fact worked on one zone? If that's the case,
>> could you
>> >>> please ensure that all the hosts in all the other zones have the new
>> >>> systemVM iso copied into them by checking the timestamps as well and
>> >>> comparing the checksums against the iso on the management server, so
>> that
>> >>> when the system VM's are recreated, they pick up the new iso.
>> >>> >
>> >>> > Thanks,
>> >>> > Pearl
>> >>> >
>> >>> > ________________________________
>> >>> > From: Ammad Syed <sy...@gmail.com>
>> >>> > Sent: Sunday, September 13, 2020 12:49 PM
>> >>> > To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
>> >>> > Subject: Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error
>> >>> >
>> >>> > Hi Andrija,
>> >>> >
>> >>> > Here is the permission of mount point in SSVM.
>> >>> >
>> >>> > root@s-25437-VM:~# df -h
>> >>> > Filesystem Size Used Avail Use% Mounted on
>> >>> > udev 233M 0 233M 0% /dev
>> >>> > tmpfs 98M 19M 80M 19% /run
>> >>> > /dev/xvda5 1.1G 773M 282M 74% /
>> >>> > tmpfs 244M 0 244M 0% /dev/shm
>> >>> > tmpfs 5.0M 0 5.0M 0% /run/lock
>> >>> > tmpfs 244M 0 244M 0% /sys/fs/cgroup
>> >>> > /dev/xvda1 92M 35M 57M 39% /boot
>> >>> > /dev/xvda6 435M 21M 410M 5% /var
>> >>> > /dev/xvda7 75M 1.6M 72M 3% /tmp
>> >>> > 172.16.10.35:/nfs/KHI02 12T 7.0T 5.1T 59%
>> >>> > /mnt/SecStorage/8ea71ccb-e493-3c7e-8bb0-97871f5c2092
>> >>> > tmpfs 49M 0 49M 0% /run/user/0
>> >>> > root@s-25437-VM:~#
>> >>> > root@s-25437-VM:~#
>> >>> > root@s-25437-VM:~# cd
>> >>> /mnt/SecStorage/8ea71ccb-e493-3c7e-8bb0-97871f5c2092
>> >>> > root@s-25437-VM
>> :/mnt/SecStorage/8ea71ccb-e493-3c7e-8bb0-97871f5c2092#
>> >>> > root@s-25437-VM
>> :/mnt/SecStorage/8ea71ccb-e493-3c7e-8bb0-97871f5c2092#
>> >>> ls
>> >>> > -lah
>> >>> > total 12K
>> >>> > drwxrwxrwx 5 root root 70 Dec 21 2018 .
>> >>> > drwxrwxrwx 3 root root 4.0K Sep 10 23:12 ..
>> >>> > drwxrwxrwx 52 root root 4.0K Aug 13 19:37 snapshots
>> >>> > drwxrwxrwx 3 root root 26 Jun 7 2013 template
>> >>> > drwxrwxrwx 98 root root 4.0K Sep 1 12:52 volumes
>> >>> >
>> >>> > -Ammad Ali
>> >>> >
>> >>> >> On Fri, Sep 11, 2020 at 6:09 PM Andrija Panic <
>> >>> andrija.panic@gmail.com>
>> >>> >> wrote:
>> >>> >>
>> >>> >> Can you share permissions of your secondary storage (mount it then
>> ls
>> >>> -lah
>> >>> >> the mount point)
>> >>> >>
>> >>> >
>> >>> > pearl.dsilva@shapeblue.com
>> >>> > www.shapeblue.com
>> >>> > 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
>> >>> > @shapeblue
>> >>> >
>> >>> >
>> >>> >
>> >>> >>> On Fri, 11 Sep 2020 at 01:39, Ammad Syed <sy...@gmail.com>
>> >>> wrote:
>> >>> >>>
>> >>> >>> Hi Andrija,
>> >>> >>>
>> >>> >>> I have performed an upgrade on my production system from 4.11.3 to
>> >>> >> 4.13.1.
>> >>> >>> Even I cleared the tags but the issue is still there.
>> >>> >>>
>> >>> >>> I have four zones and only in one zone and on specific host, the
>> >>> >>> systemVM's agent goes up but on all other zones, key injection to
>> >>> >> systemVM
>> >>> >>> is still failing on all zones and PODs. I have checked, the
>> updated
>> >>> ISO
>> >>> >> is
>> >>> >>> there on all hosts. The md5sum of systemvm.iso is same on xen
>> hosts
>> >>> and
>> >>> >> acs
>> >>> >>> host.
>> >>> >>>
>> >>> >>> Look like a weird problem. How can I troubleshoot this further ?
>> Any
>> >>> >> advise
>> >>> >>> would be appreciated.
>> >>> >>>
>> >>> >>> -Ammad
>> >>> >>>
>> >>> >>
>> >>> >>
>> >>> >> --
>> >>> >>
>> >>> >> Andrija Panić
>> >>> >>
>> >>> >
>> >>> >
>> >>> > --
>> >>> > Regards,
>> >>> >
>> >>> >
>> >>> > Syed Ammad Ali
>> >>>
>> >>
>> >>
>> >> --
>> >> Regards,
>> >>
>> >>
>> >> Syed Ammad Ali
>> >>
>> >
>> >
>> > --
>> > Regards,
>> >
>> >
>> > Syed Ammad Ali
>> >
>>
>>
>> --
>> Regards,
>>
>>
>> Syed Ammad Ali
>>
>
>
> --
>
> Andrija Panić
>
--
Andrija Panić
Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error
Posted by Andrija Panic <an...@gmail.com>.
Ammad,
what's you current status with your issue?
In logs I can see that there is some issue with SR:
2020-09-10 22:58:48,374 DEBUG [c.c.h.x.r.w.x.CitrixStartCommandWrapper]
(DirectAgent-270:ctx-3468556a) (logid:4cc5809d) 1. The VM s-25505-VM is in
Starting state.
2020-09-10 22:58:48,375 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-270:ctx-3468556a) (logid:4cc5809d) no guest OS type, start it
as HVM guest
2020-09-10 22:58:48,390 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-270:ctx-3468556a) (logid:4cc5809d) Created VM
3fa168bd-9e29-03df-52bb-a0fe2a53d390 for s-25505-VM
2020-09-10 22:58:48,394 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-270:ctx-3468556a) (logid:4cc5809d) PV args are
%template=domP%type=secstorage%host=172.16.2.42%port=8250%name=s-25505-VM%zone=10%pod=10%guid=s-25505-VM%workers=5%resource=com.cloud.storage.resource.PremiumSecondaryStorageResource%instance=SecStorage%sslcopy=false%role=templateProcessor%mtu=1500%eth2ip=175.107.206.196%eth2mask=255.255.255.0%gateway=175.107.206.1%public.network.device=eth2%eth0ip=169.254.1.184%eth0mask=255.255.0.0%eth1ip=172.16.2.56%eth1mask=255.255.255.192%mgmtcidr=
172.16.0.0/26%localgw=172.16.2.62%private.network.device=eth1%internaldns1=202.163.96.3%internaldns2=202.163.96.4%dns1=202.163.96.3%dns2=202.163.96.4%nfsVersion=null
2020-09-10 22:58:48,396 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-270:ctx-3468556a) (logid:4cc5809d) HVM args are template=domP
type=secstorage host=172.16.2.42 port=8250 name=s-25505-VM zone=10 pod=10
guid=s-25505-VM workers=5
resource=com.cloud.storage.resource.PremiumSecondaryStorageResource
instance=SecStorage sslcopy=false role=templateProcessor mtu=1500
eth2ip=175.107.206.196 eth2mask=255.255.255.0 gateway=175.107.206.1
public.network.device=eth2 eth0ip=169.254.1.184 eth0mask=255.255.0.0
eth1ip=172.16.2.56 eth1mask=255.255.255.192 mgmtcidr=172.16.0.0/26
localgw=172.16.2.62 private.network.device=eth1 internaldns1=202.163.96.3
internaldns2=202.163.96.4 dns1=202.163.96.3 dns2=202.163.96.4
nfsVersion=null
2020-09-10 22:58:48,399 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-270:ctx-3468556a) (logid:4cc5809d) Failed to find SR by name
'XenServer Tools', will try to find 'XCP-ng Tools' SR
2020-09-10 22:58:48,400 WARN [c.c.h.x.r.w.x.CitrixStartCommandWrapper]
(DirectAgent-270:ctx-3468556a) (logid:4cc5809d) Catch Exception: class
com.cloud.utils.exception.CloudRuntimeException due to
com.cloud.utils.exception.CloudRuntimeException: There are 0 SRs with name
XenServer Tools
com.cloud.utils.exception.CloudRuntimeException: There are 0 SRs with name
XenServer Tools
at
com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.createPatchVbd(CitrixResourceBase.java:1061)
Your SSVM can not even start, leave alone trying to access it etc.
Also, after the upgrade, I saw that the public SSH key was successfully
injected into the systemvm.iso, and that ISO is copied to all hosts, as
well as the current id_rsa private key - so assuming your SSVM starts
sucessfully, you should be able to access it, as a) you have the id_rsa on
each XS host, and b) systemvm.iso does contain a good public_key(inside the
authorized_keys file) - so assuming netwotk is UP (169.254...), you should
be able to access it via ssh on port 3922.
Best,
On Mon, 14 Sep 2020 at 11:39, Ammad Syed <sy...@gmail.com> wrote:
> You guys can check job-1248274 logs or SSVM s-25457-VM that has
> successfully connected agent state.
>
> Ammad Ali
>
> On Mon, Sep 14, 2020 at 2:33 PM Ammad Syed <sy...@gmail.com> wrote:
>
> > Here are the upgrade management server logs.
> >
> >
> >
> https://drive.google.com/file/d/17cxh7f-24ibnXCKvUUPqt3p1YTzlMQN8/view?usp=sharing
> >
> > Ammad Ali
> >
> > On Mon, Sep 14, 2020 at 2:17 PM Ammad Syed <sy...@gmail.com>
> wrote:
> >
> >> In addition to previous email there is only one host in one zone where
> >> systemVM agent goes up and on all other hosts on that zone agent failed.
> >>
> >> If you guys need, I can provide management server logs as well.
> >>
> >> Also is there a way to enable debugging in ACS logs to specifically find
> >> out where the problem is?
> >>
> >> Ammad Ali
> >>
> >> On Mon, Sep 14, 2020 at 10:39 AM Ammad Syed <sy...@gmail.com>
> >> wrote:
> >>
> >>> Hi Perl,
> >>>
> >>> I have taken those steps and verified that systemvm.iso is copied to
> all
> >>> hosts in all zones.
> >>>
> >>> I have recreated the systemvm and ssh to that host and checked the
> >>> md5sum of iso there and on acs. Both were same.
> >>>
> >>> However the md5sum on which systemvm was working has also same md5sum
> of
> >>> systemvm iso. The iso is getting copied successfully. The problem looks
> >>> somewhere else.
> >>>
> >>> I have checked in xenserver logs as well but didn’t find any logs that
> >>> something has failed.
> >>>
> >>> Ammad
> >>> Sent from my iPhone
> >>>
> >>> > On 14-Sep-2020, at 9:15 AM, Pearl d'Silva <
> pearl.dsilva@shapeblue.com>
> >>> wrote:
> >>> >
> >>> > Hi Ammad,
> >>> >
> >>> > Is the understanding right that the steps as mentioned by you in the
> >>> previous mail has in-fact worked on one zone? If that's the case,
> could you
> >>> please ensure that all the hosts in all the other zones have the new
> >>> systemVM iso copied into them by checking the timestamps as well and
> >>> comparing the checksums against the iso on the management server, so
> that
> >>> when the system VM's are recreated, they pick up the new iso.
> >>> >
> >>> > Thanks,
> >>> > Pearl
> >>> >
> >>> > ________________________________
> >>> > From: Ammad Syed <sy...@gmail.com>
> >>> > Sent: Sunday, September 13, 2020 12:49 PM
> >>> > To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> >>> > Subject: Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error
> >>> >
> >>> > Hi Andrija,
> >>> >
> >>> > Here is the permission of mount point in SSVM.
> >>> >
> >>> > root@s-25437-VM:~# df -h
> >>> > Filesystem Size Used Avail Use% Mounted on
> >>> > udev 233M 0 233M 0% /dev
> >>> > tmpfs 98M 19M 80M 19% /run
> >>> > /dev/xvda5 1.1G 773M 282M 74% /
> >>> > tmpfs 244M 0 244M 0% /dev/shm
> >>> > tmpfs 5.0M 0 5.0M 0% /run/lock
> >>> > tmpfs 244M 0 244M 0% /sys/fs/cgroup
> >>> > /dev/xvda1 92M 35M 57M 39% /boot
> >>> > /dev/xvda6 435M 21M 410M 5% /var
> >>> > /dev/xvda7 75M 1.6M 72M 3% /tmp
> >>> > 172.16.10.35:/nfs/KHI02 12T 7.0T 5.1T 59%
> >>> > /mnt/SecStorage/8ea71ccb-e493-3c7e-8bb0-97871f5c2092
> >>> > tmpfs 49M 0 49M 0% /run/user/0
> >>> > root@s-25437-VM:~#
> >>> > root@s-25437-VM:~#
> >>> > root@s-25437-VM:~# cd
> >>> /mnt/SecStorage/8ea71ccb-e493-3c7e-8bb0-97871f5c2092
> >>> > root@s-25437-VM
> :/mnt/SecStorage/8ea71ccb-e493-3c7e-8bb0-97871f5c2092#
> >>> > root@s-25437-VM
> :/mnt/SecStorage/8ea71ccb-e493-3c7e-8bb0-97871f5c2092#
> >>> ls
> >>> > -lah
> >>> > total 12K
> >>> > drwxrwxrwx 5 root root 70 Dec 21 2018 .
> >>> > drwxrwxrwx 3 root root 4.0K Sep 10 23:12 ..
> >>> > drwxrwxrwx 52 root root 4.0K Aug 13 19:37 snapshots
> >>> > drwxrwxrwx 3 root root 26 Jun 7 2013 template
> >>> > drwxrwxrwx 98 root root 4.0K Sep 1 12:52 volumes
> >>> >
> >>> > -Ammad Ali
> >>> >
> >>> >> On Fri, Sep 11, 2020 at 6:09 PM Andrija Panic <
> >>> andrija.panic@gmail.com>
> >>> >> wrote:
> >>> >>
> >>> >> Can you share permissions of your secondary storage (mount it then
> ls
> >>> -lah
> >>> >> the mount point)
> >>> >>
> >>> >
> >>> > pearl.dsilva@shapeblue.com
> >>> > www.shapeblue.com
> >>> > 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
> >>> > @shapeblue
> >>> >
> >>> >
> >>> >
> >>> >>> On Fri, 11 Sep 2020 at 01:39, Ammad Syed <sy...@gmail.com>
> >>> wrote:
> >>> >>>
> >>> >>> Hi Andrija,
> >>> >>>
> >>> >>> I have performed an upgrade on my production system from 4.11.3 to
> >>> >> 4.13.1.
> >>> >>> Even I cleared the tags but the issue is still there.
> >>> >>>
> >>> >>> I have four zones and only in one zone and on specific host, the
> >>> >>> systemVM's agent goes up but on all other zones, key injection to
> >>> >> systemVM
> >>> >>> is still failing on all zones and PODs. I have checked, the updated
> >>> ISO
> >>> >> is
> >>> >>> there on all hosts. The md5sum of systemvm.iso is same on xen hosts
> >>> and
> >>> >> acs
> >>> >>> host.
> >>> >>>
> >>> >>> Look like a weird problem. How can I troubleshoot this further ?
> Any
> >>> >> advise
> >>> >>> would be appreciated.
> >>> >>>
> >>> >>> -Ammad
> >>> >>>
> >>> >>
> >>> >>
> >>> >> --
> >>> >>
> >>> >> Andrija Panić
> >>> >>
> >>> >
> >>> >
> >>> > --
> >>> > Regards,
> >>> >
> >>> >
> >>> > Syed Ammad Ali
> >>>
> >>
> >>
> >> --
> >> Regards,
> >>
> >>
> >> Syed Ammad Ali
> >>
> >
> >
> > --
> > Regards,
> >
> >
> > Syed Ammad Ali
> >
>
>
> --
> Regards,
>
>
> Syed Ammad Ali
>
--
Andrija Panić
Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error
Posted by Ammad Syed <sy...@gmail.com>.
You guys can check job-1248274 logs or SSVM s-25457-VM that has
successfully connected agent state.
Ammad Ali
On Mon, Sep 14, 2020 at 2:33 PM Ammad Syed <sy...@gmail.com> wrote:
> Here are the upgrade management server logs.
>
>
> https://drive.google.com/file/d/17cxh7f-24ibnXCKvUUPqt3p1YTzlMQN8/view?usp=sharing
>
> Ammad Ali
>
> On Mon, Sep 14, 2020 at 2:17 PM Ammad Syed <sy...@gmail.com> wrote:
>
>> In addition to previous email there is only one host in one zone where
>> systemVM agent goes up and on all other hosts on that zone agent failed.
>>
>> If you guys need, I can provide management server logs as well.
>>
>> Also is there a way to enable debugging in ACS logs to specifically find
>> out where the problem is?
>>
>> Ammad Ali
>>
>> On Mon, Sep 14, 2020 at 10:39 AM Ammad Syed <sy...@gmail.com>
>> wrote:
>>
>>> Hi Perl,
>>>
>>> I have taken those steps and verified that systemvm.iso is copied to all
>>> hosts in all zones.
>>>
>>> I have recreated the systemvm and ssh to that host and checked the
>>> md5sum of iso there and on acs. Both were same.
>>>
>>> However the md5sum on which systemvm was working has also same md5sum of
>>> systemvm iso. The iso is getting copied successfully. The problem looks
>>> somewhere else.
>>>
>>> I have checked in xenserver logs as well but didn’t find any logs that
>>> something has failed.
>>>
>>> Ammad
>>> Sent from my iPhone
>>>
>>> > On 14-Sep-2020, at 9:15 AM, Pearl d'Silva <pe...@shapeblue.com>
>>> wrote:
>>> >
>>> > Hi Ammad,
>>> >
>>> > Is the understanding right that the steps as mentioned by you in the
>>> previous mail has in-fact worked on one zone? If that's the case, could you
>>> please ensure that all the hosts in all the other zones have the new
>>> systemVM iso copied into them by checking the timestamps as well and
>>> comparing the checksums against the iso on the management server, so that
>>> when the system VM's are recreated, they pick up the new iso.
>>> >
>>> > Thanks,
>>> > Pearl
>>> >
>>> > ________________________________
>>> > From: Ammad Syed <sy...@gmail.com>
>>> > Sent: Sunday, September 13, 2020 12:49 PM
>>> > To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
>>> > Subject: Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error
>>> >
>>> > Hi Andrija,
>>> >
>>> > Here is the permission of mount point in SSVM.
>>> >
>>> > root@s-25437-VM:~# df -h
>>> > Filesystem Size Used Avail Use% Mounted on
>>> > udev 233M 0 233M 0% /dev
>>> > tmpfs 98M 19M 80M 19% /run
>>> > /dev/xvda5 1.1G 773M 282M 74% /
>>> > tmpfs 244M 0 244M 0% /dev/shm
>>> > tmpfs 5.0M 0 5.0M 0% /run/lock
>>> > tmpfs 244M 0 244M 0% /sys/fs/cgroup
>>> > /dev/xvda1 92M 35M 57M 39% /boot
>>> > /dev/xvda6 435M 21M 410M 5% /var
>>> > /dev/xvda7 75M 1.6M 72M 3% /tmp
>>> > 172.16.10.35:/nfs/KHI02 12T 7.0T 5.1T 59%
>>> > /mnt/SecStorage/8ea71ccb-e493-3c7e-8bb0-97871f5c2092
>>> > tmpfs 49M 0 49M 0% /run/user/0
>>> > root@s-25437-VM:~#
>>> > root@s-25437-VM:~#
>>> > root@s-25437-VM:~# cd
>>> /mnt/SecStorage/8ea71ccb-e493-3c7e-8bb0-97871f5c2092
>>> > root@s-25437-VM:/mnt/SecStorage/8ea71ccb-e493-3c7e-8bb0-97871f5c2092#
>>> > root@s-25437-VM:/mnt/SecStorage/8ea71ccb-e493-3c7e-8bb0-97871f5c2092#
>>> ls
>>> > -lah
>>> > total 12K
>>> > drwxrwxrwx 5 root root 70 Dec 21 2018 .
>>> > drwxrwxrwx 3 root root 4.0K Sep 10 23:12 ..
>>> > drwxrwxrwx 52 root root 4.0K Aug 13 19:37 snapshots
>>> > drwxrwxrwx 3 root root 26 Jun 7 2013 template
>>> > drwxrwxrwx 98 root root 4.0K Sep 1 12:52 volumes
>>> >
>>> > -Ammad Ali
>>> >
>>> >> On Fri, Sep 11, 2020 at 6:09 PM Andrija Panic <
>>> andrija.panic@gmail.com>
>>> >> wrote:
>>> >>
>>> >> Can you share permissions of your secondary storage (mount it then ls
>>> -lah
>>> >> the mount point)
>>> >>
>>> >
>>> > pearl.dsilva@shapeblue.com
>>> > www.shapeblue.com
>>> > 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
>>> > @shapeblue
>>> >
>>> >
>>> >
>>> >>> On Fri, 11 Sep 2020 at 01:39, Ammad Syed <sy...@gmail.com>
>>> wrote:
>>> >>>
>>> >>> Hi Andrija,
>>> >>>
>>> >>> I have performed an upgrade on my production system from 4.11.3 to
>>> >> 4.13.1.
>>> >>> Even I cleared the tags but the issue is still there.
>>> >>>
>>> >>> I have four zones and only in one zone and on specific host, the
>>> >>> systemVM's agent goes up but on all other zones, key injection to
>>> >> systemVM
>>> >>> is still failing on all zones and PODs. I have checked, the updated
>>> ISO
>>> >> is
>>> >>> there on all hosts. The md5sum of systemvm.iso is same on xen hosts
>>> and
>>> >> acs
>>> >>> host.
>>> >>>
>>> >>> Look like a weird problem. How can I troubleshoot this further ? Any
>>> >> advise
>>> >>> would be appreciated.
>>> >>>
>>> >>> -Ammad
>>> >>>
>>> >>
>>> >>
>>> >> --
>>> >>
>>> >> Andrija Panić
>>> >>
>>> >
>>> >
>>> > --
>>> > Regards,
>>> >
>>> >
>>> > Syed Ammad Ali
>>>
>>
>>
>> --
>> Regards,
>>
>>
>> Syed Ammad Ali
>>
>
>
> --
> Regards,
>
>
> Syed Ammad Ali
>
--
Regards,
Syed Ammad Ali
Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error
Posted by Ammad Syed <sy...@gmail.com>.
Here are the upgrade management server logs.
https://drive.google.com/file/d/17cxh7f-24ibnXCKvUUPqt3p1YTzlMQN8/view?usp=sharing
Ammad Ali
On Mon, Sep 14, 2020 at 2:17 PM Ammad Syed <sy...@gmail.com> wrote:
> In addition to previous email there is only one host in one zone where
> systemVM agent goes up and on all other hosts on that zone agent failed.
>
> If you guys need, I can provide management server logs as well.
>
> Also is there a way to enable debugging in ACS logs to specifically find
> out where the problem is?
>
> Ammad Ali
>
> On Mon, Sep 14, 2020 at 10:39 AM Ammad Syed <sy...@gmail.com> wrote:
>
>> Hi Perl,
>>
>> I have taken those steps and verified that systemvm.iso is copied to all
>> hosts in all zones.
>>
>> I have recreated the systemvm and ssh to that host and checked the md5sum
>> of iso there and on acs. Both were same.
>>
>> However the md5sum on which systemvm was working has also same md5sum of
>> systemvm iso. The iso is getting copied successfully. The problem looks
>> somewhere else.
>>
>> I have checked in xenserver logs as well but didn’t find any logs that
>> something has failed.
>>
>> Ammad
>> Sent from my iPhone
>>
>> > On 14-Sep-2020, at 9:15 AM, Pearl d'Silva <pe...@shapeblue.com>
>> wrote:
>> >
>> > Hi Ammad,
>> >
>> > Is the understanding right that the steps as mentioned by you in the
>> previous mail has in-fact worked on one zone? If that's the case, could you
>> please ensure that all the hosts in all the other zones have the new
>> systemVM iso copied into them by checking the timestamps as well and
>> comparing the checksums against the iso on the management server, so that
>> when the system VM's are recreated, they pick up the new iso.
>> >
>> > Thanks,
>> > Pearl
>> >
>> > ________________________________
>> > From: Ammad Syed <sy...@gmail.com>
>> > Sent: Sunday, September 13, 2020 12:49 PM
>> > To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
>> > Subject: Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error
>> >
>> > Hi Andrija,
>> >
>> > Here is the permission of mount point in SSVM.
>> >
>> > root@s-25437-VM:~# df -h
>> > Filesystem Size Used Avail Use% Mounted on
>> > udev 233M 0 233M 0% /dev
>> > tmpfs 98M 19M 80M 19% /run
>> > /dev/xvda5 1.1G 773M 282M 74% /
>> > tmpfs 244M 0 244M 0% /dev/shm
>> > tmpfs 5.0M 0 5.0M 0% /run/lock
>> > tmpfs 244M 0 244M 0% /sys/fs/cgroup
>> > /dev/xvda1 92M 35M 57M 39% /boot
>> > /dev/xvda6 435M 21M 410M 5% /var
>> > /dev/xvda7 75M 1.6M 72M 3% /tmp
>> > 172.16.10.35:/nfs/KHI02 12T 7.0T 5.1T 59%
>> > /mnt/SecStorage/8ea71ccb-e493-3c7e-8bb0-97871f5c2092
>> > tmpfs 49M 0 49M 0% /run/user/0
>> > root@s-25437-VM:~#
>> > root@s-25437-VM:~#
>> > root@s-25437-VM:~# cd
>> /mnt/SecStorage/8ea71ccb-e493-3c7e-8bb0-97871f5c2092
>> > root@s-25437-VM:/mnt/SecStorage/8ea71ccb-e493-3c7e-8bb0-97871f5c2092#
>> > root@s-25437-VM:/mnt/SecStorage/8ea71ccb-e493-3c7e-8bb0-97871f5c2092#
>> ls
>> > -lah
>> > total 12K
>> > drwxrwxrwx 5 root root 70 Dec 21 2018 .
>> > drwxrwxrwx 3 root root 4.0K Sep 10 23:12 ..
>> > drwxrwxrwx 52 root root 4.0K Aug 13 19:37 snapshots
>> > drwxrwxrwx 3 root root 26 Jun 7 2013 template
>> > drwxrwxrwx 98 root root 4.0K Sep 1 12:52 volumes
>> >
>> > -Ammad Ali
>> >
>> >> On Fri, Sep 11, 2020 at 6:09 PM Andrija Panic <andrija.panic@gmail.com
>> >
>> >> wrote:
>> >>
>> >> Can you share permissions of your secondary storage (mount it then ls
>> -lah
>> >> the mount point)
>> >>
>> >
>> > pearl.dsilva@shapeblue.com
>> > www.shapeblue.com
>> > 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
>> > @shapeblue
>> >
>> >
>> >
>> >>> On Fri, 11 Sep 2020 at 01:39, Ammad Syed <sy...@gmail.com>
>> wrote:
>> >>>
>> >>> Hi Andrija,
>> >>>
>> >>> I have performed an upgrade on my production system from 4.11.3 to
>> >> 4.13.1.
>> >>> Even I cleared the tags but the issue is still there.
>> >>>
>> >>> I have four zones and only in one zone and on specific host, the
>> >>> systemVM's agent goes up but on all other zones, key injection to
>> >> systemVM
>> >>> is still failing on all zones and PODs. I have checked, the updated
>> ISO
>> >> is
>> >>> there on all hosts. The md5sum of systemvm.iso is same on xen hosts
>> and
>> >> acs
>> >>> host.
>> >>>
>> >>> Look like a weird problem. How can I troubleshoot this further ? Any
>> >> advise
>> >>> would be appreciated.
>> >>>
>> >>> -Ammad
>> >>>
>> >>
>> >>
>> >> --
>> >>
>> >> Andrija Panić
>> >>
>> >
>> >
>> > --
>> > Regards,
>> >
>> >
>> > Syed Ammad Ali
>>
>
>
> --
> Regards,
>
>
> Syed Ammad Ali
>
--
Regards,
Syed Ammad Ali
Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error
Posted by Ammad Syed <sy...@gmail.com>.
In addition to previous email there is only one host in one zone where
systemVM agent goes up and on all other hosts on that zone agent failed.
If you guys need, I can provide management server logs as well.
Also is there a way to enable debugging in ACS logs to specifically find
out where the problem is?
Ammad Ali
On Mon, Sep 14, 2020 at 10:39 AM Ammad Syed <sy...@gmail.com> wrote:
> Hi Perl,
>
> I have taken those steps and verified that systemvm.iso is copied to all
> hosts in all zones.
>
> I have recreated the systemvm and ssh to that host and checked the md5sum
> of iso there and on acs. Both were same.
>
> However the md5sum on which systemvm was working has also same md5sum of
> systemvm iso. The iso is getting copied successfully. The problem looks
> somewhere else.
>
> I have checked in xenserver logs as well but didn’t find any logs that
> something has failed.
>
> Ammad
> Sent from my iPhone
>
> > On 14-Sep-2020, at 9:15 AM, Pearl d'Silva <pe...@shapeblue.com>
> wrote:
> >
> > Hi Ammad,
> >
> > Is the understanding right that the steps as mentioned by you in the
> previous mail has in-fact worked on one zone? If that's the case, could you
> please ensure that all the hosts in all the other zones have the new
> systemVM iso copied into them by checking the timestamps as well and
> comparing the checksums against the iso on the management server, so that
> when the system VM's are recreated, they pick up the new iso.
> >
> > Thanks,
> > Pearl
> >
> > ________________________________
> > From: Ammad Syed <sy...@gmail.com>
> > Sent: Sunday, September 13, 2020 12:49 PM
> > To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> > Subject: Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error
> >
> > Hi Andrija,
> >
> > Here is the permission of mount point in SSVM.
> >
> > root@s-25437-VM:~# df -h
> > Filesystem Size Used Avail Use% Mounted on
> > udev 233M 0 233M 0% /dev
> > tmpfs 98M 19M 80M 19% /run
> > /dev/xvda5 1.1G 773M 282M 74% /
> > tmpfs 244M 0 244M 0% /dev/shm
> > tmpfs 5.0M 0 5.0M 0% /run/lock
> > tmpfs 244M 0 244M 0% /sys/fs/cgroup
> > /dev/xvda1 92M 35M 57M 39% /boot
> > /dev/xvda6 435M 21M 410M 5% /var
> > /dev/xvda7 75M 1.6M 72M 3% /tmp
> > 172.16.10.35:/nfs/KHI02 12T 7.0T 5.1T 59%
> > /mnt/SecStorage/8ea71ccb-e493-3c7e-8bb0-97871f5c2092
> > tmpfs 49M 0 49M 0% /run/user/0
> > root@s-25437-VM:~#
> > root@s-25437-VM:~#
> > root@s-25437-VM:~# cd
> /mnt/SecStorage/8ea71ccb-e493-3c7e-8bb0-97871f5c2092
> > root@s-25437-VM:/mnt/SecStorage/8ea71ccb-e493-3c7e-8bb0-97871f5c2092#
> > root@s-25437-VM:/mnt/SecStorage/8ea71ccb-e493-3c7e-8bb0-97871f5c2092# ls
> > -lah
> > total 12K
> > drwxrwxrwx 5 root root 70 Dec 21 2018 .
> > drwxrwxrwx 3 root root 4.0K Sep 10 23:12 ..
> > drwxrwxrwx 52 root root 4.0K Aug 13 19:37 snapshots
> > drwxrwxrwx 3 root root 26 Jun 7 2013 template
> > drwxrwxrwx 98 root root 4.0K Sep 1 12:52 volumes
> >
> > -Ammad Ali
> >
> >> On Fri, Sep 11, 2020 at 6:09 PM Andrija Panic <an...@gmail.com>
> >> wrote:
> >>
> >> Can you share permissions of your secondary storage (mount it then ls
> -lah
> >> the mount point)
> >>
> >
> > pearl.dsilva@shapeblue.com
> > www.shapeblue.com
> > 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
> > @shapeblue
> >
> >
> >
> >>> On Fri, 11 Sep 2020 at 01:39, Ammad Syed <sy...@gmail.com>
> wrote:
> >>>
> >>> Hi Andrija,
> >>>
> >>> I have performed an upgrade on my production system from 4.11.3 to
> >> 4.13.1.
> >>> Even I cleared the tags but the issue is still there.
> >>>
> >>> I have four zones and only in one zone and on specific host, the
> >>> systemVM's agent goes up but on all other zones, key injection to
> >> systemVM
> >>> is still failing on all zones and PODs. I have checked, the updated ISO
> >> is
> >>> there on all hosts. The md5sum of systemvm.iso is same on xen hosts and
> >> acs
> >>> host.
> >>>
> >>> Look like a weird problem. How can I troubleshoot this further ? Any
> >> advise
> >>> would be appreciated.
> >>>
> >>> -Ammad
> >>>
> >>
> >>
> >> --
> >>
> >> Andrija Panić
> >>
> >
> >
> > --
> > Regards,
> >
> >
> > Syed Ammad Ali
>
--
Regards,
Syed Ammad Ali
Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error
Posted by Ammad Syed <sy...@gmail.com>.
Hi Perl,
I have taken those steps and verified that systemvm.iso is copied to all hosts in all zones.
I have recreated the systemvm and ssh to that host and checked the md5sum of iso there and on acs. Both were same.
However the md5sum on which systemvm was working has also same md5sum of systemvm iso. The iso is getting copied successfully. The problem looks somewhere else.
I have checked in xenserver logs as well but didn’t find any logs that something has failed.
Ammad
Sent from my iPhone
> On 14-Sep-2020, at 9:15 AM, Pearl d'Silva <pe...@shapeblue.com> wrote:
>
> Hi Ammad,
>
> Is the understanding right that the steps as mentioned by you in the previous mail has in-fact worked on one zone? If that's the case, could you please ensure that all the hosts in all the other zones have the new systemVM iso copied into them by checking the timestamps as well and comparing the checksums against the iso on the management server, so that when the system VM's are recreated, they pick up the new iso.
>
> Thanks,
> Pearl
>
> ________________________________
> From: Ammad Syed <sy...@gmail.com>
> Sent: Sunday, September 13, 2020 12:49 PM
> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> Subject: Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error
>
> Hi Andrija,
>
> Here is the permission of mount point in SSVM.
>
> root@s-25437-VM:~# df -h
> Filesystem Size Used Avail Use% Mounted on
> udev 233M 0 233M 0% /dev
> tmpfs 98M 19M 80M 19% /run
> /dev/xvda5 1.1G 773M 282M 74% /
> tmpfs 244M 0 244M 0% /dev/shm
> tmpfs 5.0M 0 5.0M 0% /run/lock
> tmpfs 244M 0 244M 0% /sys/fs/cgroup
> /dev/xvda1 92M 35M 57M 39% /boot
> /dev/xvda6 435M 21M 410M 5% /var
> /dev/xvda7 75M 1.6M 72M 3% /tmp
> 172.16.10.35:/nfs/KHI02 12T 7.0T 5.1T 59%
> /mnt/SecStorage/8ea71ccb-e493-3c7e-8bb0-97871f5c2092
> tmpfs 49M 0 49M 0% /run/user/0
> root@s-25437-VM:~#
> root@s-25437-VM:~#
> root@s-25437-VM:~# cd /mnt/SecStorage/8ea71ccb-e493-3c7e-8bb0-97871f5c2092
> root@s-25437-VM:/mnt/SecStorage/8ea71ccb-e493-3c7e-8bb0-97871f5c2092#
> root@s-25437-VM:/mnt/SecStorage/8ea71ccb-e493-3c7e-8bb0-97871f5c2092# ls
> -lah
> total 12K
> drwxrwxrwx 5 root root 70 Dec 21 2018 .
> drwxrwxrwx 3 root root 4.0K Sep 10 23:12 ..
> drwxrwxrwx 52 root root 4.0K Aug 13 19:37 snapshots
> drwxrwxrwx 3 root root 26 Jun 7 2013 template
> drwxrwxrwx 98 root root 4.0K Sep 1 12:52 volumes
>
> -Ammad Ali
>
>> On Fri, Sep 11, 2020 at 6:09 PM Andrija Panic <an...@gmail.com>
>> wrote:
>>
>> Can you share permissions of your secondary storage (mount it then ls -lah
>> the mount point)
>>
>
> pearl.dsilva@shapeblue.com
> www.shapeblue.com
> 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
> @shapeblue
>
>
>
>>> On Fri, 11 Sep 2020 at 01:39, Ammad Syed <sy...@gmail.com> wrote:
>>>
>>> Hi Andrija,
>>>
>>> I have performed an upgrade on my production system from 4.11.3 to
>> 4.13.1.
>>> Even I cleared the tags but the issue is still there.
>>>
>>> I have four zones and only in one zone and on specific host, the
>>> systemVM's agent goes up but on all other zones, key injection to
>> systemVM
>>> is still failing on all zones and PODs. I have checked, the updated ISO
>> is
>>> there on all hosts. The md5sum of systemvm.iso is same on xen hosts and
>> acs
>>> host.
>>>
>>> Look like a weird problem. How can I troubleshoot this further ? Any
>> advise
>>> would be appreciated.
>>>
>>> -Ammad
>>>
>>
>>
>> --
>>
>> Andrija Panić
>>
>
>
> --
> Regards,
>
>
> Syed Ammad Ali
Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error
Posted by Pearl d'Silva <pe...@shapeblue.com>.
Hi Ammad,
Is the understanding right that the steps as mentioned by you in the previous mail has in-fact worked on one zone? If that's the case, could you please ensure that all the hosts in all the other zones have the new systemVM iso copied into them by checking the timestamps as well and comparing the checksums against the iso on the management server, so that when the system VM's are recreated, they pick up the new iso.
Thanks,
Pearl
________________________________
From: Ammad Syed <sy...@gmail.com>
Sent: Sunday, September 13, 2020 12:49 PM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error
Hi Andrija,
Here is the permission of mount point in SSVM.
root@s-25437-VM:~# df -h
Filesystem Size Used Avail Use% Mounted on
udev 233M 0 233M 0% /dev
tmpfs 98M 19M 80M 19% /run
/dev/xvda5 1.1G 773M 282M 74% /
tmpfs 244M 0 244M 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 244M 0 244M 0% /sys/fs/cgroup
/dev/xvda1 92M 35M 57M 39% /boot
/dev/xvda6 435M 21M 410M 5% /var
/dev/xvda7 75M 1.6M 72M 3% /tmp
172.16.10.35:/nfs/KHI02 12T 7.0T 5.1T 59%
/mnt/SecStorage/8ea71ccb-e493-3c7e-8bb0-97871f5c2092
tmpfs 49M 0 49M 0% /run/user/0
root@s-25437-VM:~#
root@s-25437-VM:~#
root@s-25437-VM:~# cd /mnt/SecStorage/8ea71ccb-e493-3c7e-8bb0-97871f5c2092
root@s-25437-VM:/mnt/SecStorage/8ea71ccb-e493-3c7e-8bb0-97871f5c2092#
root@s-25437-VM:/mnt/SecStorage/8ea71ccb-e493-3c7e-8bb0-97871f5c2092# ls
-lah
total 12K
drwxrwxrwx 5 root root 70 Dec 21 2018 .
drwxrwxrwx 3 root root 4.0K Sep 10 23:12 ..
drwxrwxrwx 52 root root 4.0K Aug 13 19:37 snapshots
drwxrwxrwx 3 root root 26 Jun 7 2013 template
drwxrwxrwx 98 root root 4.0K Sep 1 12:52 volumes
-Ammad Ali
On Fri, Sep 11, 2020 at 6:09 PM Andrija Panic <an...@gmail.com>
wrote:
> Can you share permissions of your secondary storage (mount it then ls -lah
> the mount point)
>
pearl.dsilva@shapeblue.com
www.shapeblue.com
3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
@shapeblue
> On Fri, 11 Sep 2020 at 01:39, Ammad Syed <sy...@gmail.com> wrote:
>
> > Hi Andrija,
> >
> > I have performed an upgrade on my production system from 4.11.3 to
> 4.13.1.
> > Even I cleared the tags but the issue is still there.
> >
> > I have four zones and only in one zone and on specific host, the
> > systemVM's agent goes up but on all other zones, key injection to
> systemVM
> > is still failing on all zones and PODs. I have checked, the updated ISO
> is
> > there on all hosts. The md5sum of systemvm.iso is same on xen hosts and
> acs
> > host.
> >
> > Look like a weird problem. How can I troubleshoot this further ? Any
> advise
> > would be appreciated.
> >
> > -Ammad
> >
>
>
> --
>
> Andrija Panić
>
--
Regards,
Syed Ammad Ali
Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error
Posted by Ammad Syed <sy...@gmail.com>.
Hi Andrija,
Here is the permission of mount point in SSVM.
root@s-25437-VM:~# df -h
Filesystem Size Used Avail Use% Mounted on
udev 233M 0 233M 0% /dev
tmpfs 98M 19M 80M 19% /run
/dev/xvda5 1.1G 773M 282M 74% /
tmpfs 244M 0 244M 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 244M 0 244M 0% /sys/fs/cgroup
/dev/xvda1 92M 35M 57M 39% /boot
/dev/xvda6 435M 21M 410M 5% /var
/dev/xvda7 75M 1.6M 72M 3% /tmp
172.16.10.35:/nfs/KHI02 12T 7.0T 5.1T 59%
/mnt/SecStorage/8ea71ccb-e493-3c7e-8bb0-97871f5c2092
tmpfs 49M 0 49M 0% /run/user/0
root@s-25437-VM:~#
root@s-25437-VM:~#
root@s-25437-VM:~# cd /mnt/SecStorage/8ea71ccb-e493-3c7e-8bb0-97871f5c2092
root@s-25437-VM:/mnt/SecStorage/8ea71ccb-e493-3c7e-8bb0-97871f5c2092#
root@s-25437-VM:/mnt/SecStorage/8ea71ccb-e493-3c7e-8bb0-97871f5c2092# ls
-lah
total 12K
drwxrwxrwx 5 root root 70 Dec 21 2018 .
drwxrwxrwx 3 root root 4.0K Sep 10 23:12 ..
drwxrwxrwx 52 root root 4.0K Aug 13 19:37 snapshots
drwxrwxrwx 3 root root 26 Jun 7 2013 template
drwxrwxrwx 98 root root 4.0K Sep 1 12:52 volumes
-Ammad Ali
On Fri, Sep 11, 2020 at 6:09 PM Andrija Panic <an...@gmail.com>
wrote:
> Can you share permissions of your secondary storage (mount it then ls -lah
> the mount point)
>
> On Fri, 11 Sep 2020 at 01:39, Ammad Syed <sy...@gmail.com> wrote:
>
> > Hi Andrija,
> >
> > I have performed an upgrade on my production system from 4.11.3 to
> 4.13.1.
> > Even I cleared the tags but the issue is still there.
> >
> > I have four zones and only in one zone and on specific host, the
> > systemVM's agent goes up but on all other zones, key injection to
> systemVM
> > is still failing on all zones and PODs. I have checked, the updated ISO
> is
> > there on all hosts. The md5sum of systemvm.iso is same on xen hosts and
> acs
> > host.
> >
> > Look like a weird problem. How can I troubleshoot this further ? Any
> advise
> > would be appreciated.
> >
> > -Ammad
> >
>
>
> --
>
> Andrija Panić
>
--
Regards,
Syed Ammad Ali
Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error
Posted by Andrija Panic <an...@gmail.com>.
Can you share permissions of your secondary storage (mount it then ls -lah
the mount point)
On Fri, 11 Sep 2020 at 01:39, Ammad Syed <sy...@gmail.com> wrote:
> Hi Andrija,
>
> I have performed an upgrade on my production system from 4.11.3 to 4.13.1.
> Even I cleared the tags but the issue is still there.
>
> I have four zones and only in one zone and on specific host, the
> systemVM's agent goes up but on all other zones, key injection to systemVM
> is still failing on all zones and PODs. I have checked, the updated ISO is
> there on all hosts. The md5sum of systemvm.iso is same on xen hosts and acs
> host.
>
> Look like a weird problem. How can I troubleshoot this further ? Any advise
> would be appreciated.
>
> -Ammad
>
--
Andrija Panić
Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error
Posted by Ammad Syed <sy...@gmail.com>.
Hi Andrija,
I have performed an upgrade on my production system from 4.11.3 to 4.13.1.
Even I cleared the tags but the issue is still there.
I have four zones and only in one zone and on specific host, the
systemVM's agent goes up but on all other zones, key injection to systemVM
is still failing on all zones and PODs. I have checked, the updated ISO is
there on all hosts. The md5sum of systemvm.iso is same on xen hosts and acs
host.
Look like a weird problem. How can I troubleshoot this further ? Any advise
would be appreciated.
-Ammad
Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error
Posted by Andrija Panic <an...@gmail.com>.
Glad to see you solved the issue!
On Mon, 7 Sep 2020 at 10:42, Ammad Syed <sy...@gmail.com> wrote:
> Hi,
>
> Thank you guys, this has fixed my problem. Here are the steps that I have
> performed.
>
> - Did a fresh installation of 4.13.1 and copied its systemvm.iso on my ACS
> in /usr/share/cloudstack-common/vms/.
> - Shutdown management server.
> - Deleted systemVMs from xencenter.
> - Executed key injection script
> "/usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh
> /root/.ssh/id_rsa.pub /root/.ssh/id_rsa
> /usr/share/cloudstack-common/vms/systemvm.iso"
> - Cleared the tags xe host-param-clear param-name=tags uuid=<uuid of the XS
> host>
> - Started management server.
> - Recreated systemVMs.
>
> Thank you guys for your support.
>
> - Ammad
>
> On Thu, Sep 3, 2020 at 11:40 AM Andrija Panic <an...@gmail.com>
> wrote:
>
> > Ammad,
> >
> > please check what Pearl has suggested, but I think the correct syntax
> > (different id_rsa,pub, specifically the one used by ACS) should be as
> > following (you can always grep your older logs for "injectkey.sh" :
> >
> > /bin/bash /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh
> > /var/cloudstack/management/.ssh/id_rsa.pub
> > /var/cloudstack/management/.ssh/id_rsa
> > /usr/share/cloudstack-common/vms/systemvm.iso
> > You should get e meaningful output if the key was injected or not
> (perhaps
> > the key inside doesn't differer, so the script will not inject it again)
> > Proceed with what Pearl has suggested as the next steps.
> >
> > Best,
> >
> > On Thu, 3 Sep 2020 at 06:16, Pearl d'Silva <pe...@shapeblue.com>
> > wrote:
> >
> > > Hi Ammad,
> > >
> > > Could you try re-injecting the keys into the systemVM, as this MAY
> happen
> > > due to stale id_rsa keys. Run the injectkeys.sh to inject the existing
> > > public ssh key into the authorized_keys file inside the systemvm.iso.
> > > Execute:
> > >
> > >
> > > /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh
> > > /root/.ssh/id_rsa.pub /root/.ssh/id_rsa
> > > /usr/share/cloudstack-common/vms/systemvm.iso'
> > >
> > > * Then replace the systemvm.iso on the hypervisor hosts. For each
> > host:
> > >
> > > 1. Login and type command - xe host-param-clear param-name=tags
> > > uuid=<uuid of the XS host>
> > > host uuid maybe obtained using: xe host-list
> > >
> > > 2. 2. Restart MS and the iso should get propagated for all.
> Check
> > > timestamp.
> > > 3. 3. Restart systemvms and they should get the iso inserted.
> > > 4.
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/SystemVm.iso#SystemVm.iso-Xen
> > >
> > > Thanks,
> > > Pearl
> > >
> > > ________________________________
> > > From: Ammad Syed <sy...@gmail.com>
> > > Sent: Wednesday, September 2, 2020 9:12 PM
> > > To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> > > Subject: Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error
> > >
> > > Guys, please advise how to troubleshoot this further ? How can i enable
> > > trace level debugging ? any help in this will be highly appreciated?
> > >
> > > Ammad
> > >
> > >
> > > pearl.dsilva@shapeblue.com
> > > www.shapeblue.com
> > > 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
> > > @shapeblue
> > >
> > >
> > >
> > > > On 27-Aug-2020, at 1:41 PM, Vivek Kumar <vivek.kumar@indiqus.com
> > .invalid>
> > > wrote:
> > > >
> > > > Pardon, the whole mail chain was not visible so replied accordingly.
> > It
> > > seems like you are doing SSH using root shell so it doesn’t matter.
> > > >
> > > > Vivek Kumar
> > > > Manager - Cloud & DevOps
> > > > IndiQus Technologies
> > > > 24*7 O +91 11 4055 1411 | M +91 7503460090
> > > > www.indiqus.com<http://www.indiqus.com> <http://indiqus.com/>
> > > >
> > > > This message is intended only for the use of the individual or entity
> > to
> > > which it is addressed and may contain information that is confidential
> > > and/or privileged. If you are not the intended recipient please delete
> > the
> > > original message and any copy of it from your computer system. You are
> > > hereby notified that any dissemination, distribution or copying of this
> > > communication is strictly prohibited unless proper authorization has
> been
> > > obtained for such action. If you have received this communication in
> > error,
> > > please notify the sender immediately. Although IndiQus attempts to
> sweep
> > > e-mail and attachments for viruses, it does not guarantee that both are
> > > virus-free and accepts no liability for any damage sustained as a
> result
> > of
> > > viruses.
> > > >
> > > >> On 27-Aug-2020, at 2:03 PM, Vivek Kumar <vi...@indiqus.com>
> > > wrote:
> > > >>
> > > >> Hello Ammad,
> > > >>
> > > >> You haven’t defined the user while logging to the system VM through
> > the
> > > Link local IP that’s why you have got the error while logging to the
> > system
> > > VM.
> > > >>
> > > >>>> ssh -i /root/.ssh/id_rsa.cloud 169.254.0.121 -p 3922
> > > >>
> > > >> Try instead below
> > > >>
> > > >>>> ssh -i /root/.ssh/id_rsa.cloud -p 3922 root@169.254.0.121
> <mailto:
> > > root@169.254.0.121>
> > > >>
> > > >> Vivek Kumar
> > > >> Manager - Cloud & DevOps
> > > >> IndiQus Technologies
> > > >> 24*7 O +91 11 4055 1411 | M +91 7503460090
> > > >> www.indiqus.com<http://www.indiqus.com> <http://indiqus.com/>
> > > >>
> > > >> This message is intended only for the use of the individual or
> entity
> > > to which it is addressed and may contain information that is
> confidential
> > > and/or privileged. If you are not the intended recipient please delete
> > the
> > > original message and any copy of it from your computer system. You are
> > > hereby notified that any dissemination, distribution or copying of this
> > > communication is strictly prohibited unless proper authorization has
> been
> > > obtained for such action. If you have received this communication in
> > error,
> > > please notify the sender immediately. Although IndiQus attempts to
> sweep
> > > e-mail and attachments for viruses, it does not guarantee that both are
> > > virus-free and accepts no liability for any damage sustained as a
> result
> > of
> > > viruses.
> > > >>
> > > >>>> On 27-Aug-2020, at 1:59 PM, Ammad Syed <syedammad83@gmail.com
> > > <ma...@gmail.com>> wrote:
> > > >>>
> > > >>> Guys, any help to troubleshoot this further would be highly
> > > appreciated.
> > > >>>
> > > >>>
> > > >>> Ammad Ali
> > > >>>> On 18-Aug-2020, at 1:05 PM, Ammad Syed <syedammad83@gmail.com
> > > <ma...@gmail.com>> wrote:
> > > >>>>
> > > >>>>
> > > >>>> Hi,
> > > >>>>
> > > >>>> The systemVM is accessible with 169.x.x.x IP from xenserver host.
> > But
> > > login with private key is denied. I am testing this on my test
> > environment.
> > > >>>>
> > > >>>> [root@xenserver ~]# ping 169.254.0.121
> > > >>>> PING 169.254.0.121 (169.254.0.121) 56(84) bytes of data.
> > > >>>> 64 bytes from 169.254.0.121: icmp_seq=1 ttl=64 time=0.328 ms
> > > >>>> 64 bytes from 169.254.0.121: icmp_seq=2 ttl=64 time=0.124 ms
> > > >>>> 64 bytes from 169.254.0.121: icmp_seq=3 ttl=64 time=0.111 ms
> > > >>>> 64 bytes from 169.254.0.121: icmp_seq=4 ttl=64 time=0.126 ms
> > > >>>> ^C
> > > >>>> --- 169.254.0.121 ping statistics ---
> > > >>>> 4 packets transmitted, 4 received, 0% packet loss, time 2997ms
> > > >>>> rtt min/avg/max/mdev = 0.111/0.172/0.328/0.090 ms
> > > >>>> [root@xenserver ~]#
> > > >>>> [root@xenserver ~]#
> > > >>>> [root@xenserver ~]# ssh -i /root/.ssh/id_rsa.cloud 169.254.0.121
> -p
> > > 3922
> > > >>>> Permission denied (publickey).
> > > >>>>
> > > >>>> I have checked by logging into the systemVM, the public key is not
> > > found in /root/.ssh/authorized_keys.
> > > >>>>
> > > >>>> I have digged further, the systemvm.iso is same on the management
> > > node and xenserver host. I have checked by mounting the ISO as well,
> the
> > > public key is there in authorized_keys in ISO.
> > > >>>>
> > > >>>> Restarting / recreating systemVM isn't fixed the issue.
> > > >>>>
> > > >>>> The systemVM is already running on HVM mode on xenserver host as
> the
> > > template OS type is Debian 8 64bit is selected.
> > > >>>>
> > > >>>> On my PRD upgrade, I have tested on one zone only. Currently I am
> > > testing on my test ACS environment which have only one zone.
> > > >>>>
> > > >>>> Ammad Ali
> > > >>>>
> > > >>>>> On Mon, Aug 17, 2020 at 11:10 PM Andrija Panic <
> > > andrija.panic@gmail.com <ma...@gmail.com>> wrote:
> > > >>>>> It was my tool (GUI based tool, that still doesn't return any
> line
> > > (???)
> > > >>>>> but I see it.
> > > >>>>>
> > > >>>>> Problem is with 169.254.1.178 not being reachable over SSH
> > > (2020-07-25
> > > >>>>> 01:45:52,097 DEBUG [c.c.h.x.r.CitrixResourceBase]
> > > >>>>> (DirectAgent-57:ctx-ded7f7f9) (logid:a75971c2) Executing command
> in
> > > VR:
> > > >>>>> /opt/cloud/bin/router_proxy.sh keystore-setup 169.254.1.178
> > > >>>>> /usr/local/cloud/systemvm/conf/agent.properties
> > > >>>>> /usr/local/cloud/systemvm/conf/cloud.jks Xe8zzqZmUp9wmABH 365
> > > >>>>> /usr/local/cloud/systemvm/conf/cloud.csr)
> > > >>>>>
> > > >>>>> Can you try to ssh from that particular XenServer (where the SSVM
> > is
> > > >>>>> running) to the IP 169.xxxxx on port 3922 (ssh port) using the
> > > private key
> > > >>>>> - if you log in to the VM via it's console (root/password) - do
> you
> > > see
> > > >>>>> with ifconfig the interface being started with 169..xxxx IP
> > address?
> > > I
> > > >>>>> assume that destroying the SSVM and starting a new one doesn't
> fix
> > > the
> > > >>>>> issue?
> > > >>>>>
> > > >>>>> You can also try changing the OS type of the systemVM template to
> > > Other
> > > >>>>> Linux 64bit, so it runs in HVM mode (instead of PV) - you'll need
> > to
> > > see ID
> > > >>>>> for that OS type from the guest_os table, and set the guest_os_id
> > > field in
> > > >>>>> the vm_template table for your systemVM template.
> > > >>>>>
> > > >>>>> Does the issue occur in other zones as well? (assuming same
> > > hypervisor type)
> > > >>>>>
> > > >>>>>
> > > >>>>> On Mon, 17 Aug 2020 at 15:31, Ammad Syed <syedammad83@gmail.com
> > > <ma...@gmail.com>> wrote:
> > > >>>>>
> > > >>>>>> I have rechecked the logs are there. Please review below lines.
> > > >>>>>>
> > > >>>>>>
> > > >>>>>>
> > > >>>>>>
> > >
> >
> C:\Users\ammad\Downloads\management-server.log-4.13\management-server.log-4.13.1-errors
> > > >>>>>> (27 hits)
> > > >>>>>> Line 409969: 2020-07-25 01:45:52,320 ERROR
> > > >>>>>> [c.c.v.VirtualMachineManagerImpl]
> > (Work-Job-Executor-58:ctx-9dc75334
> > > >>>>>> job-1207969/job-1208104 ctx-09e752ec) (logid:a75971c2) Failed to
> > > setup
> > > >>>>>> keystore and generate CSR for system vm: s-25023-VM
> > > >>>>>> Line 437413: 2020-07-25 01:51:33,845 ERROR
> > > >>>>>> [c.c.v.VirtualMachineManagerImpl]
> > (Work-Job-Executor-70:ctx-457ce4ee
> > > >>>>>> job-1207969/job-1208129 ctx-195da69e) (logid:a75971c2) Failed to
> > > setup
> > > >>>>>> keystore and generate CSR for system vm: s-25024-VM
> > > >>>>>> Line 549538: 2020-07-25 02:06:44,833 ERROR
> > > >>>>>> [c.c.v.VirtualMachineManagerImpl]
> > (Work-Job-Executor-3:ctx-4634f7e0
> > > >>>>>> job-1208154/job-1208161 ctx-05d065dc) (logid:91fc037a) Failed to
> > > setup
> > > >>>>>> keystore and generate CSR for system vm: v-25027-VM
> > > >>>>>> Line 663111: 2020-07-25 02:23:43,783 ERROR
> > > >>>>>> [c.c.v.VirtualMachineManagerImpl]
> > (Work-Job-Executor-22:ctx-fe0b857b
> > > >>>>>> job-1208154/job-1208222 ctx-ef088cd7) (logid:91fc037a) Failed to
> > > setup
> > > >>>>>> keystore and generate CSR for system vm: v-20771-VM
> > > >>>>>> Line 667060: 2020-07-25 02:24:30,221 ERROR
> > > >>>>>> [c.c.v.VirtualMachineManagerImpl]
> > (Work-Job-Executor-23:ctx-662cfe5d
> > > >>>>>> job-1208154/job-1208229 ctx-e59878e9) (logid:91fc037a) Failed to
> > > setup
> > > >>>>>> keystore and generate CSR for system vm: v-24763-VM
> > > >>>>>> Line 674269: 2020-07-25 02:25:47,345 ERROR
> > > >>>>>> [c.c.v.VirtualMachineManagerImpl]
> > (Work-Job-Executor-24:ctx-b6529613
> > > >>>>>> job-1208154/job-1208233 ctx-7da81cab) (logid:91fc037a) Failed to
> > > setup
> > > >>>>>> keystore and generate CSR for system vm: v-20773-VM
> > > >>>>>> Line 678169: 2020-07-25 02:26:20,355 ERROR
> > > >>>>>> [c.c.v.VirtualMachineManagerImpl]
> > (Work-Job-Executor-31:ctx-7a062a66
> > > >>>>>> job-1208154/job-1208244 ctx-4f48d08a) (logid:91fc037a) Failed to
> > > setup
> > > >>>>>> keystore and generate CSR for system vm: v-25027-VM
> > > >>>>>> Line 697113: 2020-07-25 02:30:09,669 ERROR
> > > >>>>>> [c.c.v.VirtualMachineManagerImpl]
> > (Work-Job-Executor-40:ctx-3be9fddc
> > > >>>>>> job-1208154/job-1208257 ctx-ed08fc43) (logid:91fc037a) Failed to
> > > setup
> > > >>>>>> keystore and generate CSR for system vm: v-20771-VM
> > > >>>>>> Line 701161: 2020-07-25 02:30:48,127 ERROR
> > > >>>>>> [c.c.v.VirtualMachineManagerImpl]
> > (Work-Job-Executor-41:ctx-4e3c666d
> > > >>>>>> job-1208155/job-1208258 ctx-df740f75) (logid:9fa7dece) Failed to
> > > setup
> > > >>>>>> keystore and generate CSR for system vm: s-24142-VM
> > > >>>>>> Line 701838: 2020-07-25 02:30:56,759 ERROR
> > > >>>>>> [c.c.v.VirtualMachineManagerImpl]
> > (Work-Job-Executor-42:ctx-04ac8e71
> > > >>>>>> job-1208154/job-1208259 ctx-36c67b5e) (logid:91fc037a) Failed to
> > > setup
> > > >>>>>> keystore and generate CSR for system vm: v-24763-VM
> > > >>>>>> Line 703865: 2020-07-25 02:31:25,123 ERROR
> > > >>>>>> [c.c.v.VirtualMachineManagerImpl]
> > (Work-Job-Executor-46:ctx-24bdbe8f
> > > >>>>>> job-1208154/job-1208266 ctx-fa500d3c) (logid:91fc037a) Failed to
> > > setup
> > > >>>>>> keystore and generate CSR for system vm: v-20773-VM
> > > >>>>>> Line 712078: 2020-07-25 02:32:28,666 ERROR
> > > >>>>>> [c.c.v.VirtualMachineManagerImpl]
> > (Work-Job-Executor-48:ctx-4d5db5c2
> > > >>>>>> job-1208155/job-1208269 ctx-b0f67c2e) (logid:9fa7dece) Failed to
> > > setup
> > > >>>>>> keystore and generate CSR for system vm: s-25033-VM
> > > >>>>>> Line 715204: 2020-07-25 02:33:02,875 ERROR
> > > >>>>>> [c.c.v.VirtualMachineManagerImpl]
> > (Work-Job-Executor-51:ctx-594da6d2
> > > >>>>>> job-1208155/job-1208273 ctx-ca115800) (logid:9fa7dece) Failed to
> > > setup
> > > >>>>>> keystore and generate CSR for system vm: s-25035-VM
> > > >>>>>> Line 743140: 2020-07-25 02:38:52,257 ERROR
> > > >>>>>> [c.c.v.VirtualMachineManagerImpl]
> > (Work-Job-Executor-59:ctx-a3f032da
> > > >>>>>> job-1208155/job-1208294 ctx-626a2716) (logid:9fa7dece) Failed to
> > > setup
> > > >>>>>> keystore and generate CSR for system vm: s-25037-VM
> > > >>>>>> Line 817646: 2020-07-25 02:47:07,127 ERROR
> > > >>>>>> [c.c.v.VirtualMachineManagerImpl]
> > (Work-Job-Executor-5:ctx-ca073bd1
> > > >>>>>> job-1208297/job-1208305 ctx-6213af3b) (logid:1a2a782c) Failed to
> > > setup
> > > >>>>>> keystore and generate CSR for system vm: s-25038-VM
> > > >>>>>> Line 847150: 2020-07-25 02:51:46,293 ERROR
> > > >>>>>> [c.c.v.VirtualMachineManagerImpl]
> > (Work-Job-Executor-8:ctx-9db8c548
> > > >>>>>> job-1208296/job-1208312 ctx-83c570ff) (logid:436d1800) Failed to
> > > setup
> > > >>>>>> keystore and generate CSR for system vm: v-25040-VM
> > > >>>>>> Line 903949: 2020-07-25 02:57:48,515 ERROR
> > > >>>>>> [c.c.v.VirtualMachineManagerImpl]
> > (Work-Job-Executor-4:ctx-f35fd59a
> > > >>>>>> job-1208320/job-1208333 ctx-f81179ac) (logid:05eacf0d) Failed to
> > > setup
> > > >>>>>> keystore and generate CSR for system vm: v-25041-VM
> > > >>>>>> Line 957628: 2020-07-25 03:04:13,286 ERROR
> > > >>>>>> [c.c.v.VirtualMachineManagerImpl]
> > (Work-Job-Executor-24:ctx-dc18956a
> > > >>>>>> job-1208320/job-1208381 ctx-b53f5d7c) (logid:05eacf0d) Failed to
> > > setup
> > > >>>>>> keystore and generate CSR for system vm: v-20771-VM
> > > >>>>>> Line 961606: 2020-07-25 03:05:00,522 ERROR
> > > >>>>>> [c.c.v.VirtualMachineManagerImpl]
> > (Work-Job-Executor-28:ctx-83e48421
> > > >>>>>> job-1208320/job-1208392 ctx-e6767f04) (logid:05eacf0d) Failed to
> > > setup
> > > >>>>>> keystore and generate CSR for system vm: v-24763-VM
> > > >>>>>> Line 966633: 2020-07-25 03:05:29,767 ERROR
> > > >>>>>> [c.c.v.VirtualMachineManagerImpl]
> > (Work-Job-Executor-30:ctx-897e2a66
> > > >>>>>> job-1208320/job-1208397 ctx-39c396e1) (logid:05eacf0d) Failed to
> > > setup
> > > >>>>>> keystore and generate CSR for system vm: v-20773-VM
> > > >>>>>> Line 968892: 2020-07-25 03:05:53,648 ERROR
> > > >>>>>> [c.c.v.VirtualMachineManagerImpl]
> > (Work-Job-Executor-36:ctx-68e45b80
> > > >>>>>> job-1208320/job-1208405 ctx-1aaf96dd) (logid:05eacf0d) Failed to
> > > setup
> > > >>>>>> keystore and generate CSR for system vm: v-25041-VM
> > > >>>>>> Line 1003036: 2020-07-25 03:13:33,582 ERROR
> > > >>>>>> [c.c.v.VirtualMachineManagerImpl]
> > (Work-Job-Executor-45:ctx-0f552e0e
> > > >>>>>> job-1208320/job-1208422 ctx-e8835fe9) (logid:05eacf0d) Failed to
> > > setup
> > > >>>>>> keystore and generate CSR for system vm: v-20771-VM
> > > >>>>>> Line 1006496: 2020-07-25 03:14:09,787 ERROR
> > > >>>>>> [c.c.v.VirtualMachineManagerImpl]
> > (Work-Job-Executor-46:ctx-08e5db24
> > > >>>>>> job-1208320/job-1208423 ctx-2cb636c2) (logid:05eacf0d) Failed to
> > > setup
> > > >>>>>> keystore and generate CSR for system vm: v-24763-VM
> > > >>>>>> Line 1008408: 2020-07-25 03:14:39,584 ERROR
> > > >>>>>> [c.c.v.VirtualMachineManagerImpl]
> > (Work-Job-Executor-47:ctx-8b9288a9
> > > >>>>>> job-1208320/job-1208424 ctx-c05d1544) (logid:05eacf0d) Failed to
> > > setup
> > > >>>>>> keystore and generate CSR for system vm: v-20773-VM
> > > >>>>>> Line 1023914: 2020-07-25 03:15:58,139 ERROR
> > > >>>>>> [c.c.v.VirtualMachineManagerImpl]
> > (Work-Job-Executor-48:ctx-4d54d717
> > > >>>>>> job-1208320/job-1208425 ctx-27e5397d) (logid:05eacf0d) Failed to
> > > setup
> > > >>>>>> keystore and generate CSR for system vm: v-25046-VM
> > > >>>>>> Line 1089568: 2020-07-25 03:24:24,534 ERROR
> > > >>>>>> [c.c.v.VirtualMachineManagerImpl]
> > (Work-Job-Executor-59:ctx-62ff8844
> > > >>>>>> job-1208320/job-1208449 ctx-d6a14f7f) (logid:05eacf0d) Failed to
> > > setup
> > > >>>>>> keystore and generate CSR for system vm: v-25048-VM
> > > >>>>>> Line 1097605: 2020-07-25 03:25:23,179 ERROR
> > > >>>>>> [c.c.v.VirtualMachineManagerImpl]
> > (Work-Job-Executor-62:ctx-24407a9e
> > > >>>>>> job-1208321/job-1208452 ctx-84f80ffa) (logid:42edff3a) Failed to
> > > setup
> > > >>>>>> keystore and generate CSR for system vm: s-25050-VM
> > > >>>>>>
> > > >>>>>> -Ammad Ali
> > > >>>>>>
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> On Mon, Aug 17, 2020 at 5:56 PM Andrija Panic <
> > > andrija.panic@gmail.com <ma...@gmail.com>>
> > > >>>>>> wrote:
> > > >>>>>>
> > > >>>>>>> Ammad, no such log lines in the file you uploaded....?
> > > >>>>>>>
> > > >>>>>>> On Mon, 17 Aug 2020 at 09:02, Ammad Syed <
> syedammad83@gmail.com
> > > <ma...@gmail.com>> wrote:
> > > >>>>>>>
> > > >>>>>>>> Hi Andrija,
> > > >>>>>>>>
> > > >>>>>>>> The error occurred only one time. Can you please try to grep
> > > "Failed to
> > > >>>>>>>> setup keystore and generate CSR" you will see all the
> systemVMs
> > > that I
> > > >>>>>>>> tried to create failed because of certificate generation
> failed.
> > > OR you
> > > >>>>>>> can
> > > >>>>>>>> grep job-1208321/job-1208452 this job as well.
> > > >>>>>>>>
> > > >>>>>>>> Also I mentioned above that same thing is happening in my test
> > > >>>>>>> environment
> > > >>>>>>>> that I created 4.11.1 then upgraded 4.11.3 > 4.13.1.
> > > >>>>>>>>
> > > >>>>>>>> Ammad Ali
> > > >>>>>>>>
> > > >>>>>>>> On Mon, Aug 17, 2020 at 11:17 AM Andrija Panic <
> > > >>>>>> andrija.panic@gmail.com <ma...@gmail.com>>
> > > >>>>>>>> wrote:
> > > >>>>>>>>
> > > >>>>>>>>> 2020-07-25 03:23:51,093 WARN [c.c.v.SystemVmLoadScanner]
> > > >>>>>>>>> (secstorage-1:ctx-50756e9b) (logid:3d7c25cb) Unexpected
> > exception
> > > >>>>>>> Failed
> > > >>>>>>>> to
> > > >>>>>>>>> fetch any free public IP address
> > > >>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Failed to
> > fetch
> > > any
> > > >>>>>>> free
> > > >>>>>>>>> public IP address
> > > >>>>>>>>>
> > > >>>>>>>>> ?
> > > >>>>>>>>>
> > > >>>>>>>>> On Sat, 15 Aug 2020 at 18:05, Ammad Syed <
> > syedammad83@gmail.com
> > > <ma...@gmail.com>>
> > > >>>>>>> wrote:
> > > >>>>>>>>>
> > > >>>>>>>>>> I have checked further, the key is successfully injected in
> > > >>>>>>>> systemvm.iso
> > > >>>>>>>>> on
> > > >>>>>>>>>> the management node and copied successfully to xenserver. I
> > have
> > > >>>>>>>> checked
> > > >>>>>>>>> on
> > > >>>>>>>>>> systemVM there is no public key found in authorized_keys of
> > > root.
> > > >>>>>>>>>>
> > > >>>>>>>>>> How can I troubleshoot this further ? how can I enable trace
> > > >>>>>> logging
> > > >>>>>>>> for
> > > >>>>>>>>>> management server to see if there is something problematic
> > > >>>>>> happening.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Ammad Ali
> > > >>>>>>>>>>
> > > >>>>>>>>>> On Sat, Aug 15, 2020 at 1:28 PM Ammad Syed <
> > > syedammad83@gmail.com <ma...@gmail.com>>
> > > >>>>>>>>> wrote:
> > > >>>>>>>>>>
> > > >>>>>>>>>>> Hi,
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> I have setup my test environment, here is what I did:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> - Installed ACS 4.11.1 and added xenserver 7.0 host in it.
> > > >>>>>>> SystemVMs
> > > >>>>>>>>> are
> > > >>>>>>>>>>> up in the zone and agents are up.
> > > >>>>>>>>>>> - Then upgraded the system to 4.11.3. Recreated systemVMs,
> > > agent
> > > >>>>>> is
> > > >>>>>>>> up
> > > >>>>>>>>>> and
> > > >>>>>>>>>>> systemVM are up with 4.11.3 systemVM version.
> > > >>>>>>>>>>> - Then upgraded the system to 4.13.1, recreated systemVMs
> are
> > > >>>>>>> running
> > > >>>>>>>>> but
> > > >>>>>>>>>>> agents are not up.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> I have checked md5sum of systemvm.iso on xenserver and
> > > management
> > > >>>>>>>>> server,
> > > >>>>>>>>>>> both are same.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> [root@xenserver iso]# md5sum
> > > >>>>>>>> /opt/xensource/packages/iso/systemvm.iso
> > > >>>>>>>>>>> baba18f156395da3a5d8208727d8f421
> > > >>>>>>>>>> /opt/xensource/packages/iso/systemvm.iso
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> [root@cloudstack-upgrade vms]# md5sum
> > > >>>>>>>>>>> /usr/share/cloudstack-common/vms/systemvm.iso
> > > >>>>>>>>>>> baba18f156395da3a5d8208727d8f421
> > > >>>>>>>>>>> /usr/share/cloudstack-common/vms/systemvm.iso
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Also the private key on xenserver root and management
> server
> > > are
> > > >>>>>>>> same.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> management server
> > > >>>>>>>>>>>
> /usr/share/cloudstack-common/scripts/vm/systemvm/id_rsa.cloud
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> xenserver path
> > > >>>>>>>>>>> /root/.ssh/id_rsa.cloud
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> - Ammad Ali
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> On Thu, Aug 13, 2020 at 11:27 AM Ammad Syed <
> > > >>>>>> syedammad83@gmail.com <ma...@gmail.com>
> > > >>>>>>>>
> > > >>>>>>>>>> wrote:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> Here is the link for download management logs.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > >
> >
> https://drive.google.com/file/d/1l6HDPguGUNaOxsc7VSaj7eaP0F2UOFjA/view?usp=sharing
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> On Thu, Aug 13, 2020 at 11:22 AM Ammad Syed <
> > > >>>>>>> syedammad83@gmail.com>
> > > >>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>> I have reverted the version back to 4.11.3. But I have
> > saved
> > > >>>>>> logs
> > > >>>>>>>>>>>>> starting from upgrade.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> I think the key has been copied successfully in system vm
> > > iso.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> 2020-07-25 00:34:17,214 INFO
> > [c.c.s.ConfigurationServerImpl]
> > > >>>>>>>>>>>>> (main:null) (logid:) Going to update systemvm iso with
> > > >>>>>> generated
> > > >>>>>>>>>> keypairs
> > > >>>>>>>>>>>>> if needed
> > > >>>>>>>>>>>>> 2020-07-25 00:34:17,214 INFO
> > [c.c.s.ConfigurationServerImpl]
> > > >>>>>>>>>>>>> (main:null) (logid:) Trying to inject public and private
> > keys
> > > >>>>>>> into
> > > >>>>>>>>>> systemvm
> > > >>>>>>>>>>>>> iso
> > > >>>>>>>>>>>>> 2020-07-25 00:34:17,217 DEBUG [c.c.u.s.Script]
> (main:null)
> > > >>>>>>> (logid:)
> > > >>>>>>>>>>>>> Looking for vms/systemvm.iso in the classpath
> > > >>>>>>>>>>>>> 2020-07-25 00:34:17,217 DEBUG [c.c.u.s.Script]
> (main:null)
> > > >>>>>>> (logid:)
> > > >>>>>>>>>>>>> System resource:
> > > >>>>>>> file:/usr/share/cloudstack-common/vms/systemvm.iso
> > > >>>>>>>>>>>>> 2020-07-25 00:34:17,217 DEBUG [c.c.u.s.Script]
> (main:null)
> > > >>>>>>> (logid:)
> > > >>>>>>>>>>>>> Absolute path =
> > > /usr/share/cloudstack-common/vms/systemvm.iso
> > > >>>>>>>>>>>>> 2020-07-25 00:34:17,218 DEBUG
> > [c.c.s.ConfigurationServerImpl]
> > > >>>>>>>>>>>>> (main:null) (logid:) Executing: /bin/bash
> > > >>>>>>>>>>>>>
> > > /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh
> > > >>>>>>>>>>>>> /var/cloudstack/management/.ssh/id_rsa.pub
> > > >>>>>>>>>>>>> /var/cloudstack/management/.ssh/id_rsa
> > > >>>>>>>>>>>>> /usr/share/cloudstack-common/vms/systemvm.iso
> > > >>>>>>>>>>>>> 2020-07-25 00:34:17,636 INFO
> > [c.c.s.ConfigurationServerImpl]
> > > >>>>>>>>>>>>> (main:null) (logid:) Injected public and private keys
> into
> > > >>>>>>> systemvm
> > > >>>>>>>>> iso
> > > >>>>>>>>>>>>> with result : null
> > > >>>>>>>>>>>>> 2020-07-25 00:34:50,613 DEBUG
> > [c.c.h.x.r.CitrixResourceBase]
> > > >>>>>>>>>>>>> (DirectAgent-1:ctx-d3dc4cf2) (logid:1d22396d) Copying
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > >
> >
> /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/xenserver65/../../../../../vms/systemvm.iso
> > > >>>>>>>>>>>>> to /opt/xensource/packages/iso on 172.16.2.22 with
> > permission
> > > >>>>>>> 0644
> > > >>>>>>>>>>>>> 2020-07-25 00:34:52,566 DEBUG
> > [c.c.h.x.r.CitrixResourceBase]
> > > >>>>>>>>>>>>> (DirectAgent-2:ctx-2537b610) (logid:29c67b7a) Copying
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > >
> >
> /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/xenserver65/../../../../../vms/systemvm.iso
> > > >>>>>>>>>>>>> to /opt/xensource/packages/iso on 172.16.2.5 with
> > permission
> > > >>>>>> 0644
> > > >>>>>>>>>>>>> 2020-07-25 00:34:53,170 DEBUG
> > [c.c.h.x.r.CitrixResourceBase]
> > > >>>>>>>>>>>>> (DirectAgent-3:ctx-168ac27d) (logid:a7862c4b) Copying
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > >
> >
> /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/xenserver65/../../../../../vms/systemvm.iso
> > > >>>>>>>>>>>>> to /opt/xensource/packages/iso on 172.16.2.9 with
> > permission
> > > >>>>>> 0644
> > > >>>>>>>>>>>>> 2020-07-25 00:34:54,621 DEBUG
> > [c.c.h.x.r.CitrixResourceBase]
> > > >>>>>>>>>>>>> (DirectAgent-6:ctx-f640fe55) (logid:0a62d4cf) Copying
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > >
> >
> /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/xenserver65/../../../../../vms/systemvm.iso
> > > >>>>>>>>>>>>> to /opt/xensource/packages/iso on 172.16.2.5 with
> > permission
> > > >>>>>> 0644
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> I have attached complete management logs. The start of
> logs
> > > is
> > > >>>>>>> the
> > > >>>>>>>>>> start
> > > >>>>>>>>>>>>> of management server after upgrade of management server
> > from
> > > >>>>>>> 4.11.3
> > > >>>>>>>>> to
> > > >>>>>>>>>>>>> 4.13.1.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Ammad Ali
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> On Thu, Aug 13, 2020 at 1:35 AM Andrija Panic <
> > > >>>>>>>>> andrija.panic@gmail.com
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Do you get an error while trying to inject ssh key into
> > the
> > > >>>>>>>>>> systemvm.iso
> > > >>>>>>>>>>>>>> (mgmt logs) , or can you confirm that the systemvm.iso
> on
> > XS
> > > >>>>>>> host
> > > >>>>>>>>> and
> > > >>>>>>>>>>>>>> the
> > > >>>>>>>>>>>>>> one on the mgmt server are identical, md5sum them (i.e.
> > has
> > > >>>>>> the
> > > >>>>>>>> iso
> > > >>>>>>>>>> been
> > > >>>>>>>>>>>>>> copied over to the XS successfully) - this might explain
> > not
> > > >>>>>>> being
> > > >>>>>>>>>> able
> > > >>>>>>>>>>>>>> to
> > > >>>>>>>>>>>>>> login via ssh with your private key.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Best,
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> On Wed, 12 Aug 2020, 09:08 Ammad Syed, <
> > > syedammad83@gmail.com
> > > >>>>>>>
> > > >>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Yes this is exactly the same issue that i faced.
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Sent from my iPhone
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> On 12-Aug-2020, at 8:35 AM, Eric Lee Green <
> > > >>>>>>>>>>>>>> eric.lee.green@gmail.com>
> > > >>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Correct, 4.11.3 template is used for 4.11.3, 4.12,
> and
> > > >>>>>>> 4.13.
> > > >>>>>>>>> 4.14
> > > >>>>>>>>>>>>>> moves
> > > >>>>>>>>>>>>>>> to the 4.14.0 template.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> There seems to be something odd happening key-wise
> > > >>>>>> sometimes
> > > >>>>>>>>> with
> > > >>>>>>>>>>>>>>> upgrades from 4.11.3 to 4.13.1 or 4.14.0. I managed
> an
> > > >>>>>>> upgrade
> > > >>>>>>>>>> from
> > > >>>>>>>>>>>>>>> 4.11.3 to 4.13.1 that *almost* worked, but the
> secondary
> > > >>>>>>> storage
> > > >>>>>>>>> VM
> > > >>>>>>>>>>>>>>> wouldn't work and thus I couldn't spawn new virtual
> > > >>>>>> machines.
> > > >>>>>>>> Same
> > > >>>>>>>>>>>>>> symptom
> > > >>>>>>>>>>>>>>> -- key error when the agent tried to ssh into it. And
> > > >>>>>> deleting
> > > >>>>>>>> it
> > > >>>>>>>>>> and
> > > >>>>>>>>>>>>>>> making it respawn didn't help. Then I tried 4.11.3 to
> > > 4.14.0
> > > >>>>>>> and
> > > >>>>>>>>>>>>>> *all* the
> > > >>>>>>>>>>>>>>> VM's failed at that point (of course, that was with the
> > new
> > > >>>>>>>>>> template).
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Right now I'm back at 4.11.3 until this can be figured
> > > >>>>>> out.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> On 8/11/2020 5:53 AM, Ammad Syed wrote:
> > > >>>>>>>>>>>>>>>>> Hi,
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> I think 4.12 and 4.13 uses same systemVM template i.e
> > > >>>>>>> 4.11.3
> > > >>>>>>>>>>>>>> version,
> > > >>>>>>>>>>>>>>> which
> > > >>>>>>>>>>>>>>>>> I already have registered. Currently I am running
> > 4.11.3
> > > >>>>>>>>> version
> > > >>>>>>>>>>>>>> of ACS.
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> MariaDB [cloud]> SELECT
> id,name,type,cross_zones,state
> > > >>>>>> FROM
> > > >>>>>>>>>>>>>>>>> cloud.vm_template WHERE name like
> > '%systemvm-xenserver%'
> > > >>>>>>> AND
> > > >>>>>>>>>>>>>> removed IS
> > > >>>>>>>>>>>>>>>>> NULL;
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>
> > >
> +------+-----------------------------+--------+-------------+----------+
> > > >>>>>>>>>>>>>>>>> | id | name | type |
> > > >>>>>>> cross_zones |
> > > >>>>>>>>>>>>>> state |
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>
> > >
> +------+-----------------------------+--------+-------------+----------+
> > > >>>>>>>>>>>>>>>>> | 337 | systemvm-xenserver-3.0.0 | SYSTEM |
> > > >>>>>>> 0 |
> > > >>>>>>>>>>>>>> Inactive |
> > > >>>>>>>>>>>>>>>>> | 418 | systemvm-xenserver-4.2 | SYSTEM |
> > > >>>>>>> 0 |
> > > >>>>>>>>>>>>>> Active |
> > > >>>>>>>>>>>>>>>>> | 472 | systemvm-xenserver-4.3 | USER |
> > > >>>>>>> 1 |
> > > >>>>>>>>>>>>>> Inactive |
> > > >>>>>>>>>>>>>>>>> | 473 | systemvm-xenserver-4.3 | USER |
> > > >>>>>>> 1 |
> > > >>>>>>>>>>>>>> Inactive |
> > > >>>>>>>>>>>>>>>>> | 474 | systemvm-xenserver-4.3 | USER |
> > > >>>>>>> 1 |
> > > >>>>>>>>>>>>>> Inactive |
> > > >>>>>>>>>>>>>>>>> | 475 | systemvm-xenserver-4.3 | USER |
> > > >>>>>>> 1 |
> > > >>>>>>>>>>>>>> Inactive |
> > > >>>>>>>>>>>>>>>>> | 476 | systemvm-xenserver-4.3 | USER |
> > > >>>>>>> 0 |
> > > >>>>>>>>>>>>>> Inactive |
> > > >>>>>>>>>>>>>>>>> | 479 | systemvm-xenserver-4.3-2 | USER |
> > > >>>>>>> 1 |
> > > >>>>>>>>>>>>>> Inactive |
> > > >>>>>>>>>>>>>>>>> | 480 | systemvm-xenserver-4.3 | SYSTEM |
> > > >>>>>>> 0 |
> > > >>>>>>>>>>>>>> Active |
> > > >>>>>>>>>>>>>>>>> | 549 | systemvm-xenserver-4.5.1 | USER |
> > > >>>>>>> 0 |
> > > >>>>>>>>>>>>>> Active |
> > > >>>>>>>>>>>>>>>>> | 550 | systemvm-xenserver-4.5.1 | SYSTEM |
> > > >>>>>>> 0 |
> > > >>>>>>>>>>>>>> Active |
> > > >>>>>>>>>>>>>>>>> | 651 | systemvm-xenserver-4.7.0 | USER |
> > > >>>>>>> 0 |
> > > >>>>>>>>>>>>>> Inactive |
> > > >>>>>>>>>>>>>>>>> | 652 | systemvm-xenserver-4.7.0 | USER |
> > > >>>>>>> 0 |
> > > >>>>>>>>>>>>>> Inactive |
> > > >>>>>>>>>>>>>>>>> | 653 | systemvm-xenserver-4.7.0 | SYSTEM |
> > > >>>>>>> 0 |
> > > >>>>>>>>>>>>>> Inactive |
> > > >>>>>>>>>>>>>>>>> | 737 | systemvm-xenserver-4.9.2 | SYSTEM |
> > > >>>>>>> 1 |
> > > >>>>>>>>>>>>>> Inactive |
> > > >>>>>>>>>>>>>>>>> | 739 | systemvm-xenserver-4.9.2-sb | SYSTEM |
> > > >>>>>>> 1 |
> > > >>>>>>>>>>>>>> Active |
> > > >>>>>>>>>>>>>>>>> | 1245 | systemvm-xenserver-4.11.1 | SYSTEM |
> > > >>>>>>> 1 |
> > > >>>>>>>>>>>>>> Active |
> > > >>>>>>>>>>>>>>>>> | 1584 | systemvm-xenserver-4.11.2 | SYSTEM |
> > > >>>>>>> 1 |
> > > >>>>>>>>>>>>>> Active |
> > > >>>>>>>>>>>>>>>>> | 1677 | systemvm-xenserver-4.11.3 | SYSTEM |
> > > >>>>>>> 1 |
> > > >>>>>>>>>>>>>> Active |
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>
> > >
> +------+-----------------------------+--------+-------------+----------+
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> - Ammad
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>> On Tue, Aug 11, 2020 at 5:17 PM Pierre-Luc Dion <
> > > >>>>>>>>>>>>>> pdion891@apache.org>
> > > >>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>> db.
> > > >>>>>>>>>>>>>>>>>> Hi Syed,
> > > >>>>>>>>>>>>>>>>>> From 4.12, the systemvm template had to be upgraded
> > > >>>>>>> because
> > > >>>>>>>> of
> > > >>>>>>>>>> OS
> > > >>>>>>>>>>>>>>> change in
> > > >>>>>>>>>>>>>>>>>> the template, moved to a latest version of Debian.
> > > >>>>>> Because
> > > >>>>>>>> of
> > > >>>>>>>>>>>>>> that,
> > > >>>>>>>>>>>>>>> some VR
> > > >>>>>>>>>>>>>>>>>> scripts have changed and make obsolete older version
> > of
> > > >>>>>>> VRs,
> > > >>>>>>>>> so
> > > >>>>>>>>>>>>>> you
> > > >>>>>>>>>>>>>>> will
> > > >>>>>>>>>>>>>>>>>> most likely have to register an updated systemvm
> > > >>>>>> templates
> > > >>>>>>>> and
> > > >>>>>>>>>>>>>> upgrade
> > > >>>>>>>>>>>>>>> your
> > > >>>>>>>>>>>>>>>>>> system VMs and VRs.
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>> Regards,
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> On Tue, Aug 11, 2020 at 6:24 AM Ammad Syed <
> > > >>>>>>>>>>>>>> syedammad83@gmail.com>
> > > >>>>>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> Hi Guys,
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> I was previously on 4.9.3 cloudstack and upgraded
> to
> > > >>>>>>> 4.11.1
> > > >>>>>>>>>> then
> > > >>>>>>>>>>>>>>> 4.11.3.
> > > >>>>>>>>>>>>>>>>>>> The version 4.11.3 is working fine since six
> months.
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> Now I have tried to upgrade my system from 4.11.3
> to
> > > >>>>>>>> 4.13.1.
> > > >>>>>>>>>> The
> > > >>>>>>>>>>>>>>> upgrade
> > > >>>>>>>>>>>>>>>>>>> goes successful. I didn't uploaded any system VM
> > > >>>>>>> template.
> > > >>>>>>>>>>>>>> However the
> > > >>>>>>>>>>>>>>>>>>> problem occured when I recreated my systemVM of
> POD,
> > > >>>>>> the
> > > >>>>>>> VM
> > > >>>>>>>>>>>>>> recreated
> > > >>>>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>>>>> its state was running but agent state was not
> getting
> > > >>>>>> up,
> > > >>>>>>>> its
> > > >>>>>>>>>>>>>> showing
> > > >>>>>>>>>>>>>>>>>> blank
> > > >>>>>>>>>>>>>>>>>>> in column.
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> Digging further via job logs, the job is failed
> with
> > > >>>>>>> error
> > > >>>>>>>>> that
> > > >>>>>>>>>>>>>>> unable to
> > > >>>>>>>>>>>>>>>>>>> execute command via ssh. Below are the logs.
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> 2020-07-25 02:30:48,126 ERROR [c.c.u.s.SshHelper]
> > > >>>>>>>>>>>>>>>>>>> (DirectAgent-211:ctx-62f09b31) (logid:9fa7dece) SSH
> > > >>>>>>>> execution
> > > >>>>>>>>>> of
> > > >>>>>>>>>>>>>>> command
> > > >>>>>>>>>>>>>>>>>>> /opt/cloud/bin/router_proxy.sh keystore-s
> > > >>>>>>>>>>>>>>>>>>> etup 169.254.2.199
> > > >>>>>>>>>>>>>> /usr/local/cloud/systemvm/conf/agent.properties
> > > >>>>>>>>>>>>>>>>>>> /usr/local/cloud/systemvm/conf/cloud.jks
> > > >>>>>> TJaQYChYBwKh7Cx9
> > > >>>>>>>> 365
> > > >>>>>>>>>>>>>>>>>>> /usr/local/cloud/systemvm/conf/clou
> > > >>>>>>>>>>>>>>>>>>> d.csr has an error status code in return. Result
> > > >>>>>> output:
> > > >>>>>>>>>>>>>>>>>>> 2020-07-25 02:30:48,127 DEBUG
> > > >>>>>>> [c.c.a.m.DirectAgentAttache]
> > > >>>>>>>>>>>>>>>>>>> (DirectAgent-211:ctx-62f09b31) (logid:9fa7dece) Seq
> > > >>>>>>>>>>>>>>>>>>> 906-3195585410596077730: Response Received:
> > > >>>>>>>>>>>>>>>>>>> 2020-07-25 02:30:48,127 DEBUG [c.c.a.t.Request]
> > > >>>>>>>>>>>>>>>>>>> (DirectAgent-211:ctx-62f09b31) (logid:9fa7dece) Seq
> > > >>>>>>>>>>>>>>>>>>> 906-3195585410596077730: Processing: { Ans: ,
> > MgmtId:
> > > >>>>>>>>>> 779271079
> > > >>>>>>>>>>>>>>>>>>> 43497, via: 906(xen-21-10-a3-khi02), Ver: v1,
> Flags:
> > > >>>>>> 10,
> > > >>>>>>>>>>>>>>>>>>> [{"org.apache.cloudstack.ca
> > > >>>>>>>>>>>>>>>>>>> .SetupKeystoreAnswer":{"result":false,"wait":0}}]
> > > >>>>>>>>>>>>>>>>>>> }
> > > >>>>>>>>>>>>>>>>>>> 2020-07-25 02:30:48,127 DEBUG [c.c.a.t.Request]
> > > >>>>>>>>>>>>>>>>>>> (Work-Job-Executor-41:ctx-4e3c666d
> > > >>>>>>> job-1208155/job-1208258
> > > >>>>>>>>>>>>>>> ctx-df740f75)
> > > >>>>>>>>>>>>>>>>>>> (logid:9fa7dece) Seq 906-319558541059607773
> > > >>>>>>>>>>>>>>>>>>> 0: Received: { Ans: , MgmtId: 77927107943497, via:
> > > >>>>>>>>>>>>>>>>>>> 906(xen-21-10-a3-khi02), Ver: v1, Flags: 10, {
> > > >>>>>>>>>>>>>> SetupKeystoreAnswer } }
> > > >>>>>>>>>>>>>>>>>>> 2020-07-25 02:30:48,127 ERROR
> > > >>>>>>>>> [c.c.v.VirtualMachineManagerImpl]
> > > >>>>>>>>>>>>>>>>>>> (Work-Job-Executor-41:ctx-4e3c666d
> > > >>>>>>> job-1208155/job-1208258
> > > >>>>>>>>>>>>>>> ctx-df740f75)
> > > >>>>>>>>>>>>>>>>>>> (logid:9fa7dece) Failed to setup keystore and
> > generate
> > > >>>>>>> CSR
> > > >>>>>>>>> for
> > > >>>>>>>>>>>>>> system
> > > >>>>>>>>>>>>>>> vm:
> > > >>>>>>>>>>>>>>>>>>> s-24142-VM
> > > >>>>>>>>>>>>>>>>>>> 2020-07-25 02:30:48,127 DEBUG
> > > >>>>>>> [c.c.v.VmWorkJobHandlerProxy]
> > > >>>>>>>>>>>>>>>>>>> (Work-Job-Executor-41:ctx-4e3c666d
> > > >>>>>>> job-1208155/job-1208258
> > > >>>>>>>>>>>>>>> ctx-df740f75)
> > > >>>>>>>>>>>>>>>>>>> (logid:9fa7dece) Done executing VM work job:
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > >
> >
> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":24142,"handlerName":"VirtualMachineManagerImpl"}
> > > >>>>>>>>>>>>>>>>>>> 2020-07-25 02:30:48,128 DEBUG
> > > >>>>>>>>> [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > >>>>>>>>>>>>>>>>>>> (Work-Job-Executor-41:ctx-4e3c666d
> > > >>>>>>> job-1208155/job-1208258
> > > >>>>>>>>>>>>>>> ctx-df740f75)
> > > >>>>>>>>>>>>>>>>>>> (logid:9fa7dece) Complete async job-1208258,
> > jobStatus:
> > > >>>>>>>>>>>>>> SUCCEEDED,
> > > >>>>>>>>>>>>>>>>>>> resultCode: 0, result: null
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> I tried to dig it further, I was unable to login
> > > >>>>>> systemVM
> > > >>>>>>>> via
> > > >>>>>>>>>>>>>> ssh from
> > > >>>>>>>>>>>>>>>>>>> xenserver host with key /root/.ssh/id_rsa.cloud
> > placed.
> > > >>>>>>>> Look
> > > >>>>>>>>>> like
> > > >>>>>>>>>>>>>>> private
> > > >>>>>>>>>>>>>>>>>>> key issue. However I am able to login on my old
> > > >>>>>>> systemVMs (
> > > >>>>>>>>> i.e
> > > >>>>>>>>>>>>>>> created
> > > >>>>>>>>>>>>>>>>>> on
> > > >>>>>>>>>>>>>>>>>>> ACS 4.11.3)
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> Also I have SSL certificate enabled for console
> proxy
> > > >>>>>> on
> > > >>>>>>> my
> > > >>>>>>>>> ACS
> > > >>>>>>>>>>>>>> 4.11.3
> > > >>>>>>>>>>>>>>>>>> and
> > > >>>>>>>>>>>>>>>>>>> I am using only xenserver 7.0 hosts.
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> I tried to disable SSL on secstorage and console
> > proxy
> > > >>>>>>> from
> > > >>>>>>>>>>>>>> global
> > > >>>>>>>>>>>>>>>>>>> settings, but still didn't worked.
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> I had a fresh installation of ACS 4.13.1 with
> > xenserver
> > > >>>>>>>> 7.0,
> > > >>>>>>>>>>>>>> systemVMs
> > > >>>>>>>>>>>>>>>>>> are
> > > >>>>>>>>>>>>>>>>>>> working fine in it.
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> Please advise.
> > > >>>>>>>>>>>>>>>>>>> --
> > > >>>>>>>>>>>>>>>>>>> Regards,
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>>> Syed Ammad Ali
> > > >>>>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> --
> > > >>>>>>>>>>>>> Regards,
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Syed Ammad Ali
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> --
> > > >>>>>>>>>>>> Regards,
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Syed Ammad Ali
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> --
> > > >>>>>>>>>>> Regards,
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Syed Ammad Ali
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>> --
> > > >>>>>>>>>> Regards,
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>> Syed Ammad Ali
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>> --
> > > >>>>>>>>>
> > > >>>>>>>>> Andrija Panić
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>> --
> > > >>>>>>>> Regards,
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>> Syed Ammad Ali
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> --
> > > >>>>>>>
> > > >>>>>>> Andrija Panić
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> --
> > > >>>>>> Regards,
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> Syed Ammad Ali
> > > >>>>>>
> > > >>>>>
> > > >>>>>
> > > >>>>> --
> > > >>>>>
> > > >>>>> Andrija Panić
> > > >>>>
> > > >>>>
> > > >>>> --
> > > >>>> Regards,
> > > >>>>
> > > >>>>
> > > >>>> Syed Ammad Ali
> > > >>
> > > >
> > >
> >
> >
> > --
> >
> > Andrija Panić
> >
>
>
> --
> Regards,
>
>
> Syed Ammad Ali
>
--
Andrija Panić
Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error
Posted by Ammad Syed <sy...@gmail.com>.
Hi,
Thank you guys, this has fixed my problem. Here are the steps that I have
performed.
- Did a fresh installation of 4.13.1 and copied its systemvm.iso on my ACS
in /usr/share/cloudstack-common/vms/.
- Shutdown management server.
- Deleted systemVMs from xencenter.
- Executed key injection script
"/usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh
/root/.ssh/id_rsa.pub /root/.ssh/id_rsa
/usr/share/cloudstack-common/vms/systemvm.iso"
- Cleared the tags xe host-param-clear param-name=tags uuid=<uuid of the XS
host>
- Started management server.
- Recreated systemVMs.
Thank you guys for your support.
- Ammad
On Thu, Sep 3, 2020 at 11:40 AM Andrija Panic <an...@gmail.com>
wrote:
> Ammad,
>
> please check what Pearl has suggested, but I think the correct syntax
> (different id_rsa,pub, specifically the one used by ACS) should be as
> following (you can always grep your older logs for "injectkey.sh" :
>
> /bin/bash /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh
> /var/cloudstack/management/.ssh/id_rsa.pub
> /var/cloudstack/management/.ssh/id_rsa
> /usr/share/cloudstack-common/vms/systemvm.iso
> You should get e meaningful output if the key was injected or not (perhaps
> the key inside doesn't differer, so the script will not inject it again)
> Proceed with what Pearl has suggested as the next steps.
>
> Best,
>
> On Thu, 3 Sep 2020 at 06:16, Pearl d'Silva <pe...@shapeblue.com>
> wrote:
>
> > Hi Ammad,
> >
> > Could you try re-injecting the keys into the systemVM, as this MAY happen
> > due to stale id_rsa keys. Run the injectkeys.sh to inject the existing
> > public ssh key into the authorized_keys file inside the systemvm.iso.
> > Execute:
> >
> >
> > /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh
> > /root/.ssh/id_rsa.pub /root/.ssh/id_rsa
> > /usr/share/cloudstack-common/vms/systemvm.iso'
> >
> > * Then replace the systemvm.iso on the hypervisor hosts. For each
> host:
> >
> > 1. Login and type command - xe host-param-clear param-name=tags
> > uuid=<uuid of the XS host>
> > host uuid maybe obtained using: xe host-list
> >
> > 2. 2. Restart MS and the iso should get propagated for all. Check
> > timestamp.
> > 3. 3. Restart systemvms and they should get the iso inserted.
> > 4.
> >
> >
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/SystemVm.iso#SystemVm.iso-Xen
> >
> > Thanks,
> > Pearl
> >
> > ________________________________
> > From: Ammad Syed <sy...@gmail.com>
> > Sent: Wednesday, September 2, 2020 9:12 PM
> > To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> > Subject: Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error
> >
> > Guys, please advise how to troubleshoot this further ? How can i enable
> > trace level debugging ? any help in this will be highly appreciated?
> >
> > Ammad
> >
> >
> > pearl.dsilva@shapeblue.com
> > www.shapeblue.com
> > 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
> > @shapeblue
> >
> >
> >
> > > On 27-Aug-2020, at 1:41 PM, Vivek Kumar <vivek.kumar@indiqus.com
> .invalid>
> > wrote:
> > >
> > > Pardon, the whole mail chain was not visible so replied accordingly.
> It
> > seems like you are doing SSH using root shell so it doesn’t matter.
> > >
> > > Vivek Kumar
> > > Manager - Cloud & DevOps
> > > IndiQus Technologies
> > > 24*7 O +91 11 4055 1411 | M +91 7503460090
> > > www.indiqus.com<http://www.indiqus.com> <http://indiqus.com/>
> > >
> > > This message is intended only for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential
> > and/or privileged. If you are not the intended recipient please delete
> the
> > original message and any copy of it from your computer system. You are
> > hereby notified that any dissemination, distribution or copying of this
> > communication is strictly prohibited unless proper authorization has been
> > obtained for such action. If you have received this communication in
> error,
> > please notify the sender immediately. Although IndiQus attempts to sweep
> > e-mail and attachments for viruses, it does not guarantee that both are
> > virus-free and accepts no liability for any damage sustained as a result
> of
> > viruses.
> > >
> > >> On 27-Aug-2020, at 2:03 PM, Vivek Kumar <vi...@indiqus.com>
> > wrote:
> > >>
> > >> Hello Ammad,
> > >>
> > >> You haven’t defined the user while logging to the system VM through
> the
> > Link local IP that’s why you have got the error while logging to the
> system
> > VM.
> > >>
> > >>>> ssh -i /root/.ssh/id_rsa.cloud 169.254.0.121 -p 3922
> > >>
> > >> Try instead below
> > >>
> > >>>> ssh -i /root/.ssh/id_rsa.cloud -p 3922 root@169.254.0.121 <mailto:
> > root@169.254.0.121>
> > >>
> > >> Vivek Kumar
> > >> Manager - Cloud & DevOps
> > >> IndiQus Technologies
> > >> 24*7 O +91 11 4055 1411 | M +91 7503460090
> > >> www.indiqus.com<http://www.indiqus.com> <http://indiqus.com/>
> > >>
> > >> This message is intended only for the use of the individual or entity
> > to which it is addressed and may contain information that is confidential
> > and/or privileged. If you are not the intended recipient please delete
> the
> > original message and any copy of it from your computer system. You are
> > hereby notified that any dissemination, distribution or copying of this
> > communication is strictly prohibited unless proper authorization has been
> > obtained for such action. If you have received this communication in
> error,
> > please notify the sender immediately. Although IndiQus attempts to sweep
> > e-mail and attachments for viruses, it does not guarantee that both are
> > virus-free and accepts no liability for any damage sustained as a result
> of
> > viruses.
> > >>
> > >>>> On 27-Aug-2020, at 1:59 PM, Ammad Syed <syedammad83@gmail.com
> > <ma...@gmail.com>> wrote:
> > >>>
> > >>> Guys, any help to troubleshoot this further would be highly
> > appreciated.
> > >>>
> > >>>
> > >>> Ammad Ali
> > >>>> On 18-Aug-2020, at 1:05 PM, Ammad Syed <syedammad83@gmail.com
> > <ma...@gmail.com>> wrote:
> > >>>>
> > >>>>
> > >>>> Hi,
> > >>>>
> > >>>> The systemVM is accessible with 169.x.x.x IP from xenserver host.
> But
> > login with private key is denied. I am testing this on my test
> environment.
> > >>>>
> > >>>> [root@xenserver ~]# ping 169.254.0.121
> > >>>> PING 169.254.0.121 (169.254.0.121) 56(84) bytes of data.
> > >>>> 64 bytes from 169.254.0.121: icmp_seq=1 ttl=64 time=0.328 ms
> > >>>> 64 bytes from 169.254.0.121: icmp_seq=2 ttl=64 time=0.124 ms
> > >>>> 64 bytes from 169.254.0.121: icmp_seq=3 ttl=64 time=0.111 ms
> > >>>> 64 bytes from 169.254.0.121: icmp_seq=4 ttl=64 time=0.126 ms
> > >>>> ^C
> > >>>> --- 169.254.0.121 ping statistics ---
> > >>>> 4 packets transmitted, 4 received, 0% packet loss, time 2997ms
> > >>>> rtt min/avg/max/mdev = 0.111/0.172/0.328/0.090 ms
> > >>>> [root@xenserver ~]#
> > >>>> [root@xenserver ~]#
> > >>>> [root@xenserver ~]# ssh -i /root/.ssh/id_rsa.cloud 169.254.0.121 -p
> > 3922
> > >>>> Permission denied (publickey).
> > >>>>
> > >>>> I have checked by logging into the systemVM, the public key is not
> > found in /root/.ssh/authorized_keys.
> > >>>>
> > >>>> I have digged further, the systemvm.iso is same on the management
> > node and xenserver host. I have checked by mounting the ISO as well, the
> > public key is there in authorized_keys in ISO.
> > >>>>
> > >>>> Restarting / recreating systemVM isn't fixed the issue.
> > >>>>
> > >>>> The systemVM is already running on HVM mode on xenserver host as the
> > template OS type is Debian 8 64bit is selected.
> > >>>>
> > >>>> On my PRD upgrade, I have tested on one zone only. Currently I am
> > testing on my test ACS environment which have only one zone.
> > >>>>
> > >>>> Ammad Ali
> > >>>>
> > >>>>> On Mon, Aug 17, 2020 at 11:10 PM Andrija Panic <
> > andrija.panic@gmail.com <ma...@gmail.com>> wrote:
> > >>>>> It was my tool (GUI based tool, that still doesn't return any line
> > (???)
> > >>>>> but I see it.
> > >>>>>
> > >>>>> Problem is with 169.254.1.178 not being reachable over SSH
> > (2020-07-25
> > >>>>> 01:45:52,097 DEBUG [c.c.h.x.r.CitrixResourceBase]
> > >>>>> (DirectAgent-57:ctx-ded7f7f9) (logid:a75971c2) Executing command in
> > VR:
> > >>>>> /opt/cloud/bin/router_proxy.sh keystore-setup 169.254.1.178
> > >>>>> /usr/local/cloud/systemvm/conf/agent.properties
> > >>>>> /usr/local/cloud/systemvm/conf/cloud.jks Xe8zzqZmUp9wmABH 365
> > >>>>> /usr/local/cloud/systemvm/conf/cloud.csr)
> > >>>>>
> > >>>>> Can you try to ssh from that particular XenServer (where the SSVM
> is
> > >>>>> running) to the IP 169.xxxxx on port 3922 (ssh port) using the
> > private key
> > >>>>> - if you log in to the VM via it's console (root/password) - do you
> > see
> > >>>>> with ifconfig the interface being started with 169..xxxx IP
> address?
> > I
> > >>>>> assume that destroying the SSVM and starting a new one doesn't fix
> > the
> > >>>>> issue?
> > >>>>>
> > >>>>> You can also try changing the OS type of the systemVM template to
> > Other
> > >>>>> Linux 64bit, so it runs in HVM mode (instead of PV) - you'll need
> to
> > see ID
> > >>>>> for that OS type from the guest_os table, and set the guest_os_id
> > field in
> > >>>>> the vm_template table for your systemVM template.
> > >>>>>
> > >>>>> Does the issue occur in other zones as well? (assuming same
> > hypervisor type)
> > >>>>>
> > >>>>>
> > >>>>> On Mon, 17 Aug 2020 at 15:31, Ammad Syed <syedammad83@gmail.com
> > <ma...@gmail.com>> wrote:
> > >>>>>
> > >>>>>> I have rechecked the logs are there. Please review below lines.
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>>
> >
> C:\Users\ammad\Downloads\management-server.log-4.13\management-server.log-4.13.1-errors
> > >>>>>> (27 hits)
> > >>>>>> Line 409969: 2020-07-25 01:45:52,320 ERROR
> > >>>>>> [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-58:ctx-9dc75334
> > >>>>>> job-1207969/job-1208104 ctx-09e752ec) (logid:a75971c2) Failed to
> > setup
> > >>>>>> keystore and generate CSR for system vm: s-25023-VM
> > >>>>>> Line 437413: 2020-07-25 01:51:33,845 ERROR
> > >>>>>> [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-70:ctx-457ce4ee
> > >>>>>> job-1207969/job-1208129 ctx-195da69e) (logid:a75971c2) Failed to
> > setup
> > >>>>>> keystore and generate CSR for system vm: s-25024-VM
> > >>>>>> Line 549538: 2020-07-25 02:06:44,833 ERROR
> > >>>>>> [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-3:ctx-4634f7e0
> > >>>>>> job-1208154/job-1208161 ctx-05d065dc) (logid:91fc037a) Failed to
> > setup
> > >>>>>> keystore and generate CSR for system vm: v-25027-VM
> > >>>>>> Line 663111: 2020-07-25 02:23:43,783 ERROR
> > >>>>>> [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-22:ctx-fe0b857b
> > >>>>>> job-1208154/job-1208222 ctx-ef088cd7) (logid:91fc037a) Failed to
> > setup
> > >>>>>> keystore and generate CSR for system vm: v-20771-VM
> > >>>>>> Line 667060: 2020-07-25 02:24:30,221 ERROR
> > >>>>>> [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-23:ctx-662cfe5d
> > >>>>>> job-1208154/job-1208229 ctx-e59878e9) (logid:91fc037a) Failed to
> > setup
> > >>>>>> keystore and generate CSR for system vm: v-24763-VM
> > >>>>>> Line 674269: 2020-07-25 02:25:47,345 ERROR
> > >>>>>> [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-24:ctx-b6529613
> > >>>>>> job-1208154/job-1208233 ctx-7da81cab) (logid:91fc037a) Failed to
> > setup
> > >>>>>> keystore and generate CSR for system vm: v-20773-VM
> > >>>>>> Line 678169: 2020-07-25 02:26:20,355 ERROR
> > >>>>>> [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-31:ctx-7a062a66
> > >>>>>> job-1208154/job-1208244 ctx-4f48d08a) (logid:91fc037a) Failed to
> > setup
> > >>>>>> keystore and generate CSR for system vm: v-25027-VM
> > >>>>>> Line 697113: 2020-07-25 02:30:09,669 ERROR
> > >>>>>> [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-40:ctx-3be9fddc
> > >>>>>> job-1208154/job-1208257 ctx-ed08fc43) (logid:91fc037a) Failed to
> > setup
> > >>>>>> keystore and generate CSR for system vm: v-20771-VM
> > >>>>>> Line 701161: 2020-07-25 02:30:48,127 ERROR
> > >>>>>> [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-41:ctx-4e3c666d
> > >>>>>> job-1208155/job-1208258 ctx-df740f75) (logid:9fa7dece) Failed to
> > setup
> > >>>>>> keystore and generate CSR for system vm: s-24142-VM
> > >>>>>> Line 701838: 2020-07-25 02:30:56,759 ERROR
> > >>>>>> [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-42:ctx-04ac8e71
> > >>>>>> job-1208154/job-1208259 ctx-36c67b5e) (logid:91fc037a) Failed to
> > setup
> > >>>>>> keystore and generate CSR for system vm: v-24763-VM
> > >>>>>> Line 703865: 2020-07-25 02:31:25,123 ERROR
> > >>>>>> [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-46:ctx-24bdbe8f
> > >>>>>> job-1208154/job-1208266 ctx-fa500d3c) (logid:91fc037a) Failed to
> > setup
> > >>>>>> keystore and generate CSR for system vm: v-20773-VM
> > >>>>>> Line 712078: 2020-07-25 02:32:28,666 ERROR
> > >>>>>> [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-48:ctx-4d5db5c2
> > >>>>>> job-1208155/job-1208269 ctx-b0f67c2e) (logid:9fa7dece) Failed to
> > setup
> > >>>>>> keystore and generate CSR for system vm: s-25033-VM
> > >>>>>> Line 715204: 2020-07-25 02:33:02,875 ERROR
> > >>>>>> [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-51:ctx-594da6d2
> > >>>>>> job-1208155/job-1208273 ctx-ca115800) (logid:9fa7dece) Failed to
> > setup
> > >>>>>> keystore and generate CSR for system vm: s-25035-VM
> > >>>>>> Line 743140: 2020-07-25 02:38:52,257 ERROR
> > >>>>>> [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-59:ctx-a3f032da
> > >>>>>> job-1208155/job-1208294 ctx-626a2716) (logid:9fa7dece) Failed to
> > setup
> > >>>>>> keystore and generate CSR for system vm: s-25037-VM
> > >>>>>> Line 817646: 2020-07-25 02:47:07,127 ERROR
> > >>>>>> [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-5:ctx-ca073bd1
> > >>>>>> job-1208297/job-1208305 ctx-6213af3b) (logid:1a2a782c) Failed to
> > setup
> > >>>>>> keystore and generate CSR for system vm: s-25038-VM
> > >>>>>> Line 847150: 2020-07-25 02:51:46,293 ERROR
> > >>>>>> [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-8:ctx-9db8c548
> > >>>>>> job-1208296/job-1208312 ctx-83c570ff) (logid:436d1800) Failed to
> > setup
> > >>>>>> keystore and generate CSR for system vm: v-25040-VM
> > >>>>>> Line 903949: 2020-07-25 02:57:48,515 ERROR
> > >>>>>> [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-4:ctx-f35fd59a
> > >>>>>> job-1208320/job-1208333 ctx-f81179ac) (logid:05eacf0d) Failed to
> > setup
> > >>>>>> keystore and generate CSR for system vm: v-25041-VM
> > >>>>>> Line 957628: 2020-07-25 03:04:13,286 ERROR
> > >>>>>> [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-24:ctx-dc18956a
> > >>>>>> job-1208320/job-1208381 ctx-b53f5d7c) (logid:05eacf0d) Failed to
> > setup
> > >>>>>> keystore and generate CSR for system vm: v-20771-VM
> > >>>>>> Line 961606: 2020-07-25 03:05:00,522 ERROR
> > >>>>>> [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-28:ctx-83e48421
> > >>>>>> job-1208320/job-1208392 ctx-e6767f04) (logid:05eacf0d) Failed to
> > setup
> > >>>>>> keystore and generate CSR for system vm: v-24763-VM
> > >>>>>> Line 966633: 2020-07-25 03:05:29,767 ERROR
> > >>>>>> [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-30:ctx-897e2a66
> > >>>>>> job-1208320/job-1208397 ctx-39c396e1) (logid:05eacf0d) Failed to
> > setup
> > >>>>>> keystore and generate CSR for system vm: v-20773-VM
> > >>>>>> Line 968892: 2020-07-25 03:05:53,648 ERROR
> > >>>>>> [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-36:ctx-68e45b80
> > >>>>>> job-1208320/job-1208405 ctx-1aaf96dd) (logid:05eacf0d) Failed to
> > setup
> > >>>>>> keystore and generate CSR for system vm: v-25041-VM
> > >>>>>> Line 1003036: 2020-07-25 03:13:33,582 ERROR
> > >>>>>> [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-45:ctx-0f552e0e
> > >>>>>> job-1208320/job-1208422 ctx-e8835fe9) (logid:05eacf0d) Failed to
> > setup
> > >>>>>> keystore and generate CSR for system vm: v-20771-VM
> > >>>>>> Line 1006496: 2020-07-25 03:14:09,787 ERROR
> > >>>>>> [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-46:ctx-08e5db24
> > >>>>>> job-1208320/job-1208423 ctx-2cb636c2) (logid:05eacf0d) Failed to
> > setup
> > >>>>>> keystore and generate CSR for system vm: v-24763-VM
> > >>>>>> Line 1008408: 2020-07-25 03:14:39,584 ERROR
> > >>>>>> [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-47:ctx-8b9288a9
> > >>>>>> job-1208320/job-1208424 ctx-c05d1544) (logid:05eacf0d) Failed to
> > setup
> > >>>>>> keystore and generate CSR for system vm: v-20773-VM
> > >>>>>> Line 1023914: 2020-07-25 03:15:58,139 ERROR
> > >>>>>> [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-48:ctx-4d54d717
> > >>>>>> job-1208320/job-1208425 ctx-27e5397d) (logid:05eacf0d) Failed to
> > setup
> > >>>>>> keystore and generate CSR for system vm: v-25046-VM
> > >>>>>> Line 1089568: 2020-07-25 03:24:24,534 ERROR
> > >>>>>> [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-59:ctx-62ff8844
> > >>>>>> job-1208320/job-1208449 ctx-d6a14f7f) (logid:05eacf0d) Failed to
> > setup
> > >>>>>> keystore and generate CSR for system vm: v-25048-VM
> > >>>>>> Line 1097605: 2020-07-25 03:25:23,179 ERROR
> > >>>>>> [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-62:ctx-24407a9e
> > >>>>>> job-1208321/job-1208452 ctx-84f80ffa) (logid:42edff3a) Failed to
> > setup
> > >>>>>> keystore and generate CSR for system vm: s-25050-VM
> > >>>>>>
> > >>>>>> -Ammad Ali
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> On Mon, Aug 17, 2020 at 5:56 PM Andrija Panic <
> > andrija.panic@gmail.com <ma...@gmail.com>>
> > >>>>>> wrote:
> > >>>>>>
> > >>>>>>> Ammad, no such log lines in the file you uploaded....?
> > >>>>>>>
> > >>>>>>> On Mon, 17 Aug 2020 at 09:02, Ammad Syed <syedammad83@gmail.com
> > <ma...@gmail.com>> wrote:
> > >>>>>>>
> > >>>>>>>> Hi Andrija,
> > >>>>>>>>
> > >>>>>>>> The error occurred only one time. Can you please try to grep
> > "Failed to
> > >>>>>>>> setup keystore and generate CSR" you will see all the systemVMs
> > that I
> > >>>>>>>> tried to create failed because of certificate generation failed.
> > OR you
> > >>>>>>> can
> > >>>>>>>> grep job-1208321/job-1208452 this job as well.
> > >>>>>>>>
> > >>>>>>>> Also I mentioned above that same thing is happening in my test
> > >>>>>>> environment
> > >>>>>>>> that I created 4.11.1 then upgraded 4.11.3 > 4.13.1.
> > >>>>>>>>
> > >>>>>>>> Ammad Ali
> > >>>>>>>>
> > >>>>>>>> On Mon, Aug 17, 2020 at 11:17 AM Andrija Panic <
> > >>>>>> andrija.panic@gmail.com <ma...@gmail.com>>
> > >>>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> 2020-07-25 03:23:51,093 WARN [c.c.v.SystemVmLoadScanner]
> > >>>>>>>>> (secstorage-1:ctx-50756e9b) (logid:3d7c25cb) Unexpected
> exception
> > >>>>>>> Failed
> > >>>>>>>> to
> > >>>>>>>>> fetch any free public IP address
> > >>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Failed to
> fetch
> > any
> > >>>>>>> free
> > >>>>>>>>> public IP address
> > >>>>>>>>>
> > >>>>>>>>> ?
> > >>>>>>>>>
> > >>>>>>>>> On Sat, 15 Aug 2020 at 18:05, Ammad Syed <
> syedammad83@gmail.com
> > <ma...@gmail.com>>
> > >>>>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> I have checked further, the key is successfully injected in
> > >>>>>>>> systemvm.iso
> > >>>>>>>>> on
> > >>>>>>>>>> the management node and copied successfully to xenserver. I
> have
> > >>>>>>>> checked
> > >>>>>>>>> on
> > >>>>>>>>>> systemVM there is no public key found in authorized_keys of
> > root.
> > >>>>>>>>>>
> > >>>>>>>>>> How can I troubleshoot this further ? how can I enable trace
> > >>>>>> logging
> > >>>>>>>> for
> > >>>>>>>>>> management server to see if there is something problematic
> > >>>>>> happening.
> > >>>>>>>>>>
> > >>>>>>>>>> Ammad Ali
> > >>>>>>>>>>
> > >>>>>>>>>> On Sat, Aug 15, 2020 at 1:28 PM Ammad Syed <
> > syedammad83@gmail.com <ma...@gmail.com>>
> > >>>>>>>>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>>> Hi,
> > >>>>>>>>>>>
> > >>>>>>>>>>> I have setup my test environment, here is what I did:
> > >>>>>>>>>>>
> > >>>>>>>>>>> - Installed ACS 4.11.1 and added xenserver 7.0 host in it.
> > >>>>>>> SystemVMs
> > >>>>>>>>> are
> > >>>>>>>>>>> up in the zone and agents are up.
> > >>>>>>>>>>> - Then upgraded the system to 4.11.3. Recreated systemVMs,
> > agent
> > >>>>>> is
> > >>>>>>>> up
> > >>>>>>>>>> and
> > >>>>>>>>>>> systemVM are up with 4.11.3 systemVM version.
> > >>>>>>>>>>> - Then upgraded the system to 4.13.1, recreated systemVMs are
> > >>>>>>> running
> > >>>>>>>>> but
> > >>>>>>>>>>> agents are not up.
> > >>>>>>>>>>>
> > >>>>>>>>>>> I have checked md5sum of systemvm.iso on xenserver and
> > management
> > >>>>>>>>> server,
> > >>>>>>>>>>> both are same.
> > >>>>>>>>>>>
> > >>>>>>>>>>> [root@xenserver iso]# md5sum
> > >>>>>>>> /opt/xensource/packages/iso/systemvm.iso
> > >>>>>>>>>>> baba18f156395da3a5d8208727d8f421
> > >>>>>>>>>> /opt/xensource/packages/iso/systemvm.iso
> > >>>>>>>>>>>
> > >>>>>>>>>>> [root@cloudstack-upgrade vms]# md5sum
> > >>>>>>>>>>> /usr/share/cloudstack-common/vms/systemvm.iso
> > >>>>>>>>>>> baba18f156395da3a5d8208727d8f421
> > >>>>>>>>>>> /usr/share/cloudstack-common/vms/systemvm.iso
> > >>>>>>>>>>>
> > >>>>>>>>>>> Also the private key on xenserver root and management server
> > are
> > >>>>>>>> same.
> > >>>>>>>>>>>
> > >>>>>>>>>>> management server
> > >>>>>>>>>>> /usr/share/cloudstack-common/scripts/vm/systemvm/id_rsa.cloud
> > >>>>>>>>>>>
> > >>>>>>>>>>> xenserver path
> > >>>>>>>>>>> /root/.ssh/id_rsa.cloud
> > >>>>>>>>>>>
> > >>>>>>>>>>> - Ammad Ali
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Thu, Aug 13, 2020 at 11:27 AM Ammad Syed <
> > >>>>>> syedammad83@gmail.com <ma...@gmail.com>
> > >>>>>>>>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Here is the link for download management logs.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> >
> https://drive.google.com/file/d/1l6HDPguGUNaOxsc7VSaj7eaP0F2UOFjA/view?usp=sharing
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On Thu, Aug 13, 2020 at 11:22 AM Ammad Syed <
> > >>>>>>> syedammad83@gmail.com>
> > >>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> I have reverted the version back to 4.11.3. But I have
> saved
> > >>>>>> logs
> > >>>>>>>>>>>>> starting from upgrade.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> I think the key has been copied successfully in system vm
> > iso.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> 2020-07-25 00:34:17,214 INFO
> [c.c.s.ConfigurationServerImpl]
> > >>>>>>>>>>>>> (main:null) (logid:) Going to update systemvm iso with
> > >>>>>> generated
> > >>>>>>>>>> keypairs
> > >>>>>>>>>>>>> if needed
> > >>>>>>>>>>>>> 2020-07-25 00:34:17,214 INFO
> [c.c.s.ConfigurationServerImpl]
> > >>>>>>>>>>>>> (main:null) (logid:) Trying to inject public and private
> keys
> > >>>>>>> into
> > >>>>>>>>>> systemvm
> > >>>>>>>>>>>>> iso
> > >>>>>>>>>>>>> 2020-07-25 00:34:17,217 DEBUG [c.c.u.s.Script] (main:null)
> > >>>>>>> (logid:)
> > >>>>>>>>>>>>> Looking for vms/systemvm.iso in the classpath
> > >>>>>>>>>>>>> 2020-07-25 00:34:17,217 DEBUG [c.c.u.s.Script] (main:null)
> > >>>>>>> (logid:)
> > >>>>>>>>>>>>> System resource:
> > >>>>>>> file:/usr/share/cloudstack-common/vms/systemvm.iso
> > >>>>>>>>>>>>> 2020-07-25 00:34:17,217 DEBUG [c.c.u.s.Script] (main:null)
> > >>>>>>> (logid:)
> > >>>>>>>>>>>>> Absolute path =
> > /usr/share/cloudstack-common/vms/systemvm.iso
> > >>>>>>>>>>>>> 2020-07-25 00:34:17,218 DEBUG
> [c.c.s.ConfigurationServerImpl]
> > >>>>>>>>>>>>> (main:null) (logid:) Executing: /bin/bash
> > >>>>>>>>>>>>>
> > /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh
> > >>>>>>>>>>>>> /var/cloudstack/management/.ssh/id_rsa.pub
> > >>>>>>>>>>>>> /var/cloudstack/management/.ssh/id_rsa
> > >>>>>>>>>>>>> /usr/share/cloudstack-common/vms/systemvm.iso
> > >>>>>>>>>>>>> 2020-07-25 00:34:17,636 INFO
> [c.c.s.ConfigurationServerImpl]
> > >>>>>>>>>>>>> (main:null) (logid:) Injected public and private keys into
> > >>>>>>> systemvm
> > >>>>>>>>> iso
> > >>>>>>>>>>>>> with result : null
> > >>>>>>>>>>>>> 2020-07-25 00:34:50,613 DEBUG
> [c.c.h.x.r.CitrixResourceBase]
> > >>>>>>>>>>>>> (DirectAgent-1:ctx-d3dc4cf2) (logid:1d22396d) Copying
> > >>>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> >
> /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/xenserver65/../../../../../vms/systemvm.iso
> > >>>>>>>>>>>>> to /opt/xensource/packages/iso on 172.16.2.22 with
> permission
> > >>>>>>> 0644
> > >>>>>>>>>>>>> 2020-07-25 00:34:52,566 DEBUG
> [c.c.h.x.r.CitrixResourceBase]
> > >>>>>>>>>>>>> (DirectAgent-2:ctx-2537b610) (logid:29c67b7a) Copying
> > >>>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> >
> /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/xenserver65/../../../../../vms/systemvm.iso
> > >>>>>>>>>>>>> to /opt/xensource/packages/iso on 172.16.2.5 with
> permission
> > >>>>>> 0644
> > >>>>>>>>>>>>> 2020-07-25 00:34:53,170 DEBUG
> [c.c.h.x.r.CitrixResourceBase]
> > >>>>>>>>>>>>> (DirectAgent-3:ctx-168ac27d) (logid:a7862c4b) Copying
> > >>>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> >
> /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/xenserver65/../../../../../vms/systemvm.iso
> > >>>>>>>>>>>>> to /opt/xensource/packages/iso on 172.16.2.9 with
> permission
> > >>>>>> 0644
> > >>>>>>>>>>>>> 2020-07-25 00:34:54,621 DEBUG
> [c.c.h.x.r.CitrixResourceBase]
> > >>>>>>>>>>>>> (DirectAgent-6:ctx-f640fe55) (logid:0a62d4cf) Copying
> > >>>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> >
> /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/xenserver65/../../../../../vms/systemvm.iso
> > >>>>>>>>>>>>> to /opt/xensource/packages/iso on 172.16.2.5 with
> permission
> > >>>>>> 0644
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> I have attached complete management logs. The start of logs
> > is
> > >>>>>>> the
> > >>>>>>>>>> start
> > >>>>>>>>>>>>> of management server after upgrade of management server
> from
> > >>>>>>> 4.11.3
> > >>>>>>>>> to
> > >>>>>>>>>>>>> 4.13.1.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Ammad Ali
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> On Thu, Aug 13, 2020 at 1:35 AM Andrija Panic <
> > >>>>>>>>> andrija.panic@gmail.com
> > >>>>>>>>>>>
> > >>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Do you get an error while trying to inject ssh key into
> the
> > >>>>>>>>>> systemvm.iso
> > >>>>>>>>>>>>>> (mgmt logs) , or can you confirm that the systemvm.iso on
> XS
> > >>>>>>> host
> > >>>>>>>>> and
> > >>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>> one on the mgmt server are identical, md5sum them (i.e.
> has
> > >>>>>> the
> > >>>>>>>> iso
> > >>>>>>>>>> been
> > >>>>>>>>>>>>>> copied over to the XS successfully) - this might explain
> not
> > >>>>>>> being
> > >>>>>>>>>> able
> > >>>>>>>>>>>>>> to
> > >>>>>>>>>>>>>> login via ssh with your private key.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Best,
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> On Wed, 12 Aug 2020, 09:08 Ammad Syed, <
> > syedammad83@gmail.com
> > >>>>>>>
> > >>>>>>>>> wrote:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Yes this is exactly the same issue that i faced.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Sent from my iPhone
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> On 12-Aug-2020, at 8:35 AM, Eric Lee Green <
> > >>>>>>>>>>>>>> eric.lee.green@gmail.com>
> > >>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Correct, 4.11.3 template is used for 4.11.3, 4.12, and
> > >>>>>>> 4.13.
> > >>>>>>>>> 4.14
> > >>>>>>>>>>>>>> moves
> > >>>>>>>>>>>>>>> to the 4.14.0 template.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> There seems to be something odd happening key-wise
> > >>>>>> sometimes
> > >>>>>>>>> with
> > >>>>>>>>>>>>>>> upgrades from 4.11.3 to 4.13.1 or 4.14.0. I managed an
> > >>>>>>> upgrade
> > >>>>>>>>>> from
> > >>>>>>>>>>>>>>> 4.11.3 to 4.13.1 that *almost* worked, but the secondary
> > >>>>>>> storage
> > >>>>>>>>> VM
> > >>>>>>>>>>>>>>> wouldn't work and thus I couldn't spawn new virtual
> > >>>>>> machines.
> > >>>>>>>> Same
> > >>>>>>>>>>>>>> symptom
> > >>>>>>>>>>>>>>> -- key error when the agent tried to ssh into it. And
> > >>>>>> deleting
> > >>>>>>>> it
> > >>>>>>>>>> and
> > >>>>>>>>>>>>>>> making it respawn didn't help. Then I tried 4.11.3 to
> > 4.14.0
> > >>>>>>> and
> > >>>>>>>>>>>>>> *all* the
> > >>>>>>>>>>>>>>> VM's failed at that point (of course, that was with the
> new
> > >>>>>>>>>> template).
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Right now I'm back at 4.11.3 until this can be figured
> > >>>>>> out.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> On 8/11/2020 5:53 AM, Ammad Syed wrote:
> > >>>>>>>>>>>>>>>>> Hi,
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> I think 4.12 and 4.13 uses same systemVM template i.e
> > >>>>>>> 4.11.3
> > >>>>>>>>>>>>>> version,
> > >>>>>>>>>>>>>>> which
> > >>>>>>>>>>>>>>>>> I already have registered. Currently I am running
> 4.11.3
> > >>>>>>>>> version
> > >>>>>>>>>>>>>> of ACS.
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> MariaDB [cloud]> SELECT id,name,type,cross_zones,state
> > >>>>>> FROM
> > >>>>>>>>>>>>>>>>> cloud.vm_template WHERE name like
> '%systemvm-xenserver%'
> > >>>>>>> AND
> > >>>>>>>>>>>>>> removed IS
> > >>>>>>>>>>>>>>>>> NULL;
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > +------+-----------------------------+--------+-------------+----------+
> > >>>>>>>>>>>>>>>>> | id | name | type |
> > >>>>>>> cross_zones |
> > >>>>>>>>>>>>>> state |
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > +------+-----------------------------+--------+-------------+----------+
> > >>>>>>>>>>>>>>>>> | 337 | systemvm-xenserver-3.0.0 | SYSTEM |
> > >>>>>>> 0 |
> > >>>>>>>>>>>>>> Inactive |
> > >>>>>>>>>>>>>>>>> | 418 | systemvm-xenserver-4.2 | SYSTEM |
> > >>>>>>> 0 |
> > >>>>>>>>>>>>>> Active |
> > >>>>>>>>>>>>>>>>> | 472 | systemvm-xenserver-4.3 | USER |
> > >>>>>>> 1 |
> > >>>>>>>>>>>>>> Inactive |
> > >>>>>>>>>>>>>>>>> | 473 | systemvm-xenserver-4.3 | USER |
> > >>>>>>> 1 |
> > >>>>>>>>>>>>>> Inactive |
> > >>>>>>>>>>>>>>>>> | 474 | systemvm-xenserver-4.3 | USER |
> > >>>>>>> 1 |
> > >>>>>>>>>>>>>> Inactive |
> > >>>>>>>>>>>>>>>>> | 475 | systemvm-xenserver-4.3 | USER |
> > >>>>>>> 1 |
> > >>>>>>>>>>>>>> Inactive |
> > >>>>>>>>>>>>>>>>> | 476 | systemvm-xenserver-4.3 | USER |
> > >>>>>>> 0 |
> > >>>>>>>>>>>>>> Inactive |
> > >>>>>>>>>>>>>>>>> | 479 | systemvm-xenserver-4.3-2 | USER |
> > >>>>>>> 1 |
> > >>>>>>>>>>>>>> Inactive |
> > >>>>>>>>>>>>>>>>> | 480 | systemvm-xenserver-4.3 | SYSTEM |
> > >>>>>>> 0 |
> > >>>>>>>>>>>>>> Active |
> > >>>>>>>>>>>>>>>>> | 549 | systemvm-xenserver-4.5.1 | USER |
> > >>>>>>> 0 |
> > >>>>>>>>>>>>>> Active |
> > >>>>>>>>>>>>>>>>> | 550 | systemvm-xenserver-4.5.1 | SYSTEM |
> > >>>>>>> 0 |
> > >>>>>>>>>>>>>> Active |
> > >>>>>>>>>>>>>>>>> | 651 | systemvm-xenserver-4.7.0 | USER |
> > >>>>>>> 0 |
> > >>>>>>>>>>>>>> Inactive |
> > >>>>>>>>>>>>>>>>> | 652 | systemvm-xenserver-4.7.0 | USER |
> > >>>>>>> 0 |
> > >>>>>>>>>>>>>> Inactive |
> > >>>>>>>>>>>>>>>>> | 653 | systemvm-xenserver-4.7.0 | SYSTEM |
> > >>>>>>> 0 |
> > >>>>>>>>>>>>>> Inactive |
> > >>>>>>>>>>>>>>>>> | 737 | systemvm-xenserver-4.9.2 | SYSTEM |
> > >>>>>>> 1 |
> > >>>>>>>>>>>>>> Inactive |
> > >>>>>>>>>>>>>>>>> | 739 | systemvm-xenserver-4.9.2-sb | SYSTEM |
> > >>>>>>> 1 |
> > >>>>>>>>>>>>>> Active |
> > >>>>>>>>>>>>>>>>> | 1245 | systemvm-xenserver-4.11.1 | SYSTEM |
> > >>>>>>> 1 |
> > >>>>>>>>>>>>>> Active |
> > >>>>>>>>>>>>>>>>> | 1584 | systemvm-xenserver-4.11.2 | SYSTEM |
> > >>>>>>> 1 |
> > >>>>>>>>>>>>>> Active |
> > >>>>>>>>>>>>>>>>> | 1677 | systemvm-xenserver-4.11.3 | SYSTEM |
> > >>>>>>> 1 |
> > >>>>>>>>>>>>>> Active |
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > +------+-----------------------------+--------+-------------+----------+
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> - Ammad
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> On Tue, Aug 11, 2020 at 5:17 PM Pierre-Luc Dion <
> > >>>>>>>>>>>>>> pdion891@apache.org>
> > >>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>> db.
> > >>>>>>>>>>>>>>>>>> Hi Syed,
> > >>>>>>>>>>>>>>>>>> From 4.12, the systemvm template had to be upgraded
> > >>>>>>> because
> > >>>>>>>> of
> > >>>>>>>>>> OS
> > >>>>>>>>>>>>>>> change in
> > >>>>>>>>>>>>>>>>>> the template, moved to a latest version of Debian.
> > >>>>>> Because
> > >>>>>>>> of
> > >>>>>>>>>>>>>> that,
> > >>>>>>>>>>>>>>> some VR
> > >>>>>>>>>>>>>>>>>> scripts have changed and make obsolete older version
> of
> > >>>>>>> VRs,
> > >>>>>>>>> so
> > >>>>>>>>>>>>>> you
> > >>>>>>>>>>>>>>> will
> > >>>>>>>>>>>>>>>>>> most likely have to register an updated systemvm
> > >>>>>> templates
> > >>>>>>>> and
> > >>>>>>>>>>>>>> upgrade
> > >>>>>>>>>>>>>>> your
> > >>>>>>>>>>>>>>>>>> system VMs and VRs.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Regards,
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> On Tue, Aug 11, 2020 at 6:24 AM Ammad Syed <
> > >>>>>>>>>>>>>> syedammad83@gmail.com>
> > >>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Hi Guys,
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> I was previously on 4.9.3 cloudstack and upgraded to
> > >>>>>>> 4.11.1
> > >>>>>>>>>> then
> > >>>>>>>>>>>>>>> 4.11.3.
> > >>>>>>>>>>>>>>>>>>> The version 4.11.3 is working fine since six months.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Now I have tried to upgrade my system from 4.11.3 to
> > >>>>>>>> 4.13.1.
> > >>>>>>>>>> The
> > >>>>>>>>>>>>>>> upgrade
> > >>>>>>>>>>>>>>>>>>> goes successful. I didn't uploaded any system VM
> > >>>>>>> template.
> > >>>>>>>>>>>>>> However the
> > >>>>>>>>>>>>>>>>>>> problem occured when I recreated my systemVM of POD,
> > >>>>>> the
> > >>>>>>> VM
> > >>>>>>>>>>>>>> recreated
> > >>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>> its state was running but agent state was not getting
> > >>>>>> up,
> > >>>>>>>> its
> > >>>>>>>>>>>>>> showing
> > >>>>>>>>>>>>>>>>>> blank
> > >>>>>>>>>>>>>>>>>>> in column.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Digging further via job logs, the job is failed with
> > >>>>>>> error
> > >>>>>>>>> that
> > >>>>>>>>>>>>>>> unable to
> > >>>>>>>>>>>>>>>>>>> execute command via ssh. Below are the logs.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> 2020-07-25 02:30:48,126 ERROR [c.c.u.s.SshHelper]
> > >>>>>>>>>>>>>>>>>>> (DirectAgent-211:ctx-62f09b31) (logid:9fa7dece) SSH
> > >>>>>>>> execution
> > >>>>>>>>>> of
> > >>>>>>>>>>>>>>> command
> > >>>>>>>>>>>>>>>>>>> /opt/cloud/bin/router_proxy.sh keystore-s
> > >>>>>>>>>>>>>>>>>>> etup 169.254.2.199
> > >>>>>>>>>>>>>> /usr/local/cloud/systemvm/conf/agent.properties
> > >>>>>>>>>>>>>>>>>>> /usr/local/cloud/systemvm/conf/cloud.jks
> > >>>>>> TJaQYChYBwKh7Cx9
> > >>>>>>>> 365
> > >>>>>>>>>>>>>>>>>>> /usr/local/cloud/systemvm/conf/clou
> > >>>>>>>>>>>>>>>>>>> d.csr has an error status code in return. Result
> > >>>>>> output:
> > >>>>>>>>>>>>>>>>>>> 2020-07-25 02:30:48,127 DEBUG
> > >>>>>>> [c.c.a.m.DirectAgentAttache]
> > >>>>>>>>>>>>>>>>>>> (DirectAgent-211:ctx-62f09b31) (logid:9fa7dece) Seq
> > >>>>>>>>>>>>>>>>>>> 906-3195585410596077730: Response Received:
> > >>>>>>>>>>>>>>>>>>> 2020-07-25 02:30:48,127 DEBUG [c.c.a.t.Request]
> > >>>>>>>>>>>>>>>>>>> (DirectAgent-211:ctx-62f09b31) (logid:9fa7dece) Seq
> > >>>>>>>>>>>>>>>>>>> 906-3195585410596077730: Processing: { Ans: ,
> MgmtId:
> > >>>>>>>>>> 779271079
> > >>>>>>>>>>>>>>>>>>> 43497, via: 906(xen-21-10-a3-khi02), Ver: v1, Flags:
> > >>>>>> 10,
> > >>>>>>>>>>>>>>>>>>> [{"org.apache.cloudstack.ca
> > >>>>>>>>>>>>>>>>>>> .SetupKeystoreAnswer":{"result":false,"wait":0}}]
> > >>>>>>>>>>>>>>>>>>> }
> > >>>>>>>>>>>>>>>>>>> 2020-07-25 02:30:48,127 DEBUG [c.c.a.t.Request]
> > >>>>>>>>>>>>>>>>>>> (Work-Job-Executor-41:ctx-4e3c666d
> > >>>>>>> job-1208155/job-1208258
> > >>>>>>>>>>>>>>> ctx-df740f75)
> > >>>>>>>>>>>>>>>>>>> (logid:9fa7dece) Seq 906-319558541059607773
> > >>>>>>>>>>>>>>>>>>> 0: Received: { Ans: , MgmtId: 77927107943497, via:
> > >>>>>>>>>>>>>>>>>>> 906(xen-21-10-a3-khi02), Ver: v1, Flags: 10, {
> > >>>>>>>>>>>>>> SetupKeystoreAnswer } }
> > >>>>>>>>>>>>>>>>>>> 2020-07-25 02:30:48,127 ERROR
> > >>>>>>>>> [c.c.v.VirtualMachineManagerImpl]
> > >>>>>>>>>>>>>>>>>>> (Work-Job-Executor-41:ctx-4e3c666d
> > >>>>>>> job-1208155/job-1208258
> > >>>>>>>>>>>>>>> ctx-df740f75)
> > >>>>>>>>>>>>>>>>>>> (logid:9fa7dece) Failed to setup keystore and
> generate
> > >>>>>>> CSR
> > >>>>>>>>> for
> > >>>>>>>>>>>>>> system
> > >>>>>>>>>>>>>>> vm:
> > >>>>>>>>>>>>>>>>>>> s-24142-VM
> > >>>>>>>>>>>>>>>>>>> 2020-07-25 02:30:48,127 DEBUG
> > >>>>>>> [c.c.v.VmWorkJobHandlerProxy]
> > >>>>>>>>>>>>>>>>>>> (Work-Job-Executor-41:ctx-4e3c666d
> > >>>>>>> job-1208155/job-1208258
> > >>>>>>>>>>>>>>> ctx-df740f75)
> > >>>>>>>>>>>>>>>>>>> (logid:9fa7dece) Done executing VM work job:
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> >
> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":24142,"handlerName":"VirtualMachineManagerImpl"}
> > >>>>>>>>>>>>>>>>>>> 2020-07-25 02:30:48,128 DEBUG
> > >>>>>>>>> [o.a.c.f.j.i.AsyncJobManagerImpl]
> > >>>>>>>>>>>>>>>>>>> (Work-Job-Executor-41:ctx-4e3c666d
> > >>>>>>> job-1208155/job-1208258
> > >>>>>>>>>>>>>>> ctx-df740f75)
> > >>>>>>>>>>>>>>>>>>> (logid:9fa7dece) Complete async job-1208258,
> jobStatus:
> > >>>>>>>>>>>>>> SUCCEEDED,
> > >>>>>>>>>>>>>>>>>>> resultCode: 0, result: null
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> I tried to dig it further, I was unable to login
> > >>>>>> systemVM
> > >>>>>>>> via
> > >>>>>>>>>>>>>> ssh from
> > >>>>>>>>>>>>>>>>>>> xenserver host with key /root/.ssh/id_rsa.cloud
> placed.
> > >>>>>>>> Look
> > >>>>>>>>>> like
> > >>>>>>>>>>>>>>> private
> > >>>>>>>>>>>>>>>>>>> key issue. However I am able to login on my old
> > >>>>>>> systemVMs (
> > >>>>>>>>> i.e
> > >>>>>>>>>>>>>>> created
> > >>>>>>>>>>>>>>>>>> on
> > >>>>>>>>>>>>>>>>>>> ACS 4.11.3)
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Also I have SSL certificate enabled for console proxy
> > >>>>>> on
> > >>>>>>> my
> > >>>>>>>>> ACS
> > >>>>>>>>>>>>>> 4.11.3
> > >>>>>>>>>>>>>>>>>> and
> > >>>>>>>>>>>>>>>>>>> I am using only xenserver 7.0 hosts.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> I tried to disable SSL on secstorage and console
> proxy
> > >>>>>>> from
> > >>>>>>>>>>>>>> global
> > >>>>>>>>>>>>>>>>>>> settings, but still didn't worked.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> I had a fresh installation of ACS 4.13.1 with
> xenserver
> > >>>>>>>> 7.0,
> > >>>>>>>>>>>>>> systemVMs
> > >>>>>>>>>>>>>>>>>> are
> > >>>>>>>>>>>>>>>>>>> working fine in it.
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Please advise.
> > >>>>>>>>>>>>>>>>>>> --
> > >>>>>>>>>>>>>>>>>>> Regards,
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>> Syed Ammad Ali
> > >>>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> --
> > >>>>>>>>>>>>> Regards,
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Syed Ammad Ali
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> --
> > >>>>>>>>>>>> Regards,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Syed Ammad Ali
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> --
> > >>>>>>>>>>> Regards,
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> Syed Ammad Ali
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> --
> > >>>>>>>>>> Regards,
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> Syed Ammad Ali
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> --
> > >>>>>>>>>
> > >>>>>>>>> Andrija Panić
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> --
> > >>>>>>>> Regards,
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> Syed Ammad Ali
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> --
> > >>>>>>>
> > >>>>>>> Andrija Panić
> > >>>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> --
> > >>>>>> Regards,
> > >>>>>>
> > >>>>>>
> > >>>>>> Syed Ammad Ali
> > >>>>>>
> > >>>>>
> > >>>>>
> > >>>>> --
> > >>>>>
> > >>>>> Andrija Panić
> > >>>>
> > >>>>
> > >>>> --
> > >>>> Regards,
> > >>>>
> > >>>>
> > >>>> Syed Ammad Ali
> > >>
> > >
> >
>
>
> --
>
> Andrija Panić
>
--
Regards,
Syed Ammad Ali
Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error
Posted by Andrija Panic <an...@gmail.com>.
Ammad,
please check what Pearl has suggested, but I think the correct syntax
(different id_rsa,pub, specifically the one used by ACS) should be as
following (you can always grep your older logs for "injectkey.sh" :
/bin/bash /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh
/var/cloudstack/management/.ssh/id_rsa.pub
/var/cloudstack/management/.ssh/id_rsa
/usr/share/cloudstack-common/vms/systemvm.iso
You should get e meaningful output if the key was injected or not (perhaps
the key inside doesn't differer, so the script will not inject it again)
Proceed with what Pearl has suggested as the next steps.
Best,
On Thu, 3 Sep 2020 at 06:16, Pearl d'Silva <pe...@shapeblue.com>
wrote:
> Hi Ammad,
>
> Could you try re-injecting the keys into the systemVM, as this MAY happen
> due to stale id_rsa keys. Run the injectkeys.sh to inject the existing
> public ssh key into the authorized_keys file inside the systemvm.iso.
> Execute:
>
>
> /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh
> /root/.ssh/id_rsa.pub /root/.ssh/id_rsa
> /usr/share/cloudstack-common/vms/systemvm.iso'
>
> * Then replace the systemvm.iso on the hypervisor hosts. For each host:
>
> 1. Login and type command - xe host-param-clear param-name=tags
> uuid=<uuid of the XS host>
> host uuid maybe obtained using: xe host-list
>
> 2. 2. Restart MS and the iso should get propagated for all. Check
> timestamp.
> 3. 3. Restart systemvms and they should get the iso inserted.
> 4.
>
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/SystemVm.iso#SystemVm.iso-Xen
>
> Thanks,
> Pearl
>
> ________________________________
> From: Ammad Syed <sy...@gmail.com>
> Sent: Wednesday, September 2, 2020 9:12 PM
> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> Subject: Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error
>
> Guys, please advise how to troubleshoot this further ? How can i enable
> trace level debugging ? any help in this will be highly appreciated?
>
> Ammad
>
>
> pearl.dsilva@shapeblue.com
> www.shapeblue.com
> 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
> @shapeblue
>
>
>
> > On 27-Aug-2020, at 1:41 PM, Vivek Kumar <vi...@indiqus.com.invalid>
> wrote:
> >
> > Pardon, the whole mail chain was not visible so replied accordingly. It
> seems like you are doing SSH using root shell so it doesn’t matter.
> >
> > Vivek Kumar
> > Manager - Cloud & DevOps
> > IndiQus Technologies
> > 24*7 O +91 11 4055 1411 | M +91 7503460090
> > www.indiqus.com<http://www.indiqus.com> <http://indiqus.com/>
> >
> > This message is intended only for the use of the individual or entity to
> which it is addressed and may contain information that is confidential
> and/or privileged. If you are not the intended recipient please delete the
> original message and any copy of it from your computer system. You are
> hereby notified that any dissemination, distribution or copying of this
> communication is strictly prohibited unless proper authorization has been
> obtained for such action. If you have received this communication in error,
> please notify the sender immediately. Although IndiQus attempts to sweep
> e-mail and attachments for viruses, it does not guarantee that both are
> virus-free and accepts no liability for any damage sustained as a result of
> viruses.
> >
> >> On 27-Aug-2020, at 2:03 PM, Vivek Kumar <vi...@indiqus.com>
> wrote:
> >>
> >> Hello Ammad,
> >>
> >> You haven’t defined the user while logging to the system VM through the
> Link local IP that’s why you have got the error while logging to the system
> VM.
> >>
> >>>> ssh -i /root/.ssh/id_rsa.cloud 169.254.0.121 -p 3922
> >>
> >> Try instead below
> >>
> >>>> ssh -i /root/.ssh/id_rsa.cloud -p 3922 root@169.254.0.121 <mailto:
> root@169.254.0.121>
> >>
> >> Vivek Kumar
> >> Manager - Cloud & DevOps
> >> IndiQus Technologies
> >> 24*7 O +91 11 4055 1411 | M +91 7503460090
> >> www.indiqus.com<http://www.indiqus.com> <http://indiqus.com/>
> >>
> >> This message is intended only for the use of the individual or entity
> to which it is addressed and may contain information that is confidential
> and/or privileged. If you are not the intended recipient please delete the
> original message and any copy of it from your computer system. You are
> hereby notified that any dissemination, distribution or copying of this
> communication is strictly prohibited unless proper authorization has been
> obtained for such action. If you have received this communication in error,
> please notify the sender immediately. Although IndiQus attempts to sweep
> e-mail and attachments for viruses, it does not guarantee that both are
> virus-free and accepts no liability for any damage sustained as a result of
> viruses.
> >>
> >>>> On 27-Aug-2020, at 1:59 PM, Ammad Syed <syedammad83@gmail.com
> <ma...@gmail.com>> wrote:
> >>>
> >>> Guys, any help to troubleshoot this further would be highly
> appreciated.
> >>>
> >>>
> >>> Ammad Ali
> >>>> On 18-Aug-2020, at 1:05 PM, Ammad Syed <syedammad83@gmail.com
> <ma...@gmail.com>> wrote:
> >>>>
> >>>>
> >>>> Hi,
> >>>>
> >>>> The systemVM is accessible with 169.x.x.x IP from xenserver host. But
> login with private key is denied. I am testing this on my test environment.
> >>>>
> >>>> [root@xenserver ~]# ping 169.254.0.121
> >>>> PING 169.254.0.121 (169.254.0.121) 56(84) bytes of data.
> >>>> 64 bytes from 169.254.0.121: icmp_seq=1 ttl=64 time=0.328 ms
> >>>> 64 bytes from 169.254.0.121: icmp_seq=2 ttl=64 time=0.124 ms
> >>>> 64 bytes from 169.254.0.121: icmp_seq=3 ttl=64 time=0.111 ms
> >>>> 64 bytes from 169.254.0.121: icmp_seq=4 ttl=64 time=0.126 ms
> >>>> ^C
> >>>> --- 169.254.0.121 ping statistics ---
> >>>> 4 packets transmitted, 4 received, 0% packet loss, time 2997ms
> >>>> rtt min/avg/max/mdev = 0.111/0.172/0.328/0.090 ms
> >>>> [root@xenserver ~]#
> >>>> [root@xenserver ~]#
> >>>> [root@xenserver ~]# ssh -i /root/.ssh/id_rsa.cloud 169.254.0.121 -p
> 3922
> >>>> Permission denied (publickey).
> >>>>
> >>>> I have checked by logging into the systemVM, the public key is not
> found in /root/.ssh/authorized_keys.
> >>>>
> >>>> I have digged further, the systemvm.iso is same on the management
> node and xenserver host. I have checked by mounting the ISO as well, the
> public key is there in authorized_keys in ISO.
> >>>>
> >>>> Restarting / recreating systemVM isn't fixed the issue.
> >>>>
> >>>> The systemVM is already running on HVM mode on xenserver host as the
> template OS type is Debian 8 64bit is selected.
> >>>>
> >>>> On my PRD upgrade, I have tested on one zone only. Currently I am
> testing on my test ACS environment which have only one zone.
> >>>>
> >>>> Ammad Ali
> >>>>
> >>>>> On Mon, Aug 17, 2020 at 11:10 PM Andrija Panic <
> andrija.panic@gmail.com <ma...@gmail.com>> wrote:
> >>>>> It was my tool (GUI based tool, that still doesn't return any line
> (???)
> >>>>> but I see it.
> >>>>>
> >>>>> Problem is with 169.254.1.178 not being reachable over SSH
> (2020-07-25
> >>>>> 01:45:52,097 DEBUG [c.c.h.x.r.CitrixResourceBase]
> >>>>> (DirectAgent-57:ctx-ded7f7f9) (logid:a75971c2) Executing command in
> VR:
> >>>>> /opt/cloud/bin/router_proxy.sh keystore-setup 169.254.1.178
> >>>>> /usr/local/cloud/systemvm/conf/agent.properties
> >>>>> /usr/local/cloud/systemvm/conf/cloud.jks Xe8zzqZmUp9wmABH 365
> >>>>> /usr/local/cloud/systemvm/conf/cloud.csr)
> >>>>>
> >>>>> Can you try to ssh from that particular XenServer (where the SSVM is
> >>>>> running) to the IP 169.xxxxx on port 3922 (ssh port) using the
> private key
> >>>>> - if you log in to the VM via it's console (root/password) - do you
> see
> >>>>> with ifconfig the interface being started with 169..xxxx IP address?
> I
> >>>>> assume that destroying the SSVM and starting a new one doesn't fix
> the
> >>>>> issue?
> >>>>>
> >>>>> You can also try changing the OS type of the systemVM template to
> Other
> >>>>> Linux 64bit, so it runs in HVM mode (instead of PV) - you'll need to
> see ID
> >>>>> for that OS type from the guest_os table, and set the guest_os_id
> field in
> >>>>> the vm_template table for your systemVM template.
> >>>>>
> >>>>> Does the issue occur in other zones as well? (assuming same
> hypervisor type)
> >>>>>
> >>>>>
> >>>>> On Mon, 17 Aug 2020 at 15:31, Ammad Syed <syedammad83@gmail.com
> <ma...@gmail.com>> wrote:
> >>>>>
> >>>>>> I have rechecked the logs are there. Please review below lines.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> C:\Users\ammad\Downloads\management-server.log-4.13\management-server.log-4.13.1-errors
> >>>>>> (27 hits)
> >>>>>> Line 409969: 2020-07-25 01:45:52,320 ERROR
> >>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-58:ctx-9dc75334
> >>>>>> job-1207969/job-1208104 ctx-09e752ec) (logid:a75971c2) Failed to
> setup
> >>>>>> keystore and generate CSR for system vm: s-25023-VM
> >>>>>> Line 437413: 2020-07-25 01:51:33,845 ERROR
> >>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-70:ctx-457ce4ee
> >>>>>> job-1207969/job-1208129 ctx-195da69e) (logid:a75971c2) Failed to
> setup
> >>>>>> keystore and generate CSR for system vm: s-25024-VM
> >>>>>> Line 549538: 2020-07-25 02:06:44,833 ERROR
> >>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-3:ctx-4634f7e0
> >>>>>> job-1208154/job-1208161 ctx-05d065dc) (logid:91fc037a) Failed to
> setup
> >>>>>> keystore and generate CSR for system vm: v-25027-VM
> >>>>>> Line 663111: 2020-07-25 02:23:43,783 ERROR
> >>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-22:ctx-fe0b857b
> >>>>>> job-1208154/job-1208222 ctx-ef088cd7) (logid:91fc037a) Failed to
> setup
> >>>>>> keystore and generate CSR for system vm: v-20771-VM
> >>>>>> Line 667060: 2020-07-25 02:24:30,221 ERROR
> >>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-23:ctx-662cfe5d
> >>>>>> job-1208154/job-1208229 ctx-e59878e9) (logid:91fc037a) Failed to
> setup
> >>>>>> keystore and generate CSR for system vm: v-24763-VM
> >>>>>> Line 674269: 2020-07-25 02:25:47,345 ERROR
> >>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-24:ctx-b6529613
> >>>>>> job-1208154/job-1208233 ctx-7da81cab) (logid:91fc037a) Failed to
> setup
> >>>>>> keystore and generate CSR for system vm: v-20773-VM
> >>>>>> Line 678169: 2020-07-25 02:26:20,355 ERROR
> >>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-31:ctx-7a062a66
> >>>>>> job-1208154/job-1208244 ctx-4f48d08a) (logid:91fc037a) Failed to
> setup
> >>>>>> keystore and generate CSR for system vm: v-25027-VM
> >>>>>> Line 697113: 2020-07-25 02:30:09,669 ERROR
> >>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-40:ctx-3be9fddc
> >>>>>> job-1208154/job-1208257 ctx-ed08fc43) (logid:91fc037a) Failed to
> setup
> >>>>>> keystore and generate CSR for system vm: v-20771-VM
> >>>>>> Line 701161: 2020-07-25 02:30:48,127 ERROR
> >>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-41:ctx-4e3c666d
> >>>>>> job-1208155/job-1208258 ctx-df740f75) (logid:9fa7dece) Failed to
> setup
> >>>>>> keystore and generate CSR for system vm: s-24142-VM
> >>>>>> Line 701838: 2020-07-25 02:30:56,759 ERROR
> >>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-42:ctx-04ac8e71
> >>>>>> job-1208154/job-1208259 ctx-36c67b5e) (logid:91fc037a) Failed to
> setup
> >>>>>> keystore and generate CSR for system vm: v-24763-VM
> >>>>>> Line 703865: 2020-07-25 02:31:25,123 ERROR
> >>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-46:ctx-24bdbe8f
> >>>>>> job-1208154/job-1208266 ctx-fa500d3c) (logid:91fc037a) Failed to
> setup
> >>>>>> keystore and generate CSR for system vm: v-20773-VM
> >>>>>> Line 712078: 2020-07-25 02:32:28,666 ERROR
> >>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-48:ctx-4d5db5c2
> >>>>>> job-1208155/job-1208269 ctx-b0f67c2e) (logid:9fa7dece) Failed to
> setup
> >>>>>> keystore and generate CSR for system vm: s-25033-VM
> >>>>>> Line 715204: 2020-07-25 02:33:02,875 ERROR
> >>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-51:ctx-594da6d2
> >>>>>> job-1208155/job-1208273 ctx-ca115800) (logid:9fa7dece) Failed to
> setup
> >>>>>> keystore and generate CSR for system vm: s-25035-VM
> >>>>>> Line 743140: 2020-07-25 02:38:52,257 ERROR
> >>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-59:ctx-a3f032da
> >>>>>> job-1208155/job-1208294 ctx-626a2716) (logid:9fa7dece) Failed to
> setup
> >>>>>> keystore and generate CSR for system vm: s-25037-VM
> >>>>>> Line 817646: 2020-07-25 02:47:07,127 ERROR
> >>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-5:ctx-ca073bd1
> >>>>>> job-1208297/job-1208305 ctx-6213af3b) (logid:1a2a782c) Failed to
> setup
> >>>>>> keystore and generate CSR for system vm: s-25038-VM
> >>>>>> Line 847150: 2020-07-25 02:51:46,293 ERROR
> >>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-8:ctx-9db8c548
> >>>>>> job-1208296/job-1208312 ctx-83c570ff) (logid:436d1800) Failed to
> setup
> >>>>>> keystore and generate CSR for system vm: v-25040-VM
> >>>>>> Line 903949: 2020-07-25 02:57:48,515 ERROR
> >>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-4:ctx-f35fd59a
> >>>>>> job-1208320/job-1208333 ctx-f81179ac) (logid:05eacf0d) Failed to
> setup
> >>>>>> keystore and generate CSR for system vm: v-25041-VM
> >>>>>> Line 957628: 2020-07-25 03:04:13,286 ERROR
> >>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-24:ctx-dc18956a
> >>>>>> job-1208320/job-1208381 ctx-b53f5d7c) (logid:05eacf0d) Failed to
> setup
> >>>>>> keystore and generate CSR for system vm: v-20771-VM
> >>>>>> Line 961606: 2020-07-25 03:05:00,522 ERROR
> >>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-28:ctx-83e48421
> >>>>>> job-1208320/job-1208392 ctx-e6767f04) (logid:05eacf0d) Failed to
> setup
> >>>>>> keystore and generate CSR for system vm: v-24763-VM
> >>>>>> Line 966633: 2020-07-25 03:05:29,767 ERROR
> >>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-30:ctx-897e2a66
> >>>>>> job-1208320/job-1208397 ctx-39c396e1) (logid:05eacf0d) Failed to
> setup
> >>>>>> keystore and generate CSR for system vm: v-20773-VM
> >>>>>> Line 968892: 2020-07-25 03:05:53,648 ERROR
> >>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-36:ctx-68e45b80
> >>>>>> job-1208320/job-1208405 ctx-1aaf96dd) (logid:05eacf0d) Failed to
> setup
> >>>>>> keystore and generate CSR for system vm: v-25041-VM
> >>>>>> Line 1003036: 2020-07-25 03:13:33,582 ERROR
> >>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-45:ctx-0f552e0e
> >>>>>> job-1208320/job-1208422 ctx-e8835fe9) (logid:05eacf0d) Failed to
> setup
> >>>>>> keystore and generate CSR for system vm: v-20771-VM
> >>>>>> Line 1006496: 2020-07-25 03:14:09,787 ERROR
> >>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-46:ctx-08e5db24
> >>>>>> job-1208320/job-1208423 ctx-2cb636c2) (logid:05eacf0d) Failed to
> setup
> >>>>>> keystore and generate CSR for system vm: v-24763-VM
> >>>>>> Line 1008408: 2020-07-25 03:14:39,584 ERROR
> >>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-47:ctx-8b9288a9
> >>>>>> job-1208320/job-1208424 ctx-c05d1544) (logid:05eacf0d) Failed to
> setup
> >>>>>> keystore and generate CSR for system vm: v-20773-VM
> >>>>>> Line 1023914: 2020-07-25 03:15:58,139 ERROR
> >>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-48:ctx-4d54d717
> >>>>>> job-1208320/job-1208425 ctx-27e5397d) (logid:05eacf0d) Failed to
> setup
> >>>>>> keystore and generate CSR for system vm: v-25046-VM
> >>>>>> Line 1089568: 2020-07-25 03:24:24,534 ERROR
> >>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-59:ctx-62ff8844
> >>>>>> job-1208320/job-1208449 ctx-d6a14f7f) (logid:05eacf0d) Failed to
> setup
> >>>>>> keystore and generate CSR for system vm: v-25048-VM
> >>>>>> Line 1097605: 2020-07-25 03:25:23,179 ERROR
> >>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-62:ctx-24407a9e
> >>>>>> job-1208321/job-1208452 ctx-84f80ffa) (logid:42edff3a) Failed to
> setup
> >>>>>> keystore and generate CSR for system vm: s-25050-VM
> >>>>>>
> >>>>>> -Ammad Ali
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Mon, Aug 17, 2020 at 5:56 PM Andrija Panic <
> andrija.panic@gmail.com <ma...@gmail.com>>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Ammad, no such log lines in the file you uploaded....?
> >>>>>>>
> >>>>>>> On Mon, 17 Aug 2020 at 09:02, Ammad Syed <syedammad83@gmail.com
> <ma...@gmail.com>> wrote:
> >>>>>>>
> >>>>>>>> Hi Andrija,
> >>>>>>>>
> >>>>>>>> The error occurred only one time. Can you please try to grep
> "Failed to
> >>>>>>>> setup keystore and generate CSR" you will see all the systemVMs
> that I
> >>>>>>>> tried to create failed because of certificate generation failed.
> OR you
> >>>>>>> can
> >>>>>>>> grep job-1208321/job-1208452 this job as well.
> >>>>>>>>
> >>>>>>>> Also I mentioned above that same thing is happening in my test
> >>>>>>> environment
> >>>>>>>> that I created 4.11.1 then upgraded 4.11.3 > 4.13.1.
> >>>>>>>>
> >>>>>>>> Ammad Ali
> >>>>>>>>
> >>>>>>>> On Mon, Aug 17, 2020 at 11:17 AM Andrija Panic <
> >>>>>> andrija.panic@gmail.com <ma...@gmail.com>>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> 2020-07-25 03:23:51,093 WARN [c.c.v.SystemVmLoadScanner]
> >>>>>>>>> (secstorage-1:ctx-50756e9b) (logid:3d7c25cb) Unexpected exception
> >>>>>>> Failed
> >>>>>>>> to
> >>>>>>>>> fetch any free public IP address
> >>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Failed to fetch
> any
> >>>>>>> free
> >>>>>>>>> public IP address
> >>>>>>>>>
> >>>>>>>>> ?
> >>>>>>>>>
> >>>>>>>>> On Sat, 15 Aug 2020 at 18:05, Ammad Syed <syedammad83@gmail.com
> <ma...@gmail.com>>
> >>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> I have checked further, the key is successfully injected in
> >>>>>>>> systemvm.iso
> >>>>>>>>> on
> >>>>>>>>>> the management node and copied successfully to xenserver. I have
> >>>>>>>> checked
> >>>>>>>>> on
> >>>>>>>>>> systemVM there is no public key found in authorized_keys of
> root.
> >>>>>>>>>>
> >>>>>>>>>> How can I troubleshoot this further ? how can I enable trace
> >>>>>> logging
> >>>>>>>> for
> >>>>>>>>>> management server to see if there is something problematic
> >>>>>> happening.
> >>>>>>>>>>
> >>>>>>>>>> Ammad Ali
> >>>>>>>>>>
> >>>>>>>>>> On Sat, Aug 15, 2020 at 1:28 PM Ammad Syed <
> syedammad83@gmail.com <ma...@gmail.com>>
> >>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Hi,
> >>>>>>>>>>>
> >>>>>>>>>>> I have setup my test environment, here is what I did:
> >>>>>>>>>>>
> >>>>>>>>>>> - Installed ACS 4.11.1 and added xenserver 7.0 host in it.
> >>>>>>> SystemVMs
> >>>>>>>>> are
> >>>>>>>>>>> up in the zone and agents are up.
> >>>>>>>>>>> - Then upgraded the system to 4.11.3. Recreated systemVMs,
> agent
> >>>>>> is
> >>>>>>>> up
> >>>>>>>>>> and
> >>>>>>>>>>> systemVM are up with 4.11.3 systemVM version.
> >>>>>>>>>>> - Then upgraded the system to 4.13.1, recreated systemVMs are
> >>>>>>> running
> >>>>>>>>> but
> >>>>>>>>>>> agents are not up.
> >>>>>>>>>>>
> >>>>>>>>>>> I have checked md5sum of systemvm.iso on xenserver and
> management
> >>>>>>>>> server,
> >>>>>>>>>>> both are same.
> >>>>>>>>>>>
> >>>>>>>>>>> [root@xenserver iso]# md5sum
> >>>>>>>> /opt/xensource/packages/iso/systemvm.iso
> >>>>>>>>>>> baba18f156395da3a5d8208727d8f421
> >>>>>>>>>> /opt/xensource/packages/iso/systemvm.iso
> >>>>>>>>>>>
> >>>>>>>>>>> [root@cloudstack-upgrade vms]# md5sum
> >>>>>>>>>>> /usr/share/cloudstack-common/vms/systemvm.iso
> >>>>>>>>>>> baba18f156395da3a5d8208727d8f421
> >>>>>>>>>>> /usr/share/cloudstack-common/vms/systemvm.iso
> >>>>>>>>>>>
> >>>>>>>>>>> Also the private key on xenserver root and management server
> are
> >>>>>>>> same.
> >>>>>>>>>>>
> >>>>>>>>>>> management server
> >>>>>>>>>>> /usr/share/cloudstack-common/scripts/vm/systemvm/id_rsa.cloud
> >>>>>>>>>>>
> >>>>>>>>>>> xenserver path
> >>>>>>>>>>> /root/.ssh/id_rsa.cloud
> >>>>>>>>>>>
> >>>>>>>>>>> - Ammad Ali
> >>>>>>>>>>>
> >>>>>>>>>>> On Thu, Aug 13, 2020 at 11:27 AM Ammad Syed <
> >>>>>> syedammad83@gmail.com <ma...@gmail.com>
> >>>>>>>>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Here is the link for download management logs.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> https://drive.google.com/file/d/1l6HDPguGUNaOxsc7VSaj7eaP0F2UOFjA/view?usp=sharing
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Thu, Aug 13, 2020 at 11:22 AM Ammad Syed <
> >>>>>>> syedammad83@gmail.com>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> I have reverted the version back to 4.11.3. But I have saved
> >>>>>> logs
> >>>>>>>>>>>>> starting from upgrade.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I think the key has been copied successfully in system vm
> iso.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> 2020-07-25 00:34:17,214 INFO [c.c.s.ConfigurationServerImpl]
> >>>>>>>>>>>>> (main:null) (logid:) Going to update systemvm iso with
> >>>>>> generated
> >>>>>>>>>> keypairs
> >>>>>>>>>>>>> if needed
> >>>>>>>>>>>>> 2020-07-25 00:34:17,214 INFO [c.c.s.ConfigurationServerImpl]
> >>>>>>>>>>>>> (main:null) (logid:) Trying to inject public and private keys
> >>>>>>> into
> >>>>>>>>>> systemvm
> >>>>>>>>>>>>> iso
> >>>>>>>>>>>>> 2020-07-25 00:34:17,217 DEBUG [c.c.u.s.Script] (main:null)
> >>>>>>> (logid:)
> >>>>>>>>>>>>> Looking for vms/systemvm.iso in the classpath
> >>>>>>>>>>>>> 2020-07-25 00:34:17,217 DEBUG [c.c.u.s.Script] (main:null)
> >>>>>>> (logid:)
> >>>>>>>>>>>>> System resource:
> >>>>>>> file:/usr/share/cloudstack-common/vms/systemvm.iso
> >>>>>>>>>>>>> 2020-07-25 00:34:17,217 DEBUG [c.c.u.s.Script] (main:null)
> >>>>>>> (logid:)
> >>>>>>>>>>>>> Absolute path =
> /usr/share/cloudstack-common/vms/systemvm.iso
> >>>>>>>>>>>>> 2020-07-25 00:34:17,218 DEBUG [c.c.s.ConfigurationServerImpl]
> >>>>>>>>>>>>> (main:null) (logid:) Executing: /bin/bash
> >>>>>>>>>>>>>
> /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh
> >>>>>>>>>>>>> /var/cloudstack/management/.ssh/id_rsa.pub
> >>>>>>>>>>>>> /var/cloudstack/management/.ssh/id_rsa
> >>>>>>>>>>>>> /usr/share/cloudstack-common/vms/systemvm.iso
> >>>>>>>>>>>>> 2020-07-25 00:34:17,636 INFO [c.c.s.ConfigurationServerImpl]
> >>>>>>>>>>>>> (main:null) (logid:) Injected public and private keys into
> >>>>>>> systemvm
> >>>>>>>>> iso
> >>>>>>>>>>>>> with result : null
> >>>>>>>>>>>>> 2020-07-25 00:34:50,613 DEBUG [c.c.h.x.r.CitrixResourceBase]
> >>>>>>>>>>>>> (DirectAgent-1:ctx-d3dc4cf2) (logid:1d22396d) Copying
> >>>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/xenserver65/../../../../../vms/systemvm.iso
> >>>>>>>>>>>>> to /opt/xensource/packages/iso on 172.16.2.22 with permission
> >>>>>>> 0644
> >>>>>>>>>>>>> 2020-07-25 00:34:52,566 DEBUG [c.c.h.x.r.CitrixResourceBase]
> >>>>>>>>>>>>> (DirectAgent-2:ctx-2537b610) (logid:29c67b7a) Copying
> >>>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/xenserver65/../../../../../vms/systemvm.iso
> >>>>>>>>>>>>> to /opt/xensource/packages/iso on 172.16.2.5 with permission
> >>>>>> 0644
> >>>>>>>>>>>>> 2020-07-25 00:34:53,170 DEBUG [c.c.h.x.r.CitrixResourceBase]
> >>>>>>>>>>>>> (DirectAgent-3:ctx-168ac27d) (logid:a7862c4b) Copying
> >>>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/xenserver65/../../../../../vms/systemvm.iso
> >>>>>>>>>>>>> to /opt/xensource/packages/iso on 172.16.2.9 with permission
> >>>>>> 0644
> >>>>>>>>>>>>> 2020-07-25 00:34:54,621 DEBUG [c.c.h.x.r.CitrixResourceBase]
> >>>>>>>>>>>>> (DirectAgent-6:ctx-f640fe55) (logid:0a62d4cf) Copying
> >>>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/xenserver65/../../../../../vms/systemvm.iso
> >>>>>>>>>>>>> to /opt/xensource/packages/iso on 172.16.2.5 with permission
> >>>>>> 0644
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I have attached complete management logs. The start of logs
> is
> >>>>>>> the
> >>>>>>>>>> start
> >>>>>>>>>>>>> of management server after upgrade of management server from
> >>>>>>> 4.11.3
> >>>>>>>>> to
> >>>>>>>>>>>>> 4.13.1.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Ammad Ali
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Thu, Aug 13, 2020 at 1:35 AM Andrija Panic <
> >>>>>>>>> andrija.panic@gmail.com
> >>>>>>>>>>>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Do you get an error while trying to inject ssh key into the
> >>>>>>>>>> systemvm.iso
> >>>>>>>>>>>>>> (mgmt logs) , or can you confirm that the systemvm.iso on XS
> >>>>>>> host
> >>>>>>>>> and
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>> one on the mgmt server are identical, md5sum them (i.e. has
> >>>>>> the
> >>>>>>>> iso
> >>>>>>>>>> been
> >>>>>>>>>>>>>> copied over to the XS successfully) - this might explain not
> >>>>>>> being
> >>>>>>>>>> able
> >>>>>>>>>>>>>> to
> >>>>>>>>>>>>>> login via ssh with your private key.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Best,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Wed, 12 Aug 2020, 09:08 Ammad Syed, <
> syedammad83@gmail.com
> >>>>>>>
> >>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Yes this is exactly the same issue that i faced.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Sent from my iPhone
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On 12-Aug-2020, at 8:35 AM, Eric Lee Green <
> >>>>>>>>>>>>>> eric.lee.green@gmail.com>
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Correct, 4.11.3 template is used for 4.11.3, 4.12, and
> >>>>>>> 4.13.
> >>>>>>>>> 4.14
> >>>>>>>>>>>>>> moves
> >>>>>>>>>>>>>>> to the 4.14.0 template.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> There seems to be something odd happening key-wise
> >>>>>> sometimes
> >>>>>>>>> with
> >>>>>>>>>>>>>>> upgrades from 4.11.3 to 4.13.1 or 4.14.0. I managed an
> >>>>>>> upgrade
> >>>>>>>>>> from
> >>>>>>>>>>>>>>> 4.11.3 to 4.13.1 that *almost* worked, but the secondary
> >>>>>>> storage
> >>>>>>>>> VM
> >>>>>>>>>>>>>>> wouldn't work and thus I couldn't spawn new virtual
> >>>>>> machines.
> >>>>>>>> Same
> >>>>>>>>>>>>>> symptom
> >>>>>>>>>>>>>>> -- key error when the agent tried to ssh into it. And
> >>>>>> deleting
> >>>>>>>> it
> >>>>>>>>>> and
> >>>>>>>>>>>>>>> making it respawn didn't help. Then I tried 4.11.3 to
> 4.14.0
> >>>>>>> and
> >>>>>>>>>>>>>> *all* the
> >>>>>>>>>>>>>>> VM's failed at that point (of course, that was with the new
> >>>>>>>>>> template).
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Right now I'm back at 4.11.3 until this can be figured
> >>>>>> out.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On 8/11/2020 5:53 AM, Ammad Syed wrote:
> >>>>>>>>>>>>>>>>> Hi,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I think 4.12 and 4.13 uses same systemVM template i.e
> >>>>>>> 4.11.3
> >>>>>>>>>>>>>> version,
> >>>>>>>>>>>>>>> which
> >>>>>>>>>>>>>>>>> I already have registered. Currently I am running 4.11.3
> >>>>>>>>> version
> >>>>>>>>>>>>>> of ACS.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> MariaDB [cloud]> SELECT id,name,type,cross_zones,state
> >>>>>> FROM
> >>>>>>>>>>>>>>>>> cloud.vm_template WHERE name like '%systemvm-xenserver%'
> >>>>>>> AND
> >>>>>>>>>>>>>> removed IS
> >>>>>>>>>>>>>>>>> NULL;
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> +------+-----------------------------+--------+-------------+----------+
> >>>>>>>>>>>>>>>>> | id | name | type |
> >>>>>>> cross_zones |
> >>>>>>>>>>>>>> state |
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> +------+-----------------------------+--------+-------------+----------+
> >>>>>>>>>>>>>>>>> | 337 | systemvm-xenserver-3.0.0 | SYSTEM |
> >>>>>>> 0 |
> >>>>>>>>>>>>>> Inactive |
> >>>>>>>>>>>>>>>>> | 418 | systemvm-xenserver-4.2 | SYSTEM |
> >>>>>>> 0 |
> >>>>>>>>>>>>>> Active |
> >>>>>>>>>>>>>>>>> | 472 | systemvm-xenserver-4.3 | USER |
> >>>>>>> 1 |
> >>>>>>>>>>>>>> Inactive |
> >>>>>>>>>>>>>>>>> | 473 | systemvm-xenserver-4.3 | USER |
> >>>>>>> 1 |
> >>>>>>>>>>>>>> Inactive |
> >>>>>>>>>>>>>>>>> | 474 | systemvm-xenserver-4.3 | USER |
> >>>>>>> 1 |
> >>>>>>>>>>>>>> Inactive |
> >>>>>>>>>>>>>>>>> | 475 | systemvm-xenserver-4.3 | USER |
> >>>>>>> 1 |
> >>>>>>>>>>>>>> Inactive |
> >>>>>>>>>>>>>>>>> | 476 | systemvm-xenserver-4.3 | USER |
> >>>>>>> 0 |
> >>>>>>>>>>>>>> Inactive |
> >>>>>>>>>>>>>>>>> | 479 | systemvm-xenserver-4.3-2 | USER |
> >>>>>>> 1 |
> >>>>>>>>>>>>>> Inactive |
> >>>>>>>>>>>>>>>>> | 480 | systemvm-xenserver-4.3 | SYSTEM |
> >>>>>>> 0 |
> >>>>>>>>>>>>>> Active |
> >>>>>>>>>>>>>>>>> | 549 | systemvm-xenserver-4.5.1 | USER |
> >>>>>>> 0 |
> >>>>>>>>>>>>>> Active |
> >>>>>>>>>>>>>>>>> | 550 | systemvm-xenserver-4.5.1 | SYSTEM |
> >>>>>>> 0 |
> >>>>>>>>>>>>>> Active |
> >>>>>>>>>>>>>>>>> | 651 | systemvm-xenserver-4.7.0 | USER |
> >>>>>>> 0 |
> >>>>>>>>>>>>>> Inactive |
> >>>>>>>>>>>>>>>>> | 652 | systemvm-xenserver-4.7.0 | USER |
> >>>>>>> 0 |
> >>>>>>>>>>>>>> Inactive |
> >>>>>>>>>>>>>>>>> | 653 | systemvm-xenserver-4.7.0 | SYSTEM |
> >>>>>>> 0 |
> >>>>>>>>>>>>>> Inactive |
> >>>>>>>>>>>>>>>>> | 737 | systemvm-xenserver-4.9.2 | SYSTEM |
> >>>>>>> 1 |
> >>>>>>>>>>>>>> Inactive |
> >>>>>>>>>>>>>>>>> | 739 | systemvm-xenserver-4.9.2-sb | SYSTEM |
> >>>>>>> 1 |
> >>>>>>>>>>>>>> Active |
> >>>>>>>>>>>>>>>>> | 1245 | systemvm-xenserver-4.11.1 | SYSTEM |
> >>>>>>> 1 |
> >>>>>>>>>>>>>> Active |
> >>>>>>>>>>>>>>>>> | 1584 | systemvm-xenserver-4.11.2 | SYSTEM |
> >>>>>>> 1 |
> >>>>>>>>>>>>>> Active |
> >>>>>>>>>>>>>>>>> | 1677 | systemvm-xenserver-4.11.3 | SYSTEM |
> >>>>>>> 1 |
> >>>>>>>>>>>>>> Active |
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> +------+-----------------------------+--------+-------------+----------+
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> - Ammad
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Tue, Aug 11, 2020 at 5:17 PM Pierre-Luc Dion <
> >>>>>>>>>>>>>> pdion891@apache.org>
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>> db.
> >>>>>>>>>>>>>>>>>> Hi Syed,
> >>>>>>>>>>>>>>>>>> From 4.12, the systemvm template had to be upgraded
> >>>>>>> because
> >>>>>>>> of
> >>>>>>>>>> OS
> >>>>>>>>>>>>>>> change in
> >>>>>>>>>>>>>>>>>> the template, moved to a latest version of Debian.
> >>>>>> Because
> >>>>>>>> of
> >>>>>>>>>>>>>> that,
> >>>>>>>>>>>>>>> some VR
> >>>>>>>>>>>>>>>>>> scripts have changed and make obsolete older version of
> >>>>>>> VRs,
> >>>>>>>>> so
> >>>>>>>>>>>>>> you
> >>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>> most likely have to register an updated systemvm
> >>>>>> templates
> >>>>>>>> and
> >>>>>>>>>>>>>> upgrade
> >>>>>>>>>>>>>>> your
> >>>>>>>>>>>>>>>>>> system VMs and VRs.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Regards,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On Tue, Aug 11, 2020 at 6:24 AM Ammad Syed <
> >>>>>>>>>>>>>> syedammad83@gmail.com>
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Hi Guys,
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> I was previously on 4.9.3 cloudstack and upgraded to
> >>>>>>> 4.11.1
> >>>>>>>>>> then
> >>>>>>>>>>>>>>> 4.11.3.
> >>>>>>>>>>>>>>>>>>> The version 4.11.3 is working fine since six months.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Now I have tried to upgrade my system from 4.11.3 to
> >>>>>>>> 4.13.1.
> >>>>>>>>>> The
> >>>>>>>>>>>>>>> upgrade
> >>>>>>>>>>>>>>>>>>> goes successful. I didn't uploaded any system VM
> >>>>>>> template.
> >>>>>>>>>>>>>> However the
> >>>>>>>>>>>>>>>>>>> problem occured when I recreated my systemVM of POD,
> >>>>>> the
> >>>>>>> VM
> >>>>>>>>>>>>>> recreated
> >>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>> its state was running but agent state was not getting
> >>>>>> up,
> >>>>>>>> its
> >>>>>>>>>>>>>> showing
> >>>>>>>>>>>>>>>>>> blank
> >>>>>>>>>>>>>>>>>>> in column.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Digging further via job logs, the job is failed with
> >>>>>>> error
> >>>>>>>>> that
> >>>>>>>>>>>>>>> unable to
> >>>>>>>>>>>>>>>>>>> execute command via ssh. Below are the logs.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> 2020-07-25 02:30:48,126 ERROR [c.c.u.s.SshHelper]
> >>>>>>>>>>>>>>>>>>> (DirectAgent-211:ctx-62f09b31) (logid:9fa7dece) SSH
> >>>>>>>> execution
> >>>>>>>>>> of
> >>>>>>>>>>>>>>> command
> >>>>>>>>>>>>>>>>>>> /opt/cloud/bin/router_proxy.sh keystore-s
> >>>>>>>>>>>>>>>>>>> etup 169.254.2.199
> >>>>>>>>>>>>>> /usr/local/cloud/systemvm/conf/agent.properties
> >>>>>>>>>>>>>>>>>>> /usr/local/cloud/systemvm/conf/cloud.jks
> >>>>>> TJaQYChYBwKh7Cx9
> >>>>>>>> 365
> >>>>>>>>>>>>>>>>>>> /usr/local/cloud/systemvm/conf/clou
> >>>>>>>>>>>>>>>>>>> d.csr has an error status code in return. Result
> >>>>>> output:
> >>>>>>>>>>>>>>>>>>> 2020-07-25 02:30:48,127 DEBUG
> >>>>>>> [c.c.a.m.DirectAgentAttache]
> >>>>>>>>>>>>>>>>>>> (DirectAgent-211:ctx-62f09b31) (logid:9fa7dece) Seq
> >>>>>>>>>>>>>>>>>>> 906-3195585410596077730: Response Received:
> >>>>>>>>>>>>>>>>>>> 2020-07-25 02:30:48,127 DEBUG [c.c.a.t.Request]
> >>>>>>>>>>>>>>>>>>> (DirectAgent-211:ctx-62f09b31) (logid:9fa7dece) Seq
> >>>>>>>>>>>>>>>>>>> 906-3195585410596077730: Processing: { Ans: , MgmtId:
> >>>>>>>>>> 779271079
> >>>>>>>>>>>>>>>>>>> 43497, via: 906(xen-21-10-a3-khi02), Ver: v1, Flags:
> >>>>>> 10,
> >>>>>>>>>>>>>>>>>>> [{"org.apache.cloudstack.ca
> >>>>>>>>>>>>>>>>>>> .SetupKeystoreAnswer":{"result":false,"wait":0}}]
> >>>>>>>>>>>>>>>>>>> }
> >>>>>>>>>>>>>>>>>>> 2020-07-25 02:30:48,127 DEBUG [c.c.a.t.Request]
> >>>>>>>>>>>>>>>>>>> (Work-Job-Executor-41:ctx-4e3c666d
> >>>>>>> job-1208155/job-1208258
> >>>>>>>>>>>>>>> ctx-df740f75)
> >>>>>>>>>>>>>>>>>>> (logid:9fa7dece) Seq 906-319558541059607773
> >>>>>>>>>>>>>>>>>>> 0: Received: { Ans: , MgmtId: 77927107943497, via:
> >>>>>>>>>>>>>>>>>>> 906(xen-21-10-a3-khi02), Ver: v1, Flags: 10, {
> >>>>>>>>>>>>>> SetupKeystoreAnswer } }
> >>>>>>>>>>>>>>>>>>> 2020-07-25 02:30:48,127 ERROR
> >>>>>>>>> [c.c.v.VirtualMachineManagerImpl]
> >>>>>>>>>>>>>>>>>>> (Work-Job-Executor-41:ctx-4e3c666d
> >>>>>>> job-1208155/job-1208258
> >>>>>>>>>>>>>>> ctx-df740f75)
> >>>>>>>>>>>>>>>>>>> (logid:9fa7dece) Failed to setup keystore and generate
> >>>>>>> CSR
> >>>>>>>>> for
> >>>>>>>>>>>>>> system
> >>>>>>>>>>>>>>> vm:
> >>>>>>>>>>>>>>>>>>> s-24142-VM
> >>>>>>>>>>>>>>>>>>> 2020-07-25 02:30:48,127 DEBUG
> >>>>>>> [c.c.v.VmWorkJobHandlerProxy]
> >>>>>>>>>>>>>>>>>>> (Work-Job-Executor-41:ctx-4e3c666d
> >>>>>>> job-1208155/job-1208258
> >>>>>>>>>>>>>>> ctx-df740f75)
> >>>>>>>>>>>>>>>>>>> (logid:9fa7dece) Done executing VM work job:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":24142,"handlerName":"VirtualMachineManagerImpl"}
> >>>>>>>>>>>>>>>>>>> 2020-07-25 02:30:48,128 DEBUG
> >>>>>>>>> [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>>>>>>>>>>>>>>>>>> (Work-Job-Executor-41:ctx-4e3c666d
> >>>>>>> job-1208155/job-1208258
> >>>>>>>>>>>>>>> ctx-df740f75)
> >>>>>>>>>>>>>>>>>>> (logid:9fa7dece) Complete async job-1208258, jobStatus:
> >>>>>>>>>>>>>> SUCCEEDED,
> >>>>>>>>>>>>>>>>>>> resultCode: 0, result: null
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> I tried to dig it further, I was unable to login
> >>>>>> systemVM
> >>>>>>>> via
> >>>>>>>>>>>>>> ssh from
> >>>>>>>>>>>>>>>>>>> xenserver host with key /root/.ssh/id_rsa.cloud placed.
> >>>>>>>> Look
> >>>>>>>>>> like
> >>>>>>>>>>>>>>> private
> >>>>>>>>>>>>>>>>>>> key issue. However I am able to login on my old
> >>>>>>> systemVMs (
> >>>>>>>>> i.e
> >>>>>>>>>>>>>>> created
> >>>>>>>>>>>>>>>>>> on
> >>>>>>>>>>>>>>>>>>> ACS 4.11.3)
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Also I have SSL certificate enabled for console proxy
> >>>>>> on
> >>>>>>> my
> >>>>>>>>> ACS
> >>>>>>>>>>>>>> 4.11.3
> >>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>> I am using only xenserver 7.0 hosts.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> I tried to disable SSL on secstorage and console proxy
> >>>>>>> from
> >>>>>>>>>>>>>> global
> >>>>>>>>>>>>>>>>>>> settings, but still didn't worked.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> I had a fresh installation of ACS 4.13.1 with xenserver
> >>>>>>>> 7.0,
> >>>>>>>>>>>>>> systemVMs
> >>>>>>>>>>>>>>>>>> are
> >>>>>>>>>>>>>>>>>>> working fine in it.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Please advise.
> >>>>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>>> Regards,
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Syed Ammad Ali
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> --
> >>>>>>>>>>>>> Regards,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Syed Ammad Ali
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> --
> >>>>>>>>>>>> Regards,
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> Syed Ammad Ali
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> --
> >>>>>>>>>>> Regards,
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Syed Ammad Ali
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> --
> >>>>>>>>>> Regards,
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Syed Ammad Ali
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>>
> >>>>>>>>> Andrija Panić
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>> Regards,
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Syed Ammad Ali
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> --
> >>>>>>>
> >>>>>>> Andrija Panić
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> Regards,
> >>>>>>
> >>>>>>
> >>>>>> Syed Ammad Ali
> >>>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>>
> >>>>> Andrija Panić
> >>>>
> >>>>
> >>>> --
> >>>> Regards,
> >>>>
> >>>>
> >>>> Syed Ammad Ali
> >>
> >
>
--
Andrija Panić
Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error
Posted by Pearl d'Silva <pe...@shapeblue.com>.
Hi Ammad,
Could you try re-injecting the keys into the systemVM, as this MAY happen due to stale id_rsa keys. Run the injectkeys.sh to inject the existing public ssh key into the authorized_keys file inside the systemvm.iso. Execute:
/usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh /root/.ssh/id_rsa.pub /root/.ssh/id_rsa /usr/share/cloudstack-common/vms/systemvm.iso'
* Then replace the systemvm.iso on the hypervisor hosts. For each host:
1. Login and type command - xe host-param-clear param-name=tags uuid=<uuid of the XS host>
host uuid maybe obtained using: xe host-list
2. 2. Restart MS and the iso should get propagated for all. Check timestamp.
3. 3. Restart systemvms and they should get the iso inserted.
4.
https://cwiki.apache.org/confluence/display/CLOUDSTACK/SystemVm.iso#SystemVm.iso-Xen
Thanks,
Pearl
________________________________
From: Ammad Syed <sy...@gmail.com>
Sent: Wednesday, September 2, 2020 9:12 PM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: Re: Cloudstack 4.11.3 to 4.13.1 SystemVMs Error
Guys, please advise how to troubleshoot this further ? How can i enable trace level debugging ? any help in this will be highly appreciated?
Ammad
pearl.dsilva@shapeblue.com
www.shapeblue.com
3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
@shapeblue
> On 27-Aug-2020, at 1:41 PM, Vivek Kumar <vi...@indiqus.com.invalid> wrote:
>
> Pardon, the whole mail chain was not visible so replied accordingly. It seems like you are doing SSH using root shell so it doesn’t matter.
>
> Vivek Kumar
> Manager - Cloud & DevOps
> IndiQus Technologies
> 24*7 O +91 11 4055 1411 | M +91 7503460090
> www.indiqus.com<http://www.indiqus.com> <http://indiqus.com/>
>
> This message is intended only for the use of the individual or entity to which it is addressed and may contain information that is confidential and/or privileged. If you are not the intended recipient please delete the original message and any copy of it from your computer system. You are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited unless proper authorization has been obtained for such action. If you have received this communication in error, please notify the sender immediately. Although IndiQus attempts to sweep e-mail and attachments for viruses, it does not guarantee that both are virus-free and accepts no liability for any damage sustained as a result of viruses.
>
>> On 27-Aug-2020, at 2:03 PM, Vivek Kumar <vi...@indiqus.com> wrote:
>>
>> Hello Ammad,
>>
>> You haven’t defined the user while logging to the system VM through the Link local IP that’s why you have got the error while logging to the system VM.
>>
>>>> ssh -i /root/.ssh/id_rsa.cloud 169.254.0.121 -p 3922
>>
>> Try instead below
>>
>>>> ssh -i /root/.ssh/id_rsa.cloud -p 3922 root@169.254.0.121 <ma...@169.254.0.121>
>>
>> Vivek Kumar
>> Manager - Cloud & DevOps
>> IndiQus Technologies
>> 24*7 O +91 11 4055 1411 | M +91 7503460090
>> www.indiqus.com<http://www.indiqus.com> <http://indiqus.com/>
>>
>> This message is intended only for the use of the individual or entity to which it is addressed and may contain information that is confidential and/or privileged. If you are not the intended recipient please delete the original message and any copy of it from your computer system. You are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited unless proper authorization has been obtained for such action. If you have received this communication in error, please notify the sender immediately. Although IndiQus attempts to sweep e-mail and attachments for viruses, it does not guarantee that both are virus-free and accepts no liability for any damage sustained as a result of viruses.
>>
>>>> On 27-Aug-2020, at 1:59 PM, Ammad Syed <syedammad83@gmail.com <ma...@gmail.com>> wrote:
>>>
>>> Guys, any help to troubleshoot this further would be highly appreciated.
>>>
>>>
>>> Ammad Ali
>>>> On 18-Aug-2020, at 1:05 PM, Ammad Syed <syedammad83@gmail.com <ma...@gmail.com>> wrote:
>>>>
>>>>
>>>> Hi,
>>>>
>>>> The systemVM is accessible with 169.x.x.x IP from xenserver host. But login with private key is denied. I am testing this on my test environment.
>>>>
>>>> [root@xenserver ~]# ping 169.254.0.121
>>>> PING 169.254.0.121 (169.254.0.121) 56(84) bytes of data.
>>>> 64 bytes from 169.254.0.121: icmp_seq=1 ttl=64 time=0.328 ms
>>>> 64 bytes from 169.254.0.121: icmp_seq=2 ttl=64 time=0.124 ms
>>>> 64 bytes from 169.254.0.121: icmp_seq=3 ttl=64 time=0.111 ms
>>>> 64 bytes from 169.254.0.121: icmp_seq=4 ttl=64 time=0.126 ms
>>>> ^C
>>>> --- 169.254.0.121 ping statistics ---
>>>> 4 packets transmitted, 4 received, 0% packet loss, time 2997ms
>>>> rtt min/avg/max/mdev = 0.111/0.172/0.328/0.090 ms
>>>> [root@xenserver ~]#
>>>> [root@xenserver ~]#
>>>> [root@xenserver ~]# ssh -i /root/.ssh/id_rsa.cloud 169.254.0.121 -p 3922
>>>> Permission denied (publickey).
>>>>
>>>> I have checked by logging into the systemVM, the public key is not found in /root/.ssh/authorized_keys.
>>>>
>>>> I have digged further, the systemvm.iso is same on the management node and xenserver host. I have checked by mounting the ISO as well, the public key is there in authorized_keys in ISO.
>>>>
>>>> Restarting / recreating systemVM isn't fixed the issue.
>>>>
>>>> The systemVM is already running on HVM mode on xenserver host as the template OS type is Debian 8 64bit is selected.
>>>>
>>>> On my PRD upgrade, I have tested on one zone only. Currently I am testing on my test ACS environment which have only one zone.
>>>>
>>>> Ammad Ali
>>>>
>>>>> On Mon, Aug 17, 2020 at 11:10 PM Andrija Panic <andrija.panic@gmail.com <ma...@gmail.com>> wrote:
>>>>> It was my tool (GUI based tool, that still doesn't return any line (???)
>>>>> but I see it.
>>>>>
>>>>> Problem is with 169.254.1.178 not being reachable over SSH (2020-07-25
>>>>> 01:45:52,097 DEBUG [c.c.h.x.r.CitrixResourceBase]
>>>>> (DirectAgent-57:ctx-ded7f7f9) (logid:a75971c2) Executing command in VR:
>>>>> /opt/cloud/bin/router_proxy.sh keystore-setup 169.254.1.178
>>>>> /usr/local/cloud/systemvm/conf/agent.properties
>>>>> /usr/local/cloud/systemvm/conf/cloud.jks Xe8zzqZmUp9wmABH 365
>>>>> /usr/local/cloud/systemvm/conf/cloud.csr)
>>>>>
>>>>> Can you try to ssh from that particular XenServer (where the SSVM is
>>>>> running) to the IP 169.xxxxx on port 3922 (ssh port) using the private key
>>>>> - if you log in to the VM via it's console (root/password) - do you see
>>>>> with ifconfig the interface being started with 169..xxxx IP address? I
>>>>> assume that destroying the SSVM and starting a new one doesn't fix the
>>>>> issue?
>>>>>
>>>>> You can also try changing the OS type of the systemVM template to Other
>>>>> Linux 64bit, so it runs in HVM mode (instead of PV) - you'll need to see ID
>>>>> for that OS type from the guest_os table, and set the guest_os_id field in
>>>>> the vm_template table for your systemVM template.
>>>>>
>>>>> Does the issue occur in other zones as well? (assuming same hypervisor type)
>>>>>
>>>>>
>>>>> On Mon, 17 Aug 2020 at 15:31, Ammad Syed <syedammad83@gmail.com <ma...@gmail.com>> wrote:
>>>>>
>>>>>> I have rechecked the logs are there. Please review below lines.
>>>>>>
>>>>>>
>>>>>>
>>>>>> C:\Users\ammad\Downloads\management-server.log-4.13\management-server.log-4.13.1-errors
>>>>>> (27 hits)
>>>>>> Line 409969: 2020-07-25 01:45:52,320 ERROR
>>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-58:ctx-9dc75334
>>>>>> job-1207969/job-1208104 ctx-09e752ec) (logid:a75971c2) Failed to setup
>>>>>> keystore and generate CSR for system vm: s-25023-VM
>>>>>> Line 437413: 2020-07-25 01:51:33,845 ERROR
>>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-70:ctx-457ce4ee
>>>>>> job-1207969/job-1208129 ctx-195da69e) (logid:a75971c2) Failed to setup
>>>>>> keystore and generate CSR for system vm: s-25024-VM
>>>>>> Line 549538: 2020-07-25 02:06:44,833 ERROR
>>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-3:ctx-4634f7e0
>>>>>> job-1208154/job-1208161 ctx-05d065dc) (logid:91fc037a) Failed to setup
>>>>>> keystore and generate CSR for system vm: v-25027-VM
>>>>>> Line 663111: 2020-07-25 02:23:43,783 ERROR
>>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-22:ctx-fe0b857b
>>>>>> job-1208154/job-1208222 ctx-ef088cd7) (logid:91fc037a) Failed to setup
>>>>>> keystore and generate CSR for system vm: v-20771-VM
>>>>>> Line 667060: 2020-07-25 02:24:30,221 ERROR
>>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-23:ctx-662cfe5d
>>>>>> job-1208154/job-1208229 ctx-e59878e9) (logid:91fc037a) Failed to setup
>>>>>> keystore and generate CSR for system vm: v-24763-VM
>>>>>> Line 674269: 2020-07-25 02:25:47,345 ERROR
>>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-24:ctx-b6529613
>>>>>> job-1208154/job-1208233 ctx-7da81cab) (logid:91fc037a) Failed to setup
>>>>>> keystore and generate CSR for system vm: v-20773-VM
>>>>>> Line 678169: 2020-07-25 02:26:20,355 ERROR
>>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-31:ctx-7a062a66
>>>>>> job-1208154/job-1208244 ctx-4f48d08a) (logid:91fc037a) Failed to setup
>>>>>> keystore and generate CSR for system vm: v-25027-VM
>>>>>> Line 697113: 2020-07-25 02:30:09,669 ERROR
>>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-40:ctx-3be9fddc
>>>>>> job-1208154/job-1208257 ctx-ed08fc43) (logid:91fc037a) Failed to setup
>>>>>> keystore and generate CSR for system vm: v-20771-VM
>>>>>> Line 701161: 2020-07-25 02:30:48,127 ERROR
>>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-41:ctx-4e3c666d
>>>>>> job-1208155/job-1208258 ctx-df740f75) (logid:9fa7dece) Failed to setup
>>>>>> keystore and generate CSR for system vm: s-24142-VM
>>>>>> Line 701838: 2020-07-25 02:30:56,759 ERROR
>>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-42:ctx-04ac8e71
>>>>>> job-1208154/job-1208259 ctx-36c67b5e) (logid:91fc037a) Failed to setup
>>>>>> keystore and generate CSR for system vm: v-24763-VM
>>>>>> Line 703865: 2020-07-25 02:31:25,123 ERROR
>>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-46:ctx-24bdbe8f
>>>>>> job-1208154/job-1208266 ctx-fa500d3c) (logid:91fc037a) Failed to setup
>>>>>> keystore and generate CSR for system vm: v-20773-VM
>>>>>> Line 712078: 2020-07-25 02:32:28,666 ERROR
>>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-48:ctx-4d5db5c2
>>>>>> job-1208155/job-1208269 ctx-b0f67c2e) (logid:9fa7dece) Failed to setup
>>>>>> keystore and generate CSR for system vm: s-25033-VM
>>>>>> Line 715204: 2020-07-25 02:33:02,875 ERROR
>>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-51:ctx-594da6d2
>>>>>> job-1208155/job-1208273 ctx-ca115800) (logid:9fa7dece) Failed to setup
>>>>>> keystore and generate CSR for system vm: s-25035-VM
>>>>>> Line 743140: 2020-07-25 02:38:52,257 ERROR
>>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-59:ctx-a3f032da
>>>>>> job-1208155/job-1208294 ctx-626a2716) (logid:9fa7dece) Failed to setup
>>>>>> keystore and generate CSR for system vm: s-25037-VM
>>>>>> Line 817646: 2020-07-25 02:47:07,127 ERROR
>>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-5:ctx-ca073bd1
>>>>>> job-1208297/job-1208305 ctx-6213af3b) (logid:1a2a782c) Failed to setup
>>>>>> keystore and generate CSR for system vm: s-25038-VM
>>>>>> Line 847150: 2020-07-25 02:51:46,293 ERROR
>>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-8:ctx-9db8c548
>>>>>> job-1208296/job-1208312 ctx-83c570ff) (logid:436d1800) Failed to setup
>>>>>> keystore and generate CSR for system vm: v-25040-VM
>>>>>> Line 903949: 2020-07-25 02:57:48,515 ERROR
>>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-4:ctx-f35fd59a
>>>>>> job-1208320/job-1208333 ctx-f81179ac) (logid:05eacf0d) Failed to setup
>>>>>> keystore and generate CSR for system vm: v-25041-VM
>>>>>> Line 957628: 2020-07-25 03:04:13,286 ERROR
>>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-24:ctx-dc18956a
>>>>>> job-1208320/job-1208381 ctx-b53f5d7c) (logid:05eacf0d) Failed to setup
>>>>>> keystore and generate CSR for system vm: v-20771-VM
>>>>>> Line 961606: 2020-07-25 03:05:00,522 ERROR
>>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-28:ctx-83e48421
>>>>>> job-1208320/job-1208392 ctx-e6767f04) (logid:05eacf0d) Failed to setup
>>>>>> keystore and generate CSR for system vm: v-24763-VM
>>>>>> Line 966633: 2020-07-25 03:05:29,767 ERROR
>>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-30:ctx-897e2a66
>>>>>> job-1208320/job-1208397 ctx-39c396e1) (logid:05eacf0d) Failed to setup
>>>>>> keystore and generate CSR for system vm: v-20773-VM
>>>>>> Line 968892: 2020-07-25 03:05:53,648 ERROR
>>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-36:ctx-68e45b80
>>>>>> job-1208320/job-1208405 ctx-1aaf96dd) (logid:05eacf0d) Failed to setup
>>>>>> keystore and generate CSR for system vm: v-25041-VM
>>>>>> Line 1003036: 2020-07-25 03:13:33,582 ERROR
>>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-45:ctx-0f552e0e
>>>>>> job-1208320/job-1208422 ctx-e8835fe9) (logid:05eacf0d) Failed to setup
>>>>>> keystore and generate CSR for system vm: v-20771-VM
>>>>>> Line 1006496: 2020-07-25 03:14:09,787 ERROR
>>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-46:ctx-08e5db24
>>>>>> job-1208320/job-1208423 ctx-2cb636c2) (logid:05eacf0d) Failed to setup
>>>>>> keystore and generate CSR for system vm: v-24763-VM
>>>>>> Line 1008408: 2020-07-25 03:14:39,584 ERROR
>>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-47:ctx-8b9288a9
>>>>>> job-1208320/job-1208424 ctx-c05d1544) (logid:05eacf0d) Failed to setup
>>>>>> keystore and generate CSR for system vm: v-20773-VM
>>>>>> Line 1023914: 2020-07-25 03:15:58,139 ERROR
>>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-48:ctx-4d54d717
>>>>>> job-1208320/job-1208425 ctx-27e5397d) (logid:05eacf0d) Failed to setup
>>>>>> keystore and generate CSR for system vm: v-25046-VM
>>>>>> Line 1089568: 2020-07-25 03:24:24,534 ERROR
>>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-59:ctx-62ff8844
>>>>>> job-1208320/job-1208449 ctx-d6a14f7f) (logid:05eacf0d) Failed to setup
>>>>>> keystore and generate CSR for system vm: v-25048-VM
>>>>>> Line 1097605: 2020-07-25 03:25:23,179 ERROR
>>>>>> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-62:ctx-24407a9e
>>>>>> job-1208321/job-1208452 ctx-84f80ffa) (logid:42edff3a) Failed to setup
>>>>>> keystore and generate CSR for system vm: s-25050-VM
>>>>>>
>>>>>> -Ammad Ali
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Aug 17, 2020 at 5:56 PM Andrija Panic <andrija.panic@gmail.com <ma...@gmail.com>>
>>>>>> wrote:
>>>>>>
>>>>>>> Ammad, no such log lines in the file you uploaded....?
>>>>>>>
>>>>>>> On Mon, 17 Aug 2020 at 09:02, Ammad Syed <syedammad83@gmail.com <ma...@gmail.com>> wrote:
>>>>>>>
>>>>>>>> Hi Andrija,
>>>>>>>>
>>>>>>>> The error occurred only one time. Can you please try to grep "Failed to
>>>>>>>> setup keystore and generate CSR" you will see all the systemVMs that I
>>>>>>>> tried to create failed because of certificate generation failed. OR you
>>>>>>> can
>>>>>>>> grep job-1208321/job-1208452 this job as well.
>>>>>>>>
>>>>>>>> Also I mentioned above that same thing is happening in my test
>>>>>>> environment
>>>>>>>> that I created 4.11.1 then upgraded 4.11.3 > 4.13.1.
>>>>>>>>
>>>>>>>> Ammad Ali
>>>>>>>>
>>>>>>>> On Mon, Aug 17, 2020 at 11:17 AM Andrija Panic <
>>>>>> andrija.panic@gmail.com <ma...@gmail.com>>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> 2020-07-25 03:23:51,093 WARN [c.c.v.SystemVmLoadScanner]
>>>>>>>>> (secstorage-1:ctx-50756e9b) (logid:3d7c25cb) Unexpected exception
>>>>>>> Failed
>>>>>>>> to
>>>>>>>>> fetch any free public IP address
>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Failed to fetch any
>>>>>>> free
>>>>>>>>> public IP address
>>>>>>>>>
>>>>>>>>> ?
>>>>>>>>>
>>>>>>>>> On Sat, 15 Aug 2020 at 18:05, Ammad Syed <syedammad83@gmail.com <ma...@gmail.com>>
>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> I have checked further, the key is successfully injected in
>>>>>>>> systemvm.iso
>>>>>>>>> on
>>>>>>>>>> the management node and copied successfully to xenserver. I have
>>>>>>>> checked
>>>>>>>>> on
>>>>>>>>>> systemVM there is no public key found in authorized_keys of root.
>>>>>>>>>>
>>>>>>>>>> How can I troubleshoot this further ? how can I enable trace
>>>>>> logging
>>>>>>>> for
>>>>>>>>>> management server to see if there is something problematic
>>>>>> happening.
>>>>>>>>>>
>>>>>>>>>> Ammad Ali
>>>>>>>>>>
>>>>>>>>>> On Sat, Aug 15, 2020 at 1:28 PM Ammad Syed <syedammad83@gmail.com <ma...@gmail.com>>
>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> I have setup my test environment, here is what I did:
>>>>>>>>>>>
>>>>>>>>>>> - Installed ACS 4.11.1 and added xenserver 7.0 host in it.
>>>>>>> SystemVMs
>>>>>>>>> are
>>>>>>>>>>> up in the zone and agents are up.
>>>>>>>>>>> - Then upgraded the system to 4.11.3. Recreated systemVMs, agent
>>>>>> is
>>>>>>>> up
>>>>>>>>>> and
>>>>>>>>>>> systemVM are up with 4.11.3 systemVM version.
>>>>>>>>>>> - Then upgraded the system to 4.13.1, recreated systemVMs are
>>>>>>> running
>>>>>>>>> but
>>>>>>>>>>> agents are not up.
>>>>>>>>>>>
>>>>>>>>>>> I have checked md5sum of systemvm.iso on xenserver and management
>>>>>>>>> server,
>>>>>>>>>>> both are same.
>>>>>>>>>>>
>>>>>>>>>>> [root@xenserver iso]# md5sum
>>>>>>>> /opt/xensource/packages/iso/systemvm.iso
>>>>>>>>>>> baba18f156395da3a5d8208727d8f421
>>>>>>>>>> /opt/xensource/packages/iso/systemvm.iso
>>>>>>>>>>>
>>>>>>>>>>> [root@cloudstack-upgrade vms]# md5sum
>>>>>>>>>>> /usr/share/cloudstack-common/vms/systemvm.iso
>>>>>>>>>>> baba18f156395da3a5d8208727d8f421
>>>>>>>>>>> /usr/share/cloudstack-common/vms/systemvm.iso
>>>>>>>>>>>
>>>>>>>>>>> Also the private key on xenserver root and management server are
>>>>>>>> same.
>>>>>>>>>>>
>>>>>>>>>>> management server
>>>>>>>>>>> /usr/share/cloudstack-common/scripts/vm/systemvm/id_rsa.cloud
>>>>>>>>>>>
>>>>>>>>>>> xenserver path
>>>>>>>>>>> /root/.ssh/id_rsa.cloud
>>>>>>>>>>>
>>>>>>>>>>> - Ammad Ali
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Aug 13, 2020 at 11:27 AM Ammad Syed <
>>>>>> syedammad83@gmail.com <ma...@gmail.com>
>>>>>>>>
>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Here is the link for download management logs.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>> https://drive.google.com/file/d/1l6HDPguGUNaOxsc7VSaj7eaP0F2UOFjA/view?usp=sharing
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Aug 13, 2020 at 11:22 AM Ammad Syed <
>>>>>>> syedammad83@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> I have reverted the version back to 4.11.3. But I have saved
>>>>>> logs
>>>>>>>>>>>>> starting from upgrade.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I think the key has been copied successfully in system vm iso.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2020-07-25 00:34:17,214 INFO [c.c.s.ConfigurationServerImpl]
>>>>>>>>>>>>> (main:null) (logid:) Going to update systemvm iso with
>>>>>> generated
>>>>>>>>>> keypairs
>>>>>>>>>>>>> if needed
>>>>>>>>>>>>> 2020-07-25 00:34:17,214 INFO [c.c.s.ConfigurationServerImpl]
>>>>>>>>>>>>> (main:null) (logid:) Trying to inject public and private keys
>>>>>>> into
>>>>>>>>>> systemvm
>>>>>>>>>>>>> iso
>>>>>>>>>>>>> 2020-07-25 00:34:17,217 DEBUG [c.c.u.s.Script] (main:null)
>>>>>>> (logid:)
>>>>>>>>>>>>> Looking for vms/systemvm.iso in the classpath
>>>>>>>>>>>>> 2020-07-25 00:34:17,217 DEBUG [c.c.u.s.Script] (main:null)
>>>>>>> (logid:)
>>>>>>>>>>>>> System resource:
>>>>>>> file:/usr/share/cloudstack-common/vms/systemvm.iso
>>>>>>>>>>>>> 2020-07-25 00:34:17,217 DEBUG [c.c.u.s.Script] (main:null)
>>>>>>> (logid:)
>>>>>>>>>>>>> Absolute path = /usr/share/cloudstack-common/vms/systemvm.iso
>>>>>>>>>>>>> 2020-07-25 00:34:17,218 DEBUG [c.c.s.ConfigurationServerImpl]
>>>>>>>>>>>>> (main:null) (logid:) Executing: /bin/bash
>>>>>>>>>>>>> /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh
>>>>>>>>>>>>> /var/cloudstack/management/.ssh/id_rsa.pub
>>>>>>>>>>>>> /var/cloudstack/management/.ssh/id_rsa
>>>>>>>>>>>>> /usr/share/cloudstack-common/vms/systemvm.iso
>>>>>>>>>>>>> 2020-07-25 00:34:17,636 INFO [c.c.s.ConfigurationServerImpl]
>>>>>>>>>>>>> (main:null) (logid:) Injected public and private keys into
>>>>>>> systemvm
>>>>>>>>> iso
>>>>>>>>>>>>> with result : null
>>>>>>>>>>>>> 2020-07-25 00:34:50,613 DEBUG [c.c.h.x.r.CitrixResourceBase]
>>>>>>>>>>>>> (DirectAgent-1:ctx-d3dc4cf2) (logid:1d22396d) Copying
>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/xenserver65/../../../../../vms/systemvm.iso
>>>>>>>>>>>>> to /opt/xensource/packages/iso on 172.16.2.22 with permission
>>>>>>> 0644
>>>>>>>>>>>>> 2020-07-25 00:34:52,566 DEBUG [c.c.h.x.r.CitrixResourceBase]
>>>>>>>>>>>>> (DirectAgent-2:ctx-2537b610) (logid:29c67b7a) Copying
>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/xenserver65/../../../../../vms/systemvm.iso
>>>>>>>>>>>>> to /opt/xensource/packages/iso on 172.16.2.5 with permission
>>>>>> 0644
>>>>>>>>>>>>> 2020-07-25 00:34:53,170 DEBUG [c.c.h.x.r.CitrixResourceBase]
>>>>>>>>>>>>> (DirectAgent-3:ctx-168ac27d) (logid:a7862c4b) Copying
>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/xenserver65/../../../../../vms/systemvm.iso
>>>>>>>>>>>>> to /opt/xensource/packages/iso on 172.16.2.9 with permission
>>>>>> 0644
>>>>>>>>>>>>> 2020-07-25 00:34:54,621 DEBUG [c.c.h.x.r.CitrixResourceBase]
>>>>>>>>>>>>> (DirectAgent-6:ctx-f640fe55) (logid:0a62d4cf) Copying
>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/xenserver65/../../../../../vms/systemvm.iso
>>>>>>>>>>>>> to /opt/xensource/packages/iso on 172.16.2.5 with permission
>>>>>> 0644
>>>>>>>>>>>>>
>>>>>>>>>>>>> I have attached complete management logs. The start of logs is
>>>>>>> the
>>>>>>>>>> start
>>>>>>>>>>>>> of management server after upgrade of management server from
>>>>>>> 4.11.3
>>>>>>>>> to
>>>>>>>>>>>>> 4.13.1.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Ammad Ali
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Aug 13, 2020 at 1:35 AM Andrija Panic <
>>>>>>>>> andrija.panic@gmail.com
>>>>>>>>>>>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Do you get an error while trying to inject ssh key into the
>>>>>>>>>> systemvm.iso
>>>>>>>>>>>>>> (mgmt logs) , or can you confirm that the systemvm.iso on XS
>>>>>>> host
>>>>>>>>> and
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> one on the mgmt server are identical, md5sum them (i.e. has
>>>>>> the
>>>>>>>> iso
>>>>>>>>>> been
>>>>>>>>>>>>>> copied over to the XS successfully) - this might explain not
>>>>>>> being
>>>>>>>>>> able
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> login via ssh with your private key.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, 12 Aug 2020, 09:08 Ammad Syed, <syedammad83@gmail.com
>>>>>>>
>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Yes this is exactly the same issue that i faced.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Sent from my iPhone
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 12-Aug-2020, at 8:35 AM, Eric Lee Green <
>>>>>>>>>>>>>> eric.lee.green@gmail.com>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Correct, 4.11.3 template is used for 4.11.3, 4.12, and
>>>>>>> 4.13.
>>>>>>>>> 4.14
>>>>>>>>>>>>>> moves
>>>>>>>>>>>>>>> to the 4.14.0 template.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> There seems to be something odd happening key-wise
>>>>>> sometimes
>>>>>>>>> with
>>>>>>>>>>>>>>> upgrades from 4.11.3 to 4.13.1 or 4.14.0. I managed an
>>>>>>> upgrade
>>>>>>>>>> from
>>>>>>>>>>>>>>> 4.11.3 to 4.13.1 that *almost* worked, but the secondary
>>>>>>> storage
>>>>>>>>> VM
>>>>>>>>>>>>>>> wouldn't work and thus I couldn't spawn new virtual
>>>>>> machines.
>>>>>>>> Same
>>>>>>>>>>>>>> symptom
>>>>>>>>>>>>>>> -- key error when the agent tried to ssh into it. And
>>>>>> deleting
>>>>>>>> it
>>>>>>>>>> and
>>>>>>>>>>>>>>> making it respawn didn't help. Then I tried 4.11.3 to 4.14.0
>>>>>>> and
>>>>>>>>>>>>>> *all* the
>>>>>>>>>>>>>>> VM's failed at that point (of course, that was with the new
>>>>>>>>>> template).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Right now I'm back at 4.11.3 until this can be figured
>>>>>> out.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On 8/11/2020 5:53 AM, Ammad Syed wrote:
>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I think 4.12 and 4.13 uses same systemVM template i.e
>>>>>>> 4.11.3
>>>>>>>>>>>>>> version,
>>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>>>> I already have registered. Currently I am running 4.11.3
>>>>>>>>> version
>>>>>>>>>>>>>> of ACS.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> MariaDB [cloud]> SELECT id,name,type,cross_zones,state
>>>>>> FROM
>>>>>>>>>>>>>>>>> cloud.vm_template WHERE name like '%systemvm-xenserver%'
>>>>>>> AND
>>>>>>>>>>>>>> removed IS
>>>>>>>>>>>>>>>>> NULL;
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>> +------+-----------------------------+--------+-------------+----------+
>>>>>>>>>>>>>>>>> | id | name | type |
>>>>>>> cross_zones |
>>>>>>>>>>>>>> state |
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>> +------+-----------------------------+--------+-------------+----------+
>>>>>>>>>>>>>>>>> | 337 | systemvm-xenserver-3.0.0 | SYSTEM |
>>>>>>> 0 |
>>>>>>>>>>>>>> Inactive |
>>>>>>>>>>>>>>>>> | 418 | systemvm-xenserver-4.2 | SYSTEM |
>>>>>>> 0 |
>>>>>>>>>>>>>> Active |
>>>>>>>>>>>>>>>>> | 472 | systemvm-xenserver-4.3 | USER |
>>>>>>> 1 |
>>>>>>>>>>>>>> Inactive |
>>>>>>>>>>>>>>>>> | 473 | systemvm-xenserver-4.3 | USER |
>>>>>>> 1 |
>>>>>>>>>>>>>> Inactive |
>>>>>>>>>>>>>>>>> | 474 | systemvm-xenserver-4.3 | USER |
>>>>>>> 1 |
>>>>>>>>>>>>>> Inactive |
>>>>>>>>>>>>>>>>> | 475 | systemvm-xenserver-4.3 | USER |
>>>>>>> 1 |
>>>>>>>>>>>>>> Inactive |
>>>>>>>>>>>>>>>>> | 476 | systemvm-xenserver-4.3 | USER |
>>>>>>> 0 |
>>>>>>>>>>>>>> Inactive |
>>>>>>>>>>>>>>>>> | 479 | systemvm-xenserver-4.3-2 | USER |
>>>>>>> 1 |
>>>>>>>>>>>>>> Inactive |
>>>>>>>>>>>>>>>>> | 480 | systemvm-xenserver-4.3 | SYSTEM |
>>>>>>> 0 |
>>>>>>>>>>>>>> Active |
>>>>>>>>>>>>>>>>> | 549 | systemvm-xenserver-4.5.1 | USER |
>>>>>>> 0 |
>>>>>>>>>>>>>> Active |
>>>>>>>>>>>>>>>>> | 550 | systemvm-xenserver-4.5.1 | SYSTEM |
>>>>>>> 0 |
>>>>>>>>>>>>>> Active |
>>>>>>>>>>>>>>>>> | 651 | systemvm-xenserver-4.7.0 | USER |
>>>>>>> 0 |
>>>>>>>>>>>>>> Inactive |
>>>>>>>>>>>>>>>>> | 652 | systemvm-xenserver-4.7.0 | USER |
>>>>>>> 0 |
>>>>>>>>>>>>>> Inactive |
>>>>>>>>>>>>>>>>> | 653 | systemvm-xenserver-4.7.0 | SYSTEM |
>>>>>>> 0 |
>>>>>>>>>>>>>> Inactive |
>>>>>>>>>>>>>>>>> | 737 | systemvm-xenserver-4.9.2 | SYSTEM |
>>>>>>> 1 |
>>>>>>>>>>>>>> Inactive |
>>>>>>>>>>>>>>>>> | 739 | systemvm-xenserver-4.9.2-sb | SYSTEM |
>>>>>>> 1 |
>>>>>>>>>>>>>> Active |
>>>>>>>>>>>>>>>>> | 1245 | systemvm-xenserver-4.11.1 | SYSTEM |
>>>>>>> 1 |
>>>>>>>>>>>>>> Active |
>>>>>>>>>>>>>>>>> | 1584 | systemvm-xenserver-4.11.2 | SYSTEM |
>>>>>>> 1 |
>>>>>>>>>>>>>> Active |
>>>>>>>>>>>>>>>>> | 1677 | systemvm-xenserver-4.11.3 | SYSTEM |
>>>>>>> 1 |
>>>>>>>>>>>>>> Active |
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>> +------+-----------------------------+--------+-------------+----------+
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> - Ammad
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Tue, Aug 11, 2020 at 5:17 PM Pierre-Luc Dion <
>>>>>>>>>>>>>> pdion891@apache.org>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>> db.
>>>>>>>>>>>>>>>>>> Hi Syed,
>>>>>>>>>>>>>>>>>> From 4.12, the systemvm template had to be upgraded
>>>>>>> because
>>>>>>>> of
>>>>>>>>>> OS
>>>>>>>>>>>>>>> change in
>>>>>>>>>>>>>>>>>> the template, moved to a latest version of Debian.
>>>>>> Because
>>>>>>>> of
>>>>>>>>>>>>>> that,
>>>>>>>>>>>>>>> some VR
>>>>>>>>>>>>>>>>>> scripts have changed and make obsolete older version of
>>>>>>> VRs,
>>>>>>>>> so
>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>> most likely have to register an updated systemvm
>>>>>> templates
>>>>>>>> and
>>>>>>>>>>>>>> upgrade
>>>>>>>>>>>>>>> your
>>>>>>>>>>>>>>>>>> system VMs and VRs.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Tue, Aug 11, 2020 at 6:24 AM Ammad Syed <
>>>>>>>>>>>>>> syedammad83@gmail.com>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi Guys,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I was previously on 4.9.3 cloudstack and upgraded to
>>>>>>> 4.11.1
>>>>>>>>>> then
>>>>>>>>>>>>>>> 4.11.3.
>>>>>>>>>>>>>>>>>>> The version 4.11.3 is working fine since six months.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Now I have tried to upgrade my system from 4.11.3 to
>>>>>>>> 4.13.1.
>>>>>>>>>> The
>>>>>>>>>>>>>>> upgrade
>>>>>>>>>>>>>>>>>>> goes successful. I didn't uploaded any system VM
>>>>>>> template.
>>>>>>>>>>>>>> However the
>>>>>>>>>>>>>>>>>>> problem occured when I recreated my systemVM of POD,
>>>>>> the
>>>>>>> VM
>>>>>>>>>>>>>> recreated
>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>> its state was running but agent state was not getting
>>>>>> up,
>>>>>>>> its
>>>>>>>>>>>>>> showing
>>>>>>>>>>>>>>>>>> blank
>>>>>>>>>>>>>>>>>>> in column.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Digging further via job logs, the job is failed with
>>>>>>> error
>>>>>>>>> that
>>>>>>>>>>>>>>> unable to
>>>>>>>>>>>>>>>>>>> execute command via ssh. Below are the logs.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 2020-07-25 02:30:48,126 ERROR [c.c.u.s.SshHelper]
>>>>>>>>>>>>>>>>>>> (DirectAgent-211:ctx-62f09b31) (logid:9fa7dece) SSH
>>>>>>>> execution
>>>>>>>>>> of
>>>>>>>>>>>>>>> command
>>>>>>>>>>>>>>>>>>> /opt/cloud/bin/router_proxy.sh keystore-s
>>>>>>>>>>>>>>>>>>> etup 169.254.2.199
>>>>>>>>>>>>>> /usr/local/cloud/systemvm/conf/agent.properties
>>>>>>>>>>>>>>>>>>> /usr/local/cloud/systemvm/conf/cloud.jks
>>>>>> TJaQYChYBwKh7Cx9
>>>>>>>> 365
>>>>>>>>>>>>>>>>>>> /usr/local/cloud/systemvm/conf/clou
>>>>>>>>>>>>>>>>>>> d.csr has an error status code in return. Result
>>>>>> output:
>>>>>>>>>>>>>>>>>>> 2020-07-25 02:30:48,127 DEBUG
>>>>>>> [c.c.a.m.DirectAgentAttache]
>>>>>>>>>>>>>>>>>>> (DirectAgent-211:ctx-62f09b31) (logid:9fa7dece) Seq
>>>>>>>>>>>>>>>>>>> 906-3195585410596077730: Response Received:
>>>>>>>>>>>>>>>>>>> 2020-07-25 02:30:48,127 DEBUG [c.c.a.t.Request]
>>>>>>>>>>>>>>>>>>> (DirectAgent-211:ctx-62f09b31) (logid:9fa7dece) Seq
>>>>>>>>>>>>>>>>>>> 906-3195585410596077730: Processing: { Ans: , MgmtId:
>>>>>>>>>> 779271079
>>>>>>>>>>>>>>>>>>> 43497, via: 906(xen-21-10-a3-khi02), Ver: v1, Flags:
>>>>>> 10,
>>>>>>>>>>>>>>>>>>> [{"org.apache.cloudstack.ca
>>>>>>>>>>>>>>>>>>> .SetupKeystoreAnswer":{"result":false,"wait":0}}]
>>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>> 2020-07-25 02:30:48,127 DEBUG [c.c.a.t.Request]
>>>>>>>>>>>>>>>>>>> (Work-Job-Executor-41:ctx-4e3c666d
>>>>>>> job-1208155/job-1208258
>>>>>>>>>>>>>>> ctx-df740f75)
>>>>>>>>>>>>>>>>>>> (logid:9fa7dece) Seq 906-319558541059607773
>>>>>>>>>>>>>>>>>>> 0: Received: { Ans: , MgmtId: 77927107943497, via:
>>>>>>>>>>>>>>>>>>> 906(xen-21-10-a3-khi02), Ver: v1, Flags: 10, {
>>>>>>>>>>>>>> SetupKeystoreAnswer } }
>>>>>>>>>>>>>>>>>>> 2020-07-25 02:30:48,127 ERROR
>>>>>>>>> [c.c.v.VirtualMachineManagerImpl]
>>>>>>>>>>>>>>>>>>> (Work-Job-Executor-41:ctx-4e3c666d
>>>>>>> job-1208155/job-1208258
>>>>>>>>>>>>>>> ctx-df740f75)
>>>>>>>>>>>>>>>>>>> (logid:9fa7dece) Failed to setup keystore and generate
>>>>>>> CSR
>>>>>>>>> for
>>>>>>>>>>>>>> system
>>>>>>>>>>>>>>> vm:
>>>>>>>>>>>>>>>>>>> s-24142-VM
>>>>>>>>>>>>>>>>>>> 2020-07-25 02:30:48,127 DEBUG
>>>>>>> [c.c.v.VmWorkJobHandlerProxy]
>>>>>>>>>>>>>>>>>>> (Work-Job-Executor-41:ctx-4e3c666d
>>>>>>> job-1208155/job-1208258
>>>>>>>>>>>>>>> ctx-df740f75)
>>>>>>>>>>>>>>>>>>> (logid:9fa7dece) Done executing VM work job:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>> com.cloud.vm.VmWorkStart{"dcId":0,"userId":1,"accountId":1,"vmId":24142,"handlerName":"VirtualMachineManagerImpl"}
>>>>>>>>>>>>>>>>>>> 2020-07-25 02:30:48,128 DEBUG
>>>>>>>>> [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>>>>>>>>>>>>>> (Work-Job-Executor-41:ctx-4e3c666d
>>>>>>> job-1208155/job-1208258
>>>>>>>>>>>>>>> ctx-df740f75)
>>>>>>>>>>>>>>>>>>> (logid:9fa7dece) Complete async job-1208258, jobStatus:
>>>>>>>>>>>>>> SUCCEEDED,
>>>>>>>>>>>>>>>>>>> resultCode: 0, result: null
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I tried to dig it further, I was unable to login
>>>>>> systemVM
>>>>>>>> via
>>>>>>>>>>>>>> ssh from
>>>>>>>>>>>>>>>>>>> xenserver host with key /root/.ssh/id_rsa.cloud placed.
>>>>>>>> Look
>>>>>>>>>> like
>>>>>>>>>>>>>>> private
>>>>>>>>>>>>>>>>>>> key issue. However I am able to login on my old
>>>>>>> systemVMs (
>>>>>>>>> i.e
>>>>>>>>>>>>>>> created
>>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>>> ACS 4.11.3)
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Also I have SSL certificate enabled for console proxy
>>>>>> on
>>>>>>> my
>>>>>>>>> ACS
>>>>>>>>>>>>>> 4.11.3
>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>> I am using only xenserver 7.0 hosts.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I tried to disable SSL on secstorage and console proxy
>>>>>>> from
>>>>>>>>>>>>>> global
>>>>>>>>>>>>>>>>>>> settings, but still didn't worked.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I had a fresh installation of ACS 4.13.1 with xenserver
>>>>>>>> 7.0,
>>>>>>>>>>>>>> systemVMs
>>>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>>> working fine in it.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Please advise.
>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Syed Ammad Ali
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Syed Ammad Ali
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Regards,
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Syed Ammad Ali
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Regards,
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Syed Ammad Ali
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Regards,
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Syed Ammad Ali
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>>
>>>>>>>>> Andrija Panić
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Regards,
>>>>>>>>
>>>>>>>>
>>>>>>>> Syed Ammad Ali
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> Andrija Panić
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Regards,
>>>>>>
>>>>>>
>>>>>> Syed Ammad Ali
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Andrija Panić
>>>>
>>>>
>>>> --
>>>> Regards,
>>>>
>>>>
>>>> Syed Ammad Ali
>>
>