You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by "Corey, Mike" <mi...@sap.com> on 2020/07/02 13:51:19 UTC
Initial SystemVM Creation Failing
Hi,
When I enable the zone, the systemvm creation begins but goes into a loop of creating and deleting VMs in vCenter. I see in the logs there is an error for creating the folder permissions on the secondary storage (see below). Running this CHMOD manually from inside the CSM kicks the same error.
Please provide suggestions.
2020-07-02 09:08:39,665 DEBUG [c.c.h.v.m.VmwareManagerImpl] (DirectAgent-66:ctx-e15c32b2 usphlmvesxt154.phl.global.corp.sap, job-16/job-35, cmd: CopyCommand) (logid:1cfa64a9) Executing: sudo chmod 1777 /var/cloudstack/mnt/VM/345050029058.2807345c
2020-07-02 09:08:39,666 DEBUG [c.c.h.v.m.VmwareManagerImpl] (DirectAgent-66:ctx-e15c32b2 usphlmvesxt154.phl.global.corp.sap, job-16/job-35, cmd: CopyCommand) (logid:1cfa64a9) Executing while with timeout : 1440000
2020-07-02 09:08:39,693 DEBUG [c.c.h.v.m.VmwareManagerImpl] (DirectAgent-66:ctx-e15c32b2 usphlmvesxt154.phl.global.corp.sap, job-16/job-35, cmd: CopyCommand) (logid:1cfa64a9) Exit value is 1
2020-07-02 09:08:39,697 DEBUG [c.c.h.v.m.VmwareManagerImpl] (DirectAgent-66:ctx-e15c32b2 usphlmvesxt154.phl.global.corp.sap, job-16/job-35, cmd: CopyCommand) (logid:1cfa64a9) chmod: changing permissions of '/var/cloudstack/mnt/VM/345050029058.2807345c': Operation not permitted
2020-07-02 09:08:39,699 WARN [c.c.h.v.m.VmwareManagerImpl] (DirectAgent-66:ctx-e15c32b2 usphlmvesxt154.phl.global.corp.sap, job-16/job-35, cmd: CopyCommand) (logid:1cfa64a9) Unable to set permissions for /var/cloudstack/mnt/VM/345050029058.2807345c due to chmod: changing permissions of '/var/cloudstack/mnt/VM/345050029058.2807345c': Operation not permitted
Lines...
2020-07-02 09:09:00,598 DEBUG [c.c.h.v.m.HostMO] (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap, job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) find VM v-2-VM on host
2020-07-02 09:09:00,598 INFO [c.c.h.v.m.HostMO] (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap, job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) VM v-2-VM not found in host cache
2020-07-02 09:09:00,598 DEBUG [c.c.h.v.m.HostMO] (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap, job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) load VM cache on host
2020-07-02 09:09:00,616 DEBUG [c.c.h.v.m.HostMO] (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap, job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) find VM v-2-VM on host
2020-07-02 09:09:00,617 DEBUG [c.c.h.v.m.HostMO] (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap, job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) VM v-2-VM found in host cache
2020-07-02 09:09:00,799 ERROR [c.c.h.v.m.VmwareManagerImpl] (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap, job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) Unable to create systemvm folder on secondary storage. location: /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm
2020-07-02 09:09:00,800 INFO [c.c.h.v.u.VmwareHelper] (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap, job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) [ignored]failed to get message for exception: Unable to create systemvm folder on secondary storage. location: /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm
2020-07-02 09:09:00,800 WARN [c.c.h.v.r.VmwareResource] (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap, job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) StartCommand failed due to Exception: com.cloud.utils.exception.CloudRuntimeException
Message: Unable to create systemvm folder on secondary storage. location: /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm
com.cloud.utils.exception.CloudRuntimeException: Unable to create systemvm folder on secondary storage. location: /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm
at com.cloud.hypervisor.vmware.manager.VmwareManagerImpl.prepareSecondaryStorageStore(VmwareManagerImpl.java:648)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy389.prepareSecondaryStorageStore(Unknown Source)
at com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:1962)
at com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:530)
at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
2020-07-02 09:09:00,804 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-17:ctx-681dc524) (logid:1cfa64a9) Seq 1-4541880224203147245: Cancelling because one of the answers is false and it is stop on error.
2020-07-02 09:09:00,804 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-17:ctx-681dc524) (logid:1cfa64a9) Seq 1-4541880224203147245: Response Received:
2020-07-02 09:09:00,805 DEBUG [c.c.a.t.Request] (DirectAgent-17:ctx-681dc524) (logid:1cfa64a9) Seq 1-4541880224203147245: Processing: { Ans: , MgmtId: 345050029058, via: 1(usphlmvesxt154.phl.global.corp.sap), Ver: v1, Flags: 110, [{"com.cloud.agent.api.StartAnswer":{"vm":{"id":2,"name":"v-2-VM","bootloader":"HVM","state":"Starting","type":"ConsoleProxy","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":1073741824,"maxRam":1073741824,"hostName":"v-2-VM","arch":"x86_64","os":"Other Linux (64-bit)","platformEmulator":"otherLinux64Guest","bootArgs":" template=domP type=consoleproxy host=10.4.32.163 port=8250 name=v-2-VM zone=1 pod=1 guid=Proxy.2 proxy_vm=2 disable_rp_filter=true eth2ip=10.4.42.25 eth2mask=255.255.255.192 gateway=10.4.42.1 eth0ip=0.0.0.0 eth0mask=0.0.0.0 eth1ip=10.4.4.35 eth1mask=255.255.252.0 mgmtcidr=10.4.32.0/24 localgw=10.4.4.1 internaldns1=10.4.202.200 dns1=10.4.202.200 dns2=10.4.12.200","enableHA":false,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"sYnwDJYgzrMKdm3ftAO-kg","params":{"vmware.reserve.mem":"false","vmware.reserve.cpu":"false","nicAdapter":"E1000"},"uuid":"fb65df73-693d-48fb-a203-c2b5e776842a","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"9a1f2ae8-6b10-4525-8a54-a175b153b789","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"50cc970f-fbe2-3b45-9cb5-b8bcdb6dbff5","id":1,"poolType":"NetworkFilesystem","host":"usphlcv300.phl.global.corp.sap","path":"/usphlmvesx_lab_cspri","port":2049,"url":"NetworkFilesystem://usphlcv300.phl.global.corp.sap/usphlmvesx_lab_cspri/?ROLE=Primary&STOREUUID=50cc970f-fbe2-3b45-9cb5-b8bcdb6dbff5","isManaged":false}},"name":"ROOT-2","size":0,"path":"ROOT-2_2","volumeId":2,"vmName":"v-2-VM","accountId":1,"format":"OVA","provisioningType":"THIN","id":2,"deviceId":0,"bytesReadRate":0,"bytesWriteRate":0,"iopsReadRate":0,"iopsWriteRate":0,"hypervisorType":"VMware","directDownload":false}},"diskSeq":0,"path":"ROOT-2_2","type":"ROOT","_details":{"storageHost":"usphlcv300.phl.global.corp.sap","managed":"false","storagePort":"2049","volumeSize":"0"}}],"nics":[{"deviceId":2,"networkRateMbps":-1,"defaultNic":true,"pxeDisable":true,"nicUuid":"ffbc1822-7907-4cdf-9e80-bb316786e04d","details":{"PromiscuousMode":"false","MacAddressChanges":"true","ForgedTransmits":"true"},"dpdkEnabled":false,"uuid":"f3a30021-a469-413d-85a8-ca1ca68ba27c","ip":"10.4.42.25","netmask":"255.255.255.192","gateway":"10.4.42.1","mac":"1e:00:e3:00:00:19","dns1":"10.4.202.200","dns2":"10.4.12.200","broadcastType":"Vlan","type":"Public","broadcastUri":"vlan://untagged","isolationUri":"vlan://untagged","isSecurityGroupEnabled":false,"name":"vSwitch0,134,vmwaresvs"},{"deviceId":0,"networkRateMbps":-1,"defaultNic":false,"pxeDisable":true,"nicUuid":"d6521be0-85b8-4836-bf8b-8461afdf593a","details":{"PromiscuousMode":"false","MacAddressChanges":"true","ForgedTransmits":"true"},"dpdkEnabled":false,"uuid":"eda0ae3d-d15d-4f91-b8db-a719cf5a94bf","mac":"02:00:31:71:00:02","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false,"name":"vSwitch0,146,vmwaresvs"},{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"pxeDisable":true,"nicUuid":"4e6201ad-91c0-4cab-bf53-9e2440025245","details":{"PromiscuousMode":"false","MacAddressChanges":"true","ForgedTransmits":"true"},"dpdkEnabled":false,"uuid":"ce88b030-c04c-4437-b60d-9910c2cc376f","ip":"10.4.4.35","netmask":"255.255.252.0","gateway":"10.4.4.1","mac":"1e:00:6b:00:00:27","broadcastType":"Native","type":"Management","isSecurityGroupEnabled":false,"name":"vSwitch0,146,vmwaresvs"}],"guestOsDetails":{},"extraConfig":{}},"result":false,"details":"StartCommand failed due to Exception: com.cloud.utils.exception.CloudRuntimeException\nMessage: Unable to create systemvm folder on secondary storage. location: /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm\n","wait":0}}] }
2020-07-02 09:09:00,805 DEBUG [c.c.a.m.AgentAttache] (DirectAgent-17:ctx-681dc524) (logid:1cfa64a9) Seq 1-4541880224203147245: No more commands found
2020-07-02 09:09:00,806 DEBUG [c.c.a.t.Request] (Work-Job-Executor-2:ctx-9aa38618 job-16/job-35 ctx-7ebd1b78) (logid:1cfa64a9) Seq 1-4541880224203147245: Received: { Ans: , MgmtId: 345050029058, via: 1(usphlmvesxt154.phl.global.corp.sap), Ver: v1, Flags: 110, { StartAnswer } }
2020-07-02 09:09:00,816 INFO [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-2:ctx-9aa38618 job-16/job-35 ctx-7ebd1b78) (logid:1cfa64a9) Unable to start VM on Host[-1-Routing] due to StartCommand failed due to Exception: com.cloud.utils.exception.CloudRuntimeException
Message: Unable to create systemvm folder on secondary storage. location: /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm
Mike Corey
Technology Senior Consultant, IT CS CTW Operation & Virtualization Service US
SAP AMERICA, INC. 3999 West Chester Pike, Newtown Square, 19073 United States
T +1 610 661 0905, M +1 484 274 2658, E mike.corey@sap.com
[cid:image002.png@01D65056.54B16610]
RE: Initial SystemVM Creation Failing
Posted by "Corey, Mike" <mi...@sap.com>.
Hi,
Has anyone replicated this error/issue I'm seeing? What part of the setup or systemvm build sets or creates the /var/cloudstack/mnt/VM/ directory and WHY would it be restrictive?
[root@ ~]# ls -al /var/cloudstack/mnt/VM/345050029058.1f7af2bf
total 12
drwxr-xr-x. 4 nobody nobody 4096 Jul 8 11:15 .
drwxr-xr-x. 3 cloud cloud 35 Jul 8 14:06 ..
drwxrwxr-x. 2 cloud cloud 4096 Jul 8 10:58 systemvm
drwxr-xr-x. 3 nobody nobody 4096 Jul 8 11:15 template
[root@usphlmvcsmt01 ~]#
-----Original Message-----
From: Corey, Mike
Sent: Wednesday, July 8, 2020 2:45 PM
To: users@cloudstack.apache.org
Subject: RE: Initial SystemVM Creation Failing
Gone over the installation now twice (reverted to snapshot of CentOS VM) and still getting weird storage/permission type issues.
The permissions and ownership of the systemvm folder looks correct.
[root@ ~]# ls -al /mnt/secondary/
total 16
drwxr-xr-x. 4 root root 4096 Jul 8 11:15 .
drwxr-xr-x. 4 root root 38 Jul 8 11:15 ..
drwxrwxrwx. 12 root root 4096 Jul 8 11:05 .snapshot
drwxrwxr-x. 2 cloud cloud 4096 Jul 8 10:58 systemvm
drwxr-xr-x. 3 root root 4096 Jul 8 11:15 template
However, I'm still seeing the copy iso error and the VMs keep creating and deleting with the log suggesting the vmx file of the VM doesn't exist on the datastore (can create files/folder on datastore identified in errror from vCenter).
2020-07-08 14:22:36,486 INFO [c.c.h.v.m.DatastoreMO] (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap, job-6/job-37, cmd: StartCommand) (logid:b828ac16) Folder v-6-VM does not exist on datastore
2020-07-08 14:22:36,494 INFO [c.c.h.v.m.DatastoreMO] (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap, job-6/job-37, cmd: StartCommand) (logid:b828ac16) Search file v-6-VM.vmx on [50cc970ffbe23b459cb5b8bcdb6dbff5]
2020-07-08 14:22:36,522 INFO [c.c.h.v.m.DatastoreMO] (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap, job-6/job-37, cmd: StartCommand) (logid:b828ac16) File [50cc970ffbe23b459cb5b8bcdb6dbff5] v-6-VM.vmx does not exist on datastore
2020-07-08 14:22:36,522 INFO [c.c.h.v.m.DatastoreMO] (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap, job-6/job-37, cmd: StartCommand) (logid:b828ac16) Searching file v-6-VM.vmx in [50cc970ffbe23b459cb5b8bcdb6dbff5]
2020-07-08 14:22:36,641 INFO [c.c.h.v.m.HypervisorHostHelper] (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap, job-6/job-37, cmd: StartCommand) (logid:b828ac16) Create blank VM. cpuCount: 1, cpuSpeed(MHz): 500, mem(Mb): 1024
2020-07-08 14:22:36,692 DEBUG [c.c.h.v.m.ClusterMO] (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap, job-6/job-37, cmd: StartCommand) (logid:b828ac16) Retrieved recommended disk controller for guest OS : otherLinux64Guest in cluster PHL-CSM01 : VirtualLsiLogicController
2020-07-08 14:22:37,115 DEBUG [c.c.h.v.m.HostMO] (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap, job-6/job-37, cmd: StartCommand) (logid:b828ac16) find VM v-6-VM on host
2020-07-08 14:22:37,115 INFO [c.c.h.v.m.HostMO] (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap, job-6/job-37, cmd: StartCommand) (logid:b828ac16) VM v-6-VM not found in host cache
2020-07-08 14:22:37,115 DEBUG [c.c.h.v.m.HostMO] (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap, job-6/job-37, cmd: StartCommand) (logid:b828ac16) load VM cache on host
2020-07-08 14:22:37,155 DEBUG [c.c.h.v.m.HostMO] (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap, job-6/job-37, cmd: StartCommand) (logid:b828ac16) find VM v-6-VM on host
2020-07-08 14:22:37,155 DEBUG [c.c.h.v.m.HostMO] (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap, job-6/job-37, cmd: StartCommand) (logid:b828ac16) VM v-6-VM found in host cache
2020-07-08 14:22:37,375 INFO [c.c.h.v.m.VmwareManagerImpl] (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap, job-6/job-37, cmd: StartCommand) (logid:b828ac16) Inject SSH key pairs before copying systemvm.iso into secondary storage
2020-07-08 14:22:37,377 INFO [c.c.s.ConfigurationServerImpl] (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap, job-6/job-37, cmd: StartCommand) (logid:b828ac16) Processing updateKeyPairs
2020-07-08 14:22:37,377 INFO [c.c.s.ConfigurationServerImpl] (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap, job-6/job-37, cmd: StartCommand) (logid:b828ac16) Keypairs already in database, updating local copy
2020-07-08 14:22:37,381 INFO [c.c.s.ConfigurationServerImpl] (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap, job-6/job-37, cmd: StartCommand) (logid:b828ac16) Going to update systemvm iso with generated keypairs if needed
2020-07-08 14:22:37,381 INFO [c.c.s.ConfigurationServerImpl] (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap, job-6/job-37, cmd: StartCommand) (logid:b828ac16) Trying to inject public and private keys into systemvm iso
2020-07-08 14:22:37,381 DEBUG [c.c.u.s.Script] (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap, job-6/job-37, cmd: StartCommand) (logid:b828ac16) Looking for scripts/vm/systemvm/injectkeys.sh in the classpath
2020-07-08 14:22:37,381 DEBUG [c.c.u.s.Script] (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap, job-6/job-37, cmd: StartCommand) (logid:b828ac16) System resource: file:/usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh
2020-07-08 14:22:37,381 DEBUG [c.c.u.s.Script] (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap, job-6/job-37, cmd: StartCommand) (logid:b828ac16) Absolute path = /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh
2020-07-08 14:22:37,381 DEBUG [c.c.u.s.Script] (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap, job-6/job-37, cmd: StartCommand) (logid:b828ac16) Looking for vms/systemvm.iso in the classpath
2020-07-08 14:22:37,381 DEBUG [c.c.u.s.Script] (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap, job-6/job-37, cmd: StartCommand) (logid:b828ac16) System resource: file:/usr/share/cloudstack-common/vms/systemvm.iso
2020-07-08 14:22:37,381 DEBUG [c.c.u.s.Script] (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap, job-6/job-37, cmd: StartCommand) (logid:b828ac16) Absolute path = /usr/share/cloudstack-common/vms/systemvm.iso
2020-07-08 14:22:37,381 DEBUG [c.c.s.ConfigurationServerImpl] (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap, job-6/job-37, cmd: StartCommand) (logid:b828ac16) Executing: /bin/bash /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh /var/cloudstack/management/.ssh/id_rsa.pub /var/cloudstack/management/.ssh/id_rsa /usr/share/cloudstack-common/vms/systemvm.iso
2020-07-08 14:22:37,384 DEBUG [c.c.s.ConfigurationServerImpl] (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap, job-6/job-37, cmd: StartCommand) (logid:b828ac16) Executing while with timeout : 3600000
2020-07-08 14:22:37,487 DEBUG [c.c.s.ConfigurationServerImpl] (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap, job-6/job-37, cmd: StartCommand) (logid:b828ac16) Execution is successful.
2020-07-08 14:22:37,494 DEBUG [c.c.s.ConfigurationServerImpl] (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap, job-6/job-37, cmd: StartCommand) (logid:b828ac16) mount: /dev/loop0 is write-protected, mounting read-only
New public key is the same as the one in the systemvm.iso, not injecting it, not modifying systemvm.iso
2020-07-08 14:22:37,494 INFO [c.c.s.ConfigurationServerImpl] (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap, job-6/job-37, cmd: StartCommand) (logid:b828ac16) The script injectkeys.sh was run with result : null
2020-07-08 14:22:37,494 INFO [c.c.h.v.m.VmwareManagerImpl] (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap, job-6/job-37, cmd: StartCommand) (logid:b828ac16) Copy System VM patch ISO file to secondary storage. source ISO: /usr/share/cloudstack-common/vms/systemvm.iso, destination: /var/cloudstack/mnt/VM/345050029058.1f7af2bf/systemvm/systemvm-4.14.0.0.iso
2020-07-08 14:22:37,495 ERROR [c.c.h.v.m.VmwareManagerImpl] (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap, job-6/job-37, cmd: StartCommand) (logid:b828ac16) Unexpected exception
java.io.FileNotFoundException: /var/cloudstack/mnt/VM/345050029058.1f7af2bf/systemvm/systemvm-4.14.0.0.iso (Permission denied)
But, the ownership/permissions look okay on that destination directory too.
[root@ ~]# ls -al /var/cloudstack/mnt/VM/345050029058.1f7af2bf
total 12
drwxr-xr-x. 4 nobody nobody 4096 Jul 8 11:15 .
drwxr-xr-x. 3 cloud cloud 35 Jul 8 14:06 ..
drwxrwxr-x. 2 cloud cloud 4096 Jul 8 10:58 systemvm
drwxr-xr-x. 3 nobody nobody 4096 Jul 8 11:15 template
[root@usphlmvcsmt01 ~]#
Any ideas?
-----Original Message-----
From: Corey, Mike
Sent: Thursday, July 2, 2020 2:31 PM
To: users@cloudstack.apache.org<ma...@cloudstack.apache.org>
Subject: RE: Initial SystemVM Creation Failing
No upgrade - this again is the fresh install of CloudStack 4.14 with VMware resources. Maybe I should note that I created the primary and secondary storage via the "Advanced" setup wizard from the UI (logged in as admin).
I performed the ownership change you suggested so now cloud owns the systemvm folder on the secondary storage. However, now I'm getting a new error regarding copying the ISO.
Should I just bag the 4.14 installation and try to get 4.13.1 running with VMware?
2020-07-02 14:19:13,314 INFO [c.c.h.v.m.VmwareManagerImpl] (DirectAgent-10:ctx-52902ebf usphlmvesxt155.phl.global.corp.sap, job-15/job-33, cmd: StartCommand) (logid:b8ba37d1) Copy System VM patch ISO file to secondary storage. source ISO: /usr/share/cloudstack-common/vms/systemvm.iso, destination: /var/cloudstack/mnt/VM/345050029058.777ecb60/systemvm/systemvm-4.14.0.0.iso
2020-07-02 14:19:13,320 ERROR [c.c.h.v.m.VmwareManagerImpl] (DirectAgent-10:ctx-52902ebf usphlmvesxt155.phl.global.corp.sap, job-15/job-33, cmd: StartCommand) (logid:b8ba37d1) Unexpected exception
java.io.FileNotFoundException: /var/cloudstack/mnt/VM/345050029058.777ecb60/systemvm/systemvm-4.14.0.0.iso (Permission denied)
at java.base/java.io.FileOutputStream.open0(Native Method)
at java.base/java.io.FileOutputStream.open(FileOutputStream.java:298)
at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:237)
at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:187)
at org.apache.commons.io.FileUtils.doCopyFile(FileUtils.java:1123)
at org.apache.commons.io.FileUtils.copyFile(FileUtils.java:1076)
at org.apache.commons.io.FileUtils.copyFile(FileUtils.java:1028)
at com.cloud.utils.FileUtil.copyfile(FileUtil.java:30)
at com.cloud.hypervisor.vmware.manager.VmwareManagerImpl.prepareSecondaryStorageStore(VmwareManagerImpl.java:661)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy389.prepareSecondaryStorageStore(Unknown Source)
at com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:1962)
at com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:530)
at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
2020-07-02 14:19:13,321 ERROR [c.c.h.v.m.VmwareManagerImpl] (DirectAgent-10:ctx-52902ebf usphlmvesxt155.phl.global.corp.sap, job-15/job-33, cmd: StartCommand) (logid:b8ba37d1) Unable to copy systemvm ISO on secondary storage. src location: /usr/share/cloudstack-common/vms/systemvm.iso, dest location: /var/cloudstack/mnt/VM/345050029058.777ecb60/systemvm/systemvm-4.14.0.0.iso
2020-07-02 14:19:13,322 INFO [c.c.h.v.u.VmwareHelper] (DirectAgent-10:ctx-52902ebf usphlmvesxt155.phl.global.corp.sap, job-15/job-33, cmd: StartCommand) (logid:b8ba37d1) [ignored]failed to get message for exception: Unable to copy systemvm ISO on secondary storage. src location: /usr/share/cloudstack-common/vms/systemvm.iso, dest location: /var/cloudstack/mnt/VM/345050029058.777ecb60/systemvm/systemvm-4.14.0.0.iso
2020-07-02 14:19:13,323 WARN [c.c.h.v.r.VmwareResource] (DirectAgent-10:ctx-52902ebf usphlmvesxt155.phl.global.corp.sap, job-15/job-33, cmd: StartCommand) (logid:b8ba37d1) StartCommand failed due to Exception: com.cloud.utils.exception.CloudRuntimeException
Message: Unable to copy systemvm ISO on secondary storage. src location: /usr/share/cloudstack-common/vms/systemvm.iso, dest location: /var/cloudstack/mnt/VM/345050029058.777ecb60/systemvm/systemvm-4.14.0.0.iso
-----Original Message-----
From: Andrija Panic <an...@gmail.com>>
Sent: Thursday, July 2, 2020 12:34 PM
To: users <us...@cloudstack.apache.org>>
Subject: Re: Initial SystemVM Creation Failing
Hi Mike,
Mount you secondary storage on the management server to some temp folder,
then run: chown -R cloud:cloud <MOUNTPOINT>/systemvm/
Other folders on Secondary Storage NFS are owned by root, but the
"systemvm" should be owned by "cloud".
I assume you got this after some ACS upgrade, or similar?
On Thu, 2 Jul 2020 at 15:51, Corey, Mike <mi...@sap.com>> wrote:
> Hi,
>
>
>
> When I enable the zone, the systemvm creation begins but goes into a loop
> of creating and deleting VMs in vCenter. I see in the logs there is an
> error for creating the folder permissions on the secondary storage (see
> below). Running this CHMOD manually from inside the CSM kicks the same
> error.
>
>
>
> Please provide suggestions.
>
>
>
> 2020-07-02 09:08:39,665 DEBUG [c.c.h.v.m.VmwareManagerImpl]
> (DirectAgent-66:ctx-e15c32b2 usphlmvesxt154.phl.global.corp.sap,
> job-16/job-35, cmd: CopyCommand) (logid:1cfa64a9) Executing: sudo chmod
> 1777 /var/cloudstack/mnt/VM/345050029058.2807345c
>
> 2020-07-02 09:08:39,666 DEBUG [c.c.h.v.m.VmwareManagerImpl]
> (DirectAgent-66:ctx-e15c32b2 usphlmvesxt154.phl.global.corp.sap,
> job-16/job-35, cmd: CopyCommand) (logid:1cfa64a9) Executing while with
> timeout : 1440000
>
> 2020-07-02 09:08:39,693 DEBUG [c.c.h.v.m.VmwareManagerImpl]
> (DirectAgent-66:ctx-e15c32b2 usphlmvesxt154.phl.global.corp.sap,
> job-16/job-35, cmd: CopyCommand) (logid:1cfa64a9) Exit value is 1
>
> 2020-07-02 09:08:39,697 DEBUG [c.c.h.v.m.VmwareManagerImpl]
> (DirectAgent-66:ctx-e15c32b2 usphlmvesxt154.phl.global.corp.sap,
> job-16/job-35, cmd: CopyCommand) (logid:1cfa64a9) chmod: changing
> permissions of ‘/var/cloudstack/mnt/VM/345050029058.2807345c’: Operation
> not permitted
>
> 2020-07-02 09:08:39,699 WARN [c.c.h.v.m.VmwareManagerImpl]
> (DirectAgent-66:ctx-e15c32b2 usphlmvesxt154.phl.global.corp.sap,
> job-16/job-35, cmd: CopyCommand) (logid:1cfa64a9) Unable to set permissions
> for /var/cloudstack/mnt/VM/345050029058.2807345c due to chmod: changing
> permissions of ‘/var/cloudstack/mnt/VM/345050029058.2807345c’: Operation
> not permitted
>
>
>
> Lines…
>
>
>
> 2020-07-02 09:09:00,598 DEBUG [c.c.h.v.m.HostMO]
> (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) find VM v-2-VM on host
>
> 2020-07-02 09:09:00,598 INFO [c.c.h.v.m.HostMO]
> (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) VM v-2-VM not found in
> host cache
>
> 2020-07-02 09:09:00,598 DEBUG [c.c.h.v.m.HostMO]
> (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) load VM cache on host
>
> 2020-07-02 09:09:00,616 DEBUG [c.c.h.v.m.HostMO]
> (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) find VM v-2-VM on host
>
> 2020-07-02 09:09:00,617 DEBUG [c.c.h.v.m.HostMO]
> (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) VM v-2-VM found in host
> cache
>
> 2020-07-02 09:09:00,799 ERROR [c.c.h.v.m.VmwareManagerImpl]
> (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) Unable to create
> systemvm folder on secondary storage. location:
> /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm
>
> 2020-07-02 09:09:00,800 INFO [c.c.h.v.u.VmwareHelper]
> (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) [ignored]failed to get
> message for exception: Unable to create systemvm folder on secondary
> storage. location: /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm
>
> 2020-07-02 09:09:00,800 WARN [c.c.h.v.r.VmwareResource]
> (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) StartCommand failed due
> to Exception: com.cloud.utils.exception.CloudRuntimeException
>
> Message: Unable to create systemvm folder on secondary storage. location:
> /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm
>
>
>
> com.cloud.utils.exception.CloudRuntimeException: Unable to create systemvm
> folder on secondary storage. location:
> /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm
>
> at
> com.cloud.hypervisor.vmware.manager.VmwareManagerImpl.prepareSecondaryStorageStore(VmwareManagerImpl.java:648)
>
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
>
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
>
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
>
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
>
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95)
>
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
>
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
>
> at
> com.sun.proxy.$Proxy389.prepareSecondaryStorageStore(Unknown Source)
>
> at
> com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:1962)
>
> at
> com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:530)
>
> at
> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
>
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>
> at
> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
>
> at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>
> at java.base/java.lang.Thread.run(Thread.java:834)
>
> 2020-07-02 09:09:00,804 DEBUG [c.c.a.m.DirectAgentAttache]
> (DirectAgent-17:ctx-681dc524) (logid:1cfa64a9) Seq 1-4541880224203147245:
> Cancelling because one of the answers is false and it is stop on error.
>
> 2020-07-02 09:09:00,804 DEBUG [c.c.a.m.DirectAgentAttache]
> (DirectAgent-17:ctx-681dc524) (logid:1cfa64a9) Seq 1-4541880224203147245:
> Response Received:
>
> 2020-07-02 09:09:00,805 DEBUG [c.c.a.t.Request]
> (DirectAgent-17:ctx-681dc524) (logid:1cfa64a9) Seq 1-4541880224203147245:
> Processing: { Ans: , MgmtId: 345050029058, via:
> 1(usphlmvesxt154.phl.global.corp.sap), Ver: v1, Flags: 110,
> [{"com.cloud.agent.api.StartAnswer":{"vm":{"id":2,"name":"v-2-VM","bootloader":"HVM","state":"Starting","type":"ConsoleProxy","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":1073741824,"maxRam":1073741824,"hostName":"v-2-VM","arch":"x86_64","os":"Other
> Linux (64-bit)","platformEmulator":"otherLinux64Guest","bootArgs":"
> template=domP type=consoleproxy host=10.4.32.163 port=8250 name=v-2-VM
> zone=1 pod=1 guid=Proxy.2 proxy_vm=2 disable_rp_filter=true
> eth2ip=10.4.42.25 eth2mask=255.255.255.192 gateway=10.4.42.1 eth0ip=0.0.0.0
> eth0mask=0.0.0.0 eth1ip=10.4.4.35 eth1mask=255.255.252.0 mgmtcidr=
> 10.4.32.0/24 localgw=10.4.4.1 internaldns1=10.4.202.200 dns1=10.4.202.200
> dns2=10.4.12.200","enableHA":false,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"sYnwDJYgzrMKdm3ftAO-kg","params":{"vmware.reserve.mem":"false","vmware.reserve.cpu":"false","nicAdapter":"E1000"},"uuid":"fb65df73-693d-48fb-a203-c2b5e776842a","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"9a1f2ae8-6b10-4525-8a54-a175b153b789","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"50cc970f-fbe2-3b45-9cb5-b8bcdb6dbff5","id":1,"poolType":"NetworkFilesystem","host":"usphlcv300.phl.global.corp.sap","path":"/usphlmvesx_lab_cspri","port":2049,"url":"NetworkFilesystem://usphlcv300.phl.global.corp.sap/usphlmvesx_lab_cspri/?ROLE=Primary&STOREUUID=50cc970f-fbe2-3b45-9cb5-b8bcdb6dbff5","isManaged":false}},"name":"ROOT-2","size":0,"path":"ROOT-2_2","volumeId":2,"vmName":"v-2-VM","accountId":1,"format":"OVA","provisioningType":"THIN","id":2,"deviceId":0,"bytesReadRate":0,"bytesWriteRate":0,"iopsReadRate":0,"iopsWriteRate":0,"hypervisorType":"VMware","directDownload":false}},"diskSeq":0,"path":"ROOT-2_2","type":"ROOT","_details":{"storageHost":"usphlcv300.phl.global.corp.sap","managed":"false","storagePort":"2049","volumeSize":"0"}}],"nics":[{"deviceId":2,"networkRateMbps":-1,"defaultNic":true,"pxeDisable":true,"nicUuid":"ffbc1822-7907-4cdf-9e80-bb316786e04d","details":{"PromiscuousMode":"false","MacAddressChanges":"true","ForgedTransmits":"true"},"dpdkEnabled":false,"uuid":"f3a30021-a469-413d-85a8-ca1ca68ba27c","ip":"10.4.42.25","netmask":"255.255.255.192","gateway":"10.4.42.1","mac":"1e:00:e3:00:00:19","dns1":"10.4.202.200","dns2":"10.4.12.200","broadcastType":"Vlan","type":"Public","broadcastUri":"vlan://untagged","isolationUri":"vlan://untagged","isSecurityGroupEnabled":false,"name":"vSwitch0,134,vmwaresvs"},{"deviceId":0,"networkRateMbps":-1,"defaultNic":false,"pxeDisable":true,"nicUuid":"d6521be0-85b8-4836-bf8b-8461afdf593a","details":{"PromiscuousMode":"false","MacAddressChanges":"true","ForgedTransmits":"true"},"dpdkEnabled":false,"uuid":"eda0ae3d-d15d-4f91-b8db-a719cf5a94bf","mac":"02:00:31:71:00:02","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false,"name":"vSwitch0,146,vmwaresvs"},{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"pxeDisable":true,"nicUuid":"4e6201ad-91c0-4cab-bf53-9e2440025245","details":{"PromiscuousMode":"false","MacAddressChanges":"true","ForgedTransmits":"true"},"dpdkEnabled":false,"uuid":"ce88b030-c04c-4437-b60d-9910c2cc376f","ip":"10.4.4.35","netmask":"255.255.252.0","gateway":"10.4.4.1","mac":"1e:00:6b:00:00:27","broadcastType":"Native","type":"Management","isSecurityGroupEnabled":false,"name":"vSwitch0,146,vmwaresvs"}],"guestOsDetails":{},"extraConfig":{}},"result":false,"details":"StartCommand
> failed due to Exception:
> com.cloud.utils.exception.CloudRuntimeException\nMessage: Unable to create
> systemvm folder on secondary storage. location:
> /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm\n","wait":0}}] }
>
> 2020-07-02 09:09:00,805 DEBUG [c.c.a.m.AgentAttache]
> (DirectAgent-17:ctx-681dc524) (logid:1cfa64a9) Seq 1-4541880224203147245:
> No more commands found
>
> 2020-07-02 09:09:00,806 DEBUG [c.c.a.t.Request]
> (Work-Job-Executor-2:ctx-9aa38618 job-16/job-35 ctx-7ebd1b78)
> (logid:1cfa64a9) Seq 1-4541880224203147245: Received: { Ans: , MgmtId:
> 345050029058, via: 1(usphlmvesxt154.phl.global.corp.sap), Ver: v1, Flags:
> 110, { StartAnswer } }
>
> 2020-07-02 09:09:00,816 INFO [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-2:ctx-9aa38618 job-16/job-35 ctx-7ebd1b78)
> (logid:1cfa64a9) Unable to start VM on Host[-1-Routing] due to StartCommand
> failed due to Exception: com.cloud.utils.exception.CloudRuntimeException
>
> Message: Unable to create systemvm folder on secondary storage. location:
> /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm
>
>
>
> *Mike Corey*
>
>
> Technology Senior Consultant, IT CS CTW Operation & Virtualization Service
> US
>
>
> *SAP AMERICA, INC.* 3999 West Chester Pike, Newtown Square, 19073 United
> States
>
>
> T +1 610 661 0905, M +1 484 274 2658, E mike.corey@sap.com<ma...@sap.com>
>
>
>
>
>
>
>
--
Andrija Panić
RE: Initial SystemVM Creation Failing
Posted by cr...@istream.today.
I thought that you are using NFS. I had the same issue in 4.14 but I did the copy manually for the systemvm and this has fixed the problem for me, maybe it is a bug.
Regards,
Cristian
-----Original Message-----
From: Corey, Mike <mi...@sap.com>
Sent: Friday, July 10, 2020 9:11 PM
To: users@cloudstack.apache.org
Subject: RE: Initial SystemVM Creation Failing
I appreciate the response, but I'm not sure where you are going.
My secondary and primary storage are mounts from a NetApp filer. Both the CentOS (CloudStack Management) and the ESXi hosts can read/write to the secondary volume. I don't understand why cloudstack is setting the ownership and permissions to nobody when it creates the systemvm folder. That doesn't seem right.
Again, being new to CloudStack I'm not sure of the underlying mechanics of the systemvm builds.
-----Original Message-----
From: cristian.c@istream.today <cr...@istream.today>
Sent: Friday, July 10, 2020 12:30 PM
To: users@cloudstack.apache.org
Subject: RE: Initial SystemVM Creation Failing
Hello
If you check your NFS configuration you will see that somewhere you have : /etc/idmapd.conf "nobody"
Regards,
Cristian
-----Original Message-----
From: Corey, Mike <mi...@sap.com>
Sent: Friday, July 10, 2020 7:01 PM
To: users@cloudstack.apache.org
Subject: RE: Initial SystemVM Creation Failing
The system VM folder is being created as owner & permissions nobody:nobody. I have to believe that is NOT SUPPOSED to be the case.
Because the directory /var/cloudstack/mnt/VM/345050029058.4f87a734 is locked down, no other steps in the systemvm creation can continue.
I need help figuring out WHY the systemVM folder in the above directory is being created as nobody:nobody.
I can't chown that directory once CloudStack creates it during the zone enablement. I can't modify the permissions to that directory once CloudStack creates it during the zone enablement. I can't copy files to that directory once CloudStack creates it during the zone enablement.
Please identify what is occurring during the creation of the systemvm directory that would cause the ownership to be set as nobody:nobody.
Mike
Message: Unable to copy systemvm ISO on secondary storage. src location: /usr/share/cloudstack-common/vms/systemvm.iso, dest location: /var/cloudstack/mnt/VM/345050029058.4f87a734/systemvm/systemvm-4.14.0.0.iso
[root@usphlmvcsmt01 VM]# pwd
/var/cloudstack/mnt/VM
[root@usphlmvcsmt01 VM]# ls -al
total 80864
drwxr-xr-x. 3 cloud cloud 55 Jul 10 11:46 .
drwxrwx---. 3 root cloud 16 Jul 9 14:40 ..
drwxr-xr-x. 5 nobody nobody 4096 Jul 10 11:31 345050029058.4f87a734
-rw-r--r--. 1 cloud cloud 82800640 Jul 10 11:43 systemvm.iso
-----Original Message-----
From: Cristian Ciobanu <cr...@istream.today>
Sent: Friday, July 10, 2020 5:26 AM
To: users@cloudstack.apache.org
Subject: Re: Initial SystemVM Creation Failing
Just try to execute the command manually, copy the system.iso from source to the destination. I had a similar issue but is was caused by the NFS permissions.
On Wed, 8 Jul 2020, 21:44 Corey, Mike, <mi...@sap.com> wrote:
> Gone over the installation now twice (reverted to snapshot of CentOS
> VM) and still getting weird storage/permission type issues.
>
> The permissions and ownership of the systemvm folder looks correct.
> [root@ ~]# ls -al /mnt/secondary/
> total 16
> drwxr-xr-x. 4 root root 4096 Jul 8 11:15 .
> drwxr-xr-x. 4 root root 38 Jul 8 11:15 ..
> drwxrwxrwx. 12 root root 4096 Jul 8 11:05 .snapshot
> drwxrwxr-x. 2 cloud cloud 4096 Jul 8 10:58 systemvm
> drwxr-xr-x. 3 root root 4096 Jul 8 11:15 template
>
> However, I'm still seeing the copy iso error and the VMs keep creating
> and deleting with the log suggesting the vmx file of the VM doesn't
> exist on the datastore (can create files/folder on datastore
> identified in errror from vCenter).
>
> 2020-07-08 14:22:36,486 INFO [c.c.h.v.m.DatastoreMO]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Folder v-6-VM does
> not exist on datastore
> 2020-07-08 14:22:36,494 INFO [c.c.h.v.m.DatastoreMO]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Search file
> v-6-VM.vmx on [50cc970ffbe23b459cb5b8bcdb6dbff5]
> 2020-07-08 14:22:36,522 INFO [c.c.h.v.m.DatastoreMO]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) File
> [50cc970ffbe23b459cb5b8bcdb6dbff5] v-6-VM.vmx does not exist on
> datastore
> 2020-07-08 14:22:36,522 INFO [c.c.h.v.m.DatastoreMO]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Searching file
> v-6-VM.vmx in [50cc970ffbe23b459cb5b8bcdb6dbff5]
> 2020-07-08 14:22:36,641 INFO [c.c.h.v.m.HypervisorHostHelper]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Create blank VM.
> cpuCount: 1, cpuSpeed(MHz): 500, mem(Mb): 1024
> 2020-07-08 14:22:36,692 DEBUG [c.c.h.v.m.ClusterMO]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Retrieved
> recommended disk controller for guest OS : otherLinux64Guest in cluster PHL-CSM01 :
> VirtualLsiLogicController
> 2020-07-08 14:22:37,115 DEBUG [c.c.h.v.m.HostMO]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) find VM v-6-VM on
> host
> 2020-07-08 14:22:37,115 INFO [c.c.h.v.m.HostMO]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) VM v-6-VM not found
> in host cache
> 2020-07-08 14:22:37,115 DEBUG [c.c.h.v.m.HostMO]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) load VM cache on
> host
> 2020-07-08 14:22:37,155 DEBUG [c.c.h.v.m.HostMO]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) find VM v-6-VM on
> host
> 2020-07-08 14:22:37,155 DEBUG [c.c.h.v.m.HostMO]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) VM v-6-VM found in
> host cache
> 2020-07-08 14:22:37,375 INFO [c.c.h.v.m.VmwareManagerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Inject SSH key pairs
> before copying systemvm.iso into secondary storage
> 2020-07-08 14:22:37,377 INFO [c.c.s.ConfigurationServerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Processing
> updateKeyPairs
> 2020-07-08 14:22:37,377 INFO [c.c.s.ConfigurationServerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Keypairs already in
> database, updating local copy
> 2020-07-08 14:22:37,381 INFO [c.c.s.ConfigurationServerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Going to update
> systemvm iso with generated keypairs if needed
> 2020-07-08 14:22:37,381 INFO [c.c.s.ConfigurationServerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Trying to inject
> public and private keys into systemvm iso
> 2020-07-08 14:22:37,381 DEBUG [c.c.u.s.Script]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Looking for
> scripts/vm/systemvm/injectkeys.sh in the classpath
> 2020-07-08 14:22:37,381 DEBUG [c.c.u.s.Script]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) System resource:
> file:/usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh
> 2020-07-08 14:22:37,381 DEBUG [c.c.u.s.Script]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Absolute path =
> /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh
> 2020-07-08 14:22:37,381 DEBUG [c.c.u.s.Script]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Looking for
> vms/systemvm.iso in the classpath
> 2020-07-08 14:22:37,381 DEBUG [c.c.u.s.Script]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) System resource:
> file:/usr/share/cloudstack-common/vms/systemvm.iso
> 2020-07-08 14:22:37,381 DEBUG [c.c.u.s.Script]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Absolute path =
> /usr/share/cloudstack-common/vms/systemvm.iso
> 2020-07-08 14:22:37,381 DEBUG [c.c.s.ConfigurationServerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Executing: /bin/bash
> /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh
> /var/cloudstack/management/.ssh/id_rsa.pub
> /var/cloudstack/management/.ssh/id_rsa
> /usr/share/cloudstack-common/vms/systemvm.iso
> 2020-07-08 14:22:37,384 DEBUG [c.c.s.ConfigurationServerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Executing while with
> timeout : 3600000
> 2020-07-08 14:22:37,487 DEBUG [c.c.s.ConfigurationServerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Execution is successful.
> 2020-07-08 14:22:37,494 DEBUG [c.c.s.ConfigurationServerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) mount: /dev/loop0 is
> write-protected, mounting read-only New public key is the same as the
> one in the systemvm.iso, not injecting it, not modifying systemvm.iso
>
> 2020-07-08 14:22:37,494 INFO [c.c.s.ConfigurationServerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) The script
> injectkeys.sh was run with result : null
> 2020-07-08 14:22:37,494 INFO [c.c.h.v.m.VmwareManagerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Copy System VM patch
> ISO file to secondary storage. source ISO:
> /usr/share/cloudstack-common/vms/systemvm.iso, destination:
> /var/cloudstack/mnt/VM/345050029058.1f7af2bf/systemvm/systemvm-4.14.0.
> 0.iso
> 2020-07-08 14:22:37,495 ERROR [c.c.h.v.m.VmwareManagerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Unexpected exception
> java.io.FileNotFoundException:
> /var/cloudstack/mnt/VM/345050029058.1f7af2bf/systemvm/systemvm-4.14.0.
> 0.iso
> (Permission denied)
>
> But, the ownership/permissions look okay on that destination directory too.
>
> [root@ ~]# ls -al /var/cloudstack/mnt/VM/345050029058.1f7af2bf
> total 12
> drwxr-xr-x. 4 nobody nobody 4096 Jul 8 11:15 .
> drwxr-xr-x. 3 cloud cloud 35 Jul 8 14:06 ..
> drwxrwxr-x. 2 cloud cloud 4096 Jul 8 10:58 systemvm drwxr-xr-x. 3
> nobody nobody 4096 Jul 8 11:15 template
> [root@usphlmvcsmt01 ~]#
>
>
> Any ideas?
>
>
> -----Original Message-----
> From: Corey, Mike
> Sent: Thursday, July 2, 2020 2:31 PM
> To: users@cloudstack.apache.org
> Subject: RE: Initial SystemVM Creation Failing
>
> No upgrade - this again is the fresh install of CloudStack 4.14 with
> VMware resources. Maybe I should note that I created the primary and
> secondary storage via the "Advanced" setup wizard from the UI (logged
> in as admin).
>
> I performed the ownership change you suggested so now cloud owns the
> systemvm folder on the secondary storage. However, now I'm getting a
> new error regarding copying the ISO.
>
> Should I just bag the 4.14 installation and try to get 4.13.1 running
> with VMware?
>
>
> 2020-07-02 14:19:13,314 INFO [c.c.h.v.m.VmwareManagerImpl]
> (DirectAgent-10:ctx-52902ebf usphlmvesxt155.phl.global.corp.sap,
> job-15/job-33, cmd: StartCommand) (logid:b8ba37d1) Copy System VM
> patch ISO file to secondary storage. source ISO:
> /usr/share/cloudstack-common/vms/systemvm.iso, destination:
> /var/cloudstack/mnt/VM/345050029058.777ecb60/systemvm/systemvm-4.14.0.
> 0.iso
> 2020-07-02 14:19:13,320 ERROR [c.c.h.v.m.VmwareManagerImpl]
> (DirectAgent-10:ctx-52902ebf usphlmvesxt155.phl.global.corp.sap,
> job-15/job-33, cmd: StartCommand) (logid:b8ba37d1) Unexpected
> exception
> java.io.FileNotFoundException:
> /var/cloudstack/mnt/VM/345050029058.777ecb60/systemvm/systemvm-4.14.0.
> 0.iso
> (Permission denied)
> at java.base/java.io.FileOutputStream.open0(Native Method)
> at java.base/java.io
> .FileOutputStream.open(FileOutputStream.java:298)
> at java.base/java.io
> .FileOutputStream.<init>(FileOutputStream.java:237)
> at java.base/java.io
> .FileOutputStream.<init>(FileOutputStream.java:187)
> at org.apache.commons.io.FileUtils.doCopyFile(FileUtils.java:1123)
> at org.apache.commons.io.FileUtils.copyFile(FileUtils.java:1076)
> at org.apache.commons.io.FileUtils.copyFile(FileUtils.java:1028)
> at com.cloud.utils.FileUtil.copyfile(FileUtil.java:30)
> at
> com.cloud.hypervisor.vmware.manager.VmwareManagerImpl.prepareSecondaryStorageStore(VmwareManagerImpl.java:661)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
> at
> com.sun.proxy.$Proxy389.prepareSecondaryStorageStore(Unknown
> Source)
> at
> com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:1962)
> at
> com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:530)
> at
> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at
> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2020-07-02 14:19:13,321 ERROR [c.c.h.v.m.VmwareManagerImpl]
> (DirectAgent-10:ctx-52902ebf usphlmvesxt155.phl.global.corp.sap,
> job-15/job-33, cmd: StartCommand) (logid:b8ba37d1) Unable to copy
> systemvm ISO on secondary storage. src location:
> /usr/share/cloudstack-common/vms/systemvm.iso, dest location:
> /var/cloudstack/mnt/VM/345050029058.777ecb60/systemvm/systemvm-4.14.0.
> 0.iso
> 2020-07-02 14:19:13,322 INFO [c.c.h.v.u.VmwareHelper]
> (DirectAgent-10:ctx-52902ebf usphlmvesxt155.phl.global.corp.sap,
> job-15/job-33, cmd: StartCommand) (logid:b8ba37d1) [ignored]failed to
> get message for exception: Unable to copy systemvm ISO on secondary storage.
> src location: /usr/share/cloudstack-common/vms/systemvm.iso, dest location:
> /var/cloudstack/mnt/VM/345050029058.777ecb60/systemvm/systemvm-4.14.0.
> 0.iso
> 2020-07-02 14:19:13,323 WARN [c.c.h.v.r.VmwareResource]
> (DirectAgent-10:ctx-52902ebf usphlmvesxt155.phl.global.corp.sap,
> job-15/job-33, cmd: StartCommand) (logid:b8ba37d1) StartCommand failed
> due to Exception: com.cloud.utils.exception.CloudRuntimeException
> Message: Unable to copy systemvm ISO on secondary storage. src location:
> /usr/share/cloudstack-common/vms/systemvm.iso, dest location:
> /var/cloudstack/mnt/VM/345050029058.777ecb60/systemvm/systemvm-4.14.0.
> 0.iso
>
>
>
>
> -----Original Message-----
> From: Andrija Panic <an...@gmail.com>
> Sent: Thursday, July 2, 2020 12:34 PM
> To: users <us...@cloudstack.apache.org>
> Subject: Re: Initial SystemVM Creation Failing
>
> Hi Mike,
>
> Mount you secondary storage on the management server to some temp folder,
> then run: chown -R cloud:cloud <MOUNTPOINT>/systemvm/
> Other folders on Secondary Storage NFS are owned by root, but the
> "systemvm" should be owned by "cloud".
>
> I assume you got this after some ACS upgrade, or similar?
>
> On Thu, 2 Jul 2020 at 15:51, Corey, Mike <mi...@sap.com> wrote:
>
> > Hi,
> >
> >
> >
> > When I enable the zone, the systemvm creation begins but goes into a
> > loop of creating and deleting VMs in vCenter. I see in the logs
> > there is an error for creating the folder permissions on the
> > secondary storage (see below). Running this CHMOD manually from
> > inside the CSM kicks the same error.
> >
> >
> >
> > Please provide suggestions.
> >
> >
> >
> > 2020-07-02 09:08:39,665 DEBUG [c.c.h.v.m.VmwareManagerImpl]
> > (DirectAgent-66:ctx-e15c32b2 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: CopyCommand) (logid:1cfa64a9) Executing: sudo
> > chmod
> > 1777 /var/cloudstack/mnt/VM/345050029058.2807345c
> >
> > 2020-07-02 09:08:39,666 DEBUG [c.c.h.v.m.VmwareManagerImpl]
> > (DirectAgent-66:ctx-e15c32b2 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: CopyCommand) (logid:1cfa64a9) Executing while
> > with timeout : 1440000
> >
> > 2020-07-02 09:08:39,693 DEBUG [c.c.h.v.m.VmwareManagerImpl]
> > (DirectAgent-66:ctx-e15c32b2 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: CopyCommand) (logid:1cfa64a9) Exit value is 1
> >
> > 2020-07-02 09:08:39,697 DEBUG [c.c.h.v.m.VmwareManagerImpl]
> > (DirectAgent-66:ctx-e15c32b2 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: CopyCommand) (logid:1cfa64a9) chmod: changing
> > permissions of ‘/var/cloudstack/mnt/VM/345050029058.2807345c’:
> > Operation not permitted
> >
> > 2020-07-02 09:08:39,699 WARN [c.c.h.v.m.VmwareManagerImpl]
> > (DirectAgent-66:ctx-e15c32b2 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: CopyCommand) (logid:1cfa64a9) Unable to set
> permissions
> > for /var/cloudstack/mnt/VM/345050029058.2807345c due to chmod:
> > changing permissions of
> > ‘/var/cloudstack/mnt/VM/345050029058.2807345c’: Operation not
> > permitted
> >
> >
> >
> > Lines…
> >
> >
> >
> > 2020-07-02 09:09:00,598 DEBUG [c.c.h.v.m.HostMO]
> > (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) find VM v-2-VM on
> > host
> >
> > 2020-07-02 09:09:00,598 INFO [c.c.h.v.m.HostMO]
> > (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) VM v-2-VM not
> > found in host cache
> >
> > 2020-07-02 09:09:00,598 DEBUG [c.c.h.v.m.HostMO]
> > (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) load VM cache on
> > host
> >
> > 2020-07-02 09:09:00,616 DEBUG [c.c.h.v.m.HostMO]
> > (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) find VM v-2-VM on
> > host
> >
> > 2020-07-02 09:09:00,617 DEBUG [c.c.h.v.m.HostMO]
> > (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) VM v-2-VM found
> > in
> host
> > cache
> >
> > 2020-07-02 09:09:00,799 ERROR [c.c.h.v.m.VmwareManagerImpl]
> > (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) Unable to create
> > systemvm folder on secondary storage. location:
> > /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm
> >
> > 2020-07-02 09:09:00,800 INFO [c.c.h.v.u.VmwareHelper]
> > (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) [ignored]failed
> > to get message for exception: Unable to create systemvm folder on
> > secondary storage. location:
> > /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm
> >
> > 2020-07-02 09:09:00,800 WARN [c.c.h.v.r.VmwareResource]
> > (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) StartCommand
> > failed
> due
> > to Exception: com.cloud.utils.exception.CloudRuntimeException
> >
> > Message: Unable to create systemvm folder on secondary storage. location:
> > /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm
> >
> >
> >
> > com.cloud.utils.exception.CloudRuntimeException: Unable to create
> systemvm
> > folder on secondary storage. location:
> > /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm
> >
> > at
> >
> com.cloud.hypervisor.vmware.manager.VmwareManagerImpl.prepareSecondary
> StorageStore(VmwareManagerImpl.java:648)
> >
> > at
> > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Nati
> > ve
> > Method)
> >
> > at
> >
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeM
> ethodAccessorImpl.java:62)
> >
> > at
> >
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Del
> egatingMethodAccessorImpl.java:43)
> >
> > at
> > java.base/java.lang.reflect.Method.invoke(Method.java:566)
> >
> > at
> >
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflectio
> n(AopUtils.java:344)
> >
> > at
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoi
> npoint(ReflectiveMethodInvocation.java:198)
> >
> > at
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(R
> eflectiveMethodInvocation.java:163)
> >
> > at
> >
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke
> (ExposeInvocationInterceptor.java:95)
> >
> > at
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(R
> eflectiveMethodInvocation.java:186)
> >
> > at
> >
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamic
> AopProxy.java:212)
> >
> > at
> > com.sun.proxy.$Proxy389.prepareSecondaryStorageStore(Unknown Source)
> >
> > at
> >
> com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareReso
> urce.java:1962)
> >
> > at
> >
> com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(Vmw
> areResource.java:530)
> >
> > at
> >
> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAge
> ntAttache.java:315)
> >
> > at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(Man
> agedContextRunnable.java:49)
> >
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.cal
> l(DefaultManagedContext.java:56)
> >
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callW
> ithContext(DefaultManagedContext.java:103)
> >
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWi
> thContext(DefaultManagedContext.java:53)
> >
> > at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(Manag
> edContextRunnable.java:46)
> >
> > at
> >
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executor
> s.java:515)
> >
> > at
> > java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> >
> > at
> >
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
> tureTask.run(ScheduledThreadPoolExecutor.java:304)
> >
> > at
> >
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
> Executor.java:1128)
> >
> > at
> >
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
> lExecutor.java:628)
> >
> > at java.base/java.lang.Thread.run(Thread.java:834)
> >
> > 2020-07-02 09:09:00,804 DEBUG [c.c.a.m.DirectAgentAttache]
> > (DirectAgent-17:ctx-681dc524) (logid:1cfa64a9) Seq 1-4541880224203147245:
> > Cancelling because one of the answers is false and it is stop on error.
> >
> > 2020-07-02 09:09:00,804 DEBUG [c.c.a.m.DirectAgentAttache]
> > (DirectAgent-17:ctx-681dc524) (logid:1cfa64a9) Seq 1-4541880224203147245:
> > Response Received:
> >
> > 2020-07-02 09:09:00,805 DEBUG [c.c.a.t.Request]
> > (DirectAgent-17:ctx-681dc524) (logid:1cfa64a9) Seq 1-4541880224203147245:
> > Processing: { Ans: , MgmtId: 345050029058, via:
> > 1(usphlmvesxt154.phl.global.corp.sap), Ver: v1, Flags: 110,
> >
> [{"com.cloud.agent.api.StartAnswer":{"vm":{"id":2,"name":"v-2-VM","boo
> tloader":"HVM","state":"Starting","type":"ConsoleProxy","cpus":1,"minS
> peed":500,"maxSpeed":500,"minRam":1073741824,"maxRam":1073741824,"host
> Name":"v-2-VM","arch":"x86_64","os":"Other
> > Linux (64-bit)","platformEmulator":"otherLinux64Guest","bootArgs":"
> > template=domP type=consoleproxy host=10.4.32.163 port=8250
> > name=v-2-VM
> > zone=1 pod=1 guid=Proxy.2 proxy_vm=2 disable_rp_filter=true
> > eth2ip=10.4.42.25 eth2mask=255.255.255.192 gateway=10.4.42.1
> eth0ip=0.0.0.0
> > eth0mask=0.0.0.0 eth1ip=10.4.4.35 eth1mask=255.255.252.0 mgmtcidr=
> > 10.4.32.0/24 localgw=10.4.4.1 internaldns1=10.4.202.200
> dns1=10.4.202.200
> >
> dns2=10.4.12.200","enableHA":false,"limitCpuUse":false,"enableDynamica
> llyScaleVm":false,"vncPassword":"sYnwDJYgzrMKdm3ftAO-kg","params":{"vm
> ware.reserve.mem":"false","vmware.reserve.cpu":"false","nicAdapter":"E
> 1000"},"uuid":"fb65df73-693d-48fb-a203-c2b5e776842a","disks":[{"data":
> {"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"9a1f2ae8-6
> b10-4525-8a54-a175b153b789","volumeType":"ROOT","dataStore":{"org.apac
> he.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"50cc970f-fbe2-3b
> 45-9cb5-b8bcdb6dbff5","id":1,"poolType":"NetworkFilesystem","host":"us
> phlcv300.phl.global.corp.sap","path":"/usphlmvesx_lab_cspri","port":20
> 49,"url":"NetworkFilesystem://usphlcv300.phl.global.corp.sap/usphlmves
> x_lab_cspri/?ROLE=Primary&STOREUUID=50cc970f-fbe2-3b45-9cb5-b8bcdb6dbf
> f5","isManaged":false}},"name":"ROOT-2","size":0,"path":"ROOT-2_2","vo
> lumeId":2,"vmName":"v-2-VM","accountId":1,"format":"OVA","provisioning
> Type":"THIN","id":2,"deviceId":0,"bytesReadRate":0,"bytesWriteRate":0,
> "iopsReadRate":0,"iopsWriteRate":0,"hypervisorType":"VMware","directDo
> wnload":false}},"diskSeq":0,"path":"ROOT-2_2","type":"ROOT","_details"
> :{"storageHost":"usphlcv300.phl.global.corp.sap","managed":"false","st
> oragePort":"2049","volumeSize":"0"}}],"nics":[{"deviceId":2,"networkRa
> teMbps":-1,"defaultNic":true,"pxeDisable":true,"nicUuid":"ffbc1822-790
> 7-4cdf-9e80-bb316786e04d","details":{"PromiscuousMode":"false","MacAdd
> ressChanges":"true","ForgedTransmits":"true"},"dpdkEnabled":false,"uui
> d":"f3a30021-a469-413d-85a8-ca1ca68ba27c","ip":"10.4.42.25","netmask":
> "255.255.255.192","gateway":"10.4.42.1","mac":"1e:00:e3:00:00:19","dns
> 1":"10.4.202.200","dns2":"10.4.12.200","broadcastType":"Vlan","type":"
> Public","broadcastUri":"vlan://untagged","isolationUri":"vlan://untagg
> ed","isSecurityGroupEnabled":false,"name":"vSwitch0,134,vmwaresvs"},{"
> deviceId":0,"networkRateMbps":-1,"defaultNic":false,"pxeDisable":true,
> "nicUuid":"d6521be0-85b8-4836-bf8b-8461afdf593a","details":{"Promiscuo
> usMode":"false","MacAddressChanges":"true","ForgedTransmits":"true"},"
> dpdkEnabled":false,"uuid":"eda0ae3d-d15d-4f91-b8db-a719cf5a94bf","mac"
> :"02:00:31:71:00:02","broadcastType":"LinkLocal","type":"Control","isS
> ecurityGroupEnabled":false,"name":"vSwitch0,146,vmwaresvs"},{"deviceId
> ":1,"networkRateMbps":-1,"defaultNic":false,"pxeDisable":true,"nicUuid
> ":"4e6201ad-91c0-4cab-bf53-9e2440025245","details":{"PromiscuousMode":
> "false","MacAddressChanges":"true","ForgedTransmits":"true"},"dpdkEnab
> led":false,"uuid":"ce88b030-c04c-4437-b60d-9910c2cc376f","ip":"10.4.4.
> 35","netmask":"255.255.252.0","gateway":"10.4.4.1","mac":"1e:00:6b:00:
> 00:27","broadcastType":"Native","type":"Management","isSecurityGroupEn
> abled":false,"name":"vSwitch0,146,vmwaresvs"}],"guestOsDetails":{},"ex
> traConfig":{}},"result":false,"details":"StartCommand
> > failed due to Exception:
> > com.cloud.utils.exception.CloudRuntimeException\nMessage: Unable to
> create
> > systemvm folder on secondary storage. location:
> > /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm\n","wait":0}}]
> > }
> >
> > 2020-07-02 09:09:00,805 DEBUG [c.c.a.m.AgentAttache]
> > (DirectAgent-17:ctx-681dc524) (logid:1cfa64a9) Seq 1-4541880224203147245:
> > No more commands found
> >
> > 2020-07-02 09:09:00,806 DEBUG [c.c.a.t.Request]
> > (Work-Job-Executor-2:ctx-9aa38618 job-16/job-35 ctx-7ebd1b78)
> > (logid:1cfa64a9) Seq 1-4541880224203147245: Received: { Ans: , MgmtId:
> > 345050029058, via: 1(usphlmvesxt154.phl.global.corp.sap), Ver: v1, Flags:
> > 110, { StartAnswer } }
> >
> > 2020-07-02 09:09:00,816 INFO [c.c.v.VirtualMachineManagerImpl]
> > (Work-Job-Executor-2:ctx-9aa38618 job-16/job-35 ctx-7ebd1b78)
> > (logid:1cfa64a9) Unable to start VM on Host[-1-Routing] due to
> StartCommand
> > failed due to Exception:
> > com.cloud.utils.exception.CloudRuntimeException
> >
> > Message: Unable to create systemvm folder on secondary storage. location:
> > /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm
> >
> >
> >
> > *Mike Corey*
> >
> >
> > Technology Senior Consultant, IT CS CTW Operation & Virtualization
> Service
> > US
> >
> >
> > *SAP AMERICA, INC.* 3999 West Chester Pike, Newtown Square, 19073
> > United States
> >
> >
> > T +1 610 661 0905, M +1 484 274 2658, E mike.corey@sap.com
> >
> >
> >
> >
> >
> >
> >
>
>
> --
>
> Andrija Panić
>
RE: Initial SystemVM Creation Failing
Posted by "Corey, Mike" <mi...@sap.com>.
Update to the forum in case others run into the same issues I've seen so far.
1 - I have to manually disable firewalld on my CentOS to get the Management UI to come up. I guess whatever firewall settings the cloudstack-management setup command runs either don't apply or save in my case.
2 - I have edited the /etc/idmapd.conf and modified/uncommented the Nobody-User=root & Nobody-Group=root
- I discovered that CSM uses NFS4 to mount the secondary storage inside the CentOS/CSM server. This got me around many of the permissions issues where "nobody" was set on the file/folder attributes.
3 - I set the global setting in CSM to use NFS version 3: secstorage.nfs.version
- This looks like it only affects how the secondary mount is connected to the ESXi host
Once I made all three of the above changes I was able to get the two system VMs deployed and running. However, I'm getting an error with a template download now. I did some of the trouble shooting that was suggested at this link (https://cwiki.apache.org/confluence/display/CLOUDSTACK/SSVM%2C+templates%2C+Secondary+storage+troubleshooting) however, the error continues and I can't use the template from the UI.
message:: Failed to register template: 6b2fc608-c603-11ea-9d37-005056959c02 with error: No route to host (Host unreachable)
I've also tried to upload an ISO form the UI and I get "Failed to upload file due to system misconfiguration. Please contact admin."
Again I've found entries in the various online forums referencing such errors; however, I can't seem to find a clear cut workaround/solution. Many thanks to the responses!
Mike
-----Original Message-----
From: Corey, Mike
Sent: Friday, July 10, 2020 2:11 PM
To: users@cloudstack.apache.org
Subject: RE: Initial SystemVM Creation Failing
I appreciate the response, but I'm not sure where you are going.
My secondary and primary storage are mounts from a NetApp filer. Both the CentOS (CloudStack Management) and the ESXi hosts can read/write to the secondary volume. I don't understand why cloudstack is setting the ownership and permissions to nobody when it creates the systemvm folder. That doesn't seem right.
Again, being new to CloudStack I'm not sure of the underlying mechanics of the systemvm builds.
-----Original Message-----
From: cristian.c@istream.today <cr...@istream.today>
Sent: Friday, July 10, 2020 12:30 PM
To: users@cloudstack.apache.org
Subject: RE: Initial SystemVM Creation Failing
Hello
If you check your NFS configuration you will see that somewhere you have : /etc/idmapd.conf "nobody"
Regards,
Cristian
-----Original Message-----
From: Corey, Mike <mi...@sap.com>
Sent: Friday, July 10, 2020 7:01 PM
To: users@cloudstack.apache.org
Subject: RE: Initial SystemVM Creation Failing
The system VM folder is being created as owner & permissions nobody:nobody. I have to believe that is NOT SUPPOSED to be the case.
Because the directory /var/cloudstack/mnt/VM/345050029058.4f87a734 is locked down, no other steps in the systemvm creation can continue.
I need help figuring out WHY the systemVM folder in the above directory is being created as nobody:nobody.
I can't chown that directory once CloudStack creates it during the zone enablement. I can't modify the permissions to that directory once CloudStack creates it during the zone enablement. I can't copy files to that directory once CloudStack creates it during the zone enablement.
Please identify what is occurring during the creation of the systemvm directory that would cause the ownership to be set as nobody:nobody.
Mike
Message: Unable to copy systemvm ISO on secondary storage. src location: /usr/share/cloudstack-common/vms/systemvm.iso, dest location: /var/cloudstack/mnt/VM/345050029058.4f87a734/systemvm/systemvm-4.14.0.0.iso
[root@usphlmvcsmt01 VM]# pwd
/var/cloudstack/mnt/VM
[root@usphlmvcsmt01 VM]# ls -al
total 80864
drwxr-xr-x. 3 cloud cloud 55 Jul 10 11:46 .
drwxrwx---. 3 root cloud 16 Jul 9 14:40 ..
drwxr-xr-x. 5 nobody nobody 4096 Jul 10 11:31 345050029058.4f87a734
-rw-r--r--. 1 cloud cloud 82800640 Jul 10 11:43 systemvm.iso
-----Original Message-----
From: Cristian Ciobanu <cr...@istream.today>
Sent: Friday, July 10, 2020 5:26 AM
To: users@cloudstack.apache.org
Subject: Re: Initial SystemVM Creation Failing
Just try to execute the command manually, copy the system.iso from source to the destination. I had a similar issue but is was caused by the NFS permissions.
On Wed, 8 Jul 2020, 21:44 Corey, Mike, <mi...@sap.com> wrote:
> Gone over the installation now twice (reverted to snapshot of CentOS
> VM) and still getting weird storage/permission type issues.
>
> The permissions and ownership of the systemvm folder looks correct.
> [root@ ~]# ls -al /mnt/secondary/
> total 16
> drwxr-xr-x. 4 root root 4096 Jul 8 11:15 .
> drwxr-xr-x. 4 root root 38 Jul 8 11:15 ..
> drwxrwxrwx. 12 root root 4096 Jul 8 11:05 .snapshot
> drwxrwxr-x. 2 cloud cloud 4096 Jul 8 10:58 systemvm
> drwxr-xr-x. 3 root root 4096 Jul 8 11:15 template
>
> However, I'm still seeing the copy iso error and the VMs keep creating
> and deleting with the log suggesting the vmx file of the VM doesn't
> exist on the datastore (can create files/folder on datastore
> identified in errror from vCenter).
>
> 2020-07-08 14:22:36,486 INFO [c.c.h.v.m.DatastoreMO]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Folder v-6-VM does
> not exist on datastore
> 2020-07-08 14:22:36,494 INFO [c.c.h.v.m.DatastoreMO]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Search file
> v-6-VM.vmx on [50cc970ffbe23b459cb5b8bcdb6dbff5]
> 2020-07-08 14:22:36,522 INFO [c.c.h.v.m.DatastoreMO]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) File
> [50cc970ffbe23b459cb5b8bcdb6dbff5] v-6-VM.vmx does not exist on
> datastore
> 2020-07-08 14:22:36,522 INFO [c.c.h.v.m.DatastoreMO]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Searching file
> v-6-VM.vmx in [50cc970ffbe23b459cb5b8bcdb6dbff5]
> 2020-07-08 14:22:36,641 INFO [c.c.h.v.m.HypervisorHostHelper]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Create blank VM.
> cpuCount: 1, cpuSpeed(MHz): 500, mem(Mb): 1024
> 2020-07-08 14:22:36,692 DEBUG [c.c.h.v.m.ClusterMO]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Retrieved
> recommended disk controller for guest OS : otherLinux64Guest in cluster PHL-CSM01 :
> VirtualLsiLogicController
> 2020-07-08 14:22:37,115 DEBUG [c.c.h.v.m.HostMO]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) find VM v-6-VM on
> host
> 2020-07-08 14:22:37,115 INFO [c.c.h.v.m.HostMO]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) VM v-6-VM not found
> in host cache
> 2020-07-08 14:22:37,115 DEBUG [c.c.h.v.m.HostMO]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) load VM cache on
> host
> 2020-07-08 14:22:37,155 DEBUG [c.c.h.v.m.HostMO]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) find VM v-6-VM on
> host
> 2020-07-08 14:22:37,155 DEBUG [c.c.h.v.m.HostMO]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) VM v-6-VM found in
> host cache
> 2020-07-08 14:22:37,375 INFO [c.c.h.v.m.VmwareManagerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Inject SSH key pairs
> before copying systemvm.iso into secondary storage
> 2020-07-08 14:22:37,377 INFO [c.c.s.ConfigurationServerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Processing
> updateKeyPairs
> 2020-07-08 14:22:37,377 INFO [c.c.s.ConfigurationServerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Keypairs already in
> database, updating local copy
> 2020-07-08 14:22:37,381 INFO [c.c.s.ConfigurationServerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Going to update
> systemvm iso with generated keypairs if needed
> 2020-07-08 14:22:37,381 INFO [c.c.s.ConfigurationServerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Trying to inject
> public and private keys into systemvm iso
> 2020-07-08 14:22:37,381 DEBUG [c.c.u.s.Script]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Looking for
> scripts/vm/systemvm/injectkeys.sh in the classpath
> 2020-07-08 14:22:37,381 DEBUG [c.c.u.s.Script]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) System resource:
> file:/usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh
> 2020-07-08 14:22:37,381 DEBUG [c.c.u.s.Script]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Absolute path =
> /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh
> 2020-07-08 14:22:37,381 DEBUG [c.c.u.s.Script]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Looking for
> vms/systemvm.iso in the classpath
> 2020-07-08 14:22:37,381 DEBUG [c.c.u.s.Script]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) System resource:
> file:/usr/share/cloudstack-common/vms/systemvm.iso
> 2020-07-08 14:22:37,381 DEBUG [c.c.u.s.Script]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Absolute path =
> /usr/share/cloudstack-common/vms/systemvm.iso
> 2020-07-08 14:22:37,381 DEBUG [c.c.s.ConfigurationServerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Executing: /bin/bash
> /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh
> /var/cloudstack/management/.ssh/id_rsa.pub
> /var/cloudstack/management/.ssh/id_rsa
> /usr/share/cloudstack-common/vms/systemvm.iso
> 2020-07-08 14:22:37,384 DEBUG [c.c.s.ConfigurationServerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Executing while with
> timeout : 3600000
> 2020-07-08 14:22:37,487 DEBUG [c.c.s.ConfigurationServerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Execution is successful.
> 2020-07-08 14:22:37,494 DEBUG [c.c.s.ConfigurationServerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) mount: /dev/loop0 is
> write-protected, mounting read-only New public key is the same as the
> one in the systemvm.iso, not injecting it, not modifying systemvm.iso
>
> 2020-07-08 14:22:37,494 INFO [c.c.s.ConfigurationServerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) The script
> injectkeys.sh was run with result : null
> 2020-07-08 14:22:37,494 INFO [c.c.h.v.m.VmwareManagerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Copy System VM patch
> ISO file to secondary storage. source ISO:
> /usr/share/cloudstack-common/vms/systemvm.iso, destination:
> /var/cloudstack/mnt/VM/345050029058.1f7af2bf/systemvm/systemvm-4.14.0.
> 0.iso
> 2020-07-08 14:22:37,495 ERROR [c.c.h.v.m.VmwareManagerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Unexpected exception
> java.io.FileNotFoundException:
> /var/cloudstack/mnt/VM/345050029058.1f7af2bf/systemvm/systemvm-4.14.0.
> 0.iso
> (Permission denied)
>
> But, the ownership/permissions look okay on that destination directory too.
>
> [root@ ~]# ls -al /var/cloudstack/mnt/VM/345050029058.1f7af2bf
> total 12
> drwxr-xr-x. 4 nobody nobody 4096 Jul 8 11:15 .
> drwxr-xr-x. 3 cloud cloud 35 Jul 8 14:06 ..
> drwxrwxr-x. 2 cloud cloud 4096 Jul 8 10:58 systemvm drwxr-xr-x. 3
> nobody nobody 4096 Jul 8 11:15 template
> [root@usphlmvcsmt01 ~]#
>
>
> Any ideas?
>
>
> -----Original Message-----
> From: Corey, Mike
> Sent: Thursday, July 2, 2020 2:31 PM
> To: users@cloudstack.apache.org
> Subject: RE: Initial SystemVM Creation Failing
>
> No upgrade - this again is the fresh install of CloudStack 4.14 with
> VMware resources. Maybe I should note that I created the primary and
> secondary storage via the "Advanced" setup wizard from the UI (logged
> in as admin).
>
> I performed the ownership change you suggested so now cloud owns the
> systemvm folder on the secondary storage. However, now I'm getting a
> new error regarding copying the ISO.
>
> Should I just bag the 4.14 installation and try to get 4.13.1 running
> with VMware?
>
>
> 2020-07-02 14:19:13,314 INFO [c.c.h.v.m.VmwareManagerImpl]
> (DirectAgent-10:ctx-52902ebf usphlmvesxt155.phl.global.corp.sap,
> job-15/job-33, cmd: StartCommand) (logid:b8ba37d1) Copy System VM
> patch ISO file to secondary storage. source ISO:
> /usr/share/cloudstack-common/vms/systemvm.iso, destination:
> /var/cloudstack/mnt/VM/345050029058.777ecb60/systemvm/systemvm-4.14.0.
> 0.iso
> 2020-07-02 14:19:13,320 ERROR [c.c.h.v.m.VmwareManagerImpl]
> (DirectAgent-10:ctx-52902ebf usphlmvesxt155.phl.global.corp.sap,
> job-15/job-33, cmd: StartCommand) (logid:b8ba37d1) Unexpected
> exception
> java.io.FileNotFoundException:
> /var/cloudstack/mnt/VM/345050029058.777ecb60/systemvm/systemvm-4.14.0.
> 0.iso
> (Permission denied)
> at java.base/java.io.FileOutputStream.open0(Native Method)
> at java.base/java.io
> .FileOutputStream.open(FileOutputStream.java:298)
> at java.base/java.io
> .FileOutputStream.<init>(FileOutputStream.java:237)
> at java.base/java.io
> .FileOutputStream.<init>(FileOutputStream.java:187)
> at org.apache.commons.io.FileUtils.doCopyFile(FileUtils.java:1123)
> at org.apache.commons.io.FileUtils.copyFile(FileUtils.java:1076)
> at org.apache.commons.io.FileUtils.copyFile(FileUtils.java:1028)
> at com.cloud.utils.FileUtil.copyfile(FileUtil.java:30)
> at
> com.cloud.hypervisor.vmware.manager.VmwareManagerImpl.prepareSecondaryStorageStore(VmwareManagerImpl.java:661)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
> at
> com.sun.proxy.$Proxy389.prepareSecondaryStorageStore(Unknown
> Source)
> at
> com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:1962)
> at
> com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:530)
> at
> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at
> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2020-07-02 14:19:13,321 ERROR [c.c.h.v.m.VmwareManagerImpl]
> (DirectAgent-10:ctx-52902ebf usphlmvesxt155.phl.global.corp.sap,
> job-15/job-33, cmd: StartCommand) (logid:b8ba37d1) Unable to copy
> systemvm ISO on secondary storage. src location:
> /usr/share/cloudstack-common/vms/systemvm.iso, dest location:
> /var/cloudstack/mnt/VM/345050029058.777ecb60/systemvm/systemvm-4.14.0.
> 0.iso
> 2020-07-02 14:19:13,322 INFO [c.c.h.v.u.VmwareHelper]
> (DirectAgent-10:ctx-52902ebf usphlmvesxt155.phl.global.corp.sap,
> job-15/job-33, cmd: StartCommand) (logid:b8ba37d1) [ignored]failed to
> get message for exception: Unable to copy systemvm ISO on secondary storage.
> src location: /usr/share/cloudstack-common/vms/systemvm.iso, dest location:
> /var/cloudstack/mnt/VM/345050029058.777ecb60/systemvm/systemvm-4.14.0.
> 0.iso
> 2020-07-02 14:19:13,323 WARN [c.c.h.v.r.VmwareResource]
> (DirectAgent-10:ctx-52902ebf usphlmvesxt155.phl.global.corp.sap,
> job-15/job-33, cmd: StartCommand) (logid:b8ba37d1) StartCommand failed
> due to Exception: com.cloud.utils.exception.CloudRuntimeException
> Message: Unable to copy systemvm ISO on secondary storage. src location:
> /usr/share/cloudstack-common/vms/systemvm.iso, dest location:
> /var/cloudstack/mnt/VM/345050029058.777ecb60/systemvm/systemvm-4.14.0.
> 0.iso
>
>
>
>
> -----Original Message-----
> From: Andrija Panic <an...@gmail.com>
> Sent: Thursday, July 2, 2020 12:34 PM
> To: users <us...@cloudstack.apache.org>
> Subject: Re: Initial SystemVM Creation Failing
>
> Hi Mike,
>
> Mount you secondary storage on the management server to some temp folder,
> then run: chown -R cloud:cloud <MOUNTPOINT>/systemvm/
> Other folders on Secondary Storage NFS are owned by root, but the
> "systemvm" should be owned by "cloud".
>
> I assume you got this after some ACS upgrade, or similar?
>
> On Thu, 2 Jul 2020 at 15:51, Corey, Mike <mi...@sap.com> wrote:
>
> > Hi,
> >
> >
> >
> > When I enable the zone, the systemvm creation begins but goes into a
> > loop of creating and deleting VMs in vCenter. I see in the logs
> > there is an error for creating the folder permissions on the
> > secondary storage (see below). Running this CHMOD manually from
> > inside the CSM kicks the same error.
> >
> >
> >
> > Please provide suggestions.
> >
> >
> >
> > 2020-07-02 09:08:39,665 DEBUG [c.c.h.v.m.VmwareManagerImpl]
> > (DirectAgent-66:ctx-e15c32b2 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: CopyCommand) (logid:1cfa64a9) Executing: sudo
> > chmod
> > 1777 /var/cloudstack/mnt/VM/345050029058.2807345c
> >
> > 2020-07-02 09:08:39,666 DEBUG [c.c.h.v.m.VmwareManagerImpl]
> > (DirectAgent-66:ctx-e15c32b2 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: CopyCommand) (logid:1cfa64a9) Executing while
> > with timeout : 1440000
> >
> > 2020-07-02 09:08:39,693 DEBUG [c.c.h.v.m.VmwareManagerImpl]
> > (DirectAgent-66:ctx-e15c32b2 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: CopyCommand) (logid:1cfa64a9) Exit value is 1
> >
> > 2020-07-02 09:08:39,697 DEBUG [c.c.h.v.m.VmwareManagerImpl]
> > (DirectAgent-66:ctx-e15c32b2 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: CopyCommand) (logid:1cfa64a9) chmod: changing
> > permissions of ‘/var/cloudstack/mnt/VM/345050029058.2807345c’:
> > Operation not permitted
> >
> > 2020-07-02 09:08:39,699 WARN [c.c.h.v.m.VmwareManagerImpl]
> > (DirectAgent-66:ctx-e15c32b2 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: CopyCommand) (logid:1cfa64a9) Unable to set
> permissions
> > for /var/cloudstack/mnt/VM/345050029058.2807345c due to chmod:
> > changing permissions of
> > ‘/var/cloudstack/mnt/VM/345050029058.2807345c’: Operation not
> > permitted
> >
> >
> >
> > Lines…
> >
> >
> >
> > 2020-07-02 09:09:00,598 DEBUG [c.c.h.v.m.HostMO]
> > (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) find VM v-2-VM on
> > host
> >
> > 2020-07-02 09:09:00,598 INFO [c.c.h.v.m.HostMO]
> > (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) VM v-2-VM not
> > found in host cache
> >
> > 2020-07-02 09:09:00,598 DEBUG [c.c.h.v.m.HostMO]
> > (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) load VM cache on
> > host
> >
> > 2020-07-02 09:09:00,616 DEBUG [c.c.h.v.m.HostMO]
> > (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) find VM v-2-VM on
> > host
> >
> > 2020-07-02 09:09:00,617 DEBUG [c.c.h.v.m.HostMO]
> > (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) VM v-2-VM found
> > in
> host
> > cache
> >
> > 2020-07-02 09:09:00,799 ERROR [c.c.h.v.m.VmwareManagerImpl]
> > (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) Unable to create
> > systemvm folder on secondary storage. location:
> > /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm
> >
> > 2020-07-02 09:09:00,800 INFO [c.c.h.v.u.VmwareHelper]
> > (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) [ignored]failed
> > to get message for exception: Unable to create systemvm folder on
> > secondary storage. location:
> > /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm
> >
> > 2020-07-02 09:09:00,800 WARN [c.c.h.v.r.VmwareResource]
> > (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) StartCommand
> > failed
> due
> > to Exception: com.cloud.utils.exception.CloudRuntimeException
> >
> > Message: Unable to create systemvm folder on secondary storage. location:
> > /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm
> >
> >
> >
> > com.cloud.utils.exception.CloudRuntimeException: Unable to create
> systemvm
> > folder on secondary storage. location:
> > /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm
> >
> > at
> >
> com.cloud.hypervisor.vmware.manager.VmwareManagerImpl.prepareSecondary
> StorageStore(VmwareManagerImpl.java:648)
> >
> > at
> > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Nati
> > ve
> > Method)
> >
> > at
> >
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeM
> ethodAccessorImpl.java:62)
> >
> > at
> >
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Del
> egatingMethodAccessorImpl.java:43)
> >
> > at
> > java.base/java.lang.reflect.Method.invoke(Method.java:566)
> >
> > at
> >
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflectio
> n(AopUtils.java:344)
> >
> > at
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoi
> npoint(ReflectiveMethodInvocation.java:198)
> >
> > at
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(R
> eflectiveMethodInvocation.java:163)
> >
> > at
> >
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke
> (ExposeInvocationInterceptor.java:95)
> >
> > at
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(R
> eflectiveMethodInvocation.java:186)
> >
> > at
> >
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamic
> AopProxy.java:212)
> >
> > at
> > com.sun.proxy.$Proxy389.prepareSecondaryStorageStore(Unknown Source)
> >
> > at
> >
> com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareReso
> urce.java:1962)
> >
> > at
> >
> com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(Vmw
> areResource.java:530)
> >
> > at
> >
> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAge
> ntAttache.java:315)
> >
> > at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(Man
> agedContextRunnable.java:49)
> >
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.cal
> l(DefaultManagedContext.java:56)
> >
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callW
> ithContext(DefaultManagedContext.java:103)
> >
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWi
> thContext(DefaultManagedContext.java:53)
> >
> > at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(Manag
> edContextRunnable.java:46)
> >
> > at
> >
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executor
> s.java:515)
> >
> > at
> > java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> >
> > at
> >
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
> tureTask.run(ScheduledThreadPoolExecutor.java:304)
> >
> > at
> >
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
> Executor.java:1128)
> >
> > at
> >
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
> lExecutor.java:628)
> >
> > at java.base/java.lang.Thread.run(Thread.java:834)
> >
> > 2020-07-02 09:09:00,804 DEBUG [c.c.a.m.DirectAgentAttache]
> > (DirectAgent-17:ctx-681dc524) (logid:1cfa64a9) Seq 1-4541880224203147245:
> > Cancelling because one of the answers is false and it is stop on error.
> >
> > 2020-07-02 09:09:00,804 DEBUG [c.c.a.m.DirectAgentAttache]
> > (DirectAgent-17:ctx-681dc524) (logid:1cfa64a9) Seq 1-4541880224203147245:
> > Response Received:
> >
> > 2020-07-02 09:09:00,805 DEBUG [c.c.a.t.Request]
> > (DirectAgent-17:ctx-681dc524) (logid:1cfa64a9) Seq 1-4541880224203147245:
> > Processing: { Ans: , MgmtId: 345050029058, via:
> > 1(usphlmvesxt154.phl.global.corp.sap), Ver: v1, Flags: 110,
> >
> [{"com.cloud.agent.api.StartAnswer":{"vm":{"id":2,"name":"v-2-VM","boo
> tloader":"HVM","state":"Starting","type":"ConsoleProxy","cpus":1,"minS
> peed":500,"maxSpeed":500,"minRam":1073741824,"maxRam":1073741824,"host
> Name":"v-2-VM","arch":"x86_64","os":"Other
> > Linux (64-bit)","platformEmulator":"otherLinux64Guest","bootArgs":"
> > template=domP type=consoleproxy host=10.4.32.163 port=8250
> > name=v-2-VM
> > zone=1 pod=1 guid=Proxy.2 proxy_vm=2 disable_rp_filter=true
> > eth2ip=10.4.42.25 eth2mask=255.255.255.192 gateway=10.4.42.1
> eth0ip=0.0.0.0
> > eth0mask=0.0.0.0 eth1ip=10.4.4.35 eth1mask=255.255.252.0 mgmtcidr=
> > 10.4.32.0/24 localgw=10.4.4.1 internaldns1=10.4.202.200
> dns1=10.4.202.200
> >
> dns2=10.4.12.200","enableHA":false,"limitCpuUse":false,"enableDynamica
> llyScaleVm":false,"vncPassword":"sYnwDJYgzrMKdm3ftAO-kg","params":{"vm
> ware.reserve.mem":"false","vmware.reserve.cpu":"false","nicAdapter":"E
> 1000"},"uuid":"fb65df73-693d-48fb-a203-c2b5e776842a","disks":[{"data":
> {"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"9a1f2ae8-6
> b10-4525-8a54-a175b153b789","volumeType":"ROOT","dataStore":{"org.apac
> he.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"50cc970f-fbe2-3b
> 45-9cb5-b8bcdb6dbff5","id":1,"poolType":"NetworkFilesystem","host":"us
> phlcv300.phl.global.corp.sap","path":"/usphlmvesx_lab_cspri","port":20
> 49,"url":"NetworkFilesystem://usphlcv300.phl.global.corp.sap/usphlmves
> x_lab_cspri/?ROLE=Primary&STOREUUID=50cc970f-fbe2-3b45-9cb5-b8bcdb6dbf
> f5","isManaged":false}},"name":"ROOT-2","size":0,"path":"ROOT-2_2","vo
> lumeId":2,"vmName":"v-2-VM","accountId":1,"format":"OVA","provisioning
> Type":"THIN","id":2,"deviceId":0,"bytesReadRate":0,"bytesWriteRate":0,
> "iopsReadRate":0,"iopsWriteRate":0,"hypervisorType":"VMware","directDo
> wnload":false}},"diskSeq":0,"path":"ROOT-2_2","type":"ROOT","_details"
> :{"storageHost":"usphlcv300.phl.global.corp.sap","managed":"false","st
> oragePort":"2049","volumeSize":"0"}}],"nics":[{"deviceId":2,"networkRa
> teMbps":-1,"defaultNic":true,"pxeDisable":true,"nicUuid":"ffbc1822-790
> 7-4cdf-9e80-bb316786e04d","details":{"PromiscuousMode":"false","MacAdd
> ressChanges":"true","ForgedTransmits":"true"},"dpdkEnabled":false,"uui
> d":"f3a30021-a469-413d-85a8-ca1ca68ba27c","ip":"10.4.42.25","netmask":
> "255.255.255.192","gateway":"10.4.42.1","mac":"1e:00:e3:00:00:19","dns
> 1":"10.4.202.200","dns2":"10.4.12.200","broadcastType":"Vlan","type":"
> Public","broadcastUri":"vlan://untagged","isolationUri":"vlan://untagg
> ed","isSecurityGroupEnabled":false,"name":"vSwitch0,134,vmwaresvs"},{"
> deviceId":0,"networkRateMbps":-1,"defaultNic":false,"pxeDisable":true,
> "nicUuid":"d6521be0-85b8-4836-bf8b-8461afdf593a","details":{"Promiscuo
> usMode":"false","MacAddressChanges":"true","ForgedTransmits":"true"},"
> dpdkEnabled":false,"uuid":"eda0ae3d-d15d-4f91-b8db-a719cf5a94bf","mac"
> :"02:00:31:71:00:02","broadcastType":"LinkLocal","type":"Control","isS
> ecurityGroupEnabled":false,"name":"vSwitch0,146,vmwaresvs"},{"deviceId
> ":1,"networkRateMbps":-1,"defaultNic":false,"pxeDisable":true,"nicUuid
> ":"4e6201ad-91c0-4cab-bf53-9e2440025245","details":{"PromiscuousMode":
> "false","MacAddressChanges":"true","ForgedTransmits":"true"},"dpdkEnab
> led":false,"uuid":"ce88b030-c04c-4437-b60d-9910c2cc376f","ip":"10.4.4.
> 35","netmask":"255.255.252.0","gateway":"10.4.4.1","mac":"1e:00:6b:00:
> 00:27","broadcastType":"Native","type":"Management","isSecurityGroupEn
> abled":false,"name":"vSwitch0,146,vmwaresvs"}],"guestOsDetails":{},"ex
> traConfig":{}},"result":false,"details":"StartCommand
> > failed due to Exception:
> > com.cloud.utils.exception.CloudRuntimeException\nMessage: Unable to
> create
> > systemvm folder on secondary storage. location:
> > /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm\n","wait":0}}]
> > }
> >
> > 2020-07-02 09:09:00,805 DEBUG [c.c.a.m.AgentAttache]
> > (DirectAgent-17:ctx-681dc524) (logid:1cfa64a9) Seq 1-4541880224203147245:
> > No more commands found
> >
> > 2020-07-02 09:09:00,806 DEBUG [c.c.a.t.Request]
> > (Work-Job-Executor-2:ctx-9aa38618 job-16/job-35 ctx-7ebd1b78)
> > (logid:1cfa64a9) Seq 1-4541880224203147245: Received: { Ans: , MgmtId:
> > 345050029058, via: 1(usphlmvesxt154.phl.global.corp.sap), Ver: v1, Flags:
> > 110, { StartAnswer } }
> >
> > 2020-07-02 09:09:00,816 INFO [c.c.v.VirtualMachineManagerImpl]
> > (Work-Job-Executor-2:ctx-9aa38618 job-16/job-35 ctx-7ebd1b78)
> > (logid:1cfa64a9) Unable to start VM on Host[-1-Routing] due to
> StartCommand
> > failed due to Exception:
> > com.cloud.utils.exception.CloudRuntimeException
> >
> > Message: Unable to create systemvm folder on secondary storage. location:
> > /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm
> >
> >
> >
> > *Mike Corey*
> >
> >
> > Technology Senior Consultant, IT CS CTW Operation & Virtualization
> Service
> > US
> >
> >
> > *SAP AMERICA, INC.* 3999 West Chester Pike, Newtown Square, 19073
> > United States
> >
> >
> > T +1 610 661 0905, M +1 484 274 2658, E mike.corey@sap.com
> >
> >
> >
> >
> >
> >
> >
>
>
> --
>
> Andrija Panić
>
RE: Initial SystemVM Creation Failing
Posted by "Corey, Mike" <mi...@sap.com>.
I appreciate the response, but I'm not sure where you are going.
My secondary and primary storage are mounts from a NetApp filer. Both the CentOS (CloudStack Management) and the ESXi hosts can read/write to the secondary volume. I don't understand why cloudstack is setting the ownership and permissions to nobody when it creates the systemvm folder. That doesn't seem right.
Again, being new to CloudStack I'm not sure of the underlying mechanics of the systemvm builds.
-----Original Message-----
From: cristian.c@istream.today <cr...@istream.today>
Sent: Friday, July 10, 2020 12:30 PM
To: users@cloudstack.apache.org
Subject: RE: Initial SystemVM Creation Failing
Hello
If you check your NFS configuration you will see that somewhere you have : /etc/idmapd.conf "nobody"
Regards,
Cristian
-----Original Message-----
From: Corey, Mike <mi...@sap.com>
Sent: Friday, July 10, 2020 7:01 PM
To: users@cloudstack.apache.org
Subject: RE: Initial SystemVM Creation Failing
The system VM folder is being created as owner & permissions nobody:nobody. I have to believe that is NOT SUPPOSED to be the case.
Because the directory /var/cloudstack/mnt/VM/345050029058.4f87a734 is locked down, no other steps in the systemvm creation can continue.
I need help figuring out WHY the systemVM folder in the above directory is being created as nobody:nobody.
I can't chown that directory once CloudStack creates it during the zone enablement. I can't modify the permissions to that directory once CloudStack creates it during the zone enablement. I can't copy files to that directory once CloudStack creates it during the zone enablement.
Please identify what is occurring during the creation of the systemvm directory that would cause the ownership to be set as nobody:nobody.
Mike
Message: Unable to copy systemvm ISO on secondary storage. src location: /usr/share/cloudstack-common/vms/systemvm.iso, dest location: /var/cloudstack/mnt/VM/345050029058.4f87a734/systemvm/systemvm-4.14.0.0.iso
[root@usphlmvcsmt01 VM]# pwd
/var/cloudstack/mnt/VM
[root@usphlmvcsmt01 VM]# ls -al
total 80864
drwxr-xr-x. 3 cloud cloud 55 Jul 10 11:46 .
drwxrwx---. 3 root cloud 16 Jul 9 14:40 ..
drwxr-xr-x. 5 nobody nobody 4096 Jul 10 11:31 345050029058.4f87a734
-rw-r--r--. 1 cloud cloud 82800640 Jul 10 11:43 systemvm.iso
-----Original Message-----
From: Cristian Ciobanu <cr...@istream.today>
Sent: Friday, July 10, 2020 5:26 AM
To: users@cloudstack.apache.org
Subject: Re: Initial SystemVM Creation Failing
Just try to execute the command manually, copy the system.iso from source to the destination. I had a similar issue but is was caused by the NFS permissions.
On Wed, 8 Jul 2020, 21:44 Corey, Mike, <mi...@sap.com> wrote:
> Gone over the installation now twice (reverted to snapshot of CentOS
> VM) and still getting weird storage/permission type issues.
>
> The permissions and ownership of the systemvm folder looks correct.
> [root@ ~]# ls -al /mnt/secondary/
> total 16
> drwxr-xr-x. 4 root root 4096 Jul 8 11:15 .
> drwxr-xr-x. 4 root root 38 Jul 8 11:15 ..
> drwxrwxrwx. 12 root root 4096 Jul 8 11:05 .snapshot
> drwxrwxr-x. 2 cloud cloud 4096 Jul 8 10:58 systemvm
> drwxr-xr-x. 3 root root 4096 Jul 8 11:15 template
>
> However, I'm still seeing the copy iso error and the VMs keep creating
> and deleting with the log suggesting the vmx file of the VM doesn't
> exist on the datastore (can create files/folder on datastore
> identified in errror from vCenter).
>
> 2020-07-08 14:22:36,486 INFO [c.c.h.v.m.DatastoreMO]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Folder v-6-VM does
> not exist on datastore
> 2020-07-08 14:22:36,494 INFO [c.c.h.v.m.DatastoreMO]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Search file
> v-6-VM.vmx on [50cc970ffbe23b459cb5b8bcdb6dbff5]
> 2020-07-08 14:22:36,522 INFO [c.c.h.v.m.DatastoreMO]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) File
> [50cc970ffbe23b459cb5b8bcdb6dbff5] v-6-VM.vmx does not exist on
> datastore
> 2020-07-08 14:22:36,522 INFO [c.c.h.v.m.DatastoreMO]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Searching file
> v-6-VM.vmx in [50cc970ffbe23b459cb5b8bcdb6dbff5]
> 2020-07-08 14:22:36,641 INFO [c.c.h.v.m.HypervisorHostHelper]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Create blank VM.
> cpuCount: 1, cpuSpeed(MHz): 500, mem(Mb): 1024
> 2020-07-08 14:22:36,692 DEBUG [c.c.h.v.m.ClusterMO]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Retrieved
> recommended disk controller for guest OS : otherLinux64Guest in cluster PHL-CSM01 :
> VirtualLsiLogicController
> 2020-07-08 14:22:37,115 DEBUG [c.c.h.v.m.HostMO]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) find VM v-6-VM on
> host
> 2020-07-08 14:22:37,115 INFO [c.c.h.v.m.HostMO]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) VM v-6-VM not found
> in host cache
> 2020-07-08 14:22:37,115 DEBUG [c.c.h.v.m.HostMO]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) load VM cache on
> host
> 2020-07-08 14:22:37,155 DEBUG [c.c.h.v.m.HostMO]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) find VM v-6-VM on
> host
> 2020-07-08 14:22:37,155 DEBUG [c.c.h.v.m.HostMO]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) VM v-6-VM found in
> host cache
> 2020-07-08 14:22:37,375 INFO [c.c.h.v.m.VmwareManagerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Inject SSH key pairs
> before copying systemvm.iso into secondary storage
> 2020-07-08 14:22:37,377 INFO [c.c.s.ConfigurationServerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Processing
> updateKeyPairs
> 2020-07-08 14:22:37,377 INFO [c.c.s.ConfigurationServerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Keypairs already in
> database, updating local copy
> 2020-07-08 14:22:37,381 INFO [c.c.s.ConfigurationServerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Going to update
> systemvm iso with generated keypairs if needed
> 2020-07-08 14:22:37,381 INFO [c.c.s.ConfigurationServerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Trying to inject
> public and private keys into systemvm iso
> 2020-07-08 14:22:37,381 DEBUG [c.c.u.s.Script]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Looking for
> scripts/vm/systemvm/injectkeys.sh in the classpath
> 2020-07-08 14:22:37,381 DEBUG [c.c.u.s.Script]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) System resource:
> file:/usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh
> 2020-07-08 14:22:37,381 DEBUG [c.c.u.s.Script]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Absolute path =
> /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh
> 2020-07-08 14:22:37,381 DEBUG [c.c.u.s.Script]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Looking for
> vms/systemvm.iso in the classpath
> 2020-07-08 14:22:37,381 DEBUG [c.c.u.s.Script]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) System resource:
> file:/usr/share/cloudstack-common/vms/systemvm.iso
> 2020-07-08 14:22:37,381 DEBUG [c.c.u.s.Script]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Absolute path =
> /usr/share/cloudstack-common/vms/systemvm.iso
> 2020-07-08 14:22:37,381 DEBUG [c.c.s.ConfigurationServerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Executing: /bin/bash
> /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh
> /var/cloudstack/management/.ssh/id_rsa.pub
> /var/cloudstack/management/.ssh/id_rsa
> /usr/share/cloudstack-common/vms/systemvm.iso
> 2020-07-08 14:22:37,384 DEBUG [c.c.s.ConfigurationServerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Executing while with
> timeout : 3600000
> 2020-07-08 14:22:37,487 DEBUG [c.c.s.ConfigurationServerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Execution is successful.
> 2020-07-08 14:22:37,494 DEBUG [c.c.s.ConfigurationServerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) mount: /dev/loop0 is
> write-protected, mounting read-only New public key is the same as the
> one in the systemvm.iso, not injecting it, not modifying systemvm.iso
>
> 2020-07-08 14:22:37,494 INFO [c.c.s.ConfigurationServerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) The script
> injectkeys.sh was run with result : null
> 2020-07-08 14:22:37,494 INFO [c.c.h.v.m.VmwareManagerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Copy System VM patch
> ISO file to secondary storage. source ISO:
> /usr/share/cloudstack-common/vms/systemvm.iso, destination:
> /var/cloudstack/mnt/VM/345050029058.1f7af2bf/systemvm/systemvm-4.14.0.
> 0.iso
> 2020-07-08 14:22:37,495 ERROR [c.c.h.v.m.VmwareManagerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Unexpected exception
> java.io.FileNotFoundException:
> /var/cloudstack/mnt/VM/345050029058.1f7af2bf/systemvm/systemvm-4.14.0.
> 0.iso
> (Permission denied)
>
> But, the ownership/permissions look okay on that destination directory too.
>
> [root@ ~]# ls -al /var/cloudstack/mnt/VM/345050029058.1f7af2bf
> total 12
> drwxr-xr-x. 4 nobody nobody 4096 Jul 8 11:15 .
> drwxr-xr-x. 3 cloud cloud 35 Jul 8 14:06 ..
> drwxrwxr-x. 2 cloud cloud 4096 Jul 8 10:58 systemvm drwxr-xr-x. 3
> nobody nobody 4096 Jul 8 11:15 template
> [root@usphlmvcsmt01 ~]#
>
>
> Any ideas?
>
>
> -----Original Message-----
> From: Corey, Mike
> Sent: Thursday, July 2, 2020 2:31 PM
> To: users@cloudstack.apache.org
> Subject: RE: Initial SystemVM Creation Failing
>
> No upgrade - this again is the fresh install of CloudStack 4.14 with
> VMware resources. Maybe I should note that I created the primary and
> secondary storage via the "Advanced" setup wizard from the UI (logged
> in as admin).
>
> I performed the ownership change you suggested so now cloud owns the
> systemvm folder on the secondary storage. However, now I'm getting a
> new error regarding copying the ISO.
>
> Should I just bag the 4.14 installation and try to get 4.13.1 running
> with VMware?
>
>
> 2020-07-02 14:19:13,314 INFO [c.c.h.v.m.VmwareManagerImpl]
> (DirectAgent-10:ctx-52902ebf usphlmvesxt155.phl.global.corp.sap,
> job-15/job-33, cmd: StartCommand) (logid:b8ba37d1) Copy System VM
> patch ISO file to secondary storage. source ISO:
> /usr/share/cloudstack-common/vms/systemvm.iso, destination:
> /var/cloudstack/mnt/VM/345050029058.777ecb60/systemvm/systemvm-4.14.0.
> 0.iso
> 2020-07-02 14:19:13,320 ERROR [c.c.h.v.m.VmwareManagerImpl]
> (DirectAgent-10:ctx-52902ebf usphlmvesxt155.phl.global.corp.sap,
> job-15/job-33, cmd: StartCommand) (logid:b8ba37d1) Unexpected
> exception
> java.io.FileNotFoundException:
> /var/cloudstack/mnt/VM/345050029058.777ecb60/systemvm/systemvm-4.14.0.
> 0.iso
> (Permission denied)
> at java.base/java.io.FileOutputStream.open0(Native Method)
> at java.base/java.io
> .FileOutputStream.open(FileOutputStream.java:298)
> at java.base/java.io
> .FileOutputStream.<init>(FileOutputStream.java:237)
> at java.base/java.io
> .FileOutputStream.<init>(FileOutputStream.java:187)
> at org.apache.commons.io.FileUtils.doCopyFile(FileUtils.java:1123)
> at org.apache.commons.io.FileUtils.copyFile(FileUtils.java:1076)
> at org.apache.commons.io.FileUtils.copyFile(FileUtils.java:1028)
> at com.cloud.utils.FileUtil.copyfile(FileUtil.java:30)
> at
> com.cloud.hypervisor.vmware.manager.VmwareManagerImpl.prepareSecondaryStorageStore(VmwareManagerImpl.java:661)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
> at
> com.sun.proxy.$Proxy389.prepareSecondaryStorageStore(Unknown
> Source)
> at
> com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:1962)
> at
> com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:530)
> at
> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at
> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2020-07-02 14:19:13,321 ERROR [c.c.h.v.m.VmwareManagerImpl]
> (DirectAgent-10:ctx-52902ebf usphlmvesxt155.phl.global.corp.sap,
> job-15/job-33, cmd: StartCommand) (logid:b8ba37d1) Unable to copy
> systemvm ISO on secondary storage. src location:
> /usr/share/cloudstack-common/vms/systemvm.iso, dest location:
> /var/cloudstack/mnt/VM/345050029058.777ecb60/systemvm/systemvm-4.14.0.
> 0.iso
> 2020-07-02 14:19:13,322 INFO [c.c.h.v.u.VmwareHelper]
> (DirectAgent-10:ctx-52902ebf usphlmvesxt155.phl.global.corp.sap,
> job-15/job-33, cmd: StartCommand) (logid:b8ba37d1) [ignored]failed to
> get message for exception: Unable to copy systemvm ISO on secondary storage.
> src location: /usr/share/cloudstack-common/vms/systemvm.iso, dest location:
> /var/cloudstack/mnt/VM/345050029058.777ecb60/systemvm/systemvm-4.14.0.
> 0.iso
> 2020-07-02 14:19:13,323 WARN [c.c.h.v.r.VmwareResource]
> (DirectAgent-10:ctx-52902ebf usphlmvesxt155.phl.global.corp.sap,
> job-15/job-33, cmd: StartCommand) (logid:b8ba37d1) StartCommand failed
> due to Exception: com.cloud.utils.exception.CloudRuntimeException
> Message: Unable to copy systemvm ISO on secondary storage. src location:
> /usr/share/cloudstack-common/vms/systemvm.iso, dest location:
> /var/cloudstack/mnt/VM/345050029058.777ecb60/systemvm/systemvm-4.14.0.
> 0.iso
>
>
>
>
> -----Original Message-----
> From: Andrija Panic <an...@gmail.com>
> Sent: Thursday, July 2, 2020 12:34 PM
> To: users <us...@cloudstack.apache.org>
> Subject: Re: Initial SystemVM Creation Failing
>
> Hi Mike,
>
> Mount you secondary storage on the management server to some temp folder,
> then run: chown -R cloud:cloud <MOUNTPOINT>/systemvm/
> Other folders on Secondary Storage NFS are owned by root, but the
> "systemvm" should be owned by "cloud".
>
> I assume you got this after some ACS upgrade, or similar?
>
> On Thu, 2 Jul 2020 at 15:51, Corey, Mike <mi...@sap.com> wrote:
>
> > Hi,
> >
> >
> >
> > When I enable the zone, the systemvm creation begins but goes into a
> > loop of creating and deleting VMs in vCenter. I see in the logs
> > there is an error for creating the folder permissions on the
> > secondary storage (see below). Running this CHMOD manually from
> > inside the CSM kicks the same error.
> >
> >
> >
> > Please provide suggestions.
> >
> >
> >
> > 2020-07-02 09:08:39,665 DEBUG [c.c.h.v.m.VmwareManagerImpl]
> > (DirectAgent-66:ctx-e15c32b2 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: CopyCommand) (logid:1cfa64a9) Executing: sudo
> > chmod
> > 1777 /var/cloudstack/mnt/VM/345050029058.2807345c
> >
> > 2020-07-02 09:08:39,666 DEBUG [c.c.h.v.m.VmwareManagerImpl]
> > (DirectAgent-66:ctx-e15c32b2 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: CopyCommand) (logid:1cfa64a9) Executing while
> > with timeout : 1440000
> >
> > 2020-07-02 09:08:39,693 DEBUG [c.c.h.v.m.VmwareManagerImpl]
> > (DirectAgent-66:ctx-e15c32b2 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: CopyCommand) (logid:1cfa64a9) Exit value is 1
> >
> > 2020-07-02 09:08:39,697 DEBUG [c.c.h.v.m.VmwareManagerImpl]
> > (DirectAgent-66:ctx-e15c32b2 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: CopyCommand) (logid:1cfa64a9) chmod: changing
> > permissions of ‘/var/cloudstack/mnt/VM/345050029058.2807345c’:
> > Operation not permitted
> >
> > 2020-07-02 09:08:39,699 WARN [c.c.h.v.m.VmwareManagerImpl]
> > (DirectAgent-66:ctx-e15c32b2 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: CopyCommand) (logid:1cfa64a9) Unable to set
> permissions
> > for /var/cloudstack/mnt/VM/345050029058.2807345c due to chmod:
> > changing permissions of
> > ‘/var/cloudstack/mnt/VM/345050029058.2807345c’: Operation not
> > permitted
> >
> >
> >
> > Lines…
> >
> >
> >
> > 2020-07-02 09:09:00,598 DEBUG [c.c.h.v.m.HostMO]
> > (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) find VM v-2-VM on
> > host
> >
> > 2020-07-02 09:09:00,598 INFO [c.c.h.v.m.HostMO]
> > (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) VM v-2-VM not
> > found in host cache
> >
> > 2020-07-02 09:09:00,598 DEBUG [c.c.h.v.m.HostMO]
> > (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) load VM cache on
> > host
> >
> > 2020-07-02 09:09:00,616 DEBUG [c.c.h.v.m.HostMO]
> > (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) find VM v-2-VM on
> > host
> >
> > 2020-07-02 09:09:00,617 DEBUG [c.c.h.v.m.HostMO]
> > (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) VM v-2-VM found
> > in
> host
> > cache
> >
> > 2020-07-02 09:09:00,799 ERROR [c.c.h.v.m.VmwareManagerImpl]
> > (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) Unable to create
> > systemvm folder on secondary storage. location:
> > /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm
> >
> > 2020-07-02 09:09:00,800 INFO [c.c.h.v.u.VmwareHelper]
> > (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) [ignored]failed
> > to get message for exception: Unable to create systemvm folder on
> > secondary storage. location:
> > /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm
> >
> > 2020-07-02 09:09:00,800 WARN [c.c.h.v.r.VmwareResource]
> > (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) StartCommand
> > failed
> due
> > to Exception: com.cloud.utils.exception.CloudRuntimeException
> >
> > Message: Unable to create systemvm folder on secondary storage. location:
> > /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm
> >
> >
> >
> > com.cloud.utils.exception.CloudRuntimeException: Unable to create
> systemvm
> > folder on secondary storage. location:
> > /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm
> >
> > at
> >
> com.cloud.hypervisor.vmware.manager.VmwareManagerImpl.prepareSecondary
> StorageStore(VmwareManagerImpl.java:648)
> >
> > at
> > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Nati
> > ve
> > Method)
> >
> > at
> >
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeM
> ethodAccessorImpl.java:62)
> >
> > at
> >
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Del
> egatingMethodAccessorImpl.java:43)
> >
> > at
> > java.base/java.lang.reflect.Method.invoke(Method.java:566)
> >
> > at
> >
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflectio
> n(AopUtils.java:344)
> >
> > at
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoi
> npoint(ReflectiveMethodInvocation.java:198)
> >
> > at
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(R
> eflectiveMethodInvocation.java:163)
> >
> > at
> >
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke
> (ExposeInvocationInterceptor.java:95)
> >
> > at
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(R
> eflectiveMethodInvocation.java:186)
> >
> > at
> >
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamic
> AopProxy.java:212)
> >
> > at
> > com.sun.proxy.$Proxy389.prepareSecondaryStorageStore(Unknown Source)
> >
> > at
> >
> com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareReso
> urce.java:1962)
> >
> > at
> >
> com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(Vmw
> areResource.java:530)
> >
> > at
> >
> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAge
> ntAttache.java:315)
> >
> > at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(Man
> agedContextRunnable.java:49)
> >
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.cal
> l(DefaultManagedContext.java:56)
> >
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callW
> ithContext(DefaultManagedContext.java:103)
> >
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWi
> thContext(DefaultManagedContext.java:53)
> >
> > at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(Manag
> edContextRunnable.java:46)
> >
> > at
> >
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executor
> s.java:515)
> >
> > at
> > java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> >
> > at
> >
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
> tureTask.run(ScheduledThreadPoolExecutor.java:304)
> >
> > at
> >
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
> Executor.java:1128)
> >
> > at
> >
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
> lExecutor.java:628)
> >
> > at java.base/java.lang.Thread.run(Thread.java:834)
> >
> > 2020-07-02 09:09:00,804 DEBUG [c.c.a.m.DirectAgentAttache]
> > (DirectAgent-17:ctx-681dc524) (logid:1cfa64a9) Seq 1-4541880224203147245:
> > Cancelling because one of the answers is false and it is stop on error.
> >
> > 2020-07-02 09:09:00,804 DEBUG [c.c.a.m.DirectAgentAttache]
> > (DirectAgent-17:ctx-681dc524) (logid:1cfa64a9) Seq 1-4541880224203147245:
> > Response Received:
> >
> > 2020-07-02 09:09:00,805 DEBUG [c.c.a.t.Request]
> > (DirectAgent-17:ctx-681dc524) (logid:1cfa64a9) Seq 1-4541880224203147245:
> > Processing: { Ans: , MgmtId: 345050029058, via:
> > 1(usphlmvesxt154.phl.global.corp.sap), Ver: v1, Flags: 110,
> >
> [{"com.cloud.agent.api.StartAnswer":{"vm":{"id":2,"name":"v-2-VM","boo
> tloader":"HVM","state":"Starting","type":"ConsoleProxy","cpus":1,"minS
> peed":500,"maxSpeed":500,"minRam":1073741824,"maxRam":1073741824,"host
> Name":"v-2-VM","arch":"x86_64","os":"Other
> > Linux (64-bit)","platformEmulator":"otherLinux64Guest","bootArgs":"
> > template=domP type=consoleproxy host=10.4.32.163 port=8250
> > name=v-2-VM
> > zone=1 pod=1 guid=Proxy.2 proxy_vm=2 disable_rp_filter=true
> > eth2ip=10.4.42.25 eth2mask=255.255.255.192 gateway=10.4.42.1
> eth0ip=0.0.0.0
> > eth0mask=0.0.0.0 eth1ip=10.4.4.35 eth1mask=255.255.252.0 mgmtcidr=
> > 10.4.32.0/24 localgw=10.4.4.1 internaldns1=10.4.202.200
> dns1=10.4.202.200
> >
> dns2=10.4.12.200","enableHA":false,"limitCpuUse":false,"enableDynamica
> llyScaleVm":false,"vncPassword":"sYnwDJYgzrMKdm3ftAO-kg","params":{"vm
> ware.reserve.mem":"false","vmware.reserve.cpu":"false","nicAdapter":"E
> 1000"},"uuid":"fb65df73-693d-48fb-a203-c2b5e776842a","disks":[{"data":
> {"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"9a1f2ae8-6
> b10-4525-8a54-a175b153b789","volumeType":"ROOT","dataStore":{"org.apac
> he.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"50cc970f-fbe2-3b
> 45-9cb5-b8bcdb6dbff5","id":1,"poolType":"NetworkFilesystem","host":"us
> phlcv300.phl.global.corp.sap","path":"/usphlmvesx_lab_cspri","port":20
> 49,"url":"NetworkFilesystem://usphlcv300.phl.global.corp.sap/usphlmves
> x_lab_cspri/?ROLE=Primary&STOREUUID=50cc970f-fbe2-3b45-9cb5-b8bcdb6dbf
> f5","isManaged":false}},"name":"ROOT-2","size":0,"path":"ROOT-2_2","vo
> lumeId":2,"vmName":"v-2-VM","accountId":1,"format":"OVA","provisioning
> Type":"THIN","id":2,"deviceId":0,"bytesReadRate":0,"bytesWriteRate":0,
> "iopsReadRate":0,"iopsWriteRate":0,"hypervisorType":"VMware","directDo
> wnload":false}},"diskSeq":0,"path":"ROOT-2_2","type":"ROOT","_details"
> :{"storageHost":"usphlcv300.phl.global.corp.sap","managed":"false","st
> oragePort":"2049","volumeSize":"0"}}],"nics":[{"deviceId":2,"networkRa
> teMbps":-1,"defaultNic":true,"pxeDisable":true,"nicUuid":"ffbc1822-790
> 7-4cdf-9e80-bb316786e04d","details":{"PromiscuousMode":"false","MacAdd
> ressChanges":"true","ForgedTransmits":"true"},"dpdkEnabled":false,"uui
> d":"f3a30021-a469-413d-85a8-ca1ca68ba27c","ip":"10.4.42.25","netmask":
> "255.255.255.192","gateway":"10.4.42.1","mac":"1e:00:e3:00:00:19","dns
> 1":"10.4.202.200","dns2":"10.4.12.200","broadcastType":"Vlan","type":"
> Public","broadcastUri":"vlan://untagged","isolationUri":"vlan://untagg
> ed","isSecurityGroupEnabled":false,"name":"vSwitch0,134,vmwaresvs"},{"
> deviceId":0,"networkRateMbps":-1,"defaultNic":false,"pxeDisable":true,
> "nicUuid":"d6521be0-85b8-4836-bf8b-8461afdf593a","details":{"Promiscuo
> usMode":"false","MacAddressChanges":"true","ForgedTransmits":"true"},"
> dpdkEnabled":false,"uuid":"eda0ae3d-d15d-4f91-b8db-a719cf5a94bf","mac"
> :"02:00:31:71:00:02","broadcastType":"LinkLocal","type":"Control","isS
> ecurityGroupEnabled":false,"name":"vSwitch0,146,vmwaresvs"},{"deviceId
> ":1,"networkRateMbps":-1,"defaultNic":false,"pxeDisable":true,"nicUuid
> ":"4e6201ad-91c0-4cab-bf53-9e2440025245","details":{"PromiscuousMode":
> "false","MacAddressChanges":"true","ForgedTransmits":"true"},"dpdkEnab
> led":false,"uuid":"ce88b030-c04c-4437-b60d-9910c2cc376f","ip":"10.4.4.
> 35","netmask":"255.255.252.0","gateway":"10.4.4.1","mac":"1e:00:6b:00:
> 00:27","broadcastType":"Native","type":"Management","isSecurityGroupEn
> abled":false,"name":"vSwitch0,146,vmwaresvs"}],"guestOsDetails":{},"ex
> traConfig":{}},"result":false,"details":"StartCommand
> > failed due to Exception:
> > com.cloud.utils.exception.CloudRuntimeException\nMessage: Unable to
> create
> > systemvm folder on secondary storage. location:
> > /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm\n","wait":0}}]
> > }
> >
> > 2020-07-02 09:09:00,805 DEBUG [c.c.a.m.AgentAttache]
> > (DirectAgent-17:ctx-681dc524) (logid:1cfa64a9) Seq 1-4541880224203147245:
> > No more commands found
> >
> > 2020-07-02 09:09:00,806 DEBUG [c.c.a.t.Request]
> > (Work-Job-Executor-2:ctx-9aa38618 job-16/job-35 ctx-7ebd1b78)
> > (logid:1cfa64a9) Seq 1-4541880224203147245: Received: { Ans: , MgmtId:
> > 345050029058, via: 1(usphlmvesxt154.phl.global.corp.sap), Ver: v1, Flags:
> > 110, { StartAnswer } }
> >
> > 2020-07-02 09:09:00,816 INFO [c.c.v.VirtualMachineManagerImpl]
> > (Work-Job-Executor-2:ctx-9aa38618 job-16/job-35 ctx-7ebd1b78)
> > (logid:1cfa64a9) Unable to start VM on Host[-1-Routing] due to
> StartCommand
> > failed due to Exception:
> > com.cloud.utils.exception.CloudRuntimeException
> >
> > Message: Unable to create systemvm folder on secondary storage. location:
> > /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm
> >
> >
> >
> > *Mike Corey*
> >
> >
> > Technology Senior Consultant, IT CS CTW Operation & Virtualization
> Service
> > US
> >
> >
> > *SAP AMERICA, INC.* 3999 West Chester Pike, Newtown Square, 19073
> > United States
> >
> >
> > T +1 610 661 0905, M +1 484 274 2658, E mike.corey@sap.com
> >
> >
> >
> >
> >
> >
> >
>
>
> --
>
> Andrija Panić
>
RE: Initial SystemVM Creation Failing
Posted by cr...@istream.today.
Hello
If you check your NFS configuration you will see that somewhere you have : /etc/idmapd.conf "nobody"
Regards,
Cristian
-----Original Message-----
From: Corey, Mike <mi...@sap.com>
Sent: Friday, July 10, 2020 7:01 PM
To: users@cloudstack.apache.org
Subject: RE: Initial SystemVM Creation Failing
The system VM folder is being created as owner & permissions nobody:nobody. I have to believe that is NOT SUPPOSED to be the case.
Because the directory /var/cloudstack/mnt/VM/345050029058.4f87a734 is locked down, no other steps in the systemvm creation can continue.
I need help figuring out WHY the systemVM folder in the above directory is being created as nobody:nobody.
I can't chown that directory once CloudStack creates it during the zone enablement. I can't modify the permissions to that directory once CloudStack creates it during the zone enablement. I can't copy files to that directory once CloudStack creates it during the zone enablement.
Please identify what is occurring during the creation of the systemvm directory that would cause the ownership to be set as nobody:nobody.
Mike
Message: Unable to copy systemvm ISO on secondary storage. src location: /usr/share/cloudstack-common/vms/systemvm.iso, dest location: /var/cloudstack/mnt/VM/345050029058.4f87a734/systemvm/systemvm-4.14.0.0.iso
[root@usphlmvcsmt01 VM]# pwd
/var/cloudstack/mnt/VM
[root@usphlmvcsmt01 VM]# ls -al
total 80864
drwxr-xr-x. 3 cloud cloud 55 Jul 10 11:46 .
drwxrwx---. 3 root cloud 16 Jul 9 14:40 ..
drwxr-xr-x. 5 nobody nobody 4096 Jul 10 11:31 345050029058.4f87a734
-rw-r--r--. 1 cloud cloud 82800640 Jul 10 11:43 systemvm.iso
-----Original Message-----
From: Cristian Ciobanu <cr...@istream.today>
Sent: Friday, July 10, 2020 5:26 AM
To: users@cloudstack.apache.org
Subject: Re: Initial SystemVM Creation Failing
Just try to execute the command manually, copy the system.iso from source to the destination. I had a similar issue but is was caused by the NFS permissions.
On Wed, 8 Jul 2020, 21:44 Corey, Mike, <mi...@sap.com> wrote:
> Gone over the installation now twice (reverted to snapshot of CentOS
> VM) and still getting weird storage/permission type issues.
>
> The permissions and ownership of the systemvm folder looks correct.
> [root@ ~]# ls -al /mnt/secondary/
> total 16
> drwxr-xr-x. 4 root root 4096 Jul 8 11:15 .
> drwxr-xr-x. 4 root root 38 Jul 8 11:15 ..
> drwxrwxrwx. 12 root root 4096 Jul 8 11:05 .snapshot
> drwxrwxr-x. 2 cloud cloud 4096 Jul 8 10:58 systemvm
> drwxr-xr-x. 3 root root 4096 Jul 8 11:15 template
>
> However, I'm still seeing the copy iso error and the VMs keep creating
> and deleting with the log suggesting the vmx file of the VM doesn't
> exist on the datastore (can create files/folder on datastore
> identified in errror from vCenter).
>
> 2020-07-08 14:22:36,486 INFO [c.c.h.v.m.DatastoreMO]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Folder v-6-VM does
> not exist on datastore
> 2020-07-08 14:22:36,494 INFO [c.c.h.v.m.DatastoreMO]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Search file
> v-6-VM.vmx on [50cc970ffbe23b459cb5b8bcdb6dbff5]
> 2020-07-08 14:22:36,522 INFO [c.c.h.v.m.DatastoreMO]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) File
> [50cc970ffbe23b459cb5b8bcdb6dbff5] v-6-VM.vmx does not exist on
> datastore
> 2020-07-08 14:22:36,522 INFO [c.c.h.v.m.DatastoreMO]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Searching file
> v-6-VM.vmx in [50cc970ffbe23b459cb5b8bcdb6dbff5]
> 2020-07-08 14:22:36,641 INFO [c.c.h.v.m.HypervisorHostHelper]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Create blank VM.
> cpuCount: 1, cpuSpeed(MHz): 500, mem(Mb): 1024
> 2020-07-08 14:22:36,692 DEBUG [c.c.h.v.m.ClusterMO]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Retrieved
> recommended disk controller for guest OS : otherLinux64Guest in cluster PHL-CSM01 :
> VirtualLsiLogicController
> 2020-07-08 14:22:37,115 DEBUG [c.c.h.v.m.HostMO]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) find VM v-6-VM on
> host
> 2020-07-08 14:22:37,115 INFO [c.c.h.v.m.HostMO]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) VM v-6-VM not found
> in host cache
> 2020-07-08 14:22:37,115 DEBUG [c.c.h.v.m.HostMO]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) load VM cache on
> host
> 2020-07-08 14:22:37,155 DEBUG [c.c.h.v.m.HostMO]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) find VM v-6-VM on
> host
> 2020-07-08 14:22:37,155 DEBUG [c.c.h.v.m.HostMO]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) VM v-6-VM found in
> host cache
> 2020-07-08 14:22:37,375 INFO [c.c.h.v.m.VmwareManagerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Inject SSH key pairs
> before copying systemvm.iso into secondary storage
> 2020-07-08 14:22:37,377 INFO [c.c.s.ConfigurationServerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Processing
> updateKeyPairs
> 2020-07-08 14:22:37,377 INFO [c.c.s.ConfigurationServerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Keypairs already in
> database, updating local copy
> 2020-07-08 14:22:37,381 INFO [c.c.s.ConfigurationServerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Going to update
> systemvm iso with generated keypairs if needed
> 2020-07-08 14:22:37,381 INFO [c.c.s.ConfigurationServerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Trying to inject
> public and private keys into systemvm iso
> 2020-07-08 14:22:37,381 DEBUG [c.c.u.s.Script]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Looking for
> scripts/vm/systemvm/injectkeys.sh in the classpath
> 2020-07-08 14:22:37,381 DEBUG [c.c.u.s.Script]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) System resource:
> file:/usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh
> 2020-07-08 14:22:37,381 DEBUG [c.c.u.s.Script]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Absolute path =
> /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh
> 2020-07-08 14:22:37,381 DEBUG [c.c.u.s.Script]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Looking for
> vms/systemvm.iso in the classpath
> 2020-07-08 14:22:37,381 DEBUG [c.c.u.s.Script]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) System resource:
> file:/usr/share/cloudstack-common/vms/systemvm.iso
> 2020-07-08 14:22:37,381 DEBUG [c.c.u.s.Script]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Absolute path =
> /usr/share/cloudstack-common/vms/systemvm.iso
> 2020-07-08 14:22:37,381 DEBUG [c.c.s.ConfigurationServerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Executing: /bin/bash
> /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh
> /var/cloudstack/management/.ssh/id_rsa.pub
> /var/cloudstack/management/.ssh/id_rsa
> /usr/share/cloudstack-common/vms/systemvm.iso
> 2020-07-08 14:22:37,384 DEBUG [c.c.s.ConfigurationServerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Executing while with
> timeout : 3600000
> 2020-07-08 14:22:37,487 DEBUG [c.c.s.ConfigurationServerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Execution is successful.
> 2020-07-08 14:22:37,494 DEBUG [c.c.s.ConfigurationServerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) mount: /dev/loop0 is
> write-protected, mounting read-only New public key is the same as the
> one in the systemvm.iso, not injecting it, not modifying systemvm.iso
>
> 2020-07-08 14:22:37,494 INFO [c.c.s.ConfigurationServerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) The script
> injectkeys.sh was run with result : null
> 2020-07-08 14:22:37,494 INFO [c.c.h.v.m.VmwareManagerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Copy System VM patch
> ISO file to secondary storage. source ISO:
> /usr/share/cloudstack-common/vms/systemvm.iso, destination:
> /var/cloudstack/mnt/VM/345050029058.1f7af2bf/systemvm/systemvm-4.14.0.
> 0.iso
> 2020-07-08 14:22:37,495 ERROR [c.c.h.v.m.VmwareManagerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Unexpected exception
> java.io.FileNotFoundException:
> /var/cloudstack/mnt/VM/345050029058.1f7af2bf/systemvm/systemvm-4.14.0.
> 0.iso
> (Permission denied)
>
> But, the ownership/permissions look okay on that destination directory too.
>
> [root@ ~]# ls -al /var/cloudstack/mnt/VM/345050029058.1f7af2bf
> total 12
> drwxr-xr-x. 4 nobody nobody 4096 Jul 8 11:15 .
> drwxr-xr-x. 3 cloud cloud 35 Jul 8 14:06 ..
> drwxrwxr-x. 2 cloud cloud 4096 Jul 8 10:58 systemvm drwxr-xr-x. 3
> nobody nobody 4096 Jul 8 11:15 template
> [root@usphlmvcsmt01 ~]#
>
>
> Any ideas?
>
>
> -----Original Message-----
> From: Corey, Mike
> Sent: Thursday, July 2, 2020 2:31 PM
> To: users@cloudstack.apache.org
> Subject: RE: Initial SystemVM Creation Failing
>
> No upgrade - this again is the fresh install of CloudStack 4.14 with
> VMware resources. Maybe I should note that I created the primary and
> secondary storage via the "Advanced" setup wizard from the UI (logged
> in as admin).
>
> I performed the ownership change you suggested so now cloud owns the
> systemvm folder on the secondary storage. However, now I'm getting a
> new error regarding copying the ISO.
>
> Should I just bag the 4.14 installation and try to get 4.13.1 running
> with VMware?
>
>
> 2020-07-02 14:19:13,314 INFO [c.c.h.v.m.VmwareManagerImpl]
> (DirectAgent-10:ctx-52902ebf usphlmvesxt155.phl.global.corp.sap,
> job-15/job-33, cmd: StartCommand) (logid:b8ba37d1) Copy System VM
> patch ISO file to secondary storage. source ISO:
> /usr/share/cloudstack-common/vms/systemvm.iso, destination:
> /var/cloudstack/mnt/VM/345050029058.777ecb60/systemvm/systemvm-4.14.0.
> 0.iso
> 2020-07-02 14:19:13,320 ERROR [c.c.h.v.m.VmwareManagerImpl]
> (DirectAgent-10:ctx-52902ebf usphlmvesxt155.phl.global.corp.sap,
> job-15/job-33, cmd: StartCommand) (logid:b8ba37d1) Unexpected
> exception
> java.io.FileNotFoundException:
> /var/cloudstack/mnt/VM/345050029058.777ecb60/systemvm/systemvm-4.14.0.
> 0.iso
> (Permission denied)
> at java.base/java.io.FileOutputStream.open0(Native Method)
> at java.base/java.io
> .FileOutputStream.open(FileOutputStream.java:298)
> at java.base/java.io
> .FileOutputStream.<init>(FileOutputStream.java:237)
> at java.base/java.io
> .FileOutputStream.<init>(FileOutputStream.java:187)
> at org.apache.commons.io.FileUtils.doCopyFile(FileUtils.java:1123)
> at org.apache.commons.io.FileUtils.copyFile(FileUtils.java:1076)
> at org.apache.commons.io.FileUtils.copyFile(FileUtils.java:1028)
> at com.cloud.utils.FileUtil.copyfile(FileUtil.java:30)
> at
> com.cloud.hypervisor.vmware.manager.VmwareManagerImpl.prepareSecondaryStorageStore(VmwareManagerImpl.java:661)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
> at
> com.sun.proxy.$Proxy389.prepareSecondaryStorageStore(Unknown
> Source)
> at
> com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:1962)
> at
> com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:530)
> at
> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at
> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2020-07-02 14:19:13,321 ERROR [c.c.h.v.m.VmwareManagerImpl]
> (DirectAgent-10:ctx-52902ebf usphlmvesxt155.phl.global.corp.sap,
> job-15/job-33, cmd: StartCommand) (logid:b8ba37d1) Unable to copy
> systemvm ISO on secondary storage. src location:
> /usr/share/cloudstack-common/vms/systemvm.iso, dest location:
> /var/cloudstack/mnt/VM/345050029058.777ecb60/systemvm/systemvm-4.14.0.
> 0.iso
> 2020-07-02 14:19:13,322 INFO [c.c.h.v.u.VmwareHelper]
> (DirectAgent-10:ctx-52902ebf usphlmvesxt155.phl.global.corp.sap,
> job-15/job-33, cmd: StartCommand) (logid:b8ba37d1) [ignored]failed to
> get message for exception: Unable to copy systemvm ISO on secondary storage.
> src location: /usr/share/cloudstack-common/vms/systemvm.iso, dest location:
> /var/cloudstack/mnt/VM/345050029058.777ecb60/systemvm/systemvm-4.14.0.
> 0.iso
> 2020-07-02 14:19:13,323 WARN [c.c.h.v.r.VmwareResource]
> (DirectAgent-10:ctx-52902ebf usphlmvesxt155.phl.global.corp.sap,
> job-15/job-33, cmd: StartCommand) (logid:b8ba37d1) StartCommand failed
> due to Exception: com.cloud.utils.exception.CloudRuntimeException
> Message: Unable to copy systemvm ISO on secondary storage. src location:
> /usr/share/cloudstack-common/vms/systemvm.iso, dest location:
> /var/cloudstack/mnt/VM/345050029058.777ecb60/systemvm/systemvm-4.14.0.
> 0.iso
>
>
>
>
> -----Original Message-----
> From: Andrija Panic <an...@gmail.com>
> Sent: Thursday, July 2, 2020 12:34 PM
> To: users <us...@cloudstack.apache.org>
> Subject: Re: Initial SystemVM Creation Failing
>
> Hi Mike,
>
> Mount you secondary storage on the management server to some temp folder,
> then run: chown -R cloud:cloud <MOUNTPOINT>/systemvm/
> Other folders on Secondary Storage NFS are owned by root, but the
> "systemvm" should be owned by "cloud".
>
> I assume you got this after some ACS upgrade, or similar?
>
> On Thu, 2 Jul 2020 at 15:51, Corey, Mike <mi...@sap.com> wrote:
>
> > Hi,
> >
> >
> >
> > When I enable the zone, the systemvm creation begins but goes into a
> > loop of creating and deleting VMs in vCenter. I see in the logs
> > there is an error for creating the folder permissions on the
> > secondary storage (see below). Running this CHMOD manually from
> > inside the CSM kicks the same error.
> >
> >
> >
> > Please provide suggestions.
> >
> >
> >
> > 2020-07-02 09:08:39,665 DEBUG [c.c.h.v.m.VmwareManagerImpl]
> > (DirectAgent-66:ctx-e15c32b2 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: CopyCommand) (logid:1cfa64a9) Executing: sudo
> > chmod
> > 1777 /var/cloudstack/mnt/VM/345050029058.2807345c
> >
> > 2020-07-02 09:08:39,666 DEBUG [c.c.h.v.m.VmwareManagerImpl]
> > (DirectAgent-66:ctx-e15c32b2 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: CopyCommand) (logid:1cfa64a9) Executing while
> > with timeout : 1440000
> >
> > 2020-07-02 09:08:39,693 DEBUG [c.c.h.v.m.VmwareManagerImpl]
> > (DirectAgent-66:ctx-e15c32b2 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: CopyCommand) (logid:1cfa64a9) Exit value is 1
> >
> > 2020-07-02 09:08:39,697 DEBUG [c.c.h.v.m.VmwareManagerImpl]
> > (DirectAgent-66:ctx-e15c32b2 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: CopyCommand) (logid:1cfa64a9) chmod: changing
> > permissions of ‘/var/cloudstack/mnt/VM/345050029058.2807345c’:
> > Operation not permitted
> >
> > 2020-07-02 09:08:39,699 WARN [c.c.h.v.m.VmwareManagerImpl]
> > (DirectAgent-66:ctx-e15c32b2 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: CopyCommand) (logid:1cfa64a9) Unable to set
> permissions
> > for /var/cloudstack/mnt/VM/345050029058.2807345c due to chmod:
> > changing permissions of
> > ‘/var/cloudstack/mnt/VM/345050029058.2807345c’: Operation not
> > permitted
> >
> >
> >
> > Lines…
> >
> >
> >
> > 2020-07-02 09:09:00,598 DEBUG [c.c.h.v.m.HostMO]
> > (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) find VM v-2-VM on
> > host
> >
> > 2020-07-02 09:09:00,598 INFO [c.c.h.v.m.HostMO]
> > (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) VM v-2-VM not
> > found in host cache
> >
> > 2020-07-02 09:09:00,598 DEBUG [c.c.h.v.m.HostMO]
> > (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) load VM cache on
> > host
> >
> > 2020-07-02 09:09:00,616 DEBUG [c.c.h.v.m.HostMO]
> > (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) find VM v-2-VM on
> > host
> >
> > 2020-07-02 09:09:00,617 DEBUG [c.c.h.v.m.HostMO]
> > (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) VM v-2-VM found
> > in
> host
> > cache
> >
> > 2020-07-02 09:09:00,799 ERROR [c.c.h.v.m.VmwareManagerImpl]
> > (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) Unable to create
> > systemvm folder on secondary storage. location:
> > /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm
> >
> > 2020-07-02 09:09:00,800 INFO [c.c.h.v.u.VmwareHelper]
> > (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) [ignored]failed
> > to get message for exception: Unable to create systemvm folder on
> > secondary storage. location:
> > /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm
> >
> > 2020-07-02 09:09:00,800 WARN [c.c.h.v.r.VmwareResource]
> > (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) StartCommand
> > failed
> due
> > to Exception: com.cloud.utils.exception.CloudRuntimeException
> >
> > Message: Unable to create systemvm folder on secondary storage. location:
> > /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm
> >
> >
> >
> > com.cloud.utils.exception.CloudRuntimeException: Unable to create
> systemvm
> > folder on secondary storage. location:
> > /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm
> >
> > at
> >
> com.cloud.hypervisor.vmware.manager.VmwareManagerImpl.prepareSecondary
> StorageStore(VmwareManagerImpl.java:648)
> >
> > at
> > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Nati
> > ve
> > Method)
> >
> > at
> >
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeM
> ethodAccessorImpl.java:62)
> >
> > at
> >
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Del
> egatingMethodAccessorImpl.java:43)
> >
> > at
> > java.base/java.lang.reflect.Method.invoke(Method.java:566)
> >
> > at
> >
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflectio
> n(AopUtils.java:344)
> >
> > at
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoi
> npoint(ReflectiveMethodInvocation.java:198)
> >
> > at
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(R
> eflectiveMethodInvocation.java:163)
> >
> > at
> >
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke
> (ExposeInvocationInterceptor.java:95)
> >
> > at
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(R
> eflectiveMethodInvocation.java:186)
> >
> > at
> >
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamic
> AopProxy.java:212)
> >
> > at
> > com.sun.proxy.$Proxy389.prepareSecondaryStorageStore(Unknown Source)
> >
> > at
> >
> com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareReso
> urce.java:1962)
> >
> > at
> >
> com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(Vmw
> areResource.java:530)
> >
> > at
> >
> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAge
> ntAttache.java:315)
> >
> > at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(Man
> agedContextRunnable.java:49)
> >
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.cal
> l(DefaultManagedContext.java:56)
> >
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callW
> ithContext(DefaultManagedContext.java:103)
> >
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWi
> thContext(DefaultManagedContext.java:53)
> >
> > at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(Manag
> edContextRunnable.java:46)
> >
> > at
> >
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executor
> s.java:515)
> >
> > at
> > java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> >
> > at
> >
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
> tureTask.run(ScheduledThreadPoolExecutor.java:304)
> >
> > at
> >
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
> Executor.java:1128)
> >
> > at
> >
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
> lExecutor.java:628)
> >
> > at java.base/java.lang.Thread.run(Thread.java:834)
> >
> > 2020-07-02 09:09:00,804 DEBUG [c.c.a.m.DirectAgentAttache]
> > (DirectAgent-17:ctx-681dc524) (logid:1cfa64a9) Seq 1-4541880224203147245:
> > Cancelling because one of the answers is false and it is stop on error.
> >
> > 2020-07-02 09:09:00,804 DEBUG [c.c.a.m.DirectAgentAttache]
> > (DirectAgent-17:ctx-681dc524) (logid:1cfa64a9) Seq 1-4541880224203147245:
> > Response Received:
> >
> > 2020-07-02 09:09:00,805 DEBUG [c.c.a.t.Request]
> > (DirectAgent-17:ctx-681dc524) (logid:1cfa64a9) Seq 1-4541880224203147245:
> > Processing: { Ans: , MgmtId: 345050029058, via:
> > 1(usphlmvesxt154.phl.global.corp.sap), Ver: v1, Flags: 110,
> >
> [{"com.cloud.agent.api.StartAnswer":{"vm":{"id":2,"name":"v-2-VM","boo
> tloader":"HVM","state":"Starting","type":"ConsoleProxy","cpus":1,"minS
> peed":500,"maxSpeed":500,"minRam":1073741824,"maxRam":1073741824,"host
> Name":"v-2-VM","arch":"x86_64","os":"Other
> > Linux (64-bit)","platformEmulator":"otherLinux64Guest","bootArgs":"
> > template=domP type=consoleproxy host=10.4.32.163 port=8250
> > name=v-2-VM
> > zone=1 pod=1 guid=Proxy.2 proxy_vm=2 disable_rp_filter=true
> > eth2ip=10.4.42.25 eth2mask=255.255.255.192 gateway=10.4.42.1
> eth0ip=0.0.0.0
> > eth0mask=0.0.0.0 eth1ip=10.4.4.35 eth1mask=255.255.252.0 mgmtcidr=
> > 10.4.32.0/24 localgw=10.4.4.1 internaldns1=10.4.202.200
> dns1=10.4.202.200
> >
> dns2=10.4.12.200","enableHA":false,"limitCpuUse":false,"enableDynamica
> llyScaleVm":false,"vncPassword":"sYnwDJYgzrMKdm3ftAO-kg","params":{"vm
> ware.reserve.mem":"false","vmware.reserve.cpu":"false","nicAdapter":"E
> 1000"},"uuid":"fb65df73-693d-48fb-a203-c2b5e776842a","disks":[{"data":
> {"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"9a1f2ae8-6
> b10-4525-8a54-a175b153b789","volumeType":"ROOT","dataStore":{"org.apac
> he.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"50cc970f-fbe2-3b
> 45-9cb5-b8bcdb6dbff5","id":1,"poolType":"NetworkFilesystem","host":"us
> phlcv300.phl.global.corp.sap","path":"/usphlmvesx_lab_cspri","port":20
> 49,"url":"NetworkFilesystem://usphlcv300.phl.global.corp.sap/usphlmves
> x_lab_cspri/?ROLE=Primary&STOREUUID=50cc970f-fbe2-3b45-9cb5-b8bcdb6dbf
> f5","isManaged":false}},"name":"ROOT-2","size":0,"path":"ROOT-2_2","vo
> lumeId":2,"vmName":"v-2-VM","accountId":1,"format":"OVA","provisioning
> Type":"THIN","id":2,"deviceId":0,"bytesReadRate":0,"bytesWriteRate":0,
> "iopsReadRate":0,"iopsWriteRate":0,"hypervisorType":"VMware","directDo
> wnload":false}},"diskSeq":0,"path":"ROOT-2_2","type":"ROOT","_details"
> :{"storageHost":"usphlcv300.phl.global.corp.sap","managed":"false","st
> oragePort":"2049","volumeSize":"0"}}],"nics":[{"deviceId":2,"networkRa
> teMbps":-1,"defaultNic":true,"pxeDisable":true,"nicUuid":"ffbc1822-790
> 7-4cdf-9e80-bb316786e04d","details":{"PromiscuousMode":"false","MacAdd
> ressChanges":"true","ForgedTransmits":"true"},"dpdkEnabled":false,"uui
> d":"f3a30021-a469-413d-85a8-ca1ca68ba27c","ip":"10.4.42.25","netmask":
> "255.255.255.192","gateway":"10.4.42.1","mac":"1e:00:e3:00:00:19","dns
> 1":"10.4.202.200","dns2":"10.4.12.200","broadcastType":"Vlan","type":"
> Public","broadcastUri":"vlan://untagged","isolationUri":"vlan://untagg
> ed","isSecurityGroupEnabled":false,"name":"vSwitch0,134,vmwaresvs"},{"
> deviceId":0,"networkRateMbps":-1,"defaultNic":false,"pxeDisable":true,
> "nicUuid":"d6521be0-85b8-4836-bf8b-8461afdf593a","details":{"Promiscuo
> usMode":"false","MacAddressChanges":"true","ForgedTransmits":"true"},"
> dpdkEnabled":false,"uuid":"eda0ae3d-d15d-4f91-b8db-a719cf5a94bf","mac"
> :"02:00:31:71:00:02","broadcastType":"LinkLocal","type":"Control","isS
> ecurityGroupEnabled":false,"name":"vSwitch0,146,vmwaresvs"},{"deviceId
> ":1,"networkRateMbps":-1,"defaultNic":false,"pxeDisable":true,"nicUuid
> ":"4e6201ad-91c0-4cab-bf53-9e2440025245","details":{"PromiscuousMode":
> "false","MacAddressChanges":"true","ForgedTransmits":"true"},"dpdkEnab
> led":false,"uuid":"ce88b030-c04c-4437-b60d-9910c2cc376f","ip":"10.4.4.
> 35","netmask":"255.255.252.0","gateway":"10.4.4.1","mac":"1e:00:6b:00:
> 00:27","broadcastType":"Native","type":"Management","isSecurityGroupEn
> abled":false,"name":"vSwitch0,146,vmwaresvs"}],"guestOsDetails":{},"ex
> traConfig":{}},"result":false,"details":"StartCommand
> > failed due to Exception:
> > com.cloud.utils.exception.CloudRuntimeException\nMessage: Unable to
> create
> > systemvm folder on secondary storage. location:
> > /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm\n","wait":0}}]
> > }
> >
> > 2020-07-02 09:09:00,805 DEBUG [c.c.a.m.AgentAttache]
> > (DirectAgent-17:ctx-681dc524) (logid:1cfa64a9) Seq 1-4541880224203147245:
> > No more commands found
> >
> > 2020-07-02 09:09:00,806 DEBUG [c.c.a.t.Request]
> > (Work-Job-Executor-2:ctx-9aa38618 job-16/job-35 ctx-7ebd1b78)
> > (logid:1cfa64a9) Seq 1-4541880224203147245: Received: { Ans: , MgmtId:
> > 345050029058, via: 1(usphlmvesxt154.phl.global.corp.sap), Ver: v1, Flags:
> > 110, { StartAnswer } }
> >
> > 2020-07-02 09:09:00,816 INFO [c.c.v.VirtualMachineManagerImpl]
> > (Work-Job-Executor-2:ctx-9aa38618 job-16/job-35 ctx-7ebd1b78)
> > (logid:1cfa64a9) Unable to start VM on Host[-1-Routing] due to
> StartCommand
> > failed due to Exception:
> > com.cloud.utils.exception.CloudRuntimeException
> >
> > Message: Unable to create systemvm folder on secondary storage. location:
> > /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm
> >
> >
> >
> > *Mike Corey*
> >
> >
> > Technology Senior Consultant, IT CS CTW Operation & Virtualization
> Service
> > US
> >
> >
> > *SAP AMERICA, INC.* 3999 West Chester Pike, Newtown Square, 19073
> > United States
> >
> >
> > T +1 610 661 0905, M +1 484 274 2658, E mike.corey@sap.com
> >
> >
> >
> >
> >
> >
> >
>
>
> --
>
> Andrija Panić
>
RE: Initial SystemVM Creation Failing
Posted by "Corey, Mike" <mi...@sap.com>.
The system VM folder is being created as owner & permissions nobody:nobody. I have to believe that is NOT SUPPOSED to be the case.
Because the directory /var/cloudstack/mnt/VM/345050029058.4f87a734 is locked down, no other steps in the systemvm creation can continue.
I need help figuring out WHY the systemVM folder in the above directory is being created as nobody:nobody.
I can't chown that directory once CloudStack creates it during the zone enablement. I can't modify the permissions to that directory once CloudStack creates it during the zone enablement. I can't copy files to that directory once CloudStack creates it during the zone enablement.
Please identify what is occurring during the creation of the systemvm directory that would cause the ownership to be set as nobody:nobody.
Mike
Message: Unable to copy systemvm ISO on secondary storage. src location: /usr/share/cloudstack-common/vms/systemvm.iso, dest location: /var/cloudstack/mnt/VM/345050029058.4f87a734/systemvm/systemvm-4.14.0.0.iso
[root@usphlmvcsmt01 VM]# pwd
/var/cloudstack/mnt/VM
[root@usphlmvcsmt01 VM]# ls -al
total 80864
drwxr-xr-x. 3 cloud cloud 55 Jul 10 11:46 .
drwxrwx---. 3 root cloud 16 Jul 9 14:40 ..
drwxr-xr-x. 5 nobody nobody 4096 Jul 10 11:31 345050029058.4f87a734
-rw-r--r--. 1 cloud cloud 82800640 Jul 10 11:43 systemvm.iso
-----Original Message-----
From: Cristian Ciobanu <cr...@istream.today>
Sent: Friday, July 10, 2020 5:26 AM
To: users@cloudstack.apache.org
Subject: Re: Initial SystemVM Creation Failing
Just try to execute the command manually, copy the system.iso from source
to the destination. I had a similar issue but is was caused by the NFS
permissions.
On Wed, 8 Jul 2020, 21:44 Corey, Mike, <mi...@sap.com> wrote:
> Gone over the installation now twice (reverted to snapshot of CentOS VM)
> and still getting weird storage/permission type issues.
>
> The permissions and ownership of the systemvm folder looks correct.
> [root@ ~]# ls -al /mnt/secondary/
> total 16
> drwxr-xr-x. 4 root root 4096 Jul 8 11:15 .
> drwxr-xr-x. 4 root root 38 Jul 8 11:15 ..
> drwxrwxrwx. 12 root root 4096 Jul 8 11:05 .snapshot
> drwxrwxr-x. 2 cloud cloud 4096 Jul 8 10:58 systemvm
> drwxr-xr-x. 3 root root 4096 Jul 8 11:15 template
>
> However, I'm still seeing the copy iso error and the VMs keep creating and
> deleting with the log suggesting the vmx file of the VM doesn't exist on
> the datastore (can create files/folder on datastore identified in errror
> from vCenter).
>
> 2020-07-08 14:22:36,486 INFO [c.c.h.v.m.DatastoreMO]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Folder v-6-VM does not
> exist on datastore
> 2020-07-08 14:22:36,494 INFO [c.c.h.v.m.DatastoreMO]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Search file v-6-VM.vmx on
> [50cc970ffbe23b459cb5b8bcdb6dbff5]
> 2020-07-08 14:22:36,522 INFO [c.c.h.v.m.DatastoreMO]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) File
> [50cc970ffbe23b459cb5b8bcdb6dbff5] v-6-VM.vmx does not exist on datastore
> 2020-07-08 14:22:36,522 INFO [c.c.h.v.m.DatastoreMO]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Searching file v-6-VM.vmx
> in [50cc970ffbe23b459cb5b8bcdb6dbff5]
> 2020-07-08 14:22:36,641 INFO [c.c.h.v.m.HypervisorHostHelper]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Create blank VM.
> cpuCount: 1, cpuSpeed(MHz): 500, mem(Mb): 1024
> 2020-07-08 14:22:36,692 DEBUG [c.c.h.v.m.ClusterMO]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Retrieved recommended
> disk controller for guest OS : otherLinux64Guest in cluster PHL-CSM01 :
> VirtualLsiLogicController
> 2020-07-08 14:22:37,115 DEBUG [c.c.h.v.m.HostMO]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) find VM v-6-VM on host
> 2020-07-08 14:22:37,115 INFO [c.c.h.v.m.HostMO]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) VM v-6-VM not found in
> host cache
> 2020-07-08 14:22:37,115 DEBUG [c.c.h.v.m.HostMO]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) load VM cache on host
> 2020-07-08 14:22:37,155 DEBUG [c.c.h.v.m.HostMO]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) find VM v-6-VM on host
> 2020-07-08 14:22:37,155 DEBUG [c.c.h.v.m.HostMO]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) VM v-6-VM found in host
> cache
> 2020-07-08 14:22:37,375 INFO [c.c.h.v.m.VmwareManagerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Inject SSH key pairs
> before copying systemvm.iso into secondary storage
> 2020-07-08 14:22:37,377 INFO [c.c.s.ConfigurationServerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Processing updateKeyPairs
> 2020-07-08 14:22:37,377 INFO [c.c.s.ConfigurationServerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Keypairs already in
> database, updating local copy
> 2020-07-08 14:22:37,381 INFO [c.c.s.ConfigurationServerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Going to update systemvm
> iso with generated keypairs if needed
> 2020-07-08 14:22:37,381 INFO [c.c.s.ConfigurationServerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Trying to inject public
> and private keys into systemvm iso
> 2020-07-08 14:22:37,381 DEBUG [c.c.u.s.Script]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Looking for
> scripts/vm/systemvm/injectkeys.sh in the classpath
> 2020-07-08 14:22:37,381 DEBUG [c.c.u.s.Script]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) System resource:
> file:/usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh
> 2020-07-08 14:22:37,381 DEBUG [c.c.u.s.Script]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Absolute path =
> /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh
> 2020-07-08 14:22:37,381 DEBUG [c.c.u.s.Script]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Looking for
> vms/systemvm.iso in the classpath
> 2020-07-08 14:22:37,381 DEBUG [c.c.u.s.Script]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) System resource:
> file:/usr/share/cloudstack-common/vms/systemvm.iso
> 2020-07-08 14:22:37,381 DEBUG [c.c.u.s.Script]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Absolute path =
> /usr/share/cloudstack-common/vms/systemvm.iso
> 2020-07-08 14:22:37,381 DEBUG [c.c.s.ConfigurationServerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Executing: /bin/bash
> /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh
> /var/cloudstack/management/.ssh/id_rsa.pub
> /var/cloudstack/management/.ssh/id_rsa
> /usr/share/cloudstack-common/vms/systemvm.iso
> 2020-07-08 14:22:37,384 DEBUG [c.c.s.ConfigurationServerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Executing while with
> timeout : 3600000
> 2020-07-08 14:22:37,487 DEBUG [c.c.s.ConfigurationServerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Execution is successful.
> 2020-07-08 14:22:37,494 DEBUG [c.c.s.ConfigurationServerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) mount: /dev/loop0 is
> write-protected, mounting read-only
> New public key is the same as the one in the systemvm.iso, not injecting
> it, not modifying systemvm.iso
>
> 2020-07-08 14:22:37,494 INFO [c.c.s.ConfigurationServerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) The script injectkeys.sh
> was run with result : null
> 2020-07-08 14:22:37,494 INFO [c.c.h.v.m.VmwareManagerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Copy System VM patch ISO
> file to secondary storage. source ISO:
> /usr/share/cloudstack-common/vms/systemvm.iso, destination:
> /var/cloudstack/mnt/VM/345050029058.1f7af2bf/systemvm/systemvm-4.14.0.0.iso
> 2020-07-08 14:22:37,495 ERROR [c.c.h.v.m.VmwareManagerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Unexpected exception
> java.io.FileNotFoundException:
> /var/cloudstack/mnt/VM/345050029058.1f7af2bf/systemvm/systemvm-4.14.0.0.iso
> (Permission denied)
>
> But, the ownership/permissions look okay on that destination directory too.
>
> [root@ ~]# ls -al /var/cloudstack/mnt/VM/345050029058.1f7af2bf
> total 12
> drwxr-xr-x. 4 nobody nobody 4096 Jul 8 11:15 .
> drwxr-xr-x. 3 cloud cloud 35 Jul 8 14:06 ..
> drwxrwxr-x. 2 cloud cloud 4096 Jul 8 10:58 systemvm
> drwxr-xr-x. 3 nobody nobody 4096 Jul 8 11:15 template
> [root@usphlmvcsmt01 ~]#
>
>
> Any ideas?
>
>
> -----Original Message-----
> From: Corey, Mike
> Sent: Thursday, July 2, 2020 2:31 PM
> To: users@cloudstack.apache.org
> Subject: RE: Initial SystemVM Creation Failing
>
> No upgrade - this again is the fresh install of CloudStack 4.14 with
> VMware resources. Maybe I should note that I created the primary and
> secondary storage via the "Advanced" setup wizard from the UI (logged in as
> admin).
>
> I performed the ownership change you suggested so now cloud owns the
> systemvm folder on the secondary storage. However, now I'm getting a new
> error regarding copying the ISO.
>
> Should I just bag the 4.14 installation and try to get 4.13.1 running with
> VMware?
>
>
> 2020-07-02 14:19:13,314 INFO [c.c.h.v.m.VmwareManagerImpl]
> (DirectAgent-10:ctx-52902ebf usphlmvesxt155.phl.global.corp.sap,
> job-15/job-33, cmd: StartCommand) (logid:b8ba37d1) Copy System VM patch ISO
> file to secondary storage. source ISO:
> /usr/share/cloudstack-common/vms/systemvm.iso, destination:
> /var/cloudstack/mnt/VM/345050029058.777ecb60/systemvm/systemvm-4.14.0.0.iso
> 2020-07-02 14:19:13,320 ERROR [c.c.h.v.m.VmwareManagerImpl]
> (DirectAgent-10:ctx-52902ebf usphlmvesxt155.phl.global.corp.sap,
> job-15/job-33, cmd: StartCommand) (logid:b8ba37d1) Unexpected exception
> java.io.FileNotFoundException:
> /var/cloudstack/mnt/VM/345050029058.777ecb60/systemvm/systemvm-4.14.0.0.iso
> (Permission denied)
> at java.base/java.io.FileOutputStream.open0(Native Method)
> at java.base/java.io
> .FileOutputStream.open(FileOutputStream.java:298)
> at java.base/java.io
> .FileOutputStream.<init>(FileOutputStream.java:237)
> at java.base/java.io
> .FileOutputStream.<init>(FileOutputStream.java:187)
> at org.apache.commons.io.FileUtils.doCopyFile(FileUtils.java:1123)
> at org.apache.commons.io.FileUtils.copyFile(FileUtils.java:1076)
> at org.apache.commons.io.FileUtils.copyFile(FileUtils.java:1028)
> at com.cloud.utils.FileUtil.copyfile(FileUtil.java:30)
> at
> com.cloud.hypervisor.vmware.manager.VmwareManagerImpl.prepareSecondaryStorageStore(VmwareManagerImpl.java:661)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
> at com.sun.proxy.$Proxy389.prepareSecondaryStorageStore(Unknown
> Source)
> at
> com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:1962)
> at
> com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:530)
> at
> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at
> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2020-07-02 14:19:13,321 ERROR [c.c.h.v.m.VmwareManagerImpl]
> (DirectAgent-10:ctx-52902ebf usphlmvesxt155.phl.global.corp.sap,
> job-15/job-33, cmd: StartCommand) (logid:b8ba37d1) Unable to copy systemvm
> ISO on secondary storage. src location:
> /usr/share/cloudstack-common/vms/systemvm.iso, dest location:
> /var/cloudstack/mnt/VM/345050029058.777ecb60/systemvm/systemvm-4.14.0.0.iso
> 2020-07-02 14:19:13,322 INFO [c.c.h.v.u.VmwareHelper]
> (DirectAgent-10:ctx-52902ebf usphlmvesxt155.phl.global.corp.sap,
> job-15/job-33, cmd: StartCommand) (logid:b8ba37d1) [ignored]failed to get
> message for exception: Unable to copy systemvm ISO on secondary storage.
> src location: /usr/share/cloudstack-common/vms/systemvm.iso, dest location:
> /var/cloudstack/mnt/VM/345050029058.777ecb60/systemvm/systemvm-4.14.0.0.iso
> 2020-07-02 14:19:13,323 WARN [c.c.h.v.r.VmwareResource]
> (DirectAgent-10:ctx-52902ebf usphlmvesxt155.phl.global.corp.sap,
> job-15/job-33, cmd: StartCommand) (logid:b8ba37d1) StartCommand failed due
> to Exception: com.cloud.utils.exception.CloudRuntimeException
> Message: Unable to copy systemvm ISO on secondary storage. src location:
> /usr/share/cloudstack-common/vms/systemvm.iso, dest location:
> /var/cloudstack/mnt/VM/345050029058.777ecb60/systemvm/systemvm-4.14.0.0.iso
>
>
>
>
> -----Original Message-----
> From: Andrija Panic <an...@gmail.com>
> Sent: Thursday, July 2, 2020 12:34 PM
> To: users <us...@cloudstack.apache.org>
> Subject: Re: Initial SystemVM Creation Failing
>
> Hi Mike,
>
> Mount you secondary storage on the management server to some temp folder,
> then run: chown -R cloud:cloud <MOUNTPOINT>/systemvm/
> Other folders on Secondary Storage NFS are owned by root, but the
> "systemvm" should be owned by "cloud".
>
> I assume you got this after some ACS upgrade, or similar?
>
> On Thu, 2 Jul 2020 at 15:51, Corey, Mike <mi...@sap.com> wrote:
>
> > Hi,
> >
> >
> >
> > When I enable the zone, the systemvm creation begins but goes into a loop
> > of creating and deleting VMs in vCenter. I see in the logs there is an
> > error for creating the folder permissions on the secondary storage (see
> > below). Running this CHMOD manually from inside the CSM kicks the same
> > error.
> >
> >
> >
> > Please provide suggestions.
> >
> >
> >
> > 2020-07-02 09:08:39,665 DEBUG [c.c.h.v.m.VmwareManagerImpl]
> > (DirectAgent-66:ctx-e15c32b2 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: CopyCommand) (logid:1cfa64a9) Executing: sudo chmod
> > 1777 /var/cloudstack/mnt/VM/345050029058.2807345c
> >
> > 2020-07-02 09:08:39,666 DEBUG [c.c.h.v.m.VmwareManagerImpl]
> > (DirectAgent-66:ctx-e15c32b2 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: CopyCommand) (logid:1cfa64a9) Executing while with
> > timeout : 1440000
> >
> > 2020-07-02 09:08:39,693 DEBUG [c.c.h.v.m.VmwareManagerImpl]
> > (DirectAgent-66:ctx-e15c32b2 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: CopyCommand) (logid:1cfa64a9) Exit value is 1
> >
> > 2020-07-02 09:08:39,697 DEBUG [c.c.h.v.m.VmwareManagerImpl]
> > (DirectAgent-66:ctx-e15c32b2 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: CopyCommand) (logid:1cfa64a9) chmod: changing
> > permissions of ‘/var/cloudstack/mnt/VM/345050029058.2807345c’: Operation
> > not permitted
> >
> > 2020-07-02 09:08:39,699 WARN [c.c.h.v.m.VmwareManagerImpl]
> > (DirectAgent-66:ctx-e15c32b2 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: CopyCommand) (logid:1cfa64a9) Unable to set
> permissions
> > for /var/cloudstack/mnt/VM/345050029058.2807345c due to chmod: changing
> > permissions of ‘/var/cloudstack/mnt/VM/345050029058.2807345c’: Operation
> > not permitted
> >
> >
> >
> > Lines…
> >
> >
> >
> > 2020-07-02 09:09:00,598 DEBUG [c.c.h.v.m.HostMO]
> > (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) find VM v-2-VM on host
> >
> > 2020-07-02 09:09:00,598 INFO [c.c.h.v.m.HostMO]
> > (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) VM v-2-VM not found in
> > host cache
> >
> > 2020-07-02 09:09:00,598 DEBUG [c.c.h.v.m.HostMO]
> > (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) load VM cache on host
> >
> > 2020-07-02 09:09:00,616 DEBUG [c.c.h.v.m.HostMO]
> > (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) find VM v-2-VM on host
> >
> > 2020-07-02 09:09:00,617 DEBUG [c.c.h.v.m.HostMO]
> > (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) VM v-2-VM found in
> host
> > cache
> >
> > 2020-07-02 09:09:00,799 ERROR [c.c.h.v.m.VmwareManagerImpl]
> > (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) Unable to create
> > systemvm folder on secondary storage. location:
> > /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm
> >
> > 2020-07-02 09:09:00,800 INFO [c.c.h.v.u.VmwareHelper]
> > (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) [ignored]failed to get
> > message for exception: Unable to create systemvm folder on secondary
> > storage. location: /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm
> >
> > 2020-07-02 09:09:00,800 WARN [c.c.h.v.r.VmwareResource]
> > (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) StartCommand failed
> due
> > to Exception: com.cloud.utils.exception.CloudRuntimeException
> >
> > Message: Unable to create systemvm folder on secondary storage. location:
> > /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm
> >
> >
> >
> > com.cloud.utils.exception.CloudRuntimeException: Unable to create
> systemvm
> > folder on secondary storage. location:
> > /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm
> >
> > at
> >
> com.cloud.hypervisor.vmware.manager.VmwareManagerImpl.prepareSecondaryStorageStore(VmwareManagerImpl.java:648)
> >
> > at
> > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> > Method)
> >
> > at
> >
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> >
> > at
> >
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >
> > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> >
> > at
> >
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
> >
> > at
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
> >
> > at
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> >
> > at
> >
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95)
> >
> > at
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
> >
> > at
> >
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
> >
> > at
> > com.sun.proxy.$Proxy389.prepareSecondaryStorageStore(Unknown Source)
> >
> > at
> >
> com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:1962)
> >
> > at
> >
> com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:530)
> >
> > at
> >
> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
> >
> > at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> >
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> >
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> >
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> >
> > at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> >
> > at
> >
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> >
> > at
> > java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> >
> > at
> >
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
> >
> > at
> >
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> >
> > at
> >
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> >
> > at java.base/java.lang.Thread.run(Thread.java:834)
> >
> > 2020-07-02 09:09:00,804 DEBUG [c.c.a.m.DirectAgentAttache]
> > (DirectAgent-17:ctx-681dc524) (logid:1cfa64a9) Seq 1-4541880224203147245:
> > Cancelling because one of the answers is false and it is stop on error.
> >
> > 2020-07-02 09:09:00,804 DEBUG [c.c.a.m.DirectAgentAttache]
> > (DirectAgent-17:ctx-681dc524) (logid:1cfa64a9) Seq 1-4541880224203147245:
> > Response Received:
> >
> > 2020-07-02 09:09:00,805 DEBUG [c.c.a.t.Request]
> > (DirectAgent-17:ctx-681dc524) (logid:1cfa64a9) Seq 1-4541880224203147245:
> > Processing: { Ans: , MgmtId: 345050029058, via:
> > 1(usphlmvesxt154.phl.global.corp.sap), Ver: v1, Flags: 110,
> >
> [{"com.cloud.agent.api.StartAnswer":{"vm":{"id":2,"name":"v-2-VM","bootloader":"HVM","state":"Starting","type":"ConsoleProxy","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":1073741824,"maxRam":1073741824,"hostName":"v-2-VM","arch":"x86_64","os":"Other
> > Linux (64-bit)","platformEmulator":"otherLinux64Guest","bootArgs":"
> > template=domP type=consoleproxy host=10.4.32.163 port=8250 name=v-2-VM
> > zone=1 pod=1 guid=Proxy.2 proxy_vm=2 disable_rp_filter=true
> > eth2ip=10.4.42.25 eth2mask=255.255.255.192 gateway=10.4.42.1
> eth0ip=0.0.0.0
> > eth0mask=0.0.0.0 eth1ip=10.4.4.35 eth1mask=255.255.252.0 mgmtcidr=
> > 10.4.32.0/24 localgw=10.4.4.1 internaldns1=10.4.202.200
> dns1=10.4.202.200
> >
> dns2=10.4.12.200","enableHA":false,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"sYnwDJYgzrMKdm3ftAO-kg","params":{"vmware.reserve.mem":"false","vmware.reserve.cpu":"false","nicAdapter":"E1000"},"uuid":"fb65df73-693d-48fb-a203-c2b5e776842a","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"9a1f2ae8-6b10-4525-8a54-a175b153b789","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"50cc970f-fbe2-3b45-9cb5-b8bcdb6dbff5","id":1,"poolType":"NetworkFilesystem","host":"usphlcv300.phl.global.corp.sap","path":"/usphlmvesx_lab_cspri","port":2049,"url":"NetworkFilesystem://usphlcv300.phl.global.corp.sap/usphlmvesx_lab_cspri/?ROLE=Primary&STOREUUID=50cc970f-fbe2-3b45-9cb5-b8bcdb6dbff5","isManaged":false}},"name":"ROOT-2","size":0,"path":"ROOT-2_2","volumeId":2,"vmName":"v-2-VM","accountId":1,"format":"OVA","provisioningType":"THIN","id":2,"deviceId":0,"bytesReadRate":0,"bytesWriteRate":0,"iopsReadRate":0,"iopsWriteRate":0,"hypervisorType":"VMware","directDownload":false}},"diskSeq":0,"path":"ROOT-2_2","type":"ROOT","_details":{"storageHost":"usphlcv300.phl.global.corp.sap","managed":"false","storagePort":"2049","volumeSize":"0"}}],"nics":[{"deviceId":2,"networkRateMbps":-1,"defaultNic":true,"pxeDisable":true,"nicUuid":"ffbc1822-7907-4cdf-9e80-bb316786e04d","details":{"PromiscuousMode":"false","MacAddressChanges":"true","ForgedTransmits":"true"},"dpdkEnabled":false,"uuid":"f3a30021-a469-413d-85a8-ca1ca68ba27c","ip":"10.4.42.25","netmask":"255.255.255.192","gateway":"10.4.42.1","mac":"1e:00:e3:00:00:19","dns1":"10.4.202.200","dns2":"10.4.12.200","broadcastType":"Vlan","type":"Public","broadcastUri":"vlan://untagged","isolationUri":"vlan://untagged","isSecurityGroupEnabled":false,"name":"vSwitch0,134,vmwaresvs"},{"deviceId":0,"networkRateMbps":-1,"defaultNic":false,"pxeDisable":true,"nicUuid":"d6521be0-85b8-4836-bf8b-8461afdf593a","details":{"PromiscuousMode":"false","MacAddressChanges":"true","ForgedTransmits":"true"},"dpdkEnabled":false,"uuid":"eda0ae3d-d15d-4f91-b8db-a719cf5a94bf","mac":"02:00:31:71:00:02","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false,"name":"vSwitch0,146,vmwaresvs"},{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"pxeDisable":true,"nicUuid":"4e6201ad-91c0-4cab-bf53-9e2440025245","details":{"PromiscuousMode":"false","MacAddressChanges":"true","ForgedTransmits":"true"},"dpdkEnabled":false,"uuid":"ce88b030-c04c-4437-b60d-9910c2cc376f","ip":"10.4.4.35","netmask":"255.255.252.0","gateway":"10.4.4.1","mac":"1e:00:6b:00:00:27","broadcastType":"Native","type":"Management","isSecurityGroupEnabled":false,"name":"vSwitch0,146,vmwaresvs"}],"guestOsDetails":{},"extraConfig":{}},"result":false,"details":"StartCommand
> > failed due to Exception:
> > com.cloud.utils.exception.CloudRuntimeException\nMessage: Unable to
> create
> > systemvm folder on secondary storage. location:
> > /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm\n","wait":0}}] }
> >
> > 2020-07-02 09:09:00,805 DEBUG [c.c.a.m.AgentAttache]
> > (DirectAgent-17:ctx-681dc524) (logid:1cfa64a9) Seq 1-4541880224203147245:
> > No more commands found
> >
> > 2020-07-02 09:09:00,806 DEBUG [c.c.a.t.Request]
> > (Work-Job-Executor-2:ctx-9aa38618 job-16/job-35 ctx-7ebd1b78)
> > (logid:1cfa64a9) Seq 1-4541880224203147245: Received: { Ans: , MgmtId:
> > 345050029058, via: 1(usphlmvesxt154.phl.global.corp.sap), Ver: v1, Flags:
> > 110, { StartAnswer } }
> >
> > 2020-07-02 09:09:00,816 INFO [c.c.v.VirtualMachineManagerImpl]
> > (Work-Job-Executor-2:ctx-9aa38618 job-16/job-35 ctx-7ebd1b78)
> > (logid:1cfa64a9) Unable to start VM on Host[-1-Routing] due to
> StartCommand
> > failed due to Exception: com.cloud.utils.exception.CloudRuntimeException
> >
> > Message: Unable to create systemvm folder on secondary storage. location:
> > /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm
> >
> >
> >
> > *Mike Corey*
> >
> >
> > Technology Senior Consultant, IT CS CTW Operation & Virtualization
> Service
> > US
> >
> >
> > *SAP AMERICA, INC.* 3999 West Chester Pike, Newtown Square, 19073 United
> > States
> >
> >
> > T +1 610 661 0905, M +1 484 274 2658, E mike.corey@sap.com
> >
> >
> >
> >
> >
> >
> >
>
>
> --
>
> Andrija Panić
>
Re: Initial SystemVM Creation Failing
Posted by Cristian Ciobanu <cr...@istream.today>.
Just try to execute the command manually, copy the system.iso from source
to the destination. I had a similar issue but is was caused by the NFS
permissions.
On Wed, 8 Jul 2020, 21:44 Corey, Mike, <mi...@sap.com> wrote:
> Gone over the installation now twice (reverted to snapshot of CentOS VM)
> and still getting weird storage/permission type issues.
>
> The permissions and ownership of the systemvm folder looks correct.
> [root@ ~]# ls -al /mnt/secondary/
> total 16
> drwxr-xr-x. 4 root root 4096 Jul 8 11:15 .
> drwxr-xr-x. 4 root root 38 Jul 8 11:15 ..
> drwxrwxrwx. 12 root root 4096 Jul 8 11:05 .snapshot
> drwxrwxr-x. 2 cloud cloud 4096 Jul 8 10:58 systemvm
> drwxr-xr-x. 3 root root 4096 Jul 8 11:15 template
>
> However, I'm still seeing the copy iso error and the VMs keep creating and
> deleting with the log suggesting the vmx file of the VM doesn't exist on
> the datastore (can create files/folder on datastore identified in errror
> from vCenter).
>
> 2020-07-08 14:22:36,486 INFO [c.c.h.v.m.DatastoreMO]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Folder v-6-VM does not
> exist on datastore
> 2020-07-08 14:22:36,494 INFO [c.c.h.v.m.DatastoreMO]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Search file v-6-VM.vmx on
> [50cc970ffbe23b459cb5b8bcdb6dbff5]
> 2020-07-08 14:22:36,522 INFO [c.c.h.v.m.DatastoreMO]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) File
> [50cc970ffbe23b459cb5b8bcdb6dbff5] v-6-VM.vmx does not exist on datastore
> 2020-07-08 14:22:36,522 INFO [c.c.h.v.m.DatastoreMO]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Searching file v-6-VM.vmx
> in [50cc970ffbe23b459cb5b8bcdb6dbff5]
> 2020-07-08 14:22:36,641 INFO [c.c.h.v.m.HypervisorHostHelper]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Create blank VM.
> cpuCount: 1, cpuSpeed(MHz): 500, mem(Mb): 1024
> 2020-07-08 14:22:36,692 DEBUG [c.c.h.v.m.ClusterMO]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Retrieved recommended
> disk controller for guest OS : otherLinux64Guest in cluster PHL-CSM01 :
> VirtualLsiLogicController
> 2020-07-08 14:22:37,115 DEBUG [c.c.h.v.m.HostMO]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) find VM v-6-VM on host
> 2020-07-08 14:22:37,115 INFO [c.c.h.v.m.HostMO]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) VM v-6-VM not found in
> host cache
> 2020-07-08 14:22:37,115 DEBUG [c.c.h.v.m.HostMO]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) load VM cache on host
> 2020-07-08 14:22:37,155 DEBUG [c.c.h.v.m.HostMO]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) find VM v-6-VM on host
> 2020-07-08 14:22:37,155 DEBUG [c.c.h.v.m.HostMO]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) VM v-6-VM found in host
> cache
> 2020-07-08 14:22:37,375 INFO [c.c.h.v.m.VmwareManagerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Inject SSH key pairs
> before copying systemvm.iso into secondary storage
> 2020-07-08 14:22:37,377 INFO [c.c.s.ConfigurationServerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Processing updateKeyPairs
> 2020-07-08 14:22:37,377 INFO [c.c.s.ConfigurationServerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Keypairs already in
> database, updating local copy
> 2020-07-08 14:22:37,381 INFO [c.c.s.ConfigurationServerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Going to update systemvm
> iso with generated keypairs if needed
> 2020-07-08 14:22:37,381 INFO [c.c.s.ConfigurationServerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Trying to inject public
> and private keys into systemvm iso
> 2020-07-08 14:22:37,381 DEBUG [c.c.u.s.Script]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Looking for
> scripts/vm/systemvm/injectkeys.sh in the classpath
> 2020-07-08 14:22:37,381 DEBUG [c.c.u.s.Script]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) System resource:
> file:/usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh
> 2020-07-08 14:22:37,381 DEBUG [c.c.u.s.Script]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Absolute path =
> /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh
> 2020-07-08 14:22:37,381 DEBUG [c.c.u.s.Script]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Looking for
> vms/systemvm.iso in the classpath
> 2020-07-08 14:22:37,381 DEBUG [c.c.u.s.Script]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) System resource:
> file:/usr/share/cloudstack-common/vms/systemvm.iso
> 2020-07-08 14:22:37,381 DEBUG [c.c.u.s.Script]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Absolute path =
> /usr/share/cloudstack-common/vms/systemvm.iso
> 2020-07-08 14:22:37,381 DEBUG [c.c.s.ConfigurationServerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Executing: /bin/bash
> /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh
> /var/cloudstack/management/.ssh/id_rsa.pub
> /var/cloudstack/management/.ssh/id_rsa
> /usr/share/cloudstack-common/vms/systemvm.iso
> 2020-07-08 14:22:37,384 DEBUG [c.c.s.ConfigurationServerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Executing while with
> timeout : 3600000
> 2020-07-08 14:22:37,487 DEBUG [c.c.s.ConfigurationServerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Execution is successful.
> 2020-07-08 14:22:37,494 DEBUG [c.c.s.ConfigurationServerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) mount: /dev/loop0 is
> write-protected, mounting read-only
> New public key is the same as the one in the systemvm.iso, not injecting
> it, not modifying systemvm.iso
>
> 2020-07-08 14:22:37,494 INFO [c.c.s.ConfigurationServerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) The script injectkeys.sh
> was run with result : null
> 2020-07-08 14:22:37,494 INFO [c.c.h.v.m.VmwareManagerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Copy System VM patch ISO
> file to secondary storage. source ISO:
> /usr/share/cloudstack-common/vms/systemvm.iso, destination:
> /var/cloudstack/mnt/VM/345050029058.1f7af2bf/systemvm/systemvm-4.14.0.0.iso
> 2020-07-08 14:22:37,495 ERROR [c.c.h.v.m.VmwareManagerImpl]
> (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap,
> job-6/job-37, cmd: StartCommand) (logid:b828ac16) Unexpected exception
> java.io.FileNotFoundException:
> /var/cloudstack/mnt/VM/345050029058.1f7af2bf/systemvm/systemvm-4.14.0.0.iso
> (Permission denied)
>
> But, the ownership/permissions look okay on that destination directory too.
>
> [root@ ~]# ls -al /var/cloudstack/mnt/VM/345050029058.1f7af2bf
> total 12
> drwxr-xr-x. 4 nobody nobody 4096 Jul 8 11:15 .
> drwxr-xr-x. 3 cloud cloud 35 Jul 8 14:06 ..
> drwxrwxr-x. 2 cloud cloud 4096 Jul 8 10:58 systemvm
> drwxr-xr-x. 3 nobody nobody 4096 Jul 8 11:15 template
> [root@usphlmvcsmt01 ~]#
>
>
> Any ideas?
>
>
> -----Original Message-----
> From: Corey, Mike
> Sent: Thursday, July 2, 2020 2:31 PM
> To: users@cloudstack.apache.org
> Subject: RE: Initial SystemVM Creation Failing
>
> No upgrade - this again is the fresh install of CloudStack 4.14 with
> VMware resources. Maybe I should note that I created the primary and
> secondary storage via the "Advanced" setup wizard from the UI (logged in as
> admin).
>
> I performed the ownership change you suggested so now cloud owns the
> systemvm folder on the secondary storage. However, now I'm getting a new
> error regarding copying the ISO.
>
> Should I just bag the 4.14 installation and try to get 4.13.1 running with
> VMware?
>
>
> 2020-07-02 14:19:13,314 INFO [c.c.h.v.m.VmwareManagerImpl]
> (DirectAgent-10:ctx-52902ebf usphlmvesxt155.phl.global.corp.sap,
> job-15/job-33, cmd: StartCommand) (logid:b8ba37d1) Copy System VM patch ISO
> file to secondary storage. source ISO:
> /usr/share/cloudstack-common/vms/systemvm.iso, destination:
> /var/cloudstack/mnt/VM/345050029058.777ecb60/systemvm/systemvm-4.14.0.0.iso
> 2020-07-02 14:19:13,320 ERROR [c.c.h.v.m.VmwareManagerImpl]
> (DirectAgent-10:ctx-52902ebf usphlmvesxt155.phl.global.corp.sap,
> job-15/job-33, cmd: StartCommand) (logid:b8ba37d1) Unexpected exception
> java.io.FileNotFoundException:
> /var/cloudstack/mnt/VM/345050029058.777ecb60/systemvm/systemvm-4.14.0.0.iso
> (Permission denied)
> at java.base/java.io.FileOutputStream.open0(Native Method)
> at java.base/java.io
> .FileOutputStream.open(FileOutputStream.java:298)
> at java.base/java.io
> .FileOutputStream.<init>(FileOutputStream.java:237)
> at java.base/java.io
> .FileOutputStream.<init>(FileOutputStream.java:187)
> at org.apache.commons.io.FileUtils.doCopyFile(FileUtils.java:1123)
> at org.apache.commons.io.FileUtils.copyFile(FileUtils.java:1076)
> at org.apache.commons.io.FileUtils.copyFile(FileUtils.java:1028)
> at com.cloud.utils.FileUtil.copyfile(FileUtil.java:30)
> at
> com.cloud.hypervisor.vmware.manager.VmwareManagerImpl.prepareSecondaryStorageStore(VmwareManagerImpl.java:661)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
> at com.sun.proxy.$Proxy389.prepareSecondaryStorageStore(Unknown
> Source)
> at
> com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:1962)
> at
> com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:530)
> at
> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at
> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2020-07-02 14:19:13,321 ERROR [c.c.h.v.m.VmwareManagerImpl]
> (DirectAgent-10:ctx-52902ebf usphlmvesxt155.phl.global.corp.sap,
> job-15/job-33, cmd: StartCommand) (logid:b8ba37d1) Unable to copy systemvm
> ISO on secondary storage. src location:
> /usr/share/cloudstack-common/vms/systemvm.iso, dest location:
> /var/cloudstack/mnt/VM/345050029058.777ecb60/systemvm/systemvm-4.14.0.0.iso
> 2020-07-02 14:19:13,322 INFO [c.c.h.v.u.VmwareHelper]
> (DirectAgent-10:ctx-52902ebf usphlmvesxt155.phl.global.corp.sap,
> job-15/job-33, cmd: StartCommand) (logid:b8ba37d1) [ignored]failed to get
> message for exception: Unable to copy systemvm ISO on secondary storage.
> src location: /usr/share/cloudstack-common/vms/systemvm.iso, dest location:
> /var/cloudstack/mnt/VM/345050029058.777ecb60/systemvm/systemvm-4.14.0.0.iso
> 2020-07-02 14:19:13,323 WARN [c.c.h.v.r.VmwareResource]
> (DirectAgent-10:ctx-52902ebf usphlmvesxt155.phl.global.corp.sap,
> job-15/job-33, cmd: StartCommand) (logid:b8ba37d1) StartCommand failed due
> to Exception: com.cloud.utils.exception.CloudRuntimeException
> Message: Unable to copy systemvm ISO on secondary storage. src location:
> /usr/share/cloudstack-common/vms/systemvm.iso, dest location:
> /var/cloudstack/mnt/VM/345050029058.777ecb60/systemvm/systemvm-4.14.0.0.iso
>
>
>
>
> -----Original Message-----
> From: Andrija Panic <an...@gmail.com>
> Sent: Thursday, July 2, 2020 12:34 PM
> To: users <us...@cloudstack.apache.org>
> Subject: Re: Initial SystemVM Creation Failing
>
> Hi Mike,
>
> Mount you secondary storage on the management server to some temp folder,
> then run: chown -R cloud:cloud <MOUNTPOINT>/systemvm/
> Other folders on Secondary Storage NFS are owned by root, but the
> "systemvm" should be owned by "cloud".
>
> I assume you got this after some ACS upgrade, or similar?
>
> On Thu, 2 Jul 2020 at 15:51, Corey, Mike <mi...@sap.com> wrote:
>
> > Hi,
> >
> >
> >
> > When I enable the zone, the systemvm creation begins but goes into a loop
> > of creating and deleting VMs in vCenter. I see in the logs there is an
> > error for creating the folder permissions on the secondary storage (see
> > below). Running this CHMOD manually from inside the CSM kicks the same
> > error.
> >
> >
> >
> > Please provide suggestions.
> >
> >
> >
> > 2020-07-02 09:08:39,665 DEBUG [c.c.h.v.m.VmwareManagerImpl]
> > (DirectAgent-66:ctx-e15c32b2 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: CopyCommand) (logid:1cfa64a9) Executing: sudo chmod
> > 1777 /var/cloudstack/mnt/VM/345050029058.2807345c
> >
> > 2020-07-02 09:08:39,666 DEBUG [c.c.h.v.m.VmwareManagerImpl]
> > (DirectAgent-66:ctx-e15c32b2 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: CopyCommand) (logid:1cfa64a9) Executing while with
> > timeout : 1440000
> >
> > 2020-07-02 09:08:39,693 DEBUG [c.c.h.v.m.VmwareManagerImpl]
> > (DirectAgent-66:ctx-e15c32b2 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: CopyCommand) (logid:1cfa64a9) Exit value is 1
> >
> > 2020-07-02 09:08:39,697 DEBUG [c.c.h.v.m.VmwareManagerImpl]
> > (DirectAgent-66:ctx-e15c32b2 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: CopyCommand) (logid:1cfa64a9) chmod: changing
> > permissions of ‘/var/cloudstack/mnt/VM/345050029058.2807345c’: Operation
> > not permitted
> >
> > 2020-07-02 09:08:39,699 WARN [c.c.h.v.m.VmwareManagerImpl]
> > (DirectAgent-66:ctx-e15c32b2 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: CopyCommand) (logid:1cfa64a9) Unable to set
> permissions
> > for /var/cloudstack/mnt/VM/345050029058.2807345c due to chmod: changing
> > permissions of ‘/var/cloudstack/mnt/VM/345050029058.2807345c’: Operation
> > not permitted
> >
> >
> >
> > Lines…
> >
> >
> >
> > 2020-07-02 09:09:00,598 DEBUG [c.c.h.v.m.HostMO]
> > (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) find VM v-2-VM on host
> >
> > 2020-07-02 09:09:00,598 INFO [c.c.h.v.m.HostMO]
> > (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) VM v-2-VM not found in
> > host cache
> >
> > 2020-07-02 09:09:00,598 DEBUG [c.c.h.v.m.HostMO]
> > (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) load VM cache on host
> >
> > 2020-07-02 09:09:00,616 DEBUG [c.c.h.v.m.HostMO]
> > (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) find VM v-2-VM on host
> >
> > 2020-07-02 09:09:00,617 DEBUG [c.c.h.v.m.HostMO]
> > (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) VM v-2-VM found in
> host
> > cache
> >
> > 2020-07-02 09:09:00,799 ERROR [c.c.h.v.m.VmwareManagerImpl]
> > (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) Unable to create
> > systemvm folder on secondary storage. location:
> > /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm
> >
> > 2020-07-02 09:09:00,800 INFO [c.c.h.v.u.VmwareHelper]
> > (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) [ignored]failed to get
> > message for exception: Unable to create systemvm folder on secondary
> > storage. location: /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm
> >
> > 2020-07-02 09:09:00,800 WARN [c.c.h.v.r.VmwareResource]
> > (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) StartCommand failed
> due
> > to Exception: com.cloud.utils.exception.CloudRuntimeException
> >
> > Message: Unable to create systemvm folder on secondary storage. location:
> > /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm
> >
> >
> >
> > com.cloud.utils.exception.CloudRuntimeException: Unable to create
> systemvm
> > folder on secondary storage. location:
> > /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm
> >
> > at
> >
> com.cloud.hypervisor.vmware.manager.VmwareManagerImpl.prepareSecondaryStorageStore(VmwareManagerImpl.java:648)
> >
> > at
> > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> > Method)
> >
> > at
> >
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> >
> > at
> >
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >
> > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> >
> > at
> >
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
> >
> > at
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
> >
> > at
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> >
> > at
> >
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95)
> >
> > at
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
> >
> > at
> >
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
> >
> > at
> > com.sun.proxy.$Proxy389.prepareSecondaryStorageStore(Unknown Source)
> >
> > at
> >
> com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:1962)
> >
> > at
> >
> com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:530)
> >
> > at
> >
> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
> >
> > at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> >
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> >
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> >
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> >
> > at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> >
> > at
> >
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> >
> > at
> > java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> >
> > at
> >
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
> >
> > at
> >
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> >
> > at
> >
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> >
> > at java.base/java.lang.Thread.run(Thread.java:834)
> >
> > 2020-07-02 09:09:00,804 DEBUG [c.c.a.m.DirectAgentAttache]
> > (DirectAgent-17:ctx-681dc524) (logid:1cfa64a9) Seq 1-4541880224203147245:
> > Cancelling because one of the answers is false and it is stop on error.
> >
> > 2020-07-02 09:09:00,804 DEBUG [c.c.a.m.DirectAgentAttache]
> > (DirectAgent-17:ctx-681dc524) (logid:1cfa64a9) Seq 1-4541880224203147245:
> > Response Received:
> >
> > 2020-07-02 09:09:00,805 DEBUG [c.c.a.t.Request]
> > (DirectAgent-17:ctx-681dc524) (logid:1cfa64a9) Seq 1-4541880224203147245:
> > Processing: { Ans: , MgmtId: 345050029058, via:
> > 1(usphlmvesxt154.phl.global.corp.sap), Ver: v1, Flags: 110,
> >
> [{"com.cloud.agent.api.StartAnswer":{"vm":{"id":2,"name":"v-2-VM","bootloader":"HVM","state":"Starting","type":"ConsoleProxy","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":1073741824,"maxRam":1073741824,"hostName":"v-2-VM","arch":"x86_64","os":"Other
> > Linux (64-bit)","platformEmulator":"otherLinux64Guest","bootArgs":"
> > template=domP type=consoleproxy host=10.4.32.163 port=8250 name=v-2-VM
> > zone=1 pod=1 guid=Proxy.2 proxy_vm=2 disable_rp_filter=true
> > eth2ip=10.4.42.25 eth2mask=255.255.255.192 gateway=10.4.42.1
> eth0ip=0.0.0.0
> > eth0mask=0.0.0.0 eth1ip=10.4.4.35 eth1mask=255.255.252.0 mgmtcidr=
> > 10.4.32.0/24 localgw=10.4.4.1 internaldns1=10.4.202.200
> dns1=10.4.202.200
> >
> dns2=10.4.12.200","enableHA":false,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"sYnwDJYgzrMKdm3ftAO-kg","params":{"vmware.reserve.mem":"false","vmware.reserve.cpu":"false","nicAdapter":"E1000"},"uuid":"fb65df73-693d-48fb-a203-c2b5e776842a","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"9a1f2ae8-6b10-4525-8a54-a175b153b789","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"50cc970f-fbe2-3b45-9cb5-b8bcdb6dbff5","id":1,"poolType":"NetworkFilesystem","host":"usphlcv300.phl.global.corp.sap","path":"/usphlmvesx_lab_cspri","port":2049,"url":"NetworkFilesystem://usphlcv300.phl.global.corp.sap/usphlmvesx_lab_cspri/?ROLE=Primary&STOREUUID=50cc970f-fbe2-3b45-9cb5-b8bcdb6dbff5","isManaged":false}},"name":"ROOT-2","size":0,"path":"ROOT-2_2","volumeId":2,"vmName":"v-2-VM","accountId":1,"format":"OVA","provisioningType":"THIN","id":2,"deviceId":0,"bytesReadRate":0,"bytesWriteRate":0,"iopsReadRate":0,"iopsWriteRate":0,"hypervisorType":"VMware","directDownload":false}},"diskSeq":0,"path":"ROOT-2_2","type":"ROOT","_details":{"storageHost":"usphlcv300.phl.global.corp.sap","managed":"false","storagePort":"2049","volumeSize":"0"}}],"nics":[{"deviceId":2,"networkRateMbps":-1,"defaultNic":true,"pxeDisable":true,"nicUuid":"ffbc1822-7907-4cdf-9e80-bb316786e04d","details":{"PromiscuousMode":"false","MacAddressChanges":"true","ForgedTransmits":"true"},"dpdkEnabled":false,"uuid":"f3a30021-a469-413d-85a8-ca1ca68ba27c","ip":"10.4.42.25","netmask":"255.255.255.192","gateway":"10.4.42.1","mac":"1e:00:e3:00:00:19","dns1":"10.4.202.200","dns2":"10.4.12.200","broadcastType":"Vlan","type":"Public","broadcastUri":"vlan://untagged","isolationUri":"vlan://untagged","isSecurityGroupEnabled":false,"name":"vSwitch0,134,vmwaresvs"},{"deviceId":0,"networkRateMbps":-1,"defaultNic":false,"pxeDisable":true,"nicUuid":"d6521be0-85b8-4836-bf8b-8461afdf593a","details":{"PromiscuousMode":"false","MacAddressChanges":"true","ForgedTransmits":"true"},"dpdkEnabled":false,"uuid":"eda0ae3d-d15d-4f91-b8db-a719cf5a94bf","mac":"02:00:31:71:00:02","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false,"name":"vSwitch0,146,vmwaresvs"},{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"pxeDisable":true,"nicUuid":"4e6201ad-91c0-4cab-bf53-9e2440025245","details":{"PromiscuousMode":"false","MacAddressChanges":"true","ForgedTransmits":"true"},"dpdkEnabled":false,"uuid":"ce88b030-c04c-4437-b60d-9910c2cc376f","ip":"10.4.4.35","netmask":"255.255.252.0","gateway":"10.4.4.1","mac":"1e:00:6b:00:00:27","broadcastType":"Native","type":"Management","isSecurityGroupEnabled":false,"name":"vSwitch0,146,vmwaresvs"}],"guestOsDetails":{},"extraConfig":{}},"result":false,"details":"StartCommand
> > failed due to Exception:
> > com.cloud.utils.exception.CloudRuntimeException\nMessage: Unable to
> create
> > systemvm folder on secondary storage. location:
> > /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm\n","wait":0}}] }
> >
> > 2020-07-02 09:09:00,805 DEBUG [c.c.a.m.AgentAttache]
> > (DirectAgent-17:ctx-681dc524) (logid:1cfa64a9) Seq 1-4541880224203147245:
> > No more commands found
> >
> > 2020-07-02 09:09:00,806 DEBUG [c.c.a.t.Request]
> > (Work-Job-Executor-2:ctx-9aa38618 job-16/job-35 ctx-7ebd1b78)
> > (logid:1cfa64a9) Seq 1-4541880224203147245: Received: { Ans: , MgmtId:
> > 345050029058, via: 1(usphlmvesxt154.phl.global.corp.sap), Ver: v1, Flags:
> > 110, { StartAnswer } }
> >
> > 2020-07-02 09:09:00,816 INFO [c.c.v.VirtualMachineManagerImpl]
> > (Work-Job-Executor-2:ctx-9aa38618 job-16/job-35 ctx-7ebd1b78)
> > (logid:1cfa64a9) Unable to start VM on Host[-1-Routing] due to
> StartCommand
> > failed due to Exception: com.cloud.utils.exception.CloudRuntimeException
> >
> > Message: Unable to create systemvm folder on secondary storage. location:
> > /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm
> >
> >
> >
> > *Mike Corey*
> >
> >
> > Technology Senior Consultant, IT CS CTW Operation & Virtualization
> Service
> > US
> >
> >
> > *SAP AMERICA, INC.* 3999 West Chester Pike, Newtown Square, 19073 United
> > States
> >
> >
> > T +1 610 661 0905, M +1 484 274 2658, E mike.corey@sap.com
> >
> >
> >
> >
> >
> >
> >
>
>
> --
>
> Andrija Panić
>
RE: Initial SystemVM Creation Failing
Posted by "Corey, Mike" <mi...@sap.com>.
Gone over the installation now twice (reverted to snapshot of CentOS VM) and still getting weird storage/permission type issues.
The permissions and ownership of the systemvm folder looks correct.
[root@ ~]# ls -al /mnt/secondary/
total 16
drwxr-xr-x. 4 root root 4096 Jul 8 11:15 .
drwxr-xr-x. 4 root root 38 Jul 8 11:15 ..
drwxrwxrwx. 12 root root 4096 Jul 8 11:05 .snapshot
drwxrwxr-x. 2 cloud cloud 4096 Jul 8 10:58 systemvm
drwxr-xr-x. 3 root root 4096 Jul 8 11:15 template
However, I'm still seeing the copy iso error and the VMs keep creating and deleting with the log suggesting the vmx file of the VM doesn't exist on the datastore (can create files/folder on datastore identified in errror from vCenter).
2020-07-08 14:22:36,486 INFO [c.c.h.v.m.DatastoreMO] (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap, job-6/job-37, cmd: StartCommand) (logid:b828ac16) Folder v-6-VM does not exist on datastore
2020-07-08 14:22:36,494 INFO [c.c.h.v.m.DatastoreMO] (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap, job-6/job-37, cmd: StartCommand) (logid:b828ac16) Search file v-6-VM.vmx on [50cc970ffbe23b459cb5b8bcdb6dbff5]
2020-07-08 14:22:36,522 INFO [c.c.h.v.m.DatastoreMO] (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap, job-6/job-37, cmd: StartCommand) (logid:b828ac16) File [50cc970ffbe23b459cb5b8bcdb6dbff5] v-6-VM.vmx does not exist on datastore
2020-07-08 14:22:36,522 INFO [c.c.h.v.m.DatastoreMO] (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap, job-6/job-37, cmd: StartCommand) (logid:b828ac16) Searching file v-6-VM.vmx in [50cc970ffbe23b459cb5b8bcdb6dbff5]
2020-07-08 14:22:36,641 INFO [c.c.h.v.m.HypervisorHostHelper] (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap, job-6/job-37, cmd: StartCommand) (logid:b828ac16) Create blank VM. cpuCount: 1, cpuSpeed(MHz): 500, mem(Mb): 1024
2020-07-08 14:22:36,692 DEBUG [c.c.h.v.m.ClusterMO] (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap, job-6/job-37, cmd: StartCommand) (logid:b828ac16) Retrieved recommended disk controller for guest OS : otherLinux64Guest in cluster PHL-CSM01 : VirtualLsiLogicController
2020-07-08 14:22:37,115 DEBUG [c.c.h.v.m.HostMO] (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap, job-6/job-37, cmd: StartCommand) (logid:b828ac16) find VM v-6-VM on host
2020-07-08 14:22:37,115 INFO [c.c.h.v.m.HostMO] (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap, job-6/job-37, cmd: StartCommand) (logid:b828ac16) VM v-6-VM not found in host cache
2020-07-08 14:22:37,115 DEBUG [c.c.h.v.m.HostMO] (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap, job-6/job-37, cmd: StartCommand) (logid:b828ac16) load VM cache on host
2020-07-08 14:22:37,155 DEBUG [c.c.h.v.m.HostMO] (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap, job-6/job-37, cmd: StartCommand) (logid:b828ac16) find VM v-6-VM on host
2020-07-08 14:22:37,155 DEBUG [c.c.h.v.m.HostMO] (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap, job-6/job-37, cmd: StartCommand) (logid:b828ac16) VM v-6-VM found in host cache
2020-07-08 14:22:37,375 INFO [c.c.h.v.m.VmwareManagerImpl] (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap, job-6/job-37, cmd: StartCommand) (logid:b828ac16) Inject SSH key pairs before copying systemvm.iso into secondary storage
2020-07-08 14:22:37,377 INFO [c.c.s.ConfigurationServerImpl] (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap, job-6/job-37, cmd: StartCommand) (logid:b828ac16) Processing updateKeyPairs
2020-07-08 14:22:37,377 INFO [c.c.s.ConfigurationServerImpl] (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap, job-6/job-37, cmd: StartCommand) (logid:b828ac16) Keypairs already in database, updating local copy
2020-07-08 14:22:37,381 INFO [c.c.s.ConfigurationServerImpl] (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap, job-6/job-37, cmd: StartCommand) (logid:b828ac16) Going to update systemvm iso with generated keypairs if needed
2020-07-08 14:22:37,381 INFO [c.c.s.ConfigurationServerImpl] (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap, job-6/job-37, cmd: StartCommand) (logid:b828ac16) Trying to inject public and private keys into systemvm iso
2020-07-08 14:22:37,381 DEBUG [c.c.u.s.Script] (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap, job-6/job-37, cmd: StartCommand) (logid:b828ac16) Looking for scripts/vm/systemvm/injectkeys.sh in the classpath
2020-07-08 14:22:37,381 DEBUG [c.c.u.s.Script] (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap, job-6/job-37, cmd: StartCommand) (logid:b828ac16) System resource: file:/usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh
2020-07-08 14:22:37,381 DEBUG [c.c.u.s.Script] (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap, job-6/job-37, cmd: StartCommand) (logid:b828ac16) Absolute path = /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh
2020-07-08 14:22:37,381 DEBUG [c.c.u.s.Script] (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap, job-6/job-37, cmd: StartCommand) (logid:b828ac16) Looking for vms/systemvm.iso in the classpath
2020-07-08 14:22:37,381 DEBUG [c.c.u.s.Script] (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap, job-6/job-37, cmd: StartCommand) (logid:b828ac16) System resource: file:/usr/share/cloudstack-common/vms/systemvm.iso
2020-07-08 14:22:37,381 DEBUG [c.c.u.s.Script] (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap, job-6/job-37, cmd: StartCommand) (logid:b828ac16) Absolute path = /usr/share/cloudstack-common/vms/systemvm.iso
2020-07-08 14:22:37,381 DEBUG [c.c.s.ConfigurationServerImpl] (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap, job-6/job-37, cmd: StartCommand) (logid:b828ac16) Executing: /bin/bash /usr/share/cloudstack-common/scripts/vm/systemvm/injectkeys.sh /var/cloudstack/management/.ssh/id_rsa.pub /var/cloudstack/management/.ssh/id_rsa /usr/share/cloudstack-common/vms/systemvm.iso
2020-07-08 14:22:37,384 DEBUG [c.c.s.ConfigurationServerImpl] (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap, job-6/job-37, cmd: StartCommand) (logid:b828ac16) Executing while with timeout : 3600000
2020-07-08 14:22:37,487 DEBUG [c.c.s.ConfigurationServerImpl] (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap, job-6/job-37, cmd: StartCommand) (logid:b828ac16) Execution is successful.
2020-07-08 14:22:37,494 DEBUG [c.c.s.ConfigurationServerImpl] (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap, job-6/job-37, cmd: StartCommand) (logid:b828ac16) mount: /dev/loop0 is write-protected, mounting read-only
New public key is the same as the one in the systemvm.iso, not injecting it, not modifying systemvm.iso
2020-07-08 14:22:37,494 INFO [c.c.s.ConfigurationServerImpl] (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap, job-6/job-37, cmd: StartCommand) (logid:b828ac16) The script injectkeys.sh was run with result : null
2020-07-08 14:22:37,494 INFO [c.c.h.v.m.VmwareManagerImpl] (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap, job-6/job-37, cmd: StartCommand) (logid:b828ac16) Copy System VM patch ISO file to secondary storage. source ISO: /usr/share/cloudstack-common/vms/systemvm.iso, destination: /var/cloudstack/mnt/VM/345050029058.1f7af2bf/systemvm/systemvm-4.14.0.0.iso
2020-07-08 14:22:37,495 ERROR [c.c.h.v.m.VmwareManagerImpl] (DirectAgent-51:ctx-7e6d5dc9 usphlmvesxt155.phl.global.corp.sap, job-6/job-37, cmd: StartCommand) (logid:b828ac16) Unexpected exception
java.io.FileNotFoundException: /var/cloudstack/mnt/VM/345050029058.1f7af2bf/systemvm/systemvm-4.14.0.0.iso (Permission denied)
But, the ownership/permissions look okay on that destination directory too.
[root@ ~]# ls -al /var/cloudstack/mnt/VM/345050029058.1f7af2bf
total 12
drwxr-xr-x. 4 nobody nobody 4096 Jul 8 11:15 .
drwxr-xr-x. 3 cloud cloud 35 Jul 8 14:06 ..
drwxrwxr-x. 2 cloud cloud 4096 Jul 8 10:58 systemvm
drwxr-xr-x. 3 nobody nobody 4096 Jul 8 11:15 template
[root@usphlmvcsmt01 ~]#
Any ideas?
-----Original Message-----
From: Corey, Mike
Sent: Thursday, July 2, 2020 2:31 PM
To: users@cloudstack.apache.org
Subject: RE: Initial SystemVM Creation Failing
No upgrade - this again is the fresh install of CloudStack 4.14 with VMware resources. Maybe I should note that I created the primary and secondary storage via the "Advanced" setup wizard from the UI (logged in as admin).
I performed the ownership change you suggested so now cloud owns the systemvm folder on the secondary storage. However, now I'm getting a new error regarding copying the ISO.
Should I just bag the 4.14 installation and try to get 4.13.1 running with VMware?
2020-07-02 14:19:13,314 INFO [c.c.h.v.m.VmwareManagerImpl] (DirectAgent-10:ctx-52902ebf usphlmvesxt155.phl.global.corp.sap, job-15/job-33, cmd: StartCommand) (logid:b8ba37d1) Copy System VM patch ISO file to secondary storage. source ISO: /usr/share/cloudstack-common/vms/systemvm.iso, destination: /var/cloudstack/mnt/VM/345050029058.777ecb60/systemvm/systemvm-4.14.0.0.iso
2020-07-02 14:19:13,320 ERROR [c.c.h.v.m.VmwareManagerImpl] (DirectAgent-10:ctx-52902ebf usphlmvesxt155.phl.global.corp.sap, job-15/job-33, cmd: StartCommand) (logid:b8ba37d1) Unexpected exception
java.io.FileNotFoundException: /var/cloudstack/mnt/VM/345050029058.777ecb60/systemvm/systemvm-4.14.0.0.iso (Permission denied)
at java.base/java.io.FileOutputStream.open0(Native Method)
at java.base/java.io.FileOutputStream.open(FileOutputStream.java:298)
at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:237)
at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:187)
at org.apache.commons.io.FileUtils.doCopyFile(FileUtils.java:1123)
at org.apache.commons.io.FileUtils.copyFile(FileUtils.java:1076)
at org.apache.commons.io.FileUtils.copyFile(FileUtils.java:1028)
at com.cloud.utils.FileUtil.copyfile(FileUtil.java:30)
at com.cloud.hypervisor.vmware.manager.VmwareManagerImpl.prepareSecondaryStorageStore(VmwareManagerImpl.java:661)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy389.prepareSecondaryStorageStore(Unknown Source)
at com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:1962)
at com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:530)
at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
2020-07-02 14:19:13,321 ERROR [c.c.h.v.m.VmwareManagerImpl] (DirectAgent-10:ctx-52902ebf usphlmvesxt155.phl.global.corp.sap, job-15/job-33, cmd: StartCommand) (logid:b8ba37d1) Unable to copy systemvm ISO on secondary storage. src location: /usr/share/cloudstack-common/vms/systemvm.iso, dest location: /var/cloudstack/mnt/VM/345050029058.777ecb60/systemvm/systemvm-4.14.0.0.iso
2020-07-02 14:19:13,322 INFO [c.c.h.v.u.VmwareHelper] (DirectAgent-10:ctx-52902ebf usphlmvesxt155.phl.global.corp.sap, job-15/job-33, cmd: StartCommand) (logid:b8ba37d1) [ignored]failed to get message for exception: Unable to copy systemvm ISO on secondary storage. src location: /usr/share/cloudstack-common/vms/systemvm.iso, dest location: /var/cloudstack/mnt/VM/345050029058.777ecb60/systemvm/systemvm-4.14.0.0.iso
2020-07-02 14:19:13,323 WARN [c.c.h.v.r.VmwareResource] (DirectAgent-10:ctx-52902ebf usphlmvesxt155.phl.global.corp.sap, job-15/job-33, cmd: StartCommand) (logid:b8ba37d1) StartCommand failed due to Exception: com.cloud.utils.exception.CloudRuntimeException
Message: Unable to copy systemvm ISO on secondary storage. src location: /usr/share/cloudstack-common/vms/systemvm.iso, dest location: /var/cloudstack/mnt/VM/345050029058.777ecb60/systemvm/systemvm-4.14.0.0.iso
-----Original Message-----
From: Andrija Panic <an...@gmail.com>
Sent: Thursday, July 2, 2020 12:34 PM
To: users <us...@cloudstack.apache.org>
Subject: Re: Initial SystemVM Creation Failing
Hi Mike,
Mount you secondary storage on the management server to some temp folder,
then run: chown -R cloud:cloud <MOUNTPOINT>/systemvm/
Other folders on Secondary Storage NFS are owned by root, but the
"systemvm" should be owned by "cloud".
I assume you got this after some ACS upgrade, or similar?
On Thu, 2 Jul 2020 at 15:51, Corey, Mike <mi...@sap.com> wrote:
> Hi,
>
>
>
> When I enable the zone, the systemvm creation begins but goes into a loop
> of creating and deleting VMs in vCenter. I see in the logs there is an
> error for creating the folder permissions on the secondary storage (see
> below). Running this CHMOD manually from inside the CSM kicks the same
> error.
>
>
>
> Please provide suggestions.
>
>
>
> 2020-07-02 09:08:39,665 DEBUG [c.c.h.v.m.VmwareManagerImpl]
> (DirectAgent-66:ctx-e15c32b2 usphlmvesxt154.phl.global.corp.sap,
> job-16/job-35, cmd: CopyCommand) (logid:1cfa64a9) Executing: sudo chmod
> 1777 /var/cloudstack/mnt/VM/345050029058.2807345c
>
> 2020-07-02 09:08:39,666 DEBUG [c.c.h.v.m.VmwareManagerImpl]
> (DirectAgent-66:ctx-e15c32b2 usphlmvesxt154.phl.global.corp.sap,
> job-16/job-35, cmd: CopyCommand) (logid:1cfa64a9) Executing while with
> timeout : 1440000
>
> 2020-07-02 09:08:39,693 DEBUG [c.c.h.v.m.VmwareManagerImpl]
> (DirectAgent-66:ctx-e15c32b2 usphlmvesxt154.phl.global.corp.sap,
> job-16/job-35, cmd: CopyCommand) (logid:1cfa64a9) Exit value is 1
>
> 2020-07-02 09:08:39,697 DEBUG [c.c.h.v.m.VmwareManagerImpl]
> (DirectAgent-66:ctx-e15c32b2 usphlmvesxt154.phl.global.corp.sap,
> job-16/job-35, cmd: CopyCommand) (logid:1cfa64a9) chmod: changing
> permissions of ‘/var/cloudstack/mnt/VM/345050029058.2807345c’: Operation
> not permitted
>
> 2020-07-02 09:08:39,699 WARN [c.c.h.v.m.VmwareManagerImpl]
> (DirectAgent-66:ctx-e15c32b2 usphlmvesxt154.phl.global.corp.sap,
> job-16/job-35, cmd: CopyCommand) (logid:1cfa64a9) Unable to set permissions
> for /var/cloudstack/mnt/VM/345050029058.2807345c due to chmod: changing
> permissions of ‘/var/cloudstack/mnt/VM/345050029058.2807345c’: Operation
> not permitted
>
>
>
> Lines…
>
>
>
> 2020-07-02 09:09:00,598 DEBUG [c.c.h.v.m.HostMO]
> (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) find VM v-2-VM on host
>
> 2020-07-02 09:09:00,598 INFO [c.c.h.v.m.HostMO]
> (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) VM v-2-VM not found in
> host cache
>
> 2020-07-02 09:09:00,598 DEBUG [c.c.h.v.m.HostMO]
> (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) load VM cache on host
>
> 2020-07-02 09:09:00,616 DEBUG [c.c.h.v.m.HostMO]
> (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) find VM v-2-VM on host
>
> 2020-07-02 09:09:00,617 DEBUG [c.c.h.v.m.HostMO]
> (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) VM v-2-VM found in host
> cache
>
> 2020-07-02 09:09:00,799 ERROR [c.c.h.v.m.VmwareManagerImpl]
> (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) Unable to create
> systemvm folder on secondary storage. location:
> /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm
>
> 2020-07-02 09:09:00,800 INFO [c.c.h.v.u.VmwareHelper]
> (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) [ignored]failed to get
> message for exception: Unable to create systemvm folder on secondary
> storage. location: /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm
>
> 2020-07-02 09:09:00,800 WARN [c.c.h.v.r.VmwareResource]
> (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) StartCommand failed due
> to Exception: com.cloud.utils.exception.CloudRuntimeException
>
> Message: Unable to create systemvm folder on secondary storage. location:
> /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm
>
>
>
> com.cloud.utils.exception.CloudRuntimeException: Unable to create systemvm
> folder on secondary storage. location:
> /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm
>
> at
> com.cloud.hypervisor.vmware.manager.VmwareManagerImpl.prepareSecondaryStorageStore(VmwareManagerImpl.java:648)
>
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
>
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
>
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
>
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
>
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95)
>
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
>
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
>
> at
> com.sun.proxy.$Proxy389.prepareSecondaryStorageStore(Unknown Source)
>
> at
> com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:1962)
>
> at
> com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:530)
>
> at
> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
>
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>
> at
> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
>
> at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>
> at java.base/java.lang.Thread.run(Thread.java:834)
>
> 2020-07-02 09:09:00,804 DEBUG [c.c.a.m.DirectAgentAttache]
> (DirectAgent-17:ctx-681dc524) (logid:1cfa64a9) Seq 1-4541880224203147245:
> Cancelling because one of the answers is false and it is stop on error.
>
> 2020-07-02 09:09:00,804 DEBUG [c.c.a.m.DirectAgentAttache]
> (DirectAgent-17:ctx-681dc524) (logid:1cfa64a9) Seq 1-4541880224203147245:
> Response Received:
>
> 2020-07-02 09:09:00,805 DEBUG [c.c.a.t.Request]
> (DirectAgent-17:ctx-681dc524) (logid:1cfa64a9) Seq 1-4541880224203147245:
> Processing: { Ans: , MgmtId: 345050029058, via:
> 1(usphlmvesxt154.phl.global.corp.sap), Ver: v1, Flags: 110,
> [{"com.cloud.agent.api.StartAnswer":{"vm":{"id":2,"name":"v-2-VM","bootloader":"HVM","state":"Starting","type":"ConsoleProxy","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":1073741824,"maxRam":1073741824,"hostName":"v-2-VM","arch":"x86_64","os":"Other
> Linux (64-bit)","platformEmulator":"otherLinux64Guest","bootArgs":"
> template=domP type=consoleproxy host=10.4.32.163 port=8250 name=v-2-VM
> zone=1 pod=1 guid=Proxy.2 proxy_vm=2 disable_rp_filter=true
> eth2ip=10.4.42.25 eth2mask=255.255.255.192 gateway=10.4.42.1 eth0ip=0.0.0.0
> eth0mask=0.0.0.0 eth1ip=10.4.4.35 eth1mask=255.255.252.0 mgmtcidr=
> 10.4.32.0/24 localgw=10.4.4.1 internaldns1=10.4.202.200 dns1=10.4.202.200
> dns2=10.4.12.200","enableHA":false,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"sYnwDJYgzrMKdm3ftAO-kg","params":{"vmware.reserve.mem":"false","vmware.reserve.cpu":"false","nicAdapter":"E1000"},"uuid":"fb65df73-693d-48fb-a203-c2b5e776842a","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"9a1f2ae8-6b10-4525-8a54-a175b153b789","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"50cc970f-fbe2-3b45-9cb5-b8bcdb6dbff5","id":1,"poolType":"NetworkFilesystem","host":"usphlcv300.phl.global.corp.sap","path":"/usphlmvesx_lab_cspri","port":2049,"url":"NetworkFilesystem://usphlcv300.phl.global.corp.sap/usphlmvesx_lab_cspri/?ROLE=Primary&STOREUUID=50cc970f-fbe2-3b45-9cb5-b8bcdb6dbff5","isManaged":false}},"name":"ROOT-2","size":0,"path":"ROOT-2_2","volumeId":2,"vmName":"v-2-VM","accountId":1,"format":"OVA","provisioningType":"THIN","id":2,"deviceId":0,"bytesReadRate":0,"bytesWriteRate":0,"iopsReadRate":0,"iopsWriteRate":0,"hypervisorType":"VMware","directDownload":false}},"diskSeq":0,"path":"ROOT-2_2","type":"ROOT","_details":{"storageHost":"usphlcv300.phl.global.corp.sap","managed":"false","storagePort":"2049","volumeSize":"0"}}],"nics":[{"deviceId":2,"networkRateMbps":-1,"defaultNic":true,"pxeDisable":true,"nicUuid":"ffbc1822-7907-4cdf-9e80-bb316786e04d","details":{"PromiscuousMode":"false","MacAddressChanges":"true","ForgedTransmits":"true"},"dpdkEnabled":false,"uuid":"f3a30021-a469-413d-85a8-ca1ca68ba27c","ip":"10.4.42.25","netmask":"255.255.255.192","gateway":"10.4.42.1","mac":"1e:00:e3:00:00:19","dns1":"10.4.202.200","dns2":"10.4.12.200","broadcastType":"Vlan","type":"Public","broadcastUri":"vlan://untagged","isolationUri":"vlan://untagged","isSecurityGroupEnabled":false,"name":"vSwitch0,134,vmwaresvs"},{"deviceId":0,"networkRateMbps":-1,"defaultNic":false,"pxeDisable":true,"nicUuid":"d6521be0-85b8-4836-bf8b-8461afdf593a","details":{"PromiscuousMode":"false","MacAddressChanges":"true","ForgedTransmits":"true"},"dpdkEnabled":false,"uuid":"eda0ae3d-d15d-4f91-b8db-a719cf5a94bf","mac":"02:00:31:71:00:02","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false,"name":"vSwitch0,146,vmwaresvs"},{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"pxeDisable":true,"nicUuid":"4e6201ad-91c0-4cab-bf53-9e2440025245","details":{"PromiscuousMode":"false","MacAddressChanges":"true","ForgedTransmits":"true"},"dpdkEnabled":false,"uuid":"ce88b030-c04c-4437-b60d-9910c2cc376f","ip":"10.4.4.35","netmask":"255.255.252.0","gateway":"10.4.4.1","mac":"1e:00:6b:00:00:27","broadcastType":"Native","type":"Management","isSecurityGroupEnabled":false,"name":"vSwitch0,146,vmwaresvs"}],"guestOsDetails":{},"extraConfig":{}},"result":false,"details":"StartCommand
> failed due to Exception:
> com.cloud.utils.exception.CloudRuntimeException\nMessage: Unable to create
> systemvm folder on secondary storage. location:
> /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm\n","wait":0}}] }
>
> 2020-07-02 09:09:00,805 DEBUG [c.c.a.m.AgentAttache]
> (DirectAgent-17:ctx-681dc524) (logid:1cfa64a9) Seq 1-4541880224203147245:
> No more commands found
>
> 2020-07-02 09:09:00,806 DEBUG [c.c.a.t.Request]
> (Work-Job-Executor-2:ctx-9aa38618 job-16/job-35 ctx-7ebd1b78)
> (logid:1cfa64a9) Seq 1-4541880224203147245: Received: { Ans: , MgmtId:
> 345050029058, via: 1(usphlmvesxt154.phl.global.corp.sap), Ver: v1, Flags:
> 110, { StartAnswer } }
>
> 2020-07-02 09:09:00,816 INFO [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-2:ctx-9aa38618 job-16/job-35 ctx-7ebd1b78)
> (logid:1cfa64a9) Unable to start VM on Host[-1-Routing] due to StartCommand
> failed due to Exception: com.cloud.utils.exception.CloudRuntimeException
>
> Message: Unable to create systemvm folder on secondary storage. location:
> /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm
>
>
>
> *Mike Corey*
>
>
> Technology Senior Consultant, IT CS CTW Operation & Virtualization Service
> US
>
>
> *SAP AMERICA, INC.* 3999 West Chester Pike, Newtown Square, 19073 United
> States
>
>
> T +1 610 661 0905, M +1 484 274 2658, E mike.corey@sap.com
>
>
>
>
>
>
>
--
Andrija Panić
Re: Initial SystemVM Creation Failing
Posted by Andrija Panic <an...@gmail.com>.
Mike,
are you using CentOS7 packages or Ubuntu?
I would delete the /systemvm/systemvm-4.14.0.0.iso file (does it actually
exist?) and restart mgmt server again (permissions on the "systemvm" folder
should be 644)
You can also verify being able to write as a "cloud" user to the mount
point/systemvm/ folders with: runuser -l cloud -c 'touch
<mountpoint/systemvm/test.file>' - if this fails then you do have some
permissions issue, but this is most certainly caused by the NFS server /
exports etc.
What you are getting is weird - there should be no issues there - and
everything should work smoothly.
Andrija
On Thu, 2 Jul 2020 at 20:31, Corey, Mike <mi...@sap.com> wrote:
> No upgrade - this again is the fresh install of CloudStack 4.14 with
> VMware resources. Maybe I should note that I created the primary and
> secondary storage via the "Advanced" setup wizard from the UI (logged in as
> admin).
>
> I performed the ownership change you suggested so now cloud owns the
> systemvm folder on the secondary storage. However, now I'm getting a new
> error regarding copying the ISO.
>
> Should I just bag the 4.14 installation and try to get 4.13.1 running with
> VMware?
>
>
> 2020-07-02 14:19:13,314 INFO [c.c.h.v.m.VmwareManagerImpl]
> (DirectAgent-10:ctx-52902ebf usphlmvesxt155.phl.global.corp.sap,
> job-15/job-33, cmd: StartCommand) (logid:b8ba37d1) Copy System VM patch ISO
> file to secondary storage. source ISO:
> /usr/share/cloudstack-common/vms/systemvm.iso, destination:
> /var/cloudstack/mnt/VM/345050029058.777ecb60/systemvm/systemvm-4.14.0.0.iso
> 2020-07-02 14:19:13,320 ERROR [c.c.h.v.m.VmwareManagerImpl]
> (DirectAgent-10:ctx-52902ebf usphlmvesxt155.phl.global.corp.sap,
> job-15/job-33, cmd: StartCommand) (logid:b8ba37d1) Unexpected exception
> java.io.FileNotFoundException:
> /var/cloudstack/mnt/VM/345050029058.777ecb60/systemvm/systemvm-4.14.0.0.iso
> (Permission denied)
> at java.base/java.io.FileOutputStream.open0(Native Method)
> at java.base/java.io
> .FileOutputStream.open(FileOutputStream.java:298)
> at java.base/java.io
> .FileOutputStream.<init>(FileOutputStream.java:237)
> at java.base/java.io
> .FileOutputStream.<init>(FileOutputStream.java:187)
> at org.apache.commons.io.FileUtils.doCopyFile(FileUtils.java:1123)
> at org.apache.commons.io.FileUtils.copyFile(FileUtils.java:1076)
> at org.apache.commons.io.FileUtils.copyFile(FileUtils.java:1028)
> at com.cloud.utils.FileUtil.copyfile(FileUtil.java:30)
> at
> com.cloud.hypervisor.vmware.manager.VmwareManagerImpl.prepareSecondaryStorageStore(VmwareManagerImpl.java:661)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
> at com.sun.proxy.$Proxy389.prepareSecondaryStorageStore(Unknown
> Source)
> at
> com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:1962)
> at
> com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:530)
> at
> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at
> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2020-07-02 14:19:13,321 ERROR [c.c.h.v.m.VmwareManagerImpl]
> (DirectAgent-10:ctx-52902ebf usphlmvesxt155.phl.global.corp.sap,
> job-15/job-33, cmd: StartCommand) (logid:b8ba37d1) Unable to copy systemvm
> ISO on secondary storage. src location:
> /usr/share/cloudstack-common/vms/systemvm.iso, dest location:
> /var/cloudstack/mnt/VM/345050029058.777ecb60/systemvm/systemvm-4.14.0.0.iso
> 2020-07-02 14:19:13,322 INFO [c.c.h.v.u.VmwareHelper]
> (DirectAgent-10:ctx-52902ebf usphlmvesxt155.phl.global.corp.sap,
> job-15/job-33, cmd: StartCommand) (logid:b8ba37d1) [ignored]failed to get
> message for exception: Unable to copy systemvm ISO on secondary storage.
> src location: /usr/share/cloudstack-common/vms/systemvm.iso, dest location:
> /var/cloudstack/mnt/VM/345050029058.777ecb60/systemvm/systemvm-4.14.0.0.iso
> 2020-07-02 14:19:13,323 WARN [c.c.h.v.r.VmwareResource]
> (DirectAgent-10:ctx-52902ebf usphlmvesxt155.phl.global.corp.sap,
> job-15/job-33, cmd: StartCommand) (logid:b8ba37d1) StartCommand failed due
> to Exception: com.cloud.utils.exception.CloudRuntimeException
> Message: Unable to copy systemvm ISO on secondary storage. src location:
> /usr/share/cloudstack-common/vms/systemvm.iso, dest location:
> /var/cloudstack/mnt/VM/345050029058.777ecb60/systemvm/systemvm-4.14.0.0.iso
>
>
>
>
> -----Original Message-----
> From: Andrija Panic <an...@gmail.com>
> Sent: Thursday, July 2, 2020 12:34 PM
> To: users <us...@cloudstack.apache.org>
> Subject: Re: Initial SystemVM Creation Failing
>
> Hi Mike,
>
> Mount you secondary storage on the management server to some temp folder,
> then run: chown -R cloud:cloud <MOUNTPOINT>/systemvm/
> Other folders on Secondary Storage NFS are owned by root, but the
> "systemvm" should be owned by "cloud".
>
> I assume you got this after some ACS upgrade, or similar?
>
> On Thu, 2 Jul 2020 at 15:51, Corey, Mike <mi...@sap.com> wrote:
>
> > Hi,
> >
> >
> >
> > When I enable the zone, the systemvm creation begins but goes into a loop
> > of creating and deleting VMs in vCenter. I see in the logs there is an
> > error for creating the folder permissions on the secondary storage (see
> > below). Running this CHMOD manually from inside the CSM kicks the same
> > error.
> >
> >
> >
> > Please provide suggestions.
> >
> >
> >
> > 2020-07-02 09:08:39,665 DEBUG [c.c.h.v.m.VmwareManagerImpl]
> > (DirectAgent-66:ctx-e15c32b2 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: CopyCommand) (logid:1cfa64a9) Executing: sudo chmod
> > 1777 /var/cloudstack/mnt/VM/345050029058.2807345c
> >
> > 2020-07-02 09:08:39,666 DEBUG [c.c.h.v.m.VmwareManagerImpl]
> > (DirectAgent-66:ctx-e15c32b2 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: CopyCommand) (logid:1cfa64a9) Executing while with
> > timeout : 1440000
> >
> > 2020-07-02 09:08:39,693 DEBUG [c.c.h.v.m.VmwareManagerImpl]
> > (DirectAgent-66:ctx-e15c32b2 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: CopyCommand) (logid:1cfa64a9) Exit value is 1
> >
> > 2020-07-02 09:08:39,697 DEBUG [c.c.h.v.m.VmwareManagerImpl]
> > (DirectAgent-66:ctx-e15c32b2 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: CopyCommand) (logid:1cfa64a9) chmod: changing
> > permissions of ‘/var/cloudstack/mnt/VM/345050029058.2807345c’: Operation
> > not permitted
> >
> > 2020-07-02 09:08:39,699 WARN [c.c.h.v.m.VmwareManagerImpl]
> > (DirectAgent-66:ctx-e15c32b2 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: CopyCommand) (logid:1cfa64a9) Unable to set
> permissions
> > for /var/cloudstack/mnt/VM/345050029058.2807345c due to chmod: changing
> > permissions of ‘/var/cloudstack/mnt/VM/345050029058.2807345c’: Operation
> > not permitted
> >
> >
> >
> > Lines…
> >
> >
> >
> > 2020-07-02 09:09:00,598 DEBUG [c.c.h.v.m.HostMO]
> > (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) find VM v-2-VM on host
> >
> > 2020-07-02 09:09:00,598 INFO [c.c.h.v.m.HostMO]
> > (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) VM v-2-VM not found in
> > host cache
> >
> > 2020-07-02 09:09:00,598 DEBUG [c.c.h.v.m.HostMO]
> > (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) load VM cache on host
> >
> > 2020-07-02 09:09:00,616 DEBUG [c.c.h.v.m.HostMO]
> > (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) find VM v-2-VM on host
> >
> > 2020-07-02 09:09:00,617 DEBUG [c.c.h.v.m.HostMO]
> > (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) VM v-2-VM found in
> host
> > cache
> >
> > 2020-07-02 09:09:00,799 ERROR [c.c.h.v.m.VmwareManagerImpl]
> > (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) Unable to create
> > systemvm folder on secondary storage. location:
> > /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm
> >
> > 2020-07-02 09:09:00,800 INFO [c.c.h.v.u.VmwareHelper]
> > (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) [ignored]failed to get
> > message for exception: Unable to create systemvm folder on secondary
> > storage. location: /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm
> >
> > 2020-07-02 09:09:00,800 WARN [c.c.h.v.r.VmwareResource]
> > (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> > job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) StartCommand failed
> due
> > to Exception: com.cloud.utils.exception.CloudRuntimeException
> >
> > Message: Unable to create systemvm folder on secondary storage. location:
> > /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm
> >
> >
> >
> > com.cloud.utils.exception.CloudRuntimeException: Unable to create
> systemvm
> > folder on secondary storage. location:
> > /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm
> >
> > at
> >
> com.cloud.hypervisor.vmware.manager.VmwareManagerImpl.prepareSecondaryStorageStore(VmwareManagerImpl.java:648)
> >
> > at
> > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> > Method)
> >
> > at
> >
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> >
> > at
> >
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >
> > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> >
> > at
> >
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
> >
> > at
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
> >
> > at
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> >
> > at
> >
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95)
> >
> > at
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
> >
> > at
> >
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
> >
> > at
> > com.sun.proxy.$Proxy389.prepareSecondaryStorageStore(Unknown Source)
> >
> > at
> >
> com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:1962)
> >
> > at
> >
> com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:530)
> >
> > at
> >
> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
> >
> > at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> >
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> >
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> >
> > at
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> >
> > at
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> >
> > at
> >
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> >
> > at
> > java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> >
> > at
> >
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
> >
> > at
> >
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> >
> > at
> >
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> >
> > at java.base/java.lang.Thread.run(Thread.java:834)
> >
> > 2020-07-02 09:09:00,804 DEBUG [c.c.a.m.DirectAgentAttache]
> > (DirectAgent-17:ctx-681dc524) (logid:1cfa64a9) Seq 1-4541880224203147245:
> > Cancelling because one of the answers is false and it is stop on error.
> >
> > 2020-07-02 09:09:00,804 DEBUG [c.c.a.m.DirectAgentAttache]
> > (DirectAgent-17:ctx-681dc524) (logid:1cfa64a9) Seq 1-4541880224203147245:
> > Response Received:
> >
> > 2020-07-02 09:09:00,805 DEBUG [c.c.a.t.Request]
> > (DirectAgent-17:ctx-681dc524) (logid:1cfa64a9) Seq 1-4541880224203147245:
> > Processing: { Ans: , MgmtId: 345050029058, via:
> > 1(usphlmvesxt154.phl.global.corp.sap), Ver: v1, Flags: 110,
> >
> [{"com.cloud.agent.api.StartAnswer":{"vm":{"id":2,"name":"v-2-VM","bootloader":"HVM","state":"Starting","type":"ConsoleProxy","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":1073741824,"maxRam":1073741824,"hostName":"v-2-VM","arch":"x86_64","os":"Other
> > Linux (64-bit)","platformEmulator":"otherLinux64Guest","bootArgs":"
> > template=domP type=consoleproxy host=10.4.32.163 port=8250 name=v-2-VM
> > zone=1 pod=1 guid=Proxy.2 proxy_vm=2 disable_rp_filter=true
> > eth2ip=10.4.42.25 eth2mask=255.255.255.192 gateway=10.4.42.1
> eth0ip=0.0.0.0
> > eth0mask=0.0.0.0 eth1ip=10.4.4.35 eth1mask=255.255.252.0 mgmtcidr=
> > 10.4.32.0/24 localgw=10.4.4.1 internaldns1=10.4.202.200
> dns1=10.4.202.200
> >
> dns2=10.4.12.200","enableHA":false,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"sYnwDJYgzrMKdm3ftAO-kg","params":{"vmware.reserve.mem":"false","vmware.reserve.cpu":"false","nicAdapter":"E1000"},"uuid":"fb65df73-693d-48fb-a203-c2b5e776842a","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"9a1f2ae8-6b10-4525-8a54-a175b153b789","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"50cc970f-fbe2-3b45-9cb5-b8bcdb6dbff5","id":1,"poolType":"NetworkFilesystem","host":"usphlcv300.phl.global.corp.sap","path":"/usphlmvesx_lab_cspri","port":2049,"url":"NetworkFilesystem://usphlcv300.phl.global.corp.sap/usphlmvesx_lab_cspri/?ROLE=Primary&STOREUUID=50cc970f-fbe2-3b45-9cb5-b8bcdb6dbff5","isManaged":false}},"name":"ROOT-2","size":0,"path":"ROOT-2_2","volumeId":2,"vmName":"v-2-VM","accountId":1,"format":"OVA","provisioningType":"THIN","id":2,"deviceId":0,"bytesReadRate":0,"bytesWriteRate":0,"iopsReadRate":0,"iopsWriteRate":0,"hypervisorType":"VMware","directDownload":false}},"diskSeq":0,"path":"ROOT-2_2","type":"ROOT","_details":{"storageHost":"usphlcv300.phl.global.corp.sap","managed":"false","storagePort":"2049","volumeSize":"0"}}],"nics":[{"deviceId":2,"networkRateMbps":-1,"defaultNic":true,"pxeDisable":true,"nicUuid":"ffbc1822-7907-4cdf-9e80-bb316786e04d","details":{"PromiscuousMode":"false","MacAddressChanges":"true","ForgedTransmits":"true"},"dpdkEnabled":false,"uuid":"f3a30021-a469-413d-85a8-ca1ca68ba27c","ip":"10.4.42.25","netmask":"255.255.255.192","gateway":"10.4.42.1","mac":"1e:00:e3:00:00:19","dns1":"10.4.202.200","dns2":"10.4.12.200","broadcastType":"Vlan","type":"Public","broadcastUri":"vlan://untagged","isolationUri":"vlan://untagged","isSecurityGroupEnabled":false,"name":"vSwitch0,134,vmwaresvs"},{"deviceId":0,"networkRateMbps":-1,"defaultNic":false,"pxeDisable":true,"nicUuid":"d6521be0-85b8-4836-bf8b-8461afdf593a","details":{"PromiscuousMode":"false","MacAddressChanges":"true","ForgedTransmits":"true"},"dpdkEnabled":false,"uuid":"eda0ae3d-d15d-4f91-b8db-a719cf5a94bf","mac":"02:00:31:71:00:02","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false,"name":"vSwitch0,146,vmwaresvs"},{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"pxeDisable":true,"nicUuid":"4e6201ad-91c0-4cab-bf53-9e2440025245","details":{"PromiscuousMode":"false","MacAddressChanges":"true","ForgedTransmits":"true"},"dpdkEnabled":false,"uuid":"ce88b030-c04c-4437-b60d-9910c2cc376f","ip":"10.4.4.35","netmask":"255.255.252.0","gateway":"10.4.4.1","mac":"1e:00:6b:00:00:27","broadcastType":"Native","type":"Management","isSecurityGroupEnabled":false,"name":"vSwitch0,146,vmwaresvs"}],"guestOsDetails":{},"extraConfig":{}},"result":false,"details":"StartCommand
> > failed due to Exception:
> > com.cloud.utils.exception.CloudRuntimeException\nMessage: Unable to
> create
> > systemvm folder on secondary storage. location:
> > /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm\n","wait":0}}] }
> >
> > 2020-07-02 09:09:00,805 DEBUG [c.c.a.m.AgentAttache]
> > (DirectAgent-17:ctx-681dc524) (logid:1cfa64a9) Seq 1-4541880224203147245:
> > No more commands found
> >
> > 2020-07-02 09:09:00,806 DEBUG [c.c.a.t.Request]
> > (Work-Job-Executor-2:ctx-9aa38618 job-16/job-35 ctx-7ebd1b78)
> > (logid:1cfa64a9) Seq 1-4541880224203147245: Received: { Ans: , MgmtId:
> > 345050029058, via: 1(usphlmvesxt154.phl.global.corp.sap), Ver: v1, Flags:
> > 110, { StartAnswer } }
> >
> > 2020-07-02 09:09:00,816 INFO [c.c.v.VirtualMachineManagerImpl]
> > (Work-Job-Executor-2:ctx-9aa38618 job-16/job-35 ctx-7ebd1b78)
> > (logid:1cfa64a9) Unable to start VM on Host[-1-Routing] due to
> StartCommand
> > failed due to Exception: com.cloud.utils.exception.CloudRuntimeException
> >
> > Message: Unable to create systemvm folder on secondary storage. location:
> > /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm
> >
> >
> >
> > *Mike Corey*
> >
> >
> > Technology Senior Consultant, IT CS CTW Operation & Virtualization
> Service
> > US
> >
> >
> > *SAP AMERICA, INC.* 3999 West Chester Pike, Newtown Square, 19073 United
> > States
> >
> >
> > T +1 610 661 0905, M +1 484 274 2658, E mike.corey@sap.com
> >
> >
> >
> >
> >
> >
> >
>
>
> --
>
> Andrija Panić
>
--
Andrija Panić
RE: Initial SystemVM Creation Failing
Posted by "Corey, Mike" <mi...@sap.com>.
No upgrade - this again is the fresh install of CloudStack 4.14 with VMware resources. Maybe I should note that I created the primary and secondary storage via the "Advanced" setup wizard from the UI (logged in as admin).
I performed the ownership change you suggested so now cloud owns the systemvm folder on the secondary storage. However, now I'm getting a new error regarding copying the ISO.
Should I just bag the 4.14 installation and try to get 4.13.1 running with VMware?
2020-07-02 14:19:13,314 INFO [c.c.h.v.m.VmwareManagerImpl] (DirectAgent-10:ctx-52902ebf usphlmvesxt155.phl.global.corp.sap, job-15/job-33, cmd: StartCommand) (logid:b8ba37d1) Copy System VM patch ISO file to secondary storage. source ISO: /usr/share/cloudstack-common/vms/systemvm.iso, destination: /var/cloudstack/mnt/VM/345050029058.777ecb60/systemvm/systemvm-4.14.0.0.iso
2020-07-02 14:19:13,320 ERROR [c.c.h.v.m.VmwareManagerImpl] (DirectAgent-10:ctx-52902ebf usphlmvesxt155.phl.global.corp.sap, job-15/job-33, cmd: StartCommand) (logid:b8ba37d1) Unexpected exception
java.io.FileNotFoundException: /var/cloudstack/mnt/VM/345050029058.777ecb60/systemvm/systemvm-4.14.0.0.iso (Permission denied)
at java.base/java.io.FileOutputStream.open0(Native Method)
at java.base/java.io.FileOutputStream.open(FileOutputStream.java:298)
at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:237)
at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:187)
at org.apache.commons.io.FileUtils.doCopyFile(FileUtils.java:1123)
at org.apache.commons.io.FileUtils.copyFile(FileUtils.java:1076)
at org.apache.commons.io.FileUtils.copyFile(FileUtils.java:1028)
at com.cloud.utils.FileUtil.copyfile(FileUtil.java:30)
at com.cloud.hypervisor.vmware.manager.VmwareManagerImpl.prepareSecondaryStorageStore(VmwareManagerImpl.java:661)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy389.prepareSecondaryStorageStore(Unknown Source)
at com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:1962)
at com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:530)
at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
2020-07-02 14:19:13,321 ERROR [c.c.h.v.m.VmwareManagerImpl] (DirectAgent-10:ctx-52902ebf usphlmvesxt155.phl.global.corp.sap, job-15/job-33, cmd: StartCommand) (logid:b8ba37d1) Unable to copy systemvm ISO on secondary storage. src location: /usr/share/cloudstack-common/vms/systemvm.iso, dest location: /var/cloudstack/mnt/VM/345050029058.777ecb60/systemvm/systemvm-4.14.0.0.iso
2020-07-02 14:19:13,322 INFO [c.c.h.v.u.VmwareHelper] (DirectAgent-10:ctx-52902ebf usphlmvesxt155.phl.global.corp.sap, job-15/job-33, cmd: StartCommand) (logid:b8ba37d1) [ignored]failed to get message for exception: Unable to copy systemvm ISO on secondary storage. src location: /usr/share/cloudstack-common/vms/systemvm.iso, dest location: /var/cloudstack/mnt/VM/345050029058.777ecb60/systemvm/systemvm-4.14.0.0.iso
2020-07-02 14:19:13,323 WARN [c.c.h.v.r.VmwareResource] (DirectAgent-10:ctx-52902ebf usphlmvesxt155.phl.global.corp.sap, job-15/job-33, cmd: StartCommand) (logid:b8ba37d1) StartCommand failed due to Exception: com.cloud.utils.exception.CloudRuntimeException
Message: Unable to copy systemvm ISO on secondary storage. src location: /usr/share/cloudstack-common/vms/systemvm.iso, dest location: /var/cloudstack/mnt/VM/345050029058.777ecb60/systemvm/systemvm-4.14.0.0.iso
-----Original Message-----
From: Andrija Panic <an...@gmail.com>
Sent: Thursday, July 2, 2020 12:34 PM
To: users <us...@cloudstack.apache.org>
Subject: Re: Initial SystemVM Creation Failing
Hi Mike,
Mount you secondary storage on the management server to some temp folder,
then run: chown -R cloud:cloud <MOUNTPOINT>/systemvm/
Other folders on Secondary Storage NFS are owned by root, but the
"systemvm" should be owned by "cloud".
I assume you got this after some ACS upgrade, or similar?
On Thu, 2 Jul 2020 at 15:51, Corey, Mike <mi...@sap.com> wrote:
> Hi,
>
>
>
> When I enable the zone, the systemvm creation begins but goes into a loop
> of creating and deleting VMs in vCenter. I see in the logs there is an
> error for creating the folder permissions on the secondary storage (see
> below). Running this CHMOD manually from inside the CSM kicks the same
> error.
>
>
>
> Please provide suggestions.
>
>
>
> 2020-07-02 09:08:39,665 DEBUG [c.c.h.v.m.VmwareManagerImpl]
> (DirectAgent-66:ctx-e15c32b2 usphlmvesxt154.phl.global.corp.sap,
> job-16/job-35, cmd: CopyCommand) (logid:1cfa64a9) Executing: sudo chmod
> 1777 /var/cloudstack/mnt/VM/345050029058.2807345c
>
> 2020-07-02 09:08:39,666 DEBUG [c.c.h.v.m.VmwareManagerImpl]
> (DirectAgent-66:ctx-e15c32b2 usphlmvesxt154.phl.global.corp.sap,
> job-16/job-35, cmd: CopyCommand) (logid:1cfa64a9) Executing while with
> timeout : 1440000
>
> 2020-07-02 09:08:39,693 DEBUG [c.c.h.v.m.VmwareManagerImpl]
> (DirectAgent-66:ctx-e15c32b2 usphlmvesxt154.phl.global.corp.sap,
> job-16/job-35, cmd: CopyCommand) (logid:1cfa64a9) Exit value is 1
>
> 2020-07-02 09:08:39,697 DEBUG [c.c.h.v.m.VmwareManagerImpl]
> (DirectAgent-66:ctx-e15c32b2 usphlmvesxt154.phl.global.corp.sap,
> job-16/job-35, cmd: CopyCommand) (logid:1cfa64a9) chmod: changing
> permissions of ‘/var/cloudstack/mnt/VM/345050029058.2807345c’: Operation
> not permitted
>
> 2020-07-02 09:08:39,699 WARN [c.c.h.v.m.VmwareManagerImpl]
> (DirectAgent-66:ctx-e15c32b2 usphlmvesxt154.phl.global.corp.sap,
> job-16/job-35, cmd: CopyCommand) (logid:1cfa64a9) Unable to set permissions
> for /var/cloudstack/mnt/VM/345050029058.2807345c due to chmod: changing
> permissions of ‘/var/cloudstack/mnt/VM/345050029058.2807345c’: Operation
> not permitted
>
>
>
> Lines…
>
>
>
> 2020-07-02 09:09:00,598 DEBUG [c.c.h.v.m.HostMO]
> (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) find VM v-2-VM on host
>
> 2020-07-02 09:09:00,598 INFO [c.c.h.v.m.HostMO]
> (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) VM v-2-VM not found in
> host cache
>
> 2020-07-02 09:09:00,598 DEBUG [c.c.h.v.m.HostMO]
> (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) load VM cache on host
>
> 2020-07-02 09:09:00,616 DEBUG [c.c.h.v.m.HostMO]
> (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) find VM v-2-VM on host
>
> 2020-07-02 09:09:00,617 DEBUG [c.c.h.v.m.HostMO]
> (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) VM v-2-VM found in host
> cache
>
> 2020-07-02 09:09:00,799 ERROR [c.c.h.v.m.VmwareManagerImpl]
> (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) Unable to create
> systemvm folder on secondary storage. location:
> /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm
>
> 2020-07-02 09:09:00,800 INFO [c.c.h.v.u.VmwareHelper]
> (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) [ignored]failed to get
> message for exception: Unable to create systemvm folder on secondary
> storage. location: /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm
>
> 2020-07-02 09:09:00,800 WARN [c.c.h.v.r.VmwareResource]
> (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) StartCommand failed due
> to Exception: com.cloud.utils.exception.CloudRuntimeException
>
> Message: Unable to create systemvm folder on secondary storage. location:
> /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm
>
>
>
> com.cloud.utils.exception.CloudRuntimeException: Unable to create systemvm
> folder on secondary storage. location:
> /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm
>
> at
> com.cloud.hypervisor.vmware.manager.VmwareManagerImpl.prepareSecondaryStorageStore(VmwareManagerImpl.java:648)
>
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
>
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
>
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
>
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
>
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95)
>
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
>
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
>
> at
> com.sun.proxy.$Proxy389.prepareSecondaryStorageStore(Unknown Source)
>
> at
> com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:1962)
>
> at
> com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:530)
>
> at
> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
>
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>
> at
> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
>
> at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>
> at java.base/java.lang.Thread.run(Thread.java:834)
>
> 2020-07-02 09:09:00,804 DEBUG [c.c.a.m.DirectAgentAttache]
> (DirectAgent-17:ctx-681dc524) (logid:1cfa64a9) Seq 1-4541880224203147245:
> Cancelling because one of the answers is false and it is stop on error.
>
> 2020-07-02 09:09:00,804 DEBUG [c.c.a.m.DirectAgentAttache]
> (DirectAgent-17:ctx-681dc524) (logid:1cfa64a9) Seq 1-4541880224203147245:
> Response Received:
>
> 2020-07-02 09:09:00,805 DEBUG [c.c.a.t.Request]
> (DirectAgent-17:ctx-681dc524) (logid:1cfa64a9) Seq 1-4541880224203147245:
> Processing: { Ans: , MgmtId: 345050029058, via:
> 1(usphlmvesxt154.phl.global.corp.sap), Ver: v1, Flags: 110,
> [{"com.cloud.agent.api.StartAnswer":{"vm":{"id":2,"name":"v-2-VM","bootloader":"HVM","state":"Starting","type":"ConsoleProxy","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":1073741824,"maxRam":1073741824,"hostName":"v-2-VM","arch":"x86_64","os":"Other
> Linux (64-bit)","platformEmulator":"otherLinux64Guest","bootArgs":"
> template=domP type=consoleproxy host=10.4.32.163 port=8250 name=v-2-VM
> zone=1 pod=1 guid=Proxy.2 proxy_vm=2 disable_rp_filter=true
> eth2ip=10.4.42.25 eth2mask=255.255.255.192 gateway=10.4.42.1 eth0ip=0.0.0.0
> eth0mask=0.0.0.0 eth1ip=10.4.4.35 eth1mask=255.255.252.0 mgmtcidr=
> 10.4.32.0/24 localgw=10.4.4.1 internaldns1=10.4.202.200 dns1=10.4.202.200
> dns2=10.4.12.200","enableHA":false,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"sYnwDJYgzrMKdm3ftAO-kg","params":{"vmware.reserve.mem":"false","vmware.reserve.cpu":"false","nicAdapter":"E1000"},"uuid":"fb65df73-693d-48fb-a203-c2b5e776842a","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"9a1f2ae8-6b10-4525-8a54-a175b153b789","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"50cc970f-fbe2-3b45-9cb5-b8bcdb6dbff5","id":1,"poolType":"NetworkFilesystem","host":"usphlcv300.phl.global.corp.sap","path":"/usphlmvesx_lab_cspri","port":2049,"url":"NetworkFilesystem://usphlcv300.phl.global.corp.sap/usphlmvesx_lab_cspri/?ROLE=Primary&STOREUUID=50cc970f-fbe2-3b45-9cb5-b8bcdb6dbff5","isManaged":false}},"name":"ROOT-2","size":0,"path":"ROOT-2_2","volumeId":2,"vmName":"v-2-VM","accountId":1,"format":"OVA","provisioningType":"THIN","id":2,"deviceId":0,"bytesReadRate":0,"bytesWriteRate":0,"iopsReadRate":0,"iopsWriteRate":0,"hypervisorType":"VMware","directDownload":false}},"diskSeq":0,"path":"ROOT-2_2","type":"ROOT","_details":{"storageHost":"usphlcv300.phl.global.corp.sap","managed":"false","storagePort":"2049","volumeSize":"0"}}],"nics":[{"deviceId":2,"networkRateMbps":-1,"defaultNic":true,"pxeDisable":true,"nicUuid":"ffbc1822-7907-4cdf-9e80-bb316786e04d","details":{"PromiscuousMode":"false","MacAddressChanges":"true","ForgedTransmits":"true"},"dpdkEnabled":false,"uuid":"f3a30021-a469-413d-85a8-ca1ca68ba27c","ip":"10.4.42.25","netmask":"255.255.255.192","gateway":"10.4.42.1","mac":"1e:00:e3:00:00:19","dns1":"10.4.202.200","dns2":"10.4.12.200","broadcastType":"Vlan","type":"Public","broadcastUri":"vlan://untagged","isolationUri":"vlan://untagged","isSecurityGroupEnabled":false,"name":"vSwitch0,134,vmwaresvs"},{"deviceId":0,"networkRateMbps":-1,"defaultNic":false,"pxeDisable":true,"nicUuid":"d6521be0-85b8-4836-bf8b-8461afdf593a","details":{"PromiscuousMode":"false","MacAddressChanges":"true","ForgedTransmits":"true"},"dpdkEnabled":false,"uuid":"eda0ae3d-d15d-4f91-b8db-a719cf5a94bf","mac":"02:00:31:71:00:02","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false,"name":"vSwitch0,146,vmwaresvs"},{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"pxeDisable":true,"nicUuid":"4e6201ad-91c0-4cab-bf53-9e2440025245","details":{"PromiscuousMode":"false","MacAddressChanges":"true","ForgedTransmits":"true"},"dpdkEnabled":false,"uuid":"ce88b030-c04c-4437-b60d-9910c2cc376f","ip":"10.4.4.35","netmask":"255.255.252.0","gateway":"10.4.4.1","mac":"1e:00:6b:00:00:27","broadcastType":"Native","type":"Management","isSecurityGroupEnabled":false,"name":"vSwitch0,146,vmwaresvs"}],"guestOsDetails":{},"extraConfig":{}},"result":false,"details":"StartCommand
> failed due to Exception:
> com.cloud.utils.exception.CloudRuntimeException\nMessage: Unable to create
> systemvm folder on secondary storage. location:
> /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm\n","wait":0}}] }
>
> 2020-07-02 09:09:00,805 DEBUG [c.c.a.m.AgentAttache]
> (DirectAgent-17:ctx-681dc524) (logid:1cfa64a9) Seq 1-4541880224203147245:
> No more commands found
>
> 2020-07-02 09:09:00,806 DEBUG [c.c.a.t.Request]
> (Work-Job-Executor-2:ctx-9aa38618 job-16/job-35 ctx-7ebd1b78)
> (logid:1cfa64a9) Seq 1-4541880224203147245: Received: { Ans: , MgmtId:
> 345050029058, via: 1(usphlmvesxt154.phl.global.corp.sap), Ver: v1, Flags:
> 110, { StartAnswer } }
>
> 2020-07-02 09:09:00,816 INFO [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-2:ctx-9aa38618 job-16/job-35 ctx-7ebd1b78)
> (logid:1cfa64a9) Unable to start VM on Host[-1-Routing] due to StartCommand
> failed due to Exception: com.cloud.utils.exception.CloudRuntimeException
>
> Message: Unable to create systemvm folder on secondary storage. location:
> /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm
>
>
>
> *Mike Corey*
>
>
> Technology Senior Consultant, IT CS CTW Operation & Virtualization Service
> US
>
>
> *SAP AMERICA, INC.* 3999 West Chester Pike, Newtown Square, 19073 United
> States
>
>
> T +1 610 661 0905, M +1 484 274 2658, E mike.corey@sap.com
>
>
>
>
>
>
>
--
Andrija Panić
Re: Initial SystemVM Creation Failing
Posted by Andrija Panic <an...@gmail.com>.
Hi Mike,
Mount you secondary storage on the management server to some temp folder,
then run: chown -R cloud:cloud <MOUNTPOINT>/systemvm/
Other folders on Secondary Storage NFS are owned by root, but the
"systemvm" should be owned by "cloud".
I assume you got this after some ACS upgrade, or similar?
On Thu, 2 Jul 2020 at 15:51, Corey, Mike <mi...@sap.com> wrote:
> Hi,
>
>
>
> When I enable the zone, the systemvm creation begins but goes into a loop
> of creating and deleting VMs in vCenter. I see in the logs there is an
> error for creating the folder permissions on the secondary storage (see
> below). Running this CHMOD manually from inside the CSM kicks the same
> error.
>
>
>
> Please provide suggestions.
>
>
>
> 2020-07-02 09:08:39,665 DEBUG [c.c.h.v.m.VmwareManagerImpl]
> (DirectAgent-66:ctx-e15c32b2 usphlmvesxt154.phl.global.corp.sap,
> job-16/job-35, cmd: CopyCommand) (logid:1cfa64a9) Executing: sudo chmod
> 1777 /var/cloudstack/mnt/VM/345050029058.2807345c
>
> 2020-07-02 09:08:39,666 DEBUG [c.c.h.v.m.VmwareManagerImpl]
> (DirectAgent-66:ctx-e15c32b2 usphlmvesxt154.phl.global.corp.sap,
> job-16/job-35, cmd: CopyCommand) (logid:1cfa64a9) Executing while with
> timeout : 1440000
>
> 2020-07-02 09:08:39,693 DEBUG [c.c.h.v.m.VmwareManagerImpl]
> (DirectAgent-66:ctx-e15c32b2 usphlmvesxt154.phl.global.corp.sap,
> job-16/job-35, cmd: CopyCommand) (logid:1cfa64a9) Exit value is 1
>
> 2020-07-02 09:08:39,697 DEBUG [c.c.h.v.m.VmwareManagerImpl]
> (DirectAgent-66:ctx-e15c32b2 usphlmvesxt154.phl.global.corp.sap,
> job-16/job-35, cmd: CopyCommand) (logid:1cfa64a9) chmod: changing
> permissions of ‘/var/cloudstack/mnt/VM/345050029058.2807345c’: Operation
> not permitted
>
> 2020-07-02 09:08:39,699 WARN [c.c.h.v.m.VmwareManagerImpl]
> (DirectAgent-66:ctx-e15c32b2 usphlmvesxt154.phl.global.corp.sap,
> job-16/job-35, cmd: CopyCommand) (logid:1cfa64a9) Unable to set permissions
> for /var/cloudstack/mnt/VM/345050029058.2807345c due to chmod: changing
> permissions of ‘/var/cloudstack/mnt/VM/345050029058.2807345c’: Operation
> not permitted
>
>
>
> Lines…
>
>
>
> 2020-07-02 09:09:00,598 DEBUG [c.c.h.v.m.HostMO]
> (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) find VM v-2-VM on host
>
> 2020-07-02 09:09:00,598 INFO [c.c.h.v.m.HostMO]
> (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) VM v-2-VM not found in
> host cache
>
> 2020-07-02 09:09:00,598 DEBUG [c.c.h.v.m.HostMO]
> (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) load VM cache on host
>
> 2020-07-02 09:09:00,616 DEBUG [c.c.h.v.m.HostMO]
> (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) find VM v-2-VM on host
>
> 2020-07-02 09:09:00,617 DEBUG [c.c.h.v.m.HostMO]
> (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) VM v-2-VM found in host
> cache
>
> 2020-07-02 09:09:00,799 ERROR [c.c.h.v.m.VmwareManagerImpl]
> (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) Unable to create
> systemvm folder on secondary storage. location:
> /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm
>
> 2020-07-02 09:09:00,800 INFO [c.c.h.v.u.VmwareHelper]
> (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) [ignored]failed to get
> message for exception: Unable to create systemvm folder on secondary
> storage. location: /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm
>
> 2020-07-02 09:09:00,800 WARN [c.c.h.v.r.VmwareResource]
> (DirectAgent-17:ctx-681dc524 usphlmvesxt154.phl.global.corp.sap,
> job-16/job-35, cmd: StartCommand) (logid:1cfa64a9) StartCommand failed due
> to Exception: com.cloud.utils.exception.CloudRuntimeException
>
> Message: Unable to create systemvm folder on secondary storage. location:
> /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm
>
>
>
> com.cloud.utils.exception.CloudRuntimeException: Unable to create systemvm
> folder on secondary storage. location:
> /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm
>
> at
> com.cloud.hypervisor.vmware.manager.VmwareManagerImpl.prepareSecondaryStorageStore(VmwareManagerImpl.java:648)
>
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
>
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
>
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
>
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
>
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95)
>
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
>
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
>
> at
> com.sun.proxy.$Proxy389.prepareSecondaryStorageStore(Unknown Source)
>
> at
> com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:1962)
>
> at
> com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:530)
>
> at
> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
>
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>
> at
> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
>
> at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>
> at java.base/java.lang.Thread.run(Thread.java:834)
>
> 2020-07-02 09:09:00,804 DEBUG [c.c.a.m.DirectAgentAttache]
> (DirectAgent-17:ctx-681dc524) (logid:1cfa64a9) Seq 1-4541880224203147245:
> Cancelling because one of the answers is false and it is stop on error.
>
> 2020-07-02 09:09:00,804 DEBUG [c.c.a.m.DirectAgentAttache]
> (DirectAgent-17:ctx-681dc524) (logid:1cfa64a9) Seq 1-4541880224203147245:
> Response Received:
>
> 2020-07-02 09:09:00,805 DEBUG [c.c.a.t.Request]
> (DirectAgent-17:ctx-681dc524) (logid:1cfa64a9) Seq 1-4541880224203147245:
> Processing: { Ans: , MgmtId: 345050029058, via:
> 1(usphlmvesxt154.phl.global.corp.sap), Ver: v1, Flags: 110,
> [{"com.cloud.agent.api.StartAnswer":{"vm":{"id":2,"name":"v-2-VM","bootloader":"HVM","state":"Starting","type":"ConsoleProxy","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":1073741824,"maxRam":1073741824,"hostName":"v-2-VM","arch":"x86_64","os":"Other
> Linux (64-bit)","platformEmulator":"otherLinux64Guest","bootArgs":"
> template=domP type=consoleproxy host=10.4.32.163 port=8250 name=v-2-VM
> zone=1 pod=1 guid=Proxy.2 proxy_vm=2 disable_rp_filter=true
> eth2ip=10.4.42.25 eth2mask=255.255.255.192 gateway=10.4.42.1 eth0ip=0.0.0.0
> eth0mask=0.0.0.0 eth1ip=10.4.4.35 eth1mask=255.255.252.0 mgmtcidr=
> 10.4.32.0/24 localgw=10.4.4.1 internaldns1=10.4.202.200 dns1=10.4.202.200
> dns2=10.4.12.200","enableHA":false,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"sYnwDJYgzrMKdm3ftAO-kg","params":{"vmware.reserve.mem":"false","vmware.reserve.cpu":"false","nicAdapter":"E1000"},"uuid":"fb65df73-693d-48fb-a203-c2b5e776842a","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"9a1f2ae8-6b10-4525-8a54-a175b153b789","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"50cc970f-fbe2-3b45-9cb5-b8bcdb6dbff5","id":1,"poolType":"NetworkFilesystem","host":"usphlcv300.phl.global.corp.sap","path":"/usphlmvesx_lab_cspri","port":2049,"url":"NetworkFilesystem://usphlcv300.phl.global.corp.sap/usphlmvesx_lab_cspri/?ROLE=Primary&STOREUUID=50cc970f-fbe2-3b45-9cb5-b8bcdb6dbff5","isManaged":false}},"name":"ROOT-2","size":0,"path":"ROOT-2_2","volumeId":2,"vmName":"v-2-VM","accountId":1,"format":"OVA","provisioningType":"THIN","id":2,"deviceId":0,"bytesReadRate":0,"bytesWriteRate":0,"iopsReadRate":0,"iopsWriteRate":0,"hypervisorType":"VMware","directDownload":false}},"diskSeq":0,"path":"ROOT-2_2","type":"ROOT","_details":{"storageHost":"usphlcv300.phl.global.corp.sap","managed":"false","storagePort":"2049","volumeSize":"0"}}],"nics":[{"deviceId":2,"networkRateMbps":-1,"defaultNic":true,"pxeDisable":true,"nicUuid":"ffbc1822-7907-4cdf-9e80-bb316786e04d","details":{"PromiscuousMode":"false","MacAddressChanges":"true","ForgedTransmits":"true"},"dpdkEnabled":false,"uuid":"f3a30021-a469-413d-85a8-ca1ca68ba27c","ip":"10.4.42.25","netmask":"255.255.255.192","gateway":"10.4.42.1","mac":"1e:00:e3:00:00:19","dns1":"10.4.202.200","dns2":"10.4.12.200","broadcastType":"Vlan","type":"Public","broadcastUri":"vlan://untagged","isolationUri":"vlan://untagged","isSecurityGroupEnabled":false,"name":"vSwitch0,134,vmwaresvs"},{"deviceId":0,"networkRateMbps":-1,"defaultNic":false,"pxeDisable":true,"nicUuid":"d6521be0-85b8-4836-bf8b-8461afdf593a","details":{"PromiscuousMode":"false","MacAddressChanges":"true","ForgedTransmits":"true"},"dpdkEnabled":false,"uuid":"eda0ae3d-d15d-4f91-b8db-a719cf5a94bf","mac":"02:00:31:71:00:02","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false,"name":"vSwitch0,146,vmwaresvs"},{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"pxeDisable":true,"nicUuid":"4e6201ad-91c0-4cab-bf53-9e2440025245","details":{"PromiscuousMode":"false","MacAddressChanges":"true","ForgedTransmits":"true"},"dpdkEnabled":false,"uuid":"ce88b030-c04c-4437-b60d-9910c2cc376f","ip":"10.4.4.35","netmask":"255.255.252.0","gateway":"10.4.4.1","mac":"1e:00:6b:00:00:27","broadcastType":"Native","type":"Management","isSecurityGroupEnabled":false,"name":"vSwitch0,146,vmwaresvs"}],"guestOsDetails":{},"extraConfig":{}},"result":false,"details":"StartCommand
> failed due to Exception:
> com.cloud.utils.exception.CloudRuntimeException\nMessage: Unable to create
> systemvm folder on secondary storage. location:
> /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm\n","wait":0}}] }
>
> 2020-07-02 09:09:00,805 DEBUG [c.c.a.m.AgentAttache]
> (DirectAgent-17:ctx-681dc524) (logid:1cfa64a9) Seq 1-4541880224203147245:
> No more commands found
>
> 2020-07-02 09:09:00,806 DEBUG [c.c.a.t.Request]
> (Work-Job-Executor-2:ctx-9aa38618 job-16/job-35 ctx-7ebd1b78)
> (logid:1cfa64a9) Seq 1-4541880224203147245: Received: { Ans: , MgmtId:
> 345050029058, via: 1(usphlmvesxt154.phl.global.corp.sap), Ver: v1, Flags:
> 110, { StartAnswer } }
>
> 2020-07-02 09:09:00,816 INFO [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-2:ctx-9aa38618 job-16/job-35 ctx-7ebd1b78)
> (logid:1cfa64a9) Unable to start VM on Host[-1-Routing] due to StartCommand
> failed due to Exception: com.cloud.utils.exception.CloudRuntimeException
>
> Message: Unable to create systemvm folder on secondary storage. location:
> /var/cloudstack/mnt/VM/345050029058.2807345c/systemvm
>
>
>
> *Mike Corey*
>
>
> Technology Senior Consultant, IT CS CTW Operation & Virtualization Service
> US
>
>
> *SAP AMERICA, INC.* 3999 West Chester Pike, Newtown Square, 19073 United
> States
>
>
> T +1 610 661 0905, M +1 484 274 2658, E mike.corey@sap.com
>
>
>
>
>
>
>
--
Andrija Panić