You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Aican France <ai...@gmail.com> on 2018/02/22 08:31:03 UTC
Cloudstack 4.11 : NFS and system VMs NIC issue
Hi All,
I’m trying to build a small cloudstack lab inside Vmware Workstation 14.0.
I’m using :
- Ubuntu 14.04.5 NFS Server for primary / secondary storage –
192.168.117.129
- Ubuntu 16.04.3 for Cloudstack management server – 192.168.117.135
- Xenserver 7.3 with latest package 2017.1206 – 192.168.117.136
All these servers hosted on the same LAN (192.168.117.x/24), provisioning
VMs IP range on the same LAN.
I’m facing some issues, but may be they are linked :
- Secondary storage on cloudstack dashboard shows 0.00KB/0.00KB
- Secondary Storage VM s-1-VM and console Proxy VM v-2-VM still
pending il Starting state.
VMs are started (Xencenter View) but when logging into VMs console, NIC are
not configured.
Even if I can see the secondary storage configured in my Infrastructure
dashboard, only one NFS mount point appears on xenserver side :
192.168.117.129:/nfs/primary/ on
/run/sr-mount/e666f671-a903-16b1-7676-ff0ad35f3db4 type nfs
(rw,relatime,vers=3,rsize=262144,wsize=262144,namlen=255,acdirmin=0,acdirmax=0,soft,proto=tcp,timeo=100,retrans=12,sec=sys,mountaddr=192.168.117.129,mountvers=3,mountport=39208,mountproto=tcp,local_lock=none,addr=192.168.117.129)
/dev/mapper/XSLocalEXT--ccb0abd6--d830--bc2e--8372--daa16a7f5d78-ccb0abd6--d830--bc2e--8372--daa16a7f5d78
on /run/sr-mount/ccb0abd6-d830-bc2e-8372-daa16a7f5d78 type ext3
(rw,relatime,data=ordered)
Management-serv.log shows :
2018-02-22 08:50:06,346 DEBUG [c.c.s.StatsCollector]
(StatsCollector-2:ctx-79cce329) (logid:3cf7734e) There is no secondary
storage VM for secondary storage host nfs://192.168.117.129/nfs/secondary
Some sample logs or the Vms :
2018-02-22 08:51:31,840 DEBUG [c.c.h.x.r.w.x.CitrixStartCommandWrapper]
(DirectAgent-10:ctx-67d15938) (logid:2b6e4061) 1. The VM s-1-VM is in
Starting state.
2018-02-22 08:51:31,874 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-10:ctx-67d15938) (logid:2b6e4061) Created VM
8e0a421d-cc05-baa4-49f2-6e75fab04d5a for s-1-VM
2018-02-22 08:51:31,880 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-10:ctx-67d15938) (logid:2b6e4061) PV args are
%template=domP%type=secstorage%host=192.168.117.135%port=8250%name=s-1-VM%zone=1%pod=1%guid=s-1-VM%workers=5%resource=org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource%instance=SecStorage%sslcopy=false%role=templateProcessor%mtu=1500%eth2ip=192.168.117.40%eth2mask=255.255.255.0%gateway=192.168.117.2%eth0ip=169.254.0.32%eth0mask=255.255.0.0%eth1ip=192.168.117.6%eth1mask=255.255.255.0%mgmtcidr=
192.168.117.0/24%localgw=192.168.117.2%private.network.device=eth1%internaldns1=192.168.117.2%dns1=8.8.8.8%nfsVersion=null
2018-02-22 08:51:49,208 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-10:ctx-67d15938) (logid:2b6e4061) Creating VIF for s-1-VM on
nic [Nic:Guest-192.168.117.40-vlan://untagged]
2018-02-22 08:51:49,234 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-10:ctx-67d15938) (logid:2b6e4061) Creating VIF for s-1-VM on
nic [Nic:Control-169.254.0.32-null]
2018-02-22 08:51:49,664 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-10:ctx-67d15938) (logid:2b6e4061) Creating VIF for s-1-VM on
nic [Nic:Management-192.168.117.6-null]
2018-02-22 08:52:01,239 INFO [c.c.h.x.r.w.x.CitrixStartCommandWrapper]
(DirectAgent-10:ctx-67d15938) (logid:2b6e4061) Programmed default network
rules for s-1-VM
2018-02-22 08:52:01,239 DEBUG [c.c.h.x.r.w.x.CitrixStartCommandWrapper]
(DirectAgent-10:ctx-67d15938) (logid:2b6e4061) 2. The VM s-1-VM is in
Running state.
2018-02-22 08:54:11,074 DEBUG [c.c.a.t.Request]
(DirectAgentCronJob-3:ctx-1258f3db) (logid:2948b9e7) Seq
1-4371306388316487683: Processing: { Ans: , MgmtId: 52231347383, via:
1(xenserver_7.3-cs4.11), Ver: v1, Flags: 10,
[{"com.cloud.agent.api.ClusterVMMetaDataSyncAnswer":{"_clusterId":1,"_vmMetaDatum":{"s-1-VM":"device-model:qemu-trad;apic:true;viridian:true;timeoffset:0;pae:true;acpi:1;hpet:true;nx:true","v-2-VM":"device-model:qemu-trad;apic:true;viridian:true;timeoffset:0;pae:true;acpi:1;hpet:true;nx:true"},"_isExecuted":false,"result":true,"wait":0}}]
}
2018-02-22 08:57:11,054 DEBUG [c.c.a.t.Request]
(DirectAgentCronJob-9:ctx-6ed1ca09) (logid:2948b9e7) Seq
1-4371306388316487683: Processing: { Ans: , MgmtId: 52231347383, via:
1(xenserver_7.3-cs4.11), Ver: v1, Flags: 10,
[{"com.cloud.agent.api.ClusterVMMetaDataSyncAnswer":{"_clusterId":1,"_vmMetaDatum":{"s-1-VM":"device-model:qemu-trad;apic:true;viridian:true;timeoffset:0;pae:true;acpi:1;hpet:true;nx:true","v-2-VM":"device-model:qemu-trad;apic:true;viridian:true;timeoffset:0;pae:true;acpi:1;hpet:true;nx:true"},"_isExecuted":false,"result":true,"wait":0}}]
}
2018-02-22 09:00:11,023 DEBUG [c.c.a.t.Request]
(DirectAgentCronJob-10:ctx-49d205af) (logid:2948b9e7) Seq
1-4371306388316487683: Processing: { Ans: , MgmtId: 52231347383, via:
1(xenserver_7.3-cs4.11), Ver: v1, Flags: 10,
[{"com.cloud.agent.api.ClusterVMMetaDataSyncAnswer":{"_clusterId":1,"_vmMetaDatum":{"s-1-VM":"device-model:qemu-trad;apic:true;viridian:true;timeoffset:0;pae:true;acpi:1;hpet:true;nx:true","v-2-VM":"device-model:qemu-trad;apic:true;viridian:true;timeoffset:0;pae:true;acpi:1;hpet:true;nx:true"},"_isExecuted":false,"result":true,"wait":0}}]
}
2018-02-22 09:03:11,034 DEBUG [c.c.a.t.Request]
(DirectAgentCronJob-2:ctx-73c4598b) (logid:2948b9e7) Seq
1-4371306388316487683: Processing: { Ans: , MgmtId: 52231347383, via:
1(xenserver_7.3-cs4.11), Ver: v1, Flags: 10,
[{"com.cloud.agent.api.ClusterVMMetaDataSyncAnswer":{"_clusterId":1,"_vmMetaDatum":{"s-1-VM":"device-model:qemu-trad;apic:true;viridian:true;timeoffset:0;pae:true;acpi:1;hpet:true;nx:true","v-2-VM":"device-model:qemu-trad;apic:true;viridian:true;timeoffset:0;pae:true;acpi:1;hpet:true;nx:true"},"_isExecuted":false,"result":true,"wait":0}}]
}
2018-02-22 09:06:11,026 DEBUG [c.c.a.t.Request]
(DirectAgentCronJob-9:ctx-271aa0a3) (logid:2948b9e7) Seq
1-4371306388316487683: Processing: { Ans: , MgmtId: 52231347383, via:
1(xenserver_7.3-cs4.11), Ver: v1, Flags: 10,
[{"com.cloud.agent.api.ClusterVMMetaDataSyncAnswer":{"_clusterId":1,"_vmMetaDatum":{"s-1-VM":"device-model:qemu-trad;apic:true;viridian:true;timeoffset:0;pae:true;acpi:1;hpet:true;nx:true","v-2-VM":"device-model:qemu-trad;apic:true;viridian:true;timeoffset:0;pae:true;acpi:1;hpet:true;nx:true"},"_isExecuted":false,"result":true,"wait":0}}]
}
2018-02-22 09:06:49,391 WARN [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-10:ctx-67d15938) (logid:2b6e4061) VM.getByNameLabel s-1-V
failed due to org.apache.xmlrpc.XmlRpcException: Failed to read server's
response: connect timed out
2018-02-22 09:06:49,395 DEBUG [c.c.a.t.Request]
(DirectAgent-10:ctx-67d15938) (logid:2b6e4061) Seq 1-4371306388316487701:
Processing: { Ans: , MgmtId: 52231347383, via: 1(xenserver_7.3-cs4.11),
Ver: v1, Flags: 10,
[{"com.cloud.agent.api.StartAnswer":{"vm":{"id":1,"name":"s-1-VM","bootloader":"PyGrub","type":"SecondaryStorageVm","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":536870912,"maxRam":536870912,"arch":"x86_64","os":"Debian
GNU/Linux 7(64-bit)","bootArgs":" template=domP type=secstorage
host=192.168.117.135 port=8250 name=s-1-VM zone=1 pod=1 guid=s-1-VM
workers=5
resource=org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource
instance=SecStorage sslcopy=false role=templateProcessor mtu=1500
eth2ip=192.168.117.40 eth2mask=255.255.255.0 gateway=192.168.117.2
eth0ip=169.254.0.32 eth0mask=255.255.0.0 eth1ip=192.168.117.6
eth1mask=255.255.255.0 mgmtcidr=192.168.117.0/24 localgw=192.168.117.2
private.network.device=eth1 internaldns1=192.168.117.2 dns1=8.8.8.8
nfsVersion=null","enableHA":false,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"-8bjKvvQvmNfG0oBQsw4pw","params":{},"uuid":"910200d6-546d-4187-a88a-ebc6d8579e50","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"6f5dc061-985e-4286-84b2-f8aeab3b2960","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"2fd0ccd5-81ff-349b-9434-7a697c4aef63","id":1,"poolType":"NetworkFilesystem","host":"192.168.117.129","path":"/nfs/primary","port":2049,"url":"NetworkFilesystem://
192.168.117.129/nfs/primary/?ROLE=Primary&STOREUUID=2fd0ccd5-81ff-349b-9434-7a697c4aef63","isManaged":false}},"name":"ROOT-1","size":3355443200,"path":"1a60d132-995a-4673-afc6-f28d53390967","volumeId":1,"vmName":"s-1-VM","accountId":1,"format":"VHD","provisioningType":"THIN","id":1,"deviceId":0,"bytesReadRate":0,"bytesWriteRate":0,"iopsReadRate":0,"iopsWriteRate":0,"hypervisorType":"XenServer"}},"diskSeq":0,"path":"1a60d132-995a-4673-afc6-f28d53390967","type":"ROOT","_details":{"storageHost":"192.168.117.129","managed":"false","storagePort":"2049","volumeSize":"3355443200"}}],"nics":[{"deviceId":2,"networkRateMbps":-1,"defaultNic":true,"pxeDisable":true,"nicUuid":"4aa7df96-2e7a-4ec7-8725-00ab1a1d1881","details":{"MacAddressChanges":"true","ForgedTransmits":"true","PromiscuousMode":"false"},"uuid":"1a92c44c-3d47-412b-b1bd-59362e9252b7","ip":"192.168.117.40","netmask":"255.255.255.0","gateway":"192.168.117.2","mac":"1e:00:ea:00:00:11","dns1":"8.8.8.8","broadcastType":"Native","type":"Guest","broadcastUri":"vlan://untagged","isSecurityGroupEnabled":true},{"deviceId":0,"networkRateMbps":-1,"defaultNic":false,"pxeDisable":true,"nicUuid":"24116728-1b20-4784-96e7-bd788b911aeb","details":{"MacAddressChanges":"true","ForgedTransmits":"true","PromiscuousMode":"false"},"uuid":"6e505d4e-1eea-4423-8d19-efd99af3cfad","ip":"169.254.0.32","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:00:20","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false},{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"pxeDisable":true,"nicUuid":"a21b23ed-3985-4d2c-8750-3f8bc2f80494","details":{"MacAddressChanges":"true","ForgedTransmits":"true","PromiscuousMode":"false"},"uuid":"1788005e-821f-4fbe-856b-f5cde2a094ff","ip":"192.168.117.6","netmask":"255.255.255.0","gateway":"192.168.117.2","mac":"1e:00:95:00:00:02","broadcastType":"Native","type":"Management","isSecurityGroupEnabled":false}],"guestOsDetails":{"xenserver.dynamicMin":"536870912","xenserver.dynamicMax":"137438953472"}},"_iqnToData":{},"result":true,"wait":0}},{"com.cloud.agent.api.check.CheckSshAnswer":{"result":false,"details":"Can
not ping System vm s-1-VMdue to:VM.getByNameLabel s-1-VM failed due to
org.apache.xmlrpc.XmlRpcException: Failed to read server's response:
connect timed out","wait":0}}] }
2018-02-22 09:06:49,410 WARN [o.a.c.s.SecondaryStorageManagerImpl]
(Work-Job-Executor-1:ctx-e231fa0d job-15/job-17 ctx-8b47dd7f)
(logid:2b6e4061) Unable to ssh to the VM: Can not ping System vm s-1-VMdue
to:VM.getByNameLabel s-1-VM failed due to
org.apache.xmlrpc.XmlRpcException: Failed to read server's response:
connect timed out
2018-02-22 09:06:49,411 INFO [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-1:ctx-e231fa0d job-15/job-17 ctx-8b47dd7f)
(logid:2b6e4061) The guru did not like the answers so stopping
VM[SecondaryStorageVm|s-1-VM]
2018-02-22 09:06:49,420 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-1:ctx-e231fa0d job-15/job-17 ctx-8b47dd7f)
(logid:2b6e4061) Seq 1-4371306388316487748: Sending { Cmd , MgmtId:
52231347383, via: 1(xenserver_7.3-cs4.11), Ver: v1, Flags: 100011,
[{"com.cloud.agent.api.StopCommand":{"isProxy":false,"checkBeforeCleanup":false,"controlIp":"169.254.0.32","forceStop":false,"volumesToDisconnect":[],"vmName":"s-1-VM","executeInSequence":false,"wait":0}}]
}
2018-02-22 09:06:49,421 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-1:ctx-e231fa0d job-15/job-17 ctx-8b47dd7f)
(logid:2b6e4061) Seq 1-4371306388316487748: Executing: { Cmd , MgmtId:
52231347383, via: 1(xenserver_7.3-cs4.11), Ver: v1, Flags: 100011,
[{"com.cloud.agent.api.StopCommand":{"isProxy":false,"checkBeforeCleanup":false,"controlIp":"169.254.0.32","forceStop":false,"volumesToDisconnect":[],"vmName":"s-1-VM","executeInSequence":false,"wait":0}}]
}
2018-02-22 09:06:54,608 DEBUG [c.c.h.x.r.w.x.CitrixStopCommandWrapper]
(DirectAgent-4:ctx-0cd4fde8) (logid:2b6e4061) 9. The VM s-1-VM is in
Stopping state
2018-02-22 09:06:56,378 INFO [c.c.h.x.r.w.x.CitrixStopCommandWrapper]
(DirectAgent-4:ctx-0cd4fde8) (logid:2b6e4061) Removed network rules for vm
s-1-VM
2018-02-22 09:07:18,120 DEBUG [c.c.h.x.r.w.x.CitrixStopCommandWrapper]
(DirectAgent-4:ctx-0cd4fde8) (logid:2b6e4061) 10. The VM s-1-VM is in
Stopped state
2018-02-22 09:07:18,120 DEBUG [c.c.a.t.Request]
(DirectAgent-4:ctx-0cd4fde8) (logid:2b6e4061) Seq 1-4371306388316487748:
Processing: { Ans: , MgmtId: 52231347383, via: 1(xenserver_7.3-cs4.11),
Ver: v1, Flags: 10,
[{"com.cloud.agent.api.StopAnswer":{"platform":"device-model:qemu-trad;apic:true;viridian:true;timeoffset:0;pae:true;acpi:1;hpet:true;nx:true","result":true,"details":"Stop
VM s-1-V Succeed","wait":0}}] }
2018-02-22 09:07:18,121 DEBUG [c.c.a.m.AgentManagerImpl]
(Work-Job-Executor-1:ctx-e231fa0d job-15/job-17 ctx-8b47dd7f)
(logid:2b6e4061) Details from executing class
com.cloud.agent.api.StopCommand: Stop VM s-1-VM Succeed
2018-02-22 09:07:18,121 ERROR [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-1:ctx-e231fa0d job-15/job-17 ctx-8b47dd7f)
(logid:2b6e4061) Failed to start instance VM[SecondaryStorageVm|s-1-VM]
2018-02-22 09:07:18,126 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-1:ctx-e231fa0d job-15/job-17 ctx-8b47dd7f)
(logid:2b6e4061) Cleaning up resources for the vm
VM[SecondaryStorageVm|s-1-VM] in Starting state
2018-02-22 09:07:18,132 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-1:ctx-e231fa0d job-15/job-17 ctx-8b47dd7f)
(logid:2b6e4061) Seq 1-4371306388316487752: Sending { Cmd , MgmtId:
52231347383, via: 1(xenserver_7.3-cs4.11), Ver: v1, Flags: 100011,
[{"com.cloud.agent.api.StopCommand":{"isProxy":false,"checkBeforeCleanup":false,"controlIp":"169.254.0.32","forceStop":false,"volumesToDisconnect":[],"vmName":"s-1-VM","executeInSequence":false,"wait":0}}]
}
2018-02-22 09:07:18,132 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-1:ctx-e231fa0d job-15/job-17 ctx-8b47dd7f)
(logid:2b6e4061) Seq 1-4371306388316487752: Executing: { Cmd , MgmtId:
52231347383, via: 1(xenserver_7.3-cs4.11), Ver: v1, Flags: 100011,
[{"com.cloud.agent.api.StopCommand":{"isProxy":false,"checkBeforeCleanup":false,"controlIp":"169.254.0.32","forceStop":false,"volumesToDisconnect":[],"vmName":"s-1-VM","executeInSequence":false,"wait":0}}]
}
2018-02-22 09:07:18,180 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-1:ctx-e231fa0d job-15/job-17 ctx-8b47dd7f)
(logid:2b6e4061) Successfully released network resources for the vm
VM[SecondaryStorageVm|s-1-VM]
2018-02-22 09:07:18,181 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-1:ctx-e231fa0d job-15/job-17 ctx-8b47dd7f)
(logid:2b6e4061) Successfully cleanued up resources for the vm
VM[SecondaryStorageVm|s-1-VM] in Starting state
2018-02-22 09:07:19,067 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-3:ctx-8727633f job-15/job-19 ctx-5416845f)
(logid:2b6e4061) VM is already stopped: VM[SecondaryStorageVm|s-1-VM]
2018-02-22 09:07:19,101 DEBUG [c.c.v.VirtualMachineManagerImpl]
(secstorage-1:ctx-1a4a5425) (logid:a272efa5) Destroying vm
VM[SecondaryStorageVm|s-1-VM]
2018-02-22 09:07:19,243 DEBUG [c.c.a.t.Request] (secstorage-1:ctx-1a4a5425)
(logid:a272efa5) Seq 1-4371306388316487753: Sending { Cmd , MgmtId:
52231347383, via: 1(xenserver_7.3-cs4.11), Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"6f5dc061-985e-4286-84b2-f8aeab3b2960","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"2fd0ccd5-81ff-349b-9434-7a697c4aef63","id":1,"poolType":"NetworkFilesystem","host":"192.168.117.129","path":"/nfs/primary","port":2049,"url":"NetworkFilesystem://
192.168.117.129/nfs/primary/?ROLE=Primary&STOREUUID=2fd0ccd5-81ff-349b-9434-7a697c4aef63","isManaged":false}},"name":"ROOT-1","size":3355443200,"path":"1a60d132-995a-4673-afc6-f28d53390967","volumeId":1,"vmName":"s-1-VM","accountId":1,"format":"VHD","provisioningType":"THIN","id":1,"deviceId":0,"hypervisorType":"XenServer"}},"wait":0}}]
}
2018-02-22 09:07:19,244 DEBUG [c.c.a.t.Request] (secstorage-1:ctx-1a4a5425)
(logid:a272efa5) Seq 1-4371306388316487753: Executing: { Cmd , MgmtId:
52231347383, via: 1(xenserver_7.3-cs4.11), Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"6f5dc061-985e-4286-84b2-f8aeab3b2960","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"2fd0ccd5-81ff-349b-9434-7a697c4aef63","id":1,"poolType":"NetworkFilesystem","host":"192.168.117.129","path":"/nfs/primary","port":2049,"url":"NetworkFilesystem://
192.168.117.129/nfs/primary/?ROLE=Primary&STOREUUID=2fd0ccd5-81ff-349b-9434-7a697c4aef63","isManaged":false}},"name":"ROOT-1","size":3355443200,"path":"1a60d132-995a-4673-afc6-f28d53390967","volumeId":1,"vmName":"s-1-VM","accountId":1,"format":"VHD","provisioningType":"THIN","id":1,"deviceId":0,"hypervisorType":"XenServer"}},"wait":0}}]
}
2018-02-22 09:07:19,853 DEBUG [c.c.v.VirtualMachineManagerImpl]
(secstorage-1:ctx-1a4a5425) (logid:a272efa5) Expunged
VM[SecondaryStorageVm|s-1-VM]
Do you have any ideas about what could be wrong?
Thank you,
Thomas
Re: Cloudstack 4.11 : NFS and system VMs NIC issue
Posted by Khosrow Moossavi <km...@cloudops.com>.
I don't think this is conclusive yet. It states that your XenServer is
capable of running HVM VMs.
try this:
xe vm-list uuid=<uuid_of_system_vm_or_vr_in_question> params=hvm,name-label
if it's false, I'm not sure what's happening but if it's true I guess you
have three options:
1) wait for the fix on 4.11.0.1
2) downgrade your XenServer to 7.1.0
3) use another hypervisor (since you're building a lab)
On Fri, Feb 23, 2018 at 4:37 AM, Aican France <ai...@gmail.com> wrote:
> Hi Khosrow,
>
> I think you are right with the HVM type/mode in Xenserver :
>
> [root@xenserver-7 /]# xl info
> host : xenserver-7
> release : 4.4.0+10
> version : #1 SMP Wed Nov 29 16:48:41 UTC 2017
> machine : x86_64
> nr_cpus : 4
> max_cpu_id : 127
> nr_nodes : 1
> cores_per_socket : 1
> threads_per_core : 1
> cpu_mhz : 2496
> hw_caps :
> 078bfbff:f7fa3223:2c100800:00000121:0000000b:009c6fbb:00000000:00000100
> virt_caps : hvm
> total_memory : 4095
> free_memory : 1542
> sharing_freed_memory : 0
> sharing_used_memory : 0
> outstanding_claims : 0
> free_cpus : 0
> xen_major : 4
> xen_minor : 7
> xen_extra : .4-3.2
> xen_version : 4.7.4-3.2
> xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
> hvm-3.0-x86_32p hvm-3.0-x86_64
> xen_scheduler : credit
> xen_pagesize : 4096
> platform_params : virt_start=0xffff800000000000
> xen_changeset : bcc9e245aafb, pq ea94f077a1eb
> xen_commandline : dom0_mem=752M,max:752M watchdog ucode=scan
> dom0_max_vcpus=4 crashkernel=192M,below=4G console=vga vga=mode-0x0311
> cc_compiler : gcc (GCC) 7.2.0
> cc_compile_by : mockbuild
> cc_compile_domain : [unknown]
> cc_compile_date : Fri Jan 19 00:06:46 UTC 2018
> build_id : 50176d8c3bdf10b460e45c4b7399e0d9418c9495
> xend_config_format : 4
>
> So what would you suggest? Is there a way to solve that behaviour?
> Or should I try with an other hypervisor like KVM?
>
> BR
> Thomas
>
> 2018-02-23 3:46 GMT+01:00 Khosrow Moossavi <km...@cloudops.com>:
>
> > Hi Aican
> >
> > I've seen similar behavior but in our case it was custom 4.10 (somehow
> > similar to 4.11 in this case) and XenServer 7.1 not 7.3.
> >
> > Can you check in which mode VMs are started on XenServer? Since it's 7.3
> > I'm assuming they are brought up in HVM.
> >
> > If this is the case there's an open PR (#2465) to fix this in 4.11.0.1.
> >
> > Khosrow Moossavi
> > CloudOps
> >
> > On Feb 22, 2018 09:08, "Paul Angus" <pa...@shapeblue.com> wrote:
> >
> > Hi Aican,
> >
> > I don’t think that XenServer 7.3 is supported.
> >
> >
> > Kind regards,
> >
> > Paul Angus
> >
> > paul.angus@shapeblue.com
> > www.shapeblue.com
> > 53 Chandos Place, Covent Garden, London WC2N 4HSUK
> > @shapeblue
> >
> >
> >
> >
> > -----Original Message-----
> > From: Aican France [mailto:aican.fr@gmail.com]
> > Sent: 22 February 2018 08:31
> > To: users@cloudstack.apache.org
> > Subject: Cloudstack 4.11 : NFS and system VMs NIC issue
> >
> > Hi All,
> >
> >
> >
> > I’m trying to build a small cloudstack lab inside Vmware Workstation
> 14.0.
> >
> >
> >
> > I’m using :
> >
> > - Ubuntu 14.04.5 NFS Server for primary / secondary storage –
> > 192.168.117.129
> >
> > - Ubuntu 16.04.3 for Cloudstack management server –
> 192.168.117.135
> >
> > - Xenserver 7.3 with latest package 2017.1206 – 192.168.117.136
> >
> >
> >
> > All these servers hosted on the same LAN (192.168.117.x/24), provisioning
> > VMs IP range on the same LAN.
> >
> >
> >
> > I’m facing some issues, but may be they are linked :
> >
> > - Secondary storage on cloudstack dashboard shows 0.00KB/0.00KB
> >
> > - Secondary Storage VM s-1-VM and console Proxy VM v-2-VM still
> > pending il Starting state.
> >
> >
> >
> > VMs are started (Xencenter View) but when logging into VMs console, NIC
> are
> > not configured.
> >
> >
> >
> > Even if I can see the secondary storage configured in my Infrastructure
> > dashboard, only one NFS mount point appears on xenserver side :
> >
> > 192.168.117.129:/nfs/primary/ on
> > /run/sr-mount/e666f671-a903-16b1-7676-ff0ad35f3db4 type nfs
> > (rw,relatime,vers=3,rsize=262144,wsize=262144,namlen=
> > 255,acdirmin=0,acdirmax=0,soft,proto=tcp,timeo=100,
> > retrans=12,sec=sys,mountaddr=192.168.117.129,mountvers=3,
> > mountport=39208,mountproto=tcp,local_lock=none,addr=192.168.117.129)
> >
> > /dev/mapper/XSLocalEXT--ccb0abd6--d830--bc2e--8372--
> > daa16a7f5d78-ccb0abd6--d830--bc2e--8372--daa16a7f5d78
> > on /run/sr-mount/ccb0abd6-d830-bc2e-8372-daa16a7f5d78 type ext3
> > (rw,relatime,data=ordered)
> >
> >
> >
> > Management-serv.log shows :
> >
> > 2018-02-22 08:50:06,346 DEBUG [c.c.s.StatsCollector]
> > (StatsCollector-2:ctx-79cce329) (logid:3cf7734e) There is no secondary
> > storage VM for secondary storage host nfs://192.168.117.129/nfs/
> secondary
> >
> >
> >
> > Some sample logs or the Vms :
> >
> >
> >
> > 2018-02-22 08:51:31,840 DEBUG [c.c.h.x.r.w.x.CitrixStartCommandWrapper]
> > (DirectAgent-10:ctx-67d15938) (logid:2b6e4061) 1. The VM s-1-VM is in
> > Starting state.
> >
> > 2018-02-22 08:51:31,874 DEBUG [c.c.h.x.r.CitrixResourceBase]
> > (DirectAgent-10:ctx-67d15938) (logid:2b6e4061) Created VM
> > 8e0a421d-cc05-baa4-49f2-6e75fab04d5a for s-1-VM
> >
> > 2018-02-22 08:51:31,880 DEBUG [c.c.h.x.r.CitrixResourceBase]
> > (DirectAgent-10:ctx-67d15938) (logid:2b6e4061) PV args are
> > %template=domP%type=secstorage%host=192.168.117.
> 135%port=8250%name=s-1-VM%
> > zone=1%pod=1%guid=s-1-VM%workers=5%resource=org.apache.
> > cloudstack.storage.resource.NfsSecondaryStorageResource%
> > instance=SecStorage%sslcopy=false%role=templateProcessor%
> > mtu=1500%eth2ip=192.168.117.40%eth2mask=255.255.255.0%
> > gateway=192.168.117.2%eth0ip=169.254.0.32%eth0mask=255.255.
> > 0.0%eth1ip=192.168.117.6%eth1mask=255.255.255.0%mgmtcidr=
> > 192.168.117.0/24%localgw=192.168.117.2%private.network.
> > device=eth1%internaldns1=192.168.117.2%dns1=8.8.8.8%nfsVersion=null
> >
> > 2018-02-22 08:51:49,208 DEBUG [c.c.h.x.r.CitrixResourceBase]
> > (DirectAgent-10:ctx-67d15938) (logid:2b6e4061) Creating VIF for s-1-VM on
> > nic [Nic:Guest-192.168.117.40-vlan://untagged]
> >
> > 2018-02-22 08:51:49,234 DEBUG [c.c.h.x.r.CitrixResourceBase]
> > (DirectAgent-10:ctx-67d15938) (logid:2b6e4061) Creating VIF for s-1-VM on
> > nic [Nic:Control-169.254.0.32-null]
> >
> > 2018-02-22 08:51:49,664 DEBUG [c.c.h.x.r.CitrixResourceBase]
> > (DirectAgent-10:ctx-67d15938) (logid:2b6e4061) Creating VIF for s-1-VM on
> > nic [Nic:Management-192.168.117.6-null]
> >
> > 2018-02-22 08:52:01,239 INFO [c.c.h.x.r.w.x.CitrixStartCommandWrapper]
> > (DirectAgent-10:ctx-67d15938) (logid:2b6e4061) Programmed default network
> > rules for s-1-VM
> >
> > 2018-02-22 08:52:01,239 DEBUG [c.c.h.x.r.w.x.CitrixStartCommandWrapper]
> > (DirectAgent-10:ctx-67d15938) (logid:2b6e4061) 2. The VM s-1-VM is in
> > Running state.
> >
> > 2018-02-22 08:54:11,074 DEBUG [c.c.a.t.Request]
> > (DirectAgentCronJob-3:ctx-1258f3db) (logid:2948b9e7) Seq
> > 1-4371306388316487683: Processing: { Ans: , MgmtId: 52231347383, via:
> > 1(xenserver_7.3-cs4.11), Ver: v1, Flags: 10, [{"com.cloud.agent.api.
> > ClusterVMMetaDataSyncAnswer":{"_clusterId":1,"_vmMetaDatum":
> > {"s-1-VM":"device-model:qemu-trad;apic:true;viridian:true;
> > timeoffset:0;pae:true;acpi:1;hpet:true;nx:true","v-2-VM":"
> > device-model:qemu-trad;apic:true;viridian:true;timeoffset:
> > 0;pae:true;acpi:1;hpet:true;nx:true"},"_isExecuted":false,
> > "result":true,"wait":0}}]
> > }
> >
> > 2018-02-22 08:57:11,054 DEBUG [c.c.a.t.Request]
> > (DirectAgentCronJob-9:ctx-6ed1ca09) (logid:2948b9e7) Seq
> > 1-4371306388316487683: Processing: { Ans: , MgmtId: 52231347383, via:
> > 1(xenserver_7.3-cs4.11), Ver: v1, Flags: 10, [{"com.cloud.agent.api.
> > ClusterVMMetaDataSyncAnswer":{"_clusterId":1,"_vmMetaDatum":
> > {"s-1-VM":"device-model:qemu-trad;apic:true;viridian:true;
> > timeoffset:0;pae:true;acpi:1;hpet:true;nx:true","v-2-VM":"
> > device-model:qemu-trad;apic:true;viridian:true;timeoffset:
> > 0;pae:true;acpi:1;hpet:true;nx:true"},"_isExecuted":false,
> > "result":true,"wait":0}}]
> > }
> >
> > 2018-02-22 09:00:11,023 DEBUG [c.c.a.t.Request]
> > (DirectAgentCronJob-10:ctx-49d205af) (logid:2948b9e7) Seq
> > 1-4371306388316487683: Processing: { Ans: , MgmtId: 52231347383, via:
> > 1(xenserver_7.3-cs4.11), Ver: v1, Flags: 10, [{"com.cloud.agent.api.
> > ClusterVMMetaDataSyncAnswer":{"_clusterId":1,"_vmMetaDatum":
> > {"s-1-VM":"device-model:qemu-trad;apic:true;viridian:true;
> > timeoffset:0;pae:true;acpi:1;hpet:true;nx:true","v-2-VM":"
> > device-model:qemu-trad;apic:true;viridian:true;timeoffset:
> > 0;pae:true;acpi:1;hpet:true;nx:true"},"_isExecuted":false,
> > "result":true,"wait":0}}]
> > }
> >
> > 2018-02-22 09:03:11,034 DEBUG [c.c.a.t.Request]
> > (DirectAgentCronJob-2:ctx-73c4598b) (logid:2948b9e7) Seq
> > 1-4371306388316487683: Processing: { Ans: , MgmtId: 52231347383, via:
> > 1(xenserver_7.3-cs4.11), Ver: v1, Flags: 10, [{"com.cloud.agent.api.
> > ClusterVMMetaDataSyncAnswer":{"_clusterId":1,"_vmMetaDatum":
> > {"s-1-VM":"device-model:qemu-trad;apic:true;viridian:true;
> > timeoffset:0;pae:true;acpi:1;hpet:true;nx:true","v-2-VM":"
> > device-model:qemu-trad;apic:true;viridian:true;timeoffset:
> > 0;pae:true;acpi:1;hpet:true;nx:true"},"_isExecuted":false,
> > "result":true,"wait":0}}]
> > }
> >
> > 2018-02-22 09:06:11,026 DEBUG [c.c.a.t.Request]
> > (DirectAgentCronJob-9:ctx-271aa0a3) (logid:2948b9e7) Seq
> > 1-4371306388316487683: Processing: { Ans: , MgmtId: 52231347383, via:
> > 1(xenserver_7.3-cs4.11), Ver: v1, Flags: 10, [{"com.cloud.agent.api.
> > ClusterVMMetaDataSyncAnswer":{"_clusterId":1,"_vmMetaDatum":
> > {"s-1-VM":"device-model:qemu-trad;apic:true;viridian:true;
> > timeoffset:0;pae:true;acpi:1;hpet:true;nx:true","v-2-VM":"
> > device-model:qemu-trad;apic:true;viridian:true;timeoffset:
> > 0;pae:true;acpi:1;hpet:true;nx:true"},"_isExecuted":false,
> > "result":true,"wait":0}}]
> > }
> >
> > 2018-02-22 09:06:49,391 WARN [c.c.h.x.r.CitrixResourceBase]
> > (DirectAgent-10:ctx-67d15938) (logid:2b6e4061) VM.getByNameLabel s-1-V
> > failed due to org.apache.xmlrpc.XmlRpcException: Failed to read server's
> > response: connect timed out
> >
> > 2018-02-22 09:06:49,395 DEBUG [c.c.a.t.Request]
> > (DirectAgent-10:ctx-67d15938) (logid:2b6e4061) Seq 1-4371306388316487701:
> > Processing: { Ans: , MgmtId: 52231347383, via: 1(xenserver_7.3-cs4.11),
> > Ver: v1, Flags: 10,
> > [{"com.cloud.agent.api.StartAnswer":{"vm":{"id":1,"
> > name":"s-1-VM","bootloader":"PyGrub","type":"
> > SecondaryStorageVm","cpus":1,"
> > minSpeed":500,"maxSpeed":500,"minRam":536870912,"maxRam":
> > 536870912,"arch":"x86_64","os":"Debian
> > GNU/Linux 7(64-bit)","bootArgs":" template=domP type=secstorage
> > host=192.168.117.135 port=8250 name=s-1-VM zone=1 pod=1 guid=s-1-VM
> > workers=5
> > resource=org.apache.cloudstack.storage.resource.
> > NfsSecondaryStorageResource
> > instance=SecStorage sslcopy=false role=templateProcessor mtu=1500
> > eth2ip=192.168.117.40 eth2mask=255.255.255.0 gateway=192.168.117.2
> > eth0ip=169.254.0.32 eth0mask=255.255.0.0 eth1ip=192.168.117.6
> > eth1mask=255.255.255.0 mgmtcidr=192.168.117.0/24 localgw=192.168.117.2
> > private.network.device=eth1 internaldns1=192.168.117.2 dns1=8.8.8.8
> > nfsVersion=null","enableHA":false,"limitCpuUse":false,"
> > enableDynamicallyScaleVm":false,"vncPassword":"-8bjKvvQvmNfG0oBQsw4pw","
> > params":{},"uuid":"910200d6-546d-4187-a88a-ebc6d8579e50","
> > disks":[{"data":{"org.apache.cloudstack.storage.to.
> > VolumeObjectTO":{"uuid":"
> > 6f5dc061-985e-4286-84b2-f8aeab3b2960","volumeType":"
> > ROOT","dataStore":{"org.
> > apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"
> > 2fd0ccd5-81ff-349b-9434-7a697c4aef63","id":1,"
> > poolType":"NetworkFilesystem",
> > "host":"192.168.117.129","path":"/nfs/primary","port":
> > 2049,"url":"NetworkFilesystem://
> > 192.168.117.129/nfs/primary/?ROLE=Primary&STOREUUID=
> > 2fd0ccd5-81ff-349b-9434-
> > 7a697c4aef63","isManaged":false}},"name":"ROOT-1","size"
> > :3355443200,"path":"1a60d132-995a-4673-afc6-f28d53390967","
> > volumeId":1,"vmName":"s-1-VM","accountId":1,"format":"VHD","
> > provisioningType":"THIN","id":1,"deviceId":0,"bytesReadRate"
> > :0,"bytesWriteRate":0,"iopsReadRate":0,"iopsWriteRate":0,"
> > hypervisorType":"XenServer"}},"diskSeq":0,"path":"1a60d132-
> > 995a-4673-afc6-f28d53390967","type":"ROOT","_details":{"
> > storageHost":"192.168.117.129","managed":"false","storagePort":"2049","
> > volumeSize":"3355443200"}}],"nics":[{"deviceId":2,"networkRateMbps":-1,"
> > defaultNic":true,"pxeDisable":true,"nicUuid":"4aa7df96-2e7a-
> > 4ec7-8725-00ab1a1d1881","details":{"MacAddressChanges":
> > "true","ForgedTransmits":"true","PromiscuousMode":"
> > false"},"uuid":"1a92c44c-3d47-412b-b1bd-59362e9252b7","ip":"
> > 192.168.117.40","netmask":"255.255.255.0","gateway":"192.
> > 168.117.2","mac":"1e:00:ea:00:00:11","dns1":"8.8.8.8","
> > broadcastType":"Native","type":"Guest","broadcastUri":"vlan:
> //untagged","
> > isSecurityGroupEnabled":true},{"deviceId":0,"networkRateMbps":-1,"
> > defaultNic":false,"pxeDisable":true,"nicUuid":"24116728-
> > 1b20-4784-96e7-bd788b911aeb","details":{"MacAddressChanges":
> > "true","ForgedTransmits":"true","PromiscuousMode":"
> > false"},"uuid":"6e505d4e-1eea-4423-8d19-efd99af3cfad","ip":"
> > 169.254.0.32","netmask":"255.255.0.0","gateway":"169.254.0.
> > 1","mac":"0e:00:a9:fe:00:20","broadcastType":"LinkLocal","
> > type":"Control","
> > isSecurityGroupEnabled":false},{"deviceId":1,"networkRateMbps":-1,"
> > defaultNic":false,"pxeDisable":true,"nicUuid":"a21b23ed-
> > 3985-4d2c-8750-3f8bc2f80494","details":{"MacAddressChanges":
> > "true","ForgedTransmits":"true","PromiscuousMode":"
> > false"},"uuid":"1788005e-821f-4fbe-856b-f5cde2a094ff","ip":"
> > 192.168.117.6","netmask":"255.255.255.0","gateway":"192.168.
> > 117.2","mac":"1e:00:95:00:00:02","broadcastType":"Native","
> > type":"Management","isSecurityGroupEnabled":false}],"guestOsDetails":{"
> > xenserver.dynamicMin":"536870912","xenserver.
> > dynamicMax":"137438953472"}},"
> > _iqnToData":{},"result":true,"wait":0}},{"com.cloud.agent.
> > api.check.CheckSshAnswer":{"result":false,"details":"Can
> > not ping System vm s-1-VMdue to:VM.getByNameLabel s-1-VM failed due to
> > org.apache.xmlrpc.XmlRpcException: Failed to read server's response:
> > connect timed out","wait":0}}] }
> >
> > 2018-02-22 09:06:49,410 WARN [o.a.c.s.SecondaryStorageManagerImpl]
> > (Work-Job-Executor-1:ctx-e231fa0d job-15/job-17 ctx-8b47dd7f)
> > (logid:2b6e4061) Unable to ssh to the VM: Can not ping System vm
> s-1-VMdue
> > to:VM.getByNameLabel s-1-VM failed due to
> > org.apache.xmlrpc.XmlRpcException: Failed to read server's response:
> > connect timed out
> >
> > 2018-02-22 09:06:49,411 INFO [c.c.v.VirtualMachineManagerImpl]
> > (Work-Job-Executor-1:ctx-e231fa0d job-15/job-17 ctx-8b47dd7f)
> > (logid:2b6e4061) The guru did not like the answers so stopping
> > VM[SecondaryStorageVm|s-1-VM]
> >
> > 2018-02-22 09:06:49,420 DEBUG [c.c.a.t.Request]
> > (Work-Job-Executor-1:ctx-e231fa0d
> > job-15/job-17 ctx-8b47dd7f)
> > (logid:2b6e4061) Seq 1-4371306388316487748: Sending { Cmd , MgmtId:
> > 52231347383, via: 1(xenserver_7.3-cs4.11), Ver: v1, Flags: 100011,
> > [{"com.cloud.agent.api.StopCommand":{"isProxy":false,
> > "checkBeforeCleanup":false,"controlIp":"169.254.0.32","
> forceStop":false,"
> > volumesToDisconnect":[],"vmName":"s-1-VM","executeInSequence":false,"
> > wait":0}}]
> > }
> >
> > 2018-02-22 09:06:49,421 DEBUG [c.c.a.t.Request]
> > (Work-Job-Executor-1:ctx-e231fa0d
> > job-15/job-17 ctx-8b47dd7f)
> > (logid:2b6e4061) Seq 1-4371306388316487748: Executing: { Cmd , MgmtId:
> > 52231347383, via: 1(xenserver_7.3-cs4.11), Ver: v1, Flags: 100011,
> > [{"com.cloud.agent.api.StopCommand":{"isProxy":false,
> > "checkBeforeCleanup":false,"controlIp":"169.254.0.32","
> forceStop":false,"
> > volumesToDisconnect":[],"vmName":"s-1-VM","executeInSequence":false,"
> > wait":0}}]
> > }
> >
> > 2018-02-22 09:06:54,608 DEBUG [c.c.h.x.r.w.x.CitrixStopCommandWrapper]
> > (DirectAgent-4:ctx-0cd4fde8) (logid:2b6e4061) 9. The VM s-1-VM is in
> > Stopping state
> >
> > 2018-02-22 09:06:56,378 INFO [c.c.h.x.r.w.x.CitrixStopCommandWrapper]
> > (DirectAgent-4:ctx-0cd4fde8) (logid:2b6e4061) Removed network rules for
> vm
> > s-1-VM
> >
> > 2018-02-22 09:07:18,120 DEBUG [c.c.h.x.r.w.x.CitrixStopCommandWrapper]
> > (DirectAgent-4:ctx-0cd4fde8) (logid:2b6e4061) 10. The VM s-1-VM is in
> > Stopped state
> >
> > 2018-02-22 09:07:18,120 DEBUG [c.c.a.t.Request]
> > (DirectAgent-4:ctx-0cd4fde8) (logid:2b6e4061) Seq 1-4371306388316487748:
> > Processing: { Ans: , MgmtId: 52231347383, via: 1(xenserver_7.3-cs4.11),
> > Ver: v1, Flags: 10,
> > [{"com.cloud.agent.api.StopAnswer":{"platform":"
> > device-model:qemu-trad;apic:
> > true;viridian:true;timeoffset:0;pae:true;acpi:1;hpet:true;
> > nx:true","result":true,"details":"Stop
> > VM s-1-V Succeed","wait":0}}] }
> >
> > 2018-02-22 09:07:18,121 DEBUG [c.c.a.m.AgentManagerImpl]
> > (Work-Job-Executor-1:ctx-e231fa0d job-15/job-17 ctx-8b47dd7f)
> > (logid:2b6e4061) Details from executing class
> > com.cloud.agent.api.StopCommand: Stop VM s-1-VM Succeed
> >
> > 2018-02-22 09:07:18,121 ERROR [c.c.v.VirtualMachineManagerImpl]
> > (Work-Job-Executor-1:ctx-e231fa0d job-15/job-17 ctx-8b47dd7f)
> > (logid:2b6e4061) Failed to start instance VM[SecondaryStorageVm|s-1-VM]
> >
> > 2018-02-22 09:07:18,126 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > (Work-Job-Executor-1:ctx-e231fa0d job-15/job-17 ctx-8b47dd7f)
> > (logid:2b6e4061) Cleaning up resources for the vm
> > VM[SecondaryStorageVm|s-1-VM] in Starting state
> >
> > 2018-02-22 09:07:18,132 DEBUG [c.c.a.t.Request]
> > (Work-Job-Executor-1:ctx-e231fa0d
> > job-15/job-17 ctx-8b47dd7f)
> > (logid:2b6e4061) Seq 1-4371306388316487752: Sending { Cmd , MgmtId:
> > 52231347383, via: 1(xenserver_7.3-cs4.11), Ver: v1, Flags: 100011,
> > [{"com.cloud.agent.api.StopCommand":{"isProxy":false,
> > "checkBeforeCleanup":false,"controlIp":"169.254.0.32","
> forceStop":false,"
> > volumesToDisconnect":[],"vmName":"s-1-VM","executeInSequence":false,"
> > wait":0}}]
> > }
> >
> > 2018-02-22 09:07:18,132 DEBUG [c.c.a.t.Request]
> > (Work-Job-Executor-1:ctx-e231fa0d
> > job-15/job-17 ctx-8b47dd7f)
> > (logid:2b6e4061) Seq 1-4371306388316487752: Executing: { Cmd , MgmtId:
> > 52231347383, via: 1(xenserver_7.3-cs4.11), Ver: v1, Flags: 100011,
> > [{"com.cloud.agent.api.StopCommand":{"isProxy":false,
> > "checkBeforeCleanup":false,"controlIp":"169.254.0.32","
> forceStop":false,"
> > volumesToDisconnect":[],"vmName":"s-1-VM","executeInSequence":false,"
> > wait":0}}]
> > }
> >
> > 2018-02-22 09:07:18,180 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > (Work-Job-Executor-1:ctx-e231fa0d job-15/job-17 ctx-8b47dd7f)
> > (logid:2b6e4061) Successfully released network resources for the vm
> > VM[SecondaryStorageVm|s-1-VM]
> >
> > 2018-02-22 09:07:18,181 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > (Work-Job-Executor-1:ctx-e231fa0d job-15/job-17 ctx-8b47dd7f)
> > (logid:2b6e4061) Successfully cleanued up resources for the vm
> > VM[SecondaryStorageVm|s-1-VM] in Starting state
> >
> > 2018-02-22 09:07:19,067 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > (Work-Job-Executor-3:ctx-8727633f job-15/job-19 ctx-5416845f)
> > (logid:2b6e4061) VM is already stopped: VM[SecondaryStorageVm|s-1-VM]
> >
> > 2018-02-22 09:07:19,101 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > (secstorage-1:ctx-1a4a5425) (logid:a272efa5) Destroying vm
> > VM[SecondaryStorageVm|s-1-VM]
> >
> > 2018-02-22 09:07:19,243 DEBUG [c.c.a.t.Request]
> (secstorage-1:ctx-1a4a5425)
> > (logid:a272efa5) Seq 1-4371306388316487753: Sending { Cmd , MgmtId:
> > 52231347383, via: 1(xenserver_7.3-cs4.11), Ver: v1, Flags: 100011,
> > [{"org.apache.cloudstack.storage.command.DeleteCommand"
> > :{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"
> > 6f5dc061-985e-4286-84b2-f8aeab3b2960","volumeType":"
> > ROOT","dataStore":{"org.
> > apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"
> > 2fd0ccd5-81ff-349b-9434-7a697c4aef63","id":1,"
> > poolType":"NetworkFilesystem",
> > "host":"192.168.117.129","path":"/nfs/primary","port":
> > 2049,"url":"NetworkFilesystem://
> > 192.168.117.129/nfs/primary/?ROLE=Primary&STOREUUID=
> > 2fd0ccd5-81ff-349b-9434-
> > 7a697c4aef63","isManaged":false}},"name":"ROOT-1","size"
> > :3355443200,"path":"1a60d132-995a-4673-afc6-f28d53390967","
> > volumeId":1,"vmName":"s-1-VM","accountId":1,"format":"VHD","
> > provisioningType":"THIN","id":1,"deviceId":0,"
> > hypervisorType":"XenServer"}},
> > "wait":0}}]
> > }
> >
> > 2018-02-22 09:07:19,244 DEBUG [c.c.a.t.Request]
> (secstorage-1:ctx-1a4a5425)
> > (logid:a272efa5) Seq 1-4371306388316487753: Executing: { Cmd , MgmtId:
> > 52231347383, via: 1(xenserver_7.3-cs4.11), Ver: v1, Flags: 100011,
> > [{"org.apache.cloudstack.storage.command.DeleteCommand"
> > :{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"
> > 6f5dc061-985e-4286-84b2-f8aeab3b2960","volumeType":"
> > ROOT","dataStore":{"org.
> > apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"
> > 2fd0ccd5-81ff-349b-9434-7a697c4aef63","id":1,"
> > poolType":"NetworkFilesystem",
> > "host":"192.168.117.129","path":"/nfs/primary","port":
> > 2049,"url":"NetworkFilesystem://
> > 192.168.117.129/nfs/primary/?ROLE=Primary&STOREUUID=
> > 2fd0ccd5-81ff-349b-9434-
> > 7a697c4aef63","isManaged":false}},"name":"ROOT-1","size"
> > :3355443200,"path":"1a60d132-995a-4673-afc6-f28d53390967","
> > volumeId":1,"vmName":"s-1-VM","accountId":1,"format":"VHD","
> > provisioningType":"THIN","id":1,"deviceId":0,"
> > hypervisorType":"XenServer"}},
> > "wait":0}}]
> > }
> >
> > 2018-02-22 09:07:19,853 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > (secstorage-1:ctx-1a4a5425) (logid:a272efa5) Expunged
> > VM[SecondaryStorageVm|s-1-VM]
> >
> >
> >
> >
> >
> > Do you have any ideas about what could be wrong?
> >
> >
> >
> > Thank you,
> >
> >
> >
> > Thomas
> >
>
Re: Cloudstack 4.11 : NFS and system VMs NIC issue
Posted by Aican France <ai...@gmail.com>.
Hi Khosrow,
I think you are right with the HVM type/mode in Xenserver :
[root@xenserver-7 /]# xl info
host : xenserver-7
release : 4.4.0+10
version : #1 SMP Wed Nov 29 16:48:41 UTC 2017
machine : x86_64
nr_cpus : 4
max_cpu_id : 127
nr_nodes : 1
cores_per_socket : 1
threads_per_core : 1
cpu_mhz : 2496
hw_caps :
078bfbff:f7fa3223:2c100800:00000121:0000000b:009c6fbb:00000000:00000100
virt_caps : hvm
total_memory : 4095
free_memory : 1542
sharing_freed_memory : 0
sharing_used_memory : 0
outstanding_claims : 0
free_cpus : 0
xen_major : 4
xen_minor : 7
xen_extra : .4-3.2
xen_version : 4.7.4-3.2
xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler : credit
xen_pagesize : 4096
platform_params : virt_start=0xffff800000000000
xen_changeset : bcc9e245aafb, pq ea94f077a1eb
xen_commandline : dom0_mem=752M,max:752M watchdog ucode=scan
dom0_max_vcpus=4 crashkernel=192M,below=4G console=vga vga=mode-0x0311
cc_compiler : gcc (GCC) 7.2.0
cc_compile_by : mockbuild
cc_compile_domain : [unknown]
cc_compile_date : Fri Jan 19 00:06:46 UTC 2018
build_id : 50176d8c3bdf10b460e45c4b7399e0d9418c9495
xend_config_format : 4
So what would you suggest? Is there a way to solve that behaviour?
Or should I try with an other hypervisor like KVM?
BR
Thomas
2018-02-23 3:46 GMT+01:00 Khosrow Moossavi <km...@cloudops.com>:
> Hi Aican
>
> I've seen similar behavior but in our case it was custom 4.10 (somehow
> similar to 4.11 in this case) and XenServer 7.1 not 7.3.
>
> Can you check in which mode VMs are started on XenServer? Since it's 7.3
> I'm assuming they are brought up in HVM.
>
> If this is the case there's an open PR (#2465) to fix this in 4.11.0.1.
>
> Khosrow Moossavi
> CloudOps
>
> On Feb 22, 2018 09:08, "Paul Angus" <pa...@shapeblue.com> wrote:
>
> Hi Aican,
>
> I don’t think that XenServer 7.3 is supported.
>
>
> Kind regards,
>
> Paul Angus
>
> paul.angus@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London WC2N 4HSUK
> @shapeblue
>
>
>
>
> -----Original Message-----
> From: Aican France [mailto:aican.fr@gmail.com]
> Sent: 22 February 2018 08:31
> To: users@cloudstack.apache.org
> Subject: Cloudstack 4.11 : NFS and system VMs NIC issue
>
> Hi All,
>
>
>
> I’m trying to build a small cloudstack lab inside Vmware Workstation 14.0.
>
>
>
> I’m using :
>
> - Ubuntu 14.04.5 NFS Server for primary / secondary storage –
> 192.168.117.129
>
> - Ubuntu 16.04.3 for Cloudstack management server – 192.168.117.135
>
> - Xenserver 7.3 with latest package 2017.1206 – 192.168.117.136
>
>
>
> All these servers hosted on the same LAN (192.168.117.x/24), provisioning
> VMs IP range on the same LAN.
>
>
>
> I’m facing some issues, but may be they are linked :
>
> - Secondary storage on cloudstack dashboard shows 0.00KB/0.00KB
>
> - Secondary Storage VM s-1-VM and console Proxy VM v-2-VM still
> pending il Starting state.
>
>
>
> VMs are started (Xencenter View) but when logging into VMs console, NIC are
> not configured.
>
>
>
> Even if I can see the secondary storage configured in my Infrastructure
> dashboard, only one NFS mount point appears on xenserver side :
>
> 192.168.117.129:/nfs/primary/ on
> /run/sr-mount/e666f671-a903-16b1-7676-ff0ad35f3db4 type nfs
> (rw,relatime,vers=3,rsize=262144,wsize=262144,namlen=
> 255,acdirmin=0,acdirmax=0,soft,proto=tcp,timeo=100,
> retrans=12,sec=sys,mountaddr=192.168.117.129,mountvers=3,
> mountport=39208,mountproto=tcp,local_lock=none,addr=192.168.117.129)
>
> /dev/mapper/XSLocalEXT--ccb0abd6--d830--bc2e--8372--
> daa16a7f5d78-ccb0abd6--d830--bc2e--8372--daa16a7f5d78
> on /run/sr-mount/ccb0abd6-d830-bc2e-8372-daa16a7f5d78 type ext3
> (rw,relatime,data=ordered)
>
>
>
> Management-serv.log shows :
>
> 2018-02-22 08:50:06,346 DEBUG [c.c.s.StatsCollector]
> (StatsCollector-2:ctx-79cce329) (logid:3cf7734e) There is no secondary
> storage VM for secondary storage host nfs://192.168.117.129/nfs/secondary
>
>
>
> Some sample logs or the Vms :
>
>
>
> 2018-02-22 08:51:31,840 DEBUG [c.c.h.x.r.w.x.CitrixStartCommandWrapper]
> (DirectAgent-10:ctx-67d15938) (logid:2b6e4061) 1. The VM s-1-VM is in
> Starting state.
>
> 2018-02-22 08:51:31,874 DEBUG [c.c.h.x.r.CitrixResourceBase]
> (DirectAgent-10:ctx-67d15938) (logid:2b6e4061) Created VM
> 8e0a421d-cc05-baa4-49f2-6e75fab04d5a for s-1-VM
>
> 2018-02-22 08:51:31,880 DEBUG [c.c.h.x.r.CitrixResourceBase]
> (DirectAgent-10:ctx-67d15938) (logid:2b6e4061) PV args are
> %template=domP%type=secstorage%host=192.168.117.135%port=8250%name=s-1-VM%
> zone=1%pod=1%guid=s-1-VM%workers=5%resource=org.apache.
> cloudstack.storage.resource.NfsSecondaryStorageResource%
> instance=SecStorage%sslcopy=false%role=templateProcessor%
> mtu=1500%eth2ip=192.168.117.40%eth2mask=255.255.255.0%
> gateway=192.168.117.2%eth0ip=169.254.0.32%eth0mask=255.255.
> 0.0%eth1ip=192.168.117.6%eth1mask=255.255.255.0%mgmtcidr=
> 192.168.117.0/24%localgw=192.168.117.2%private.network.
> device=eth1%internaldns1=192.168.117.2%dns1=8.8.8.8%nfsVersion=null
>
> 2018-02-22 08:51:49,208 DEBUG [c.c.h.x.r.CitrixResourceBase]
> (DirectAgent-10:ctx-67d15938) (logid:2b6e4061) Creating VIF for s-1-VM on
> nic [Nic:Guest-192.168.117.40-vlan://untagged]
>
> 2018-02-22 08:51:49,234 DEBUG [c.c.h.x.r.CitrixResourceBase]
> (DirectAgent-10:ctx-67d15938) (logid:2b6e4061) Creating VIF for s-1-VM on
> nic [Nic:Control-169.254.0.32-null]
>
> 2018-02-22 08:51:49,664 DEBUG [c.c.h.x.r.CitrixResourceBase]
> (DirectAgent-10:ctx-67d15938) (logid:2b6e4061) Creating VIF for s-1-VM on
> nic [Nic:Management-192.168.117.6-null]
>
> 2018-02-22 08:52:01,239 INFO [c.c.h.x.r.w.x.CitrixStartCommandWrapper]
> (DirectAgent-10:ctx-67d15938) (logid:2b6e4061) Programmed default network
> rules for s-1-VM
>
> 2018-02-22 08:52:01,239 DEBUG [c.c.h.x.r.w.x.CitrixStartCommandWrapper]
> (DirectAgent-10:ctx-67d15938) (logid:2b6e4061) 2. The VM s-1-VM is in
> Running state.
>
> 2018-02-22 08:54:11,074 DEBUG [c.c.a.t.Request]
> (DirectAgentCronJob-3:ctx-1258f3db) (logid:2948b9e7) Seq
> 1-4371306388316487683: Processing: { Ans: , MgmtId: 52231347383, via:
> 1(xenserver_7.3-cs4.11), Ver: v1, Flags: 10, [{"com.cloud.agent.api.
> ClusterVMMetaDataSyncAnswer":{"_clusterId":1,"_vmMetaDatum":
> {"s-1-VM":"device-model:qemu-trad;apic:true;viridian:true;
> timeoffset:0;pae:true;acpi:1;hpet:true;nx:true","v-2-VM":"
> device-model:qemu-trad;apic:true;viridian:true;timeoffset:
> 0;pae:true;acpi:1;hpet:true;nx:true"},"_isExecuted":false,
> "result":true,"wait":0}}]
> }
>
> 2018-02-22 08:57:11,054 DEBUG [c.c.a.t.Request]
> (DirectAgentCronJob-9:ctx-6ed1ca09) (logid:2948b9e7) Seq
> 1-4371306388316487683: Processing: { Ans: , MgmtId: 52231347383, via:
> 1(xenserver_7.3-cs4.11), Ver: v1, Flags: 10, [{"com.cloud.agent.api.
> ClusterVMMetaDataSyncAnswer":{"_clusterId":1,"_vmMetaDatum":
> {"s-1-VM":"device-model:qemu-trad;apic:true;viridian:true;
> timeoffset:0;pae:true;acpi:1;hpet:true;nx:true","v-2-VM":"
> device-model:qemu-trad;apic:true;viridian:true;timeoffset:
> 0;pae:true;acpi:1;hpet:true;nx:true"},"_isExecuted":false,
> "result":true,"wait":0}}]
> }
>
> 2018-02-22 09:00:11,023 DEBUG [c.c.a.t.Request]
> (DirectAgentCronJob-10:ctx-49d205af) (logid:2948b9e7) Seq
> 1-4371306388316487683: Processing: { Ans: , MgmtId: 52231347383, via:
> 1(xenserver_7.3-cs4.11), Ver: v1, Flags: 10, [{"com.cloud.agent.api.
> ClusterVMMetaDataSyncAnswer":{"_clusterId":1,"_vmMetaDatum":
> {"s-1-VM":"device-model:qemu-trad;apic:true;viridian:true;
> timeoffset:0;pae:true;acpi:1;hpet:true;nx:true","v-2-VM":"
> device-model:qemu-trad;apic:true;viridian:true;timeoffset:
> 0;pae:true;acpi:1;hpet:true;nx:true"},"_isExecuted":false,
> "result":true,"wait":0}}]
> }
>
> 2018-02-22 09:03:11,034 DEBUG [c.c.a.t.Request]
> (DirectAgentCronJob-2:ctx-73c4598b) (logid:2948b9e7) Seq
> 1-4371306388316487683: Processing: { Ans: , MgmtId: 52231347383, via:
> 1(xenserver_7.3-cs4.11), Ver: v1, Flags: 10, [{"com.cloud.agent.api.
> ClusterVMMetaDataSyncAnswer":{"_clusterId":1,"_vmMetaDatum":
> {"s-1-VM":"device-model:qemu-trad;apic:true;viridian:true;
> timeoffset:0;pae:true;acpi:1;hpet:true;nx:true","v-2-VM":"
> device-model:qemu-trad;apic:true;viridian:true;timeoffset:
> 0;pae:true;acpi:1;hpet:true;nx:true"},"_isExecuted":false,
> "result":true,"wait":0}}]
> }
>
> 2018-02-22 09:06:11,026 DEBUG [c.c.a.t.Request]
> (DirectAgentCronJob-9:ctx-271aa0a3) (logid:2948b9e7) Seq
> 1-4371306388316487683: Processing: { Ans: , MgmtId: 52231347383, via:
> 1(xenserver_7.3-cs4.11), Ver: v1, Flags: 10, [{"com.cloud.agent.api.
> ClusterVMMetaDataSyncAnswer":{"_clusterId":1,"_vmMetaDatum":
> {"s-1-VM":"device-model:qemu-trad;apic:true;viridian:true;
> timeoffset:0;pae:true;acpi:1;hpet:true;nx:true","v-2-VM":"
> device-model:qemu-trad;apic:true;viridian:true;timeoffset:
> 0;pae:true;acpi:1;hpet:true;nx:true"},"_isExecuted":false,
> "result":true,"wait":0}}]
> }
>
> 2018-02-22 09:06:49,391 WARN [c.c.h.x.r.CitrixResourceBase]
> (DirectAgent-10:ctx-67d15938) (logid:2b6e4061) VM.getByNameLabel s-1-V
> failed due to org.apache.xmlrpc.XmlRpcException: Failed to read server's
> response: connect timed out
>
> 2018-02-22 09:06:49,395 DEBUG [c.c.a.t.Request]
> (DirectAgent-10:ctx-67d15938) (logid:2b6e4061) Seq 1-4371306388316487701:
> Processing: { Ans: , MgmtId: 52231347383, via: 1(xenserver_7.3-cs4.11),
> Ver: v1, Flags: 10,
> [{"com.cloud.agent.api.StartAnswer":{"vm":{"id":1,"
> name":"s-1-VM","bootloader":"PyGrub","type":"
> SecondaryStorageVm","cpus":1,"
> minSpeed":500,"maxSpeed":500,"minRam":536870912,"maxRam":
> 536870912,"arch":"x86_64","os":"Debian
> GNU/Linux 7(64-bit)","bootArgs":" template=domP type=secstorage
> host=192.168.117.135 port=8250 name=s-1-VM zone=1 pod=1 guid=s-1-VM
> workers=5
> resource=org.apache.cloudstack.storage.resource.
> NfsSecondaryStorageResource
> instance=SecStorage sslcopy=false role=templateProcessor mtu=1500
> eth2ip=192.168.117.40 eth2mask=255.255.255.0 gateway=192.168.117.2
> eth0ip=169.254.0.32 eth0mask=255.255.0.0 eth1ip=192.168.117.6
> eth1mask=255.255.255.0 mgmtcidr=192.168.117.0/24 localgw=192.168.117.2
> private.network.device=eth1 internaldns1=192.168.117.2 dns1=8.8.8.8
> nfsVersion=null","enableHA":false,"limitCpuUse":false,"
> enableDynamicallyScaleVm":false,"vncPassword":"-8bjKvvQvmNfG0oBQsw4pw","
> params":{},"uuid":"910200d6-546d-4187-a88a-ebc6d8579e50","
> disks":[{"data":{"org.apache.cloudstack.storage.to.
> VolumeObjectTO":{"uuid":"
> 6f5dc061-985e-4286-84b2-f8aeab3b2960","volumeType":"
> ROOT","dataStore":{"org.
> apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"
> 2fd0ccd5-81ff-349b-9434-7a697c4aef63","id":1,"
> poolType":"NetworkFilesystem",
> "host":"192.168.117.129","path":"/nfs/primary","port":
> 2049,"url":"NetworkFilesystem://
> 192.168.117.129/nfs/primary/?ROLE=Primary&STOREUUID=
> 2fd0ccd5-81ff-349b-9434-
> 7a697c4aef63","isManaged":false}},"name":"ROOT-1","size"
> :3355443200,"path":"1a60d132-995a-4673-afc6-f28d53390967","
> volumeId":1,"vmName":"s-1-VM","accountId":1,"format":"VHD","
> provisioningType":"THIN","id":1,"deviceId":0,"bytesReadRate"
> :0,"bytesWriteRate":0,"iopsReadRate":0,"iopsWriteRate":0,"
> hypervisorType":"XenServer"}},"diskSeq":0,"path":"1a60d132-
> 995a-4673-afc6-f28d53390967","type":"ROOT","_details":{"
> storageHost":"192.168.117.129","managed":"false","storagePort":"2049","
> volumeSize":"3355443200"}}],"nics":[{"deviceId":2,"networkRateMbps":-1,"
> defaultNic":true,"pxeDisable":true,"nicUuid":"4aa7df96-2e7a-
> 4ec7-8725-00ab1a1d1881","details":{"MacAddressChanges":
> "true","ForgedTransmits":"true","PromiscuousMode":"
> false"},"uuid":"1a92c44c-3d47-412b-b1bd-59362e9252b7","ip":"
> 192.168.117.40","netmask":"255.255.255.0","gateway":"192.
> 168.117.2","mac":"1e:00:ea:00:00:11","dns1":"8.8.8.8","
> broadcastType":"Native","type":"Guest","broadcastUri":"vlan://untagged","
> isSecurityGroupEnabled":true},{"deviceId":0,"networkRateMbps":-1,"
> defaultNic":false,"pxeDisable":true,"nicUuid":"24116728-
> 1b20-4784-96e7-bd788b911aeb","details":{"MacAddressChanges":
> "true","ForgedTransmits":"true","PromiscuousMode":"
> false"},"uuid":"6e505d4e-1eea-4423-8d19-efd99af3cfad","ip":"
> 169.254.0.32","netmask":"255.255.0.0","gateway":"169.254.0.
> 1","mac":"0e:00:a9:fe:00:20","broadcastType":"LinkLocal","
> type":"Control","
> isSecurityGroupEnabled":false},{"deviceId":1,"networkRateMbps":-1,"
> defaultNic":false,"pxeDisable":true,"nicUuid":"a21b23ed-
> 3985-4d2c-8750-3f8bc2f80494","details":{"MacAddressChanges":
> "true","ForgedTransmits":"true","PromiscuousMode":"
> false"},"uuid":"1788005e-821f-4fbe-856b-f5cde2a094ff","ip":"
> 192.168.117.6","netmask":"255.255.255.0","gateway":"192.168.
> 117.2","mac":"1e:00:95:00:00:02","broadcastType":"Native","
> type":"Management","isSecurityGroupEnabled":false}],"guestOsDetails":{"
> xenserver.dynamicMin":"536870912","xenserver.
> dynamicMax":"137438953472"}},"
> _iqnToData":{},"result":true,"wait":0}},{"com.cloud.agent.
> api.check.CheckSshAnswer":{"result":false,"details":"Can
> not ping System vm s-1-VMdue to:VM.getByNameLabel s-1-VM failed due to
> org.apache.xmlrpc.XmlRpcException: Failed to read server's response:
> connect timed out","wait":0}}] }
>
> 2018-02-22 09:06:49,410 WARN [o.a.c.s.SecondaryStorageManagerImpl]
> (Work-Job-Executor-1:ctx-e231fa0d job-15/job-17 ctx-8b47dd7f)
> (logid:2b6e4061) Unable to ssh to the VM: Can not ping System vm s-1-VMdue
> to:VM.getByNameLabel s-1-VM failed due to
> org.apache.xmlrpc.XmlRpcException: Failed to read server's response:
> connect timed out
>
> 2018-02-22 09:06:49,411 INFO [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-1:ctx-e231fa0d job-15/job-17 ctx-8b47dd7f)
> (logid:2b6e4061) The guru did not like the answers so stopping
> VM[SecondaryStorageVm|s-1-VM]
>
> 2018-02-22 09:06:49,420 DEBUG [c.c.a.t.Request]
> (Work-Job-Executor-1:ctx-e231fa0d
> job-15/job-17 ctx-8b47dd7f)
> (logid:2b6e4061) Seq 1-4371306388316487748: Sending { Cmd , MgmtId:
> 52231347383, via: 1(xenserver_7.3-cs4.11), Ver: v1, Flags: 100011,
> [{"com.cloud.agent.api.StopCommand":{"isProxy":false,
> "checkBeforeCleanup":false,"controlIp":"169.254.0.32","forceStop":false,"
> volumesToDisconnect":[],"vmName":"s-1-VM","executeInSequence":false,"
> wait":0}}]
> }
>
> 2018-02-22 09:06:49,421 DEBUG [c.c.a.t.Request]
> (Work-Job-Executor-1:ctx-e231fa0d
> job-15/job-17 ctx-8b47dd7f)
> (logid:2b6e4061) Seq 1-4371306388316487748: Executing: { Cmd , MgmtId:
> 52231347383, via: 1(xenserver_7.3-cs4.11), Ver: v1, Flags: 100011,
> [{"com.cloud.agent.api.StopCommand":{"isProxy":false,
> "checkBeforeCleanup":false,"controlIp":"169.254.0.32","forceStop":false,"
> volumesToDisconnect":[],"vmName":"s-1-VM","executeInSequence":false,"
> wait":0}}]
> }
>
> 2018-02-22 09:06:54,608 DEBUG [c.c.h.x.r.w.x.CitrixStopCommandWrapper]
> (DirectAgent-4:ctx-0cd4fde8) (logid:2b6e4061) 9. The VM s-1-VM is in
> Stopping state
>
> 2018-02-22 09:06:56,378 INFO [c.c.h.x.r.w.x.CitrixStopCommandWrapper]
> (DirectAgent-4:ctx-0cd4fde8) (logid:2b6e4061) Removed network rules for vm
> s-1-VM
>
> 2018-02-22 09:07:18,120 DEBUG [c.c.h.x.r.w.x.CitrixStopCommandWrapper]
> (DirectAgent-4:ctx-0cd4fde8) (logid:2b6e4061) 10. The VM s-1-VM is in
> Stopped state
>
> 2018-02-22 09:07:18,120 DEBUG [c.c.a.t.Request]
> (DirectAgent-4:ctx-0cd4fde8) (logid:2b6e4061) Seq 1-4371306388316487748:
> Processing: { Ans: , MgmtId: 52231347383, via: 1(xenserver_7.3-cs4.11),
> Ver: v1, Flags: 10,
> [{"com.cloud.agent.api.StopAnswer":{"platform":"
> device-model:qemu-trad;apic:
> true;viridian:true;timeoffset:0;pae:true;acpi:1;hpet:true;
> nx:true","result":true,"details":"Stop
> VM s-1-V Succeed","wait":0}}] }
>
> 2018-02-22 09:07:18,121 DEBUG [c.c.a.m.AgentManagerImpl]
> (Work-Job-Executor-1:ctx-e231fa0d job-15/job-17 ctx-8b47dd7f)
> (logid:2b6e4061) Details from executing class
> com.cloud.agent.api.StopCommand: Stop VM s-1-VM Succeed
>
> 2018-02-22 09:07:18,121 ERROR [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-1:ctx-e231fa0d job-15/job-17 ctx-8b47dd7f)
> (logid:2b6e4061) Failed to start instance VM[SecondaryStorageVm|s-1-VM]
>
> 2018-02-22 09:07:18,126 DEBUG [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-1:ctx-e231fa0d job-15/job-17 ctx-8b47dd7f)
> (logid:2b6e4061) Cleaning up resources for the vm
> VM[SecondaryStorageVm|s-1-VM] in Starting state
>
> 2018-02-22 09:07:18,132 DEBUG [c.c.a.t.Request]
> (Work-Job-Executor-1:ctx-e231fa0d
> job-15/job-17 ctx-8b47dd7f)
> (logid:2b6e4061) Seq 1-4371306388316487752: Sending { Cmd , MgmtId:
> 52231347383, via: 1(xenserver_7.3-cs4.11), Ver: v1, Flags: 100011,
> [{"com.cloud.agent.api.StopCommand":{"isProxy":false,
> "checkBeforeCleanup":false,"controlIp":"169.254.0.32","forceStop":false,"
> volumesToDisconnect":[],"vmName":"s-1-VM","executeInSequence":false,"
> wait":0}}]
> }
>
> 2018-02-22 09:07:18,132 DEBUG [c.c.a.t.Request]
> (Work-Job-Executor-1:ctx-e231fa0d
> job-15/job-17 ctx-8b47dd7f)
> (logid:2b6e4061) Seq 1-4371306388316487752: Executing: { Cmd , MgmtId:
> 52231347383, via: 1(xenserver_7.3-cs4.11), Ver: v1, Flags: 100011,
> [{"com.cloud.agent.api.StopCommand":{"isProxy":false,
> "checkBeforeCleanup":false,"controlIp":"169.254.0.32","forceStop":false,"
> volumesToDisconnect":[],"vmName":"s-1-VM","executeInSequence":false,"
> wait":0}}]
> }
>
> 2018-02-22 09:07:18,180 DEBUG [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-1:ctx-e231fa0d job-15/job-17 ctx-8b47dd7f)
> (logid:2b6e4061) Successfully released network resources for the vm
> VM[SecondaryStorageVm|s-1-VM]
>
> 2018-02-22 09:07:18,181 DEBUG [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-1:ctx-e231fa0d job-15/job-17 ctx-8b47dd7f)
> (logid:2b6e4061) Successfully cleanued up resources for the vm
> VM[SecondaryStorageVm|s-1-VM] in Starting state
>
> 2018-02-22 09:07:19,067 DEBUG [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-3:ctx-8727633f job-15/job-19 ctx-5416845f)
> (logid:2b6e4061) VM is already stopped: VM[SecondaryStorageVm|s-1-VM]
>
> 2018-02-22 09:07:19,101 DEBUG [c.c.v.VirtualMachineManagerImpl]
> (secstorage-1:ctx-1a4a5425) (logid:a272efa5) Destroying vm
> VM[SecondaryStorageVm|s-1-VM]
>
> 2018-02-22 09:07:19,243 DEBUG [c.c.a.t.Request] (secstorage-1:ctx-1a4a5425)
> (logid:a272efa5) Seq 1-4371306388316487753: Sending { Cmd , MgmtId:
> 52231347383, via: 1(xenserver_7.3-cs4.11), Ver: v1, Flags: 100011,
> [{"org.apache.cloudstack.storage.command.DeleteCommand"
> :{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"
> 6f5dc061-985e-4286-84b2-f8aeab3b2960","volumeType":"
> ROOT","dataStore":{"org.
> apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"
> 2fd0ccd5-81ff-349b-9434-7a697c4aef63","id":1,"
> poolType":"NetworkFilesystem",
> "host":"192.168.117.129","path":"/nfs/primary","port":
> 2049,"url":"NetworkFilesystem://
> 192.168.117.129/nfs/primary/?ROLE=Primary&STOREUUID=
> 2fd0ccd5-81ff-349b-9434-
> 7a697c4aef63","isManaged":false}},"name":"ROOT-1","size"
> :3355443200,"path":"1a60d132-995a-4673-afc6-f28d53390967","
> volumeId":1,"vmName":"s-1-VM","accountId":1,"format":"VHD","
> provisioningType":"THIN","id":1,"deviceId":0,"
> hypervisorType":"XenServer"}},
> "wait":0}}]
> }
>
> 2018-02-22 09:07:19,244 DEBUG [c.c.a.t.Request] (secstorage-1:ctx-1a4a5425)
> (logid:a272efa5) Seq 1-4371306388316487753: Executing: { Cmd , MgmtId:
> 52231347383, via: 1(xenserver_7.3-cs4.11), Ver: v1, Flags: 100011,
> [{"org.apache.cloudstack.storage.command.DeleteCommand"
> :{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"
> 6f5dc061-985e-4286-84b2-f8aeab3b2960","volumeType":"
> ROOT","dataStore":{"org.
> apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"
> 2fd0ccd5-81ff-349b-9434-7a697c4aef63","id":1,"
> poolType":"NetworkFilesystem",
> "host":"192.168.117.129","path":"/nfs/primary","port":
> 2049,"url":"NetworkFilesystem://
> 192.168.117.129/nfs/primary/?ROLE=Primary&STOREUUID=
> 2fd0ccd5-81ff-349b-9434-
> 7a697c4aef63","isManaged":false}},"name":"ROOT-1","size"
> :3355443200,"path":"1a60d132-995a-4673-afc6-f28d53390967","
> volumeId":1,"vmName":"s-1-VM","accountId":1,"format":"VHD","
> provisioningType":"THIN","id":1,"deviceId":0,"
> hypervisorType":"XenServer"}},
> "wait":0}}]
> }
>
> 2018-02-22 09:07:19,853 DEBUG [c.c.v.VirtualMachineManagerImpl]
> (secstorage-1:ctx-1a4a5425) (logid:a272efa5) Expunged
> VM[SecondaryStorageVm|s-1-VM]
>
>
>
>
>
> Do you have any ideas about what could be wrong?
>
>
>
> Thank you,
>
>
>
> Thomas
>
RE: Cloudstack 4.11 : NFS and system VMs NIC issue
Posted by Khosrow Moossavi <km...@cloudops.com>.
Hi Aican
I've seen similar behavior but in our case it was custom 4.10 (somehow
similar to 4.11 in this case) and XenServer 7.1 not 7.3.
Can you check in which mode VMs are started on XenServer? Since it's 7.3
I'm assuming they are brought up in HVM.
If this is the case there's an open PR (#2465) to fix this in 4.11.0.1.
Khosrow Moossavi
CloudOps
On Feb 22, 2018 09:08, "Paul Angus" <pa...@shapeblue.com> wrote:
Hi Aican,
I don’t think that XenServer 7.3 is supported.
Kind regards,
Paul Angus
paul.angus@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
-----Original Message-----
From: Aican France [mailto:aican.fr@gmail.com]
Sent: 22 February 2018 08:31
To: users@cloudstack.apache.org
Subject: Cloudstack 4.11 : NFS and system VMs NIC issue
Hi All,
I’m trying to build a small cloudstack lab inside Vmware Workstation 14.0.
I’m using :
- Ubuntu 14.04.5 NFS Server for primary / secondary storage –
192.168.117.129
- Ubuntu 16.04.3 for Cloudstack management server – 192.168.117.135
- Xenserver 7.3 with latest package 2017.1206 – 192.168.117.136
All these servers hosted on the same LAN (192.168.117.x/24), provisioning
VMs IP range on the same LAN.
I’m facing some issues, but may be they are linked :
- Secondary storage on cloudstack dashboard shows 0.00KB/0.00KB
- Secondary Storage VM s-1-VM and console Proxy VM v-2-VM still
pending il Starting state.
VMs are started (Xencenter View) but when logging into VMs console, NIC are
not configured.
Even if I can see the secondary storage configured in my Infrastructure
dashboard, only one NFS mount point appears on xenserver side :
192.168.117.129:/nfs/primary/ on
/run/sr-mount/e666f671-a903-16b1-7676-ff0ad35f3db4 type nfs
(rw,relatime,vers=3,rsize=262144,wsize=262144,namlen=
255,acdirmin=0,acdirmax=0,soft,proto=tcp,timeo=100,
retrans=12,sec=sys,mountaddr=192.168.117.129,mountvers=3,
mountport=39208,mountproto=tcp,local_lock=none,addr=192.168.117.129)
/dev/mapper/XSLocalEXT--ccb0abd6--d830--bc2e--8372--
daa16a7f5d78-ccb0abd6--d830--bc2e--8372--daa16a7f5d78
on /run/sr-mount/ccb0abd6-d830-bc2e-8372-daa16a7f5d78 type ext3
(rw,relatime,data=ordered)
Management-serv.log shows :
2018-02-22 08:50:06,346 DEBUG [c.c.s.StatsCollector]
(StatsCollector-2:ctx-79cce329) (logid:3cf7734e) There is no secondary
storage VM for secondary storage host nfs://192.168.117.129/nfs/secondary
Some sample logs or the Vms :
2018-02-22 08:51:31,840 DEBUG [c.c.h.x.r.w.x.CitrixStartCommandWrapper]
(DirectAgent-10:ctx-67d15938) (logid:2b6e4061) 1. The VM s-1-VM is in
Starting state.
2018-02-22 08:51:31,874 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-10:ctx-67d15938) (logid:2b6e4061) Created VM
8e0a421d-cc05-baa4-49f2-6e75fab04d5a for s-1-VM
2018-02-22 08:51:31,880 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-10:ctx-67d15938) (logid:2b6e4061) PV args are
%template=domP%type=secstorage%host=192.168.117.135%port=8250%name=s-1-VM%
zone=1%pod=1%guid=s-1-VM%workers=5%resource=org.apache.
cloudstack.storage.resource.NfsSecondaryStorageResource%
instance=SecStorage%sslcopy=false%role=templateProcessor%
mtu=1500%eth2ip=192.168.117.40%eth2mask=255.255.255.0%
gateway=192.168.117.2%eth0ip=169.254.0.32%eth0mask=255.255.
0.0%eth1ip=192.168.117.6%eth1mask=255.255.255.0%mgmtcidr=
192.168.117.0/24%localgw=192.168.117.2%private.network.
device=eth1%internaldns1=192.168.117.2%dns1=8.8.8.8%nfsVersion=null
2018-02-22 08:51:49,208 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-10:ctx-67d15938) (logid:2b6e4061) Creating VIF for s-1-VM on
nic [Nic:Guest-192.168.117.40-vlan://untagged]
2018-02-22 08:51:49,234 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-10:ctx-67d15938) (logid:2b6e4061) Creating VIF for s-1-VM on
nic [Nic:Control-169.254.0.32-null]
2018-02-22 08:51:49,664 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-10:ctx-67d15938) (logid:2b6e4061) Creating VIF for s-1-VM on
nic [Nic:Management-192.168.117.6-null]
2018-02-22 08:52:01,239 INFO [c.c.h.x.r.w.x.CitrixStartCommandWrapper]
(DirectAgent-10:ctx-67d15938) (logid:2b6e4061) Programmed default network
rules for s-1-VM
2018-02-22 08:52:01,239 DEBUG [c.c.h.x.r.w.x.CitrixStartCommandWrapper]
(DirectAgent-10:ctx-67d15938) (logid:2b6e4061) 2. The VM s-1-VM is in
Running state.
2018-02-22 08:54:11,074 DEBUG [c.c.a.t.Request]
(DirectAgentCronJob-3:ctx-1258f3db) (logid:2948b9e7) Seq
1-4371306388316487683: Processing: { Ans: , MgmtId: 52231347383, via:
1(xenserver_7.3-cs4.11), Ver: v1, Flags: 10, [{"com.cloud.agent.api.
ClusterVMMetaDataSyncAnswer":{"_clusterId":1,"_vmMetaDatum":
{"s-1-VM":"device-model:qemu-trad;apic:true;viridian:true;
timeoffset:0;pae:true;acpi:1;hpet:true;nx:true","v-2-VM":"
device-model:qemu-trad;apic:true;viridian:true;timeoffset:
0;pae:true;acpi:1;hpet:true;nx:true"},"_isExecuted":false,
"result":true,"wait":0}}]
}
2018-02-22 08:57:11,054 DEBUG [c.c.a.t.Request]
(DirectAgentCronJob-9:ctx-6ed1ca09) (logid:2948b9e7) Seq
1-4371306388316487683: Processing: { Ans: , MgmtId: 52231347383, via:
1(xenserver_7.3-cs4.11), Ver: v1, Flags: 10, [{"com.cloud.agent.api.
ClusterVMMetaDataSyncAnswer":{"_clusterId":1,"_vmMetaDatum":
{"s-1-VM":"device-model:qemu-trad;apic:true;viridian:true;
timeoffset:0;pae:true;acpi:1;hpet:true;nx:true","v-2-VM":"
device-model:qemu-trad;apic:true;viridian:true;timeoffset:
0;pae:true;acpi:1;hpet:true;nx:true"},"_isExecuted":false,
"result":true,"wait":0}}]
}
2018-02-22 09:00:11,023 DEBUG [c.c.a.t.Request]
(DirectAgentCronJob-10:ctx-49d205af) (logid:2948b9e7) Seq
1-4371306388316487683: Processing: { Ans: , MgmtId: 52231347383, via:
1(xenserver_7.3-cs4.11), Ver: v1, Flags: 10, [{"com.cloud.agent.api.
ClusterVMMetaDataSyncAnswer":{"_clusterId":1,"_vmMetaDatum":
{"s-1-VM":"device-model:qemu-trad;apic:true;viridian:true;
timeoffset:0;pae:true;acpi:1;hpet:true;nx:true","v-2-VM":"
device-model:qemu-trad;apic:true;viridian:true;timeoffset:
0;pae:true;acpi:1;hpet:true;nx:true"},"_isExecuted":false,
"result":true,"wait":0}}]
}
2018-02-22 09:03:11,034 DEBUG [c.c.a.t.Request]
(DirectAgentCronJob-2:ctx-73c4598b) (logid:2948b9e7) Seq
1-4371306388316487683: Processing: { Ans: , MgmtId: 52231347383, via:
1(xenserver_7.3-cs4.11), Ver: v1, Flags: 10, [{"com.cloud.agent.api.
ClusterVMMetaDataSyncAnswer":{"_clusterId":1,"_vmMetaDatum":
{"s-1-VM":"device-model:qemu-trad;apic:true;viridian:true;
timeoffset:0;pae:true;acpi:1;hpet:true;nx:true","v-2-VM":"
device-model:qemu-trad;apic:true;viridian:true;timeoffset:
0;pae:true;acpi:1;hpet:true;nx:true"},"_isExecuted":false,
"result":true,"wait":0}}]
}
2018-02-22 09:06:11,026 DEBUG [c.c.a.t.Request]
(DirectAgentCronJob-9:ctx-271aa0a3) (logid:2948b9e7) Seq
1-4371306388316487683: Processing: { Ans: , MgmtId: 52231347383, via:
1(xenserver_7.3-cs4.11), Ver: v1, Flags: 10, [{"com.cloud.agent.api.
ClusterVMMetaDataSyncAnswer":{"_clusterId":1,"_vmMetaDatum":
{"s-1-VM":"device-model:qemu-trad;apic:true;viridian:true;
timeoffset:0;pae:true;acpi:1;hpet:true;nx:true","v-2-VM":"
device-model:qemu-trad;apic:true;viridian:true;timeoffset:
0;pae:true;acpi:1;hpet:true;nx:true"},"_isExecuted":false,
"result":true,"wait":0}}]
}
2018-02-22 09:06:49,391 WARN [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-10:ctx-67d15938) (logid:2b6e4061) VM.getByNameLabel s-1-V
failed due to org.apache.xmlrpc.XmlRpcException: Failed to read server's
response: connect timed out
2018-02-22 09:06:49,395 DEBUG [c.c.a.t.Request]
(DirectAgent-10:ctx-67d15938) (logid:2b6e4061) Seq 1-4371306388316487701:
Processing: { Ans: , MgmtId: 52231347383, via: 1(xenserver_7.3-cs4.11),
Ver: v1, Flags: 10,
[{"com.cloud.agent.api.StartAnswer":{"vm":{"id":1,"
name":"s-1-VM","bootloader":"PyGrub","type":"SecondaryStorageVm","cpus":1,"
minSpeed":500,"maxSpeed":500,"minRam":536870912,"maxRam":
536870912,"arch":"x86_64","os":"Debian
GNU/Linux 7(64-bit)","bootArgs":" template=domP type=secstorage
host=192.168.117.135 port=8250 name=s-1-VM zone=1 pod=1 guid=s-1-VM
workers=5
resource=org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource
instance=SecStorage sslcopy=false role=templateProcessor mtu=1500
eth2ip=192.168.117.40 eth2mask=255.255.255.0 gateway=192.168.117.2
eth0ip=169.254.0.32 eth0mask=255.255.0.0 eth1ip=192.168.117.6
eth1mask=255.255.255.0 mgmtcidr=192.168.117.0/24 localgw=192.168.117.2
private.network.device=eth1 internaldns1=192.168.117.2 dns1=8.8.8.8
nfsVersion=null","enableHA":false,"limitCpuUse":false,"
enableDynamicallyScaleVm":false,"vncPassword":"-8bjKvvQvmNfG0oBQsw4pw","
params":{},"uuid":"910200d6-546d-4187-a88a-ebc6d8579e50","
disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"
6f5dc061-985e-4286-84b2-f8aeab3b2960","volumeType":"ROOT","dataStore":{"org.
apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"
2fd0ccd5-81ff-349b-9434-7a697c4aef63","id":1,"poolType":"NetworkFilesystem",
"host":"192.168.117.129","path":"/nfs/primary","port":
2049,"url":"NetworkFilesystem://
192.168.117.129/nfs/primary/?ROLE=Primary&STOREUUID=2fd0ccd5-81ff-349b-9434-
7a697c4aef63","isManaged":false}},"name":"ROOT-1","size"
:3355443200,"path":"1a60d132-995a-4673-afc6-f28d53390967","
volumeId":1,"vmName":"s-1-VM","accountId":1,"format":"VHD","
provisioningType":"THIN","id":1,"deviceId":0,"bytesReadRate"
:0,"bytesWriteRate":0,"iopsReadRate":0,"iopsWriteRate":0,"
hypervisorType":"XenServer"}},"diskSeq":0,"path":"1a60d132-
995a-4673-afc6-f28d53390967","type":"ROOT","_details":{"
storageHost":"192.168.117.129","managed":"false","storagePort":"2049","
volumeSize":"3355443200"}}],"nics":[{"deviceId":2,"networkRateMbps":-1,"
defaultNic":true,"pxeDisable":true,"nicUuid":"4aa7df96-2e7a-
4ec7-8725-00ab1a1d1881","details":{"MacAddressChanges":
"true","ForgedTransmits":"true","PromiscuousMode":"
false"},"uuid":"1a92c44c-3d47-412b-b1bd-59362e9252b7","ip":"
192.168.117.40","netmask":"255.255.255.0","gateway":"192.
168.117.2","mac":"1e:00:ea:00:00:11","dns1":"8.8.8.8","
broadcastType":"Native","type":"Guest","broadcastUri":"vlan://untagged","
isSecurityGroupEnabled":true},{"deviceId":0,"networkRateMbps":-1,"
defaultNic":false,"pxeDisable":true,"nicUuid":"24116728-
1b20-4784-96e7-bd788b911aeb","details":{"MacAddressChanges":
"true","ForgedTransmits":"true","PromiscuousMode":"
false"},"uuid":"6e505d4e-1eea-4423-8d19-efd99af3cfad","ip":"
169.254.0.32","netmask":"255.255.0.0","gateway":"169.254.0.
1","mac":"0e:00:a9:fe:00:20","broadcastType":"LinkLocal","type":"Control","
isSecurityGroupEnabled":false},{"deviceId":1,"networkRateMbps":-1,"
defaultNic":false,"pxeDisable":true,"nicUuid":"a21b23ed-
3985-4d2c-8750-3f8bc2f80494","details":{"MacAddressChanges":
"true","ForgedTransmits":"true","PromiscuousMode":"
false"},"uuid":"1788005e-821f-4fbe-856b-f5cde2a094ff","ip":"
192.168.117.6","netmask":"255.255.255.0","gateway":"192.168.
117.2","mac":"1e:00:95:00:00:02","broadcastType":"Native","
type":"Management","isSecurityGroupEnabled":false}],"guestOsDetails":{"
xenserver.dynamicMin":"536870912","xenserver.dynamicMax":"137438953472"}},"
_iqnToData":{},"result":true,"wait":0}},{"com.cloud.agent.
api.check.CheckSshAnswer":{"result":false,"details":"Can
not ping System vm s-1-VMdue to:VM.getByNameLabel s-1-VM failed due to
org.apache.xmlrpc.XmlRpcException: Failed to read server's response:
connect timed out","wait":0}}] }
2018-02-22 09:06:49,410 WARN [o.a.c.s.SecondaryStorageManagerImpl]
(Work-Job-Executor-1:ctx-e231fa0d job-15/job-17 ctx-8b47dd7f)
(logid:2b6e4061) Unable to ssh to the VM: Can not ping System vm s-1-VMdue
to:VM.getByNameLabel s-1-VM failed due to
org.apache.xmlrpc.XmlRpcException: Failed to read server's response:
connect timed out
2018-02-22 09:06:49,411 INFO [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-1:ctx-e231fa0d job-15/job-17 ctx-8b47dd7f)
(logid:2b6e4061) The guru did not like the answers so stopping
VM[SecondaryStorageVm|s-1-VM]
2018-02-22 09:06:49,420 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-1:ctx-e231fa0d
job-15/job-17 ctx-8b47dd7f)
(logid:2b6e4061) Seq 1-4371306388316487748: Sending { Cmd , MgmtId:
52231347383, via: 1(xenserver_7.3-cs4.11), Ver: v1, Flags: 100011,
[{"com.cloud.agent.api.StopCommand":{"isProxy":false,
"checkBeforeCleanup":false,"controlIp":"169.254.0.32","forceStop":false,"
volumesToDisconnect":[],"vmName":"s-1-VM","executeInSequence":false,"
wait":0}}]
}
2018-02-22 09:06:49,421 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-1:ctx-e231fa0d
job-15/job-17 ctx-8b47dd7f)
(logid:2b6e4061) Seq 1-4371306388316487748: Executing: { Cmd , MgmtId:
52231347383, via: 1(xenserver_7.3-cs4.11), Ver: v1, Flags: 100011,
[{"com.cloud.agent.api.StopCommand":{"isProxy":false,
"checkBeforeCleanup":false,"controlIp":"169.254.0.32","forceStop":false,"
volumesToDisconnect":[],"vmName":"s-1-VM","executeInSequence":false,"
wait":0}}]
}
2018-02-22 09:06:54,608 DEBUG [c.c.h.x.r.w.x.CitrixStopCommandWrapper]
(DirectAgent-4:ctx-0cd4fde8) (logid:2b6e4061) 9. The VM s-1-VM is in
Stopping state
2018-02-22 09:06:56,378 INFO [c.c.h.x.r.w.x.CitrixStopCommandWrapper]
(DirectAgent-4:ctx-0cd4fde8) (logid:2b6e4061) Removed network rules for vm
s-1-VM
2018-02-22 09:07:18,120 DEBUG [c.c.h.x.r.w.x.CitrixStopCommandWrapper]
(DirectAgent-4:ctx-0cd4fde8) (logid:2b6e4061) 10. The VM s-1-VM is in
Stopped state
2018-02-22 09:07:18,120 DEBUG [c.c.a.t.Request]
(DirectAgent-4:ctx-0cd4fde8) (logid:2b6e4061) Seq 1-4371306388316487748:
Processing: { Ans: , MgmtId: 52231347383, via: 1(xenserver_7.3-cs4.11),
Ver: v1, Flags: 10,
[{"com.cloud.agent.api.StopAnswer":{"platform":"device-model:qemu-trad;apic:
true;viridian:true;timeoffset:0;pae:true;acpi:1;hpet:true;
nx:true","result":true,"details":"Stop
VM s-1-V Succeed","wait":0}}] }
2018-02-22 09:07:18,121 DEBUG [c.c.a.m.AgentManagerImpl]
(Work-Job-Executor-1:ctx-e231fa0d job-15/job-17 ctx-8b47dd7f)
(logid:2b6e4061) Details from executing class
com.cloud.agent.api.StopCommand: Stop VM s-1-VM Succeed
2018-02-22 09:07:18,121 ERROR [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-1:ctx-e231fa0d job-15/job-17 ctx-8b47dd7f)
(logid:2b6e4061) Failed to start instance VM[SecondaryStorageVm|s-1-VM]
2018-02-22 09:07:18,126 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-1:ctx-e231fa0d job-15/job-17 ctx-8b47dd7f)
(logid:2b6e4061) Cleaning up resources for the vm
VM[SecondaryStorageVm|s-1-VM] in Starting state
2018-02-22 09:07:18,132 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-1:ctx-e231fa0d
job-15/job-17 ctx-8b47dd7f)
(logid:2b6e4061) Seq 1-4371306388316487752: Sending { Cmd , MgmtId:
52231347383, via: 1(xenserver_7.3-cs4.11), Ver: v1, Flags: 100011,
[{"com.cloud.agent.api.StopCommand":{"isProxy":false,
"checkBeforeCleanup":false,"controlIp":"169.254.0.32","forceStop":false,"
volumesToDisconnect":[],"vmName":"s-1-VM","executeInSequence":false,"
wait":0}}]
}
2018-02-22 09:07:18,132 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-1:ctx-e231fa0d
job-15/job-17 ctx-8b47dd7f)
(logid:2b6e4061) Seq 1-4371306388316487752: Executing: { Cmd , MgmtId:
52231347383, via: 1(xenserver_7.3-cs4.11), Ver: v1, Flags: 100011,
[{"com.cloud.agent.api.StopCommand":{"isProxy":false,
"checkBeforeCleanup":false,"controlIp":"169.254.0.32","forceStop":false,"
volumesToDisconnect":[],"vmName":"s-1-VM","executeInSequence":false,"
wait":0}}]
}
2018-02-22 09:07:18,180 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-1:ctx-e231fa0d job-15/job-17 ctx-8b47dd7f)
(logid:2b6e4061) Successfully released network resources for the vm
VM[SecondaryStorageVm|s-1-VM]
2018-02-22 09:07:18,181 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-1:ctx-e231fa0d job-15/job-17 ctx-8b47dd7f)
(logid:2b6e4061) Successfully cleanued up resources for the vm
VM[SecondaryStorageVm|s-1-VM] in Starting state
2018-02-22 09:07:19,067 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-3:ctx-8727633f job-15/job-19 ctx-5416845f)
(logid:2b6e4061) VM is already stopped: VM[SecondaryStorageVm|s-1-VM]
2018-02-22 09:07:19,101 DEBUG [c.c.v.VirtualMachineManagerImpl]
(secstorage-1:ctx-1a4a5425) (logid:a272efa5) Destroying vm
VM[SecondaryStorageVm|s-1-VM]
2018-02-22 09:07:19,243 DEBUG [c.c.a.t.Request] (secstorage-1:ctx-1a4a5425)
(logid:a272efa5) Seq 1-4371306388316487753: Sending { Cmd , MgmtId:
52231347383, via: 1(xenserver_7.3-cs4.11), Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.DeleteCommand"
:{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"
6f5dc061-985e-4286-84b2-f8aeab3b2960","volumeType":"ROOT","dataStore":{"org.
apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"
2fd0ccd5-81ff-349b-9434-7a697c4aef63","id":1,"poolType":"NetworkFilesystem",
"host":"192.168.117.129","path":"/nfs/primary","port":
2049,"url":"NetworkFilesystem://
192.168.117.129/nfs/primary/?ROLE=Primary&STOREUUID=2fd0ccd5-81ff-349b-9434-
7a697c4aef63","isManaged":false}},"name":"ROOT-1","size"
:3355443200,"path":"1a60d132-995a-4673-afc6-f28d53390967","
volumeId":1,"vmName":"s-1-VM","accountId":1,"format":"VHD","
provisioningType":"THIN","id":1,"deviceId":0,"hypervisorType":"XenServer"}},
"wait":0}}]
}
2018-02-22 09:07:19,244 DEBUG [c.c.a.t.Request] (secstorage-1:ctx-1a4a5425)
(logid:a272efa5) Seq 1-4371306388316487753: Executing: { Cmd , MgmtId:
52231347383, via: 1(xenserver_7.3-cs4.11), Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.DeleteCommand"
:{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"
6f5dc061-985e-4286-84b2-f8aeab3b2960","volumeType":"ROOT","dataStore":{"org.
apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"
2fd0ccd5-81ff-349b-9434-7a697c4aef63","id":1,"poolType":"NetworkFilesystem",
"host":"192.168.117.129","path":"/nfs/primary","port":
2049,"url":"NetworkFilesystem://
192.168.117.129/nfs/primary/?ROLE=Primary&STOREUUID=2fd0ccd5-81ff-349b-9434-
7a697c4aef63","isManaged":false}},"name":"ROOT-1","size"
:3355443200,"path":"1a60d132-995a-4673-afc6-f28d53390967","
volumeId":1,"vmName":"s-1-VM","accountId":1,"format":"VHD","
provisioningType":"THIN","id":1,"deviceId":0,"hypervisorType":"XenServer"}},
"wait":0}}]
}
2018-02-22 09:07:19,853 DEBUG [c.c.v.VirtualMachineManagerImpl]
(secstorage-1:ctx-1a4a5425) (logid:a272efa5) Expunged
VM[SecondaryStorageVm|s-1-VM]
Do you have any ideas about what could be wrong?
Thank you,
Thomas
RE: Cloudstack 4.11 : NFS and system VMs NIC issue
Posted by Paul Angus <pa...@shapeblue.com>.
Hi Aican,
I don’t think that XenServer 7.3 is supported.
Kind regards,
Paul Angus
paul.angus@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
-----Original Message-----
From: Aican France [mailto:aican.fr@gmail.com]
Sent: 22 February 2018 08:31
To: users@cloudstack.apache.org
Subject: Cloudstack 4.11 : NFS and system VMs NIC issue
Hi All,
I’m trying to build a small cloudstack lab inside Vmware Workstation 14.0.
I’m using :
- Ubuntu 14.04.5 NFS Server for primary / secondary storage –
192.168.117.129
- Ubuntu 16.04.3 for Cloudstack management server – 192.168.117.135
- Xenserver 7.3 with latest package 2017.1206 – 192.168.117.136
All these servers hosted on the same LAN (192.168.117.x/24), provisioning VMs IP range on the same LAN.
I’m facing some issues, but may be they are linked :
- Secondary storage on cloudstack dashboard shows 0.00KB/0.00KB
- Secondary Storage VM s-1-VM and console Proxy VM v-2-VM still
pending il Starting state.
VMs are started (Xencenter View) but when logging into VMs console, NIC are not configured.
Even if I can see the secondary storage configured in my Infrastructure dashboard, only one NFS mount point appears on xenserver side :
192.168.117.129:/nfs/primary/ on
/run/sr-mount/e666f671-a903-16b1-7676-ff0ad35f3db4 type nfs
(rw,relatime,vers=3,rsize=262144,wsize=262144,namlen=255,acdirmin=0,acdirmax=0,soft,proto=tcp,timeo=100,retrans=12,sec=sys,mountaddr=192.168.117.129,mountvers=3,mountport=39208,mountproto=tcp,local_lock=none,addr=192.168.117.129)
/dev/mapper/XSLocalEXT--ccb0abd6--d830--bc2e--8372--daa16a7f5d78-ccb0abd6--d830--bc2e--8372--daa16a7f5d78
on /run/sr-mount/ccb0abd6-d830-bc2e-8372-daa16a7f5d78 type ext3
(rw,relatime,data=ordered)
Management-serv.log shows :
2018-02-22 08:50:06,346 DEBUG [c.c.s.StatsCollector]
(StatsCollector-2:ctx-79cce329) (logid:3cf7734e) There is no secondary storage VM for secondary storage host nfs://192.168.117.129/nfs/secondary
Some sample logs or the Vms :
2018-02-22 08:51:31,840 DEBUG [c.c.h.x.r.w.x.CitrixStartCommandWrapper]
(DirectAgent-10:ctx-67d15938) (logid:2b6e4061) 1. The VM s-1-VM is in Starting state.
2018-02-22 08:51:31,874 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-10:ctx-67d15938) (logid:2b6e4061) Created VM 8e0a421d-cc05-baa4-49f2-6e75fab04d5a for s-1-VM
2018-02-22 08:51:31,880 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-10:ctx-67d15938) (logid:2b6e4061) PV args are %template=domP%type=secstorage%host=192.168.117.135%port=8250%name=s-1-VM%zone=1%pod=1%guid=s-1-VM%workers=5%resource=org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource%instance=SecStorage%sslcopy=false%role=templateProcessor%mtu=1500%eth2ip=192.168.117.40%eth2mask=255.255.255.0%gateway=192.168.117.2%eth0ip=169.254.0.32%eth0mask=255.255.0.0%eth1ip=192.168.117.6%eth1mask=255.255.255.0%mgmtcidr=
192.168.117.0/24%localgw=192.168.117.2%private.network.device=eth1%internaldns1=192.168.117.2%dns1=8.8.8.8%nfsVersion=null
2018-02-22 08:51:49,208 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-10:ctx-67d15938) (logid:2b6e4061) Creating VIF for s-1-VM on nic [Nic:Guest-192.168.117.40-vlan://untagged]
2018-02-22 08:51:49,234 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-10:ctx-67d15938) (logid:2b6e4061) Creating VIF for s-1-VM on nic [Nic:Control-169.254.0.32-null]
2018-02-22 08:51:49,664 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-10:ctx-67d15938) (logid:2b6e4061) Creating VIF for s-1-VM on nic [Nic:Management-192.168.117.6-null]
2018-02-22 08:52:01,239 INFO [c.c.h.x.r.w.x.CitrixStartCommandWrapper]
(DirectAgent-10:ctx-67d15938) (logid:2b6e4061) Programmed default network rules for s-1-VM
2018-02-22 08:52:01,239 DEBUG [c.c.h.x.r.w.x.CitrixStartCommandWrapper]
(DirectAgent-10:ctx-67d15938) (logid:2b6e4061) 2. The VM s-1-VM is in Running state.
2018-02-22 08:54:11,074 DEBUG [c.c.a.t.Request]
(DirectAgentCronJob-3:ctx-1258f3db) (logid:2948b9e7) Seq
1-4371306388316487683: Processing: { Ans: , MgmtId: 52231347383, via:
1(xenserver_7.3-cs4.11), Ver: v1, Flags: 10, [{"com.cloud.agent.api.ClusterVMMetaDataSyncAnswer":{"_clusterId":1,"_vmMetaDatum":{"s-1-VM":"device-model:qemu-trad;apic:true;viridian:true;timeoffset:0;pae:true;acpi:1;hpet:true;nx:true","v-2-VM":"device-model:qemu-trad;apic:true;viridian:true;timeoffset:0;pae:true;acpi:1;hpet:true;nx:true"},"_isExecuted":false,"result":true,"wait":0}}]
}
2018-02-22 08:57:11,054 DEBUG [c.c.a.t.Request]
(DirectAgentCronJob-9:ctx-6ed1ca09) (logid:2948b9e7) Seq
1-4371306388316487683: Processing: { Ans: , MgmtId: 52231347383, via:
1(xenserver_7.3-cs4.11), Ver: v1, Flags: 10, [{"com.cloud.agent.api.ClusterVMMetaDataSyncAnswer":{"_clusterId":1,"_vmMetaDatum":{"s-1-VM":"device-model:qemu-trad;apic:true;viridian:true;timeoffset:0;pae:true;acpi:1;hpet:true;nx:true","v-2-VM":"device-model:qemu-trad;apic:true;viridian:true;timeoffset:0;pae:true;acpi:1;hpet:true;nx:true"},"_isExecuted":false,"result":true,"wait":0}}]
}
2018-02-22 09:00:11,023 DEBUG [c.c.a.t.Request]
(DirectAgentCronJob-10:ctx-49d205af) (logid:2948b9e7) Seq
1-4371306388316487683: Processing: { Ans: , MgmtId: 52231347383, via:
1(xenserver_7.3-cs4.11), Ver: v1, Flags: 10, [{"com.cloud.agent.api.ClusterVMMetaDataSyncAnswer":{"_clusterId":1,"_vmMetaDatum":{"s-1-VM":"device-model:qemu-trad;apic:true;viridian:true;timeoffset:0;pae:true;acpi:1;hpet:true;nx:true","v-2-VM":"device-model:qemu-trad;apic:true;viridian:true;timeoffset:0;pae:true;acpi:1;hpet:true;nx:true"},"_isExecuted":false,"result":true,"wait":0}}]
}
2018-02-22 09:03:11,034 DEBUG [c.c.a.t.Request]
(DirectAgentCronJob-2:ctx-73c4598b) (logid:2948b9e7) Seq
1-4371306388316487683: Processing: { Ans: , MgmtId: 52231347383, via:
1(xenserver_7.3-cs4.11), Ver: v1, Flags: 10, [{"com.cloud.agent.api.ClusterVMMetaDataSyncAnswer":{"_clusterId":1,"_vmMetaDatum":{"s-1-VM":"device-model:qemu-trad;apic:true;viridian:true;timeoffset:0;pae:true;acpi:1;hpet:true;nx:true","v-2-VM":"device-model:qemu-trad;apic:true;viridian:true;timeoffset:0;pae:true;acpi:1;hpet:true;nx:true"},"_isExecuted":false,"result":true,"wait":0}}]
}
2018-02-22 09:06:11,026 DEBUG [c.c.a.t.Request]
(DirectAgentCronJob-9:ctx-271aa0a3) (logid:2948b9e7) Seq
1-4371306388316487683: Processing: { Ans: , MgmtId: 52231347383, via:
1(xenserver_7.3-cs4.11), Ver: v1, Flags: 10, [{"com.cloud.agent.api.ClusterVMMetaDataSyncAnswer":{"_clusterId":1,"_vmMetaDatum":{"s-1-VM":"device-model:qemu-trad;apic:true;viridian:true;timeoffset:0;pae:true;acpi:1;hpet:true;nx:true","v-2-VM":"device-model:qemu-trad;apic:true;viridian:true;timeoffset:0;pae:true;acpi:1;hpet:true;nx:true"},"_isExecuted":false,"result":true,"wait":0}}]
}
2018-02-22 09:06:49,391 WARN [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-10:ctx-67d15938) (logid:2b6e4061) VM.getByNameLabel s-1-V failed due to org.apache.xmlrpc.XmlRpcException: Failed to read server's
response: connect timed out
2018-02-22 09:06:49,395 DEBUG [c.c.a.t.Request]
(DirectAgent-10:ctx-67d15938) (logid:2b6e4061) Seq 1-4371306388316487701:
Processing: { Ans: , MgmtId: 52231347383, via: 1(xenserver_7.3-cs4.11),
Ver: v1, Flags: 10,
[{"com.cloud.agent.api.StartAnswer":{"vm":{"id":1,"name":"s-1-VM","bootloader":"PyGrub","type":"SecondaryStorageVm","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":536870912,"maxRam":536870912,"arch":"x86_64","os":"Debian
GNU/Linux 7(64-bit)","bootArgs":" template=domP type=secstorage
host=192.168.117.135 port=8250 name=s-1-VM zone=1 pod=1 guid=s-1-VM
workers=5
resource=org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource
instance=SecStorage sslcopy=false role=templateProcessor mtu=1500
eth2ip=192.168.117.40 eth2mask=255.255.255.0 gateway=192.168.117.2
eth0ip=169.254.0.32 eth0mask=255.255.0.0 eth1ip=192.168.117.6
eth1mask=255.255.255.0 mgmtcidr=192.168.117.0/24 localgw=192.168.117.2
private.network.device=eth1 internaldns1=192.168.117.2 dns1=8.8.8.8 nfsVersion=null","enableHA":false,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"-8bjKvvQvmNfG0oBQsw4pw","params":{},"uuid":"910200d6-546d-4187-a88a-ebc6d8579e50","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"6f5dc061-985e-4286-84b2-f8aeab3b2960","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"2fd0ccd5-81ff-349b-9434-7a697c4aef63","id":1,"poolType":"NetworkFilesystem","host":"192.168.117.129","path":"/nfs/primary","port":2049,"url":"NetworkFilesystem://
192.168.117.129/nfs/primary/?ROLE=Primary&STOREUUID=2fd0ccd5-81ff-349b-9434-7a697c4aef63","isManaged":false}},"name":"ROOT-1","size":3355443200,"path":"1a60d132-995a-4673-afc6-f28d53390967","volumeId":1,"vmName":"s-1-VM","accountId":1,"format":"VHD","provisioningType":"THIN","id":1,"deviceId":0,"bytesReadRate":0,"bytesWriteRate":0,"iopsReadRate":0,"iopsWriteRate":0,"hypervisorType":"XenServer"}},"diskSeq":0,"path":"1a60d132-995a-4673-afc6-f28d53390967","type":"ROOT","_details":{"storageHost":"192.168.117.129","managed":"false","storagePort":"2049","volumeSize":"3355443200"}}],"nics":[{"deviceId":2,"networkRateMbps":-1,"defaultNic":true,"pxeDisable":true,"nicUuid":"4aa7df96-2e7a-4ec7-8725-00ab1a1d1881","details":{"MacAddressChanges":"true","ForgedTransmits":"true","PromiscuousMode":"false"},"uuid":"1a92c44c-3d47-412b-b1bd-59362e9252b7","ip":"192.168.117.40","netmask":"255.255.255.0","gateway":"192.168.117.2","mac":"1e:00:ea:00:00:11","dns1":"8.8.8.8","broadcastType":"Native","type":"Guest","broadcastUri":"vlan://untagged","isSecurityGroupEnabled":true},{"deviceId":0,"networkRateMbps":-1,"defaultNic":false,"pxeDisable":true,"nicUuid":"24116728-1b20-4784-96e7-bd788b911aeb","details":{"MacAddressChanges":"true","ForgedTransmits":"true","PromiscuousMode":"false"},"uuid":"6e505d4e-1eea-4423-8d19-efd99af3cfad","ip":"169.254.0.32","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:00:20","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false},{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"pxeDisable":true,"nicUuid":"a21b23ed-3985-4d2c-8750-3f8bc2f80494","details":{"MacAddressChanges":"true","ForgedTransmits":"true","PromiscuousMode":"false"},"uuid":"1788005e-821f-4fbe-856b-f5cde2a094ff","ip":"192.168.117.6","netmask":"255.255.255.0","gateway":"192.168.117.2","mac":"1e:00:95:00:00:02","broadcastType":"Native","type":"Management","isSecurityGroupEnabled":false}],"guestOsDetails":{"xenserver.dynamicMin":"536870912","xenserver.dynamicMax":"137438953472"}},"_iqnToData":{},"result":true,"wait":0}},{"com.cloud.agent.api.check.CheckSshAnswer":{"result":false,"details":"Can
not ping System vm s-1-VMdue to:VM.getByNameLabel s-1-VM failed due to
org.apache.xmlrpc.XmlRpcException: Failed to read server's response:
connect timed out","wait":0}}] }
2018-02-22 09:06:49,410 WARN [o.a.c.s.SecondaryStorageManagerImpl]
(Work-Job-Executor-1:ctx-e231fa0d job-15/job-17 ctx-8b47dd7f)
(logid:2b6e4061) Unable to ssh to the VM: Can not ping System vm s-1-VMdue to:VM.getByNameLabel s-1-VM failed due to
org.apache.xmlrpc.XmlRpcException: Failed to read server's response:
connect timed out
2018-02-22 09:06:49,411 INFO [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-1:ctx-e231fa0d job-15/job-17 ctx-8b47dd7f)
(logid:2b6e4061) The guru did not like the answers so stopping VM[SecondaryStorageVm|s-1-VM]
2018-02-22 09:06:49,420 DEBUG [c.c.a.t.Request] (Work-Job-Executor-1:ctx-e231fa0d job-15/job-17 ctx-8b47dd7f)
(logid:2b6e4061) Seq 1-4371306388316487748: Sending { Cmd , MgmtId:
52231347383, via: 1(xenserver_7.3-cs4.11), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"checkBeforeCleanup":false,"controlIp":"169.254.0.32","forceStop":false,"volumesToDisconnect":[],"vmName":"s-1-VM","executeInSequence":false,"wait":0}}]
}
2018-02-22 09:06:49,421 DEBUG [c.c.a.t.Request] (Work-Job-Executor-1:ctx-e231fa0d job-15/job-17 ctx-8b47dd7f)
(logid:2b6e4061) Seq 1-4371306388316487748: Executing: { Cmd , MgmtId:
52231347383, via: 1(xenserver_7.3-cs4.11), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"checkBeforeCleanup":false,"controlIp":"169.254.0.32","forceStop":false,"volumesToDisconnect":[],"vmName":"s-1-VM","executeInSequence":false,"wait":0}}]
}
2018-02-22 09:06:54,608 DEBUG [c.c.h.x.r.w.x.CitrixStopCommandWrapper]
(DirectAgent-4:ctx-0cd4fde8) (logid:2b6e4061) 9. The VM s-1-VM is in Stopping state
2018-02-22 09:06:56,378 INFO [c.c.h.x.r.w.x.CitrixStopCommandWrapper]
(DirectAgent-4:ctx-0cd4fde8) (logid:2b6e4061) Removed network rules for vm s-1-VM
2018-02-22 09:07:18,120 DEBUG [c.c.h.x.r.w.x.CitrixStopCommandWrapper]
(DirectAgent-4:ctx-0cd4fde8) (logid:2b6e4061) 10. The VM s-1-VM is in Stopped state
2018-02-22 09:07:18,120 DEBUG [c.c.a.t.Request]
(DirectAgent-4:ctx-0cd4fde8) (logid:2b6e4061) Seq 1-4371306388316487748:
Processing: { Ans: , MgmtId: 52231347383, via: 1(xenserver_7.3-cs4.11),
Ver: v1, Flags: 10,
[{"com.cloud.agent.api.StopAnswer":{"platform":"device-model:qemu-trad;apic:true;viridian:true;timeoffset:0;pae:true;acpi:1;hpet:true;nx:true","result":true,"details":"Stop
VM s-1-V Succeed","wait":0}}] }
2018-02-22 09:07:18,121 DEBUG [c.c.a.m.AgentManagerImpl] (Work-Job-Executor-1:ctx-e231fa0d job-15/job-17 ctx-8b47dd7f)
(logid:2b6e4061) Details from executing class
com.cloud.agent.api.StopCommand: Stop VM s-1-VM Succeed
2018-02-22 09:07:18,121 ERROR [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-1:ctx-e231fa0d job-15/job-17 ctx-8b47dd7f)
(logid:2b6e4061) Failed to start instance VM[SecondaryStorageVm|s-1-VM]
2018-02-22 09:07:18,126 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-1:ctx-e231fa0d job-15/job-17 ctx-8b47dd7f)
(logid:2b6e4061) Cleaning up resources for the vm VM[SecondaryStorageVm|s-1-VM] in Starting state
2018-02-22 09:07:18,132 DEBUG [c.c.a.t.Request] (Work-Job-Executor-1:ctx-e231fa0d job-15/job-17 ctx-8b47dd7f)
(logid:2b6e4061) Seq 1-4371306388316487752: Sending { Cmd , MgmtId:
52231347383, via: 1(xenserver_7.3-cs4.11), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"checkBeforeCleanup":false,"controlIp":"169.254.0.32","forceStop":false,"volumesToDisconnect":[],"vmName":"s-1-VM","executeInSequence":false,"wait":0}}]
}
2018-02-22 09:07:18,132 DEBUG [c.c.a.t.Request] (Work-Job-Executor-1:ctx-e231fa0d job-15/job-17 ctx-8b47dd7f)
(logid:2b6e4061) Seq 1-4371306388316487752: Executing: { Cmd , MgmtId:
52231347383, via: 1(xenserver_7.3-cs4.11), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"checkBeforeCleanup":false,"controlIp":"169.254.0.32","forceStop":false,"volumesToDisconnect":[],"vmName":"s-1-VM","executeInSequence":false,"wait":0}}]
}
2018-02-22 09:07:18,180 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-1:ctx-e231fa0d job-15/job-17 ctx-8b47dd7f)
(logid:2b6e4061) Successfully released network resources for the vm VM[SecondaryStorageVm|s-1-VM]
2018-02-22 09:07:18,181 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-1:ctx-e231fa0d job-15/job-17 ctx-8b47dd7f)
(logid:2b6e4061) Successfully cleanued up resources for the vm VM[SecondaryStorageVm|s-1-VM] in Starting state
2018-02-22 09:07:19,067 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-3:ctx-8727633f job-15/job-19 ctx-5416845f)
(logid:2b6e4061) VM is already stopped: VM[SecondaryStorageVm|s-1-VM]
2018-02-22 09:07:19,101 DEBUG [c.c.v.VirtualMachineManagerImpl]
(secstorage-1:ctx-1a4a5425) (logid:a272efa5) Destroying vm VM[SecondaryStorageVm|s-1-VM]
2018-02-22 09:07:19,243 DEBUG [c.c.a.t.Request] (secstorage-1:ctx-1a4a5425)
(logid:a272efa5) Seq 1-4371306388316487753: Sending { Cmd , MgmtId:
52231347383, via: 1(xenserver_7.3-cs4.11), Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"6f5dc061-985e-4286-84b2-f8aeab3b2960","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"2fd0ccd5-81ff-349b-9434-7a697c4aef63","id":1,"poolType":"NetworkFilesystem","host":"192.168.117.129","path":"/nfs/primary","port":2049,"url":"NetworkFilesystem://
192.168.117.129/nfs/primary/?ROLE=Primary&STOREUUID=2fd0ccd5-81ff-349b-9434-7a697c4aef63","isManaged":false}},"name":"ROOT-1","size":3355443200,"path":"1a60d132-995a-4673-afc6-f28d53390967","volumeId":1,"vmName":"s-1-VM","accountId":1,"format":"VHD","provisioningType":"THIN","id":1,"deviceId":0,"hypervisorType":"XenServer"}},"wait":0}}]
}
2018-02-22 09:07:19,244 DEBUG [c.c.a.t.Request] (secstorage-1:ctx-1a4a5425)
(logid:a272efa5) Seq 1-4371306388316487753: Executing: { Cmd , MgmtId:
52231347383, via: 1(xenserver_7.3-cs4.11), Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"6f5dc061-985e-4286-84b2-f8aeab3b2960","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"2fd0ccd5-81ff-349b-9434-7a697c4aef63","id":1,"poolType":"NetworkFilesystem","host":"192.168.117.129","path":"/nfs/primary","port":2049,"url":"NetworkFilesystem://
192.168.117.129/nfs/primary/?ROLE=Primary&STOREUUID=2fd0ccd5-81ff-349b-9434-7a697c4aef63","isManaged":false}},"name":"ROOT-1","size":3355443200,"path":"1a60d132-995a-4673-afc6-f28d53390967","volumeId":1,"vmName":"s-1-VM","accountId":1,"format":"VHD","provisioningType":"THIN","id":1,"deviceId":0,"hypervisorType":"XenServer"}},"wait":0}}]
}
2018-02-22 09:07:19,853 DEBUG [c.c.v.VirtualMachineManagerImpl]
(secstorage-1:ctx-1a4a5425) (logid:a272efa5) Expunged VM[SecondaryStorageVm|s-1-VM]
Do you have any ideas about what could be wrong?
Thank you,
Thomas