System VM's - Starting

[Jeremy Peterson <jp...@acentek.net>]

I added the NFS for secondary storage this morning and I was expecting the system vm's to be UP,but at this time they are in a Starting state.

I've pasted my /var/log/cloudstack/management/management-server.log to pastebin

http://pastebin.com/UXJEbcat

My cloudstack setup is.

Two physical hosts Flex-H1.flexhost.local and Flex-H2.flexhost.local
I am using CLVM as my Primary with the help of http://www.slideshare.net/MarcusLSorensen/cloud-stack-clvm
I am using a NFS as secondary of my Management server for now that isn't my final goal.

I am looking at my log and the things that jump out are the java runtime exception.

2014-09-30 08:48:09,581 DEBUG [c.c.a.t.Request] (StatsCollector-2:ctx-34ed0e9d) Seq 1-792595064: Received:  { Ans: , MgmtId: 345050433729, via: 1, Ver: v1, Flags: 10, { GetHostStatsAnswer } }
2014-09-30 08:48:10,223 DEBUG [c.c.a.t.Request] (StatsCollector-2:ctx-34ed0e9d) Seq 2-2053047772: Received:  { Ans: , MgmtId: 345050433729, via: 2, Ver: v1, Flags: 10, { GetHostStatsAnswer } }
2014-09-30 08:48:26,436 DEBUG [c.c.a.t.Request] (AgentManager-Handler-13:null) Seq 1-792595056: Processing:  { Ans: , MgmtId: 345050433729, via: 1, Ver: v1, Flags: 10, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"com.cloud.utils.exception.CloudRuntimeException: org.libvirt.LibvirtException: internal error Child process (/bin/mount Flex-CSMS.flexhost.local:/mnt/secondary/template/tmpl/1/3 /mnt/7a4f0f30-05b1-3f2f-9acf-b5827be546c0) unexpected exit status 32: mount.nfs: Connection timed out\n","wait":0}}] }
2014-09-30 08:48:26,437 DEBUG [c.c.a.t.Request] (secstorage-1:ctx-9fb0b338) Seq 1-792595056: Received:  { Ans: , MgmtId: 345050433729, via: 1, Ver: v1, Flags: 10, { CopyCmdAnswer } }
2014-09-30 08:48:26,438 ERROR [c.c.v.VirtualMachineManagerImpl] (secstorage-1:ctx-9fb0b338) Failed to start instance VM[SecondaryStorageVm|s-35-VM]
java.lang.RuntimeException: InvocationTargetException when invoking RPC callback for command: createVolumeFromBaseImageCallBack

So I looked into my NFS shares.

[root@flex-csms mnt]# mount -t nfs Flex-H1.flexhost.local:/export/secondary /mnt/secondary
mount.nfs: /mnt/secondary is busy or already mounted
[root@flex-csms mnt]# ls
secondary  template
[root@flex-csms mnt]# cd secondary/
[root@flex-csms secondary]# ls
[root@flex-csms secondary]# touch me
[root@flex-csms secondary]# ls
me
[root@Flex-H1 export]# cd secondary/
[root@Flex-H1 secondary]# ls
me

As you see the mount is mounted and I can touch me and then if I go over to h1 if I ls the secondary directory the me file shows up.

Flex-H1:/export/secondary
                      232G  3.4G  217G   2% /mnt/secondary

And when I check df -h  I see the storage is mounted.

Thank you.