[Solve] Re: Issue with CS 4.1 on CentOS/KVM with iSCSI/CLVM: lost volume group

[Milamber <mi...@apache.org>]

Le 13/04/2013 23:37, Milamber a ecrit :
>
>
> Le 13/04/2013 23:29, Milamber a ecrit :
>> Hello,
>>
>> I've a issue with a CloudStack 4.1 installation on CentOS 6.4 with 
>> the hypervisor KVM.
>>
>> The primary storage is CLVM on iSCSI array (Dell Equallogic) on 3 
>> nodes (hosts) with 1 Cloudmgr server.
>>
>> I've installed and configure a CLVM with locking_type = 3 and 
>> wait_for_locks = 30. Cman like the slides: 
>> http://fr.slideshare.net/MarcusLSorensen/cloud-stack-clvm.
>> I have a volume group share over iscsi on each node.
>>
>> The issue come when I've 2 ou 3 nodes up. When I try to create a new 
>> instance (from ISO or Template), somes times (> 70%) the creation 
>> failed. (That works some times.
>> (==> no problem when I only have one active node, I can create a lot 
>> of VM simultaneous without issue)

It's solve.

My volume group have been create *without* the cluster mark (option 
--clustered in vgcreate)

I made a "vgchange --clustered vg_baie" on one node, puts in maintenance 
mode all nodes and Primary storage, stop clvm, restart cman, start clvm, 
disable maintenance modes. And works!

Milamber


>>
>> After some log inspection, I've found that the volume is "disappear" 
>> from the node (which create the VM).
>>
>> In libvirtd log, we have when it's failed:
>> 2013-04-13 20:41:56.812+0000: 28373: error : 
>> virStorageBackendVolOpenCheckMode:1023 : cannot stat file 
>> '/dev/vg_baie/23761696-cca7-4d09-a65f-fa9d88fbd987': No such file or 
>> directory
>> 2013-04-13 20:41:56.841+0000: 28373: error : virCommandWait:2345 : 
>> internal error Child process (/sbin/vgchange -aln vg_baie) unexpected 
>> exit status 5:   Can't deactivate volume group "vg_baie" with 16 open 
>> logical volume(s)
>>
>
> Please note: the volume group is correctly displays with a virsh 
> pool-list command before the creation VM's attempt.
>
>
>> (a virsh pool-list show that the vg has disappeared, I must restart 
>> cloudstack-agent for enabled it. Note: the pool isn't with the 
>> autostart option)
>>
>> On management.log, we have:
>>
>> 2013-04-13 22:58:02,022 DEBUG [cloud.storage.StorageManagerImpl] 
>> (Job-Executor-38:job-228) Creating volume: Vol[68|vm=68|ROOT]
>> 2013-04-13 22:58:02,022 DEBUG [cloud.storage.StorageManagerImpl] 
>> (Job-Executor-38:job-228) Trying to create in Pool[200|CLVM]
>> 2013-04-13 22:58:02,027 DEBUG [agent.transport.Request] 
>> (Job-Executor-38:job-228) Seq 1-1321467941: Sending  { Cmd , MgmtId: 
>> 130177108533, via: 1, Ver: v1, Flags: 100111, 
>> [{"storage.CreateCommand":{"volId":68,"pool":{"id":200,"uuid":"3425857b-3c3c-4e2a-8eba-115cefbe982b","host":"localhost","path":"/vg_baie","port":0,"type":"CLVM"},"diskCharacteristics":{"size":107374182400,"tags":[],"type":"ROOT","name":"ROOT-68","useLocalStorage":false,"recreatable":false,"diskOfferingId":5,"volumeId":68,"hyperType":"KVM"},"wait":0}}] 
>> }
>> 2013-04-13 22:58:02,077 DEBUG [agent.transport.Request] 
>> (AgentManager-Handler-2:null) Seq 1-1321467941: Processing:  { Ans: , 
>> MgmtId: 130177108533, via: 1, Ver: v1, Flags: 110, 
>> [{"storage.CreateAnswer":{"requestTemplateReload":false,"result":false,"details":"Exception: 
>> com.cloud.utils.exception.CloudRuntimeException\nMessage: 
>> org.libvirt.LibvirtException: Storage pool not found: no pool with 
>> matching uuid\nStack: 
>> com.cloud.utils.exception.CloudRuntimeException: 
>> org.libvirt.LibvirtException: Storage pool not found: no pool with 
>> matching uuid\n\tat 
>> com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.getStoragePool(LibvirtStorageAdaptor.java:427)\n\tat 
>> com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getStoragePool(KVMStoragePoolManager.java:71)\n\tat 
>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:1272)\n\tat 
>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1073)\n\tat 
>> com.cloud.agent.Agent.processRequest(Agent.java:525)\n\tat 
>> com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:852)\n\tat com.cloud.utils.nio.Task.run(Task.java:83)\n\tat 
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)\n\tat 
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n\tat 
>> java.lang.Thread.run(Thread.java:679)\n","wait":0}}] }
>> 2013-04-13 22:58:02,077 DEBUG [agent.manager.AgentAttache] 
>> (AgentManager-Handler-2:null) Seq 1-1321467941: No more commands found
>> 2013-04-13 22:58:02,077 DEBUG [agent.transport.Request] 
>> (Job-Executor-38:job-228) Seq 1-1321467941: Received:  { Ans: , 
>> MgmtId: 130177108533, via: 1, Ver: v1, Flags: 110, { CreateAnswer } }
>> 2013-04-13 22:58:02,077 DEBUG [cloud.storage.StorageManagerImpl] 
>> (Job-Executor-38:job-228) Unable to create volume Vol[68|vm=68|ROOT]
>> 2013-04-13 22:58:02,079 INFO  [cloud.vm.VirtualMachineManagerImpl] 
>> (Job-Executor-38:job-228) Unable to contact resource.
>> com.cloud.exception.StorageUnavailableException: Resource 
>> [StoragePool:200] is unreachable: Unable to create Vol[68|vm=68|ROOT]
>>         at 
>> com.cloud.storage.StorageManagerImpl.prepare(StorageManagerImpl.java:3483)
>>         at 
>> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:748)
>>         at 
>> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:471)
>>         at 
>> org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:212)
>>         at 
>> org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:209)
>>         at 
>> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3860)
>>         at 
>> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3458)
>>         at 
>> com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3444)
>>         at 
>> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>>         at 
>> org.apache.cloudstack.api.command.user.vm.DeployVMCmd.execute(DeployVMCmd.java:379)
>>         at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:162)
>>         at 
>> com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:437)
>>         at 
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>         at 
>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>>         at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>>         at 
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
>>         at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>         at java.lang.Thread.run(Thread.java:679)
>> 2013-04-13 22:58:02,081 DEBUG [cloud.vm.VirtualMachineManagerImpl] 
>> (Job-Executor-38:job-228) Cleaning up resources for the vm 
>> VM[User|Centos100go] in Starting state
>> 2013-04-13 22:58:02,083 DEBUG [agent.transport.Request] 
>> (Job-Executor-38:job-228) Seq 1-1321467942: Sending  { Cmd , MgmtId: 
>> 130177108533, via: 1, Ver: v1, Flags: 100111, 
>> [{"StopCommand":{"isProxy":false,"vmName":"i-2-68-VM","wait":0}}] }
>> (and after a message: 
>> com.cloud.exception.InsufficientServerCapacityException: Unable to 
>> create a deployment for VM[User...)
>>
>>
>> Perhaps that is a configuration issue on CLVM ? or iSCSI attach?
>>
>> I'm not sure if CloudStack support the iSCSI/CLVM with KVM 
>> hypervisor? yes/no.
>>
>> I must change for a NFS primary storage (over iSCSI) ? or a 
>> SharedMountPoint (GFS?).
>>
>> That is very annoying for me, please have some ideas?
>>
>> Thanks.
>>
>> Milamber
>>
>>
>
>