You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Tamas Monos <ta...@veber.co.uk> on 2012/06/14 13:58:21 UTC
Permanent snapshot problem bound to a specific volume
Hi,
Last night I tried to create a snapshot manually from a volume.
The mangement server was performing the operation unfortunately got power cycled through the process :(
This caused preventing me to do any further snapshots as the system said backup is already in progress on that volume. Please try again later.
After hours it was still stuck saying the same (while vmware was not doing anything at all) so I went to the 'snapshots' table in the DB and marked its state from 'BackingUP' to 'Error'.
This allowed me to create snapshots again and started to export the OVF template just fine (seen the progress via vcenter). After about an hour the export failed around 15%.
Since then I cannot create any snapshots of that VM while can make from others with no problems.
The log:
2012-06-14 10:42:19,918 WARN [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-8:job-1063) Cleanup failed due to BackupSnapshotCommand exception: java.lang.Exception: Unable to finish the whole process to package as a OVA file
2012-06-14 10:42:19,918 DEBUG [storage.snapshot.SnapshotManagerImpl] (Job-Executor-8:job-1063) The result for com.cloud.agent.api.BackupSnapshotCommand is BackupSnapshotCommand exception: java.lang.Exception: Unable to finish the whole process to package as a OVA file
2012-06-14 10:42:19,919 WARN [cloud.vm.VirtualMachineManagerImpl] (AgentManager-Handler-3:null) Cleanup failed due to BackupSnapshotCommand exception: java.lang.Exception: Unable to finish the whole process to package as a OVA file
2012-06-14 10:44:39,289 DEBUG [agent.transport.Request] (AgentManager-Handler-2:null) Seq 10-577440210: Processing: { Ans: , MgmtId: 345049202747, via: 10, Ver: v1, Flags: 10, [{"BackupSnapshotAnswer":{"full":true,"result":false,"details":"BackupSnapshotCommand exception: java.lang.ArrayIndexOutOfBoundsException: 0\ncom.cloud.hypervisor.vmware.util.VmwareHelper.prepareDiskDevice(VmwareHelper.java:220)\ncom.cloud.hypervisor.vmware.mo.VirtualMachineMO.cloneFromDiskChain(VirtualMachineMO.java:1474)\ncom.cloud.hypervisor.vmware.mo.VirtualMachineMO.cloneFromCurrentSnapshot(VirtualMachineMO.java:1450)\ncom.cloud.hypervisor.vmware.manager.VmwareStorageManagerImpl.exportVolumeToSecondaryStroage(VmwareStorageManagerImpl.java:747)\ncom.cloud.hypervisor.vmware.manager.VmwareStorageManagerImpl.backupSnapshotToSecondaryStorage(VmwareStorageManagerImpl.java:714)\ncom.cloud.hypervisor.vmware.manager.VmwareStorageManagerImpl.execute(VmwareStorageManagerImpl.java:217)\ncom.cloud.storage.resource.VmwareSecondaryStorageResourceHandler.execute(VmwareSecondaryStorageResourceHandler.java:109)\ncom.cloud.storage.resource.VmwareSecondaryStorageResourceHandler.executeRequest(VmwareSecondaryStorageResourceHandler.java:67)\ncom.cloud.storage.resource.PremiumSecondaryStorageResource.executeRequest(PremiumSecondaryStorageResource.java:50)\ncom.cloud.agent.Agent.processRequest(Agent.java:517)\ncom.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:827)\ncom.cloud.utils.nio.Task.run(Task.java:79)\njava.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)\njava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)\njava.lang.Thread.run(Thread.java:662)\n","wait":0}}] }
2012-06-14 10:44:39,289 DEBUG [agent.transport.Request] (Job-Executor-8:job-1063) Seq 10-577440210: Received: { Ans: , MgmtId: 345049202747, via: 10, Ver: v1, Flags: 10, { BackupSnapshotAnswer } }
2012-06-14 10:44:39,323 WARN [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-8:job-1063) Cleanup failed due to BackupSnapshotCommand exception: java.lang.ArrayIndexOutOfBoundsException: 0
2012-06-14 10:44:39,324 DEBUG [storage.snapshot.SnapshotManagerImpl] (Job-Executor-8:job-1063) The result for com.cloud.agent.api.BackupSnapshotCommand is BackupSnapshotCommand exception: java.lang.ArrayIndexOutOfBoundsException: 0
2012-06-14 10:44:39,324 WARN [cloud.vm.VirtualMachineManagerImpl] (AgentManager-Handler-2:null) Cleanup failed due to BackupSnapshotCommand exception: java.lang.ArrayIndexOutOfBoundsException: 0
2012-06-14 10:46:39,325 DEBUG [storage.snapshot.SnapshotManagerImpl] (Job-Executor-8:job-1063) Retrying...
2012-06-14 10:46:39,329 WARN [storage.snapshot.SnapshotManagerImpl] (Job-Executor-8:job-1063) Storage unavailable
com.cloud.exception.StorageUnavailableException: Resource [StoragePool:201] is unreachable: Unable to send command to the pool 201 due to there is no enabled hosts up in this cluster
2012-06-14 10:46:39,330 WARN [storage.snapshot.SnapshotManagerImpl] (Job-Executor-8:job-1063) Failed to back up snapshot on secondary storage, deleting the record from the DB
2012-06-14 10:46:39,377 ERROR [cloud.api.ApiDispatcher] (Job-Executor-8:job-1063) Exception while executing CreateSnapshotCmd:
com.cloud.utils.exception.CloudRuntimeException: Created snapshot: com.cloud.storage.SnapshotVO$$EnhancerByCGLIB$$aaba1dbd@44374434<ma...@44374434> on primary but failed to backup on secondary
At the end it says Storage unavailable... I can parallel create other snapshots with no problem so the storage is there.
That java.lang.ArrayIndexOutOfBoundsException is quite scary though.
Any advice or magic could sort this problem would be appreciated.
Regards
Tamas Monos DDI +44(0)2034687012
Chief Technical Office +44(0)2034687000
Veber: The Hosting Specialists Fax +44(0)871 522 7057
http://www.veber.co.uk<http://www.veber.co.uk/>
Follow us on Twitter: www.twitter.com/veberhost<http://www.twitter.com/veberhost>
Follow us on Facebook: www.facebook.com/veberhost<http://www.facebook.com/veberhost>
RE: Permanent snapshot problem bound to a specific volume
Posted by Tamas Monos <ta...@veber.co.uk>.
Hi,
Any suggestions how to attach a debugger to a running CS management server to find out what is causing the java.lang.ArrayIndexOutOfBoundsException?
Regards
Tamas Monos DDI +44(0)2034687012
Chief Technical Office +44(0)2034687000
Veber: The Hosting Specialists Fax +44(0)871 522 7057
http://www.veber.co.uk
Follow us on Twitter: www.twitter.com/veberhost
Follow us on Facebook: www.facebook.com/veberhost
-----Original Message-----
From: Tamas Monos [mailto:tamasm@veber.co.uk]
Sent: 14 June 2012 12:58
To: cloudstack-dev@incubator.apache.org
Subject: Permanent snapshot problem bound to a specific volume
Hi,
Last night I tried to create a snapshot manually from a volume.
The mangement server was performing the operation unfortunately got power cycled through the process :( This caused preventing me to do any further snapshots as the system said backup is already in progress on that volume. Please try again later.
After hours it was still stuck saying the same (while vmware was not doing anything at all) so I went to the 'snapshots' table in the DB and marked its state from 'BackingUP' to 'Error'.
This allowed me to create snapshots again and started to export the OVF template just fine (seen the progress via vcenter). After about an hour the export failed around 15%.
Since then I cannot create any snapshots of that VM while can make from others with no problems.
The log:
2012-06-14 10:42:19,918 WARN [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-8:job-1063) Cleanup failed due to BackupSnapshotCommand exception: java.lang.Exception: Unable to finish the whole process to package as a OVA file
2012-06-14 10:42:19,918 DEBUG [storage.snapshot.SnapshotManagerImpl] (Job-Executor-8:job-1063) The result for com.cloud.agent.api.BackupSnapshotCommand is BackupSnapshotCommand exception: java.lang.Exception: Unable to finish the whole process to package as a OVA file
2012-06-14 10:42:19,919 WARN [cloud.vm.VirtualMachineManagerImpl] (AgentManager-Handler-3:null) Cleanup failed due to BackupSnapshotCommand exception: java.lang.Exception: Unable to finish the whole process to package as a OVA file
2012-06-14 10:44:39,289 DEBUG [agent.transport.Request] (AgentManager-Handler-2:null) Seq 10-577440210: Processing: { Ans: , MgmtId: 345049202747, via: 10, Ver: v1, Flags: 10, [{"BackupSnapshotAnswer":{"full":true,"result":false,"details":"BackupSnapshotCommand exception: java.lang.ArrayIndexOutOfBoundsException: 0\ncom.cloud.hypervisor.vmware.util.VmwareHelper.prepareDiskDevice(VmwareHelper.java:220)\ncom.cloud.hypervisor.vmware.mo.VirtualMachineMO.cloneFromDiskChain(VirtualMachineMO.java:1474)\ncom.cloud.hypervisor.vmware.mo.VirtualMachineMO.cloneFromCurrentSnapshot(VirtualMachineMO.java:1450)\ncom.cloud.hypervisor.vmware.manager.VmwareStorageManagerImpl.exportVolumeToSecondaryStroage(VmwareStorageManagerImpl.java:747)\ncom.cloud.hypervisor.vmware.manager.VmwareStorageManagerImpl.backupSnapshotToSecondaryStorage(VmwareStorageManagerImpl.java:714)\ncom.cloud.hypervisor.vmware.manager.VmwareStorageManagerImpl.execute(VmwareStorageManagerImpl.java:217)\ncom.cloud.storage.resource.VmwareSecondaryStorageResourceHandler.execute(VmwareSecondaryStorageResourceHandler.java:109)\ncom.cloud.storage.resource.VmwareSecondaryStorageResourceHandler.executeRequest(VmwareSecondaryStorageResourceHandler.java:67)\ncom.cloud.storage.resource.PremiumSecondaryStorageResource.executeRequest(PremiumSecondaryStorageResource.java:50)\ncom.cloud.agent.Agent.processRequest(Agent.java:517)\ncom.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:827)\ncom.cloud.utils.nio.Task.run(Task.java:79)\njava.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)\njava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)\njava.lang.Thread.run(Thread.java:662)\n","wait":0}}] }
2012-06-14 10:44:39,289 DEBUG [agent.transport.Request] (Job-Executor-8:job-1063) Seq 10-577440210: Received: { Ans: , MgmtId: 345049202747, via: 10, Ver: v1, Flags: 10, { BackupSnapshotAnswer } }
2012-06-14 10:44:39,323 WARN [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-8:job-1063) Cleanup failed due to BackupSnapshotCommand exception: java.lang.ArrayIndexOutOfBoundsException: 0
2012-06-14 10:44:39,324 DEBUG [storage.snapshot.SnapshotManagerImpl] (Job-Executor-8:job-1063) The result for com.cloud.agent.api.BackupSnapshotCommand is BackupSnapshotCommand exception: java.lang.ArrayIndexOutOfBoundsException: 0
2012-06-14 10:44:39,324 WARN [cloud.vm.VirtualMachineManagerImpl] (AgentManager-Handler-2:null) Cleanup failed due to BackupSnapshotCommand exception: java.lang.ArrayIndexOutOfBoundsException: 0
2012-06-14 10:46:39,325 DEBUG [storage.snapshot.SnapshotManagerImpl] (Job-Executor-8:job-1063) Retrying...
2012-06-14 10:46:39,329 WARN [storage.snapshot.SnapshotManagerImpl] (Job-Executor-8:job-1063) Storage unavailable
com.cloud.exception.StorageUnavailableException: Resource [StoragePool:201] is unreachable: Unable to send command to the pool 201 due to there is no enabled hosts up in this cluster
2012-06-14 10:46:39,330 WARN [storage.snapshot.SnapshotManagerImpl] (Job-Executor-8:job-1063) Failed to back up snapshot on secondary storage, deleting the record from the DB
2012-06-14 10:46:39,377 ERROR [cloud.api.ApiDispatcher] (Job-Executor-8:job-1063) Exception while executing CreateSnapshotCmd:
com.cloud.utils.exception.CloudRuntimeException: Created snapshot: com.cloud.storage.SnapshotVO$$EnhancerByCGLIB$$aaba1dbd@44374434<ma...@44374434> on primary but failed to backup on secondary
At the end it says Storage unavailable... I can parallel create other snapshots with no problem so the storage is there.
That java.lang.ArrayIndexOutOfBoundsException is quite scary though.
Any advice or magic could sort this problem would be appreciated.
Regards
Tamas Monos DDI +44(0)2034687012
Chief Technical Office +44(0)2034687000
Veber: The Hosting Specialists Fax +44(0)871 522 7057
http://www.veber.co.uk<http://www.veber.co.uk/>
Follow us on Twitter: www.twitter.com/veberhost<http://www.twitter.com/veberhost>
Follow us on Facebook: www.facebook.com/veberhost<http://www.facebook.com/veberhost>