You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@cloudstack.apache.org by "Thomas O'Dowd (JIRA)" <ji...@apache.org> on 2013/07/29 10:15:48 UTC
[jira] [Reopened] (CLOUDSTACK-3229) Object_Store_Refactor -
Snapshot fails due to an internal error
[ https://issues.apache.org/jira/browse/CLOUDSTACK-3229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas O'Dowd reopened CLOUDSTACK-3229:
---------------------------------------
I've finally got a chance to test this again today. Unfortunately, I still am not able to create a snapshot.
INFO [user.snapshot.CreateSnapshotCmd] (Job-Executor-2:job-9 = [ eedf4da9-2eba-49b8-b17d-bff66af8125f ]) VOLSS: createSnapshotCmd starts:1375084903140
WARN [xen.resource.CitrixResourceBase] (DirectAgent-45:) Task failed! Task record: uuid: f121c4fa-dab7-ba96-b5e0-70520d03e49d
nameLabel: Async.host.call_plugin
nameDescription:
allowedOperations: []
currentOperations: {}
created: Mon Jul 29 17:03:28 JST 2013
finished: Mon Jul 29 17:03:28 JST 2013
status: failure
residentOn: com.xensource.xenapi.Host@3ba61fa8
progress: 1.0
type: <none/>
result:
errorInfo: [XENAPI_MISSING_PLUGIN, s3xen]
otherConfig: {}
subtaskOf: com.xensource.xenapi.Task@aaf13f6f
subtasks: []
WARN [xen.resource.CitrixResourceBase] (DirectAgent-45:) callHostPlugin failed for cmd: s3 with args operation: put, accessKey: 00ba9a7f9a8142b070c3, secretKey: pe5sLXnndBnsOHiMT/IhbJW995kAy1/+HK9+14Uc, endPoint: s3.cloudian.com:18080, connectionTimeout: 60000, socketTimeout: 60000, directory: /var/run/sr-mount/6462814c-e367-4035-0711-93b5f8578f83, iSCSIFlag: false, filename: 48db220b-ca8a-4d20-ae82-ce88fcd840a0.vhd, maxErrorRetry: 5, key: snapshots/48db220b-ca8a-4d20-ae82-ce88fcd840a0, due to Task failed! Task record: uuid: f121c4fa-dab7-ba96-b5e0-70520d03e49d
nameLabel: Async.host.call_plugin
nameDescription:
allowedOperations: []
currentOperations: {}
created: Mon Jul 29 17:03:28 JST 2013
finished: Mon Jul 29 17:03:28 JST 2013
status: failure
residentOn: com.xensource.xenapi.Host@3ba61fa8
progress: 1.0
type: <none/>
result:
errorInfo: [XENAPI_MISSING_PLUGIN, s3xen]
otherConfig: {}
subtaskOf: com.xensource.xenapi.Task@aaf13f6f
subtasks: []
Task failed! Task record: uuid: f121c4fa-dab7-ba96-b5e0-70520d03e49d
nameLabel: Async.host.call_plugin
nameDescription:
allowedOperations: []
currentOperations: {}
created: Mon Jul 29 17:03:28 JST 2013
finished: Mon Jul 29 17:03:28 JST 2013
status: failure
residentOn: com.xensource.xenapi.Host@3ba61fa8
progress: 1.0
type: <none/>
result:
errorInfo: [XENAPI_MISSING_PLUGIN, s3xen]
otherConfig: {}
subtaskOf: com.xensource.xenapi.Task@aaf13f6f
subtasks: []
at com.cloud.hypervisor.xen.resource.CitrixResourceBase.checkForSuccess(CitrixResourceBase.java:3637)
at com.cloud.hypervisor.xen.resource.CitrixResourceBase.callHostPluginAsync(CitrixResourceBase.java:3951)
at com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.backupSnapshotToS3(XenServerStorageProcessor.java:1133)
at com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.backupSnapshot(XenServerStorageProcessor.java:1314)
at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:84)
at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:49)
at com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:623)
at com.cloud.hypervisor.xen.resource.XcpOssResource.executeRequest(XcpOssResource.java:140)
at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
WARN [xen.resource.CitrixResourceBase] (DirectAgent-45:) callHostPlugin failed for cmd: deleteSnapshotBackup with args secondaryStorageMountPath: 192.168.56.10:/opt/storage/cache/, localMountPoint: /var/run/cloud_mount/49ab0a32-3113-3ee2-bb24-3e65acb96ce9, path: snapshots/2/3, backupUUID: 48db220b-ca8a-4d20-ae82-ce88fcd840a0.vhd, due to There was a failure communicating with the plugin.
WARN [xen.resource.XenServerStorageProcessor] (DirectAgent-45:) BackupSnapshot Failed due to callHostPlugin failed for cmd: deleteSnapshotBackup with args secondaryStorageMountPath: 192.168.56.10:/opt/storage/cache/, localMountPoint: /var/run/cloud_mount/49ab0a32-3113-3ee2-bb24-3e65acb96ce9, path: snapshots/2/3, backupUUID: 48db220b-ca8a-4d20-ae82-ce88fcd840a0.vhd, due to There was a failure communicating with the plugin.
com.cloud.utils.exception.CloudRuntimeException: callHostPlugin failed for cmd: deleteSnapshotBackup with args secondaryStorageMountPath: 192.168.56.10:/opt/storage/cache/, localMountPoint: /var/run/cloud_mount/49ab0a32-3113-3ee2-bb24-3e65acb96ce9, path: snapshots/2/3, backupUUID: 48db220b-ca8a-4d20-ae82-ce88fcd840a0.vhd, due to There was a failure communicating with the plugin.
at com.cloud.hypervisor.xen.resource.CitrixResourceBase.callHostPlugin(CitrixResourceBase.java:4231)
at com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.deleteSnapshotBackup(XenServerStorageProcessor.java:1050)
at com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.backupSnapshot(XenServerStorageProcessor.java:1317)
at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:84)
at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:49)
at com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:623)
at com.cloud.hypervisor.xen.resource.XcpOssResource.executeRequest(XcpOssResource.java:140)
at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
> Object_Store_Refactor - Snapshot fails due to an internal error
> ---------------------------------------------------------------
>
> Key: CLOUDSTACK-3229
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-3229
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the default.)
> Affects Versions: 4.2.0
> Environment: chrome on linux
> devcloud
> Cloudian or Amazon S3 Object store
> Reporter: Thomas O'Dowd
> Assignee: Min Chen
> Priority: Critical
>
> Assuming initial devcloud state...
> I added a cache for the S3 storage like this.
> on devcloud machine as root:
> # mkdir /opt/storage/cache
> # vi /etc/exports (and append this line)
> /opt/storage/cache *(rw,no_subtree_check,no_root_squash,fsid=9999)
> # exportfs -a
> On Mgmt server GUI:
> 1. navigate to infrastructure -> secondary storage
> 2. delete the NFS SS.
> 3. add S3 storage for Cloudian (I used 60000 as the timeouts - assuming millis). I used the /opt/storage/cache thing as the s3 cache.
> 4. nav to templates
> 5. register a new template (I uploaded tinyLinux again as "mytiny" (5.3 64bit)).
> 6. confirm with s3cmd that 2 objects are now on S3.
> --------- s3 objects -------
> template/tmpl/1/1/routing-1/acton-systemvm-02062012.vhd.bz2 2013-06-27T03:01:46.203Z None 140616708 "b533e7b65219439ee7fca0146ddd7ffa-27"
> template/tmpl/2/201/201-2-ae9e9409-4c8e-3ad8-a62f-abec7a35fe26/tinylinux.vhd 2013-06-27T03:04:06.730Z None 50430464 "4afac316e865adf74ca1a8039fae7399-10"
> --------- s3 objects -------
> 7. I restarted the management server at this point which actually resulted in another object on S3.
> --------- the new s3 object -------
> template/tmpl/1/5/tiny Linux/ttylinux_pv.vhd 2013-06-27T03:43:26.494Z None 50430464 "4afac316e865adf74ca1a8039fae7399-10"
> --------- the new s3 object -------
> 8. Go to instance and create a new choosing the "mytiny" template which we registered.
> 9. launch it after selecting all defaults.
> 10. wait until it starts.
> 11. nav to storage. I see ROOT-8. Click on this to open.
> 12. click the camera to take the snapshot.
> after a pause I get a popup
> "Failed to create snapshot due to an internal error creating snapshot for volume 8"
> Also on the mgmt terminal I get the following log entry (only 1):
> INFO [user.snapshot.CreateSnapshotCmd] (Job-Executor-8:job-16) VOLSS: createSnapshotCmd starts:1372321251009
> If I check the "view snapshots" button under storage, I can however see the snapshot. It says its on primary. I'm expecting it to go to secondary storage though. Nothing is in the S3 logs and no snapshots.
> If I try to delete that snapshot from here I get this error in the logs:
> ERROR [cloud.async.AsyncJobManagerImpl] (Job-Executor-12:job-20) Unexpected exception while executing org.apache.cloudstack.api.command.user.snapshot.DeleteSnapshotCmd
> com.cloud.utils.exception.CloudRuntimeException: Failed to delete snapshot:com.cloud.exception.InvalidParameterValueException: Can't delete snapshotshot 4 due to it is not in BackedUp Status
> at com.cloud.storage.snapshot.SnapshotManagerImpl.deleteSnapshot(SnapshotManagerImpl.java:513)
> at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at org.apache.cloudstack.api.command.user.snapshot.DeleteSnapshotCmd.execute(DeleteSnapshotCmd.java:96)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:155)
> at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:437)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> If I navigate to instance, my instance, and try to take a vm snapshot from here, I get a different pop-up which says:
> "There is other active volume snapshot tasks on the instance to which the volume is attached, please try again later"
> And I get an exception:
> ERROR [cloud.api.ApiServer] (352129314@qtp-2110413789-32:) unhandled exception executing api command: createVMSnapshot
> com.cloud.utils.exception.CloudRuntimeException: There is other active volume snapshot tasks on the instance to which the volume is attached, please try again later.
> at com.cloud.vm.snapshot.VMSnapshotManagerImpl.allocVMSnapshot(VMSnapshotManagerImpl.java:299)
> at org.apache.cloudstack.api.command.user.vmsnapshot.CreateVMSnapshotCmd.create(CreateVMSnapshotCmd.java:78)
> at com.cloud.api.ApiDispatcher.dispatchCreateCmd(ApiDispatcher.java:101)
> at com.cloud.api.ApiServer.queueCommand(ApiServer.java:475)
> at com.cloud.api.ApiServer.handleRequest(ApiServer.java:371)
> at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:304)
> at com.cloud.api.ApiServlet.doGet(ApiServlet.java:66)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
> at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
> at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
> at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:326)
> at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
> at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
> at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
> at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> There are no requests going to the S3 storage for the snap-shotting that I can see and its the only secondary storage that I have setup.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira