You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by "deepti dohare (JIRA)" <ji...@apache.org> on 2012/10/16 07:57:03 UTC
[jira] [Created] (CLOUDSTACK-356) snapshot errors with multiple
secondary storage in one zone
deepti dohare created CLOUDSTACK-356:
----------------------------------------
Summary: snapshot errors with multiple secondary storage in one zone
Key: CLOUDSTACK-356
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-356
Project: CloudStack
Issue Type: Bug
Components: Snapshot
Affects Versions: pre-4.0.0
Reporter: deepti dohare
Fix For: 4.1.0
BRIEF SYNOPSIS:
========================
Snapshots are failing inder some cases where the host has already mounted other
secondary storage and snapshot command has been issued to it
ENVIRONMENT
====================
CS 2.2.14 + XEN server
DETAILED DESCRIPTION
====================
1) Noticcing following error for some snapshots
2012-07-23 02:16:38,120 WARN [xen.resource.CitrixResourceBase]
(DirectAgent-281:null) Task failed! Task record: uuid:
f0642eda-6e40-8bca-1ffb-bbf52a0e307e
status: FAILURE
errorInfo: [XENAPI_PLUGIN_FAILURE, backupSnapshot, SROSError, Error reporting
error, unknown key File
/var/run/cloud_mount/1/snapshots/35/2521/a05404cd-e2c7-41cf-9972-aa019ab79494.vhd
does not exist.]
2) Cluster has investigated the issue and they found that mount points are not
identical for the xenservers to the secondary storages across a cluster (df) :
* host 1 : 10.16.36.64:/RAID10-DATA01/secsatxen002/snapshots 2147483648
240044032 1907439616 12% /var/run/cloud_mount/1/snapshots
* Host 2 : 10.16.36.64:/RAID10-DATA01/secsatxen001/snapshots 2147483648
569533440 1577950208 27% /var/run/cloud_mount/1/snapshots
Therefore, the contents are not identical in the 2 secondary storages (this is
right to my understanding, because multiple secondary storage permit to scale
horizontally) :
* Host 1 :
* -rw-r--r--+ 1 nobody nobody 307M 2012-07-23 16:11
07ba46c7-0908-4402-b20a-5ddd29a65d17.vhd
* -rw-r--r--+ 1 nobody nobody 301M 2012-07-18 02:18
1c378e6a-0eba-48d9-8545-ab9fa09b9ba0.vhd
* -rw-r--r--+ 1 nobody nobody 59M 2012-07-19 02:18
3b7e8c19-ba9c-433e-ac18-f52d4ad42be9.vhd
* -rw-r--r--+ 1 nobody nobody 97M 2012-07-20 02:19
4644362a-764d-4c66-a038-168127007a5f.vhd
* -rw-r--r--+ 1 nobody nobody 47M 2012-07-23 02:18
72e54c85-8245-4b84-b4b5-5a1ef2b44e52.vhd
* -rw-r--r--+ 1 nobody nobody 69M 2012-07-21 02:18
76301f19-fcc0-43ac-adab-cde2cf23bd87.vhd
* -rw-r--r--+ 1 nobody nobody 57M 2012-07-22 02:18
a05404cd-e2c7-41cf-9972-aa019ab79494.vhd
* Host 2 : empty
A simple workaround is to migrate the corresponding instance to the host with
the right mount. After further checking, this behavior is encountered on all
clusters, only rarely because only applying to instances that :
* Have a scheduled snapshot retention policy
* Have either migrated, been restarted on a different host than previous run
Though workaround is feasible but it is difficult to predict when the user
seems this probelm.
please check if there is any problem in cloudstack logic to ensure right mount
secondary mount point is available and create it if it doesnt exist
If this behaviour confirms to be a bug,please file one bug in jira.
REPRO STEPS
==================
You can try reproducing the scenario by havign two secondary storages and
continuously taking snapshots
EXPECTED BEHAVIOR
==================
Cloudstack should create correct mount of secondary storage if it doesnt exist.
It should never use other secondary storage mount point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CLOUDSTACK-356) snapshot errors with multiple
secondary storage in one zone
Posted by "deepti dohare (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CLOUDSTACK-356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476773#comment-13476773 ]
deepti dohare commented on CLOUDSTACK-356:
------------------------------------------
Submitted a patch on Review Board: https://reviews.apache.org/r/7594/
> snapshot errors with multiple secondary storage in one zone
> -----------------------------------------------------------
>
> Key: CLOUDSTACK-356
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-356
> Project: CloudStack
> Issue Type: Bug
> Components: Snapshot
> Affects Versions: pre-4.0.0
> Reporter: deepti dohare
> Assignee: deepti dohare
> Fix For: 4.1.0
>
>
> BRIEF SYNOPSIS:
> ========================
> Snapshots are failing inder some cases where the host has already mounted other
> secondary storage and snapshot command has been issued to it
> ENVIRONMENT
> ====================
> CS 2.2.14 + XEN server
> DETAILED DESCRIPTION
> ====================
> 1) Noticcing following error for some snapshots
> 2012-07-23 02:16:38,120 WARN [xen.resource.CitrixResourceBase]
> (DirectAgent-281:null) Task failed! Task record: uuid:
> f0642eda-6e40-8bca-1ffb-bbf52a0e307e
> status: FAILURE
> errorInfo: [XENAPI_PLUGIN_FAILURE, backupSnapshot, SROSError, Error reporting
> error, unknown key File
> /var/run/cloud_mount/1/snapshots/35/2521/a05404cd-e2c7-41cf-9972-aa019ab79494.vhd
> does not exist.]
> 2) Cluster has investigated the issue and they found that mount points are not
> identical for the xenservers to the secondary storages across a cluster (df) :
> * host 1 : 10.16.36.64:/RAID10-DATA01/secsatxen002/snapshots 2147483648
> 240044032 1907439616 12% /var/run/cloud_mount/1/snapshots
> * Host 2 : 10.16.36.64:/RAID10-DATA01/secsatxen001/snapshots 2147483648
> 569533440 1577950208 27% /var/run/cloud_mount/1/snapshots
> Therefore, the contents are not identical in the 2 secondary storages (this is
> right to my understanding, because multiple secondary storage permit to scale
> horizontally) :
> * Host 1 :
> * -rw-r--r--+ 1 nobody nobody 307M 2012-07-23 16:11
> 07ba46c7-0908-4402-b20a-5ddd29a65d17.vhd
> * -rw-r--r--+ 1 nobody nobody 301M 2012-07-18 02:18
> 1c378e6a-0eba-48d9-8545-ab9fa09b9ba0.vhd
> * -rw-r--r--+ 1 nobody nobody 59M 2012-07-19 02:18
> 3b7e8c19-ba9c-433e-ac18-f52d4ad42be9.vhd
> * -rw-r--r--+ 1 nobody nobody 97M 2012-07-20 02:19
> 4644362a-764d-4c66-a038-168127007a5f.vhd
> * -rw-r--r--+ 1 nobody nobody 47M 2012-07-23 02:18
> 72e54c85-8245-4b84-b4b5-5a1ef2b44e52.vhd
> * -rw-r--r--+ 1 nobody nobody 69M 2012-07-21 02:18
> 76301f19-fcc0-43ac-adab-cde2cf23bd87.vhd
> * -rw-r--r--+ 1 nobody nobody 57M 2012-07-22 02:18
> a05404cd-e2c7-41cf-9972-aa019ab79494.vhd
> * Host 2 : empty
> A simple workaround is to migrate the corresponding instance to the host with
> the right mount. After further checking, this behavior is encountered on all
> clusters, only rarely because only applying to instances that :
> * Have a scheduled snapshot retention policy
> * Have either migrated, been restarted on a different host than previous run
> Though workaround is feasible but it is difficult to predict when the user
> seems this probelm.
> please check if there is any problem in cloudstack logic to ensure right mount
> secondary mount point is available and create it if it doesnt exist
> If this behaviour confirms to be a bug,please file one bug in jira.
> REPRO STEPS
> ==================
> You can try reproducing the scenario by havign two secondary storages and
> continuously taking snapshots
> EXPECTED BEHAVIOR
> ==================
> Cloudstack should create correct mount of secondary storage if it doesnt exist.
> It should never use other secondary storage mount point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (CLOUDSTACK-356) snapshot errors with multiple
secondary storage in one zone
Posted by "deepti dohare (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CLOUDSTACK-356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
deepti dohare reassigned CLOUDSTACK-356:
----------------------------------------
Assignee: deepti dohare
> snapshot errors with multiple secondary storage in one zone
> -----------------------------------------------------------
>
> Key: CLOUDSTACK-356
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-356
> Project: CloudStack
> Issue Type: Bug
> Components: Snapshot
> Affects Versions: pre-4.0.0
> Reporter: deepti dohare
> Assignee: deepti dohare
> Fix For: 4.1.0
>
>
> BRIEF SYNOPSIS:
> ========================
> Snapshots are failing inder some cases where the host has already mounted other
> secondary storage and snapshot command has been issued to it
> ENVIRONMENT
> ====================
> CS 2.2.14 + XEN server
> DETAILED DESCRIPTION
> ====================
> 1) Noticcing following error for some snapshots
> 2012-07-23 02:16:38,120 WARN [xen.resource.CitrixResourceBase]
> (DirectAgent-281:null) Task failed! Task record: uuid:
> f0642eda-6e40-8bca-1ffb-bbf52a0e307e
> status: FAILURE
> errorInfo: [XENAPI_PLUGIN_FAILURE, backupSnapshot, SROSError, Error reporting
> error, unknown key File
> /var/run/cloud_mount/1/snapshots/35/2521/a05404cd-e2c7-41cf-9972-aa019ab79494.vhd
> does not exist.]
> 2) Cluster has investigated the issue and they found that mount points are not
> identical for the xenservers to the secondary storages across a cluster (df) :
> * host 1 : 10.16.36.64:/RAID10-DATA01/secsatxen002/snapshots 2147483648
> 240044032 1907439616 12% /var/run/cloud_mount/1/snapshots
> * Host 2 : 10.16.36.64:/RAID10-DATA01/secsatxen001/snapshots 2147483648
> 569533440 1577950208 27% /var/run/cloud_mount/1/snapshots
> Therefore, the contents are not identical in the 2 secondary storages (this is
> right to my understanding, because multiple secondary storage permit to scale
> horizontally) :
> * Host 1 :
> * -rw-r--r--+ 1 nobody nobody 307M 2012-07-23 16:11
> 07ba46c7-0908-4402-b20a-5ddd29a65d17.vhd
> * -rw-r--r--+ 1 nobody nobody 301M 2012-07-18 02:18
> 1c378e6a-0eba-48d9-8545-ab9fa09b9ba0.vhd
> * -rw-r--r--+ 1 nobody nobody 59M 2012-07-19 02:18
> 3b7e8c19-ba9c-433e-ac18-f52d4ad42be9.vhd
> * -rw-r--r--+ 1 nobody nobody 97M 2012-07-20 02:19
> 4644362a-764d-4c66-a038-168127007a5f.vhd
> * -rw-r--r--+ 1 nobody nobody 47M 2012-07-23 02:18
> 72e54c85-8245-4b84-b4b5-5a1ef2b44e52.vhd
> * -rw-r--r--+ 1 nobody nobody 69M 2012-07-21 02:18
> 76301f19-fcc0-43ac-adab-cde2cf23bd87.vhd
> * -rw-r--r--+ 1 nobody nobody 57M 2012-07-22 02:18
> a05404cd-e2c7-41cf-9972-aa019ab79494.vhd
> * Host 2 : empty
> A simple workaround is to migrate the corresponding instance to the host with
> the right mount. After further checking, this behavior is encountered on all
> clusters, only rarely because only applying to instances that :
> * Have a scheduled snapshot retention policy
> * Have either migrated, been restarted on a different host than previous run
> Though workaround is feasible but it is difficult to predict when the user
> seems this probelm.
> please check if there is any problem in cloudstack logic to ensure right mount
> secondary mount point is available and create it if it doesnt exist
> If this behaviour confirms to be a bug,please file one bug in jira.
> REPRO STEPS
> ==================
> You can try reproducing the scenario by havign two secondary storages and
> continuously taking snapshots
> EXPECTED BEHAVIOR
> ==================
> Cloudstack should create correct mount of secondary storage if it doesnt exist.
> It should never use other secondary storage mount point.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira