You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by GitBox <gi...@apache.org> on 2020/11/24 04:09:09 UTC
[GitHub] [cloudstack] GabrielBrascher opened a new issue #4498: RBD Snapshot fail when snapshot.backup.to.secondary: true
GabrielBrascher opened a new issue #4498:
URL: https://github.com/apache/cloudstack/issues/4498
<!--
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release and master branch are affected too.
Always add information AFTER of these HTML comments, but no need to delete the comments.
-->
##### ISSUE TYPE
<!-- Pick one below and delete the rest -->
* Bug Report
##### COMPONENT NAME
<!--
Categorize the issue, e.g. API, VR, VPN, UI, etc.
-->
~~~
API
~~~
##### CLOUDSTACK VERSION
<!--
New line separated list of affected versions, commit ID for issues on master branch.
-->
~~~
master*
~~~
*tested on master, but might be on previous release as well
##### CONFIGURATION
<!--
Information about the configuration if relevant, e.g. basic network, advanced networking, etc. N/A otherwise
-->
Hypervisor: KVM
Primary Storage: RBD
Secondary Storage: NFS
Global settings: `snapshot.backup.to.secondary: true`
##### SUMMARY
<!-- Explain the problem/feature briefly -->
Taking snapshots for RBD volumes fail when when global settings parameter `snapshot.backup.to.secondary` is set to `true`. if `snapshot.backup.to.secondary` is set to `false` then snapshotting RBD volumes works fine.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] GabrielBrascher edited a comment on issue #4498: RBD Snapshot fail when snapshot.backup.to.secondary: true
Posted by GitBox <gi...@apache.org>.
GabrielBrascher edited a comment on issue #4498:
URL: https://github.com/apache/cloudstack/issues/4498#issuecomment-732977112
@rhtyd ceph snapshot work really well indeed; however, it throws an exception when taking RBD volume snapshots in case of the Zone has `snapshot.backup.to.secondary` set to **true**.
#### Scenario where it can happen:
For instance, if there is a zone with local storage, NFS, and RBD as primary storage pools it could be interesting to have `snapshot.backup.to.secondary` enabled (at least for the local storage and NFS). The problem is tha tif this is enabled then all RBD snapshots will fail. At least it should raise a warn then taking RBD volume snapshots such as _"backup on secondray pool is not supported for RBD snapshots, snapshot is kept only on the primary storage"_.
#### Exception example
For now it fails as it tries to access the RBD storage pool locally, failing to parse configuration files and pool the path.
```
2020-11-20 20:23:59,916 ERROR [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-4:ctx-3de4fd53 job-57289/job-57290 ctx-3e53cbdd) (logid:05d7071d) Take snapshot: 12888 failed
com.cloud.utils.exception.CloudRuntimeException: org.apache.cloudstack.utils.qemu.QemuImgException: 2020-11-20 20:23:18.810175 7f96fa5b48c0 -1 Errors while parsing config file!2020-11-20 20:23:18.810179 7f96fa5b48c0 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No
such file or directory2020-11-20 20:23:18.810181 7f96fa5b48c0 -1 parse_file: cannot open ~/.ceph/ceph.conf: (2) No such file or directory2020-11-20 20:23:18.810182 7f96fa5b48c0 -1 parse_file: cannot open ceph.conf: (2) No such file or directoryqemu-img: Could not open 'r
bd:rbd/9aaef3ea-4d3a-40d0-ab36-5e3db7393a50@28d4acf2-6886-4da1-925e-62910dcf2fd4:mon_host=ceph.address\\:6789:auth_supported=cephx:id=cloudstack:key=BGCADKsHdh7BWChDABVsp5WHinnDCBOZ91dHfgw==:rbd_default_format=2:client_mount_timeout=30': invalid conf option Н�I^]
V: No such file or directory
at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:301)
at org.apache.cloudstack.storage.snapshot.DefaultSnapshotStrategy.backupSnapshot(DefaultSnapshotStrategy.java:171)
```
#### Proposal of next steps
There is two things that can be done:
**A.** fix the handling allowing multiple stortage pools to handle snapshot backup, as described above;
**B.** evaulate alternatives to implement RBD snapshots backup.
My proposal: I will check first the alternative **A** as a bug fix, creating a PR (for 4.15.1, or 4.16?). Later I can evaluate viability of implementing **B** as a new feature.
**Note:** I do not consider this issue a blocker for 4.15.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] GabrielBrascher commented on issue #4498: RBD Snapshot fail when snapshot.backup.to.secondary: true
Posted by GitBox <gi...@apache.org>.
GabrielBrascher commented on issue #4498:
URL: https://github.com/apache/cloudstack/issues/4498#issuecomment-754597373
Fixed by #4568.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] GabrielBrascher closed issue #4498: RBD Snapshot fail when snapshot.backup.to.secondary: true
Posted by GitBox <gi...@apache.org>.
GabrielBrascher closed issue #4498:
URL: https://github.com/apache/cloudstack/issues/4498
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd edited a comment on issue #4498: RBD Snapshot fail when snapshot.backup.to.secondary: true
Posted by GitBox <gi...@apache.org>.
rhtyd edited a comment on issue #4498:
URL: https://github.com/apache/cloudstack/issues/4498#issuecomment-732945627
That is fine as backup to secondary storage is not supported @GabrielBrascher ? (I've a 4.14 env where I've seen this to work, snapshots are on ceph pool itself)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on issue #4498: RBD Snapshot fail when snapshot.backup.to.secondary: true
Posted by GitBox <gi...@apache.org>.
rhtyd commented on issue #4498:
URL: https://github.com/apache/cloudstack/issues/4498#issuecomment-732945627
That is fine as backup to secondary storage is not supported @GabrielBrascher ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] GabrielBrascher commented on issue #4498: RBD Snapshot fail when snapshot.backup.to.secondary: true
Posted by GitBox <gi...@apache.org>.
GabrielBrascher commented on issue #4498:
URL: https://github.com/apache/cloudstack/issues/4498#issuecomment-732977112
@rhtyd ceph snapshot work really well indeed; however, it throws an exception when taking RBD volume snapshots in case of the Zone has `snapshot.backup.to.secondary` set to **true**.
#### Scenario wher it can happen:
For instance, if there is a zone with local storage, NFS, and RBD as primary storage pools it could be interesting to have `snapshot.backup.to.secondary` enabled (at least for the local storage and NFS). The problem is tha tif this is enabled then all RBD snapshots will fail. At least it should raise a warn then taking RBD volume snapshots such as _"backup on secondray pool is not supported for RBD snapshots, snapshot is kept only on the primary storage"_.
#### Exception example
For now it fails as it tries to access the RBD storage pool locally, failing to parse configuration files and pool the path.
```
2020-11-20 20:23:59,916 ERROR [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-4:ctx-3de4fd53 job-57289/job-57290 ctx-3e53cbdd) (logid:05d7071d) Take snapshot: 12888 failed
com.cloud.utils.exception.CloudRuntimeException: org.apache.cloudstack.utils.qemu.QemuImgException: 2020-11-20 20:23:18.810175 7f96fa5b48c0 -1 Errors while parsing config file!2020-11-20 20:23:18.810179 7f96fa5b48c0 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No
such file or directory2020-11-20 20:23:18.810181 7f96fa5b48c0 -1 parse_file: cannot open ~/.ceph/ceph.conf: (2) No such file or directory2020-11-20 20:23:18.810182 7f96fa5b48c0 -1 parse_file: cannot open ceph.conf: (2) No such file or directoryqemu-img: Could not open 'r
bd:rbd/9aaef3ea-4d3a-40d0-ab36-5e3db7393a50@28d4acf2-6886-4da1-925e-62910dcf2fd4:mon_host=ceph.address\\:6789:auth_supported=cephx:id=cloudstack:key=BGCADKsHdh7BWChDABVsp5WHinnDCBOZ91dHfgw==:rbd_default_format=2:client_mount_timeout=30': invalid conf option Н�I^]
V: No such file or directory
at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:301)
at org.apache.cloudstack.storage.snapshot.DefaultSnapshotStrategy.backupSnapshot(DefaultSnapshotStrategy.java:171)
```
#### Proposal of next steps
There is two things that can be done:
**A.** fix the handling allowing multiple stortage pools to handle snapshot backup, as described above;
**B.** evaulate alternatives to implement RBD snapshots backup.
My proposal: I will check first the alternative **A** as a bug fix, creating a PR (for 4.15.1, or 4.16?). Later I can evaluate viability of implementing **B** as a new feature.
**Note:** I do not consider this issue a blocker for 4.15.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] GabrielBrascher edited a comment on issue #4498: RBD Snapshot fail when snapshot.backup.to.secondary: true
Posted by GitBox <gi...@apache.org>.
GabrielBrascher edited a comment on issue #4498:
URL: https://github.com/apache/cloudstack/issues/4498#issuecomment-732977112
@rhtyd ceph snapshot work really well indeed; however, it throws an exception when taking RBD volume snapshots in case of the Zone has `snapshot.backup.to.secondary` set to **true**.
#### Scenario where it can happen:
For instance, if there is a zone with local storage, NFS, and RBD as primary storage pools it could be interesting to have `snapshot.backup.to.secondary` enabled (at least for the local storage and NFS). The problem is that if backup is enabled then all RBD snapshots fail. At least it should allow the snapshot and then raise a warn such as _"backup on secondary pool is not supported for RBD snapshots, RBD snapshots will remain only on the primary storage"_.
#### Exception example:
For now it fails as it tries to access the RBD storage pool locally, failing to parse configuration files and pool the path.
```
2020-11-20 20:23:59,916 ERROR [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-4:ctx-3de4fd53 job-57289/job-57290 ctx-3e53cbdd) (logid:05d7071d) Take snapshot: 12888 failed
com.cloud.utils.exception.CloudRuntimeException: org.apache.cloudstack.utils.qemu.QemuImgException: 2020-11-20 20:23:18.810175 7f96fa5b48c0 -1 Errors while parsing config file!2020-11-20 20:23:18.810179 7f96fa5b48c0 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No
such file or directory2020-11-20 20:23:18.810181 7f96fa5b48c0 -1 parse_file: cannot open ~/.ceph/ceph.conf: (2) No such file or directory2020-11-20 20:23:18.810182 7f96fa5b48c0 -1 parse_file: cannot open ceph.conf: (2) No such file or directoryqemu-img: Could not open 'r
bd:rbd/9aaef3ea-4d3a-40d0-ab36-5e3db7393a50@28d4acf2-6886-4da1-925e-62910dcf2fd4:mon_host=ceph.address\\:6789:auth_supported=cephx:id=cloudstack:key=BGCADKsHdh7BWChDABVsp5WHinnDCBOZ91dHfgw==:rbd_default_format=2:client_mount_timeout=30': invalid conf option Н�I^]
V: No such file or directory
at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:301)
at org.apache.cloudstack.storage.snapshot.DefaultSnapshotStrategy.backupSnapshot(DefaultSnapshotStrategy.java:171)
```
#### Proposal of next steps:
There is two things that can be done:
**A.** fix the handling allowing multiple stortage pools to handle snapshot backup, as described above;
**B.** evaulate alternatives to implement RBD snapshots backup.
My proposal: I will check first the alternative **A** as a bug fix, creating a PR (for 4.15.1, or 4.16?). Later I can evaluate viability of implementing **B** as a new feature.
**Note:** I do not consider this issue a blocker for 4.15.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org