You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by GitBox <gi...@apache.org> on 2022/04/05 19:57:44 UTC
[GitHub] [cloudstack] nvazquez opened a new issue, #6209: Revert volume to snapshot fails after volume migration
nvazquez opened a new issue, #6209:
URL: https://github.com/apache/cloudstack/issues/6209
##### ISSUE TYPE
* Bug Report
* Improvement Request
* Enhancement Request
##### COMPONENT NAME
~~~
Storage
~~~
##### CLOUDSTACK VERSION
~~~
4.16.1 and main
~~~
##### CONFIGURATION
Set `snapshot.backup.to.secondary` to true
##### OS / ENVIRONMENT
Tested KVM on NFS storage
##### SUMMARY
Revert volume to snapshot after migrating a volume to another storage fails with error:
````
2022-04-05 14:52:25,411 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-12:ctx-e7abb0ff job-48) (logid:8178b033) Unexpected exception while executing org.apache.cloudstack.api.command.user.snapshot.RevertSnapshotCmd
com.cloud.utils.exception.CloudRuntimeException: Cannot find an entry for snapshot 1 on primary storage pools
at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.revertSnapshot(SnapshotServiceImpl.java:433)
at org.apache.cloudstack.storage.snapshot.DefaultSnapshotStrategy.revertSnapshot(DefaultSnapshotStrategy.java:393)
at com.cloud.storage.snapshot.SnapshotManagerImpl.revertSnapshot(SnapshotManagerImpl.java:330)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy218.revertSnapshot(Unknown Source)
at org.apache.cloudstack.api.command.user.snapshot.RevertSnapshotCmd.execute(RevertSnapshotCmd.java:101)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:156)
at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
````
##### STEPS TO REPRODUCE
~~~
- Take VM snapshots (VM running)
- Stop VM
- Migrate VM with volume
- Try revert volume to snapshot
~~~
##### EXPECTED RESULTS
~~~
Revert succeeds
~~~
##### ACTUAL RESULTS
~~~
Failure (logs above)
~~~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] nvazquez commented on issue #6209: Revert volume to snapshot failures
Posted by GitBox <gi...@apache.org>.
nvazquez commented on issue #6209:
URL: https://github.com/apache/cloudstack/issues/6209#issuecomment-1103076669
Moved to the next milestone
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rohityadavcloud closed issue #6209: Revert volume to snapshot failures
Posted by GitBox <gi...@apache.org>.
rohityadavcloud closed issue #6209: Revert volume to snapshot failures
URL: https://github.com/apache/cloudstack/issues/6209
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] weizhouapache commented on issue #6209: Revert volume to snapshot failures
Posted by GitBox <gi...@apache.org>.
weizhouapache commented on issue #6209:
URL: https://github.com/apache/cloudstack/issues/6209#issuecomment-1102694473
@nvazquez @slavkap @Pearl1594
my opinion, for kvm with NFS storage,
(1) the record should be removed from DB, because no snapshot exists in primary storage.
(2) the primary datastore info should be optional (not required) when revert snapshot on KVM, because kvm agent tries to copy snapshot from secondary storage. the operation should succeed even if primary datastore info is not provided. however, from code it looks it is required.
(3) it looks ceph storage and managed storage are not impacted. correct me if I am wrong
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] slavkap commented on issue #6209: Revert volume to snapshot fails after volume migration
Posted by GitBox <gi...@apache.org>.
slavkap commented on issue #6209:
URL: https://github.com/apache/cloudstack/issues/6209#issuecomment-1089306858
@nvazquez, this can also be hit when migrating a volume to another storage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] nvazquez commented on issue #6209: Revert volume to snapshot fails after volume migration
Posted by GitBox <gi...@apache.org>.
nvazquez commented on issue #6209:
URL: https://github.com/apache/cloudstack/issues/6209#issuecomment-1089265571
@slavkap @weizhouapache this might be related to #5433 and should find a solution for both issues
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] nvazquez commented on issue #6209: Revert volume to snapshot failures
Posted by GitBox <gi...@apache.org>.
nvazquez commented on issue #6209:
URL: https://github.com/apache/cloudstack/issues/6209#issuecomment-1103080677
Thanks @weizhouapache - this would need further investigation for different storage and also considering stock qemu and qemu-ev packages on KVM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on issue #6209: Revert volume to snapshot failures
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on issue #6209:
URL: https://github.com/apache/cloudstack/issues/6209#issuecomment-1090220241
I think that makes sense @nvazquez @slavkap
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] nvazquez commented on issue #6209: Revert volume to snapshot fails after volume migration
Posted by GitBox <gi...@apache.org>.
nvazquez commented on issue #6209:
URL: https://github.com/apache/cloudstack/issues/6209#issuecomment-1089725273
Thanks @slavkap I've updated the description
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] nvazquez commented on issue #6209: Revert volume to snapshot failures
Posted by GitBox <gi...@apache.org>.
nvazquez commented on issue #6209:
URL: https://github.com/apache/cloudstack/issues/6209#issuecomment-1090175963
@slavkap @Pearl1594 opening this for discussion: since https://github.com/apache/cloudstack/pull/1740 snapshots on primary storage are cleaned up after migration. Internally a new volume DB record is created after the volume is migrated and the old one is destroyed, along with snapshots and that seems to cause the issue (for the reinstall VM case I think that is also the case).
As an example, before the volume migration:
````
MariaDB [cloud]> select id, store_id, store_role, snapshot_id, state, volume_id from snapshot_store_ref where snapshot_id = 5;
+----+----------+------------+-------------+-------+-----------+
| id | store_id | store_role | snapshot_id | state | volume_id |
+----+----------+------------+-------------+-------+-----------+
| 9 | 1 | Primary | 5 | Ready | 13 |
| 10 | 1 | Image | 5 | Ready | 13 |
+----+----------+------------+-------------+-------+-----------+
2 rows in set (0.00 sec)
````
After volume migration:
````
MariaDB [cloud]> select id, store_id, store_role, snapshot_id, state, volume_id from snapshot_store_ref where snapshot_id = 5;
+----+----------+------------+-------------+-------+-----------+
| id | store_id | store_role | snapshot_id | state | volume_id |
+----+----------+------------+-------------+-------+-----------+
| 10 | 1 | Image | 5 | Ready | 14 |
+----+----------+------------+-------------+-------+-----------+
1 rows in set (0.00 sec)
````
As discussed offline @slavkap proposed the storage driver checks if the references on `snapshot_store_ref` should be deleted for Primary role - @Pearl1594 what do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] nvazquez commented on issue #6209: Revert volume to snapshot failures
Posted by GitBox <gi...@apache.org>.
nvazquez commented on issue #6209:
URL: https://github.com/apache/cloudstack/issues/6209#issuecomment-1102687654
Ping @weizhouapache @DaanHoogland @rohityadavcloud @andrijapanicsb what's your take on the approach to be taken for solving this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rohityadavcloud commented on issue #6209: Revert volume to snapshot failures
Posted by GitBox <gi...@apache.org>.
rohityadavcloud commented on issue #6209:
URL: https://github.com/apache/cloudstack/issues/6209#issuecomment-1189864909
Merged in https://github.com/apache/cloudstack/pull/6527
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org