You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@cloudstack.apache.org by GitBox <gi...@apache.org> on 2020/03/16 13:22:15 UTC

[GitHub] [cloudstack] DaanHoogland opened a new pull request #3969: Snapshot deletion issues

DaanHoogland opened a new pull request #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969
 
 
   ## Description
   <!--- Describe your changes in detail -->
   copy of PR #3649 on joint account for easier cooperation.
   
   Tested both cases:
   - Snapshot backed up on Secondary (stored on primary and secondary) 
   - Snapshot stored only on primary storage.
   
   This fix works for KVM  and XenServer.
   
   **Note**: I changed the name of the strategy from _XenserverSnapshotStrategy_ to _DefaultSnapshotStrategy_ due to the fact that the "Xenserver" strategy handles also Ceph, KVM, and do return `StrategyPriority.DEFAULT` in any case except _REVERT_.
   
   ```
       @Override
       public StrategyPriority canHandle(Snapshot snapshot, SnapshotOperation op) {
           if (SnapshotOperation.REVERT.equals(op)) {
               long volumeId = snapshot.getVolumeId();
               VolumeVO volumeVO = volumeDao.findById(volumeId);
   
               if (volumeVO != null && ImageFormat.QCOW2.equals(volumeVO.getFormat())) {
                   return StrategyPriority.DEFAULT;
               }
               return StrategyPriority.CANT_HANDLE;
           }
           return StrategyPriority.DEFAULT;
       }
   ```
   
   Fixes: #3646
   
   ## Types of changes
   <!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
   - [ ] Breaking change (fix or feature that would cause existing functionality to change)
   - [ ] New feature (non-breaking change which adds functionality)
   - [x] Bug fix (non-breaking change which fixes an issue)
   - [ ] Enhancement (improves an existing feature and functionality)
   - [ ] Cleanup (Code refactoring and cleanup, that may add test cases)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] DaanHoogland removed a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

DaanHoogland removed a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-609730934
 
 
   @blueorangutan test

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] Slair1 commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

Slair1 commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-613075769
 
 
   > but you can test 4.13.1 RC1 and see if VMs are suspended as expected - perhaps some parts of the code were refractored or so...
   
   @andrijapanicsb @DaanHoogland from our testing it is just gone... so strange - i hope other commits didn't get dropped somehow.  I even checked the "blame" and I don't see a refactor - the last change in branch 4.13 was 4-years ago for the section of code.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] andrijapanicsb commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

andrijapanicsb commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-613085691
 
 
   @Slair1 do you mind creating a PR for MASTER (not 4.13 - we already have RC1 for vote today) to add back those few lines... it's incredible - the global desc that the VM will be suspended is STILL there in 4.12/4.13/master, but the "vm.suspend" code is 
   @DaanHoogland your opinion?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] DaanHoogland commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

DaanHoogland commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-610262406
 
 
   @blueorangutan test matrix

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan removed a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan removed a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-599630848
 
 
   @DaanHoogland a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-606342663
 
 
   <b>Trillian test result (tid-1337)</b>
   Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
   Total time taken: 27846 seconds
   Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3969-t1337-kvm-centos7.zip
   Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
   Smoke tests completed. 76 look OK, 1 have error(s)
   Only failed tests results shown below:
   
   
   Test | Result | Time (s) | Test File
   --- | --- | --- | ---
   test_02_vpc_privategw_static_routes | `Failure` | 173.80 | test_privategw_acl.py
   test_03_vpc_privategw_restart_vpc_cleanup | `Failure` | 180.22 | test_privategw_acl.py
   test_04_rvpc_privategw_static_routes | `Failure` | 238.01 | test_privategw_acl.py
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] andrijapanicsb commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

andrijapanicsb commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-607319803
 
 
   when NOT backing up the snap to secondary - the snap fails to be deleted....damn it 😢 
   
   2020-04-01 15:29:22,117 DEBUG [c.c.a.ApiServlet] (qtp504527234-17:ctx-3353a255 ctx-cc5b2b2c) (logid:0f9171aa) ===END===  10.1.0.1 -- GET  command=deleteSnapshot&id=1e63d646-eadd-4a1d-b8b8-ddec90fc9e0e&response=json&_=1585754941759
   2020-04-01 15:29:22,133 DEBUG [o.a.c.s.s.DefaultSnapshotStrategy] (API-Job-Executor-3:ctx-c4bb5e71 job-284 ctx-75e649e7) (logid:37b859e0) Can't find snapshot
   [snapshot id: 19] on backup storage
   2020-04-01 15:29:22,141 DEBUG [o.a.c.s.s.SnapshotObject] (API-Job-Executor-3:ctx-c4bb5e71 job-284 ctx-75e649e7) (logid:37b859e0) Failed to update state:java.lang.NullPointerException
   2020-04-01 15:29:22,142 DEBUG [c.c.s.s.SnapshotManagerImpl] (API-Job-Executor-3:ctx-c4bb5e71 job-284 ctx-75e649e7) (logid:37b859e0) Failed to delete snapshot: 19:java.lang.NullPointerException
   2020-04-01 15:29:22,147 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-3:ctx-c4bb5e71 job-284) (logid:37b859e0) Unexpected exception while executing org.apache.cloudstack.api.command.user.snapshot.DeleteSnapshotCmd
   com.cloud.utils.exception.CloudRuntimeException: Failed to delete snapshot:java.lang.NullPointerException
           at com.cloud.storage.snapshot.SnapshotManagerImpl.deleteSnapshot(SnapshotManagerImpl.java:611)
           at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
           at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
           at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
           at java.lang.reflect.Method.invoke(Method.java:498)
           at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
           at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
           at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
           at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
           at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
           at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
           at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
           at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
           at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
           at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
           at com.sun.proxy.$Proxy199.deleteSnapshot(Unknown Source)
           at org.apache.cloudstack.api.command.user.snapshot.DeleteSnapshotCmd.execute(DeleteSnapshotCmd.java:103)
           at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:156)
           at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
           at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:586)
           at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
           at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
           at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
           at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
           at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
           at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:534)
           at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
           at java.util.concurrent.FutureTask.run(FutureTask.java:266)
           at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
           at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
           at java.lang.Thread.run(Thread.java:748)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-609040966
 
 
   Packaging result: ✔centos7 ✔debian. JID-1125

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan removed a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan removed a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-600171769
 
 
   Packaging result: ✖centos6 ✖centos7 ✔debian. JID-1067

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-610224060
 
 
   @DaanHoogland a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan removed a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan removed a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-609380649
 
 
   <b>Trillian test result (tid-1359)</b>
   Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
   Total time taken: 60212 seconds
   Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3969-t1359-kvm-centos7.zip
   Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
   Intermittent failure detected: /marvin/tests/smoke/test_public_ip_range.py
   Intermittent failure detected: /marvin/tests/smoke/test_reset_vm_on_reboot.py
   Intermittent failure detected: /marvin/tests/smoke/test_resource_accounting.py
   Intermittent failure detected: /marvin/tests/smoke/test_router_dhcphosts.py
   Intermittent failure detected: /marvin/tests/smoke/test_router_dns.py
   Intermittent failure detected: /marvin/tests/smoke/test_router_dnsservice.py
   Intermittent failure detected: /marvin/tests/smoke/test_routers_iptables_default_policy.py
   Intermittent failure detected: /marvin/tests/smoke/test_routers_network_ops.py
   Intermittent failure detected: /marvin/tests/smoke/test_routers.py
   Intermittent failure detected: /marvin/tests/smoke/test_secondary_storage.py
   Intermittent failure detected: /marvin/tests/smoke/test_service_offerings.py
   Intermittent failure detected: /marvin/tests/smoke/test_snapshots.py
   Intermittent failure detected: /marvin/tests/smoke/test_ssvm.py
   Intermittent failure detected: /marvin/tests/smoke/test_templates.py
   Intermittent failure detected: /marvin/tests/smoke/test_usage.py
   Intermittent failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
   Intermittent failure detected: /marvin/tests/smoke/test_vm_snapshots.py
   Intermittent failure detected: /marvin/tests/smoke/test_volumes.py
   Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
   Intermittent failure detected: /marvin/tests/smoke/test_vpc_router_nics.py
   Intermittent failure detected: /marvin/tests/smoke/test_vpc_vpn.py
   Intermittent failure detected: /marvin/tests/smoke/test_hostha_kvm.py
   Smoke tests completed. 55 look OK, 22 have error(s)
   Only failed tests results shown below:
   
   
   Test | Result | Time (s) | Test File
   --- | --- | --- | ---
   test_02_vpc_privategw_static_routes | `Failure` | 177.05 | test_privategw_acl.py
   test_03_vpc_privategw_restart_vpc_cleanup | `Failure` | 178.03 | test_privategw_acl.py
   test_04_rvpc_privategw_static_routes | `Failure` | 242.25 | test_privategw_acl.py
   ContextSuite context=TestResetVmOnReboot>:setup | `Error` | 0.00 | test_reset_vm_on_reboot.py
   ContextSuite context=TestRAMCPUResourceAccounting>:setup | `Error` | 0.00 | test_resource_accounting.py
   ContextSuite context=TestRouterDHCPHosts>:setup | `Error` | 0.00 | test_router_dhcphosts.py
   ContextSuite context=TestRouterDHCPOpts>:setup | `Error` | 0.00 | test_router_dhcphosts.py
   ContextSuite context=TestRouterDns>:setup | `Error` | 0.00 | test_router_dns.py
   test_01_sys_vm_start | `Failure` | 0.06 | test_secondary_storage.py
   ContextSuite context=TestRouterDnsService>:setup | `Error` | 0.00 | test_router_dnsservice.py
   ContextSuite context=TestRouterIpTablesPolicies>:setup | `Error` | 0.00 | test_routers_iptables_default_policy.py
   ContextSuite context=TestVPCIpTablesPolicies>:setup | `Error` | 0.00 | test_routers_iptables_default_policy.py
   ContextSuite context=TestIsolatedNetworks>:setup | `Error` | 0.00 | test_routers_network_ops.py
   ContextSuite context=TestRedundantIsolateNetworks>:setup | `Error` | 0.00 | test_routers_network_ops.py
   ContextSuite context=TestRouterServices>:setup | `Error` | 0.00 | test_routers.py
   ContextSuite context=TestCpuCapServiceOfferings>:setup | `Error` | 0.00 | test_service_offerings.py
   ContextSuite context=TestServiceOfferings>:setup | `Error` | 0.14 | test_service_offerings.py
   ContextSuite context=TestSnapshotRootDisk>:setup | `Error` | 0.00 | test_snapshots.py
   test_01_list_sec_storage_vm | `Failure` | 0.03 | test_ssvm.py
   test_02_list_cpvm_vm | `Failure` | 0.02 | test_ssvm.py
   test_03_ssvm_internals | `Failure` | 0.03 | test_ssvm.py
   test_04_cpvm_internals | `Failure` | 0.02 | test_ssvm.py
   test_05_stop_ssvm | `Failure` | 0.02 | test_ssvm.py
   test_06_stop_cpvm | `Failure` | 0.03 | test_ssvm.py
   test_07_reboot_ssvm | `Failure` | 0.03 | test_ssvm.py
   test_08_reboot_cpvm | `Failure` | 0.03 | test_ssvm.py
   test_09_destroy_ssvm | `Failure` | 0.02 | test_ssvm.py
   test_10_destroy_cpvm | `Failure` | 0.03 | test_ssvm.py
   test_02_create_template_with_checksum_sha1 | `Error` | 65.41 | test_templates.py
   test_03_create_template_with_checksum_sha256 | `Error` | 65.44 | test_templates.py
   test_04_create_template_with_checksum_md5 | `Error` | 65.36 | test_templates.py
   test_05_create_template_with_no_checksum | `Error` | 65.37 | test_templates.py
   test_02_deploy_vm_from_direct_download_template | `Error` | 1.18 | test_templates.py
   test_03_deploy_vm_wrong_checksum | `Error` | 1.26 | test_templates.py
   ContextSuite context=TestTemplates>:setup | `Error` | 17.64 | test_templates.py
   ContextSuite context=TestISOUsage>:setup | `Error` | 0.00 | test_usage.py
   ContextSuite context=TestLBRuleUsage>:setup | `Error` | 0.00 | test_usage.py
   ContextSuite context=TestNatRuleUsage>:setup | `Error` | 0.00 | test_usage.py
   ContextSuite context=TestPublicIPUsage>:setup | `Error` | 0.00 | test_usage.py
   ContextSuite context=TestSnapshotUsage>:setup | `Error` | 0.00 | test_usage.py
   ContextSuite context=TestVmUsage>:setup | `Error` | 0.00 | test_usage.py
   ContextSuite context=TestVolumeUsage>:setup | `Error` | 0.00 | test_usage.py
   ContextSuite context=TestVpnUsage>:setup | `Error` | 0.00 | test_usage.py
   ContextSuite context=Test01DeployVM>:setup | `Error` | 0.00 | test_vm_life_cycle.py
   ContextSuite context=Test02VMLifeCycle>:setup | `Error` | 0.00 | test_vm_life_cycle.py
   test_14_secure_to_secure_vm_migration | `Error` | 11.27 | test_vm_life_cycle.py
   test_15_secured_to_nonsecured_vm_migration | `Error` | 73.87 | test_vm_life_cycle.py
   test_16_nonsecured_to_secured_vm_migration | `Error` | 1.16 | test_vm_life_cycle.py
   ContextSuite context=TestVmSnapshot>:setup | `Error` | 1.56 | test_vm_snapshots.py
   ContextSuite context=TestCreateVolume>:setup | `Error` | 0.00 | test_volumes.py
   ContextSuite context=TestVolumes>:setup | `Error` | 0.00 | test_volumes.py
   ContextSuite context=TestVPCRedundancy>:setup | `Error` | 0.00 | test_vpc_redundant.py
   ContextSuite context=TestVPCNics>:setup | `Error` | 0.00 | test_vpc_router_nics.py
   ContextSuite context=TestRVPCSite2SiteVpn>:setup | `Error` | 0.00 | test_vpc_vpn.py
   ContextSuite context=TestVPCSite2SiteVPNMultipleOptions>:setup | `Error` | 0.00 | test_vpc_vpn.py
   ContextSuite context=TestVpcRemoteAccessVpn>:setup | `Error` | 0.00 | test_vpc_vpn.py
   ContextSuite context=TestVpcSite2SiteVpn>:setup | `Error` | 0.00 | test_vpc_vpn.py
   test_disable_oobm_ha_state_ineligible | `Error` | 1516.40 | test_hostha_kvm.py
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] andrijapanicsb commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

andrijapanicsb commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-610827367
 
 
   XS test that needs investigation:
   
   2020-04-07 16:45:40,456 - CRITICAL - EXCEPTION: test_03_list_snapshots: ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.7/site-packages/nose/suite.py", line 228, in run\n    self.tearDown()\n', '  File "/usr/lib/python2.7/site-packages/nose/suite.py", line 351, in tearDown\n    self.teardownContext(ancestor)\n', '  File "/usr/lib/python2.7/site-packages/nose/suite.py", line 367, in teardownContext\n    try_run(context, names)\n', '  File "/usr/lib/python2.7/site-packages/nose/util.py", line 471, in try_run\n    return func()\n', '  File "/marvin/tests/smoke/test_list_ids_parameter.py", line 209, in tearDownClass\n    raise Exception("Warning: Exception during cleanup : %s" % e)\n', "Exception: Warning: Exception during cleanup : Job failed: {jobprocstatus : 0, created : u'2020-04-07T16:46:13+0000', jobresult : {errorcode : 530, errortext : u**'Failed to delete snapshot'**}, cmd : u'org.apache.cloudstack.api.command.user.snapshot.DeleteSnapshotCmd', userid : u'964f52cb-78d6-11ea-bc32-1e00400108b3', jobstatus : 2, jobid : u'8806aed0-896c-4264-801e-d570217f4ae0', jobresultcode : 530, jobinstanceid : u'4cdc1214-942c-436c-b6ae-738f3220b1cf', jobresulttype : u'object', jobinstancetype : u'Snapshot', completed : u'2020-04-07T16:46:13+0000', accountid : u'964f4d54-78d6-11ea-bc32-1e00400108b3'}\n"]
   
   VMware tests that failed:
   
   'org.apache.cloudstack.api.command.admin.vm.MigrateVMCmd\', userid : u\'3d686233-78da-11ea-8ab1-1e009101089c\', jobstatus : 2, jobid : u\'aace648a-e269-49f8-982b-3c07bf5baa11\', jobresultcode : 530, jobresulttype : u\'object\', jobresult : {errorcode : 530, errortext : u"MigrationCommand failed due to Exception: java.lang.RuntimeException\\nMessage: **The target host does not support the virtual machine\'s current hardware requirements.**\\n"}, accountid : u\'3d685acc-78da-11ea-8ab1-1e009101089c\'}\n']
   That is funny...

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] andrijapanicsb edited a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

andrijapanicsb edited a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-609950065
 
 
   Latest testing of the deletion stuff reports below
   
   (previously I have tested normal operations works fine - convert to this to that, revert (KVM only), etc, for all 3 hypervisors)
   
   ## snapshot.backup.to.secondary=FALSE
   
   Expected to work ONLY with KVM+Ceph (not other variants, not other hypervisors)
   (but the basic create/delete snap was tested with KVM+NFS to make sure nothing broken in that specific flow)
   create, delete snap works fine - GC kicks in, no exceptions - after GC runs, the "PRIMARY" (role_type=primary) row in snapshots_store_ref is left undeleted (with other snapshot.backup.to.secondary=TRUE testing as well)
   
   @rafaelweingartner please test with Ceph if possible
   
   ## snapshot.backup.to.secondary=TRUE
   
   - snaps can be created, operated with, deleted fine
   - deleted from file systems as expected (State changed to Destroyed in snapshots and snapshot_store_ref tables - "removed"date NOT set in the main "snapshots" table
   - GC will remove the IMAGE row (role_type=image) from the snapshot_store_ref table and set the "removed" date in the main "snapshots" table
   - GC will NOT remove the PRIMARY row from the snapshot_store_ref table (any of possibly many rows for the same volume (i.e. many different snapss of the same volume)
   - For XS - snaps chains deletion is broken - everything works fine, except that no snap is deleted in the Primary Storage until the "VHD chain too long" happens one day - solution go to XenCenter and remove all the snaps by name (dig snap names from DB)
   
   ### GC garbage workaround
   
   Remove rows from snapshot_store_ref table:
   
   > delete from snapshot_store_ref where snapshot_id in (select id from snapshots where status="Destroyed" and removed is NOT NULL); 
   
   (removed not NULL = means that the GC has run and left the garbage behind)
   
   or the safer one (volume is removed)
   
   > delete from snapshot_store_ref where snapshot_id in (select id from snapshots where status="Destroyed" and removed is NOT NULL) and volume_id in (select id from volumes where state="Expunged" and removed is NOT NULL);

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] andrijapanicsb edited a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

andrijapanicsb edited a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-609950065
 
 
   Latest testing of the deletion stuff reports below
   
   (previously I have tested normal operations works fine - convert to this to that, revert (KVM only), etc, for all 3 hypervisors)
   
   ## snapshot.backup.to.secondary=FALSE
   
   Expected to work ONLY with KVM+Ceph (not other variants, not other hypervisors)
   (but the basic create/delete snap was tested with KVM+NFS to make sure nothing broken in that specific flow)
   create, delete snap works fine - GC kicks in, no exceptions - after GC runs, the "PRIMARY" (role_type=primary) row in snapshots_store_ref is left undeleted (with other snapshot.backup.to.secondary=TRUE testing as well)
   
   ## snapshot.backup.to.secondary=TRUE
   
   - snaps can be created, operated with, deleted fine
   - deleted from file systems as expected (State changed to Destroyed in snapshots and snapshot_store_ref tables - "removed"date NOT set in the main "snapshots" table
   - GC will remove the IMAGE row (role_type=image) from the snapshot_store_ref table and set the "removed" date in the main "snapshots" table
   - GC will NOT remove the PRIMARY row from the snapshot_store_ref table (any of possibly many rows for the same volume (i.e. many different snapss of the same volume)
   - For XS - snaps chains deletion is broken - everything works fine, except that no snap is deleted in the Primary Storage until the "VHD chain too long" happens one day - solution go to XenCenter and remove all the snaps by name (dig snap names from DB)
   
   ### GC garbage workaround
   
   Remove rows from snapshot_store_ref table:
   
   > delete from snapshot_store_ref where snapshot_id in (select id from snapshots where status="Destroyed" and removed is NOT NULL); 
   
   (removed not NULL = means that the GC has run and left the garbage behind)
   
   or the safer one (volume is removed)
   
   > delete from snapshot_store_ref where snapshot_id in (select id from snapshots where status="Destroyed" and removed is NOT NULL) and volume_id in (select id from volumes where state="Expunged" and removed is NOT NULL);

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] harikrishna-patnala commented on a change in pull request #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

harikrishna-patnala commented on a change in pull request #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#discussion_r403937237
 
 

 ##########
 File path: engine/storage/snapshot/src/main/java/org/apache/cloudstack/storage/snapshot/DefaultSnapshotStrategy.java
 ##########
 @@ -285,7 +284,27 @@ public boolean deleteSnapshot(Long snapshotId) {
             }
         }
 
-        boolean deletedOnPrimary = deleteSnapshotOnPrimary(snapshotId);
+        boolean deletedOnPrimary = false;
+        snapshotVO = snapshotDao.findById(snapshotId);
+        SnapshotInfo snapshotOnPrimaryInfo = snapshotDataFactory.getSnapshot(snapshotId, DataStoreRole.Primary);
+        if (snapshotVO != null && snapshotVO.getState() == Snapshot.State.Destroyed) {
+            deletedOnPrimary = deleteSnapshotOnPrimary(snapshotId, snapshotOnPrimaryInfo);
+        } else {
+            // This is to handle snapshots which are created only on primary when snapshot.backup.to.secondary is set to false. 
 
 Review comment:
   There is will be only 'Primary' store entry in snapshot_store_ref table for a snapshot created when snapshot.backup.to.secondary is set to false. so deletion of snapshot on secondary store will fail and it won't mark snapshot as destroyed and then goes to 'else' case. In other case (snapshot.backup.to.secondary is set to true) 'Image' and 'Primary' entries will be there in which it goes to 'if' case. 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] andrijapanicsb removed a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

andrijapanicsb removed a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-609044924
 
 
   @blueorangutan test

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan removed a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan removed a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-607219952
 
 
   @andrijapanicsb a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] DaanHoogland commented on a change in pull request #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

DaanHoogland commented on a change in pull request #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#discussion_r403918034
 
 

 ##########
 File path: engine/storage/snapshot/src/main/java/org/apache/cloudstack/storage/snapshot/DefaultSnapshotStrategy.java
 ##########
 @@ -285,7 +284,27 @@ public boolean deleteSnapshot(Long snapshotId) {
             }
         }
 
-        boolean deletedOnPrimary = deleteSnapshotOnPrimary(snapshotId);
+        boolean deletedOnPrimary = false;
+        snapshotVO = snapshotDao.findById(snapshotId);
+        SnapshotInfo snapshotOnPrimaryInfo = snapshotDataFactory.getSnapshot(snapshotId, DataStoreRole.Primary);
+        if (snapshotVO != null && snapshotVO.getState() == Snapshot.State.Destroyed) {
+            deletedOnPrimary = deleteSnapshotOnPrimary(snapshotId, snapshotOnPrimaryInfo);
+        } else {
+            // This is to handle snapshots which are created only on primary when snapshot.backup.to.secondary is set to false. 
+            SnapshotObject obj = (SnapshotObject) snapshotOnPrimaryInfo;
+            try {
+                obj.processEvent(Snapshot.Event.DestroyRequested);
+                deletedOnPrimary = deleteSnapshotOnPrimary(snapshotId, snapshotOnPrimaryInfo);
+                if (!deletedOnPrimary) {
+                    obj.processEvent(Snapshot.Event.OperationFailed);
+                } else {
+                    obj.processEvent(Snapshot.Event.OperationSucceeded);
+                }
+            } catch (NoTransitionException e) {
+            s_logger.debug("Failed to set the state to destroying: ", e);
+            return false;
 
 Review comment:
   indent?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-600159692
 
 
   @GabrielBrascher a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan removed a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan removed a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-605782913
 
 
   Packaging result: ✔centos7 ✔debian. JID-1102

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-612006022
 
 
   @andrijapanicsb a Trillian-Jenkins test job (centos7 mgmt + xenserver-71) has been kicked to run smoke tests

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] harikrishna-patnala commented on a change in pull request #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

harikrishna-patnala commented on a change in pull request #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#discussion_r403937237
 
 

 ##########
 File path: engine/storage/snapshot/src/main/java/org/apache/cloudstack/storage/snapshot/DefaultSnapshotStrategy.java
 ##########
 @@ -285,7 +284,27 @@ public boolean deleteSnapshot(Long snapshotId) {
             }
         }
 
-        boolean deletedOnPrimary = deleteSnapshotOnPrimary(snapshotId);
+        boolean deletedOnPrimary = false;
+        snapshotVO = snapshotDao.findById(snapshotId);
+        SnapshotInfo snapshotOnPrimaryInfo = snapshotDataFactory.getSnapshot(snapshotId, DataStoreRole.Primary);
+        if (snapshotVO != null && snapshotVO.getState() == Snapshot.State.Destroyed) {
+            deletedOnPrimary = deleteSnapshotOnPrimary(snapshotId, snapshotOnPrimaryInfo);
+        } else {
+            // This is to handle snapshots which are created only on primary when snapshot.backup.to.secondary is set to false. 
 
 Review comment:
   There will be only 'Primary' store entry in snapshot_store_ref table for a snapshot created when snapshot.backup.to.secondary is set to false. so deletion of snapshot on secondary store will fail and it won't mark snapshot as destroyed and then goes to 'else' case. In other case (snapshot.backup.to.secondary is set to true) 'Image' and 'Primary' entries will be there in which it goes to 'if' case. 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-612006718
 
 
   @andrijapanicsb a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-609045028
 
 
   @andrijapanicsb a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] DaanHoogland commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

DaanHoogland commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-611679538
 
 
   @Slair1 :
   ```
   $ git branch --contains c94ee1454d3ecdf1bf371fd18c8e3ec0631d3092
     4.11
     4.12
     4.13
   ...
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] andrijapanicsb commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

andrijapanicsb commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-611587549
 
 
   @blueorangutan package

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan removed a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan removed a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-609045028
 
 
   @andrijapanicsb a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] DaanHoogland commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

DaanHoogland commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-609963113
 
 
   @blueorangutan package

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] DaanHoogland commented on a change in pull request #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

DaanHoogland commented on a change in pull request #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#discussion_r406054109
 
 

 ##########
 File path: engine/storage/snapshot/src/main/java/org/apache/cloudstack/storage/snapshot/DefaultSnapshotStrategy.java
 ##########
 @@ -287,18 +287,25 @@ private boolean deleteOnPrimaryIfNeeded(Long snapshotId) {
             // Here we handle snapshots which are to be deleted but are not marked as destroyed yet.
             // This may occur for instance when they are created only on primary because
             // snapshot.backup.to.secondary was set to false.
-            SnapshotObject obj = (SnapshotObject) snapshotOnPrimaryInfo;
-            try {
-                obj.processEvent(Snapshot.Event.DestroyRequested);
-                deletedOnPrimary = deleteSnapshotOnPrimary(snapshotId, snapshotOnPrimaryInfo);
-                if (!deletedOnPrimary) {
-                    obj.processEvent(Snapshot.Event.OperationFailed);
-                } else {
-                    obj.processEvent(Snapshot.Event.OperationSucceeded);
+            if (snapshotOnPrimaryInfo == null) {
+                s_logger.debug("Snapshot id:" + snapshotId + " not found on primary storage, skipping snapshot deletion on primary and marking it destroyed");
+                snapshotVO.setState(Snapshot.State.Destroyed);
+                snapshotDao.update(snapshotId, snapshotVO);
+                return true;
 
 Review comment:
   no extra return please; this means `deletedOnPrimary = true` and we fall through?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] davidjumani commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

davidjumani commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-605906705
 
 
   @GabrielBrascher That is happening in our environment. However, when I create a first snapshot of a data volume, it also makes a copy in the secondary storage. Successive snapshots do not get created in the secondary storage but when I try to restore them or create a new volume from them it fails, while I can only use the first snapshot that I had created to create a new volume / restore the snapshot.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-599555347
 
 
   Packaging result: ✖centos6 ✔centos7 ✔debian. JID-1056

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] DaanHoogland commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

DaanHoogland commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-599542351
 
 
   @blueorangutan package

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] andrijapanicsb commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

andrijapanicsb commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-609950065
 
 
   Latest testing of the deletion stuff reports below
   
   (previously I have tested normal operations works fine - convert to this to that, revert (KVM only), etc, for all 3 hypervisors)
   
   ## snapshot.backup.to.secondary=FALSE
   
   Expected to work ONLY with KVM+Ceph (not other variants, not other hypervisors)
   (but the basic create/delete snap was tested with KVM+NFS to make sure nothing broken in that specific flow)
   create, delete snap works fine - GC kicks in, no exceptions - after GC runs, the "PRIMARY" (role_type=primary) row in snapshots_store_ref is left undeleted (with other snapshot.backup.to.secondary=TRUE testing as well)
   
   @rafaelweingartner please test with Ceph if possible
   
   ## snapshot.backup.to.secondary=TRUE
   
   - snaps can be created, operated with, deleted fine
   -- deleted from file systems as expected (State changed to Destroyed in snapshots and snapshot_store_ref tables - "removed"date NOT set in the main "snapshots" table
   --- GC will remove the IMAGE row (role_type=image) from the snapshot_store_ref table and set the "removed" date in the main "snapshots" table
   -- GC will NOT remove the PRIMARY row from the snapshot_store_ref table (any of possibly many rows for the same volume (i.e. many different snapss of the same volume)
   -- When a volume (root/data) is removed, then only first/latest (not sure) PRIMARY row in the snapshot_store_ref table- all other PRIMARY rows for a particular volume are left - i.e. DB garbage is left
   
   ### GC garbage workaround
   
   Remove rows from snapshot_store_ref table:
   
   > delete from snapshot_store_ref where snapshot_id in (select id from snapshots where status="Destroyed" and removed is NOT NULL); 
   
   (removed not NULL = means that the GC has run and left the garbage behind)
   
   or the safer one (volume is removed)
   
   > delete from snapshot_store_ref where snapshot_id in (select id from snapshots where status="Destroyed" and removed is NOT NULL) and volume_id in (select id from volumes where state="Expunged" and removed is NOT NULL);

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] andrijapanicsb commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

andrijapanicsb commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-611681037
 
 
   So it does @Slair1 ...

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan removed a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan removed a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-600490266
 
 
   @DaanHoogland a Trillian-Jenkins matrix job (centos7 mgmt + xs71, centos7 mgmt + vmware67, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] DaanHoogland opened a new pull request #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

DaanHoogland opened a new pull request #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969
 
 
   ## Description
   <!--- Describe your changes in detail -->
   copy of PR #3649 on joint account for easier cooperation.
   
   Tested both cases:
   - Snapshot backed up on Secondary (stored on primary and secondary) 
   - Snapshot stored only on primary storage.
   
   This fix works for KVM  and XenServer.
   
   **Note**: I changed the name of the strategy from _XenserverSnapshotStrategy_ to _DefaultSnapshotStrategy_ due to the fact that the "Xenserver" strategy handles also Ceph, KVM, and do return `StrategyPriority.DEFAULT` in any case except _REVERT_.
   
   ```
       @Override
       public StrategyPriority canHandle(Snapshot snapshot, SnapshotOperation op) {
           if (SnapshotOperation.REVERT.equals(op)) {
               long volumeId = snapshot.getVolumeId();
               VolumeVO volumeVO = volumeDao.findById(volumeId);
   
               if (volumeVO != null && ImageFormat.QCOW2.equals(volumeVO.getFormat())) {
                   return StrategyPriority.DEFAULT;
               }
               return StrategyPriority.CANT_HANDLE;
           }
           return StrategyPriority.DEFAULT;
       }
   ```
   
   Fixes: #3646
   
   ## Types of changes
   <!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
   - [ ] Breaking change (fix or feature that would cause existing functionality to change)
   - [ ] New feature (non-breaking change which adds functionality)
   - [x] Bug fix (non-breaking change which fixes an issue)
   - [ ] Enhancement (improves an existing feature and functionality)
   - [ ] Cleanup (Code refactoring and cleanup, that may add test cases)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-611869083
 
 
   <b>Trillian test result (tid-1401)</b>
   Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server 7
   Total time taken: 32576 seconds
   Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3969-t1401-vmware-67u3.zip
   Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
   Intermittent failure detected: /marvin/tests/smoke/test_ssvm.py
   Smoke tests completed. 76 look OK, 1 have error(s)
   Only failed tests results shown below:
   
   
   Test | Result | Time (s) | Test File
   --- | --- | --- | ---
   test_02_vpc_privategw_static_routes | `Failure` | 258.83 | test_privategw_acl.py
   test_03_vpc_privategw_restart_vpc_cleanup | `Failure` | 253.80 | test_privategw_acl.py
   test_04_rvpc_privategw_static_routes | `Failure` | 379.55 | test_privategw_acl.py
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan removed a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan removed a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-611274236
 
 
   <b>Trillian test result (tid-1391)</b>
   Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7
   Total time taken: 57777 seconds
   Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3969-t1391-xenserver-71.zip
   Intermittent failure detected: /marvin/tests/smoke/test_list_ids_parameter.py
   Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
   Intermittent failure detected: /marvin/tests/smoke/test_public_ip_range.py
   Intermittent failure detected: /marvin/tests/smoke/test_reset_vm_on_reboot.py
   Intermittent failure detected: /marvin/tests/smoke/test_resource_accounting.py
   Intermittent failure detected: /marvin/tests/smoke/test_router_dhcphosts.py
   Intermittent failure detected: /marvin/tests/smoke/test_router_dns.py
   Intermittent failure detected: /marvin/tests/smoke/test_router_dnsservice.py
   Intermittent failure detected: /marvin/tests/smoke/test_routers_iptables_default_policy.py
   Intermittent failure detected: /marvin/tests/smoke/test_routers_network_ops.py
   Intermittent failure detected: /marvin/tests/smoke/test_routers.py
   Intermittent failure detected: /marvin/tests/smoke/test_scale_vm.py
   Intermittent failure detected: /marvin/tests/smoke/test_secondary_storage.py
   Intermittent failure detected: /marvin/tests/smoke/test_service_offerings.py
   Intermittent failure detected: /marvin/tests/smoke/test_snapshots.py
   Intermittent failure detected: /marvin/tests/smoke/test_ssvm.py
   Intermittent failure detected: /marvin/tests/smoke/test_templates.py
   Intermittent failure detected: /marvin/tests/smoke/test_usage.py
   Intermittent failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
   Intermittent failure detected: /marvin/tests/smoke/test_vm_snapshots.py
   Intermittent failure detected: /marvin/tests/smoke/test_volumes.py
   Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
   Intermittent failure detected: /marvin/tests/smoke/test_vpc_router_nics.py
   Intermittent failure detected: /marvin/tests/smoke/test_vpc_vpn.py
   Smoke tests completed. 54 look OK, 23 have error(s)
   Only failed tests results shown below:
   
   
   Test | Result | Time (s) | Test File
   --- | --- | --- | ---
   ContextSuite context=TestListIdsParams>:teardown | `Error` | 1.09 | test_list_ids_parameter.py
   test_02_vpc_privategw_static_routes | `Failure` | 265.00 | test_privategw_acl.py
   test_03_vpc_privategw_restart_vpc_cleanup | `Failure` | 280.63 | test_privategw_acl.py
   test_04_rvpc_privategw_static_routes | `Failure` | 322.51 | test_privategw_acl.py
   ContextSuite context=TestResetVmOnReboot>:setup | `Error` | 0.00 | test_reset_vm_on_reboot.py
   ContextSuite context=TestRAMCPUResourceAccounting>:setup | `Error` | 0.00 | test_resource_accounting.py
   ContextSuite context=TestScaleVm>:setup | `Error` | 0.00 | test_scale_vm.py
   ContextSuite context=TestRouterDHCPHosts>:setup | `Error` | 0.00 | test_router_dhcphosts.py
   ContextSuite context=TestRouterDHCPOpts>:setup | `Error` | 0.00 | test_router_dhcphosts.py
   ContextSuite context=TestRouterDns>:setup | `Error` | 0.00 | test_router_dns.py
   ContextSuite context=TestRouterDnsService>:setup | `Error` | 0.00 | test_router_dnsservice.py
   ContextSuite context=TestRouterIpTablesPolicies>:setup | `Error` | 0.00 | test_routers_iptables_default_policy.py
   ContextSuite context=TestVPCIpTablesPolicies>:setup | `Error` | 0.00 | test_routers_iptables_default_policy.py
   ContextSuite context=TestIsolatedNetworks>:setup | `Error` | 0.00 | test_routers_network_ops.py
   ContextSuite context=TestRedundantIsolateNetworks>:setup | `Error` | 0.00 | test_routers_network_ops.py
   ContextSuite context=TestRouterServices>:setup | `Error` | 0.00 | test_routers.py
   test_01_sys_vm_start | `Failure` | 0.07 | test_secondary_storage.py
   ContextSuite context=TestServiceOfferings>:setup | `Error` | 1511.31 | test_service_offerings.py
   ContextSuite context=TestSnapshotRootDisk>:setup | `Error` | 0.00 | test_snapshots.py
   test_01_list_sec_storage_vm | `Failure` | 0.03 | test_ssvm.py
   test_02_list_cpvm_vm | `Failure` | 0.02 | test_ssvm.py
   test_03_ssvm_internals | `Failure` | 0.03 | test_ssvm.py
   test_04_cpvm_internals | `Failure` | 0.02 | test_ssvm.py
   test_05_stop_ssvm | `Failure` | 0.02 | test_ssvm.py
   test_06_stop_cpvm | `Failure` | 0.02 | test_ssvm.py
   test_07_reboot_ssvm | `Failure` | 0.02 | test_ssvm.py
   test_08_reboot_cpvm | `Failure` | 0.02 | test_ssvm.py
   test_09_destroy_ssvm | `Failure` | 0.02 | test_ssvm.py
   test_10_destroy_cpvm | `Failure` | 0.03 | test_ssvm.py
   test_02_create_template_with_checksum_sha1 | `Error` | 65.35 | test_templates.py
   test_03_create_template_with_checksum_sha256 | `Error` | 65.36 | test_templates.py
   test_04_create_template_with_checksum_md5 | `Error` | 65.34 | test_templates.py
   test_05_create_template_with_no_checksum | `Error` | 65.35 | test_templates.py
   ContextSuite context=TestTemplates>:setup | `Error` | 1.94 | test_templates.py
   ContextSuite context=TestISOUsage>:setup | `Error` | 0.00 | test_usage.py
   ContextSuite context=TestLBRuleUsage>:setup | `Error` | 0.00 | test_usage.py
   ContextSuite context=TestNatRuleUsage>:setup | `Error` | 0.00 | test_usage.py
   ContextSuite context=TestPublicIPUsage>:setup | `Error` | 0.00 | test_usage.py
   ContextSuite context=TestSnapshotUsage>:setup | `Error` | 0.00 | test_usage.py
   ContextSuite context=TestVmUsage>:setup | `Error` | 0.00 | test_usage.py
   ContextSuite context=TestVolumeUsage>:setup | `Error` | 0.00 | test_usage.py
   ContextSuite context=TestVpnUsage>:setup | `Error` | 0.00 | test_usage.py
   ContextSuite context=Test01DeployVM>:setup | `Error` | 0.00 | test_vm_life_cycle.py
   ContextSuite context=Test02VMLifeCycle>:setup | `Error` | 0.00 | test_vm_life_cycle.py
   test_change_service_offering_for_vm_with_snapshots | `Error` | 1.18 | test_vm_snapshots.py
   ContextSuite context=TestVmSnapshot>:setup | `Error` | 8.95 | test_vm_snapshots.py
   ContextSuite context=TestCreateVolume>:setup | `Error` | 0.00 | test_volumes.py
   ContextSuite context=TestVolumes>:setup | `Error` | 0.00 | test_volumes.py
   ContextSuite context=TestVPCRedundancy>:setup | `Error` | 0.00 | test_vpc_redundant.py
   ContextSuite context=TestVPCNics>:setup | `Error` | 0.00 | test_vpc_router_nics.py
   ContextSuite context=TestRVPCSite2SiteVpn>:setup | `Error` | 0.00 | test_vpc_vpn.py
   ContextSuite context=TestVPCSite2SiteVPNMultipleOptions>:setup | `Error` | 0.00 | test_vpc_vpn.py
   ContextSuite context=TestVpcRemoteAccessVpn>:setup | `Error` | 0.00 | test_vpc_vpn.py
   ContextSuite context=TestVpcSite2SiteVpn>:setup | `Error` | 0.00 | test_vpc_vpn.py
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] andrijapanicsb commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

andrijapanicsb commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-611703714
 
 
   @Slair1 I can't, I'm off as from 10min ago (late night, time for beer) - perhaps @DaanHoogland can confirm once more tomorrow....

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-610700547
 
 
   <b>Trillian test result (tid-1385)</b>
   Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server 7
   Total time taken: 39186 seconds
   Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3969-t1385-vmware-67u3.zip
   Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
   Intermittent failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
   Smoke tests completed. 75 look OK, 2 have error(s)
   Only failed tests results shown below:
   
   
   Test | Result | Time (s) | Test File
   --- | --- | --- | ---
   test_02_vpc_privategw_static_routes | `Failure` | 317.48 | test_privategw_acl.py
   test_03_vpc_privategw_restart_vpc_cleanup | `Failure` | 308.34 | test_privategw_acl.py
   test_04_rvpc_privategw_static_routes | `Failure` | 519.60 | test_privategw_acl.py
   test_11_migrate_vm | `Error` | 55.87 | test_vm_life_cycle.py
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] andrijapanicsb commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

andrijapanicsb commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-606129284
 
 
   @blueorangutan test matrix

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] DaanHoogland closed pull request #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

DaanHoogland closed pull request #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] DaanHoogland commented on a change in pull request #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

DaanHoogland commented on a change in pull request #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#discussion_r404344872
 
 

 ##########
 File path: engine/storage/snapshot/src/main/java/org/apache/cloudstack/storage/snapshot/DefaultSnapshotStrategy.java
 ##########
 @@ -257,21 +257,23 @@ public boolean deleteSnapshot(Long snapshotId) {
             return true;
         }
 
-        if (!Snapshot.State.BackedUp.equals(snapshotVO.getState()) && !Snapshot.State.Error.equals(snapshotVO.getState()) &&
+        if (!Snapshot.State.BackedUp.equals(snapshotVO.getState()) &&
                 !Snapshot.State.Destroying.equals(snapshotVO.getState())) {
             throw new InvalidParameterValueException("Can't delete snapshotshot " + snapshotId + " due to it is in " + snapshotVO.getState() + " Status");
         }
 
-        boolean deletedOnSecondary = deleteOnSecondaryIfNeeded(snapshotId);
+        Boolean deletedOnSecondary = deleteOnSecondaryIfNeeded(snapshotId);
         boolean deletedOnPrimary = deleteOnPrimaryIfNeeded(snapshotId);
 
         if (deletedOnPrimary) {
             s_logger.debug(String.format("Successfully deleted snapshot (id: %d) on primary storage.", snapshotId));
-        } else if (deletedOnSecondary) {
-            s_logger.debug(String.format("The snapshot was deleted on secondary storage. Could not find/delete snapshot (id: %d) on primary storage.", snapshotId));
+        } else {
+            s_logger.debug(String.format("The snapshot (id: %d) could not be found/deleted on primary storage.", snapshotId));
         }
-
-        return deletedOnSecondary || deletedOnPrimary;
+        if (null == deletedOnSecondary && deletedOnSecondary) {
 
 Review comment:
   yes should be !=, tnx (it's late)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-606129630
 
 
   @andrijapanicsb a Trillian-Jenkins matrix job (centos7 mgmt + xs71, centos7 mgmt + vmware67, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] andrijapanicsb commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

andrijapanicsb commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-607219803
 
 
   @blueorangutan package

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan removed a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan removed a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-600858038
 
 
   Packaging result: ✖centos6 ✔centos7 ✔debian. JID-1072

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] andrijapanicsb removed a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

andrijapanicsb removed a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-606129284
 
 
   @blueorangutan test matrix

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] DaanHoogland edited a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

DaanHoogland edited a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-599556425
 
 
   using KVM/NFS, I tested this scenario:
   
   1. created VM with data disk
   2. both disks snapshotted
   3. both deleted
   4. both marked deleted in the DB and removed secondary
   5. snapshot_store_ref has entries for both disks on primary and on secondary, all four are marked 'Destroyed'.
   6. no changes in primary everything had already been clean (not sure if the snapshots were removed before)
   
   basically different results than reported by @GabrielBrascher .
   
   this is with @GabrielBrascher 's fix in this PR.
   that would mean we are good to go, but it could also mean there are unknown circumstances that create garbage. (cc @andrijapanicsb )

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan removed a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan removed a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-612000101
 
 
   <b>Trillian test result (tid-1400)</b>
   Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
   Total time taken: 60513 seconds
   Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3969-t1400-kvm-centos7.zip
   Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
   Intermittent failure detected: /marvin/tests/smoke/test_public_ip_range.py
   Intermittent failure detected: /marvin/tests/smoke/test_reset_vm_on_reboot.py
   Intermittent failure detected: /marvin/tests/smoke/test_resource_accounting.py
   Intermittent failure detected: /marvin/tests/smoke/test_router_dhcphosts.py
   Intermittent failure detected: /marvin/tests/smoke/test_router_dns.py
   Intermittent failure detected: /marvin/tests/smoke/test_router_dnsservice.py
   Intermittent failure detected: /marvin/tests/smoke/test_routers_iptables_default_policy.py
   Intermittent failure detected: /marvin/tests/smoke/test_routers_network_ops.py
   Intermittent failure detected: /marvin/tests/smoke/test_routers.py
   Intermittent failure detected: /marvin/tests/smoke/test_secondary_storage.py
   Intermittent failure detected: /marvin/tests/smoke/test_service_offerings.py
   Intermittent failure detected: /marvin/tests/smoke/test_snapshots.py
   Intermittent failure detected: /marvin/tests/smoke/test_ssvm.py
   Intermittent failure detected: /marvin/tests/smoke/test_templates.py
   Intermittent failure detected: /marvin/tests/smoke/test_usage.py
   Intermittent failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
   Intermittent failure detected: /marvin/tests/smoke/test_vm_snapshots.py
   Intermittent failure detected: /marvin/tests/smoke/test_volumes.py
   Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
   Intermittent failure detected: /marvin/tests/smoke/test_vpc_router_nics.py
   Intermittent failure detected: /marvin/tests/smoke/test_vpc_vpn.py
   Intermittent failure detected: /marvin/tests/smoke/test_hostha_kvm.py
   Smoke tests completed. 55 look OK, 22 have error(s)
   Only failed tests results shown below:
   
   
   Test | Result | Time (s) | Test File
   --- | --- | --- | ---
   test_02_vpc_privategw_static_routes | `Failure` | 182.50 | test_privategw_acl.py
   test_03_vpc_privategw_restart_vpc_cleanup | `Failure` | 181.23 | test_privategw_acl.py
   test_04_rvpc_privategw_static_routes | `Failure` | 259.01 | test_privategw_acl.py
   ContextSuite context=TestResetVmOnReboot>:setup | `Error` | 0.00 | test_reset_vm_on_reboot.py
   ContextSuite context=TestRAMCPUResourceAccounting>:setup | `Error` | 0.00 | test_resource_accounting.py
   ContextSuite context=TestRouterDHCPHosts>:setup | `Error` | 0.00 | test_router_dhcphosts.py
   ContextSuite context=TestRouterDHCPOpts>:setup | `Error` | 0.00 | test_router_dhcphosts.py
   ContextSuite context=TestRouterDns>:setup | `Error` | 0.00 | test_router_dns.py
   test_01_sys_vm_start | `Failure` | 0.07 | test_secondary_storage.py
   ContextSuite context=TestRouterDnsService>:setup | `Error` | 0.00 | test_router_dnsservice.py
   ContextSuite context=TestRouterIpTablesPolicies>:setup | `Error` | 0.00 | test_routers_iptables_default_policy.py
   ContextSuite context=TestVPCIpTablesPolicies>:setup | `Error` | 0.00 | test_routers_iptables_default_policy.py
   ContextSuite context=TestIsolatedNetworks>:setup | `Error` | 0.00 | test_routers_network_ops.py
   ContextSuite context=TestRedundantIsolateNetworks>:setup | `Error` | 0.00 | test_routers_network_ops.py
   ContextSuite context=TestRouterServices>:setup | `Error` | 0.00 | test_routers.py
   ContextSuite context=TestCpuCapServiceOfferings>:setup | `Error` | 0.00 | test_service_offerings.py
   ContextSuite context=TestServiceOfferings>:setup | `Error` | 0.17 | test_service_offerings.py
   ContextSuite context=TestSnapshotRootDisk>:setup | `Error` | 0.00 | test_snapshots.py
   test_01_list_sec_storage_vm | `Failure` | 0.03 | test_ssvm.py
   test_02_list_cpvm_vm | `Failure` | 0.03 | test_ssvm.py
   test_03_ssvm_internals | `Failure` | 0.03 | test_ssvm.py
   test_04_cpvm_internals | `Failure` | 0.03 | test_ssvm.py
   test_05_stop_ssvm | `Failure` | 0.03 | test_ssvm.py
   test_06_stop_cpvm | `Failure` | 0.02 | test_ssvm.py
   test_07_reboot_ssvm | `Failure` | 0.03 | test_ssvm.py
   test_08_reboot_cpvm | `Failure` | 0.03 | test_ssvm.py
   test_09_destroy_ssvm | `Failure` | 0.02 | test_ssvm.py
   test_10_destroy_cpvm | `Failure` | 0.03 | test_ssvm.py
   test_02_create_template_with_checksum_sha1 | `Error` | 65.37 | test_templates.py
   test_03_create_template_with_checksum_sha256 | `Error` | 65.34 | test_templates.py
   test_04_create_template_with_checksum_md5 | `Error` | 65.36 | test_templates.py
   test_05_create_template_with_no_checksum | `Error` | 65.36 | test_templates.py
   test_02_deploy_vm_from_direct_download_template | `Error` | 1.19 | test_templates.py
   test_03_deploy_vm_wrong_checksum | `Error` | 1.24 | test_templates.py
   ContextSuite context=TestTemplates>:setup | `Error` | 16.56 | test_templates.py
   ContextSuite context=TestISOUsage>:setup | `Error` | 0.00 | test_usage.py
   ContextSuite context=TestLBRuleUsage>:setup | `Error` | 0.00 | test_usage.py
   ContextSuite context=TestNatRuleUsage>:setup | `Error` | 0.00 | test_usage.py
   ContextSuite context=TestPublicIPUsage>:setup | `Error` | 0.00 | test_usage.py
   ContextSuite context=TestSnapshotUsage>:setup | `Error` | 0.00 | test_usage.py
   ContextSuite context=TestVmUsage>:setup | `Error` | 0.00 | test_usage.py
   ContextSuite context=TestVolumeUsage>:setup | `Error` | 0.00 | test_usage.py
   ContextSuite context=TestVpnUsage>:setup | `Error` | 0.00 | test_usage.py
   ContextSuite context=Test01DeployVM>:setup | `Error` | 0.00 | test_vm_life_cycle.py
   ContextSuite context=Test02VMLifeCycle>:setup | `Error` | 0.00 | test_vm_life_cycle.py
   test_14_secure_to_secure_vm_migration | `Error` | 11.28 | test_vm_life_cycle.py
   test_15_secured_to_nonsecured_vm_migration | `Error` | 73.93 | test_vm_life_cycle.py
   test_16_nonsecured_to_secured_vm_migration | `Error` | 1.20 | test_vm_life_cycle.py
   ContextSuite context=TestVmSnapshot>:setup | `Error` | 1.57 | test_vm_snapshots.py
   ContextSuite context=TestCreateVolume>:setup | `Error` | 0.00 | test_volumes.py
   ContextSuite context=TestVolumes>:setup | `Error` | 0.00 | test_volumes.py
   ContextSuite context=TestVPCRedundancy>:setup | `Error` | 0.00 | test_vpc_redundant.py
   ContextSuite context=TestVPCNics>:setup | `Error` | 0.00 | test_vpc_router_nics.py
   ContextSuite context=TestRVPCSite2SiteVpn>:setup | `Error` | 0.00 | test_vpc_vpn.py
   ContextSuite context=TestVPCSite2SiteVPNMultipleOptions>:setup | `Error` | 0.00 | test_vpc_vpn.py
   ContextSuite context=TestVpcRemoteAccessVpn>:setup | `Error` | 0.00 | test_vpc_vpn.py
   ContextSuite context=TestVpcSite2SiteVpn>:setup | `Error` | 0.00 | test_vpc_vpn.py
   test_disable_oobm_ha_state_ineligible | `Error` | 1512.14 | test_hostha_kvm.py
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-611444146
 
 
   <b>Trillian test result (tid-1393)</b>
   Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server 7
   Total time taken: 34628 seconds
   Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3969-t1393-vmware-67u3.zip
   Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
   Smoke tests completed. 76 look OK, 1 have error(s)
   Only failed tests results shown below:
   
   
   Test | Result | Time (s) | Test File
   --- | --- | --- | ---
   test_02_vpc_privategw_static_routes | `Failure` | 277.12 | test_privategw_acl.py
   test_03_vpc_privategw_restart_vpc_cleanup | `Failure` | 259.81 | test_privategw_acl.py
   test_04_rvpc_privategw_static_routes | `Failure` | 399.76 | test_privategw_acl.py
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-611688665
 
 
   @andrijapanicsb a Trillian-Jenkins matrix job (centos7 mgmt + xs71, centos7 mgmt + vmware67, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] andrijapanicsb commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

andrijapanicsb commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-611688138
 
 
   Ape, do your thing!  @blueorangutan test matrix

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] DaanHoogland commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

DaanHoogland commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-600174457
 
 
   @blueorangutan package

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-611274236
 
 
   <b>Trillian test result (tid-1391)</b>
   Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7
   Total time taken: 57777 seconds
   Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3969-t1391-xenserver-71.zip
   Intermittent failure detected: /marvin/tests/smoke/test_list_ids_parameter.py
   Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
   Intermittent failure detected: /marvin/tests/smoke/test_public_ip_range.py
   Intermittent failure detected: /marvin/tests/smoke/test_reset_vm_on_reboot.py
   Intermittent failure detected: /marvin/tests/smoke/test_resource_accounting.py
   Intermittent failure detected: /marvin/tests/smoke/test_router_dhcphosts.py
   Intermittent failure detected: /marvin/tests/smoke/test_router_dns.py
   Intermittent failure detected: /marvin/tests/smoke/test_router_dnsservice.py
   Intermittent failure detected: /marvin/tests/smoke/test_routers_iptables_default_policy.py
   Intermittent failure detected: /marvin/tests/smoke/test_routers_network_ops.py
   Intermittent failure detected: /marvin/tests/smoke/test_routers.py
   Intermittent failure detected: /marvin/tests/smoke/test_scale_vm.py
   Intermittent failure detected: /marvin/tests/smoke/test_secondary_storage.py
   Intermittent failure detected: /marvin/tests/smoke/test_service_offerings.py
   Intermittent failure detected: /marvin/tests/smoke/test_snapshots.py
   Intermittent failure detected: /marvin/tests/smoke/test_ssvm.py
   Intermittent failure detected: /marvin/tests/smoke/test_templates.py
   Intermittent failure detected: /marvin/tests/smoke/test_usage.py
   Intermittent failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
   Intermittent failure detected: /marvin/tests/smoke/test_vm_snapshots.py
   Intermittent failure detected: /marvin/tests/smoke/test_volumes.py
   Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
   Intermittent failure detected: /marvin/tests/smoke/test_vpc_router_nics.py
   Intermittent failure detected: /marvin/tests/smoke/test_vpc_vpn.py
   Smoke tests completed. 54 look OK, 23 have error(s)
   Only failed tests results shown below:
   
   
   Test | Result | Time (s) | Test File
   --- | --- | --- | ---
   ContextSuite context=TestListIdsParams>:teardown | `Error` | 1.09 | test_list_ids_parameter.py
   test_02_vpc_privategw_static_routes | `Failure` | 265.00 | test_privategw_acl.py
   test_03_vpc_privategw_restart_vpc_cleanup | `Failure` | 280.63 | test_privategw_acl.py
   test_04_rvpc_privategw_static_routes | `Failure` | 322.51 | test_privategw_acl.py
   ContextSuite context=TestResetVmOnReboot>:setup | `Error` | 0.00 | test_reset_vm_on_reboot.py
   ContextSuite context=TestRAMCPUResourceAccounting>:setup | `Error` | 0.00 | test_resource_accounting.py
   ContextSuite context=TestScaleVm>:setup | `Error` | 0.00 | test_scale_vm.py
   ContextSuite context=TestRouterDHCPHosts>:setup | `Error` | 0.00 | test_router_dhcphosts.py
   ContextSuite context=TestRouterDHCPOpts>:setup | `Error` | 0.00 | test_router_dhcphosts.py
   ContextSuite context=TestRouterDns>:setup | `Error` | 0.00 | test_router_dns.py
   ContextSuite context=TestRouterDnsService>:setup | `Error` | 0.00 | test_router_dnsservice.py
   ContextSuite context=TestRouterIpTablesPolicies>:setup | `Error` | 0.00 | test_routers_iptables_default_policy.py
   ContextSuite context=TestVPCIpTablesPolicies>:setup | `Error` | 0.00 | test_routers_iptables_default_policy.py
   ContextSuite context=TestIsolatedNetworks>:setup | `Error` | 0.00 | test_routers_network_ops.py
   ContextSuite context=TestRedundantIsolateNetworks>:setup | `Error` | 0.00 | test_routers_network_ops.py
   ContextSuite context=TestRouterServices>:setup | `Error` | 0.00 | test_routers.py
   test_01_sys_vm_start | `Failure` | 0.07 | test_secondary_storage.py
   ContextSuite context=TestServiceOfferings>:setup | `Error` | 1511.31 | test_service_offerings.py
   ContextSuite context=TestSnapshotRootDisk>:setup | `Error` | 0.00 | test_snapshots.py
   test_01_list_sec_storage_vm | `Failure` | 0.03 | test_ssvm.py
   test_02_list_cpvm_vm | `Failure` | 0.02 | test_ssvm.py
   test_03_ssvm_internals | `Failure` | 0.03 | test_ssvm.py
   test_04_cpvm_internals | `Failure` | 0.02 | test_ssvm.py
   test_05_stop_ssvm | `Failure` | 0.02 | test_ssvm.py
   test_06_stop_cpvm | `Failure` | 0.02 | test_ssvm.py
   test_07_reboot_ssvm | `Failure` | 0.02 | test_ssvm.py
   test_08_reboot_cpvm | `Failure` | 0.02 | test_ssvm.py
   test_09_destroy_ssvm | `Failure` | 0.02 | test_ssvm.py
   test_10_destroy_cpvm | `Failure` | 0.03 | test_ssvm.py
   test_02_create_template_with_checksum_sha1 | `Error` | 65.35 | test_templates.py
   test_03_create_template_with_checksum_sha256 | `Error` | 65.36 | test_templates.py
   test_04_create_template_with_checksum_md5 | `Error` | 65.34 | test_templates.py
   test_05_create_template_with_no_checksum | `Error` | 65.35 | test_templates.py
   ContextSuite context=TestTemplates>:setup | `Error` | 1.94 | test_templates.py
   ContextSuite context=TestISOUsage>:setup | `Error` | 0.00 | test_usage.py
   ContextSuite context=TestLBRuleUsage>:setup | `Error` | 0.00 | test_usage.py
   ContextSuite context=TestNatRuleUsage>:setup | `Error` | 0.00 | test_usage.py
   ContextSuite context=TestPublicIPUsage>:setup | `Error` | 0.00 | test_usage.py
   ContextSuite context=TestSnapshotUsage>:setup | `Error` | 0.00 | test_usage.py
   ContextSuite context=TestVmUsage>:setup | `Error` | 0.00 | test_usage.py
   ContextSuite context=TestVolumeUsage>:setup | `Error` | 0.00 | test_usage.py
   ContextSuite context=TestVpnUsage>:setup | `Error` | 0.00 | test_usage.py
   ContextSuite context=Test01DeployVM>:setup | `Error` | 0.00 | test_vm_life_cycle.py
   ContextSuite context=Test02VMLifeCycle>:setup | `Error` | 0.00 | test_vm_life_cycle.py
   test_change_service_offering_for_vm_with_snapshots | `Error` | 1.18 | test_vm_snapshots.py
   ContextSuite context=TestVmSnapshot>:setup | `Error` | 8.95 | test_vm_snapshots.py
   ContextSuite context=TestCreateVolume>:setup | `Error` | 0.00 | test_volumes.py
   ContextSuite context=TestVolumes>:setup | `Error` | 0.00 | test_volumes.py
   ContextSuite context=TestVPCRedundancy>:setup | `Error` | 0.00 | test_vpc_redundant.py
   ContextSuite context=TestVPCNics>:setup | `Error` | 0.00 | test_vpc_router_nics.py
   ContextSuite context=TestRVPCSite2SiteVpn>:setup | `Error` | 0.00 | test_vpc_vpn.py
   ContextSuite context=TestVPCSite2SiteVPNMultipleOptions>:setup | `Error` | 0.00 | test_vpc_vpn.py
   ContextSuite context=TestVpcRemoteAccessVpn>:setup | `Error` | 0.00 | test_vpc_vpn.py
   ContextSuite context=TestVpcSite2SiteVpn>:setup | `Error` | 0.00 | test_vpc_vpn.py
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] andrijapanicsb commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

andrijapanicsb commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-609044924
 
 
   @blueorangutan test

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] andrijapanicsb commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

andrijapanicsb commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-613655243
 
 
   tested, confirmed the issue with latest centos7 host (centos5.5 ext3 guest - simulated high IO load while deleting the snap and broke the VM)
   
   Raised PR (for master only) https://github.com/apache/cloudstack/pull/4029

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan removed a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan removed a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-609038645
 
 
   @nvazquez a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] andrijapanicsb commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

andrijapanicsb commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-613054117
 
 
   I don't see it myself, despite @DaanHoogland confirming the commit being present in 4.13 branch... Some regression by someone? 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan removed a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan removed a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-610224060
 
 
   @DaanHoogland a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan removed a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan removed a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-610701895
 
 
   <b>Trillian test result (tid-1386)</b>
   Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7
   Total time taken: 40680 seconds
   Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3969-t1386-xenserver-71.zip
   Intermittent failure detected: /marvin/tests/smoke/test_iso.py
   Intermittent failure detected: /marvin/tests/smoke/test_list_ids_parameter.py
   Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
   Intermittent failure detected: /marvin/tests/smoke/test_scale_vm.py
   Intermittent failure detected: /marvin/tests/smoke/test_vpc_vpn.py
   Smoke tests completed. 74 look OK, 3 have error(s)
   Only failed tests results shown below:
   
   
   Test | Result | Time (s) | Test File
   --- | --- | --- | ---
   ContextSuite context=TestListIdsParams>:teardown | `Error` | 1.09 | test_list_ids_parameter.py
   test_02_vpc_privategw_static_routes | `Failure` | 361.62 | test_privategw_acl.py
   test_03_vpc_privategw_restart_vpc_cleanup | `Failure` | 293.85 | test_privategw_acl.py
   test_04_rvpc_privategw_static_routes | `Failure` | 469.87 | test_privategw_acl.py
   test_01_scale_vm | `Failure` | 33.81 | test_scale_vm.py
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan removed a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan removed a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-599590064
 
 
   Packaging result: ✖centos6 ✔centos7 ✔debian. JID-1057

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-611437737
 
 
   Packaging result: ✔centos7 ✔debian. JID-1158

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan removed a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan removed a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-599555347
 
 
   Packaging result: ✖centos6 ✔centos7 ✔debian. JID-1056

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-609999930
 
 
   @DaanHoogland a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] GabrielBrascher commented on a change in pull request #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

GabrielBrascher commented on a change in pull request #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#discussion_r404319082
 
 

 ##########
 File path: engine/storage/snapshot/src/main/java/org/apache/cloudstack/storage/snapshot/DefaultSnapshotStrategy.java
 ##########
 @@ -257,21 +257,23 @@ public boolean deleteSnapshot(Long snapshotId) {
             return true;
         }
 
-        if (!Snapshot.State.BackedUp.equals(snapshotVO.getState()) && !Snapshot.State.Error.equals(snapshotVO.getState()) &&
+        if (!Snapshot.State.BackedUp.equals(snapshotVO.getState()) &&
                 !Snapshot.State.Destroying.equals(snapshotVO.getState())) {
             throw new InvalidParameterValueException("Can't delete snapshotshot " + snapshotId + " due to it is in " + snapshotVO.getState() + " Status");
         }
 
-        boolean deletedOnSecondary = deleteOnSecondaryIfNeeded(snapshotId);
+        Boolean deletedOnSecondary = deleteOnSecondaryIfNeeded(snapshotId);
         boolean deletedOnPrimary = deleteOnPrimaryIfNeeded(snapshotId);
 
         if (deletedOnPrimary) {
             s_logger.debug(String.format("Successfully deleted snapshot (id: %d) on primary storage.", snapshotId));
-        } else if (deletedOnSecondary) {
-            s_logger.debug(String.format("The snapshot was deleted on secondary storage. Could not find/delete snapshot (id: %d) on primary storage.", snapshotId));
+        } else {
+            s_logger.debug(String.format("The snapshot (id: %d) could not be found/deleted on primary storage.", snapshotId));
         }
-
-        return deletedOnSecondary || deletedOnPrimary;
+        if (null == deletedOnSecondary && deletedOnSecondary) {
 
 Review comment:
   I guess that instead of `null == deletedOnSecondary && deletedOnSecondary` the it should be `null == deletedOnPrimary && deletedOnSecondary`, right? Or `deletedOnSecondary != null && deletedOnSecondary`.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] harikrishna-patnala commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

harikrishna-patnala commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-610234553
 
 
   > @GabrielBrascher @andrijapanicsb (@harikrishna-patnala ) that last commit seems to be a state never tested. I don't think the code allows or should allow for a NoSuchTransitionException so I do not know how we can force it to happen.
   
   NoSuchTransitionException won't occur at all if all state transitions are defined properly. So only way I can think of testing is in dev testing that too by changing the state intermediately on setting breakpoints. 
   Also here we are not doing anything in the code, returning null or returning false does the same logging.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] andrijapanicsb removed a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

andrijapanicsb removed a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-600847619
 
 
   @blueorangutan package

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] DaanHoogland removed a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

DaanHoogland removed a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-600489931
 
 
   @blueorangutan test matrix

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] DaanHoogland closed pull request #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

DaanHoogland closed pull request #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-600185007
 
 
   Packaging result: ✖centos6 ✔centos7 ✔debian. JID-1068

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] andrijapanicsb commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

andrijapanicsb commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-611704299
 
 
   Is this for volume or VM snapshots @Slair1 ? - as volume snapshots are NOT possible for KVM while the VM is running (someone thought that's interesting to do that, due to qcow2 safety...so now it's useless for KVM...)
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan removed a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan removed a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-609999930
 
 
   @DaanHoogland a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-609729679
 
 
   Packaging result: ✔centos7 ✔debian. JID-1128

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-611990883
 
 
   <b>Trillian test result (tid-1399)</b>
   Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7
   Total time taken: 58379 seconds
   Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3969-t1399-xenserver-71.zip
   Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
   Intermittent failure detected: /marvin/tests/smoke/test_public_ip_range.py
   Intermittent failure detected: /marvin/tests/smoke/test_reset_vm_on_reboot.py
   Intermittent failure detected: /marvin/tests/smoke/test_resource_accounting.py
   Intermittent failure detected: /marvin/tests/smoke/test_router_dhcphosts.py
   Intermittent failure detected: /marvin/tests/smoke/test_router_dns.py
   Intermittent failure detected: /marvin/tests/smoke/test_router_dnsservice.py
   Intermittent failure detected: /marvin/tests/smoke/test_routers_iptables_default_policy.py
   Intermittent failure detected: /marvin/tests/smoke/test_routers_network_ops.py
   Intermittent failure detected: /marvin/tests/smoke/test_routers.py
   Intermittent failure detected: /marvin/tests/smoke/test_scale_vm.py
   Intermittent failure detected: /marvin/tests/smoke/test_secondary_storage.py
   Intermittent failure detected: /marvin/tests/smoke/test_service_offerings.py
   Intermittent failure detected: /marvin/tests/smoke/test_snapshots.py
   Intermittent failure detected: /marvin/tests/smoke/test_ssvm.py
   Intermittent failure detected: /marvin/tests/smoke/test_templates.py
   Intermittent failure detected: /marvin/tests/smoke/test_usage.py
   Intermittent failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
   Intermittent failure detected: /marvin/tests/smoke/test_vm_snapshots.py
   Intermittent failure detected: /marvin/tests/smoke/test_volumes.py
   Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
   Intermittent failure detected: /marvin/tests/smoke/test_vpc_router_nics.py
   Intermittent failure detected: /marvin/tests/smoke/test_vpc_vpn.py
   Smoke tests completed. 55 look OK, 22 have error(s)
   Only failed tests results shown below:
   
   
   Test | Result | Time (s) | Test File
   --- | --- | --- | ---
   test_02_vpc_privategw_static_routes | `Failure` | 251.57 | test_privategw_acl.py
   test_03_vpc_privategw_restart_vpc_cleanup | `Failure` | 217.06 | test_privategw_acl.py
   test_04_rvpc_privategw_static_routes | `Failure` | 326.43 | test_privategw_acl.py
   ContextSuite context=TestResetVmOnReboot>:setup | `Error` | 0.00 | test_reset_vm_on_reboot.py
   ContextSuite context=TestRAMCPUResourceAccounting>:setup | `Error` | 0.00 | test_resource_accounting.py
   ContextSuite context=TestScaleVm>:setup | `Error` | 0.00 | test_scale_vm.py
   ContextSuite context=TestRouterDHCPHosts>:setup | `Error` | 0.00 | test_router_dhcphosts.py
   ContextSuite context=TestRouterDHCPOpts>:setup | `Error` | 0.00 | test_router_dhcphosts.py
   ContextSuite context=TestRouterDns>:setup | `Error` | 0.00 | test_router_dns.py
   ContextSuite context=TestRouterDnsService>:setup | `Error` | 0.00 | test_router_dnsservice.py
   ContextSuite context=TestRouterIpTablesPolicies>:setup | `Error` | 0.00 | test_routers_iptables_default_policy.py
   ContextSuite context=TestVPCIpTablesPolicies>:setup | `Error` | 0.00 | test_routers_iptables_default_policy.py
   ContextSuite context=TestIsolatedNetworks>:setup | `Error` | 0.00 | test_routers_network_ops.py
   ContextSuite context=TestRedundantIsolateNetworks>:setup | `Error` | 0.00 | test_routers_network_ops.py
   ContextSuite context=TestRouterServices>:setup | `Error` | 0.00 | test_routers.py
   test_01_sys_vm_start | `Failure` | 0.07 | test_secondary_storage.py
   ContextSuite context=TestServiceOfferings>:setup | `Error` | 1511.89 | test_service_offerings.py
   ContextSuite context=TestSnapshotRootDisk>:setup | `Error` | 0.00 | test_snapshots.py
   test_01_list_sec_storage_vm | `Failure` | 0.03 | test_ssvm.py
   test_02_list_cpvm_vm | `Failure` | 0.02 | test_ssvm.py
   test_03_ssvm_internals | `Failure` | 0.02 | test_ssvm.py
   test_04_cpvm_internals | `Failure` | 0.02 | test_ssvm.py
   test_05_stop_ssvm | `Failure` | 0.02 | test_ssvm.py
   test_06_stop_cpvm | `Failure` | 0.02 | test_ssvm.py
   test_07_reboot_ssvm | `Failure` | 0.03 | test_ssvm.py
   test_08_reboot_cpvm | `Failure` | 0.02 | test_ssvm.py
   test_09_destroy_ssvm | `Failure` | 0.02 | test_ssvm.py
   test_10_destroy_cpvm | `Failure` | 0.02 | test_ssvm.py
   test_02_create_template_with_checksum_sha1 | `Error` | 65.38 | test_templates.py
   test_03_create_template_with_checksum_sha256 | `Error` | 65.38 | test_templates.py
   test_04_create_template_with_checksum_md5 | `Error` | 65.40 | test_templates.py
   test_05_create_template_with_no_checksum | `Error` | 65.43 | test_templates.py
   ContextSuite context=TestTemplates>:setup | `Error` | 1.96 | test_templates.py
   ContextSuite context=TestISOUsage>:setup | `Error` | 0.00 | test_usage.py
   ContextSuite context=TestLBRuleUsage>:setup | `Error` | 0.00 | test_usage.py
   ContextSuite context=TestNatRuleUsage>:setup | `Error` | 0.00 | test_usage.py
   ContextSuite context=TestPublicIPUsage>:setup | `Error` | 0.00 | test_usage.py
   ContextSuite context=TestSnapshotUsage>:setup | `Error` | 0.00 | test_usage.py
   ContextSuite context=TestVmUsage>:setup | `Error` | 0.00 | test_usage.py
   ContextSuite context=TestVolumeUsage>:setup | `Error` | 0.00 | test_usage.py
   ContextSuite context=TestVpnUsage>:setup | `Error` | 0.00 | test_usage.py
   ContextSuite context=Test01DeployVM>:setup | `Error` | 0.00 | test_vm_life_cycle.py
   ContextSuite context=Test02VMLifeCycle>:setup | `Error` | 0.00 | test_vm_life_cycle.py
   test_change_service_offering_for_vm_with_snapshots | `Error` | 1.19 | test_vm_snapshots.py
   ContextSuite context=TestVmSnapshot>:setup | `Error` | 8.99 | test_vm_snapshots.py
   ContextSuite context=TestCreateVolume>:setup | `Error` | 0.00 | test_volumes.py
   ContextSuite context=TestVolumes>:setup | `Error` | 0.00 | test_volumes.py
   ContextSuite context=TestVPCRedundancy>:setup | `Error` | 0.00 | test_vpc_redundant.py
   ContextSuite context=TestVPCNics>:setup | `Error` | 0.00 | test_vpc_router_nics.py
   ContextSuite context=TestRVPCSite2SiteVpn>:setup | `Error` | 0.00 | test_vpc_vpn.py
   ContextSuite context=TestVPCSite2SiteVPNMultipleOptions>:setup | `Error` | 0.00 | test_vpc_vpn.py
   ContextSuite context=TestVpcRemoteAccessVpn>:setup | `Error` | 0.00 | test_vpc_vpn.py
   ContextSuite context=TestVpcSite2SiteVpn>:setup | `Error` | 0.00 | test_vpc_vpn.py
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] andrijapanicsb commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

andrijapanicsb commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-602768635
 
 
   Not good (still testing a few more scenarios and will update this comment once done):
   
   ## XENSERVER (7.1):
   
   ### snapshot.backup.to.secondary = TRUE (default)
   MANUAL SNAPS: LGTM
   - deletion works fine - removed from both primary/secondary store, DB records in snapshost_store_reg marked as DESTROYED, all good.
   SCHEDULED SNAPS: MAYBE OK - someone should advise
   - for disk snaps we are making snap chains (parent_snapshot_id pointing to the ID of the previous scheduled snapshot) and while DB says that older snaps are deleted, it's not really true on storage side, but as those are snap diff snaps (chain)  - it makes sense that it doesn't delete the older/parent snap while there are child/linked snaps), but an older snap can be selectively deleted manually - which is a bit strange, as it would break the snap chain)
   
   ### snapshot.backup.to.secondary = FALSE
   MANUAL SNAPS: LGTM
   - deletion works fine - removed from both primary store, DB records in snapshost_store_ref marked as DESTROYED, all good.
   SCHEDULED SNAPS:
   - TO BE TESTED - will edit here once it completes+
   
   
   
   
   ## KVM (CENT7.1)
   
   ### snapshot.backup.to.secondary TRUE (default)
   MANUAL SNAPS: LGTM
   - deletion works fine - removed from both primary/secondary store, DB records in snapshost_store_reg marked as DESTROYED, all good.
   - I also see we are NOT keeping the snap on the QCOW2 file (good, as we made a copy to sec storage) - and even though the "PRIMARY" record (store_role) is still created and kept as "Ready" in snapshost_store_ref - it does get properly marked as DESTROYED when snap is deleted with no exception
   SCHEDULED SNAPS:
   - works fine, created and deleted as expected in both DB and on the primary/secondary stor.
   
   ### snapshot.backup.to.secondary = FALSE
   MANUAL SNAPS: BROKEN
   - can't even create manual KVM volume snap
   SCHEDULED SNAPS: NA
   - not tested as manual ones fail
   
   
   ### VMware not tested, 
   but will re-test that it's NOT affected once we have everything else fixed.
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] DaanHoogland opened a new pull request #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

DaanHoogland opened a new pull request #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969
 
 
   ## Description
   <!--- Describe your changes in detail -->
   copy of PR #3649 on joint account for easier cooperation.
   
   Tested both cases:
   - Snapshot backed up on Secondary (stored on primary and secondary) 
   - Snapshot stored only on primary storage.
   
   This fix works for KVM  and XenServer.
   
   **Note**: I changed the name of the strategy from _XenserverSnapshotStrategy_ to _DefaultSnapshotStrategy_ due to the fact that the "Xenserver" strategy handles also Ceph, KVM, and do return `StrategyPriority.DEFAULT` in any case except _REVERT_.
   
   ```
       @Override
       public StrategyPriority canHandle(Snapshot snapshot, SnapshotOperation op) {
           if (SnapshotOperation.REVERT.equals(op)) {
               long volumeId = snapshot.getVolumeId();
               VolumeVO volumeVO = volumeDao.findById(volumeId);
   
               if (volumeVO != null && ImageFormat.QCOW2.equals(volumeVO.getFormat())) {
                   return StrategyPriority.DEFAULT;
               }
               return StrategyPriority.CANT_HANDLE;
           }
           return StrategyPriority.DEFAULT;
       }
   ```
   
   Fixes: #3646
   
   ## Types of changes
   <!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
   - [ ] Breaking change (fix or feature that would cause existing functionality to change)
   - [ ] New feature (non-breaking change which adds functionality)
   - [x] Bug fix (non-breaking change which fixes an issue)
   - [ ] Enhancement (improves an existing feature and functionality)
   - [ ] Cleanup (Code refactoring and cleanup, that may add test cases)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] Slair1 commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

Slair1 commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-613032091
 
 
   > > So it does @Slair1 ...
   > 
   > @andrijapanicsb could you double-check the code? When i manually check 4.13, i don't see the lines added in the screenshot below from #3194 .
   > ![image](https://user-images.githubusercontent.com/17735593/78931434-90f0a200-7a6b-11ea-829c-59a93703a5f9.png)
   
   
   @andrijapanicsb just wanted to ping you on this again thanks!

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] andrijapanicsb commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

andrijapanicsb commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-613612963
 
 
   Clear, I did a bit of reading myself. Say, what about corruption when creating snapshots? - in recent qemu releases (2.10) qemu-img actually opens qcow2 with read-write access instead of read-only previously (don't ask me why, documented behavior). This would probably mean that even backing up the snap to Secondary Storage (with qemu 2.10) will not work (requires a use of --force-share flag, which we don't have implemented in ACS) unless the VM is suspended - did you have a change to test qemu 2.10 (that is separate Centos SIG repo), etc? What about corruption while creating snapshots with stock qemu version 1.53?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] DaanHoogland removed a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

DaanHoogland removed a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-600174457
 
 
   @blueorangutan package

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-609722360
 
 
   @DaanHoogland a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] DaanHoogland removed a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

DaanHoogland removed a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-609963113
 
 
   @blueorangutan package

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] andrijapanicsb edited a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

andrijapanicsb edited a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-607319803
 
 
   This failure is due to last commit:
   
   when NOT backing up the snap to secondary - the snap fails to be deleted....damn it 😢 
   
   2020-04-01 15:29:22,117 DEBUG [c.c.a.ApiServlet] (qtp504527234-17:ctx-3353a255 ctx-cc5b2b2c) (logid:0f9171aa) ===END===  10.1.0.1 -- GET  command=deleteSnapshot&id=1e63d646-eadd-4a1d-b8b8-ddec90fc9e0e&response=json&_=1585754941759
   2020-04-01 15:29:22,133 DEBUG [o.a.c.s.s.DefaultSnapshotStrategy] (API-Job-Executor-3:ctx-c4bb5e71 job-284 ctx-75e649e7) (logid:37b859e0) Can't find snapshot
   [snapshot id: 19] on backup storage
   2020-04-01 15:29:22,141 DEBUG [o.a.c.s.s.SnapshotObject] (API-Job-Executor-3:ctx-c4bb5e71 job-284 ctx-75e649e7) (logid:37b859e0) Failed to update state:java.lang.NullPointerException
   2020-04-01 15:29:22,142 DEBUG [c.c.s.s.SnapshotManagerImpl] (API-Job-Executor-3:ctx-c4bb5e71 job-284 ctx-75e649e7) (logid:37b859e0) Failed to delete snapshot: 19:java.lang.NullPointerException
   2020-04-01 15:29:22,147 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-3:ctx-c4bb5e71 job-284) (logid:37b859e0) Unexpected exception while executing org.apache.cloudstack.api.command.user.snapshot.DeleteSnapshotCmd
   com.cloud.utils.exception.CloudRuntimeException: Failed to delete snapshot:java.lang.NullPointerException
           at com.cloud.storage.snapshot.SnapshotManagerImpl.deleteSnapshot(SnapshotManagerImpl.java:611)
           at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
           at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
           at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
           at java.lang.reflect.Method.invoke(Method.java:498)
           at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
           at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
           at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
           at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
           at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
           at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
           at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
           at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
           at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
           at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
           at com.sun.proxy.$Proxy199.deleteSnapshot(Unknown Source)
           at org.apache.cloudstack.api.command.user.snapshot.DeleteSnapshotCmd.execute(DeleteSnapshotCmd.java:103)
           at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:156)
           at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
           at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:586)
           at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
           at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
           at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
           at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
           at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
           at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:534)
           at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
           at java.util.concurrent.FutureTask.run(FutureTask.java:266)
           at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
           at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
           at java.lang.Thread.run(Thread.java:748)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan removed a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan removed a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-610078296
 
 
   <b>Trillian test result (tid-1366)</b>
   Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server 7
   Total time taken: 37855 seconds
   Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3969-t1366-vmware-67u3.zip
   Intermittent failure detected: /marvin/tests/smoke/test_deploy_vm_root_resize.py
   Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
   Intermittent failure detected: /marvin/tests/smoke/test_ssvm.py
   Intermittent failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
   Smoke tests completed. 75 look OK, 2 have error(s)
   Only failed tests results shown below:
   
   
   Test | Result | Time (s) | Test File
   --- | --- | --- | ---
   test_02_vpc_privategw_static_routes | `Failure` | 317.01 | test_privategw_acl.py
   test_03_vpc_privategw_restart_vpc_cleanup | `Failure` | 299.80 | test_privategw_acl.py
   test_04_rvpc_privategw_static_routes | `Failure` | 455.71 | test_privategw_acl.py
   test_11_migrate_vm | `Error` | 50.84 | test_vm_life_cycle.py
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-599630848
 
 
   @DaanHoogland a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] harikrishna-patnala commented on a change in pull request #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

harikrishna-patnala commented on a change in pull request #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#discussion_r400886597
 
 

 ##########
 File path: engine/storage/snapshot/src/main/java/org/apache/cloudstack/storage/snapshot/DefaultSnapshotStrategy.java
 ##########
 @@ -269,63 +262,87 @@ public boolean deleteSnapshot(Long snapshotId) {
             throw new InvalidParameterValueException("Can't delete snapshotshot " + snapshotId + " due to it is in " + snapshotVO.getState() + " Status");
         }
 
-        // first mark the snapshot as destroyed, so that ui can't see it, but we
-        // may not destroy the snapshot on the storage, as other snapshots may
-        // depend on it.
         SnapshotInfo snapshotOnImage = snapshotDataFactory.getSnapshot(snapshotId, DataStoreRole.Image);
-        if (snapshotOnImage == null) {
-            s_logger.debug("Can't find snapshot on backup storage, delete it in db");
-            snapshotDao.remove(snapshotId);
-            return true;
-        }
-
-        SnapshotObject obj = (SnapshotObject)snapshotOnImage;
-        try {
-            obj.processEvent(Snapshot.Event.DestroyRequested);
-            List<VolumeDetailVO> volumesFromSnapshot;
-            volumesFromSnapshot = _volumeDetailsDaoImpl.findDetails("SNAPSHOT_ID", String.valueOf(snapshotId), null);
 
-            if (volumesFromSnapshot.size() > 0) {
-                try {
-                    obj.processEvent(Snapshot.Event.OperationFailed);
-                } catch (NoTransitionException e1) {
-                    s_logger.debug("Failed to change snapshot state: " + e1.toString());
+        boolean deletedOnSecondary = false;
+        if (snapshotOnImage == null) {
+            s_logger.debug(String.format("Can't find snapshot [snapshot id: %d] on backup storage", snapshotId));
 
 Review comment:
   Yes David, snapshotDao.remove(snapshotId) needs to be moved here. 
   cc @GabrielBrascher 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-599590064
 
 
   Packaging result: ✖centos6 ✔centos7 ✔debian. JID-1057

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan removed a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan removed a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-610262736
 
 
   @DaanHoogland a Trillian-Jenkins matrix job (centos7 mgmt + xs71, centos7 mgmt + vmware67, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] andrijapanicsb commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

andrijapanicsb commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-613090734
 
 
   @Slair1 - the bug you shared (RHEL bug), was marked as WON'T FIX - and was raised against qemu-kvm-0.12.1.2-2.355.el6_4.1.x86_64
   
   Can you please describe in which env (versions of qemu) you are seeing the corruption (as master/4.14 does NOT even support centos6)
   The issue happened during emerging 4.11 into 4.12 branch - we identified WHEN it was messed up, but not HOW (which is now irrelevant)
   
   I would like to first REPRODUCE this issue on centos7+ instead of just suspending the VM (now that we think to AGAIN introduce the suspend thing)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] andrijapanicsb edited a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

andrijapanicsb edited a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-602768635
 
 
   Not good (still testing a few more scenarios and will update this comment once done):
   
   ## XENSERVER (7.1):
   
   ### snapshot.backup.to.secondary = TRUE (default)
   MANUAL SNAPS: LGTM
   - deletion works fine - removed from both primary/secondary store, DB records in snapshost_store_reg marked as DESTROYED, all good.
   SCHEDULED SNAPS: MAYBE OK - someone should advise
   - for disk snaps we are making snap chains (parent_snapshot_id pointing to the ID of the previous scheduled snapshot) and while DB says that older snaps are deleted, it's not really true on storage side, but as those are snap diff snaps (chain)  - it makes sense that it doesn't delete the older/parent snap while there are child/linked snaps), but an older snap can be selectively deleted manually - which is a bit strange, as it would break the snap chain)
   
   ### snapshot.backup.to.secondary = FALSE
   
   MANUAL SNAPS: LGTM
   
   deletion works fine - removed from both primary store, DB records in snapshost_store_ref marked as DESTROYED, all good.
   
   SCHEDULED SNAPS:
   
   TO BE TESTED - will edit here once it completes+
   
   
   
   
   ## KVM (CENT7.1)
   
   ### snapshot.backup.to.secondary TRUE (default)
   MANUAL SNAPS: LGTM
   - deletion works fine - removed from both primary/secondary store, DB records in snapshost_store_reg marked as DESTROYED, all good.
   - I also see we are NOT keeping the snap on the QCOW2 file (good, as we made a copy to sec storage) - and even though the "PRIMARY" record (store_role) is still created and kept as "Ready" in snapshost_store_ref - it does get properly marked as DESTROYED when snap is deleted with no exception
   SCHEDULED SNAPS:
   - works fine, created and deleted as expected in both DB and on the primary/secondary stor.
   
   ### snapshot.backup.to.secondary = FALSE
   MANUAL SNAPS: BROKEN
   - can't even create manual KVM volume snap
   SCHEDULED SNAPS: NA
   - not tested as manual ones fail
   
   
   ### VMware not tested, 
   but will re-test that it's NOT affected once we have everything else fixed.
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] DaanHoogland commented on a change in pull request #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

DaanHoogland commented on a change in pull request #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#discussion_r403915764
 
 

 ##########
 File path: engine/storage/snapshot/src/main/java/org/apache/cloudstack/storage/snapshot/DefaultSnapshotStrategy.java
 ##########
 @@ -285,7 +284,27 @@ public boolean deleteSnapshot(Long snapshotId) {
             }
         }
 
-        boolean deletedOnPrimary = deleteSnapshotOnPrimary(snapshotId);
+        boolean deletedOnPrimary = false;
+        snapshotVO = snapshotDao.findById(snapshotId);
+        SnapshotInfo snapshotOnPrimaryInfo = snapshotDataFactory.getSnapshot(snapshotId, DataStoreRole.Primary);
+        if (snapshotVO != null && snapshotVO.getState() == Snapshot.State.Destroyed) {
+            deletedOnPrimary = deleteSnapshotOnPrimary(snapshotId, snapshotOnPrimaryInfo);
+        } else {
+            // This is to handle snapshots which are created only on primary when snapshot.backup.to.secondary is set to false. 
+            SnapshotObject obj = (SnapshotObject) snapshotOnPrimaryInfo;
+            try {
+                obj.processEvent(Snapshot.Event.DestroyRequested);
+                deletedOnPrimary = deleteSnapshotOnPrimary(snapshotId, snapshotOnPrimaryInfo);
+                if (!deletedOnPrimary) {
+                    obj.processEvent(Snapshot.Event.OperationFailed);
+                } else {
+                    obj.processEvent(Snapshot.Event.OperationSucceeded);
+                }
+            } catch (NoTransitionException e) {
+            s_logger.debug("Failed to set the state to destroying: ", e);
+            return false;
+            }
 
 Review comment:
   and than factor this bit out again?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] DaanHoogland removed a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

DaanHoogland removed a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-599542351
 
 
   @blueorangutan package

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] DaanHoogland removed a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

DaanHoogland removed a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-609721773
 
 
   @blueorangutan package

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-610701895
 
 
   <b>Trillian test result (tid-1386)</b>
   Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7
   Total time taken: 40680 seconds
   Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3969-t1386-xenserver-71.zip
   Intermittent failure detected: /marvin/tests/smoke/test_iso.py
   Intermittent failure detected: /marvin/tests/smoke/test_list_ids_parameter.py
   Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
   Intermittent failure detected: /marvin/tests/smoke/test_scale_vm.py
   Intermittent failure detected: /marvin/tests/smoke/test_vpc_vpn.py
   Smoke tests completed. 74 look OK, 3 have error(s)
   Only failed tests results shown below:
   
   
   Test | Result | Time (s) | Test File
   --- | --- | --- | ---
   ContextSuite context=TestListIdsParams>:teardown | `Error` | 1.09 | test_list_ids_parameter.py
   test_02_vpc_privategw_static_routes | `Failure` | 361.62 | test_privategw_acl.py
   test_03_vpc_privategw_restart_vpc_cleanup | `Failure` | 293.85 | test_privategw_acl.py
   test_04_rvpc_privategw_static_routes | `Failure` | 469.87 | test_privategw_acl.py
   test_01_scale_vm | `Failure` | 33.81 | test_scale_vm.py
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-606348503
 
 
   <b>Trillian test result (tid-1336)</b>
   Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7
   Total time taken: 29085 seconds
   Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3969-t1336-xenserver-71.zip
   Intermittent failure detected: /marvin/tests/smoke/test_list_ids_parameter.py
   Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
   Intermittent failure detected: /marvin/tests/smoke/test_scale_vm.py
   Smoke tests completed. 74 look OK, 3 have error(s)
   Only failed tests results shown below:
   
   
   Test | Result | Time (s) | Test File
   --- | --- | --- | ---
   ContextSuite context=TestListIdsParams>:teardown | `Error` | 1.08 | test_list_ids_parameter.py
   test_02_vpc_privategw_static_routes | `Failure` | 227.08 | test_privategw_acl.py
   test_03_vpc_privategw_restart_vpc_cleanup | `Failure` | 214.64 | test_privategw_acl.py
   test_04_rvpc_privategw_static_routes | `Failure` | 326.09 | test_privategw_acl.py
   test_01_scale_vm | `Failure` | 20.60 | test_scale_vm.py
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] Slair1 edited a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

Slair1 edited a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-611706617
 
 
   > Is this for volume or VM snapshots @Slair1 ? - as volume snapshots are NOT possible for KVM while the VM is running (someone thought that's interesting to do that, due to qcow2 safety...so now it's useless for KVM...)
   
   It is for volume snapshots, but you're correct the global setting kvm.snapshot.enabled needs to be enabled.  But this brings up another point, we are wondering if we need to do a suspend before deleting VM Snapshots also.  From our examination of the code, it looks like both VM Snapshots and Volume Snapshots use the save libvirt libraries to take and delete the snapshot... so I would think they could both have the same corruption issue if the snapshot is deleted while the VM is powered-on and not suspended during the snapshot delete...

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-612000101
 
 
   <b>Trillian test result (tid-1400)</b>
   Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
   Total time taken: 60513 seconds
   Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3969-t1400-kvm-centos7.zip
   Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
   Intermittent failure detected: /marvin/tests/smoke/test_public_ip_range.py
   Intermittent failure detected: /marvin/tests/smoke/test_reset_vm_on_reboot.py
   Intermittent failure detected: /marvin/tests/smoke/test_resource_accounting.py
   Intermittent failure detected: /marvin/tests/smoke/test_router_dhcphosts.py
   Intermittent failure detected: /marvin/tests/smoke/test_router_dns.py
   Intermittent failure detected: /marvin/tests/smoke/test_router_dnsservice.py
   Intermittent failure detected: /marvin/tests/smoke/test_routers_iptables_default_policy.py
   Intermittent failure detected: /marvin/tests/smoke/test_routers_network_ops.py
   Intermittent failure detected: /marvin/tests/smoke/test_routers.py
   Intermittent failure detected: /marvin/tests/smoke/test_secondary_storage.py
   Intermittent failure detected: /marvin/tests/smoke/test_service_offerings.py
   Intermittent failure detected: /marvin/tests/smoke/test_snapshots.py
   Intermittent failure detected: /marvin/tests/smoke/test_ssvm.py
   Intermittent failure detected: /marvin/tests/smoke/test_templates.py
   Intermittent failure detected: /marvin/tests/smoke/test_usage.py
   Intermittent failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
   Intermittent failure detected: /marvin/tests/smoke/test_vm_snapshots.py
   Intermittent failure detected: /marvin/tests/smoke/test_volumes.py
   Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
   Intermittent failure detected: /marvin/tests/smoke/test_vpc_router_nics.py
   Intermittent failure detected: /marvin/tests/smoke/test_vpc_vpn.py
   Intermittent failure detected: /marvin/tests/smoke/test_hostha_kvm.py
   Smoke tests completed. 55 look OK, 22 have error(s)
   Only failed tests results shown below:
   
   
   Test | Result | Time (s) | Test File
   --- | --- | --- | ---
   test_02_vpc_privategw_static_routes | `Failure` | 182.50 | test_privategw_acl.py
   test_03_vpc_privategw_restart_vpc_cleanup | `Failure` | 181.23 | test_privategw_acl.py
   test_04_rvpc_privategw_static_routes | `Failure` | 259.01 | test_privategw_acl.py
   ContextSuite context=TestResetVmOnReboot>:setup | `Error` | 0.00 | test_reset_vm_on_reboot.py
   ContextSuite context=TestRAMCPUResourceAccounting>:setup | `Error` | 0.00 | test_resource_accounting.py
   ContextSuite context=TestRouterDHCPHosts>:setup | `Error` | 0.00 | test_router_dhcphosts.py
   ContextSuite context=TestRouterDHCPOpts>:setup | `Error` | 0.00 | test_router_dhcphosts.py
   ContextSuite context=TestRouterDns>:setup | `Error` | 0.00 | test_router_dns.py
   test_01_sys_vm_start | `Failure` | 0.07 | test_secondary_storage.py
   ContextSuite context=TestRouterDnsService>:setup | `Error` | 0.00 | test_router_dnsservice.py
   ContextSuite context=TestRouterIpTablesPolicies>:setup | `Error` | 0.00 | test_routers_iptables_default_policy.py
   ContextSuite context=TestVPCIpTablesPolicies>:setup | `Error` | 0.00 | test_routers_iptables_default_policy.py
   ContextSuite context=TestIsolatedNetworks>:setup | `Error` | 0.00 | test_routers_network_ops.py
   ContextSuite context=TestRedundantIsolateNetworks>:setup | `Error` | 0.00 | test_routers_network_ops.py
   ContextSuite context=TestRouterServices>:setup | `Error` | 0.00 | test_routers.py
   ContextSuite context=TestCpuCapServiceOfferings>:setup | `Error` | 0.00 | test_service_offerings.py
   ContextSuite context=TestServiceOfferings>:setup | `Error` | 0.17 | test_service_offerings.py
   ContextSuite context=TestSnapshotRootDisk>:setup | `Error` | 0.00 | test_snapshots.py
   test_01_list_sec_storage_vm | `Failure` | 0.03 | test_ssvm.py
   test_02_list_cpvm_vm | `Failure` | 0.03 | test_ssvm.py
   test_03_ssvm_internals | `Failure` | 0.03 | test_ssvm.py
   test_04_cpvm_internals | `Failure` | 0.03 | test_ssvm.py
   test_05_stop_ssvm | `Failure` | 0.03 | test_ssvm.py
   test_06_stop_cpvm | `Failure` | 0.02 | test_ssvm.py
   test_07_reboot_ssvm | `Failure` | 0.03 | test_ssvm.py
   test_08_reboot_cpvm | `Failure` | 0.03 | test_ssvm.py
   test_09_destroy_ssvm | `Failure` | 0.02 | test_ssvm.py
   test_10_destroy_cpvm | `Failure` | 0.03 | test_ssvm.py
   test_02_create_template_with_checksum_sha1 | `Error` | 65.37 | test_templates.py
   test_03_create_template_with_checksum_sha256 | `Error` | 65.34 | test_templates.py
   test_04_create_template_with_checksum_md5 | `Error` | 65.36 | test_templates.py
   test_05_create_template_with_no_checksum | `Error` | 65.36 | test_templates.py
   test_02_deploy_vm_from_direct_download_template | `Error` | 1.19 | test_templates.py
   test_03_deploy_vm_wrong_checksum | `Error` | 1.24 | test_templates.py
   ContextSuite context=TestTemplates>:setup | `Error` | 16.56 | test_templates.py
   ContextSuite context=TestISOUsage>:setup | `Error` | 0.00 | test_usage.py
   ContextSuite context=TestLBRuleUsage>:setup | `Error` | 0.00 | test_usage.py
   ContextSuite context=TestNatRuleUsage>:setup | `Error` | 0.00 | test_usage.py
   ContextSuite context=TestPublicIPUsage>:setup | `Error` | 0.00 | test_usage.py
   ContextSuite context=TestSnapshotUsage>:setup | `Error` | 0.00 | test_usage.py
   ContextSuite context=TestVmUsage>:setup | `Error` | 0.00 | test_usage.py
   ContextSuite context=TestVolumeUsage>:setup | `Error` | 0.00 | test_usage.py
   ContextSuite context=TestVpnUsage>:setup | `Error` | 0.00 | test_usage.py
   ContextSuite context=Test01DeployVM>:setup | `Error` | 0.00 | test_vm_life_cycle.py
   ContextSuite context=Test02VMLifeCycle>:setup | `Error` | 0.00 | test_vm_life_cycle.py
   test_14_secure_to_secure_vm_migration | `Error` | 11.28 | test_vm_life_cycle.py
   test_15_secured_to_nonsecured_vm_migration | `Error` | 73.93 | test_vm_life_cycle.py
   test_16_nonsecured_to_secured_vm_migration | `Error` | 1.20 | test_vm_life_cycle.py
   ContextSuite context=TestVmSnapshot>:setup | `Error` | 1.57 | test_vm_snapshots.py
   ContextSuite context=TestCreateVolume>:setup | `Error` | 0.00 | test_volumes.py
   ContextSuite context=TestVolumes>:setup | `Error` | 0.00 | test_volumes.py
   ContextSuite context=TestVPCRedundancy>:setup | `Error` | 0.00 | test_vpc_redundant.py
   ContextSuite context=TestVPCNics>:setup | `Error` | 0.00 | test_vpc_router_nics.py
   ContextSuite context=TestRVPCSite2SiteVpn>:setup | `Error` | 0.00 | test_vpc_vpn.py
   ContextSuite context=TestVPCSite2SiteVPNMultipleOptions>:setup | `Error` | 0.00 | test_vpc_vpn.py
   ContextSuite context=TestVpcRemoteAccessVpn>:setup | `Error` | 0.00 | test_vpc_vpn.py
   ContextSuite context=TestVpcSite2SiteVpn>:setup | `Error` | 0.00 | test_vpc_vpn.py
   test_disable_oobm_ha_state_ineligible | `Error` | 1512.14 | test_hostha_kvm.py
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-611740124
 
 
   <b>Trillian test result (tid-1397)</b>
   Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server 7
   Total time taken: 36094 seconds
   Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3969-t1397-vmware-67u3.zip
   Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
   Smoke tests completed. 76 look OK, 1 have error(s)
   Only failed tests results shown below:
   
   
   Test | Result | Time (s) | Test File
   --- | --- | --- | ---
   test_02_vpc_privategw_static_routes | `Failure` | 305.78 | test_privategw_acl.py
   test_03_vpc_privategw_restart_vpc_cleanup | `Failure` | 310.61 | test_privategw_acl.py
   test_04_rvpc_privategw_static_routes | `Failure` | 480.69 | test_privategw_acl.py
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] DaanHoogland closed pull request #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

DaanHoogland closed pull request #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] andrijapanicsb commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

andrijapanicsb commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-606101797
 
 
   I proposed that we continue/merge this PR as it is (I want to confirm a few more things with testing, specifically that VMware is still not affected) and close the "garbage" problem as it seems solved.
   
   For the KVM+NFS/local very bad behaviour when snapshot.backup.to.secondary=false - that is traced back to the original implementation, which seems only done for KVM + CEPH, and so I propose that we simply change the description of this global settinh to ... Applicable only for KVM = Ceph"
   
   @GabrielBrascher sounds ok to you?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] andrijapanicsb edited a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

andrijapanicsb edited a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-602768635
 
 
   Not good (still testing a few more scenarios and will update this comment once done):
   
   ## XENSERVER (7.1) + NFS
   
   ### snapshot.backup.to.secondary = TRUE (default)
   MANUAL SNAPS: LGTM
   - deletion works fine - removed from both primary/secondary store, DB records in snapshost_store_reg marked as DESTROYED, all good.
   
   SCHEDULED SNAPS: MAYBE OK - someone should advise
   - for disk snaps we are making snap chains (parent_snapshot_id pointing to the ID of the previous scheduled snapshot) and while DB says that older snaps are deleted, it's not really true on storage side, but as those are snap diff snaps (chain)  - it makes sense that it doesn't delete the older/parent snap while there are child/linked snaps), but an older snap can be selectively deleted manually - which is a bit strange, as it would break the snap chain)
   
   ### snapshot.backup.to.secondary = FALSE
   
   MANUAL SNAPS: LGTM
   deletion works fine - removed from both primary store, DB records in snapshost_store_ref marked as DESTROYED, all good.
   
   SCHEDULED SNAPS:
   TO BE TESTED - will edit here once it completes+
   
   
   
   
   ## KVM (CENT7.1) + NFS
   
   ### snapshot.backup.to.secondary TRUE (default)
   MANUAL SNAPS: LGTM
   - deletion works fine - removed from both primary/secondary store, DB records in snapshost_store_reg marked as DESTROYED, all good.
   - I also see we are NOT keeping the snap on the QCOW2 file (good, as we made a copy to sec storage) - and even though the "PRIMARY" record (store_role) is still created and kept as "Ready" in snapshost_store_ref - it does get properly marked as DESTROYED when snap is deleted with no exception
   
   SCHEDULED SNAPS:
   - works fine, created and deleted as expected in both DB and on the primary/secondary stor.
   
   ### snapshot.backup.to.secondary = FALSE
   MANUAL SNAPS: BROKEN
   - can't even create manual KVM volume snap
   
   SCHEDULED SNAPS: NA
   - not tested as manual ones fail
   
   
   ### VMware not tested, 
   but will re-test that it's NOT affected once we have everything else fixed.
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] DaanHoogland opened a new pull request #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

DaanHoogland opened a new pull request #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969
 
 
   ## Description
   <!--- Describe your changes in detail -->
   copy of PR #3649 on joint account for easier cooperation.
   
   Tested both cases:
   - Snapshot backed up on Secondary (stored on primary and secondary) 
   - Snapshot stored only on primary storage.
   
   This fix works for KVM  and XenServer.
   
   **Note**: I changed the name of the strategy from _XenserverSnapshotStrategy_ to _DefaultSnapshotStrategy_ due to the fact that the "Xenserver" strategy handles also Ceph, KVM, and do return `StrategyPriority.DEFAULT` in any case except _REVERT_.
   
   ```
       @Override
       public StrategyPriority canHandle(Snapshot snapshot, SnapshotOperation op) {
           if (SnapshotOperation.REVERT.equals(op)) {
               long volumeId = snapshot.getVolumeId();
               VolumeVO volumeVO = volumeDao.findById(volumeId);
   
               if (volumeVO != null && ImageFormat.QCOW2.equals(volumeVO.getFormat())) {
                   return StrategyPriority.DEFAULT;
               }
               return StrategyPriority.CANT_HANDLE;
           }
           return StrategyPriority.DEFAULT;
       }
   ```
   
   Fixes: #3646
   
   ## Types of changes
   <!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
   - [ ] Breaking change (fix or feature that would cause existing functionality to change)
   - [ ] New feature (non-breaking change which adds functionality)
   - [x] Bug fix (non-breaking change which fixes an issue)
   - [ ] Enhancement (improves an existing feature and functionality)
   - [ ] Cleanup (Code refactoring and cleanup, that may add test cases)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] andrijapanicsb commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

andrijapanicsb commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-612005865
 
 
   @blueorangutan test centos7 xenserver-71

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan removed a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan removed a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-611990883
 
 
   <b>Trillian test result (tid-1399)</b>
   Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7
   Total time taken: 58379 seconds
   Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3969-t1399-xenserver-71.zip
   Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
   Intermittent failure detected: /marvin/tests/smoke/test_public_ip_range.py
   Intermittent failure detected: /marvin/tests/smoke/test_reset_vm_on_reboot.py
   Intermittent failure detected: /marvin/tests/smoke/test_resource_accounting.py
   Intermittent failure detected: /marvin/tests/smoke/test_router_dhcphosts.py
   Intermittent failure detected: /marvin/tests/smoke/test_router_dns.py
   Intermittent failure detected: /marvin/tests/smoke/test_router_dnsservice.py
   Intermittent failure detected: /marvin/tests/smoke/test_routers_iptables_default_policy.py
   Intermittent failure detected: /marvin/tests/smoke/test_routers_network_ops.py
   Intermittent failure detected: /marvin/tests/smoke/test_routers.py
   Intermittent failure detected: /marvin/tests/smoke/test_scale_vm.py
   Intermittent failure detected: /marvin/tests/smoke/test_secondary_storage.py
   Intermittent failure detected: /marvin/tests/smoke/test_service_offerings.py
   Intermittent failure detected: /marvin/tests/smoke/test_snapshots.py
   Intermittent failure detected: /marvin/tests/smoke/test_ssvm.py
   Intermittent failure detected: /marvin/tests/smoke/test_templates.py
   Intermittent failure detected: /marvin/tests/smoke/test_usage.py
   Intermittent failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
   Intermittent failure detected: /marvin/tests/smoke/test_vm_snapshots.py
   Intermittent failure detected: /marvin/tests/smoke/test_volumes.py
   Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
   Intermittent failure detected: /marvin/tests/smoke/test_vpc_router_nics.py
   Intermittent failure detected: /marvin/tests/smoke/test_vpc_vpn.py
   Smoke tests completed. 55 look OK, 22 have error(s)
   Only failed tests results shown below:
   
   
   Test | Result | Time (s) | Test File
   --- | --- | --- | ---
   test_02_vpc_privategw_static_routes | `Failure` | 251.57 | test_privategw_acl.py
   test_03_vpc_privategw_restart_vpc_cleanup | `Failure` | 217.06 | test_privategw_acl.py
   test_04_rvpc_privategw_static_routes | `Failure` | 326.43 | test_privategw_acl.py
   ContextSuite context=TestResetVmOnReboot>:setup | `Error` | 0.00 | test_reset_vm_on_reboot.py
   ContextSuite context=TestRAMCPUResourceAccounting>:setup | `Error` | 0.00 | test_resource_accounting.py
   ContextSuite context=TestScaleVm>:setup | `Error` | 0.00 | test_scale_vm.py
   ContextSuite context=TestRouterDHCPHosts>:setup | `Error` | 0.00 | test_router_dhcphosts.py
   ContextSuite context=TestRouterDHCPOpts>:setup | `Error` | 0.00 | test_router_dhcphosts.py
   ContextSuite context=TestRouterDns>:setup | `Error` | 0.00 | test_router_dns.py
   ContextSuite context=TestRouterDnsService>:setup | `Error` | 0.00 | test_router_dnsservice.py
   ContextSuite context=TestRouterIpTablesPolicies>:setup | `Error` | 0.00 | test_routers_iptables_default_policy.py
   ContextSuite context=TestVPCIpTablesPolicies>:setup | `Error` | 0.00 | test_routers_iptables_default_policy.py
   ContextSuite context=TestIsolatedNetworks>:setup | `Error` | 0.00 | test_routers_network_ops.py
   ContextSuite context=TestRedundantIsolateNetworks>:setup | `Error` | 0.00 | test_routers_network_ops.py
   ContextSuite context=TestRouterServices>:setup | `Error` | 0.00 | test_routers.py
   test_01_sys_vm_start | `Failure` | 0.07 | test_secondary_storage.py
   ContextSuite context=TestServiceOfferings>:setup | `Error` | 1511.89 | test_service_offerings.py
   ContextSuite context=TestSnapshotRootDisk>:setup | `Error` | 0.00 | test_snapshots.py
   test_01_list_sec_storage_vm | `Failure` | 0.03 | test_ssvm.py
   test_02_list_cpvm_vm | `Failure` | 0.02 | test_ssvm.py
   test_03_ssvm_internals | `Failure` | 0.02 | test_ssvm.py
   test_04_cpvm_internals | `Failure` | 0.02 | test_ssvm.py
   test_05_stop_ssvm | `Failure` | 0.02 | test_ssvm.py
   test_06_stop_cpvm | `Failure` | 0.02 | test_ssvm.py
   test_07_reboot_ssvm | `Failure` | 0.03 | test_ssvm.py
   test_08_reboot_cpvm | `Failure` | 0.02 | test_ssvm.py
   test_09_destroy_ssvm | `Failure` | 0.02 | test_ssvm.py
   test_10_destroy_cpvm | `Failure` | 0.02 | test_ssvm.py
   test_02_create_template_with_checksum_sha1 | `Error` | 65.38 | test_templates.py
   test_03_create_template_with_checksum_sha256 | `Error` | 65.38 | test_templates.py
   test_04_create_template_with_checksum_md5 | `Error` | 65.40 | test_templates.py
   test_05_create_template_with_no_checksum | `Error` | 65.43 | test_templates.py
   ContextSuite context=TestTemplates>:setup | `Error` | 1.96 | test_templates.py
   ContextSuite context=TestISOUsage>:setup | `Error` | 0.00 | test_usage.py
   ContextSuite context=TestLBRuleUsage>:setup | `Error` | 0.00 | test_usage.py
   ContextSuite context=TestNatRuleUsage>:setup | `Error` | 0.00 | test_usage.py
   ContextSuite context=TestPublicIPUsage>:setup | `Error` | 0.00 | test_usage.py
   ContextSuite context=TestSnapshotUsage>:setup | `Error` | 0.00 | test_usage.py
   ContextSuite context=TestVmUsage>:setup | `Error` | 0.00 | test_usage.py
   ContextSuite context=TestVolumeUsage>:setup | `Error` | 0.00 | test_usage.py
   ContextSuite context=TestVpnUsage>:setup | `Error` | 0.00 | test_usage.py
   ContextSuite context=Test01DeployVM>:setup | `Error` | 0.00 | test_vm_life_cycle.py
   ContextSuite context=Test02VMLifeCycle>:setup | `Error` | 0.00 | test_vm_life_cycle.py
   test_change_service_offering_for_vm_with_snapshots | `Error` | 1.19 | test_vm_snapshots.py
   ContextSuite context=TestVmSnapshot>:setup | `Error` | 8.99 | test_vm_snapshots.py
   ContextSuite context=TestCreateVolume>:setup | `Error` | 0.00 | test_volumes.py
   ContextSuite context=TestVolumes>:setup | `Error` | 0.00 | test_volumes.py
   ContextSuite context=TestVPCRedundancy>:setup | `Error` | 0.00 | test_vpc_redundant.py
   ContextSuite context=TestVPCNics>:setup | `Error` | 0.00 | test_vpc_router_nics.py
   ContextSuite context=TestRVPCSite2SiteVpn>:setup | `Error` | 0.00 | test_vpc_vpn.py
   ContextSuite context=TestVPCSite2SiteVPNMultipleOptions>:setup | `Error` | 0.00 | test_vpc_vpn.py
   ContextSuite context=TestVpcRemoteAccessVpn>:setup | `Error` | 0.00 | test_vpc_vpn.py
   ContextSuite context=TestVpcSite2SiteVpn>:setup | `Error` | 0.00 | test_vpc_vpn.py
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-600171769
 
 
   Packaging result: ✖centos6 ✖centos7 ✔debian. JID-1067

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] DaanHoogland commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

DaanHoogland commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-610223450
 
 
   @GabrielBrascher @andrijapanicsb (@harikrishna-patnala ) that last commit seems to be a state never tested. I don't think the code allows or should allow for a NoSuchTransitionException so I do not know how we can force it to happen.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] DaanHoogland removed a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

DaanHoogland removed a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-610223673
 
 
   @blueorangutan package

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-599542708
 
 
   @DaanHoogland a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] GabrielBrascher commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

GabrielBrascher commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-604436002
 
 
   @andrijapanicsb thank you very much for all the tests. Here follows an update on  `snapshot.backup.to.secondary = FALSE`.
   
   I have been testing this implementation (tested with 4.13.1 and also on a port to 4.14.0). The snapshotting with `snapshot.backup.to.secondary = FALSE` seems to be working fine for me. Created and deleted on (i) XenServer + NFS, (ii) KVM + NFS, (iii) KVM + Ceph.
   
   Environment:
   - CloudStack management node: Ubuntu 18.04
   - KVM nodes: Ubuntu 18.04
   - XenServer 6.5
   
   Can you please share more details on the issue that you had? Are there exceptions on management or the agent node? Is the DB  being correctly updated (snapshots, volumes, snapshot_store_ref)?
   
   Note that snapshots on NFS as primary storage is a bit different than on secondary. The snapshot handling for NFS (primary) is executed with the help of the [managesnapshot.sh](https://github.com/apache/cloudstack/blob/da865f08d30df5e177154049215fb6f4b54e5252/scripts/storage/qcow2/managesnapshot.sh#L72) script. That script on KVM + NFS (primary) creates a lvm, which makes a bit tricky to find the snapshot volume compared to other approaches (e.g. ceph: `rbd snap ls foo`, nfs secondary: `ls -lah /export/secondary/snapshots/`, XenServer: `xe vdi-list`).
   
   Basically the script runs a lvm lvcreate: `lvm lvcreate --size ${lv_bytes}b --name "${snapshotname}-cow" ${vg} >&2 || return 2`. Let's say that the volume path is `/mnt/<uuid primary>/<uuid volume>`.
   
   Thus, the snapshot path will become something as `/mnt/<uuid primary>/<uuid volume>/<uuid snapshot>`. However, one looking at the NFS storage via `ls /mnt/<uuid primary>/` will see something as `/mnt/<uuid primary>/<uuid volume>`.
   
   Is that happening on your test environment?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] andrijapanicsb edited a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

andrijapanicsb edited a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-602768635
 
 
   Not good (still testing a few more scenarios and will update this comment once done):
   
   ## XENSERVER (7.1):
   
   ### snapshot.backup.to.secondary = TRUE (default)
   MANUAL SNAPS: LGTM
   - deletion works fine - removed from both primary/secondary store, DB records in snapshost_store_reg marked as DESTROYED, all good.
   SCHEDULED SNAPS: MAYBE OK - someone should advise
   - for disk snaps we are making snap chains (parent_snapshot_id pointing to the ID of the previous scheduled snapshot) and while DB says that older snaps are deleted, it's not really true on storage side, but as those are snap diff snaps (chain)  - it makes sense that it doesn't delete the older/parent snap while there are child/linked snaps), but an older snap can be selectively deleted manually - which is a bit strange, as it would break the snap chain)
   
   ### snapshot.backup.to.secondary = FALSE
   
   MANUAL SNAPS: LGTM
   - deletion works fine - removed from both primary store, DB records in snapshost_store_ref marked as DESTROYED, all good.
   
   SCHEDULED SNAPS:
   - TO BE TESTED - will edit here once it completes+
   
   
   
   
   ## KVM (CENT7.1)
   
   ### snapshot.backup.to.secondary TRUE (default)
   MANUAL SNAPS: LGTM
   - deletion works fine - removed from both primary/secondary store, DB records in snapshost_store_reg marked as DESTROYED, all good.
   - I also see we are NOT keeping the snap on the QCOW2 file (good, as we made a copy to sec storage) - and even though the "PRIMARY" record (store_role) is still created and kept as "Ready" in snapshost_store_ref - it does get properly marked as DESTROYED when snap is deleted with no exception
   SCHEDULED SNAPS:
   - works fine, created and deleted as expected in both DB and on the primary/secondary stor.
   
   ### snapshot.backup.to.secondary = FALSE
   MANUAL SNAPS: BROKEN
   - can't even create manual KVM volume snap
   SCHEDULED SNAPS: NA
   - not tested as manual ones fail
   
   
   ### VMware not tested, 
   but will re-test that it's NOT affected once we have everything else fixed.
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] andrijapanicsb edited a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

andrijapanicsb edited a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-609950065
 
 
   Latest testing of the deletion stuff reports below
   
   (previously I have tested normal operations works fine - convert to this to that, revert (KVM only), etc, for all 3 hypervisors)
   
   ## snapshot.backup.to.secondary=FALSE
   
   Expected to work ONLY with KVM+Ceph (not other variants, not other hypervisors)
   (but the basic create/delete snap was tested with KVM+NFS to make sure nothing broken in that specific flow)
   create, delete snap works fine - GC kicks in, no exceptions - after GC runs, the "PRIMARY" (role_type=primary) row in snapshots_store_ref is left undeleted (with other snapshot.backup.to.secondary=TRUE testing as well)
   
   @rafaelweingartner please test with Ceph if possible
   
   ## snapshot.backup.to.secondary=TRUE
   
   - snaps can be created, operated with, deleted fine
   - deleted from file systems as expected (State changed to Destroyed in snapshots and snapshot_store_ref tables - "removed"date NOT set in the main "snapshots" table
   - GC will remove the IMAGE row (role_type=image) from the snapshot_store_ref table and set the "removed" date in the main "snapshots" table
   - GC will NOT remove the PRIMARY row from the snapshot_store_ref table (any of possibly many rows for the same volume (i.e. many different snapss of the same volume)
   - When a volume (root/data) is removed, then only first/latest (not sure) PRIMARY row in the snapshot_store_ref table- all other PRIMARY rows for a particular volume are left - i.e. DB garbage is left
   
   ### GC garbage workaround
   
   Remove rows from snapshot_store_ref table:
   
   > delete from snapshot_store_ref where snapshot_id in (select id from snapshots where status="Destroyed" and removed is NOT NULL); 
   
   (removed not NULL = means that the GC has run and left the garbage behind)
   
   or the safer one (volume is removed)
   
   > delete from snapshot_store_ref where snapshot_id in (select id from snapshots where status="Destroyed" and removed is NOT NULL) and volume_id in (select id from volumes where state="Expunged" and removed is NOT NULL);

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] andrijapanicsb commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

andrijapanicsb commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-610295953
 
 
   @GabrielBrascher @DaanHoogland I've tested deleting the snaps while SSVM was destroyed - indeed the delete action is "successful" but the file is left on Secondary Storage - however, it gets deleted once GC kicks in (assuming SSVM is up again at that moment) - I've confirmed this explicitly.
   
   To me this all looks good - certainly MUCH better than what we started with a few months ago - and if there are still some edge cases, we can touch that subject in 4.15 or similar.
   
   @harikrishna-patnala @GabrielBrascher @DaanHoogland - if anyone is OK with this, please LGTM it so we have a consensus - if not, please explain why

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] GabrielBrascher commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

GabrielBrascher commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-610143513
 
 
   I did a few tests and it does not delete on secondary storage.
   
   Tests considered the following scenarios (with `snapshot.backup.to.secondary=TRUE`):
   1.  KVM + Ceph (primary) & NFS (scondary).
   
   Result: snapshot deleted properly on primary storage; snapshot stays on secondary. The snapshot state is Ready at table `snapshot_store_ref`.
   
   2.  KVM + NFS (primary & secondary).
   
   Result: snapshot deleted properly on primary storage; snapshot stays on secondary. The snapshot state is Destroying at table `snapshot_store_ref`.
   
   I am debugging and re-checking previous commits to find at which point this stopped working.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan removed a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan removed a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-599639234
 
 
   Packaging result: ✖centos6 ✔centos7 ✔debian. JID-1058

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] andrijapanicsb edited a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

andrijapanicsb edited a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-602768635
 
 
   Not good (still testing a few more scenarios and will update this comment once done):
   
   ## XENSERVER (7.1) + NFS
   
   ### snapshot.backup.to.secondary = TRUE (default)
   MANUAL SNAPS: LGTM
   - deletion works fine - removed from both primary/secondary store, DB records in snapshost_store_reg marked as DESTROYED, all good.
   
   SCHEDULED SNAPS: MAYBE OK - someone should advise
   - for disk snaps we are making snap chains (parent_snapshot_id pointing to the ID of the previous scheduled snapshot) and while DB says that older snaps are deleted, it's not really true on storage side, but as those are snap diff snaps (chain)  - it makes sense that it doesn't delete the older/parent snap while there are child/linked snaps), but an older snap can be selectively deleted manually - which is a bit strange, as it would break the snap chain)
   
   ### snapshot.backup.to.secondary = FALSE
   
   MANUAL SNAPS: LGTM
   deletion works fine - removed from both primary store, DB records in snapshost_store_ref marked as DESTROYED, all good.
   
   SCHEDULED SNAPS:
   Works fine - garbage deleted in both DB and on Primary NFS server
   
   
   
   ## KVM (CENT7.1) + NFS
   
   ### snapshot.backup.to.secondary TRUE (default)
   MANUAL SNAPS: LGTM
   - deletion works fine - removed from both primary/secondary store, DB records in snapshost_store_reg marked as DESTROYED, all good.
   - I also see we are NOT keeping the snap on the QCOW2 file (good, as we made a copy to sec storage) - and even though the "PRIMARY" record (store_role) is still created and kept as "Ready" in snapshost_store_ref - it does get properly marked as DESTROYED when snap is deleted with no exception
   
   SCHEDULED SNAPS:
   - works fine, created and deleted as expected in both DB and on the primary/secondary stor.
   
   ### snapshot.backup.to.secondary = FALSE
   MANUAL SNAPS: BROKEN
   - can't even create manual KVM volume snap
   
   SCHEDULED SNAPS: NA
   - not tested as manual ones fail
   
   
   ### VMware not tested, 
   but will re-test that it's NOT affected once we have everything else fixed.
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] andrijapanicsb edited a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

andrijapanicsb edited a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-602768635
 
 
   Not good (still testing a few more scenarios and will update this comment once done):
   
   ## XENSERVER (7.1):
   
   ### snapshot.backup.to.secondary = TRUE (default)
   MANUAL SNAPS: LGTM
   - deletion works fine - removed from both primary/secondary store, DB records in snapshost_store_reg marked as DESTROYED, all good.
   SCHEDULED SNAPS: MAYBE OK - someone should advise
   - for disk snaps we are making snap chains (parent_snapshot_id pointing to the ID of the previous scheduled snapshot) and while DB says that older snaps are deleted, it's not really true on storage side, but as those are snap diff snaps (chain)  - it makes sense that it doesn't delete the older/parent snap while there are child/linked snaps), but an older snap can be selectively deleted manually - which is a bit strange, as it would break the snap chain)
   
   ### snapshot.backup.to.secondary = FALSE
   
   MANUAL SNAPS: LGTM
   deletion works fine - removed from both primary store, DB records in snapshost_store_ref marked as DESTROYED, all good.
   
   SCHEDULED SNAPS:
   TO BE TESTED - will edit here once it completes+
   
   
   
   
   ## KVM (CENT7.1)
   
   ### snapshot.backup.to.secondary TRUE (default)
   MANUAL SNAPS: LGTM
   - deletion works fine - removed from both primary/secondary store, DB records in snapshost_store_reg marked as DESTROYED, all good.
   - I also see we are NOT keeping the snap on the QCOW2 file (good, as we made a copy to sec storage) - and even though the "PRIMARY" record (store_role) is still created and kept as "Ready" in snapshost_store_ref - it does get properly marked as DESTROYED when snap is deleted with no exception
   SCHEDULED SNAPS:
   - works fine, created and deleted as expected in both DB and on the primary/secondary stor.
   
   ### snapshot.backup.to.secondary = FALSE
   MANUAL SNAPS: BROKEN
   - can't even create manual KVM volume snap
   SCHEDULED SNAPS: NA
   - not tested as manual ones fail
   
   
   ### VMware not tested, 
   but will re-test that it's NOT affected once we have everything else fixed.
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-607230649
 
 
   Packaging result: ✔centos7 ✔debian. JID-1113

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan removed a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan removed a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-610531014
 
 
   <b>Trillian test result (tid-1376)</b>
   Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
   Total time taken: 30256 seconds
   Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3969-t1376-kvm-centos7.zip
   Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
   Smoke tests completed. 76 look OK, 1 have error(s)
   Only failed tests results shown below:
   
   
   Test | Result | Time (s) | Test File
   --- | --- | --- | ---
   test_02_vpc_privategw_static_routes | `Failure` | 178.45 | test_privategw_acl.py
   test_03_vpc_privategw_restart_vpc_cleanup | `Failure` | 179.41 | test_privategw_acl.py
   test_04_rvpc_privategw_static_routes | `Failure` | 254.57 | test_privategw_acl.py
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] andrijapanicsb commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

andrijapanicsb commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-611252468
 
 
   @blueorangutan test centos7 vmware-67u3

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] GabrielBrascher commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

GabrielBrascher commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-602799830
 
 
   @andrijapanicsb thanks for all the tests, I am going to check this issue.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] andrijapanicsb commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

andrijapanicsb commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-610018095
 
 
   Tested the latest commit - to confirm proper logging:
   
   ## snapshot.backup.to.secondary=TRUE
   ### Normal case - 2 rows present, for both IMAGE/PRIMARY
   
   > 2020-04-06 20:14:44,739 DEBUG [o.a.c.s.s.DefaultSnapshotStrategy] (API-Job-Executor-2:ctx-4eecdad6 job-420 ctx-67c357b4) (logid:a8e3d3a0) Successfully deleted snapshot (id: 48) on primary storage.
   > 2020-04-06 20:14:44,739 DEBUG [o.a.c.s.s.DefaultSnapshotStrategy] (API-Job-Executor-2:ctx-4eecdad6 job-420 ctx-67c357b4) (logid:a8e3d3a0) Successfully deleted snapshot (id: 48) on secondary storage.
   
   ### case 1 - PRIMARY row manually removed from snapshot_store_ref table - only the IMAGE rows present
    (not normal, or otherwise possible to happen on it's own)
   
   > 2020-04-06 20:16:27,982 DEBUG [o.a.c.s.s.DefaultSnapshotStrategy] (API-Job-Executor-4:ctx-9fadd1ca job-423 ctx-5e982b61) (logid:b6224956) Snapshot (id: 49) **has been deleted on secondary storage.**
   > 2020-04-06 20:16:27,985 DEBUG [o.a.c.s.s.DefaultSnapshotStrategy] (API-Job-Executor-4:ctx-9fadd1ca job-423 ctx-5e982b61) (logid:b6224956) The snapshot (id: 49) **could not be found/deleted on primary storage.**
   > 2020-04-06 20:16:27,985 DEBUG [o.a.c.s.s.DefaultSnapshotStrategy] (API-Job-Executor-4:ctx-9fadd1ca job-423 ctx-5e982b61) (logid:b6224956) Successfully deleted snapshot (id: 49) on secondary storage.
   (and GC removed the IMAGE row - so all good, didn't break or something)
   
   ### case 2 - I removed manually the IMAGE row from snap_store_ref, only PRIMARY left
   
   2020-04-06 20:18:27,146 DEBUG [o.a.c.s.s.DefaultSnapshotStrategy] (API-Job-Executor-6:ctx-fd12e158 job-426 ctx-30f38c9c) (logid:13bbc143) Can't find snapshot [snapshot id: 50] on backup storage
   ...
   2020-04-06 20:18:27,475 DEBUG [o.a.c.s.s.DefaultSnapshotStrategy] (API-Job-Executor-6:ctx-fd12e158 job-426 ctx-30f38c9c) (logid:13bbc143) Successfully deleted snapshot (id: 50) on primary storage.
   
   So all good.
   
   @GabrielBrascher  after your testing, if you are happy, I plan to merge this one and shift any outstanding (DB garbage for PRIMARY in snap_store_ref) to a new use #4018 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] DaanHoogland removed a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

DaanHoogland removed a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-610262406
 
 
   @blueorangutan test matrix

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] andrijapanicsb commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

andrijapanicsb commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-613055232
 
 
   but you can test 4.13.1 RC1 and see if VMs are suspended as expected - perhaps some parts of the code were refractored or so... 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] andrijapanicsb commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

andrijapanicsb commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-610296032
 
 
   (will wait for test matrix to finish)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] harikrishna-patnala commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

harikrishna-patnala commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-608490677

Have seen couple of issues when handling deletesnapshot for snapshots backed up only in primary storage and snapshots backed up in secondary storage. I have made a fix for that but unable to push the changes due to some access issues. I'll update the PR once I resolve my access issue.0

Following are the test cases I covered after the fix is made
1. Create data volumes vol1 and vol2
2. Set global parameter snapshots.backup.to.secondary = true
3. Create snapshot snap1vol1 from vol1
4. Set global parameter snapshots.backup.to.secondary = false
5. Create snapshot snap2vol2 from vol2
6. Create snapshot snap3vol1 form vol1
7. Delete snap2vol2 // This is deletion of snapshot which is created only in primary and no entries in snapshot_store_ref table against store_role ‘Image’
8. Delete snap3vol1 // This is deletion of snapshot which is created only in primary but MS can get a snapshot_store_ref entry against store_role ‘Image’ with id of vol1 because snap1vol1 are created in both primary and secondary storage from vol1
9. Delete snap1vol1 // This is deletion of snapshot which is created both in primary and secondary

Attached the patch if someone can add it on behalf of me.
[0001-Fix-deletesnapshot-worflow-to-handle-both-snapshots-.patch.zip](https://github.com/apache/cloudstack/files/4428067/0001-Fix-deletesnapshot-worflow-to-handle-both-snapshots-.patch.zip)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

[GitHub] [cloudstack] nvazquez commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

nvazquez commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-609038460
 
 
   @GabrielBrascher @harikrishna-patnala I have pushed Hari's commit to this PR
   @blueorangutan package

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] andrijapanicsb closed pull request #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

andrijapanicsb closed pull request #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-611597661
 
 
   Packaging result: ✔centos7 ✔debian. JID-1159

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] andrijapanicsb edited a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

andrijapanicsb edited a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-602768635
 
 
   Not good (still testing a few more scenarios and will update this comment once done):
   
   ## XENSERVER (7.1):
   
   ### snapshot.backup.to.secondary = TRUE (default)
   MANUAL SNAPS: LGTM
   - deletion works fine - removed from both primary/secondary store, DB records in snapshost_store_reg marked as DESTROYED, all good.
   SCHEDULED SNAPS: MAYBE OK - someone should advise
   - for disk snaps we are making snap chains (parent_snapshot_id pointing to the ID of the previous scheduled snapshot) and while DB says that older snaps are deleted, it's not really true on storage side, but as those are snap diff snaps (chain)  - it makes sense that it doesn't delete the older/parent snap while there are child/linked snaps), but an older snap can be selectively deleted manually - which is a bit strange, as it would break the snap chain)
   
   ### snapshot.backup.to.secondary = FALSE
   
   ####MANUAL SNAPS: LGTM
   deletion works fine - removed from both primary store, DB records in snapshost_store_ref marked as DESTROYED, all good.
   
   ####SCHEDULED SNAPS:
   TO BE TESTED - will edit here once it completes+
   
   
   
   
   ## KVM (CENT7.1)
   
   ### snapshot.backup.to.secondary TRUE (default)
   MANUAL SNAPS: LGTM
   - deletion works fine - removed from both primary/secondary store, DB records in snapshost_store_reg marked as DESTROYED, all good.
   - I also see we are NOT keeping the snap on the QCOW2 file (good, as we made a copy to sec storage) - and even though the "PRIMARY" record (store_role) is still created and kept as "Ready" in snapshost_store_ref - it does get properly marked as DESTROYED when snap is deleted with no exception
   SCHEDULED SNAPS:
   - works fine, created and deleted as expected in both DB and on the primary/secondary stor.
   
   ### snapshot.backup.to.secondary = FALSE
   MANUAL SNAPS: BROKEN
   - can't even create manual KVM volume snap
   SCHEDULED SNAPS: NA
   - not tested as manual ones fail
   
   
   ### VMware not tested, 
   but will re-test that it's NOT affected once we have everything else fixed.
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-609731418
 
 
   @DaanHoogland a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] Slair1 commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

Slair1 commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-611565790
 
 
   Is this branch (4.13) missing this related PR? #3194 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] andrijapanicsb removed a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

andrijapanicsb removed a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-607219803
 
 
   @blueorangutan package

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] GabrielBrascher commented on a change in pull request #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

GabrielBrascher commented on a change in pull request #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#discussion_r393297087
 
 

 ##########
 File path: engine/storage/snapshot/src/main/java/org/apache/cloudstack/storage/snapshot/DefaultSnapshotStrategy.java
 ##########
 @@ -269,63 +262,87 @@ public boolean deleteSnapshot(Long snapshotId) {
             throw new InvalidParameterValueException("Can't delete snapshotshot " + snapshotId + " due to it is in " + snapshotVO.getState() + " Status");
         }
 
-        // first mark the snapshot as destroyed, so that ui can't see it, but we
-        // may not destroy the snapshot on the storage, as other snapshots may
-        // depend on it.
         SnapshotInfo snapshotOnImage = snapshotDataFactory.getSnapshot(snapshotId, DataStoreRole.Image);
-        if (snapshotOnImage == null) {
-            s_logger.debug("Can't find snapshot on backup storage, delete it in db");
-            snapshotDao.remove(snapshotId);
-            return true;
-        }
-
-        SnapshotObject obj = (SnapshotObject)snapshotOnImage;
-        try {
-            obj.processEvent(Snapshot.Event.DestroyRequested);
-            List<VolumeDetailVO> volumesFromSnapshot;
-            volumesFromSnapshot = _volumeDetailsDaoImpl.findDetails("SNAPSHOT_ID", String.valueOf(snapshotId), null);
 
-            if (volumesFromSnapshot.size() > 0) {
-                try {
-                    obj.processEvent(Snapshot.Event.OperationFailed);
-                } catch (NoTransitionException e1) {
-                    s_logger.debug("Failed to change snapshot state: " + e1.toString());
+        boolean deletedOnSecondary = false;
+        if (snapshotOnImage == null) {
+            s_logger.debug(String.format("Can't find snapshot [snapshot id: %d] on backup storage", snapshotId));
+        } else {
+            SnapshotObject obj = (SnapshotObject)snapshotOnImage;
+            try {
+                deletedOnSecondary = deleteSnapshotOnSecundaryStorage(snapshotId, snapshotOnImage, obj);
 
 Review comment:
   going to fix the "secundary" typo

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] davidjumani commented on a change in pull request #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

davidjumani commented on a change in pull request #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#discussion_r400883706
 
 

 ##########
 File path: engine/storage/snapshot/src/main/java/org/apache/cloudstack/storage/snapshot/DefaultSnapshotStrategy.java
 ##########
 @@ -269,63 +262,87 @@ public boolean deleteSnapshot(Long snapshotId) {
             throw new InvalidParameterValueException("Can't delete snapshotshot " + snapshotId + " due to it is in " + snapshotVO.getState() + " Status");
         }
 
-        // first mark the snapshot as destroyed, so that ui can't see it, but we
-        // may not destroy the snapshot on the storage, as other snapshots may
-        // depend on it.
         SnapshotInfo snapshotOnImage = snapshotDataFactory.getSnapshot(snapshotId, DataStoreRole.Image);
-        if (snapshotOnImage == null) {
-            s_logger.debug("Can't find snapshot on backup storage, delete it in db");
-            snapshotDao.remove(snapshotId);
-            return true;
-        }
-
-        SnapshotObject obj = (SnapshotObject)snapshotOnImage;
-        try {
-            obj.processEvent(Snapshot.Event.DestroyRequested);
-            List<VolumeDetailVO> volumesFromSnapshot;
-            volumesFromSnapshot = _volumeDetailsDaoImpl.findDetails("SNAPSHOT_ID", String.valueOf(snapshotId), null);
 
-            if (volumesFromSnapshot.size() > 0) {
-                try {
-                    obj.processEvent(Snapshot.Event.OperationFailed);
-                } catch (NoTransitionException e1) {
-                    s_logger.debug("Failed to change snapshot state: " + e1.toString());
+        boolean deletedOnSecondary = false;
+        if (snapshotOnImage == null) {
+            s_logger.debug(String.format("Can't find snapshot [snapshot id: %d] on backup storage", snapshotId));
 
 Review comment:
   Can remove the snapshot entry here instead ?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] andrijapanicsb removed a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

andrijapanicsb removed a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-611252468
 
 
   @blueorangutan test centos7 vmware-67u3

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan removed a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan removed a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-600847953
 
 
   @andrijapanicsb a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan removed a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan removed a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-610700547
 
 
   <b>Trillian test result (tid-1385)</b>
   Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server 7
   Total time taken: 39186 seconds
   Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3969-t1385-vmware-67u3.zip
   Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
   Intermittent failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
   Smoke tests completed. 75 look OK, 2 have error(s)
   Only failed tests results shown below:
   
   
   Test | Result | Time (s) | Test File
   --- | --- | --- | ---
   test_02_vpc_privategw_static_routes | `Failure` | 317.48 | test_privategw_acl.py
   test_03_vpc_privategw_restart_vpc_cleanup | `Failure` | 308.34 | test_privategw_acl.py
   test_04_rvpc_privategw_static_routes | `Failure` | 519.60 | test_privategw_acl.py
   test_11_migrate_vm | `Error` | 55.87 | test_vm_life_cycle.py
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan removed a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan removed a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-609722360
 
 
   @DaanHoogland a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-612222927
 
 
   <b>Trillian test result (tid-1404)</b>
   Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
   Total time taken: 30295 seconds
   Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3969-t1404-kvm-centos7.zip
   Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
   Smoke tests completed. 76 look OK, 1 have error(s)
   Only failed tests results shown below:
   
   
   Test | Result | Time (s) | Test File
   --- | --- | --- | ---
   test_02_vpc_privategw_static_routes | `Failure` | 186.51 | test_privategw_acl.py
   test_03_vpc_privategw_restart_vpc_cleanup | `Failure` | 179.71 | test_privategw_acl.py
   test_04_rvpc_privategw_static_routes | `Failure` | 244.52 | test_privategw_acl.py
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] DaanHoogland commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

DaanHoogland commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-609999593
 
 
   @blueorangutan package

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] GabrielBrascher commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

GabrielBrascher commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-610149382
 
 
   @DaanHoogland @andrijapanicsb please ignore the previous test results.
   
   The issues that I got were related with SSVM not working (VM running, but agent was down). Therefore, those failures when deleting snapshot on secondary storage are not related with this PR.
   
   The fact that the deletion worked without any exception even with SSVM failing to delete de snapshot on secondary is a problem. It could lead to snapshot garbage issue on secondary storage. But that is not critical and we can (should) leave such issue for a future release.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-607219952
 
 
   @andrijapanicsb a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-600175161
 
 
   @DaanHoogland a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] andrijapanicsb removed a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

andrijapanicsb removed a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-610827367
 
 
   XS test that needs investigation:
   
   2020-04-07 16:45:40,456 - CRITICAL - EXCEPTION: test_03_list_snapshots: ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.7/site-packages/nose/suite.py", line 228, in run\n    self.tearDown()\n', '  File "/usr/lib/python2.7/site-packages/nose/suite.py", line 351, in tearDown\n    self.teardownContext(ancestor)\n', '  File "/usr/lib/python2.7/site-packages/nose/suite.py", line 367, in teardownContext\n    try_run(context, names)\n', '  File "/usr/lib/python2.7/site-packages/nose/util.py", line 471, in try_run\n    return func()\n', '  File "/marvin/tests/smoke/test_list_ids_parameter.py", line 209, in tearDownClass\n    raise Exception("Warning: Exception during cleanup : %s" % e)\n', "Exception: Warning: Exception during cleanup : Job failed: {jobprocstatus : 0, created : u'2020-04-07T16:46:13+0000', jobresult : {errorcode : 530, errortext : u**'Failed to delete snapshot'**}, cmd : u'org.apache.cloudstack.api.command.user.snapshot.DeleteSnapshotCmd', userid : u'964f52cb-78d6-11ea-bc32-1e00400108b3', jobstatus : 2, jobid : u'8806aed0-896c-4264-801e-d570217f4ae0', jobresultcode : 530, jobinstanceid : u'4cdc1214-942c-436c-b6ae-738f3220b1cf', jobresulttype : u'object', jobinstancetype : u'Snapshot', completed : u'2020-04-07T16:46:13+0000', accountid : u'964f4d54-78d6-11ea-bc32-1e00400108b3'}\n"]
   
   VMware tests that failed:
   
   'org.apache.cloudstack.api.command.admin.vm.MigrateVMCmd\', userid : u\'3d686233-78da-11ea-8ab1-1e009101089c\', jobstatus : 2, jobid : u\'aace648a-e269-49f8-982b-3c07bf5baa11\', jobresultcode : 530, jobresulttype : u\'object\', jobresult : {errorcode : 530, errortext : u"MigrationCommand failed due to Exception: java.lang.RuntimeException\\nMessage: **The target host does not support the virtual machine\'s current hardware requirements.**\\n"}, accountid : u\'3d685acc-78da-11ea-8ab1-1e009101089c\'}\n']
   That is funny...

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] Slair1 commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

Slair1 commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-611702879
 
 
   > So it does @Slair1 ...
   
   @andrijapanicsb could you double-check the code?  When i manually check 4.13, i don't see the lines added in the screenshot below from #3194 .  
   ![image](https://user-images.githubusercontent.com/17735593/78931434-90f0a200-7a6b-11ea-829c-59a93703a5f9.png)
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-611252716
 
 
   @andrijapanicsb a Trillian-Jenkins test job (centos7 mgmt + vmware-67u3) has been kicked to run smoke tests

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan removed a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan removed a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-606348503
 
 
   <b>Trillian test result (tid-1336)</b>
   Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7
   Total time taken: 29085 seconds
   Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3969-t1336-xenserver-71.zip
   Intermittent failure detected: /marvin/tests/smoke/test_list_ids_parameter.py
   Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
   Intermittent failure detected: /marvin/tests/smoke/test_scale_vm.py
   Smoke tests completed. 74 look OK, 3 have error(s)
   Only failed tests results shown below:
   
   
   Test | Result | Time (s) | Test File
   --- | --- | --- | ---
   ContextSuite context=TestListIdsParams>:teardown | `Error` | 1.08 | test_list_ids_parameter.py
   test_02_vpc_privategw_static_routes | `Failure` | 227.08 | test_privategw_acl.py
   test_03_vpc_privategw_restart_vpc_cleanup | `Failure` | 214.64 | test_privategw_acl.py
   test_04_rvpc_privategw_static_routes | `Failure` | 326.09 | test_privategw_acl.py
   test_01_scale_vm | `Failure` | 20.60 | test_scale_vm.py
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] DaanHoogland commented on a change in pull request #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

DaanHoogland commented on a change in pull request #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#discussion_r403915720
 
 

 ##########
 File path: engine/storage/snapshot/src/main/java/org/apache/cloudstack/storage/snapshot/DefaultSnapshotStrategy.java
 ##########
 @@ -285,7 +284,27 @@ public boolean deleteSnapshot(Long snapshotId) {
             }
         }
 
-        boolean deletedOnPrimary = deleteSnapshotOnPrimary(snapshotId);
+        boolean deletedOnPrimary = false;
+        snapshotVO = snapshotDao.findById(snapshotId);
+        SnapshotInfo snapshotOnPrimaryInfo = snapshotDataFactory.getSnapshot(snapshotId, DataStoreRole.Primary);
+        if (snapshotVO != null && snapshotVO.getState() == Snapshot.State.Destroyed) {
+            deletedOnPrimary = deleteSnapshotOnPrimary(snapshotId, snapshotOnPrimaryInfo);
+        } else {
+            // This is to handle snapshots which are created only on primary when snapshot.backup.to.secondary is set to false. 
+            SnapshotObject obj = (SnapshotObject) snapshotOnPrimaryInfo;
+            try {
+                obj.processEvent(Snapshot.Event.DestroyRequested);
+                deletedOnPrimary = deleteSnapshotOnPrimary(snapshotId, snapshotOnPrimaryInfo);
+                if (!deletedOnPrimary) {
+                    obj.processEvent(Snapshot.Event.OperationFailed);
+                } else {
+                    obj.processEvent(Snapshot.Event.OperationSucceeded);
+                }
+            } catch (NoTransitionException e) {
+            s_logger.debug("Failed to set the state to destroying: ", e);
+            return false;
+            }
+        }
 
 Review comment:
   @harikrishna-patnala can you factor this bit out, please?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-610262736
 
 
   @DaanHoogland a Trillian-Jenkins matrix job (centos7 mgmt + xs71, centos7 mgmt + vmware67, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] andrijapanicsb commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

andrijapanicsb commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-610019986
 
 
   not gonna run ANY test or package rebuild after the last commit - a single word was changed in the comment/logging (from "backup" to "secondary")

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] GabrielBrascher removed a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

GabrielBrascher removed a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-600081395
 
 
   @blueorangutan package

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] GabrielBrascher commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

GabrielBrascher commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-606109537
 
 
   @andrijapanicsb sounds good to me, considering the raised context and all tests done so far. Let's run a few more tests then.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-612221456
 
 
   <b>Trillian test result (tid-1403)</b>
   Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7
   Total time taken: 30283 seconds
   Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3969-t1403-xenserver-71.zip
   Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
   Intermittent failure detected: /marvin/tests/smoke/test_scale_vm.py
   Smoke tests completed. 75 look OK, 2 have error(s)
   Only failed tests results shown below:
   
   
   Test | Result | Time (s) | Test File
   --- | --- | --- | ---
   test_02_vpc_privategw_static_routes | `Failure` | 252.71 | test_privategw_acl.py
   test_03_vpc_privategw_restart_vpc_cleanup | `Failure` | 244.28 | test_privategw_acl.py
   test_04_rvpc_privategw_static_routes | `Failure` | 323.21 | test_privategw_acl.py
   test_01_scale_vm | `Failure` | 24.57 | test_scale_vm.py
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan removed a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan removed a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-609963735
 
 
   @DaanHoogland a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] DaanHoogland commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

DaanHoogland commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-612362608
 
 
   those were spurious failures obviously, @andrijapanicsb as they passed after applying insanity. If you manual verification checks out we should merge and release.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] GabrielBrascher commented on a change in pull request #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

GabrielBrascher commented on a change in pull request #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#discussion_r401695072
 
 

 ##########
 File path: engine/storage/snapshot/src/main/java/org/apache/cloudstack/storage/snapshot/DefaultSnapshotStrategy.java
 ##########
 @@ -269,63 +262,87 @@ public boolean deleteSnapshot(Long snapshotId) {
             throw new InvalidParameterValueException("Can't delete snapshotshot " + snapshotId + " due to it is in " + snapshotVO.getState() + " Status");
         }
 
-        // first mark the snapshot as destroyed, so that ui can't see it, but we
-        // may not destroy the snapshot on the storage, as other snapshots may
-        // depend on it.
         SnapshotInfo snapshotOnImage = snapshotDataFactory.getSnapshot(snapshotId, DataStoreRole.Image);
-        if (snapshotOnImage == null) {
-            s_logger.debug("Can't find snapshot on backup storage, delete it in db");
-            snapshotDao.remove(snapshotId);
-            return true;
-        }
-
-        SnapshotObject obj = (SnapshotObject)snapshotOnImage;
-        try {
-            obj.processEvent(Snapshot.Event.DestroyRequested);
-            List<VolumeDetailVO> volumesFromSnapshot;
-            volumesFromSnapshot = _volumeDetailsDaoImpl.findDetails("SNAPSHOT_ID", String.valueOf(snapshotId), null);
 
-            if (volumesFromSnapshot.size() > 0) {
-                try {
-                    obj.processEvent(Snapshot.Event.OperationFailed);
-                } catch (NoTransitionException e1) {
-                    s_logger.debug("Failed to change snapshot state: " + e1.toString());
+        boolean deletedOnSecondary = false;
+        if (snapshotOnImage == null) {
+            s_logger.debug(String.format("Can't find snapshot [snapshot id: %d] on backup storage", snapshotId));
+        } else {
+            SnapshotObject obj = (SnapshotObject)snapshotOnImage;
+            try {
+                deletedOnSecondary = deleteSnapshotOnSecondaryStorage(snapshotId, snapshotOnImage, obj);
+                if (!deletedOnSecondary) {
+                    s_logger.debug(
+                            String.format("Failed to find/delete snapshot (id: %d) on secondary storage. Still necessary to check and delete snapshot on primary storage.",
+                                    snapshotId));
+                } else {
+                    s_logger.debug(String.format("Snapshot (id: %d) has been deleted on secondary storage.", snapshotId));
                 }
-                throw new InvalidParameterValueException("Unable to perform delete operation, Snapshot with id: " + snapshotId + " is in use  ");
+            } catch (NoTransitionException e) {
+                s_logger.debug("Failed to set the state to destroying: ", e);
+                return false;
             }
-        } catch (NoTransitionException e) {
-            s_logger.debug("Failed to set the state to destroying: ", e);
-            return false;
         }
 
-        try {
-            boolean result = deleteSnapshotChain(snapshotOnImage);
-            obj.processEvent(Snapshot.Event.OperationSucceeded);
-            if (result) {
-                //snapshot is deleted on backup storage, need to delete it on primary storage
-                SnapshotDataStoreVO snapshotOnPrimary = snapshotStoreDao.findBySnapshot(snapshotId, DataStoreRole.Primary);
-                if (snapshotOnPrimary != null) {
-                    SnapshotInfo snapshotOnPrimaryInfo = snapshotDataFactory.getSnapshot(snapshotId, DataStoreRole.Primary);
-                    long volumeId = snapshotOnPrimary.getVolumeId();
-                    VolumeVO volumeVO = volumeDao.findById(volumeId);
-                    if (((PrimaryDataStoreImpl)snapshotOnPrimaryInfo.getDataStore()).getPoolType() == StoragePoolType.RBD && volumeVO != null) {
-                        snapshotSvr.deleteSnapshot(snapshotOnPrimaryInfo);
-                    }
-                    snapshotOnPrimary.setState(State.Destroyed);
-                    snapshotStoreDao.update(snapshotOnPrimary.getId(), snapshotOnPrimary);
-                }
-            }
-        } catch (Exception e) {
-            s_logger.debug("Failed to delete snapshot: ", e);
+        boolean deletedOnPrimary = deleteSnapshotOnPrimary(snapshotId);
+        if (deletedOnPrimary) {
+            s_logger.debug(String.format("Successfully deleted snapshot (id: %d) on primary storage.", snapshotId));
+        } else if (deletedOnSecondary) {
+            s_logger.debug(String.format("The snapshot was deleted on secondary storage. Could not find/delete snapshot (id: %d) on primary storage.", snapshotId));
+        }
+        return deletedOnSecondary || deletedOnPrimary;
+    }
+
+    /**
+     * Deletes the snapshot on secondary storage.
+     * It can return false when the snapshot was stored on primary storage and not backed up on secondary; therefore, the snapshot should also be deleted on primary storage even when this method returns false.
+     */
+    private boolean deleteSnapshotOnSecondaryStorage(Long snapshotId, SnapshotInfo snapshotOnImage, SnapshotObject obj) throws NoTransitionException {
+        obj.processEvent(Snapshot.Event.DestroyRequested);
+        List<VolumeDetailVO> volumesFromSnapshot;
+        volumesFromSnapshot = _volumeDetailsDaoImpl.findDetails("SNAPSHOT_ID", String.valueOf(snapshotId), null);
+
+        if (volumesFromSnapshot.size() > 0) {
             try {
                 obj.processEvent(Snapshot.Event.OperationFailed);
             } catch (NoTransitionException e1) {
-                s_logger.debug("Failed to change snapshot state: " + e.toString());
+                s_logger.debug("Failed to change snapshot state: " + e1.toString());
             }
-            return false;
+            throw new InvalidParameterValueException("Unable to perform delete operation, Snapshot with id: " + snapshotId + " is in use  ");
         }
 
-        return true;
+        boolean result = deleteSnapshotChain(snapshotOnImage);
+        obj.processEvent(Snapshot.Event.OperationSucceeded);
+        return result;
+    }
+
+    /**
+     * Deletes the snapshot on primary storage. It can return false when the snapshot was not stored on primary storage; however this does not means that it failed to delete the snapshot. </br>
+     * In case of failure, it will throw one of the following exceptions: CloudRuntimeException, InterruptedException, or ExecutionException. </br>
+     */
+    private boolean deleteSnapshotOnPrimary(Long snapshotId) {
+        SnapshotDataStoreVO snapshotOnPrimary = snapshotStoreDao.findBySnapshot(snapshotId, DataStoreRole.Primary);
+        SnapshotInfo snapshotOnPrimaryInfo = snapshotDataFactory.getSnapshot(snapshotId, DataStoreRole.Primary);
+        if (isSnapshotOnPrimaryStorage(snapshotId) && snapshotSvr.deleteSnapshot(snapshotOnPrimaryInfo)) {
+            snapshotOnPrimary.setState(State.Destroyed);
+            snapshotStoreDao.update(snapshotOnPrimary.getId(), snapshotOnPrimary);
+            snapshotDao.remove(snapshotId);
 
 Review comment:
   That looks like the issue indeed, I will take a second look at this.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] andrijapanicsb commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

andrijapanicsb commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-600847619
 
 
   @blueorangutan package

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] GabrielBrascher commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

GabrielBrascher commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-605983772
 
 
   Got it @davidjumani, thanks for the feedback.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] DaanHoogland commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

DaanHoogland commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-609999763
 
 
   yes should be !=, tnx (it's late)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan removed a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan removed a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-606129630
 
 
   @andrijapanicsb a Trillian-Jenkins matrix job (centos7 mgmt + xs71, centos7 mgmt + vmware67, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-606368942
 
 
   <b>Trillian test result (tid-1338)</b>
   Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server 7
   Total time taken: 31949 seconds
   Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3969-t1338-vmware-67u3.zip
   Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
   Intermittent failure detected: /marvin/tests/smoke/test_ssvm.py
   Intermittent failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
   Smoke tests completed. 75 look OK, 2 have error(s)
   Only failed tests results shown below:
   
   
   Test | Result | Time (s) | Test File
   --- | --- | --- | ---
   test_02_vpc_privategw_static_routes | `Failure` | 251.32 | test_privategw_acl.py
   test_03_vpc_privategw_restart_vpc_cleanup | `Failure` | 270.92 | test_privategw_acl.py
   test_04_rvpc_privategw_static_routes | `Failure` | 381.98 | test_privategw_acl.py
   test_11_migrate_vm | `Error` | 52.82 | test_vm_life_cycle.py
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] DaanHoogland commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

DaanHoogland commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-599630656
 
 
   @blueorangutan package

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] Slair1 commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

Slair1 commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-613496467
 
 
   > @Slair1 - the bug you shared (RHEL bug), was marked as WON'T FIX - and was raised against qemu-kvm-0.12.1.2-2.355.el6_4.1.x86_64
   > 
   > Can you please describe in which env (versions of qemu) you are seeing the corruption (as master/4.14 does NOT even support centos6)
   > The issue happened during emerging 4.11 into 4.12 branch - we identified WHEN it was messed up, but not HOW (which is now irrelevant)
   > 
   > I would like to first REPRODUCE this issue on centos7+ instead of just suspending the VM (now that we think to AGAIN introduce the suspend thing)
   
   Thanks @andrijapanicsb .  We have always been on CentOS7 - never 6.  We have had many corrupt VMs from using the scheduled volume snapshot functionality...  We had to turn off the ability to do volume snapshots while a VM is powered-on...  We also stopped using VM Snapshots since the Delete VM Snapshot functionality deletes snapshots the same way as volume snapshots are deleted from the qcow2 files (after the snap is backed up to secondary storage).  We are hoping the pause VM, delete snapshot, resume VM fixes this - we are still testing

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] Slair1 edited a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

Slair1 edited a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-611706617
 
 
   > Is this for volume or VM snapshots @Slair1 ? - as volume snapshots are NOT possible for KVM while the VM is running (someone thought that's interesting to do that, due to qcow2 safety...so now it's useless for KVM...)
   
   It is for volume snapshots, but you're correct the global setting kvm.snapshot.enabled needs to be enabled.  But this brings up another point, we are wondering if we need to do a suspend before deleting VM Snapshots also.  From our examination of the code, it looks like both VM Snapshots and Volume Snapshots use the java libvirt libraries to take and delete the snapshot... so I would think they could both have the same corruption issue if the snapshot is deleted while the VM is powered-on and not suspended during the snapshot delete...

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] DaanHoogland commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

DaanHoogland commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-609721773
 
 
   @blueorangutan package

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] andrijapanicsb edited a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

andrijapanicsb edited a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-610018095
 
 
   Tested the latest commit - to confirm proper logging:
   
   ## snapshot.backup.to.secondary=TRUE
   ### Normal case - 2 rows present, for both IMAGE/PRIMARY
   
   > 2020-04-06 20:14:44,739 DEBUG [o.a.c.s.s.DefaultSnapshotStrategy] (API-Job-Executor-2:ctx-4eecdad6 job-420 ctx-67c357b4) (logid:a8e3d3a0) Successfully deleted snapshot (id: 48) on primary storage.
   > 2020-04-06 20:14:44,739 DEBUG [o.a.c.s.s.DefaultSnapshotStrategy] (API-Job-Executor-2:ctx-4eecdad6 job-420 ctx-67c357b4) (logid:a8e3d3a0) Successfully deleted snapshot (id: 48) on secondary storage.
   
   ### case 1 - PRIMARY row manually removed from snapshot_store_ref table - only the IMAGE rows present
    (not normal, or otherwise possible to happen on it's own)
   
   > 2020-04-06 20:16:27,982 DEBUG [o.a.c.s.s.DefaultSnapshotStrategy] (API-Job-Executor-4:ctx-9fadd1ca job-423 ctx-5e982b61) (logid:b6224956) Snapshot (id: 49) **has been deleted on secondary storage.**
   > 2020-04-06 20:16:27,985 DEBUG [o.a.c.s.s.DefaultSnapshotStrategy] (API-Job-Executor-4:ctx-9fadd1ca job-423 ctx-5e982b61) (logid:b6224956) The snapshot (id: 49) **could not be found/deleted on primary storage.**
   > 2020-04-06 20:16:27,985 DEBUG [o.a.c.s.s.DefaultSnapshotStrategy] (API-Job-Executor-4:ctx-9fadd1ca job-423 ctx-5e982b61) (logid:b6224956) Successfully deleted snapshot (id: 49) on secondary storage.
   (and GC removed the IMAGE row - so all good, didn't break or something)
   
   ### case 2 - I removed manually the IMAGE row from snap_store_ref, only PRIMARY left
   
   > 2020-04-06 20:18:27,146 DEBUG [o.a.c.s.s.DefaultSnapshotStrategy] (API-Job-Executor-6:ctx-fd12e158 job-426 ctx-30f38c9c) (logid:13bbc143) Can't find snapshot [snapshot id: 50] on backup storage
   > ...
   > 2020-04-06 20:18:27,475 DEBUG [o.a.c.s.s.DefaultSnapshotStrategy] (API-Job-Executor-6:ctx-fd12e158 job-426 ctx-30f38c9c) (logid:13bbc143) Successfully deleted snapshot (id: 50) on primary storage.
   
   So all good.
   
   @GabrielBrascher  after your testing, if you are happy, I plan to merge this one and shift any outstanding (DB garbage for PRIMARY in snap_store_ref) to a new use #4018 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] harikrishna-patnala commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

harikrishna-patnala commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-610311500
 
 
   Based on the last testing that I have done and @andrijapanicsb's test results this is in good shape now.
   LGTM

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan removed a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan removed a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-600175161
 
 
   @DaanHoogland a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] GabrielBrascher commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

GabrielBrascher commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-608559503
 
 
   @harikrishna-patnala thanks for sharing that commit. Can you please share a branch with that commit?
   
   The access issue is due to the fact that Apache branches are open only for committers.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-610531014
 
 
   <b>Trillian test result (tid-1376)</b>
   Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
   Total time taken: 30256 seconds
   Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3969-t1376-kvm-centos7.zip
   Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
   Smoke tests completed. 76 look OK, 1 have error(s)
   Only failed tests results shown below:
   
   
   Test | Result | Time (s) | Test File
   --- | --- | --- | ---
   test_02_vpc_privategw_static_routes | `Failure` | 178.45 | test_privategw_acl.py
   test_03_vpc_privategw_restart_vpc_cleanup | `Failure` | 179.41 | test_privategw_acl.py
   test_04_rvpc_privategw_static_routes | `Failure` | 254.57 | test_privategw_acl.py
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan removed a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan removed a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-611740124
 
 
   <b>Trillian test result (tid-1397)</b>
   Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server 7
   Total time taken: 36094 seconds
   Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3969-t1397-vmware-67u3.zip
   Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
   Smoke tests completed. 76 look OK, 1 have error(s)
   Only failed tests results shown below:
   
   
   Test | Result | Time (s) | Test File
   --- | --- | --- | ---
   test_02_vpc_privategw_static_routes | `Failure` | 305.78 | test_privategw_acl.py
   test_03_vpc_privategw_restart_vpc_cleanup | `Failure` | 310.61 | test_privategw_acl.py
   test_04_rvpc_privategw_static_routes | `Failure` | 480.69 | test_privategw_acl.py
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] DaanHoogland commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

DaanHoogland commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-609730934
 
 
   @blueorangutan test

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-599577063
 
 
   @DaanHoogland a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] DaanHoogland removed a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

DaanHoogland removed a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-599576442
 
 
   @blueorangutan package

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan removed a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan removed a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-609729679
 
 
   Packaging result: ✔centos7 ✔debian. JID-1128

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] andrijapanicsb merged pull request #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

andrijapanicsb merged pull request #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan removed a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan removed a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-609731418
 
 
   @DaanHoogland a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] DaanHoogland opened a new pull request #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

DaanHoogland opened a new pull request #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969
 
 
   ## Description
   <!--- Describe your changes in detail -->
   copy of PR #3649 on joint account for easier cooperation.
   
   Tested both cases:
   - Snapshot backed up on Secondary (stored on primary and secondary) 
   - Snapshot stored only on primary storage.
   
   This fix works for KVM  and XenServer.
   
   **Note**: I changed the name of the strategy from _XenserverSnapshotStrategy_ to _DefaultSnapshotStrategy_ due to the fact that the "Xenserver" strategy handles also Ceph, KVM, and do return `StrategyPriority.DEFAULT` in any case except _REVERT_.
   
   ```
       @Override
       public StrategyPriority canHandle(Snapshot snapshot, SnapshotOperation op) {
           if (SnapshotOperation.REVERT.equals(op)) {
               long volumeId = snapshot.getVolumeId();
               VolumeVO volumeVO = volumeDao.findById(volumeId);
   
               if (volumeVO != null && ImageFormat.QCOW2.equals(volumeVO.getFormat())) {
                   return StrategyPriority.DEFAULT;
               }
               return StrategyPriority.CANT_HANDLE;
           }
           return StrategyPriority.DEFAULT;
       }
   ```
   
   Fixes: #3646
   
   ## Types of changes
   <!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
   - [ ] Breaking change (fix or feature that would cause existing functionality to change)
   - [ ] New feature (non-breaking change which adds functionality)
   - [x] Bug fix (non-breaking change which fixes an issue)
   - [ ] Enhancement (improves an existing feature and functionality)
   - [ ] Cleanup (Code refactoring and cleanup, that may add test cases)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] andrijapanicsb commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

andrijapanicsb commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-606064729
 
 
   So - yes, when using KVM+NFS and snapshot.backup.to.secondary=false - there is all kind of mess with snapshots happening - both in 4.11.3 and in master - which implies lack of proper code implementation for this combo.
   
   The rest looks good and I'm tempted to give this specific PR a LGTM (after some more testing of what was already tested before as good)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-605782913
 
 
   Packaging result: ✔centos7 ✔debian. JID-1102

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] GabrielBrascher closed pull request #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

GabrielBrascher closed pull request #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] andrijapanicsb edited a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

andrijapanicsb edited a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-602768635
 
 
   Not good (still testing a few more scenarios and will update this comment once done):
   
   ## XENSERVER (7.1):
   
   ### snapshot.backup.to.secondary = TRUE (default)
   MANUAL SNAPS: LGTM
   - deletion works fine - removed from both primary/secondary store, DB records in snapshost_store_reg marked as DESTROYED, all good.
   
   SCHEDULED SNAPS: MAYBE OK - someone should advise
   - for disk snaps we are making snap chains (parent_snapshot_id pointing to the ID of the previous scheduled snapshot) and while DB says that older snaps are deleted, it's not really true on storage side, but as those are snap diff snaps (chain)  - it makes sense that it doesn't delete the older/parent snap while there are child/linked snaps), but an older snap can be selectively deleted manually - which is a bit strange, as it would break the snap chain)
   
   ### snapshot.backup.to.secondary = FALSE
   
   MANUAL SNAPS: LGTM
   deletion works fine - removed from both primary store, DB records in snapshost_store_ref marked as DESTROYED, all good.
   
   SCHEDULED SNAPS:
   TO BE TESTED - will edit here once it completes+
   
   
   
   
   ## KVM (CENT7.1)
   
   ### snapshot.backup.to.secondary TRUE (default)
   MANUAL SNAPS: LGTM
   - deletion works fine - removed from both primary/secondary store, DB records in snapshost_store_reg marked as DESTROYED, all good.
   - I also see we are NOT keeping the snap on the QCOW2 file (good, as we made a copy to sec storage) - and even though the "PRIMARY" record (store_role) is still created and kept as "Ready" in snapshost_store_ref - it does get properly marked as DESTROYED when snap is deleted with no exception
   
   SCHEDULED SNAPS:
   - works fine, created and deleted as expected in both DB and on the primary/secondary stor.
   
   ### snapshot.backup.to.secondary = FALSE
   MANUAL SNAPS: BROKEN
   - can't even create manual KVM volume snap
   
   SCHEDULED SNAPS: NA
   - not tested as manual ones fail
   
   
   ### VMware not tested, 
   but will re-test that it's NOT affected once we have everything else fixed.
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan removed a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan removed a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-599542708
 
 
   @DaanHoogland a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-609971789
 
 
   Packaging result: ✔centos7 ✔debian. JID-1134

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-611588074
 
 
   @andrijapanicsb a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] davidjumani commented on a change in pull request #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

davidjumani commented on a change in pull request #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#discussion_r400787736
 
 

 ##########
 File path: engine/storage/snapshot/src/main/java/org/apache/cloudstack/storage/snapshot/DefaultSnapshotStrategy.java
 ##########
 @@ -269,63 +262,87 @@ public boolean deleteSnapshot(Long snapshotId) {
             throw new InvalidParameterValueException("Can't delete snapshotshot " + snapshotId + " due to it is in " + snapshotVO.getState() + " Status");
         }
 
-        // first mark the snapshot as destroyed, so that ui can't see it, but we
-        // may not destroy the snapshot on the storage, as other snapshots may
-        // depend on it.
         SnapshotInfo snapshotOnImage = snapshotDataFactory.getSnapshot(snapshotId, DataStoreRole.Image);
-        if (snapshotOnImage == null) {
-            s_logger.debug("Can't find snapshot on backup storage, delete it in db");
-            snapshotDao.remove(snapshotId);
-            return true;
-        }
-
-        SnapshotObject obj = (SnapshotObject)snapshotOnImage;
-        try {
-            obj.processEvent(Snapshot.Event.DestroyRequested);
-            List<VolumeDetailVO> volumesFromSnapshot;
-            volumesFromSnapshot = _volumeDetailsDaoImpl.findDetails("SNAPSHOT_ID", String.valueOf(snapshotId), null);
 
-            if (volumesFromSnapshot.size() > 0) {
-                try {
-                    obj.processEvent(Snapshot.Event.OperationFailed);
-                } catch (NoTransitionException e1) {
-                    s_logger.debug("Failed to change snapshot state: " + e1.toString());
+        boolean deletedOnSecondary = false;
+        if (snapshotOnImage == null) {
+            s_logger.debug(String.format("Can't find snapshot [snapshot id: %d] on backup storage", snapshotId));
+        } else {
+            SnapshotObject obj = (SnapshotObject)snapshotOnImage;
+            try {
+                deletedOnSecondary = deleteSnapshotOnSecondaryStorage(snapshotId, snapshotOnImage, obj);
+                if (!deletedOnSecondary) {
+                    s_logger.debug(
+                            String.format("Failed to find/delete snapshot (id: %d) on secondary storage. Still necessary to check and delete snapshot on primary storage.",
+                                    snapshotId));
+                } else {
+                    s_logger.debug(String.format("Snapshot (id: %d) has been deleted on secondary storage.", snapshotId));
                 }
-                throw new InvalidParameterValueException("Unable to perform delete operation, Snapshot with id: " + snapshotId + " is in use  ");
+            } catch (NoTransitionException e) {
+                s_logger.debug("Failed to set the state to destroying: ", e);
+                return false;
             }
-        } catch (NoTransitionException e) {
-            s_logger.debug("Failed to set the state to destroying: ", e);
-            return false;
         }
 
-        try {
-            boolean result = deleteSnapshotChain(snapshotOnImage);
-            obj.processEvent(Snapshot.Event.OperationSucceeded);
-            if (result) {
-                //snapshot is deleted on backup storage, need to delete it on primary storage
-                SnapshotDataStoreVO snapshotOnPrimary = snapshotStoreDao.findBySnapshot(snapshotId, DataStoreRole.Primary);
-                if (snapshotOnPrimary != null) {
-                    SnapshotInfo snapshotOnPrimaryInfo = snapshotDataFactory.getSnapshot(snapshotId, DataStoreRole.Primary);
-                    long volumeId = snapshotOnPrimary.getVolumeId();
-                    VolumeVO volumeVO = volumeDao.findById(volumeId);
-                    if (((PrimaryDataStoreImpl)snapshotOnPrimaryInfo.getDataStore()).getPoolType() == StoragePoolType.RBD && volumeVO != null) {
-                        snapshotSvr.deleteSnapshot(snapshotOnPrimaryInfo);
-                    }
-                    snapshotOnPrimary.setState(State.Destroyed);
-                    snapshotStoreDao.update(snapshotOnPrimary.getId(), snapshotOnPrimary);
-                }
-            }
-        } catch (Exception e) {
-            s_logger.debug("Failed to delete snapshot: ", e);
+        boolean deletedOnPrimary = deleteSnapshotOnPrimary(snapshotId);
+        if (deletedOnPrimary) {
+            s_logger.debug(String.format("Successfully deleted snapshot (id: %d) on primary storage.", snapshotId));
+        } else if (deletedOnSecondary) {
+            s_logger.debug(String.format("The snapshot was deleted on secondary storage. Could not find/delete snapshot (id: %d) on primary storage.", snapshotId));
+        }
+        return deletedOnSecondary || deletedOnPrimary;
+    }
+
+    /**
+     * Deletes the snapshot on secondary storage.
+     * It can return false when the snapshot was stored on primary storage and not backed up on secondary; therefore, the snapshot should also be deleted on primary storage even when this method returns false.
+     */
+    private boolean deleteSnapshotOnSecondaryStorage(Long snapshotId, SnapshotInfo snapshotOnImage, SnapshotObject obj) throws NoTransitionException {
+        obj.processEvent(Snapshot.Event.DestroyRequested);
+        List<VolumeDetailVO> volumesFromSnapshot;
+        volumesFromSnapshot = _volumeDetailsDaoImpl.findDetails("SNAPSHOT_ID", String.valueOf(snapshotId), null);
+
+        if (volumesFromSnapshot.size() > 0) {
             try {
                 obj.processEvent(Snapshot.Event.OperationFailed);
             } catch (NoTransitionException e1) {
-                s_logger.debug("Failed to change snapshot state: " + e.toString());
+                s_logger.debug("Failed to change snapshot state: " + e1.toString());
             }
-            return false;
+            throw new InvalidParameterValueException("Unable to perform delete operation, Snapshot with id: " + snapshotId + " is in use  ");
         }
 
-        return true;
+        boolean result = deleteSnapshotChain(snapshotOnImage);
+        obj.processEvent(Snapshot.Event.OperationSucceeded);
+        return result;
+    }
+
+    /**
+     * Deletes the snapshot on primary storage. It can return false when the snapshot was not stored on primary storage; however this does not means that it failed to delete the snapshot. </br>
+     * In case of failure, it will throw one of the following exceptions: CloudRuntimeException, InterruptedException, or ExecutionException. </br>
+     */
+    private boolean deleteSnapshotOnPrimary(Long snapshotId) {
+        SnapshotDataStoreVO snapshotOnPrimary = snapshotStoreDao.findBySnapshot(snapshotId, DataStoreRole.Primary);
+        SnapshotInfo snapshotOnPrimaryInfo = snapshotDataFactory.getSnapshot(snapshotId, DataStoreRole.Primary);
+        if (isSnapshotOnPrimaryStorage(snapshotId) && snapshotSvr.deleteSnapshot(snapshotOnPrimaryInfo)) {
+            snapshotOnPrimary.setState(State.Destroyed);
+            snapshotStoreDao.update(snapshotOnPrimary.getId(), snapshotOnPrimary);
+            snapshotDao.remove(snapshotId);
 
 Review comment:
   Perhaps this is leading to the record having a removed date on the snapshots table which later on causes issues during GC ?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-600490266
 
 
   @DaanHoogland a Trillian-Jenkins matrix job (centos7 mgmt + xs71, centos7 mgmt + vmware67, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] andrijapanicsb commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

andrijapanicsb commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-607322344
 
 
   I've made other extensive tests and everything works fine when snapshot.backup.to.secondary=TRUE - including GC on the snapshot_store_ref table.
   
   But snap can't be deleted when snapshot.backup.to.secondary=FALSE

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-610232215
 
 
   Packaging result: ✔centos7 ✔debian. JID-1139

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] DaanHoogland removed a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

DaanHoogland removed a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-609999763
 
 
   yes should be !=, tnx (it's late)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan removed a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan removed a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-610009843
 
 
   Packaging result: ✔centos7 ✔debian. JID-1135

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] DaanHoogland opened a new pull request #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

DaanHoogland opened a new pull request #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969
 
 
   ## Description
   <!--- Describe your changes in detail -->
   copy of PR #3649 on joint account for easier cooperation.
   
   Tested both cases:
   - Snapshot backed up on Secondary (stored on primary and secondary) 
   - Snapshot stored only on primary storage.
   
   This fix works for KVM  and XenServer.
   
   **Note**: I changed the name of the strategy from _XenserverSnapshotStrategy_ to _DefaultSnapshotStrategy_ due to the fact that the "Xenserver" strategy handles also Ceph, KVM, and do return `StrategyPriority.DEFAULT` in any case except _REVERT_.
   
   ```
       @Override
       public StrategyPriority canHandle(Snapshot snapshot, SnapshotOperation op) {
           if (SnapshotOperation.REVERT.equals(op)) {
               long volumeId = snapshot.getVolumeId();
               VolumeVO volumeVO = volumeDao.findById(volumeId);
   
               if (volumeVO != null && ImageFormat.QCOW2.equals(volumeVO.getFormat())) {
                   return StrategyPriority.DEFAULT;
               }
               return StrategyPriority.CANT_HANDLE;
           }
           return StrategyPriority.DEFAULT;
       }
   ```
   
   Fixes: #3646
   
   ## Types of changes
   <!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
   - [ ] Breaking change (fix or feature that would cause existing functionality to change)
   - [ ] New feature (non-breaking change which adds functionality)
   - [x] Bug fix (non-breaking change which fixes an issue)
   - [ ] Enhancement (improves an existing feature and functionality)
   - [ ] Cleanup (Code refactoring and cleanup, that may add test cases)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-600858038
 
 
   Packaging result: ✖centos6 ✔centos7 ✔debian. JID-1072

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] DaanHoogland removed a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

DaanHoogland removed a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-599630656
 
 
   @blueorangutan package

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-609963735
 
 
   @DaanHoogland a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] andrijapanicsb edited a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

andrijapanicsb edited a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-613655243
 
 
   tested, confirmed the issue with latest centos7 host (centos5.5 ext3 guest - simulated high IO load while deleting the snap and broke the VM)
   
   Raised PR (for master only) https://github.com/apache/cloudstack/pull/4029
   
   EDIT: well, I "broke" the image (qemu-img check showing thousands of errors) but not the EXT3 inside the VM - EXT3 can't be broken as it seems (retrying testing with centos7 and the famous XFS which has the tendency to self-crash for fun...)
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] DaanHoogland opened a new pull request #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

DaanHoogland opened a new pull request #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969
 
 
   ## Description
   <!--- Describe your changes in detail -->
   copy of PR #3649 on joint account for easier cooperation.
   
   Tested both cases:
   - Snapshot backed up on Secondary (stored on primary and secondary) 
   - Snapshot stored only on primary storage.
   
   This fix works for KVM  and XenServer.
   
   **Note**: I changed the name of the strategy from _XenserverSnapshotStrategy_ to _DefaultSnapshotStrategy_ due to the fact that the "Xenserver" strategy handles also Ceph, KVM, and do return `StrategyPriority.DEFAULT` in any case except _REVERT_.
   
   ```
       @Override
       public StrategyPriority canHandle(Snapshot snapshot, SnapshotOperation op) {
           if (SnapshotOperation.REVERT.equals(op)) {
               long volumeId = snapshot.getVolumeId();
               VolumeVO volumeVO = volumeDao.findById(volumeId);
   
               if (volumeVO != null && ImageFormat.QCOW2.equals(volumeVO.getFormat())) {
                   return StrategyPriority.DEFAULT;
               }
               return StrategyPriority.CANT_HANDLE;
           }
           return StrategyPriority.DEFAULT;
       }
   ```
   
   Fixes: #3646
   
   ## Types of changes
   <!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
   - [ ] Breaking change (fix or feature that would cause existing functionality to change)
   - [ ] New feature (non-breaking change which adds functionality)
   - [x] Bug fix (non-breaking change which fixes an issue)
   - [ ] Enhancement (improves an existing feature and functionality)
   - [ ] Cleanup (Code refactoring and cleanup, that may add test cases)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] GabrielBrascher commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

GabrielBrascher commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-600081395
 
 
   @blueorangutan package

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] DaanHoogland commented on a change in pull request #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

DaanHoogland commented on a change in pull request #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#discussion_r406056673
 
 

 ##########
 File path: engine/storage/snapshot/src/main/java/org/apache/cloudstack/storage/snapshot/DefaultSnapshotStrategy.java
 ##########
 @@ -352,14 +359,21 @@ private boolean deleteSnapshotOnSecondaryStorage(Long snapshotId, SnapshotInfo s
     }
 
     /**
-     * Deletes the snapshot on primary storage. It can return false when the snapshot was not stored on primary storage; however this does not means that it failed to delete the snapshot. </br>
-     * In case of failure, it will throw one of the following exceptions: CloudRuntimeException, InterruptedException, or ExecutionException. </br>
+     * Deletes the snapshot on primary storage. It returns true when the snapshot was not found on primary storage; </br>
+     * In case of failure while deleting the snapshot, it will throw one of the following exceptions: CloudRuntimeException, InterruptedException, or ExecutionException. </br>
      */
     private boolean deleteSnapshotOnPrimary(Long snapshotId, SnapshotInfo snapshotOnPrimaryInfo) {
         SnapshotDataStoreVO snapshotOnPrimary = snapshotStoreDao.findBySnapshot(snapshotId, DataStoreRole.Primary);
-        if (isSnapshotOnPrimaryStorage(snapshotId) && snapshotSvr.deleteSnapshot(snapshotOnPrimaryInfo)) {
-            snapshotOnPrimary.setState(State.Destroyed);
-            snapshotStoreDao.update(snapshotOnPrimary.getId(), snapshotOnPrimary);
+        if (isSnapshotOnPrimaryStorage(snapshotId)) {
+            s_logger.debug("Snapshot reference is found on primary storage for snapshot id:" + snapshotId + ", performing snapshot deletion on primary");
 
 Review comment:
   can you surround debug statements that include string manupulation in teh paramater list with `if (LOG.isDebugEnables()) {}` please

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan removed a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan removed a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-609971789
 
 
   Packaging result: ✔centos7 ✔debian. JID-1134

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] GabrielBrascher commented on a change in pull request #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

GabrielBrascher commented on a change in pull request #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#discussion_r401680996
 
 

 ##########
 File path: engine/storage/snapshot/src/main/java/org/apache/cloudstack/storage/snapshot/DefaultSnapshotStrategy.java
 ##########
 @@ -269,63 +262,87 @@ public boolean deleteSnapshot(Long snapshotId) {
             throw new InvalidParameterValueException("Can't delete snapshotshot " + snapshotId + " due to it is in " + snapshotVO.getState() + " Status");
         }
 
-        // first mark the snapshot as destroyed, so that ui can't see it, but we
-        // may not destroy the snapshot on the storage, as other snapshots may
-        // depend on it.
         SnapshotInfo snapshotOnImage = snapshotDataFactory.getSnapshot(snapshotId, DataStoreRole.Image);
-        if (snapshotOnImage == null) {
-            s_logger.debug("Can't find snapshot on backup storage, delete it in db");
-            snapshotDao.remove(snapshotId);
-            return true;
-        }
-
-        SnapshotObject obj = (SnapshotObject)snapshotOnImage;
-        try {
-            obj.processEvent(Snapshot.Event.DestroyRequested);
-            List<VolumeDetailVO> volumesFromSnapshot;
-            volumesFromSnapshot = _volumeDetailsDaoImpl.findDetails("SNAPSHOT_ID", String.valueOf(snapshotId), null);
 
-            if (volumesFromSnapshot.size() > 0) {
-                try {
-                    obj.processEvent(Snapshot.Event.OperationFailed);
-                } catch (NoTransitionException e1) {
-                    s_logger.debug("Failed to change snapshot state: " + e1.toString());
+        boolean deletedOnSecondary = false;
+        if (snapshotOnImage == null) {
+            s_logger.debug(String.format("Can't find snapshot [snapshot id: %d] on backup storage", snapshotId));
 
 Review comment:
   @davidjumani @harikrishna-patnala it might work placing `snapshotDao.remove(snapshotId)` here, indeed. 
   
   However, I see a potential problem in the case thre is no snapshot backed up on secondary and deletion fails at the primary storage. In such a case it would remove the snapshot entry before removing the snapshot on primary storage. Not sure if this could lead to a snapshot garbage issue on primary storage (instead of on secondary).
   
   What do you think? Does this observation make sense?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-609380649
 
 
   <b>Trillian test result (tid-1359)</b>
   Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
   Total time taken: 60212 seconds
   Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3969-t1359-kvm-centos7.zip
   Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
   Intermittent failure detected: /marvin/tests/smoke/test_public_ip_range.py
   Intermittent failure detected: /marvin/tests/smoke/test_reset_vm_on_reboot.py
   Intermittent failure detected: /marvin/tests/smoke/test_resource_accounting.py
   Intermittent failure detected: /marvin/tests/smoke/test_router_dhcphosts.py
   Intermittent failure detected: /marvin/tests/smoke/test_router_dns.py
   Intermittent failure detected: /marvin/tests/smoke/test_router_dnsservice.py
   Intermittent failure detected: /marvin/tests/smoke/test_routers_iptables_default_policy.py
   Intermittent failure detected: /marvin/tests/smoke/test_routers_network_ops.py
   Intermittent failure detected: /marvin/tests/smoke/test_routers.py
   Intermittent failure detected: /marvin/tests/smoke/test_secondary_storage.py
   Intermittent failure detected: /marvin/tests/smoke/test_service_offerings.py
   Intermittent failure detected: /marvin/tests/smoke/test_snapshots.py
   Intermittent failure detected: /marvin/tests/smoke/test_ssvm.py
   Intermittent failure detected: /marvin/tests/smoke/test_templates.py
   Intermittent failure detected: /marvin/tests/smoke/test_usage.py
   Intermittent failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
   Intermittent failure detected: /marvin/tests/smoke/test_vm_snapshots.py
   Intermittent failure detected: /marvin/tests/smoke/test_volumes.py
   Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
   Intermittent failure detected: /marvin/tests/smoke/test_vpc_router_nics.py
   Intermittent failure detected: /marvin/tests/smoke/test_vpc_vpn.py
   Intermittent failure detected: /marvin/tests/smoke/test_hostha_kvm.py
   Smoke tests completed. 55 look OK, 22 have error(s)
   Only failed tests results shown below:
   
   
   Test | Result | Time (s) | Test File
   --- | --- | --- | ---
   test_02_vpc_privategw_static_routes | `Failure` | 177.05 | test_privategw_acl.py
   test_03_vpc_privategw_restart_vpc_cleanup | `Failure` | 178.03 | test_privategw_acl.py
   test_04_rvpc_privategw_static_routes | `Failure` | 242.25 | test_privategw_acl.py
   ContextSuite context=TestResetVmOnReboot>:setup | `Error` | 0.00 | test_reset_vm_on_reboot.py
   ContextSuite context=TestRAMCPUResourceAccounting>:setup | `Error` | 0.00 | test_resource_accounting.py
   ContextSuite context=TestRouterDHCPHosts>:setup | `Error` | 0.00 | test_router_dhcphosts.py
   ContextSuite context=TestRouterDHCPOpts>:setup | `Error` | 0.00 | test_router_dhcphosts.py
   ContextSuite context=TestRouterDns>:setup | `Error` | 0.00 | test_router_dns.py
   test_01_sys_vm_start | `Failure` | 0.06 | test_secondary_storage.py
   ContextSuite context=TestRouterDnsService>:setup | `Error` | 0.00 | test_router_dnsservice.py
   ContextSuite context=TestRouterIpTablesPolicies>:setup | `Error` | 0.00 | test_routers_iptables_default_policy.py
   ContextSuite context=TestVPCIpTablesPolicies>:setup | `Error` | 0.00 | test_routers_iptables_default_policy.py
   ContextSuite context=TestIsolatedNetworks>:setup | `Error` | 0.00 | test_routers_network_ops.py
   ContextSuite context=TestRedundantIsolateNetworks>:setup | `Error` | 0.00 | test_routers_network_ops.py
   ContextSuite context=TestRouterServices>:setup | `Error` | 0.00 | test_routers.py
   ContextSuite context=TestCpuCapServiceOfferings>:setup | `Error` | 0.00 | test_service_offerings.py
   ContextSuite context=TestServiceOfferings>:setup | `Error` | 0.14 | test_service_offerings.py
   ContextSuite context=TestSnapshotRootDisk>:setup | `Error` | 0.00 | test_snapshots.py
   test_01_list_sec_storage_vm | `Failure` | 0.03 | test_ssvm.py
   test_02_list_cpvm_vm | `Failure` | 0.02 | test_ssvm.py
   test_03_ssvm_internals | `Failure` | 0.03 | test_ssvm.py
   test_04_cpvm_internals | `Failure` | 0.02 | test_ssvm.py
   test_05_stop_ssvm | `Failure` | 0.02 | test_ssvm.py
   test_06_stop_cpvm | `Failure` | 0.03 | test_ssvm.py
   test_07_reboot_ssvm | `Failure` | 0.03 | test_ssvm.py
   test_08_reboot_cpvm | `Failure` | 0.03 | test_ssvm.py
   test_09_destroy_ssvm | `Failure` | 0.02 | test_ssvm.py
   test_10_destroy_cpvm | `Failure` | 0.03 | test_ssvm.py
   test_02_create_template_with_checksum_sha1 | `Error` | 65.41 | test_templates.py
   test_03_create_template_with_checksum_sha256 | `Error` | 65.44 | test_templates.py
   test_04_create_template_with_checksum_md5 | `Error` | 65.36 | test_templates.py
   test_05_create_template_with_no_checksum | `Error` | 65.37 | test_templates.py
   test_02_deploy_vm_from_direct_download_template | `Error` | 1.18 | test_templates.py
   test_03_deploy_vm_wrong_checksum | `Error` | 1.26 | test_templates.py
   ContextSuite context=TestTemplates>:setup | `Error` | 17.64 | test_templates.py
   ContextSuite context=TestISOUsage>:setup | `Error` | 0.00 | test_usage.py
   ContextSuite context=TestLBRuleUsage>:setup | `Error` | 0.00 | test_usage.py
   ContextSuite context=TestNatRuleUsage>:setup | `Error` | 0.00 | test_usage.py
   ContextSuite context=TestPublicIPUsage>:setup | `Error` | 0.00 | test_usage.py
   ContextSuite context=TestSnapshotUsage>:setup | `Error` | 0.00 | test_usage.py
   ContextSuite context=TestVmUsage>:setup | `Error` | 0.00 | test_usage.py
   ContextSuite context=TestVolumeUsage>:setup | `Error` | 0.00 | test_usage.py
   ContextSuite context=TestVpnUsage>:setup | `Error` | 0.00 | test_usage.py
   ContextSuite context=Test01DeployVM>:setup | `Error` | 0.00 | test_vm_life_cycle.py
   ContextSuite context=Test02VMLifeCycle>:setup | `Error` | 0.00 | test_vm_life_cycle.py
   test_14_secure_to_secure_vm_migration | `Error` | 11.27 | test_vm_life_cycle.py
   test_15_secured_to_nonsecured_vm_migration | `Error` | 73.87 | test_vm_life_cycle.py
   test_16_nonsecured_to_secured_vm_migration | `Error` | 1.16 | test_vm_life_cycle.py
   ContextSuite context=TestVmSnapshot>:setup | `Error` | 1.56 | test_vm_snapshots.py
   ContextSuite context=TestCreateVolume>:setup | `Error` | 0.00 | test_volumes.py
   ContextSuite context=TestVolumes>:setup | `Error` | 0.00 | test_volumes.py
   ContextSuite context=TestVPCRedundancy>:setup | `Error` | 0.00 | test_vpc_redundant.py
   ContextSuite context=TestVPCNics>:setup | `Error` | 0.00 | test_vpc_router_nics.py
   ContextSuite context=TestRVPCSite2SiteVpn>:setup | `Error` | 0.00 | test_vpc_vpn.py
   ContextSuite context=TestVPCSite2SiteVPNMultipleOptions>:setup | `Error` | 0.00 | test_vpc_vpn.py
   ContextSuite context=TestVpcRemoteAccessVpn>:setup | `Error` | 0.00 | test_vpc_vpn.py
   ContextSuite context=TestVpcSite2SiteVpn>:setup | `Error` | 0.00 | test_vpc_vpn.py
   test_disable_oobm_ha_state_ineligible | `Error` | 1516.40 | test_hostha_kvm.py
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan removed a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan removed a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-610232215
 
 
   Packaging result: ✔centos7 ✔debian. JID-1139

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] DaanHoogland removed a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

DaanHoogland removed a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-609999593
 
 
   @blueorangutan package

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] Slair1 commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

Slair1 commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-611706617
 
 
   > Is this for volume or VM snapshots @Slair1 ? - as volume snapshots are NOT possible for KVM while the VM is running (someone thought that's interesting to do that, due to qcow2 safety...so now it's useless for KVM...)
   
   It is for volume snapshots, but and you're correct the global setting kvm.snapshot.enabled needs to be enabled.  But this brings up another point, we are wondering if we need to do a suspend before deleting VM Snapshots also.  From our examination of the code, it looks like both VM Snapshots and Volume Snapshots use the save libvirt libraries to take and delete the snapshot... so I would think they could both have the same corruption issue if the snapshot is deleted while the VM is powered-on and not suspended during the snapshot delete...

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] andrijapanicsb closed pull request #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

andrijapanicsb closed pull request #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] harikrishna-patnala commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

harikrishna-patnala commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-609734614
 
 
   Code changes on the last commit from @DaanHoogland looks good to me. 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan removed a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan removed a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-611437737
 
 
   Packaging result: ✔centos7 ✔debian. JID-1158

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] DaanHoogland commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

DaanHoogland commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-599556425
 
 
   using KVM/NFS, I tested this scenario:
   
   1. created VM with data disk
   2. both disks snapshotted
   3. both deleted
   4. both marked deleted in the DB and removed secondary
   5. no changes in primary (not sure if the snapshots were removed in between
   
   basically different results than reported by @GabrielBrascher .
   
   this is with @GabrielBrascher 's fix in this PR.
   that would mean we are good to go, but it could also mean there are unknown circumstances that create garbage. (cc @andrijapanicsb )

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan removed a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan removed a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-609040966
 
 
   Packaging result: ✔centos7 ✔debian. JID-1125

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] andrijapanicsb commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

andrijapanicsb commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-612006447
 
 
   @blueorangutan test

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan removed a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan removed a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-600159692
 
 
   @GabrielBrascher a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan removed a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan removed a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-607230649
 
 
   Packaging result: ✔centos7 ✔debian. JID-1113

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan removed a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan removed a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-606368942
 
 
   <b>Trillian test result (tid-1338)</b>
   Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server 7
   Total time taken: 31949 seconds
   Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3969-t1338-vmware-67u3.zip
   Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
   Intermittent failure detected: /marvin/tests/smoke/test_ssvm.py
   Intermittent failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
   Smoke tests completed. 75 look OK, 2 have error(s)
   Only failed tests results shown below:
   
   
   Test | Result | Time (s) | Test File
   --- | --- | --- | ---
   test_02_vpc_privategw_static_routes | `Failure` | 251.32 | test_privategw_acl.py
   test_03_vpc_privategw_restart_vpc_cleanup | `Failure` | 270.92 | test_privategw_acl.py
   test_04_rvpc_privategw_static_routes | `Failure` | 381.98 | test_privategw_acl.py
   test_11_migrate_vm | `Error` | 52.82 | test_vm_life_cycle.py
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan removed a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan removed a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-610024472
 
 
   <b>Trillian test result (tid-1365)</b>
   Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
   Total time taken: 31944 seconds
   Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3969-t1365-kvm-centos7.zip
   Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
   Intermittent failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
   Smoke tests completed. 76 look OK, 1 have error(s)
   Only failed tests results shown below:
   
   
   Test | Result | Time (s) | Test File
   --- | --- | --- | ---
   test_02_vpc_privategw_static_routes | `Failure` | 185.43 | test_privategw_acl.py
   test_03_vpc_privategw_restart_vpc_cleanup | `Failure` | 220.51 | test_privategw_acl.py
   test_04_rvpc_privategw_static_routes | `Failure` | 280.49 | test_privategw_acl.py
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] davidjumani commented on a change in pull request #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

davidjumani commented on a change in pull request #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#discussion_r400883706
 
 

 ##########
 File path: engine/storage/snapshot/src/main/java/org/apache/cloudstack/storage/snapshot/DefaultSnapshotStrategy.java
 ##########
 @@ -269,63 +262,87 @@ public boolean deleteSnapshot(Long snapshotId) {
             throw new InvalidParameterValueException("Can't delete snapshotshot " + snapshotId + " due to it is in " + snapshotVO.getState() + " Status");
         }
 
-        // first mark the snapshot as destroyed, so that ui can't see it, but we
-        // may not destroy the snapshot on the storage, as other snapshots may
-        // depend on it.
         SnapshotInfo snapshotOnImage = snapshotDataFactory.getSnapshot(snapshotId, DataStoreRole.Image);
-        if (snapshotOnImage == null) {
-            s_logger.debug("Can't find snapshot on backup storage, delete it in db");
-            snapshotDao.remove(snapshotId);
-            return true;
-        }
-
-        SnapshotObject obj = (SnapshotObject)snapshotOnImage;
-        try {
-            obj.processEvent(Snapshot.Event.DestroyRequested);
-            List<VolumeDetailVO> volumesFromSnapshot;
-            volumesFromSnapshot = _volumeDetailsDaoImpl.findDetails("SNAPSHOT_ID", String.valueOf(snapshotId), null);
 
-            if (volumesFromSnapshot.size() > 0) {
-                try {
-                    obj.processEvent(Snapshot.Event.OperationFailed);
-                } catch (NoTransitionException e1) {
-                    s_logger.debug("Failed to change snapshot state: " + e1.toString());
+        boolean deletedOnSecondary = false;
+        if (snapshotOnImage == null) {
+            s_logger.debug(String.format("Can't find snapshot [snapshot id: %d] on backup storage", snapshotId));
 
 Review comment:
   Can remove the snapshot entry here instead!

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] DaanHoogland commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

DaanHoogland commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-599576442
 
 
   @blueorangutan package

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] andrijapanicsb edited a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

andrijapanicsb edited a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-602768635
 
 
   Not good (still testing a few more scenarios and will update this comment once done):
   
   ## XENSERVER (7.1):
   
   ### snapshot.backup.to.secondary = TRUE (default)
   MANUAL SNAPS: LGTM
   - deletion works fine - removed from both primary/secondary store, DB records in snapshost_store_reg marked as DESTROYED, all good.
   SCHEDULED SNAPS: MAYBE OK - someone should advise
   - for disk snaps we are making snap chains (parent_snapshot_id pointing to the ID of the previous scheduled snapshot) and while DB says that older snaps are deleted, it's not really true on storage side, but as those are snap diff snaps (chain)  - it makes sense that it doesn't delete the older/parent snap while there are child/linked snaps), but an older snap can be selectively deleted manually - which is a bit strange, as it would break the snap chain)
   
   ### snapshot.backup.to.secondary = FALSE
   MANUAL SNAPS: LGTM
   - deletion works fine - removed from both primary store, DB records in snapshost_store_ref marked as DESTROYED, all good.
   
   SCHEDULED SNAPS:
   - TO BE TESTED - will edit here once it completes+
   
   
   
   
   ## KVM (CENT7.1)
   
   ### snapshot.backup.to.secondary TRUE (default)
   MANUAL SNAPS: LGTM
   - deletion works fine - removed from both primary/secondary store, DB records in snapshost_store_reg marked as DESTROYED, all good.
   - I also see we are NOT keeping the snap on the QCOW2 file (good, as we made a copy to sec storage) - and even though the "PRIMARY" record (store_role) is still created and kept as "Ready" in snapshost_store_ref - it does get properly marked as DESTROYED when snap is deleted with no exception
   SCHEDULED SNAPS:
   - works fine, created and deleted as expected in both DB and on the primary/secondary stor.
   
   ### snapshot.backup.to.secondary = FALSE
   MANUAL SNAPS: BROKEN
   - can't even create manual KVM volume snap
   SCHEDULED SNAPS: NA
   - not tested as manual ones fail
   
   
   ### VMware not tested, 
   but will re-test that it's NOT affected once we have everything else fixed.
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-610078296
 
 
   <b>Trillian test result (tid-1366)</b>
   Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server 7
   Total time taken: 37855 seconds
   Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3969-t1366-vmware-67u3.zip
   Intermittent failure detected: /marvin/tests/smoke/test_deploy_vm_root_resize.py
   Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
   Intermittent failure detected: /marvin/tests/smoke/test_ssvm.py
   Intermittent failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
   Smoke tests completed. 75 look OK, 2 have error(s)
   Only failed tests results shown below:
   
   
   Test | Result | Time (s) | Test File
   --- | --- | --- | ---
   test_02_vpc_privategw_static_routes | `Failure` | 317.01 | test_privategw_acl.py
   test_03_vpc_privategw_restart_vpc_cleanup | `Failure` | 299.80 | test_privategw_acl.py
   test_04_rvpc_privategw_static_routes | `Failure` | 455.71 | test_privategw_acl.py
   test_11_migrate_vm | `Error` | 50.84 | test_vm_life_cycle.py
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] harikrishna-patnala edited a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

harikrishna-patnala edited a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-610234553
 
 
   > @GabrielBrascher @andrijapanicsb (@harikrishna-patnala ) that last commit seems to be a state never tested. I don't think the code allows or should allow for a NoSuchTransitionException so I do not know how we can force it to happen.
   
   NoSuchTransitionException won't occur at all if all state transitions are defined properly. So only way I can think of testing is in dev testing that too by changing the state intermediately on setting breakpoints. 
   Also here returning null or returning false does the same logging.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan removed a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan removed a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-611444146
 
 
   <b>Trillian test result (tid-1393)</b>
   Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server 7
   Total time taken: 34628 seconds
   Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3969-t1393-vmware-67u3.zip
   Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
   Smoke tests completed. 76 look OK, 1 have error(s)
   Only failed tests results shown below:
   
   
   Test | Result | Time (s) | Test File
   --- | --- | --- | ---
   test_02_vpc_privategw_static_routes | `Failure` | 277.12 | test_privategw_acl.py
   test_03_vpc_privategw_restart_vpc_cleanup | `Failure` | 259.81 | test_privategw_acl.py
   test_04_rvpc_privategw_static_routes | `Failure` | 399.76 | test_privategw_acl.py
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan removed a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan removed a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-599577063
 
 
   @DaanHoogland a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] DaanHoogland commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

DaanHoogland commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-610223673
 
 
   @blueorangutan package

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] DaanHoogland commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

DaanHoogland commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-600489931
 
 
   @blueorangutan test matrix

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-600847953
 
 
   @andrijapanicsb a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan removed a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan removed a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-600185007
 
 
   Packaging result: ✖centos6 ✔centos7 ✔debian. JID-1068

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-609038645
 
 
   @nvazquez a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-610009843
 
 
   Packaging result: ✔centos7 ✔debian. JID-1135

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] DaanHoogland commented on a change in pull request #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

DaanHoogland commented on a change in pull request #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#discussion_r403917585
 
 

 ##########
 File path: engine/storage/snapshot/src/main/java/org/apache/cloudstack/storage/snapshot/DefaultSnapshotStrategy.java
 ##########
 @@ -285,7 +284,27 @@ public boolean deleteSnapshot(Long snapshotId) {
             }
         }
 
-        boolean deletedOnPrimary = deleteSnapshotOnPrimary(snapshotId);
+        boolean deletedOnPrimary = false;
+        snapshotVO = snapshotDao.findById(snapshotId);
+        SnapshotInfo snapshotOnPrimaryInfo = snapshotDataFactory.getSnapshot(snapshotId, DataStoreRole.Primary);
+        if (snapshotVO != null && snapshotVO.getState() == Snapshot.State.Destroyed) {
+            deletedOnPrimary = deleteSnapshotOnPrimary(snapshotId, snapshotOnPrimaryInfo);
+        } else {
+            // This is to handle snapshots which are created only on primary when snapshot.backup.to.secondary is set to false. 
 
 Review comment:
   this comment does not seem to be in alignment with the code. How do we conclude that we are at this point in code because of the global setting 'snapshot.backup.to.secondary'?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan removed a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan removed a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-611252716
 
 
   @andrijapanicsb a Trillian-Jenkins test job (centos7 mgmt + vmware-67u3) has been kicked to run smoke tests

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-599639234
 
 
   Packaging result: ✖centos6 ✔centos7 ✔debian. JID-1058

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] andrijapanicsb edited a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

andrijapanicsb edited a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-602768635
 
 
   Not good (still testing a few more scenarios and will update this comment once done):
   
   ## XENSERVER (7.1):
   
   ### snapshot.backup.to.secondary = TRUE (default)
   MANUAL SNAPS: LGTM
   - deletion works fine - removed from both primary/secondary store, DB records in snapshost_store_reg marked as DESTROYED, all good.
   SCHEDULED SNAPS: MAYBE OK - someone should advise
   - for disk snaps we are making snap chains (parent_snapshot_id pointing to the ID of the previous scheduled snapshot) and while DB says that older snaps are deleted, it's not really true on storage side, but as those are snap diff snaps (chain)  - it makes sense that it doesn't delete the older/parent snap while there are child/linked snaps), but an older snap can be selectively deleted manually - which is a bit strange, as it would break the snap chain)
   
   ### snapshot.backup.to.secondary = FALSE
   
   MANUAL SNAPS: LGTM
   deletion works fine - removed from both primary store, DB records in snapshost_store_ref marked as DESTROYED, all good.
   
   SCHEDULED SNAPS:
   TO BE TESTED - will edit here once it completes+
   
   
   
   
   ## KVM (CENT7.1)
   
   ### snapshot.backup.to.secondary TRUE (default)
   MANUAL SNAPS: LGTM
   - deletion works fine - removed from both primary/secondary store, DB records in snapshost_store_reg marked as DESTROYED, all good.
   - I also see we are NOT keeping the snap on the QCOW2 file (good, as we made a copy to sec storage) - and even though the "PRIMARY" record (store_role) is still created and kept as "Ready" in snapshost_store_ref - it does get properly marked as DESTROYED when snap is deleted with no exception
   SCHEDULED SNAPS:
   - works fine, created and deleted as expected in both DB and on the primary/secondary stor.
   
   ### snapshot.backup.to.secondary = FALSE
   MANUAL SNAPS: BROKEN
   - can't even create manual KVM volume snap
   SCHEDULED SNAPS: NA
   - not tested as manual ones fail
   
   
   ### VMware not tested, 
   but will re-test that it's NOT affected once we have everything else fixed.
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] harikrishna-patnala commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

harikrishna-patnala commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-608618008
 
 
   > @harikrishna-patnala thanks for sharing that commit. Can you please share a branch with that commit?
   > 
   > The access issue is due to the fact that Apache branches are open only for committers.
   
   You may please apply this patch on this branch and update the PR. I could not create a private branch, I'm a committer though but have to check about access rights.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] andrijapanicsb edited a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

andrijapanicsb edited a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-602768635
 
 
   Not good (still testing a few more scenarios and will update this comment once done):
   
   ## XENSERVER (7.1):
   
   ### snapshot.backup.to.secondary = TRUE (default)
   MANUAL SNAPS: LGTM
   - deletion works fine - removed from both primary/secondary store, DB records in snapshost_store_reg marked as DESTROYED, all good.
   
   SCHEDULED SNAPS: MAYBE OK - someone should advise
   - for disk snaps we are making snap chains (parent_snapshot_id pointing to the ID of the previous scheduled snapshot) and while DB says that older snaps are deleted, it's not really true on storage side, but as those are snap diff snaps (chain)  - it makes sense that it doesn't delete the older/parent snap while there are child/linked snaps), but an older snap can be selectively deleted manually - which is a bit strange, as it would break the snap chain)
   
   ### snapshot.backup.to.secondary = FALSE
   
   MANUAL SNAPS: LGTM
   deletion works fine - removed from both primary store, DB records in snapshost_store_ref marked as DESTROYED, all good.
   
   SCHEDULED SNAPS:
   TO BE TESTED - will edit here once it completes+
   
   
   
   
   ## KVM (CENT7.1)
   
   ### snapshot.backup.to.secondary TRUE (default)
   MANUAL SNAPS: LGTM
   - deletion works fine - removed from both primary/secondary store, DB records in snapshost_store_reg marked as DESTROYED, all good.
   - I also see we are NOT keeping the snap on the QCOW2 file (good, as we made a copy to sec storage) - and even though the "PRIMARY" record (store_role) is still created and kept as "Ready" in snapshost_store_ref - it does get properly marked as DESTROYED when snap is deleted with no exception
   SCHEDULED SNAPS:
   - works fine, created and deleted as expected in both DB and on the primary/secondary stor.
   
   ### snapshot.backup.to.secondary = FALSE
   MANUAL SNAPS: BROKEN
   - can't even create manual KVM volume snap
   SCHEDULED SNAPS: NA
   - not tested as manual ones fail
   
   
   ### VMware not tested, 
   but will re-test that it's NOT affected once we have everything else fixed.
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan removed a comment on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan removed a comment on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-606342663
 
 
   <b>Trillian test result (tid-1337)</b>
   Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
   Total time taken: 27846 seconds
   Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3969-t1337-kvm-centos7.zip
   Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
   Smoke tests completed. 76 look OK, 1 have error(s)
   Only failed tests results shown below:
   
   
   Test | Result | Time (s) | Test File
   --- | --- | --- | ---
   test_02_vpc_privategw_static_routes | `Failure` | 173.80 | test_privategw_acl.py
   test_03_vpc_privategw_restart_vpc_cleanup | `Failure` | 180.22 | test_privategw_acl.py
   test_04_rvpc_privategw_static_routes | `Failure` | 238.01 | test_privategw_acl.py
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] blueorangutan commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

blueorangutan commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-610024472
 
 
   <b>Trillian test result (tid-1365)</b>
   Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
   Total time taken: 31944 seconds
   Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3969-t1365-kvm-centos7.zip
   Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
   Intermittent failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
   Smoke tests completed. 76 look OK, 1 have error(s)
   Only failed tests results shown below:
   
   
   Test | Result | Time (s) | Test File
   --- | --- | --- | ---
   test_02_vpc_privategw_static_routes | `Failure` | 185.43 | test_privategw_acl.py
   test_03_vpc_privategw_restart_vpc_cleanup | `Failure` | 220.51 | test_privategw_acl.py
   test_04_rvpc_privategw_static_routes | `Failure` | 280.49 | test_privategw_acl.py
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [cloudstack] andrijapanicsb commented on issue #3969: Snapshot deletion issues

Posted by GitBox <gi...@apache.org>.

andrijapanicsb commented on issue #3969: Snapshot deletion issues
URL: https://github.com/apache/cloudstack/pull/3969#issuecomment-613081297
 
 
   Agree, it's gone/messed up (trying to understand what happened with some of my colleagues who are good with git) - some code refactoring happened between from 4.11 to 4.12, and it seems gone during this... (from 4.12 and onwards) - so it's NOT new issue (that code missing)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services