You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by GitBox <gi...@apache.org> on 2020/08/10 07:43:36 UTC
[GitHub] [cloudstack] Pearl1594 opened a new pull request #4053: Secondary Storage Usage Improvements
Pearl1594 opened a new pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053
## Description
This feature enables the following:
- Balanced migration of data objects from source Image store to destination Image store(s)
- Complete migration of data
- setting an image store to read-only
- viewing download progress of templates across all data stores
Related Primate PR: https://github.com/apache/cloudstack-primate/pull/326
## Types of changes
<!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
- [X] New feature (non-breaking change which adds functionality)
- [ ] Bug fix (non-breaking change which fixes an issue)
- [X] Enhancement (improves an existing feature and functionality)
- [ ] Cleanup (Code refactoring and cleanup, that may add test cases)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-682586222
@Pearl1594 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-671796705
Packaging result: ✔centos7 ✔debian. JID-1704
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-651703800
@Pearl1594 a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Management
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-623262952
Packaging result: ✔centos7 ✔debian. JID-1207
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
rhtyd commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-648556867
@blueorangutan test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-647409846
@blueorangutan package
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-657420280
@blueorangutan test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-651703526
@blueorangutan test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-648715239
@Pearl1594 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-651689167
@blueorangutan package
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on a change in pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on a change in pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#discussion_r489250509
##########
File path: test.sh
##########
@@ -0,0 +1 @@
+cks
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd closed pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
rhtyd closed pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-688066397
@Pearl1594 a Trillian-Jenkins matrix job (centos7 mgmt + xs71, centos7 mgmt + vmware67, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-682447995
Packaging result: ✖centos7 ✖centos8 ✖debian. JID-1839
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-672781143
@blueorangutan test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd closed pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
rhtyd closed pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-691019104
@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-693769791
<b>Trillian test result (tid-2753)</b>
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 67152 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4053-t2753-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_domain_vpc_offerings.py
Intermittent failure detected: /marvin/tests/smoke/test_internal_lb.py
Intermittent failure detected: /marvin/tests/smoke/test_iso.py
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_supported_versions.py
Intermittent failure detected: /marvin/tests/smoke/test_password_server.py
Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermittent failure detected: /marvin/tests/smoke/test_host_maintenance.py
Intermittent failure detected: /marvin/tests/smoke/test_hostha_kvm.py
Smoke tests completed. 80 look OK, 5 have error(s)
Only failed tests results shown below:
Test | Result | Time (s) | Test File
--- | --- | --- | ---
test_04_extract_Iso | `Failure` | 128.37 | test_iso.py
test_01_add_delete_kubernetes_supported_version | `Error` | 1812.30 | test_kubernetes_supported_versions.py
test_02_vpc_privategw_static_routes | `Failure` | 218.84 | test_privategw_acl.py
test_03_vpc_privategw_restart_vpc_cleanup | `Failure` | 207.93 | test_privategw_acl.py
test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL | `Error` | 583.06 | test_vpc_redundant.py
test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers | `Failure` | 476.67 | test_vpc_redundant.py
test_hostha_kvm_host_fencing | `Error` | 163.16 | test_hostha_kvm.py
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-688081570
@Pearl1594 a Trillian-Jenkins matrix job (centos7 mgmt + xs71, centos7 mgmt + vmware67, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] vladimirpetrov commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
vladimirpetrov commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-675109375
@blueorangutan test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-685237529
<b>Trillian test result (tid-2635)</b>
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 53526 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4053-t2635-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_router_dhcphosts.py
Intermittent failure detected: /marvin/tests/smoke/test_routers_network_ops.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermittent failure detected: /marvin/tests/smoke/test_hostha_kvm.py
Smoke tests completed. 82 look OK, 3 have error(s)
Only failed tests results shown below:
Test | Result | Time (s) | Test File
--- | --- | --- | ---
test_01_isolate_network_FW_PF_default_routes_egress_true | `Failure` | 132.96 | test_routers_network_ops.py
test_01_RVR_Network_FW_PF_SSH_default_routes_egress_true | `Failure` | 202.82 | test_routers_network_ops.py
test_04_rvpc_network_garbage_collector_nics | `Error` | 3845.21 | test_vpc_redundant.py
test_hostha_kvm_host_fencing | `Error` | 175.00 | test_hostha_kvm.py
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-690998255
Packaging result: ✖centos7 ✖centos8 ✖debian. JID-1962
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on a change in pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on a change in pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#discussion_r422977801
##########
File path: api/src/main/java/org/apache/cloudstack/api/ApiConstants.java
##########
@@ -777,6 +780,7 @@
public static final String EXITCODE = "exitcode";
public static final String TARGET_ID = "targetid";
public static final String FILES = "files";
+ public static final String FROM = "from";
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-651097835
Packaging result: ✔centos7 ✔debian. JID-1485
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-671209737
@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-682831661
@blueorangutan test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-674556730
<b>Trillian test result (tid-2467)</b>
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 41595 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4053-t2467-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_clusters.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermittent failure detected: /marvin/tests/smoke/test_hostha_kvm.py
Smoke tests completed. 82 look OK, 3 have error(s)
Only failed tests results shown below:
Test | Result | Time (s) | Test File
--- | --- | --- | ---
test_01_deploy_kubernetes_cluster | `Error` | 0.06 | test_kubernetes_clusters.py
test_02_deploy_kubernetes_ha_cluster | `Error` | 0.03 | test_kubernetes_clusters.py
test_04_deploy_and_upgrade_kubernetes_cluster | `Error` | 0.03 | test_kubernetes_clusters.py
test_05_deploy_and_upgrade_kubernetes_ha_cluster | `Error` | 0.03 | test_kubernetes_clusters.py
test_06_deploy_and_invalid_upgrade_kubernetes_cluster | `Error` | 0.03 | test_kubernetes_clusters.py
test_07_deploy_and_scale_kubernetes_cluster | `Error` | 0.03 | test_kubernetes_clusters.py
test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers | `Failure` | 323.00 | test_vpc_redundant.py
test_hostha_kvm_host_fencing | `Error` | 176.20 | test_hostha_kvm.py
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-682585760
@blueorangutan package
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-651100113
@Pearl1594 a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-674474733
@Pearl1594 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-682465773
@rhtyd seems like the the Nfs minor version feature is causing some conflict in datatypes on this branch/PR. I'll look into it and will re-kick packaging
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-690988336
@Pearl1594 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-638570834
@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-657957355
@Pearl1594 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-671797104
@blueorangutan test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-691032877
@Pearl1594 a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-674483128
@blueorangutan test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-650012673
@Pearl1594 a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-632689806
Packaging result: ✔centos7 ✔debian. JID-1239
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-651099453
@blueorangutan test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on a change in pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
rhtyd commented on a change in pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#discussion_r489228115
##########
File path: test.sh
##########
@@ -0,0 +1 @@
+cks
Review comment:
@Pearl1594 can you remove this file?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-667544572
@Pearl1594 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-684743092
@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-631713090
Packaging result: ✔centos7 ✔debian. JID-1238
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
rhtyd commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-638570653
@blueorangutan test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-691366518
<b>Trillian test result (tid-2721)</b>
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 46427 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4053-t2721-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_diagnostics.py
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_clusters.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermittent failure detected: /marvin/tests/smoke/test_hostha_kvm.py
Smoke tests completed. 83 look OK, 2 have error(s)
Only failed tests results shown below:
Test | Result | Time (s) | Test File
--- | --- | --- | ---
test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL | `Failure` | 346.27 | test_vpc_redundant.py
test_hostha_kvm_host_fencing | `Error` | 167.08 | test_hostha_kvm.py
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-667544435
@blueorangutan package
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-663387544
@blueorangutan package
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-693219588
@blueorangutan test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-682802843
Packaging result: ✔centos7 ✔centos8 ✔debian. JID-1849
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] DaanHoogland commented on a change in pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
DaanHoogland commented on a change in pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#discussion_r427807563
##########
File path: server/src/main/java/com/cloud/configuration/Config.java
##########
@@ -16,6 +16,15 @@
// under the License.
package com.cloud.configuration;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.StringTokenizer;
+
+import org.apache.cloudstack.engine.orchestration.service.NetworkOrchestrationService;
+import org.apache.cloudstack.engine.subsystem.api.storage.StoragePoolAllocator;
+import org.apache.cloudstack.framework.config.ConfigKey;
+
Review comment:
no significant changes in this file
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
rhtyd commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-673940125
@blueorangutan test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-691448052
@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-650000196
@blueorangutan package
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-651082970
@Pearl1594 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-693827065
@rhtyd - ready to merge - test failures seem unrelated to this PR
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-691450429
Packaging result: ✖centos7 ✖centos8 ✖debian. JID-1971
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-691849940
@Pearl1594 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on a change in pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on a change in pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#discussion_r427322103
##########
File path: engine/orchestration/src/main/java/org/apache/cloudstack/engine/orchestration/DataMigrationUtility.java
##########
@@ -0,0 +1,262 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.cloudstack.engine.orchestration;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+
+import javax.inject.Inject;
+
+import org.apache.cloudstack.engine.subsystem.api.storage.DataObject;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.engine.subsystem.api.storage.ObjectInDataStoreStateMachine;
+import org.apache.cloudstack.engine.subsystem.api.storage.SnapshotDataFactory;
+import org.apache.cloudstack.engine.subsystem.api.storage.SnapshotInfo;
+import org.apache.cloudstack.engine.subsystem.api.storage.TemplateDataFactory;
+import org.apache.cloudstack.engine.subsystem.api.storage.VolumeDataFactory;
+import org.apache.cloudstack.storage.ImageStoreService;
+import org.apache.cloudstack.storage.datastore.db.SnapshotDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.SnapshotDataStoreVO;
+import org.apache.cloudstack.storage.datastore.db.TemplateDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.TemplateDataStoreVO;
+import org.apache.cloudstack.storage.datastore.db.VolumeDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.VolumeDataStoreVO;
+import org.apache.log4j.Logger;
+
+import com.cloud.host.HostVO;
+import com.cloud.host.Status;
+import com.cloud.host.dao.HostDao;
+import com.cloud.storage.DataStoreRole;
+import com.cloud.storage.SnapshotVO;
+import com.cloud.storage.VMTemplateVO;
+import com.cloud.storage.dao.SnapshotDao;
+import com.cloud.storage.dao.VMTemplateDao;
+import com.cloud.utils.Pair;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.vm.SecondaryStorageVm;
+import com.cloud.vm.SecondaryStorageVmVO;
+import com.cloud.vm.VirtualMachine;
+import com.cloud.vm.dao.SecondaryStorageVmDao;
+
+public class DataMigrationUtility {
+ @Inject
+ SecondaryStorageVmDao secStorageVmDao;
+ @Inject
+ TemplateDataStoreDao templateDataStoreDao;
+ @Inject
+ SnapshotDataStoreDao snapshotDataStoreDao;
+ @Inject
+ VolumeDataStoreDao volumeDataStoreDao;
+ @Inject
+ VMTemplateDao templateDao;
+ @Inject
+ VolumeDataFactory volumeFactory;
+ @Inject
+ TemplateDataFactory templateFactory;
+ @Inject
+ SnapshotDataFactory snapshotFactory;
+ @Inject
+ HostDao hostDao;
+ @Inject
+ SnapshotDao snapshotDao;
+
+ private static final Logger s_logger = Logger.getLogger(DataMigrationUtility.class);
+
+ /** This function verifies if the given image store comprises of data objects that are not in either the "Ready" or
+ * "Allocated" state - in such a case, if the migration policy is complete, the migration is terminated
+ */
+ private boolean filesReady(Long srcDataStoreId) {
+ String[] validStates = new String[]{"Ready", "Allocated"};
+ boolean isReady = true;
+ List<TemplateDataStoreVO> templates = templateDataStoreDao.listByStoreId(srcDataStoreId);
+ for (TemplateDataStoreVO template : templates) {
+ isReady &= (Arrays.asList(validStates).contains(template.getState().toString()));
+ }
+ List<SnapshotDataStoreVO> snapshots = snapshotDataStoreDao.listByStoreId(srcDataStoreId, DataStoreRole.Image);
+ for (SnapshotDataStoreVO snapshot : snapshots) {
+ isReady &= (Arrays.asList(validStates).contains(snapshot.getState().toString()));
+ }
+ List<VolumeDataStoreVO> volumes = volumeDataStoreDao.listByStoreId(srcDataStoreId);
+ for (VolumeDataStoreVO volume : volumes) {
+ isReady &= (Arrays.asList(validStates).contains(volume.getState().toString()));
+ }
+ return isReady;
+ }
+
+ protected void checkIfCompleteMigrationPossible(ImageStoreService.MigrationPolicy policy, Long srcDataStoreId) {
+ if (policy == ImageStoreService.MigrationPolicy.COMPLETE) {
+ if (!filesReady(srcDataStoreId)) {
+ throw new CloudRuntimeException("Complete migration failed as there are data objects which are not Ready");
+ }
+ }
+ return;
+ }
+
+ protected Long getFileSize(DataObject file, Map<DataObject, Pair<List<SnapshotInfo>, Long>> snapshotChain) {
+ Long size = file.getSize();
+ Pair<List<SnapshotInfo>, Long> chain = snapshotChain.get(file);
+ if (file instanceof SnapshotInfo && chain.first() != null) {
+ size = chain.second();
+ }
+ return size;
+ }
+
+ /**
+ * Sorts the datastores in decreasing order of their free capacities, so as to make
+ * an informed decision of picking the datastore with maximum free capactiy for migration
+ */
+ protected List<Long> sortDataStores(Map<Long, Pair<Long, Long>> storageCapacities) {
+ List<Map.Entry<Long, Pair<Long, Long>>> list =
+ new LinkedList<Map.Entry<Long, Pair<Long, Long>>>((storageCapacities.entrySet()));
+
+ Collections.sort(list, new Comparator<Map.Entry<Long, Pair<Long, Long>>>() {
+ @Override
+ public int compare(Map.Entry<Long, Pair<Long, Long>> e1, Map.Entry<Long, Pair<Long, Long>> e2) {
+ return e2.getValue().first() > e1.getValue().first() ? 1 : -1;
+ }
+ });
+ HashMap<Long, Pair<Long, Long>> temp = new LinkedHashMap<>();
+ for (Map.Entry<Long, Pair<Long, Long>> value : list) {
+ temp.put(value.getKey(), value.getValue());
+ }
+
+ return new ArrayList<>(temp.keySet());
+ }
+
+ protected List<DataObject> getSortedValidSourcesList(DataStore srcDataStore, Map<DataObject, Pair<List<SnapshotInfo>, Long>> snapshotChains) {
+ List<DataObject> files = new ArrayList<>();
+ files.addAll(getAllValidTemplates(srcDataStore));
+ files.addAll(getAllValidSnapshotsAndChains(srcDataStore, snapshotChains));
+ files.addAll(getAllValidVolumes(srcDataStore));
+
+ files = sortFilesOnSize(files, snapshotChains);
+
+ return files;
+ }
+
+ protected List<DataObject> sortFilesOnSize(List<DataObject> files, Map<DataObject, Pair<List<SnapshotInfo>, Long>> snapshotChains) {
+ Collections.sort(files, new Comparator<DataObject>() {
+ @Override
+ public int compare(DataObject o1, DataObject o2) {
+ Long size1 = o1.getSize();
+ Long size2 = o2.getSize();
+ if (o1 instanceof SnapshotInfo) {
+ size1 = snapshotChains.get(o1).second();
+ }
+ if (o2 instanceof SnapshotInfo) {
+ size2 = snapshotChains.get(o2).second();
+ }
+ return (int) (size2 - size1);
+ }
+ });
+ return files;
+ }
+
+ // Gets list of all valid templates, i.e, templates in "Ready" state for migration
+ protected List<DataObject> getAllValidTemplates(DataStore srcDataStore) {
+
+ List<DataObject> files = new LinkedList<>();
+ List<TemplateDataStoreVO> templates = templateDataStoreDao.listByStoreId(srcDataStore.getId());
+ for (TemplateDataStoreVO template : templates) {
+ VMTemplateVO templateVO = templateDao.findById(template.getTemplateId());
+ if (template.getState() == ObjectInDataStoreStateMachine.State.Ready && !templateVO.isPublicTemplate()) {
+ files.add(templateFactory.getTemplate(template.getTemplateId(), srcDataStore));
+ }
+ }
+ return files;
+ }
+
+ /** Returns parent snapshots and snapshots that do not have any children; snapshotChains comprises of the snapshot chain info
+ * for each parent snapshot and the cumulative size of the chain - this is done to ensure that all the snapshots in a chain
+ * are migrated to the same datastore
+ */
Review comment:
basic idea is to follow the snapshot creation logic, if there is a chain they all go to the same datastore - otherwise snapshot is created in the datastore with max free capacity
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-650012279
@blueorangutan test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-691657250
Packaging result: ✖centos7 ✖centos8 ✖debian. JID-1983
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
rhtyd commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-657407616
I checked Travis has passed/skipped Vladi's test.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-683224878
@Pearl1594 a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
rhtyd commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-682448588
@blueorangutan package
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on a change in pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on a change in pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#discussion_r422035093
##########
File path: engine/storage/src/main/java/org/apache/cloudstack/storage/image/BaseImageStoreDriverImpl.java
##########
@@ -373,32 +375,26 @@ public void copyAsync(DataObject srcdata, DataObject destData, AsyncCompletionCa
s_logger.error(errMsg);
answer = new Answer(cmd, false, errMsg);
} else {
- boolean sent = false;
- // Find the first endpoint to which the command can be sent to
- for (EndPoint ep : eps) {
- if (getCopyCmdsCountToSpecificSSVM(ep.getId()) >= maxConcurrentCopyOpsPerSSVM) {
- continue;
+ // select endpoint with least number of commands running on them
+ EndPoint endPoint = null;
+ Long epId = ssvmWithLeastMigrateJobs();
+ if (epId == null) {
+ Collections.shuffle(eps);
+ endPoint = eps.get(0);
+ } else {
+ List<EndPoint> remainingEps = eps.stream().filter(ep -> ep.getId() != epId ).collect(Collectors.toList());
+ if (!remainingEps.isEmpty()) {
+ Collections.shuffle(remainingEps);
+ endPoint = remainingEps.get(0);
+ } else {
+ endPoint = _defaultEpSelector.getEndPointFromHostId(epId);
}
-
- CommandExecLogVO execLog = new CommandExecLogVO(ep.getId(), _secStorageVmDao.findByInstanceName(hostDao.findById(ep.getId()).getName()).getId(), cmd.getClass().getSimpleName(), 1);
- Long cmdExecId = _cmdExecLogDao.persist(execLog).getId();
- answer = ep.sendMessage(cmd);
- answer.setContextParam("cmd", cmdExecId.toString());
- sent = true;
- break;
- }
- // If both SSVMs are pre-occupied with tasks, choose the SSVM with least migrate jobs
- if (!sent) {
- // Picking endpoint with least number of copy commands running on it
- Long epId = ssvmWithLeastMigrateJobs();
- EndPoint endPoint = _defaultEpSelector.getEndPointFromHostId(epId);
- CommandExecLogVO execLog = new CommandExecLogVO(epId, _secStorageVmDao.findByInstanceName(hostDao.findById(epId).getName()).getId(), cmd.getClass().getSimpleName(), 1);
- Long cmdExecId = _cmdExecLogDao.persist(execLog).getId();
- answer = endPoint.sendMessage(cmd);
- answer.setContextParam("cmd", cmdExecId.toString());
}
+ CommandExecLogVO execLog = new CommandExecLogVO(endPoint.getId(), _secStorageVmDao.findByInstanceName(hostDao.findById(endPoint.getId()).getName()).getId(), cmd.getClass().getSimpleName(), 1);
+ Long cmdExecId = _cmdExecLogDao.persist(execLog).getId();
+ answer = endPoint.sendMessage(cmd);
+ answer.setContextParam("cmd", cmdExecId.toString());
Review comment:
Done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-648557262
@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
rhtyd commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-684561720
@blueorangutan package
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on a change in pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on a change in pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#discussion_r422080022
##########
File path: api/src/main/java/org/apache/cloudstack/api/ApiConstants.java
##########
@@ -777,6 +780,7 @@
public static final String EXITCODE = "exitcode";
public static final String TARGET_ID = "targetid";
public static final String FILES = "files";
+ public static final String FROM = "from";
Review comment:
the API is more:
migrate secondarystoragedata from=<> migrateto=<> migrationtype=<> ..
However, I could refactor that
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-676801804
<b>Trillian test result (tid-2508)</b>
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 50044 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4053-t2508-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_iso.py
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_supported_versions.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermittent failure detected: /marvin/tests/smoke/test_hostha_kvm.py
Smoke tests completed. 82 look OK, 3 have error(s)
Only failed tests results shown below:
Test | Result | Time (s) | Test File
--- | --- | --- | ---
test_01_add_delete_kubernetes_supported_version | `Error` | 486.63 | test_kubernetes_supported_versions.py
test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers | `Failure` | 428.31 | test_vpc_redundant.py
test_04_rvpc_network_garbage_collector_nics | `Error` | 3818.41 | test_vpc_redundant.py
test_hostha_kvm_host_fencing | `Error` | 372.68 | test_hostha_kvm.py
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-648217775
@blueorangutan package
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-663387849
@Pearl1594 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-648765687
<b>Trillian test result (tid-1825)</b>
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 28122 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4053-t1825-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_direct_download.py
Intermittent failure detected: /marvin/tests/smoke/test_list_ids_parameter.py
Intermittent failure detected: /marvin/tests/smoke/test_nic.py
Intermittent failure detected: /marvin/tests/smoke/test_templates.py
Intermittent failure detected: /marvin/tests/smoke/test_usage.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_router_nics.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_vpn.py
Intermittent failure detected: /marvin/tests/smoke/test_hostha_kvm.py
Smoke tests completed. 74 look OK, 9 have error(s)
Only failed tests results shown below:
Test | Result | Time (s) | Test File
--- | --- | --- | ---
ContextSuite context=TestDirectDownloadTemplates>:setup | `Error` | 0.00 | test_direct_download.py
ContextSuite context=TestListIdsParams>:setup | `Error` | 0.00 | test_list_ids_parameter.py
test_01_nic | `Error` | 52.22 | test_nic.py
test_01_create_template | `Error` | 28.75 | test_templates.py
test_CreateTemplateWithDuplicateName | `Error` | 35.80 | test_templates.py
test_01_register_template_direct_download_flag | `Error` | 0.10 | test_templates.py
test_02_deploy_vm_from_direct_download_template | `Error` | 0.00 | test_templates.py
test_03_deploy_vm_wrong_checksum | `Error` | 0.10 | test_templates.py
ContextSuite context=TestTemplates>:setup | `Error` | 223.76 | test_templates.py
test_01_template_usage | `Error` | 23.64 | test_usage.py
ContextSuite context=TestVPCRedundancy>:setup | `Error` | 0.00 | test_vpc_redundant.py
ContextSuite context=TestVPCNics>:setup | `Error` | 0.00 | test_vpc_router_nics.py
ContextSuite context=TestRVPCSite2SiteVpn>:setup | `Error` | 0.00 | test_vpc_vpn.py
ContextSuite context=TestVPCSite2SiteVPNMultipleOptions>:setup | `Error` | 0.00 | test_vpc_vpn.py
ContextSuite context=TestVpcRemoteAccessVpn>:setup | `Error` | 0.00 | test_vpc_vpn.py
ContextSuite context=TestVpcSite2SiteVpn>:setup | `Error` | 0.00 | test_vpc_vpn.py
test_disable_oobm_ha_state_ineligible | `Error` | 0.29 | test_hostha_kvm.py
test_hostha_configure_default_driver | `Error` | 0.29 | test_hostha_kvm.py
test_hostha_enable_ha_when_host_disabled | `Error` | 0.36 | test_hostha_kvm.py
test_hostha_enable_ha_when_host_disconected | `Error` | 0.28 | test_hostha_kvm.py
test_hostha_enable_ha_when_host_in_maintenance | `Error` | 0.27 | test_hostha_kvm.py
test_hostha_kvm_host_degraded | `Error` | 0.27 | test_hostha_kvm.py
test_hostha_kvm_host_fencing | `Error` | 0.33 | test_hostha_kvm.py
test_hostha_kvm_host_recovering | `Error` | 0.25 | test_hostha_kvm.py
test_remove_ha_provider_not_possible | `Error` | 0.28 | test_hostha_kvm.py
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-674483235
@Pearl1594 a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-647901352
Packaging result: ✔centos7 ✔debian. JID-1428
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-648620978
@Pearl1594 a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-653726094
<b>Trillian test result (tid-1962)</b>
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 59216 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4053-t1962-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_clusters.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Smoke tests completed. 82 look OK, 1 have error(s)
Only failed tests results shown below:
Test | Result | Time (s) | Test File
--- | --- | --- | ---
test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers | `Error` | 331.79 | test_vpc_redundant.py
test_04_rvpc_network_garbage_collector_nics | `Error` | 3843.35 | test_vpc_redundant.py
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-675952620
Packaging result: ✔centos7 ✔centos8 ✔debian. JID-1784
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd closed pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
rhtyd closed pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-651689853
@Pearl1594 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-648600922
@Pearl1594 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-644551647
@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-688080693
@blueorangutan test matrix
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-688064890
Packaging result: ✔centos7 ✔centos8 ✔debian. JID-1906
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-687650333
@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
rhtyd commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-644551475
@blueorangutan package
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-682461437
Packaging result: ✖centos7 ✖centos8 ✖debian. JID-1847
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-663791809
<b>Trillian test result (tid-2189)</b>
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 59546 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4053-t2189-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_clusters.py
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_supported_versions.py
Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
Intermittent failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Smoke tests completed. 81 look OK, 2 have error(s)
Only failed tests results shown below:
Test | Result | Time (s) | Test File
--- | --- | --- | ---
test_02_vpc_privategw_static_routes | `Failure` | 372.36 | test_privategw_acl.py
test_03_vpc_privategw_restart_vpc_cleanup | `Failure` | 304.95 | test_privategw_acl.py
ContextSuite context=Test01DeployVM>:setup | `Error` | 0.00 | test_vm_life_cycle.py
ContextSuite context=Test02VMLifeCycle>:setup | `Error` | 0.00 | test_vm_life_cycle.py
ContextSuite context=Test03SecuredVmMigration>:setup | `Error` | 0.00 | test_vm_life_cycle.py
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd closed pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
rhtyd closed pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-638078834
@vladimirpetrov a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-689870700
@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-692503998
<b>Trillian test result (tid-2733)</b>
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 70668 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4053-t2733-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_internal_lb.py
Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_vpn.py
Intermittent failure detected: /marvin/tests/smoke/test_hostha_kvm.py
Smoke tests completed. 81 look OK, 4 have error(s)
Only failed tests results shown below:
Test | Result | Time (s) | Test File
--- | --- | --- | ---
test_03_vpc_privategw_restart_vpc_cleanup | `Failure` | 358.67 | test_privategw_acl.py
test_04_rvpc_privategw_static_routes | `Failure` | 563.47 | test_privategw_acl.py
test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL | `Failure` | 450.45 | test_vpc_redundant.py
test_02_redundant_VPC_default_routes | `Failure` | 447.19 | test_vpc_redundant.py
test_01_redundant_vpc_site2site_vpn | `Failure` | 737.25 | test_vpc_vpn.py
test_01_vpc_site2site_vpn_multiple_options | `Error` | 695.53 | test_vpc_vpn.py
test_01_vpc_site2site_vpn | `Failure` | 465.04 | test_vpc_vpn.py
test_hostha_enable_ha_when_host_disabled | `Error` | 1.58 | test_hostha_kvm.py
test_hostha_enable_ha_when_host_in_maintenance | `Error` | 301.71 | test_hostha_kvm.py
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-651082218
@blueorangutan package
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-688050783
@Pearl1594 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-690969340
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
rhtyd commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-691018623
Env issue, trying again
@blueorangutan package
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
rhtyd commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-691018623
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-642568293
Packaging result: ✔centos7 ✔debian. JID-1344
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-632770636
Yes @DaanHoogland - that's how it's stored in the DB (for vmware)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-638091590
Packaging result: ✔centos7 ✔debian. JID-1270
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on a change in pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on a change in pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#discussion_r427101218
##########
File path: engine/orchestration/src/main/java/org/apache/cloudstack/engine/orchestration/DataMigrationUtility.java
##########
@@ -0,0 +1,262 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.cloudstack.engine.orchestration;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+
+import javax.inject.Inject;
+
+import org.apache.cloudstack.engine.subsystem.api.storage.DataObject;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.engine.subsystem.api.storage.ObjectInDataStoreStateMachine;
+import org.apache.cloudstack.engine.subsystem.api.storage.SnapshotDataFactory;
+import org.apache.cloudstack.engine.subsystem.api.storage.SnapshotInfo;
+import org.apache.cloudstack.engine.subsystem.api.storage.TemplateDataFactory;
+import org.apache.cloudstack.engine.subsystem.api.storage.VolumeDataFactory;
+import org.apache.cloudstack.storage.ImageStoreService;
+import org.apache.cloudstack.storage.datastore.db.SnapshotDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.SnapshotDataStoreVO;
+import org.apache.cloudstack.storage.datastore.db.TemplateDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.TemplateDataStoreVO;
+import org.apache.cloudstack.storage.datastore.db.VolumeDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.VolumeDataStoreVO;
+import org.apache.log4j.Logger;
+
+import com.cloud.host.HostVO;
+import com.cloud.host.Status;
+import com.cloud.host.dao.HostDao;
+import com.cloud.storage.DataStoreRole;
+import com.cloud.storage.SnapshotVO;
+import com.cloud.storage.VMTemplateVO;
+import com.cloud.storage.dao.SnapshotDao;
+import com.cloud.storage.dao.VMTemplateDao;
+import com.cloud.utils.Pair;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.vm.SecondaryStorageVm;
+import com.cloud.vm.SecondaryStorageVmVO;
+import com.cloud.vm.VirtualMachine;
+import com.cloud.vm.dao.SecondaryStorageVmDao;
+
+public class DataMigrationUtility {
+ @Inject
+ SecondaryStorageVmDao secStorageVmDao;
+ @Inject
+ TemplateDataStoreDao templateDataStoreDao;
+ @Inject
+ SnapshotDataStoreDao snapshotDataStoreDao;
+ @Inject
+ VolumeDataStoreDao volumeDataStoreDao;
+ @Inject
+ VMTemplateDao templateDao;
+ @Inject
+ VolumeDataFactory volumeFactory;
+ @Inject
+ TemplateDataFactory templateFactory;
+ @Inject
+ SnapshotDataFactory snapshotFactory;
+ @Inject
+ HostDao hostDao;
+ @Inject
+ SnapshotDao snapshotDao;
+
+ private static final Logger s_logger = Logger.getLogger(DataMigrationUtility.class);
+
+ /** This function verifies if the given image store comprises of data objects that are not in either the "Ready" or
+ * "Allocated" state - in such a case, if the migration policy is complete, the migration is terminated
+ */
+ private boolean filesReady(Long srcDataStoreId) {
+ String[] validStates = new String[]{"Ready", "Allocated"};
+ boolean isReady = true;
+ List<TemplateDataStoreVO> templates = templateDataStoreDao.listByStoreId(srcDataStoreId);
+ for (TemplateDataStoreVO template : templates) {
+ isReady &= (Arrays.asList(validStates).contains(template.getState().toString()));
+ }
+ List<SnapshotDataStoreVO> snapshots = snapshotDataStoreDao.listByStoreId(srcDataStoreId, DataStoreRole.Image);
+ for (SnapshotDataStoreVO snapshot : snapshots) {
+ isReady &= (Arrays.asList(validStates).contains(snapshot.getState().toString()));
+ }
+ List<VolumeDataStoreVO> volumes = volumeDataStoreDao.listByStoreId(srcDataStoreId);
+ for (VolumeDataStoreVO volume : volumes) {
+ isReady &= (Arrays.asList(validStates).contains(volume.getState().toString()));
+ }
+ return isReady;
+ }
+
+ protected void checkIfCompleteMigrationPossible(ImageStoreService.MigrationPolicy policy, Long srcDataStoreId) {
+ if (policy == ImageStoreService.MigrationPolicy.COMPLETE) {
+ if (!filesReady(srcDataStoreId)) {
+ throw new CloudRuntimeException("Complete migration failed as there are data objects which are not Ready");
+ }
+ }
+ return;
+ }
+
+ protected Long getFileSize(DataObject file, Map<DataObject, Pair<List<SnapshotInfo>, Long>> snapshotChain) {
+ Long size = file.getSize();
+ Pair<List<SnapshotInfo>, Long> chain = snapshotChain.get(file);
+ if (file instanceof SnapshotInfo && chain.first() != null) {
+ size = chain.second();
+ }
+ return size;
+ }
+
+ /**
+ * Sorts the datastores in decreasing order of their free capacities, so as to make
+ * an informed decision of picking the datastore with maximum free capactiy for migration
+ */
+ protected List<Long> sortDataStores(Map<Long, Pair<Long, Long>> storageCapacities) {
+ List<Map.Entry<Long, Pair<Long, Long>>> list =
+ new LinkedList<Map.Entry<Long, Pair<Long, Long>>>((storageCapacities.entrySet()));
+
+ Collections.sort(list, new Comparator<Map.Entry<Long, Pair<Long, Long>>>() {
+ @Override
+ public int compare(Map.Entry<Long, Pair<Long, Long>> e1, Map.Entry<Long, Pair<Long, Long>> e2) {
+ return e2.getValue().first() > e1.getValue().first() ? 1 : -1;
Review comment:
They get sorted properly even on ignoring the equality case
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-642583139
@blueorangutan test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-631701443
@vladimirpetrov a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-676926547
<b>Trillian test result (tid-2500)</b>
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 70795 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4053-t2500-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_clusters.py
Intermittent failure detected: /marvin/tests/smoke/test_public_ip_range.py
Intermittent failure detected: /marvin/tests/smoke/test_reset_vm_on_reboot.py
Intermittent failure detected: /marvin/tests/smoke/test_resource_accounting.py
Intermittent failure detected: /marvin/tests/smoke/test_router_dhcphosts.py
Intermittent failure detected: /marvin/tests/smoke/test_router_dns.py
Intermittent failure detected: /marvin/tests/smoke/test_router_dnsservice.py
Intermittent failure detected: /marvin/tests/smoke/test_routers_iptables_default_policy.py
Intermittent failure detected: /marvin/tests/smoke/test_routers_network_ops.py
Intermittent failure detected: /marvin/tests/smoke/test_routers.py
Intermittent failure detected: /marvin/tests/smoke/test_secondary_storage.py
Intermittent failure detected: /marvin/tests/smoke/test_service_offerings.py
Intermittent failure detected: /marvin/tests/smoke/test_snapshots.py
Intermittent failure detected: /marvin/tests/smoke/test_ssvm.py
Intermittent failure detected: /marvin/tests/smoke/test_templates.py
Intermittent failure detected: /marvin/tests/smoke/test_usage.py
Intermittent failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
Intermittent failure detected: /marvin/tests/smoke/test_vm_snapshots.py
Intermittent failure detected: /marvin/tests/smoke/test_volumes.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_router_nics.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_vpn.py
Intermittent failure detected: /marvin/tests/smoke/test_hostha_kvm.py
Smoke tests completed. 63 look OK, 22 have error(s)
Only failed tests results shown below:
Test | Result | Time (s) | Test File
--- | --- | --- | ---
test_01_deploy_kubernetes_cluster | `Error` | 0.16 | test_kubernetes_clusters.py
test_02_deploy_kubernetes_ha_cluster | `Error` | 0.07 | test_kubernetes_clusters.py
test_04_deploy_and_upgrade_kubernetes_cluster | `Error` | 0.05 | test_kubernetes_clusters.py
test_05_deploy_and_upgrade_kubernetes_ha_cluster | `Error` | 0.06 | test_kubernetes_clusters.py
test_06_deploy_and_invalid_upgrade_kubernetes_cluster | `Error` | 0.06 | test_kubernetes_clusters.py
test_07_deploy_and_scale_kubernetes_cluster | `Error` | 0.05 | test_kubernetes_clusters.py
ContextSuite context=TestResetVmOnReboot>:setup | `Error` | 0.00 | test_reset_vm_on_reboot.py
ContextSuite context=TestRouterDHCPHosts>:setup | `Error` | 0.00 | test_router_dhcphosts.py
ContextSuite context=TestRouterDHCPOpts>:setup | `Error` | 0.00 | test_router_dhcphosts.py
ContextSuite context=TestRAMCPUResourceAccounting>:setup | `Error` | 0.00 | test_resource_accounting.py
ContextSuite context=TestRouterDns>:setup | `Error` | 0.00 | test_router_dns.py
ContextSuite context=TestRouterDnsService>:setup | `Error` | 0.00 | test_router_dnsservice.py
ContextSuite context=TestRouterIpTablesPolicies>:setup | `Error` | 0.00 | test_routers_iptables_default_policy.py
ContextSuite context=TestVPCIpTablesPolicies>:setup | `Error` | 0.00 | test_routers_iptables_default_policy.py
ContextSuite context=TestIsolatedNetworks>:setup | `Error` | 0.00 | test_routers_network_ops.py
ContextSuite context=TestRedundantIsolateNetworks>:setup | `Error` | 0.00 | test_routers_network_ops.py
ContextSuite context=TestRouterServices>:setup | `Error` | 0.00 | test_routers.py
test_01_sys_vm_start | `Failure` | 0.25 | test_secondary_storage.py
ContextSuite context=TestCpuCapServiceOfferings>:setup | `Error` | 0.00 | test_service_offerings.py
ContextSuite context=TestServiceOfferings>:setup | `Error` | 0.17 | test_service_offerings.py
ContextSuite context=TestSnapshotRootDisk>:setup | `Error` | 0.00 | test_snapshots.py
test_01_list_sec_storage_vm | `Failure` | 0.04 | test_ssvm.py
test_02_list_cpvm_vm | `Failure` | 0.04 | test_ssvm.py
test_03_ssvm_internals | `Failure` | 0.04 | test_ssvm.py
test_04_cpvm_internals | `Failure` | 0.04 | test_ssvm.py
test_05_stop_ssvm | `Failure` | 0.04 | test_ssvm.py
test_06_stop_cpvm | `Failure` | 0.04 | test_ssvm.py
test_07_reboot_ssvm | `Failure` | 0.04 | test_ssvm.py
test_08_reboot_cpvm | `Failure` | 0.04 | test_ssvm.py
test_09_destroy_ssvm | `Failure` | 0.04 | test_ssvm.py
test_10_destroy_cpvm | `Failure` | 0.04 | test_ssvm.py
test_02_create_template_with_checksum_sha1 | `Error` | 65.58 | test_templates.py
test_03_create_template_with_checksum_sha256 | `Error` | 65.57 | test_templates.py
test_04_create_template_with_checksum_md5 | `Error` | 65.60 | test_templates.py
test_05_create_template_with_no_checksum | `Error` | 65.58 | test_templates.py
test_02_deploy_vm_from_direct_download_template | `Error` | 1.33 | test_templates.py
test_03_deploy_vm_wrong_checksum | `Error` | 1.39 | test_templates.py
ContextSuite context=TestTemplates>:setup | `Error` | 18.95 | test_templates.py
ContextSuite context=TestISOUsage>:setup | `Error` | 0.00 | test_usage.py
ContextSuite context=TestLBRuleUsage>:setup | `Error` | 0.00 | test_usage.py
ContextSuite context=TestNatRuleUsage>:setup | `Error` | 0.00 | test_usage.py
ContextSuite context=TestPublicIPUsage>:setup | `Error` | 0.00 | test_usage.py
ContextSuite context=TestSnapshotUsage>:setup | `Error` | 0.00 | test_usage.py
ContextSuite context=TestVmUsage>:setup | `Error` | 0.00 | test_usage.py
ContextSuite context=TestVolumeUsage>:setup | `Error` | 0.00 | test_usage.py
ContextSuite context=TestVpnUsage>:setup | `Error` | 0.00 | test_usage.py
ContextSuite context=Test01DeployVM>:setup | `Error` | 0.00 | test_vm_life_cycle.py
ContextSuite context=Test02VMLifeCycle>:setup | `Error` | 0.00 | test_vm_life_cycle.py
test_14_secure_to_secure_vm_migration | `Error` | 11.45 | test_vm_life_cycle.py
test_15_secured_to_nonsecured_vm_migration | `Error` | 74.26 | test_vm_life_cycle.py
test_16_nonsecured_to_secured_vm_migration | `Error` | 1.31 | test_vm_life_cycle.py
ContextSuite context=TestVmSnapshot>:setup | `Error` | 2.12 | test_vm_snapshots.py
ContextSuite context=TestCreateVolume>:setup | `Error` | 0.00 | test_volumes.py
ContextSuite context=TestVolumes>:setup | `Error` | 0.00 | test_volumes.py
ContextSuite context=TestVPCRedundancy>:setup | `Error` | 0.00 | test_vpc_redundant.py
ContextSuite context=TestVPCNics>:setup | `Error` | 0.00 | test_vpc_router_nics.py
ContextSuite context=TestRVPCSite2SiteVpn>:setup | `Error` | 0.00 | test_vpc_vpn.py
ContextSuite context=TestVPCSite2SiteVPNMultipleOptions>:setup | `Error` | 0.00 | test_vpc_vpn.py
ContextSuite context=TestVpcRemoteAccessVpn>:setup | `Error` | 0.00 | test_vpc_vpn.py
ContextSuite context=TestVpcSite2SiteVpn>:setup | `Error` | 0.00 | test_vpc_vpn.py
test_disable_oobm_ha_state_ineligible | `Error` | 1514.88 | test_hostha_kvm.py
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-657420705
@Pearl1594 a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-657957091
@blueorangutan package
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-657403191
@blueorangutan package
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-690969340
@Pearl1594 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-691623942
Packaging result: ✖centos7 ✔centos8 ✔debian. JID-1980
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] vladimirpetrov commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
vladimirpetrov commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-657400311
New Marvin tests for this feature have been added. @rhtyd and @Pearl1594 - please review them.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
rhtyd commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-675407604
Test manually rekicked
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on a change in pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on a change in pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#discussion_r426744052
##########
File path: engine/storage/src/main/java/org/apache/cloudstack/storage/datastore/ObjectInDataStoreManagerImpl.java
##########
@@ -104,6 +104,14 @@ public ObjectInDataStoreManagerImpl() {
// TODO: further investigate why an extra event is sent when it is
// alreay Ready for DownloadListener
stateMachines.addTransition(State.Ready, Event.OperationSuccessed, State.Ready);
+ // State transitions for data object migration
+ stateMachines.addTransition(State.Ready, Event.MigrationRequested, State.Migrating);
+ stateMachines.addTransition(State.Ready, Event.CopyRequested, State.Copying);
+ stateMachines.addTransition(State.Allocated, Event.MigrationRequested, State.Migrating);
+ stateMachines.addTransition(State.Migrating, Event.MigrationFailed, State.Failed);
+ stateMachines.addTransition(State.Migrating, Event.MigrationSucceeded, State.Destroyed);
+ stateMachines.addTransition(State.Migrating, Event.OperationSuccessed, State.Ready);
Review comment:
I've reused what existed before
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
rhtyd commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-691967139
@blueorangutan test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-648600651
@blueorangutan package
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-657965940
@Pearl1594 a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-646031974
@Pearl1594 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-650010674
Packaging result: ✔centos7 ✔debian. JID-1467
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
rhtyd commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-689890005
@blueorangutan package
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-674944878
Packaging result: ✔centos7 ✔debian. JID-1773
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-672560586
@Pearl1594 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-675109890
@vladimirpetrov a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-691032225
Packaging result: ✔centos7 ✔centos8 ✔debian. JID-1964
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-652769454
@Pearl1594 a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-650000348
@Pearl1594 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-657415814
The tests LGTM, however, will run the smoke tests once
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-674665036
@Pearl1594 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-647410117
@Pearl1594 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] DaanHoogland commented on a change in pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
DaanHoogland commented on a change in pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#discussion_r422041219
##########
File path: api/src/main/java/org/apache/cloudstack/api/ApiConstants.java
##########
@@ -777,6 +780,7 @@
public static final String EXITCODE = "exitcode";
public static final String TARGET_ID = "targetid";
public static final String FILES = "files";
+ public static final String FROM = "from";
Review comment:
how is this used in the API? I could imagine a name like srcStore or srcImage would be more appropriate.
##########
File path: api/src/main/java/com/cloud/storage/ImageStoreService.java
##########
@@ -0,0 +1,29 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package com.cloud.storage;
Review comment:
new classes should go into org.apache.cloudstack.*
##########
File path: api/src/main/java/org/apache/cloudstack/api/command/admin/storage/MigrateSecondaryStorageDataCmd.java
##########
@@ -0,0 +1,124 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.cloudstack.api.command.admin.storage;
+
+import java.util.List;
+
+import org.apache.cloudstack.acl.RoleType;
+import org.apache.cloudstack.api.APICommand;
+import org.apache.cloudstack.api.ApiConstants;
+import org.apache.cloudstack.api.BaseAsyncCmd;
+import org.apache.cloudstack.api.Parameter;
+import org.apache.cloudstack.api.ServerApiException;
+import org.apache.cloudstack.api.response.ImageStoreResponse;
+import org.apache.cloudstack.api.response.MigrationResponse;
+import org.apache.cloudstack.context.CallContext;
+import org.apache.log4j.Logger;
+
+import com.cloud.event.EventTypes;
+import com.cloud.exception.ConcurrentOperationException;
+import com.cloud.exception.InsufficientCapacityException;
+import com.cloud.exception.NetworkRuleConflictException;
+import com.cloud.exception.ResourceAllocationException;
+import com.cloud.exception.ResourceUnavailableException;
+import com.cloud.storage.ImageStore;
+import com.cloud.utils.StringUtils;
+
+@APICommand(name = MigrateSecondaryStorageDataCmd.APINAME,
+ description = "migrates data objects from one secondary storage to destination image store(s)",
+ responseObject = MigrationResponse.class,
+ entityType = {ImageStore.class},
+ requestHasSensitiveInfo = false,
+ responseHasSensitiveInfo = false,
+ since = "4.14.0",
+ authorized = {RoleType.Admin})
+public class MigrateSecondaryStorageDataCmd extends BaseAsyncCmd {
+
+ public static final Logger s_logger = Logger.getLogger(MigrateSecondaryStorageDataCmd.class.getName());
+
+ public static final String APINAME = "migrateSecondaryStorageData";
+
+ /////////////////////////////////////////////////////
+ //////////////// API parameters /////////////////////
+ /////////////////////////////////////////////////////
+
+ @Parameter(name = ApiConstants.FROM,
+ type = CommandType.UUID,
+ entityType = ImageStoreResponse.class,
+ description = "id of the image store from where the data is to be migrated",
+ required = true)
+ private Long id;
+
+ @Parameter(name = ApiConstants.MIGRATE_TO,
+ type = CommandType.LIST,
+ collectionType = CommandType.UUID,
+ entityType = ImageStoreResponse.class,
+ description = "id of the destination secondary storage pool to which the templates are to be migrated to",
+ required = true)
+ private List<Long> migrateTo;
Review comment:
I think these should be called `sourcePool` and `destinationPools`. In light of conventions `srcpool` and destpools` are best, I think.
##########
File path: api/src/main/java/org/apache/cloudstack/api/command/admin/storage/MigrateSecondaryStorageDataCmd.java
##########
@@ -0,0 +1,124 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.cloudstack.api.command.admin.storage;
+
+import java.util.List;
+
+import org.apache.cloudstack.acl.RoleType;
+import org.apache.cloudstack.api.APICommand;
+import org.apache.cloudstack.api.ApiConstants;
+import org.apache.cloudstack.api.BaseAsyncCmd;
+import org.apache.cloudstack.api.Parameter;
+import org.apache.cloudstack.api.ServerApiException;
+import org.apache.cloudstack.api.response.ImageStoreResponse;
+import org.apache.cloudstack.api.response.MigrationResponse;
+import org.apache.cloudstack.context.CallContext;
+import org.apache.log4j.Logger;
+
+import com.cloud.event.EventTypes;
+import com.cloud.exception.ConcurrentOperationException;
+import com.cloud.exception.InsufficientCapacityException;
+import com.cloud.exception.NetworkRuleConflictException;
+import com.cloud.exception.ResourceAllocationException;
+import com.cloud.exception.ResourceUnavailableException;
+import com.cloud.storage.ImageStore;
+import com.cloud.utils.StringUtils;
+
+@APICommand(name = MigrateSecondaryStorageDataCmd.APINAME,
+ description = "migrates data objects from one secondary storage to destination image store(s)",
+ responseObject = MigrationResponse.class,
+ entityType = {ImageStore.class},
+ requestHasSensitiveInfo = false,
+ responseHasSensitiveInfo = false,
+ since = "4.14.0",
+ authorized = {RoleType.Admin})
+public class MigrateSecondaryStorageDataCmd extends BaseAsyncCmd {
+
+ public static final Logger s_logger = Logger.getLogger(MigrateSecondaryStorageDataCmd.class.getName());
+
+ public static final String APINAME = "migrateSecondaryStorageData";
+
+ /////////////////////////////////////////////////////
+ //////////////// API parameters /////////////////////
+ /////////////////////////////////////////////////////
+
+ @Parameter(name = ApiConstants.FROM,
+ type = CommandType.UUID,
+ entityType = ImageStoreResponse.class,
+ description = "id of the image store from where the data is to be migrated",
+ required = true)
+ private Long id;
+
+ @Parameter(name = ApiConstants.MIGRATE_TO,
+ type = CommandType.LIST,
+ collectionType = CommandType.UUID,
+ entityType = ImageStoreResponse.class,
+ description = "id of the destination secondary storage pool to which the templates are to be migrated to",
+ required = true)
+ private List<Long> migrateTo;
+
+ @Parameter(name = ApiConstants.MIGRATION_TYPE,
+ type = CommandType.STRING,
+ description = "Balance: if you want data to be distributed evenly among the destination stores, " +
+ "Complete: If you want to migrate the entire data from source image store to the destination store(s)",
+ required = true)
Review comment:
why is this required, isn't `Complete` a reasonable default? making this required lays unnecessary burdon on the user.
##########
File path: api/src/main/java/org/apache/cloudstack/api/command/admin/storage/MigrateSecondaryStorageDataCmd.java
##########
@@ -0,0 +1,124 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.cloudstack.api.command.admin.storage;
+
+import java.util.List;
+
+import org.apache.cloudstack.acl.RoleType;
+import org.apache.cloudstack.api.APICommand;
+import org.apache.cloudstack.api.ApiConstants;
+import org.apache.cloudstack.api.BaseAsyncCmd;
+import org.apache.cloudstack.api.Parameter;
+import org.apache.cloudstack.api.ServerApiException;
+import org.apache.cloudstack.api.response.ImageStoreResponse;
+import org.apache.cloudstack.api.response.MigrationResponse;
+import org.apache.cloudstack.context.CallContext;
+import org.apache.log4j.Logger;
+
+import com.cloud.event.EventTypes;
+import com.cloud.exception.ConcurrentOperationException;
+import com.cloud.exception.InsufficientCapacityException;
+import com.cloud.exception.NetworkRuleConflictException;
+import com.cloud.exception.ResourceAllocationException;
+import com.cloud.exception.ResourceUnavailableException;
+import com.cloud.storage.ImageStore;
+import com.cloud.utils.StringUtils;
+
+@APICommand(name = MigrateSecondaryStorageDataCmd.APINAME,
+ description = "migrates data objects from one secondary storage to destination image store(s)",
+ responseObject = MigrationResponse.class,
+ entityType = {ImageStore.class},
+ requestHasSensitiveInfo = false,
+ responseHasSensitiveInfo = false,
+ since = "4.14.0",
+ authorized = {RoleType.Admin})
+public class MigrateSecondaryStorageDataCmd extends BaseAsyncCmd {
+
+ public static final Logger s_logger = Logger.getLogger(MigrateSecondaryStorageDataCmd.class.getName());
+
+ public static final String APINAME = "migrateSecondaryStorageData";
+
+ /////////////////////////////////////////////////////
+ //////////////// API parameters /////////////////////
+ /////////////////////////////////////////////////////
+
+ @Parameter(name = ApiConstants.FROM,
+ type = CommandType.UUID,
+ entityType = ImageStoreResponse.class,
+ description = "id of the image store from where the data is to be migrated",
+ required = true)
+ private Long id;
+
+ @Parameter(name = ApiConstants.MIGRATE_TO,
+ type = CommandType.LIST,
+ collectionType = CommandType.UUID,
+ entityType = ImageStoreResponse.class,
+ description = "id of the destination secondary storage pool to which the templates are to be migrated to",
Review comment:
`id(s) of the destination storage pool(s)` and remove the last `to` from the sentence
##########
File path: api/src/main/java/org/apache/cloudstack/api/command/admin/storage/UpdateImageStoreCmd.java
##########
@@ -0,0 +1,93 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.cloudstack.api.command.admin.storage;
+
+import org.apache.cloudstack.api.APICommand;
+import org.apache.cloudstack.api.ApiConstants;
+import org.apache.cloudstack.api.ApiErrorCode;
+import org.apache.cloudstack.api.BaseCmd;
+import org.apache.cloudstack.api.Parameter;
+import org.apache.cloudstack.api.ServerApiException;
+import org.apache.cloudstack.api.response.ImageStoreResponse;
+import org.apache.cloudstack.context.CallContext;
+import org.apache.log4j.Logger;
+
+import com.cloud.exception.ConcurrentOperationException;
+import com.cloud.exception.InsufficientCapacityException;
+import com.cloud.exception.NetworkRuleConflictException;
+import com.cloud.exception.ResourceAllocationException;
+import com.cloud.exception.ResourceUnavailableException;
+import com.cloud.storage.ImageStore;
+
+@APICommand(name = UpdateImageStoreCmd.APINAME, description = "Updates image store read-only status", responseObject = ImageStoreResponse.class, entityType = {ImageStore.class},
+ requestHasSensitiveInfo = false, responseHasSensitiveInfo = false)
+public class UpdateImageStoreCmd extends BaseCmd {
+ private static final Logger LOG = Logger.getLogger(UpdateImageStoreCmd.class.getName());
+ public static final String APINAME = "updateImageStore";
+
+ /////////////////////////////////////////////////////
+ //////////////// API parameters /////////////////////
+ /////////////////////////////////////////////////////
+ @Parameter(name = ApiConstants.ID, type = CommandType.UUID, entityType = ImageStoreResponse.class, required = true, description = "Image Store UUID")
+ private Long id;
+
+ @Parameter(name = ApiConstants.READ_ONLY, type = CommandType.BOOLEAN, required = true, description = "If set to true, it designates the corresponding image store to read-only")
Review comment:
can you expand on the usefullness of a store being read-only in this - or the API description? I.E. something like (in better English): "useful for deprecating image stores during storage migrations"
##########
File path: api/src/main/java/org/apache/cloudstack/api/command/admin/storage/MigrateSecondaryStorageDataCmd.java
##########
@@ -0,0 +1,124 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.cloudstack.api.command.admin.storage;
+
+import java.util.List;
+
+import org.apache.cloudstack.acl.RoleType;
+import org.apache.cloudstack.api.APICommand;
+import org.apache.cloudstack.api.ApiConstants;
+import org.apache.cloudstack.api.BaseAsyncCmd;
+import org.apache.cloudstack.api.Parameter;
+import org.apache.cloudstack.api.ServerApiException;
+import org.apache.cloudstack.api.response.ImageStoreResponse;
+import org.apache.cloudstack.api.response.MigrationResponse;
+import org.apache.cloudstack.context.CallContext;
+import org.apache.log4j.Logger;
+
+import com.cloud.event.EventTypes;
+import com.cloud.exception.ConcurrentOperationException;
+import com.cloud.exception.InsufficientCapacityException;
+import com.cloud.exception.NetworkRuleConflictException;
+import com.cloud.exception.ResourceAllocationException;
+import com.cloud.exception.ResourceUnavailableException;
+import com.cloud.storage.ImageStore;
+import com.cloud.utils.StringUtils;
+
+@APICommand(name = MigrateSecondaryStorageDataCmd.APINAME,
+ description = "migrates data objects from one secondary storage to destination image store(s)",
+ responseObject = MigrationResponse.class,
+ entityType = {ImageStore.class},
+ requestHasSensitiveInfo = false,
+ responseHasSensitiveInfo = false,
+ since = "4.14.0",
+ authorized = {RoleType.Admin})
+public class MigrateSecondaryStorageDataCmd extends BaseAsyncCmd {
+
+ public static final Logger s_logger = Logger.getLogger(MigrateSecondaryStorageDataCmd.class.getName());
+
+ public static final String APINAME = "migrateSecondaryStorageData";
+
+ /////////////////////////////////////////////////////
+ //////////////// API parameters /////////////////////
+ /////////////////////////////////////////////////////
+
+ @Parameter(name = ApiConstants.FROM,
+ type = CommandType.UUID,
+ entityType = ImageStoreResponse.class,
+ description = "id of the image store from where the data is to be migrated",
+ required = true)
+ private Long id;
+
+ @Parameter(name = ApiConstants.MIGRATE_TO,
+ type = CommandType.LIST,
+ collectionType = CommandType.UUID,
+ entityType = ImageStoreResponse.class,
+ description = "id of the destination secondary storage pool to which the templates are to be migrated to",
+ required = true)
+ private List<Long> migrateTo;
+
+ @Parameter(name = ApiConstants.MIGRATION_TYPE,
+ type = CommandType.STRING,
+ description = "Balance: if you want data to be distributed evenly among the destination stores, " +
+ "Complete: If you want to migrate the entire data from source image store to the destination store(s)",
+ required = true)
+ private String migrationType;
+
+ /////////////////////////////////////////////////////
+ /////////////////// Accessors ///////////////////////
+ /////////////////////////////////////////////////////
+
+ public Long getId() {
+ return id;
+ }
+
+ public List<Long> getMigrateTo() {
+ return migrateTo;
+ }
+
+ public String getMigrationType() {
+ return migrationType;
+ }
+
+ @Override
+ public String getEventType() {
+ return EventTypes.EVENT_FILE_MIGRATE;
+ }
+
+ @Override
+ public String getEventDescription() {
+ return "Attempting to migrate files/data objects " + "from : " + this.getId() + " to: " + StringUtils.join(getMigrateTo(), ",");
+ }
+
+ @Override
+ public void execute() throws ResourceUnavailableException, InsufficientCapacityException, ServerApiException, ConcurrentOperationException, ResourceAllocationException, NetworkRuleConflictException {
Review comment:
Why throw so many different exceptions? we can catch most of them and give nice consise error messages in a ServerApiException, right?
Most of these are Runtime exception.
Mentioning them here is also redundant as they are declared in BaseCmd as well.
##########
File path: api/src/main/java/org/apache/cloudstack/api/response/MigrationResponse.java
##########
@@ -0,0 +1,73 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.cloudstack.api.response;
+
+import org.apache.cloudstack.api.BaseResponse;
+import org.apache.cloudstack.api.EntityReference;
+
+import com.cloud.serializer.Param;
+import com.cloud.storage.ImageStore;
+import com.google.gson.annotations.SerializedName;
+
+@EntityReference(value = ImageStore.class)
+public class MigrationResponse extends BaseResponse {
+ @SerializedName("message")
+ @Param(description = "Response message")
Review comment:
A user might wonder what this means: "Where does the messsage come from and what will it signify for me?" Can you expand on that, please?
##########
File path: api/src/main/java/org/apache/cloudstack/api/command/admin/storage/UpdateImageStoreCmd.java
##########
@@ -0,0 +1,93 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.cloudstack.api.command.admin.storage;
+
+import org.apache.cloudstack.api.APICommand;
+import org.apache.cloudstack.api.ApiConstants;
+import org.apache.cloudstack.api.ApiErrorCode;
+import org.apache.cloudstack.api.BaseCmd;
+import org.apache.cloudstack.api.Parameter;
+import org.apache.cloudstack.api.ServerApiException;
+import org.apache.cloudstack.api.response.ImageStoreResponse;
+import org.apache.cloudstack.context.CallContext;
+import org.apache.log4j.Logger;
+
+import com.cloud.exception.ConcurrentOperationException;
+import com.cloud.exception.InsufficientCapacityException;
+import com.cloud.exception.NetworkRuleConflictException;
+import com.cloud.exception.ResourceAllocationException;
+import com.cloud.exception.ResourceUnavailableException;
+import com.cloud.storage.ImageStore;
+
+@APICommand(name = UpdateImageStoreCmd.APINAME, description = "Updates image store read-only status", responseObject = ImageStoreResponse.class, entityType = {ImageStore.class},
+ requestHasSensitiveInfo = false, responseHasSensitiveInfo = false)
+public class UpdateImageStoreCmd extends BaseCmd {
+ private static final Logger LOG = Logger.getLogger(UpdateImageStoreCmd.class.getName());
+ public static final String APINAME = "updateImageStore";
+
+ /////////////////////////////////////////////////////
+ //////////////// API parameters /////////////////////
+ /////////////////////////////////////////////////////
+ @Parameter(name = ApiConstants.ID, type = CommandType.UUID, entityType = ImageStoreResponse.class, required = true, description = "Image Store UUID")
+ private Long id;
+
+ @Parameter(name = ApiConstants.READ_ONLY, type = CommandType.BOOLEAN, required = true, description = "If set to true, it designates the corresponding image store to read-only")
+ private Boolean readonly;
+
+ /////////////////////////////////////////////////////
+ /////////////////// Accessors ///////////////////////
+ /////////////////////////////////////////////////////
+
+ public Long getId() {
+ return id;
+ }
+
+ public Boolean getReadonly() {
+ return readonly;
+ }
+
+ /////////////////////////////////////////////////////
+ /////////////// API Implementation///////////////////
+ /////////////////////////////////////////////////////
+
+ @Override
+ public void execute() throws ResourceUnavailableException, InsufficientCapacityException, ServerApiException, ConcurrentOperationException, ResourceAllocationException, NetworkRuleConflictException {
Review comment:
same as above in `MigrateSecondaryStorageDataCmd`
##########
File path: engine/schema/src/main/java/com/cloud/secstorage/CommandExecLogDaoImpl.java
##########
@@ -43,4 +48,15 @@ public void expungeExpiredRecords(Date cutTime) {
sc.setParameters("created", cutTime);
expunge(sc);
}
+
+ @Override
+ public Integer getCopyCmdCountForSSVM(Long id) {
+ SearchCriteria<CommandExecLogVO> sc = CommandSearch.create();
+ sc.setParameters("host_id", id);
+ sc.setParameters("command_name", "CopyCommand");
+ List<CommandExecLogVO> copyCmds = customSearch(sc, null);
+ return copyCmds.size();
+ }
+
+
Review comment:
?
##########
File path: engine/orchestration/src/main/java/org/apache/cloudstack/engine/orchestration/VolumeOrchestrator.java
##########
@@ -141,7 +141,6 @@
import com.cloud.vm.dao.UserVmDao;
public class VolumeOrchestrator extends ManagerBase implements VolumeOrchestrationService, Configurable {
-
Review comment:
makes sense but will lead possibly to unecessary conflicts during rebases, which are inevitable for this PR
##########
File path: engine/schema/src/main/java/org/apache/cloudstack/storage/datastore/db/ImageStoreDaoImpl.java
##########
@@ -91,6 +91,20 @@ public ImageStoreVO findByName(String name) {
return listBy(sc);
}
+ @Override
+ public List<ImageStoreVO> findByScopeExcludingReadOnly(ZoneScope scope) {
+ SearchCriteria<ImageStoreVO> sc = createSearchCriteria();
+ sc.addAnd("role", SearchCriteria.Op.EQ, DataStoreRole.Image);
+ if (scope.getScopeId() != null) {
+ SearchCriteria<ImageStoreVO> scc = createSearchCriteria();
+ scc.addOr("scope", SearchCriteria.Op.EQ, ScopeType.REGION);
+ scc.addOr("dcId", SearchCriteria.Op.EQ, scope.getScopeId());
+ sc.addAnd("scope", SearchCriteria.Op.SC, scc);
+ sc.addAnd("readonly", SearchCriteria.Op.EQ, Boolean.FALSE);
Review comment:
so this line could be surrounded by `if (readonlyExcludedFlag == true) {}`
##########
File path: engine/schema/src/main/java/org/apache/cloudstack/storage/datastore/db/ImageStoreDao.java
##########
@@ -31,6 +31,8 @@
List<ImageStoreVO> findByScope(ZoneScope scope);
+ List<ImageStoreVO> findByScopeExcludingReadOnly(ZoneScope scope);
Review comment:
this method name is confusing: `findByScope` but than the scope is allready set to `ZoneScope`? Wouldn't `findByZone` be better?
Also the method above could probably just be extended with a flag for `excludeReadOnly`.
##########
File path: engine/orchestration/src/main/java/org/apache/cloudstack/engine/orchestration/StorageOrchestrator.java
##########
@@ -0,0 +1,629 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.cloudstack.engine.orchestration;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.Hashtable;
+import java.util.LinkedHashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+import javax.inject.Inject;
+import javax.naming.ConfigurationException;
+
+import org.apache.cloudstack.api.response.MigrationResponse;
+import org.apache.cloudstack.engine.orchestration.service.StorageOrchestrationService;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataObject;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStoreManager;
+import org.apache.cloudstack.engine.subsystem.api.storage.ObjectInDataStoreStateMachine;
+import org.apache.cloudstack.engine.subsystem.api.storage.SecondaryStorageService;
+import org.apache.cloudstack.engine.subsystem.api.storage.SecondaryStorageService.DataObjectResult;
+import org.apache.cloudstack.engine.subsystem.api.storage.SnapshotDataFactory;
+import org.apache.cloudstack.engine.subsystem.api.storage.SnapshotInfo;
+import org.apache.cloudstack.engine.subsystem.api.storage.TemplateDataFactory;
+import org.apache.cloudstack.engine.subsystem.api.storage.VolumeDataFactory;
+import org.apache.cloudstack.framework.async.AsyncCallFuture;
+import org.apache.cloudstack.framework.config.ConfigKey;
+import org.apache.cloudstack.framework.config.Configurable;
+import org.apache.cloudstack.framework.config.dao.ConfigurationDao;
+import org.apache.cloudstack.framework.jobs.AsyncJobManager;
+import org.apache.cloudstack.storage.datastore.db.ImageStoreDao;
+import org.apache.cloudstack.storage.datastore.db.SnapshotDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.SnapshotDataStoreVO;
+import org.apache.cloudstack.storage.datastore.db.TemplateDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.TemplateDataStoreVO;
+import org.apache.cloudstack.storage.datastore.db.VolumeDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.VolumeDataStoreVO;
+import org.apache.commons.math3.stat.descriptive.moment.Mean;
+import org.apache.commons.math3.stat.descriptive.moment.StandardDeviation;
+import org.apache.log4j.Logger;
+
+import com.cloud.configuration.Config;
+import com.cloud.host.HostVO;
+import com.cloud.host.Status;
+import com.cloud.host.dao.HostDao;
+import com.cloud.server.StatsCollector;
+import com.cloud.storage.DataStoreRole;
+import com.cloud.storage.ImageStoreService.MigrationPolicy;
+import com.cloud.storage.SnapshotVO;
+import com.cloud.storage.StorageService;
+import com.cloud.storage.StorageStats;
+import com.cloud.storage.VMTemplateVO;
+import com.cloud.storage.dao.SnapshotDao;
+import com.cloud.storage.dao.VMTemplateDao;
+import com.cloud.utils.NumbersUtil;
+import com.cloud.utils.Pair;
+import com.cloud.utils.StringUtils;
+import com.cloud.utils.component.ManagerBase;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.vm.SecondaryStorageVm;
+import com.cloud.vm.SecondaryStorageVmVO;
+import com.cloud.vm.VirtualMachine;
+import com.cloud.vm.dao.SecondaryStorageVmDao;
+
+public class StorageOrchestrator extends ManagerBase implements StorageOrchestrationService, Configurable {
+
+ private static final Logger s_logger = Logger.getLogger(StorageOrchestrator.class);
+ @Inject
+ TemplateDataStoreDao templateDataStoreDao;
+ @Inject
+ SnapshotDataStoreDao snapshotDataStoreDao;
+ @Inject
+ VolumeDataStoreDao volumeDataStoreDao;
+ @Inject
+ VolumeDataFactory volumeFactory;
+ @Inject
+ VMTemplateDao templateDao;
+ @Inject
+ TemplateDataFactory templateFactory;
+ @Inject
+ SnapshotDao snapshotDao;
+ @Inject
+ SnapshotDataFactory snapshotFactory;
+ @Inject
+ DataStoreManager dataStoreManager;
+ @Inject
+ ImageStoreDao imageStoreDao;
+ @Inject
+ StatsCollector statsCollector;
+ @Inject
+ public StorageService storageService;
+ @Inject
+ SecondaryStorageVmDao secStorageVmDao;
+ @Inject
+ ConfigurationDao configDao;
+ @Inject
+ HostDao hostDao;
+ @Inject
+ private AsyncJobManager jobMgr;
+ @Inject
+ private SecondaryStorageService secStgSrv;
+
+ ConfigKey<Double> ImageStoreImbalanceThreshold = new ConfigKey<>("Advanced", Double.class,
+ "image.store.imbalance.threshold",
+ "0.1",
+ "The storage imbalance threshold that is compared with the standard deviation percentage for a storage utilization metric. " +
+ "The value is a percentage in decimal format.",
+ true, ConfigKey.Scope.Global);
+
+ Integer numConcurrentCopyTasksPerSSVM = 2;
+
+ private double imageStoreCapacityThreshold = 0.90;
+
+ @Override
+ public String getConfigComponentName() {
+ return StorageOrchestrationService.class.getName();
+ }
+
+ @Override
+ public ConfigKey<?>[] getConfigKeys() {
+ return new ConfigKey<?>[]{ImageStoreImbalanceThreshold};
+ }
+
+ static class MigrateBlockingQueue<T> extends ArrayBlockingQueue<T> {
+
+ MigrateBlockingQueue(int size) {
+ super(size);
+ }
+
+ public boolean offer(T task) {
+ try {
+ this.put(task);
+ } catch (InterruptedException e) {
+ throw new RuntimeException(e);
+ }
+ return true;
+ }
+ }
+
+ @Override
+ public boolean configure(String name, Map<String, Object> params) throws ConfigurationException {
+ numConcurrentCopyTasksPerSSVM = NumbersUtil.parseInt(configDao.getValue(Config.SecStorageMaxMigrateSessions.key()), 2);
+ return true;
+ }
+
+ @Override
+ public MigrationResponse migrateData(Long srcDataStoreId, List<Long> destDatastores, MigrationPolicy migrationPolicy) {
Review comment:
this method is way too long. please modularise it into byte size chunks and factor it out to a worker - or utility class?
I can not set a rule but in my opinion any method larger than 20 line or any class larger than 300 lines is probably not separating concerns.
every comment in this method should probably be converted to a good method name and extracted, and so should many top level blocks (if-, for- and while statements
##########
File path: engine/orchestration/src/main/java/org/apache/cloudstack/engine/orchestration/StorageOrchestrator.java
##########
@@ -0,0 +1,629 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.cloudstack.engine.orchestration;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.Hashtable;
+import java.util.LinkedHashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+import javax.inject.Inject;
+import javax.naming.ConfigurationException;
+
+import org.apache.cloudstack.api.response.MigrationResponse;
+import org.apache.cloudstack.engine.orchestration.service.StorageOrchestrationService;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataObject;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStoreManager;
+import org.apache.cloudstack.engine.subsystem.api.storage.ObjectInDataStoreStateMachine;
+import org.apache.cloudstack.engine.subsystem.api.storage.SecondaryStorageService;
+import org.apache.cloudstack.engine.subsystem.api.storage.SecondaryStorageService.DataObjectResult;
+import org.apache.cloudstack.engine.subsystem.api.storage.SnapshotDataFactory;
+import org.apache.cloudstack.engine.subsystem.api.storage.SnapshotInfo;
+import org.apache.cloudstack.engine.subsystem.api.storage.TemplateDataFactory;
+import org.apache.cloudstack.engine.subsystem.api.storage.VolumeDataFactory;
+import org.apache.cloudstack.framework.async.AsyncCallFuture;
+import org.apache.cloudstack.framework.config.ConfigKey;
+import org.apache.cloudstack.framework.config.Configurable;
+import org.apache.cloudstack.framework.config.dao.ConfigurationDao;
+import org.apache.cloudstack.framework.jobs.AsyncJobManager;
+import org.apache.cloudstack.storage.datastore.db.ImageStoreDao;
+import org.apache.cloudstack.storage.datastore.db.SnapshotDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.SnapshotDataStoreVO;
+import org.apache.cloudstack.storage.datastore.db.TemplateDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.TemplateDataStoreVO;
+import org.apache.cloudstack.storage.datastore.db.VolumeDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.VolumeDataStoreVO;
+import org.apache.commons.math3.stat.descriptive.moment.Mean;
+import org.apache.commons.math3.stat.descriptive.moment.StandardDeviation;
+import org.apache.log4j.Logger;
+
+import com.cloud.configuration.Config;
+import com.cloud.host.HostVO;
+import com.cloud.host.Status;
+import com.cloud.host.dao.HostDao;
+import com.cloud.server.StatsCollector;
+import com.cloud.storage.DataStoreRole;
+import com.cloud.storage.ImageStoreService.MigrationPolicy;
+import com.cloud.storage.SnapshotVO;
+import com.cloud.storage.StorageService;
+import com.cloud.storage.StorageStats;
+import com.cloud.storage.VMTemplateVO;
+import com.cloud.storage.dao.SnapshotDao;
+import com.cloud.storage.dao.VMTemplateDao;
+import com.cloud.utils.NumbersUtil;
+import com.cloud.utils.Pair;
+import com.cloud.utils.StringUtils;
+import com.cloud.utils.component.ManagerBase;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.vm.SecondaryStorageVm;
+import com.cloud.vm.SecondaryStorageVmVO;
+import com.cloud.vm.VirtualMachine;
+import com.cloud.vm.dao.SecondaryStorageVmDao;
+
+public class StorageOrchestrator extends ManagerBase implements StorageOrchestrationService, Configurable {
+
+ private static final Logger s_logger = Logger.getLogger(StorageOrchestrator.class);
+ @Inject
+ TemplateDataStoreDao templateDataStoreDao;
+ @Inject
+ SnapshotDataStoreDao snapshotDataStoreDao;
+ @Inject
+ VolumeDataStoreDao volumeDataStoreDao;
+ @Inject
+ VolumeDataFactory volumeFactory;
+ @Inject
+ VMTemplateDao templateDao;
+ @Inject
+ TemplateDataFactory templateFactory;
+ @Inject
+ SnapshotDao snapshotDao;
+ @Inject
+ SnapshotDataFactory snapshotFactory;
+ @Inject
+ DataStoreManager dataStoreManager;
+ @Inject
+ ImageStoreDao imageStoreDao;
+ @Inject
+ StatsCollector statsCollector;
+ @Inject
+ public StorageService storageService;
+ @Inject
+ SecondaryStorageVmDao secStorageVmDao;
+ @Inject
+ ConfigurationDao configDao;
+ @Inject
+ HostDao hostDao;
+ @Inject
+ private AsyncJobManager jobMgr;
+ @Inject
+ private SecondaryStorageService secStgSrv;
+
+ ConfigKey<Double> ImageStoreImbalanceThreshold = new ConfigKey<>("Advanced", Double.class,
+ "image.store.imbalance.threshold",
+ "0.1",
+ "The storage imbalance threshold that is compared with the standard deviation percentage for a storage utilization metric. " +
+ "The value is a percentage in decimal format.",
+ true, ConfigKey.Scope.Global);
+
+ Integer numConcurrentCopyTasksPerSSVM = 2;
+
+ private double imageStoreCapacityThreshold = 0.90;
+
+ @Override
+ public String getConfigComponentName() {
+ return StorageOrchestrationService.class.getName();
+ }
+
+ @Override
+ public ConfigKey<?>[] getConfigKeys() {
+ return new ConfigKey<?>[]{ImageStoreImbalanceThreshold};
+ }
+
+ static class MigrateBlockingQueue<T> extends ArrayBlockingQueue<T> {
+
+ MigrateBlockingQueue(int size) {
+ super(size);
+ }
+
+ public boolean offer(T task) {
+ try {
+ this.put(task);
+ } catch (InterruptedException e) {
+ throw new RuntimeException(e);
+ }
+ return true;
+ }
+ }
+
+ @Override
+ public boolean configure(String name, Map<String, Object> params) throws ConfigurationException {
+ numConcurrentCopyTasksPerSSVM = NumbersUtil.parseInt(configDao.getValue(Config.SecStorageMaxMigrateSessions.key()), 2);
+ return true;
+ }
+
+ @Override
+ public MigrationResponse migrateData(Long srcDataStoreId, List<Long> destDatastores, MigrationPolicy migrationPolicy) {
+ List<DataObject> files = new LinkedList<>();
+ int successCount = 0;
+ boolean success = true;
+ String message = null;
+
+ if (migrationPolicy == MigrationPolicy.COMPLETE) {
+ if (!filesReady(srcDataStoreId)) {
+ throw new CloudRuntimeException("Complete migration failed as there are data objects which are not Ready");
+ }
+ }
+
+ DataStore srcDatastore = dataStoreManager.getDataStore(srcDataStoreId, DataStoreRole.Image);
+ Map<DataObject, Pair<List<SnapshotInfo>, Long>> snapshotChains = new HashMap<>();
+ files.addAll(getAllValidTemplates(srcDatastore));
+ files.addAll(getAllValidSnapshotChains(srcDatastore, snapshotChains));
+ files.addAll(getAllValidVolumes(srcDatastore));
Review comment:
or: `prepareSourcesList(...)`
##########
File path: engine/orchestration/src/main/java/org/apache/cloudstack/engine/orchestration/StorageOrchestrator.java
##########
@@ -0,0 +1,629 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.cloudstack.engine.orchestration;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.Hashtable;
+import java.util.LinkedHashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+import javax.inject.Inject;
+import javax.naming.ConfigurationException;
+
+import org.apache.cloudstack.api.response.MigrationResponse;
+import org.apache.cloudstack.engine.orchestration.service.StorageOrchestrationService;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataObject;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStoreManager;
+import org.apache.cloudstack.engine.subsystem.api.storage.ObjectInDataStoreStateMachine;
+import org.apache.cloudstack.engine.subsystem.api.storage.SecondaryStorageService;
+import org.apache.cloudstack.engine.subsystem.api.storage.SecondaryStorageService.DataObjectResult;
+import org.apache.cloudstack.engine.subsystem.api.storage.SnapshotDataFactory;
+import org.apache.cloudstack.engine.subsystem.api.storage.SnapshotInfo;
+import org.apache.cloudstack.engine.subsystem.api.storage.TemplateDataFactory;
+import org.apache.cloudstack.engine.subsystem.api.storage.VolumeDataFactory;
+import org.apache.cloudstack.framework.async.AsyncCallFuture;
+import org.apache.cloudstack.framework.config.ConfigKey;
+import org.apache.cloudstack.framework.config.Configurable;
+import org.apache.cloudstack.framework.config.dao.ConfigurationDao;
+import org.apache.cloudstack.framework.jobs.AsyncJobManager;
+import org.apache.cloudstack.storage.datastore.db.ImageStoreDao;
+import org.apache.cloudstack.storage.datastore.db.SnapshotDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.SnapshotDataStoreVO;
+import org.apache.cloudstack.storage.datastore.db.TemplateDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.TemplateDataStoreVO;
+import org.apache.cloudstack.storage.datastore.db.VolumeDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.VolumeDataStoreVO;
+import org.apache.commons.math3.stat.descriptive.moment.Mean;
+import org.apache.commons.math3.stat.descriptive.moment.StandardDeviation;
+import org.apache.log4j.Logger;
+
+import com.cloud.configuration.Config;
+import com.cloud.host.HostVO;
+import com.cloud.host.Status;
+import com.cloud.host.dao.HostDao;
+import com.cloud.server.StatsCollector;
+import com.cloud.storage.DataStoreRole;
+import com.cloud.storage.ImageStoreService.MigrationPolicy;
+import com.cloud.storage.SnapshotVO;
+import com.cloud.storage.StorageService;
+import com.cloud.storage.StorageStats;
+import com.cloud.storage.VMTemplateVO;
+import com.cloud.storage.dao.SnapshotDao;
+import com.cloud.storage.dao.VMTemplateDao;
+import com.cloud.utils.NumbersUtil;
+import com.cloud.utils.Pair;
+import com.cloud.utils.StringUtils;
+import com.cloud.utils.component.ManagerBase;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.vm.SecondaryStorageVm;
+import com.cloud.vm.SecondaryStorageVmVO;
+import com.cloud.vm.VirtualMachine;
+import com.cloud.vm.dao.SecondaryStorageVmDao;
+
+public class StorageOrchestrator extends ManagerBase implements StorageOrchestrationService, Configurable {
+
+ private static final Logger s_logger = Logger.getLogger(StorageOrchestrator.class);
+ @Inject
+ TemplateDataStoreDao templateDataStoreDao;
+ @Inject
+ SnapshotDataStoreDao snapshotDataStoreDao;
+ @Inject
+ VolumeDataStoreDao volumeDataStoreDao;
+ @Inject
+ VolumeDataFactory volumeFactory;
+ @Inject
+ VMTemplateDao templateDao;
+ @Inject
+ TemplateDataFactory templateFactory;
+ @Inject
+ SnapshotDao snapshotDao;
+ @Inject
+ SnapshotDataFactory snapshotFactory;
+ @Inject
+ DataStoreManager dataStoreManager;
+ @Inject
+ ImageStoreDao imageStoreDao;
+ @Inject
+ StatsCollector statsCollector;
+ @Inject
+ public StorageService storageService;
+ @Inject
+ SecondaryStorageVmDao secStorageVmDao;
+ @Inject
+ ConfigurationDao configDao;
+ @Inject
+ HostDao hostDao;
+ @Inject
+ private AsyncJobManager jobMgr;
+ @Inject
+ private SecondaryStorageService secStgSrv;
+
+ ConfigKey<Double> ImageStoreImbalanceThreshold = new ConfigKey<>("Advanced", Double.class,
+ "image.store.imbalance.threshold",
+ "0.1",
+ "The storage imbalance threshold that is compared with the standard deviation percentage for a storage utilization metric. " +
+ "The value is a percentage in decimal format.",
+ true, ConfigKey.Scope.Global);
+
+ Integer numConcurrentCopyTasksPerSSVM = 2;
+
+ private double imageStoreCapacityThreshold = 0.90;
+
+ @Override
+ public String getConfigComponentName() {
+ return StorageOrchestrationService.class.getName();
+ }
+
+ @Override
+ public ConfigKey<?>[] getConfigKeys() {
+ return new ConfigKey<?>[]{ImageStoreImbalanceThreshold};
+ }
+
+ static class MigrateBlockingQueue<T> extends ArrayBlockingQueue<T> {
+
+ MigrateBlockingQueue(int size) {
+ super(size);
+ }
+
+ public boolean offer(T task) {
+ try {
+ this.put(task);
+ } catch (InterruptedException e) {
+ throw new RuntimeException(e);
+ }
+ return true;
+ }
+ }
+
+ @Override
+ public boolean configure(String name, Map<String, Object> params) throws ConfigurationException {
+ numConcurrentCopyTasksPerSSVM = NumbersUtil.parseInt(configDao.getValue(Config.SecStorageMaxMigrateSessions.key()), 2);
+ return true;
+ }
+
+ @Override
+ public MigrationResponse migrateData(Long srcDataStoreId, List<Long> destDatastores, MigrationPolicy migrationPolicy) {
+ List<DataObject> files = new LinkedList<>();
+ int successCount = 0;
+ boolean success = true;
+ String message = null;
+
+ if (migrationPolicy == MigrationPolicy.COMPLETE) {
+ if (!filesReady(srcDataStoreId)) {
+ throw new CloudRuntimeException("Complete migration failed as there are data objects which are not Ready");
+ }
+ }
Review comment:
example extraction: above code could go in `checkCompleteMigrationFor ReadyVolumesOnly(...)`
##########
File path: engine/storage/datamotion/src/main/java/org/apache/cloudstack/storage/motion/DataMotionServiceImpl.java
##########
@@ -39,9 +33,14 @@
import org.apache.cloudstack.engine.subsystem.api.storage.StorageStrategyFactory;
import org.apache.cloudstack.engine.subsystem.api.storage.VolumeInfo;
import org.apache.cloudstack.framework.async.AsyncCompletionCallback;
+import org.apache.log4j.Logger;
+import org.springframework.stereotype.Component;
import com.cloud.agent.api.to.VirtualMachineTO;
import com.cloud.host.Host;
+import com.cloud.storage.Volume;
+import com.cloud.storage.VolumeVO;
+import com.cloud.storage.dao.VolumeDao;
Review comment:
unecessary conlict potential
##########
File path: engine/orchestration/src/main/java/org/apache/cloudstack/engine/orchestration/StorageOrchestrator.java
##########
@@ -0,0 +1,629 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.cloudstack.engine.orchestration;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.Hashtable;
+import java.util.LinkedHashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+import javax.inject.Inject;
+import javax.naming.ConfigurationException;
+
+import org.apache.cloudstack.api.response.MigrationResponse;
+import org.apache.cloudstack.engine.orchestration.service.StorageOrchestrationService;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataObject;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStoreManager;
+import org.apache.cloudstack.engine.subsystem.api.storage.ObjectInDataStoreStateMachine;
+import org.apache.cloudstack.engine.subsystem.api.storage.SecondaryStorageService;
+import org.apache.cloudstack.engine.subsystem.api.storage.SecondaryStorageService.DataObjectResult;
+import org.apache.cloudstack.engine.subsystem.api.storage.SnapshotDataFactory;
+import org.apache.cloudstack.engine.subsystem.api.storage.SnapshotInfo;
+import org.apache.cloudstack.engine.subsystem.api.storage.TemplateDataFactory;
+import org.apache.cloudstack.engine.subsystem.api.storage.VolumeDataFactory;
+import org.apache.cloudstack.framework.async.AsyncCallFuture;
+import org.apache.cloudstack.framework.config.ConfigKey;
+import org.apache.cloudstack.framework.config.Configurable;
+import org.apache.cloudstack.framework.config.dao.ConfigurationDao;
+import org.apache.cloudstack.framework.jobs.AsyncJobManager;
+import org.apache.cloudstack.storage.datastore.db.ImageStoreDao;
+import org.apache.cloudstack.storage.datastore.db.SnapshotDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.SnapshotDataStoreVO;
+import org.apache.cloudstack.storage.datastore.db.TemplateDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.TemplateDataStoreVO;
+import org.apache.cloudstack.storage.datastore.db.VolumeDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.VolumeDataStoreVO;
+import org.apache.commons.math3.stat.descriptive.moment.Mean;
+import org.apache.commons.math3.stat.descriptive.moment.StandardDeviation;
+import org.apache.log4j.Logger;
+
+import com.cloud.configuration.Config;
+import com.cloud.host.HostVO;
+import com.cloud.host.Status;
+import com.cloud.host.dao.HostDao;
+import com.cloud.server.StatsCollector;
+import com.cloud.storage.DataStoreRole;
+import com.cloud.storage.ImageStoreService.MigrationPolicy;
+import com.cloud.storage.SnapshotVO;
+import com.cloud.storage.StorageService;
+import com.cloud.storage.StorageStats;
+import com.cloud.storage.VMTemplateVO;
+import com.cloud.storage.dao.SnapshotDao;
+import com.cloud.storage.dao.VMTemplateDao;
+import com.cloud.utils.NumbersUtil;
+import com.cloud.utils.Pair;
+import com.cloud.utils.StringUtils;
+import com.cloud.utils.component.ManagerBase;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.vm.SecondaryStorageVm;
+import com.cloud.vm.SecondaryStorageVmVO;
+import com.cloud.vm.VirtualMachine;
+import com.cloud.vm.dao.SecondaryStorageVmDao;
+
+public class StorageOrchestrator extends ManagerBase implements StorageOrchestrationService, Configurable {
+
+ private static final Logger s_logger = Logger.getLogger(StorageOrchestrator.class);
+ @Inject
+ TemplateDataStoreDao templateDataStoreDao;
+ @Inject
+ SnapshotDataStoreDao snapshotDataStoreDao;
+ @Inject
+ VolumeDataStoreDao volumeDataStoreDao;
+ @Inject
+ VolumeDataFactory volumeFactory;
+ @Inject
+ VMTemplateDao templateDao;
+ @Inject
+ TemplateDataFactory templateFactory;
+ @Inject
+ SnapshotDao snapshotDao;
+ @Inject
+ SnapshotDataFactory snapshotFactory;
+ @Inject
+ DataStoreManager dataStoreManager;
+ @Inject
+ ImageStoreDao imageStoreDao;
+ @Inject
+ StatsCollector statsCollector;
+ @Inject
+ public StorageService storageService;
+ @Inject
+ SecondaryStorageVmDao secStorageVmDao;
+ @Inject
+ ConfigurationDao configDao;
+ @Inject
+ HostDao hostDao;
+ @Inject
+ private AsyncJobManager jobMgr;
+ @Inject
+ private SecondaryStorageService secStgSrv;
+
+ ConfigKey<Double> ImageStoreImbalanceThreshold = new ConfigKey<>("Advanced", Double.class,
+ "image.store.imbalance.threshold",
+ "0.1",
+ "The storage imbalance threshold that is compared with the standard deviation percentage for a storage utilization metric. " +
+ "The value is a percentage in decimal format.",
+ true, ConfigKey.Scope.Global);
+
+ Integer numConcurrentCopyTasksPerSSVM = 2;
+
+ private double imageStoreCapacityThreshold = 0.90;
+
+ @Override
+ public String getConfigComponentName() {
+ return StorageOrchestrationService.class.getName();
+ }
+
+ @Override
+ public ConfigKey<?>[] getConfigKeys() {
+ return new ConfigKey<?>[]{ImageStoreImbalanceThreshold};
+ }
+
+ static class MigrateBlockingQueue<T> extends ArrayBlockingQueue<T> {
+
+ MigrateBlockingQueue(int size) {
+ super(size);
+ }
+
+ public boolean offer(T task) {
+ try {
+ this.put(task);
+ } catch (InterruptedException e) {
+ throw new RuntimeException(e);
+ }
+ return true;
+ }
+ }
+
+ @Override
+ public boolean configure(String name, Map<String, Object> params) throws ConfigurationException {
+ numConcurrentCopyTasksPerSSVM = NumbersUtil.parseInt(configDao.getValue(Config.SecStorageMaxMigrateSessions.key()), 2);
+ return true;
+ }
+
+ @Override
+ public MigrationResponse migrateData(Long srcDataStoreId, List<Long> destDatastores, MigrationPolicy migrationPolicy) {
+ List<DataObject> files = new LinkedList<>();
+ int successCount = 0;
+ boolean success = true;
+ String message = null;
+
+ if (migrationPolicy == MigrationPolicy.COMPLETE) {
+ if (!filesReady(srcDataStoreId)) {
+ throw new CloudRuntimeException("Complete migration failed as there are data objects which are not Ready");
+ }
+ }
+
+ DataStore srcDatastore = dataStoreManager.getDataStore(srcDataStoreId, DataStoreRole.Image);
+ Map<DataObject, Pair<List<SnapshotInfo>, Long>> snapshotChains = new HashMap<>();
+ files.addAll(getAllValidTemplates(srcDatastore));
+ files.addAll(getAllValidSnapshotChains(srcDatastore, snapshotChains));
+ files.addAll(getAllValidVolumes(srcDatastore));
+
+ Collections.sort(files, new Comparator<DataObject>() {
+ @Override
+ public int compare(DataObject o1, DataObject o2) {
+ Long size1 = o1.getSize();
+ Long size2 = o2.getSize();
+ if (o1 instanceof SnapshotInfo) {
+ size1 = snapshotChains.get(o1).second();
+ }
+ if (o2 instanceof SnapshotInfo) {
+ size2 = snapshotChains.get(o2).second();
+ }
+ //return o2.getSize() > o1.getSize() ? 1 : -1;
+ return size2 > size1 ? 1 : -1;
+ }
+ });
Review comment:
`sortSources(...), should probably be called from `prepareSourcesList()`
etc.
etc.
hope you get my drift. The code looks good but this is about maintainability. happy to discuss.
##########
File path: engine/api/src/main/java/org/apache/cloudstack/engine/subsystem/api/storage/ObjectInDataStoreStateMachine.java
##########
@@ -49,8 +50,12 @@ public String getDescription() {
DestroyRequested,
OperationSuccessed,
OperationFailed,
+ // Added as volume converts migrationrequested to copyrequested - VolumeObject.java
Review comment:
I don't understand this comment, should it still be here or was it only there as development aid?
##########
File path: engine/orchestration/src/main/java/org/apache/cloudstack/engine/orchestration/StorageOrchestrator.java
##########
@@ -0,0 +1,629 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.cloudstack.engine.orchestration;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.Hashtable;
+import java.util.LinkedHashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+import javax.inject.Inject;
+import javax.naming.ConfigurationException;
+
+import org.apache.cloudstack.api.response.MigrationResponse;
+import org.apache.cloudstack.engine.orchestration.service.StorageOrchestrationService;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataObject;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStoreManager;
+import org.apache.cloudstack.engine.subsystem.api.storage.ObjectInDataStoreStateMachine;
+import org.apache.cloudstack.engine.subsystem.api.storage.SecondaryStorageService;
+import org.apache.cloudstack.engine.subsystem.api.storage.SecondaryStorageService.DataObjectResult;
+import org.apache.cloudstack.engine.subsystem.api.storage.SnapshotDataFactory;
+import org.apache.cloudstack.engine.subsystem.api.storage.SnapshotInfo;
+import org.apache.cloudstack.engine.subsystem.api.storage.TemplateDataFactory;
+import org.apache.cloudstack.engine.subsystem.api.storage.VolumeDataFactory;
+import org.apache.cloudstack.framework.async.AsyncCallFuture;
+import org.apache.cloudstack.framework.config.ConfigKey;
+import org.apache.cloudstack.framework.config.Configurable;
+import org.apache.cloudstack.framework.config.dao.ConfigurationDao;
+import org.apache.cloudstack.framework.jobs.AsyncJobManager;
+import org.apache.cloudstack.storage.datastore.db.ImageStoreDao;
+import org.apache.cloudstack.storage.datastore.db.SnapshotDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.SnapshotDataStoreVO;
+import org.apache.cloudstack.storage.datastore.db.TemplateDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.TemplateDataStoreVO;
+import org.apache.cloudstack.storage.datastore.db.VolumeDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.VolumeDataStoreVO;
+import org.apache.commons.math3.stat.descriptive.moment.Mean;
+import org.apache.commons.math3.stat.descriptive.moment.StandardDeviation;
+import org.apache.log4j.Logger;
+
+import com.cloud.configuration.Config;
+import com.cloud.host.HostVO;
+import com.cloud.host.Status;
+import com.cloud.host.dao.HostDao;
+import com.cloud.server.StatsCollector;
+import com.cloud.storage.DataStoreRole;
+import com.cloud.storage.ImageStoreService.MigrationPolicy;
+import com.cloud.storage.SnapshotVO;
+import com.cloud.storage.StorageService;
+import com.cloud.storage.StorageStats;
+import com.cloud.storage.VMTemplateVO;
+import com.cloud.storage.dao.SnapshotDao;
+import com.cloud.storage.dao.VMTemplateDao;
+import com.cloud.utils.NumbersUtil;
+import com.cloud.utils.Pair;
+import com.cloud.utils.StringUtils;
+import com.cloud.utils.component.ManagerBase;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.vm.SecondaryStorageVm;
+import com.cloud.vm.SecondaryStorageVmVO;
+import com.cloud.vm.VirtualMachine;
+import com.cloud.vm.dao.SecondaryStorageVmDao;
+
+public class StorageOrchestrator extends ManagerBase implements StorageOrchestrationService, Configurable {
+
+ private static final Logger s_logger = Logger.getLogger(StorageOrchestrator.class);
+ @Inject
+ TemplateDataStoreDao templateDataStoreDao;
+ @Inject
+ SnapshotDataStoreDao snapshotDataStoreDao;
+ @Inject
+ VolumeDataStoreDao volumeDataStoreDao;
+ @Inject
+ VolumeDataFactory volumeFactory;
+ @Inject
+ VMTemplateDao templateDao;
+ @Inject
+ TemplateDataFactory templateFactory;
+ @Inject
+ SnapshotDao snapshotDao;
+ @Inject
+ SnapshotDataFactory snapshotFactory;
+ @Inject
+ DataStoreManager dataStoreManager;
+ @Inject
+ ImageStoreDao imageStoreDao;
+ @Inject
+ StatsCollector statsCollector;
+ @Inject
+ public StorageService storageService;
+ @Inject
+ SecondaryStorageVmDao secStorageVmDao;
+ @Inject
+ ConfigurationDao configDao;
+ @Inject
+ HostDao hostDao;
+ @Inject
+ private AsyncJobManager jobMgr;
+ @Inject
+ private SecondaryStorageService secStgSrv;
+
+ ConfigKey<Double> ImageStoreImbalanceThreshold = new ConfigKey<>("Advanced", Double.class,
+ "image.store.imbalance.threshold",
+ "0.1",
+ "The storage imbalance threshold that is compared with the standard deviation percentage for a storage utilization metric. " +
+ "The value is a percentage in decimal format.",
+ true, ConfigKey.Scope.Global);
+
+ Integer numConcurrentCopyTasksPerSSVM = 2;
+
+ private double imageStoreCapacityThreshold = 0.90;
+
+ @Override
+ public String getConfigComponentName() {
+ return StorageOrchestrationService.class.getName();
+ }
+
+ @Override
+ public ConfigKey<?>[] getConfigKeys() {
+ return new ConfigKey<?>[]{ImageStoreImbalanceThreshold};
+ }
+
+ static class MigrateBlockingQueue<T> extends ArrayBlockingQueue<T> {
+
+ MigrateBlockingQueue(int size) {
+ super(size);
+ }
+
+ public boolean offer(T task) {
+ try {
+ this.put(task);
+ } catch (InterruptedException e) {
+ throw new RuntimeException(e);
+ }
+ return true;
+ }
+ }
+
+ @Override
+ public boolean configure(String name, Map<String, Object> params) throws ConfigurationException {
+ numConcurrentCopyTasksPerSSVM = NumbersUtil.parseInt(configDao.getValue(Config.SecStorageMaxMigrateSessions.key()), 2);
+ return true;
+ }
+
+ @Override
+ public MigrationResponse migrateData(Long srcDataStoreId, List<Long> destDatastores, MigrationPolicy migrationPolicy) {
+ List<DataObject> files = new LinkedList<>();
+ int successCount = 0;
+ boolean success = true;
+ String message = null;
+
+ if (migrationPolicy == MigrationPolicy.COMPLETE) {
+ if (!filesReady(srcDataStoreId)) {
+ throw new CloudRuntimeException("Complete migration failed as there are data objects which are not Ready");
+ }
+ }
+
+ DataStore srcDatastore = dataStoreManager.getDataStore(srcDataStoreId, DataStoreRole.Image);
+ Map<DataObject, Pair<List<SnapshotInfo>, Long>> snapshotChains = new HashMap<>();
+ files.addAll(getAllValidTemplates(srcDatastore));
+ files.addAll(getAllValidSnapshotChains(srcDatastore, snapshotChains));
+ files.addAll(getAllValidVolumes(srcDatastore));
+
+ Collections.sort(files, new Comparator<DataObject>() {
+ @Override
+ public int compare(DataObject o1, DataObject o2) {
+ Long size1 = o1.getSize();
+ Long size2 = o2.getSize();
+ if (o1 instanceof SnapshotInfo) {
+ size1 = snapshotChains.get(o1).second();
+ }
+ if (o2 instanceof SnapshotInfo) {
+ size2 = snapshotChains.get(o2).second();
+ }
+ //return o2.getSize() > o1.getSize() ? 1 : -1;
+ return size2 > size1 ? 1 : -1;
Review comment:
so never 0? Can we return size2 - size1?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-690969340
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-671768083
@DaanHoogland a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-646031103
@blueorangutan package
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-691032668
@blueorangutan test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-652765366
Packaging result: ✔centos7 ✔debian. JID-1503
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-672754775
@Pearl1594 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-651349706
<b>Trillian test result (tid-1901)</b>
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 26221 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4053-t1901-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_direct_download.py
Intermittent failure detected: /marvin/tests/smoke/test_list_ids_parameter.py
Intermittent failure detected: /marvin/tests/smoke/test_templates.py
Intermittent failure detected: /marvin/tests/smoke/test_usage.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_router_nics.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_vpn.py
Intermittent failure detected: /marvin/tests/smoke/test_hostha_kvm.py
Smoke tests completed. 75 look OK, 8 have error(s)
Only failed tests results shown below:
Test | Result | Time (s) | Test File
--- | --- | --- | ---
ContextSuite context=TestDirectDownloadTemplates>:setup | `Error` | 0.00 | test_direct_download.py
ContextSuite context=TestListIdsParams>:setup | `Error` | 0.00 | test_list_ids_parameter.py
test_01_create_template | `Error` | 18.48 | test_templates.py
test_CreateTemplateWithDuplicateName | `Error` | 18.48 | test_templates.py
test_01_register_template_direct_download_flag | `Error` | 0.29 | test_templates.py
test_02_deploy_vm_from_direct_download_template | `Error` | 0.00 | test_templates.py
test_03_deploy_vm_wrong_checksum | `Error` | 0.07 | test_templates.py
ContextSuite context=TestTemplates>:setup | `Error` | 228.28 | test_templates.py
test_01_template_usage | `Error` | 18.55 | test_usage.py
ContextSuite context=TestVPCRedundancy>:setup | `Error` | 0.00 | test_vpc_redundant.py
ContextSuite context=TestVPCNics>:setup | `Error` | 0.00 | test_vpc_router_nics.py
ContextSuite context=TestRVPCSite2SiteVpn>:setup | `Error` | 0.00 | test_vpc_vpn.py
ContextSuite context=TestVPCSite2SiteVPNMultipleOptions>:setup | `Error` | 0.00 | test_vpc_vpn.py
ContextSuite context=TestVpcRemoteAccessVpn>:setup | `Error` | 0.00 | test_vpc_vpn.py
ContextSuite context=TestVpcSite2SiteVpn>:setup | `Error` | 0.00 | test_vpc_vpn.py
test_disable_oobm_ha_state_ineligible | `Error` | 0.23 | test_hostha_kvm.py
test_hostha_configure_default_driver | `Error` | 0.23 | test_hostha_kvm.py
test_hostha_enable_ha_when_host_disabled | `Error` | 0.22 | test_hostha_kvm.py
test_hostha_enable_ha_when_host_disconected | `Error` | 0.22 | test_hostha_kvm.py
test_hostha_enable_ha_when_host_in_maintenance | `Error` | 0.23 | test_hostha_kvm.py
test_hostha_kvm_host_degraded | `Error` | 0.23 | test_hostha_kvm.py
test_hostha_kvm_host_fencing | `Error` | 0.22 | test_hostha_kvm.py
test_hostha_kvm_host_recovering | `Error` | 0.23 | test_hostha_kvm.py
test_remove_ha_provider_not_possible | `Error` | 0.23 | test_hostha_kvm.py
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-632678471
@vladimirpetrov a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-691018067
Packaging result: ✖centos7 ✖centos8 ✖debian. JID-1963
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-674676488
@blueorangutan test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-649998954
Packaging result: ✖centos7 ✖debian. JID-1466
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-653537691
@Pearl1594 a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
rhtyd commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-657406850
lgtm did a quick one, let's see if they pass a round of smoketests. @Pearl1594 please review Vladi's new marvin tests, thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-650312640
<b>Trillian test result (tid-1856)</b>
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 37202 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4053-t1856-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_direct_download.py
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_clusters.py
Intermittent failure detected: /marvin/tests/smoke/test_list_ids_parameter.py
Intermittent failure detected: /marvin/tests/smoke/test_templates.py
Intermittent failure detected: /marvin/tests/smoke/test_usage.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_router_nics.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_vpn.py
Intermittent failure detected: /marvin/tests/smoke/test_hostha_kvm.py
Smoke tests completed. 75 look OK, 8 have error(s)
Only failed tests results shown below:
Test | Result | Time (s) | Test File
--- | --- | --- | ---
ContextSuite context=TestDirectDownloadTemplates>:setup | `Error` | 0.00 | test_direct_download.py
ContextSuite context=TestListIdsParams>:setup | `Error` | 0.00 | test_list_ids_parameter.py
test_01_create_template | `Error` | 55.11 | test_templates.py
test_CreateTemplateWithDuplicateName | `Error` | 32.93 | test_templates.py
test_01_register_template_direct_download_flag | `Error` | 0.13 | test_templates.py
test_02_deploy_vm_from_direct_download_template | `Error` | 0.00 | test_templates.py
test_03_deploy_vm_wrong_checksum | `Error` | 0.12 | test_templates.py
ContextSuite context=TestTemplates>:setup | `Error` | 223.66 | test_templates.py
test_01_template_usage | `Error` | 64.68 | test_usage.py
ContextSuite context=TestVPCRedundancy>:setup | `Error` | 0.00 | test_vpc_redundant.py
ContextSuite context=TestVPCNics>:setup | `Error` | 0.00 | test_vpc_router_nics.py
ContextSuite context=TestRVPCSite2SiteVpn>:setup | `Error` | 0.00 | test_vpc_vpn.py
ContextSuite context=TestVPCSite2SiteVPNMultipleOptions>:setup | `Error` | 0.00 | test_vpc_vpn.py
ContextSuite context=TestVpcRemoteAccessVpn>:setup | `Error` | 0.00 | test_vpc_vpn.py
ContextSuite context=TestVpcSite2SiteVpn>:setup | `Error` | 0.00 | test_vpc_vpn.py
test_disable_oobm_ha_state_ineligible | `Error` | 2.17 | test_hostha_kvm.py
test_hostha_configure_default_driver | `Error` | 2.01 | test_hostha_kvm.py
test_hostha_enable_ha_when_host_disabled | `Error` | 2.31 | test_hostha_kvm.py
test_hostha_enable_ha_when_host_disconected | `Error` | 1.50 | test_hostha_kvm.py
test_hostha_enable_ha_when_host_in_maintenance | `Error` | 0.44 | test_hostha_kvm.py
test_hostha_kvm_host_degraded | `Error` | 0.41 | test_hostha_kvm.py
test_hostha_kvm_host_fencing | `Error` | 0.43 | test_hostha_kvm.py
test_hostha_kvm_host_recovering | `Error` | 0.42 | test_hostha_kvm.py
test_remove_ha_provider_not_possible | `Error` | 0.46 | test_hostha_kvm.py
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on a change in pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on a change in pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#discussion_r427057420
##########
File path: engine/orchestration/src/main/java/org/apache/cloudstack/engine/orchestration/DataMigrationUtility.java
##########
@@ -0,0 +1,262 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.cloudstack.engine.orchestration;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+
+import javax.inject.Inject;
+
+import org.apache.cloudstack.engine.subsystem.api.storage.DataObject;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.engine.subsystem.api.storage.ObjectInDataStoreStateMachine;
+import org.apache.cloudstack.engine.subsystem.api.storage.SnapshotDataFactory;
+import org.apache.cloudstack.engine.subsystem.api.storage.SnapshotInfo;
+import org.apache.cloudstack.engine.subsystem.api.storage.TemplateDataFactory;
+import org.apache.cloudstack.engine.subsystem.api.storage.VolumeDataFactory;
+import org.apache.cloudstack.storage.ImageStoreService;
+import org.apache.cloudstack.storage.datastore.db.SnapshotDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.SnapshotDataStoreVO;
+import org.apache.cloudstack.storage.datastore.db.TemplateDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.TemplateDataStoreVO;
+import org.apache.cloudstack.storage.datastore.db.VolumeDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.VolumeDataStoreVO;
+import org.apache.log4j.Logger;
+
+import com.cloud.host.HostVO;
+import com.cloud.host.Status;
+import com.cloud.host.dao.HostDao;
+import com.cloud.storage.DataStoreRole;
+import com.cloud.storage.SnapshotVO;
+import com.cloud.storage.VMTemplateVO;
+import com.cloud.storage.dao.SnapshotDao;
+import com.cloud.storage.dao.VMTemplateDao;
+import com.cloud.utils.Pair;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.vm.SecondaryStorageVm;
+import com.cloud.vm.SecondaryStorageVmVO;
+import com.cloud.vm.VirtualMachine;
+import com.cloud.vm.dao.SecondaryStorageVmDao;
+
+public class DataMigrationUtility {
+ @Inject
+ SecondaryStorageVmDao secStorageVmDao;
+ @Inject
+ TemplateDataStoreDao templateDataStoreDao;
+ @Inject
+ SnapshotDataStoreDao snapshotDataStoreDao;
+ @Inject
+ VolumeDataStoreDao volumeDataStoreDao;
+ @Inject
+ VMTemplateDao templateDao;
+ @Inject
+ VolumeDataFactory volumeFactory;
+ @Inject
+ TemplateDataFactory templateFactory;
+ @Inject
+ SnapshotDataFactory snapshotFactory;
+ @Inject
+ HostDao hostDao;
+ @Inject
+ SnapshotDao snapshotDao;
+
+ private static final Logger s_logger = Logger.getLogger(DataMigrationUtility.class);
+
+ /** This function verifies if the given image store comprises of data objects that are not in either the "Ready" or
+ * "Allocated" state - in such a case, if the migration policy is complete, the migration is terminated
+ */
+ private boolean filesReady(Long srcDataStoreId) {
+ String[] validStates = new String[]{"Ready", "Allocated"};
+ boolean isReady = true;
+ List<TemplateDataStoreVO> templates = templateDataStoreDao.listByStoreId(srcDataStoreId);
+ for (TemplateDataStoreVO template : templates) {
+ isReady &= (Arrays.asList(validStates).contains(template.getState().toString()));
+ }
+ List<SnapshotDataStoreVO> snapshots = snapshotDataStoreDao.listByStoreId(srcDataStoreId, DataStoreRole.Image);
+ for (SnapshotDataStoreVO snapshot : snapshots) {
+ isReady &= (Arrays.asList(validStates).contains(snapshot.getState().toString()));
+ }
+ List<VolumeDataStoreVO> volumes = volumeDataStoreDao.listByStoreId(srcDataStoreId);
+ for (VolumeDataStoreVO volume : volumes) {
+ isReady &= (Arrays.asList(validStates).contains(volume.getState().toString()));
+ }
+ return isReady;
+ }
+
+ protected void checkIfCompleteMigrationPossible(ImageStoreService.MigrationPolicy policy, Long srcDataStoreId) {
+ if (policy == ImageStoreService.MigrationPolicy.COMPLETE) {
+ if (!filesReady(srcDataStoreId)) {
+ throw new CloudRuntimeException("Complete migration failed as there are data objects which are not Ready");
+ }
+ }
+ return;
+ }
+
+ protected Long getFileSize(DataObject file, Map<DataObject, Pair<List<SnapshotInfo>, Long>> snapshotChain) {
+ Long size = file.getSize();
+ Pair<List<SnapshotInfo>, Long> chain = snapshotChain.get(file);
+ if (file instanceof SnapshotInfo && chain.first() != null) {
+ size = chain.second();
+ }
+ return size;
+ }
+
+ /**
+ * Sorts the datastores in decreasing order of their free capacities, so as to make
+ * an informed decision of picking the datastore with maximum free capactiy for migration
+ */
+ protected List<Long> sortDataStores(Map<Long, Pair<Long, Long>> storageCapacities) {
+ List<Map.Entry<Long, Pair<Long, Long>>> list =
+ new LinkedList<Map.Entry<Long, Pair<Long, Long>>>((storageCapacities.entrySet()));
+
+ Collections.sort(list, new Comparator<Map.Entry<Long, Pair<Long, Long>>>() {
+ @Override
+ public int compare(Map.Entry<Long, Pair<Long, Long>> e1, Map.Entry<Long, Pair<Long, Long>> e2) {
+ return e2.getValue().first() > e1.getValue().first() ? 1 : -1;
+ }
+ });
+ HashMap<Long, Pair<Long, Long>> temp = new LinkedHashMap<>();
+ for (Map.Entry<Long, Pair<Long, Long>> value : list) {
+ temp.put(value.getKey(), value.getValue());
+ }
+
+ return new ArrayList<>(temp.keySet());
+ }
+
+ protected List<DataObject> getSortedValidSourcesList(DataStore srcDataStore, Map<DataObject, Pair<List<SnapshotInfo>, Long>> snapshotChains) {
+ List<DataObject> files = new ArrayList<>();
+ files.addAll(getAllValidTemplates(srcDataStore));
+ files.addAll(getAllValidSnapshotsAndChains(srcDataStore, snapshotChains));
+ files.addAll(getAllValidVolumes(srcDataStore));
+
+ files = sortFilesOnSize(files, snapshotChains);
+
+ return files;
+ }
+
+ protected List<DataObject> sortFilesOnSize(List<DataObject> files, Map<DataObject, Pair<List<SnapshotInfo>, Long>> snapshotChains) {
+ Collections.sort(files, new Comparator<DataObject>() {
+ @Override
+ public int compare(DataObject o1, DataObject o2) {
+ Long size1 = o1.getSize();
+ Long size2 = o2.getSize();
+ if (o1 instanceof SnapshotInfo) {
+ size1 = snapshotChains.get(o1).second();
+ }
+ if (o2 instanceof SnapshotInfo) {
+ size2 = snapshotChains.get(o2).second();
+ }
+ return (int) (size2 - size1);
+ }
+ });
+ return files;
+ }
+
+ // Gets list of all valid templates, i.e, templates in "Ready" state for migration
+ protected List<DataObject> getAllValidTemplates(DataStore srcDataStore) {
+
+ List<DataObject> files = new LinkedList<>();
+ List<TemplateDataStoreVO> templates = templateDataStoreDao.listByStoreId(srcDataStore.getId());
+ for (TemplateDataStoreVO template : templates) {
+ VMTemplateVO templateVO = templateDao.findById(template.getTemplateId());
+ if (template.getState() == ObjectInDataStoreStateMachine.State.Ready && !templateVO.isPublicTemplate()) {
+ files.add(templateFactory.getTemplate(template.getTemplateId(), srcDataStore));
+ }
+ }
+ return files;
+ }
+
+ /** Returns parent snapshots and snapshots that do not have any children; snapshotChains comprises of the snapshot chain info
+ * for each parent snapshot and the cumulative size of the chain - this is done to ensure that all the snapshots in a chain
+ * are migrated to the same datastore
+ */
Review comment:
@DaanHoogland Could you please expand on this
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-682722465
@Pearl1594 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] DaanHoogland commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
DaanHoogland commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-671767594
@blueorangutan package
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-675915211
@Pearl1594 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-648620773
@blueorangutan test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-648714739
@blueorangutan package
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-672346280
<b>Trillian test result (tid-2357)</b>
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 52233 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4053-t2357-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_supported_versions.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Smoke tests completed. 81 look OK, 2 have error(s)
Only failed tests results shown below:
Test | Result | Time (s) | Test File
--- | --- | --- | ---
test_01_add_delete_kubernetes_supported_version | `Error` | 1807.53 | test_kubernetes_supported_versions.py
test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL | `Failure` | 411.01 | test_vpc_redundant.py
test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers | `Error` | 344.84 | test_vpc_redundant.py
test_04_rvpc_network_garbage_collector_nics | `Error` | 3855.47 | test_vpc_redundant.py
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-663398343
Packaging result: ✔centos7 ✔debian. JID-1600
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
rhtyd commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-684741055
@blueorangutan test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] vladimirpetrov commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
vladimirpetrov commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-631701133
@blueorangutan package
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-653537140
@blueorangutan test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
rhtyd commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-642534794
@Pearl1594 can you fix the conflict and kick packaging and a smoke test?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on a change in pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on a change in pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#discussion_r422978447
##########
File path: engine/orchestration/src/main/java/org/apache/cloudstack/engine/orchestration/StorageOrchestrator.java
##########
@@ -0,0 +1,629 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.cloudstack.engine.orchestration;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.Hashtable;
+import java.util.LinkedHashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+import javax.inject.Inject;
+import javax.naming.ConfigurationException;
+
+import org.apache.cloudstack.api.response.MigrationResponse;
+import org.apache.cloudstack.engine.orchestration.service.StorageOrchestrationService;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataObject;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStoreManager;
+import org.apache.cloudstack.engine.subsystem.api.storage.ObjectInDataStoreStateMachine;
+import org.apache.cloudstack.engine.subsystem.api.storage.SecondaryStorageService;
+import org.apache.cloudstack.engine.subsystem.api.storage.SecondaryStorageService.DataObjectResult;
+import org.apache.cloudstack.engine.subsystem.api.storage.SnapshotDataFactory;
+import org.apache.cloudstack.engine.subsystem.api.storage.SnapshotInfo;
+import org.apache.cloudstack.engine.subsystem.api.storage.TemplateDataFactory;
+import org.apache.cloudstack.engine.subsystem.api.storage.VolumeDataFactory;
+import org.apache.cloudstack.framework.async.AsyncCallFuture;
+import org.apache.cloudstack.framework.config.ConfigKey;
+import org.apache.cloudstack.framework.config.Configurable;
+import org.apache.cloudstack.framework.config.dao.ConfigurationDao;
+import org.apache.cloudstack.framework.jobs.AsyncJobManager;
+import org.apache.cloudstack.storage.datastore.db.ImageStoreDao;
+import org.apache.cloudstack.storage.datastore.db.SnapshotDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.SnapshotDataStoreVO;
+import org.apache.cloudstack.storage.datastore.db.TemplateDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.TemplateDataStoreVO;
+import org.apache.cloudstack.storage.datastore.db.VolumeDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.VolumeDataStoreVO;
+import org.apache.commons.math3.stat.descriptive.moment.Mean;
+import org.apache.commons.math3.stat.descriptive.moment.StandardDeviation;
+import org.apache.log4j.Logger;
+
+import com.cloud.configuration.Config;
+import com.cloud.host.HostVO;
+import com.cloud.host.Status;
+import com.cloud.host.dao.HostDao;
+import com.cloud.server.StatsCollector;
+import com.cloud.storage.DataStoreRole;
+import com.cloud.storage.ImageStoreService.MigrationPolicy;
+import com.cloud.storage.SnapshotVO;
+import com.cloud.storage.StorageService;
+import com.cloud.storage.StorageStats;
+import com.cloud.storage.VMTemplateVO;
+import com.cloud.storage.dao.SnapshotDao;
+import com.cloud.storage.dao.VMTemplateDao;
+import com.cloud.utils.NumbersUtil;
+import com.cloud.utils.Pair;
+import com.cloud.utils.StringUtils;
+import com.cloud.utils.component.ManagerBase;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.vm.SecondaryStorageVm;
+import com.cloud.vm.SecondaryStorageVmVO;
+import com.cloud.vm.VirtualMachine;
+import com.cloud.vm.dao.SecondaryStorageVmDao;
+
+public class StorageOrchestrator extends ManagerBase implements StorageOrchestrationService, Configurable {
+
+ private static final Logger s_logger = Logger.getLogger(StorageOrchestrator.class);
+ @Inject
+ TemplateDataStoreDao templateDataStoreDao;
+ @Inject
+ SnapshotDataStoreDao snapshotDataStoreDao;
+ @Inject
+ VolumeDataStoreDao volumeDataStoreDao;
+ @Inject
+ VolumeDataFactory volumeFactory;
+ @Inject
+ VMTemplateDao templateDao;
+ @Inject
+ TemplateDataFactory templateFactory;
+ @Inject
+ SnapshotDao snapshotDao;
+ @Inject
+ SnapshotDataFactory snapshotFactory;
+ @Inject
+ DataStoreManager dataStoreManager;
+ @Inject
+ ImageStoreDao imageStoreDao;
+ @Inject
+ StatsCollector statsCollector;
+ @Inject
+ public StorageService storageService;
+ @Inject
+ SecondaryStorageVmDao secStorageVmDao;
+ @Inject
+ ConfigurationDao configDao;
+ @Inject
+ HostDao hostDao;
+ @Inject
+ private AsyncJobManager jobMgr;
+ @Inject
+ private SecondaryStorageService secStgSrv;
+
+ ConfigKey<Double> ImageStoreImbalanceThreshold = new ConfigKey<>("Advanced", Double.class,
+ "image.store.imbalance.threshold",
+ "0.1",
+ "The storage imbalance threshold that is compared with the standard deviation percentage for a storage utilization metric. " +
+ "The value is a percentage in decimal format.",
+ true, ConfigKey.Scope.Global);
+
+ Integer numConcurrentCopyTasksPerSSVM = 2;
+
+ private double imageStoreCapacityThreshold = 0.90;
+
+ @Override
+ public String getConfigComponentName() {
+ return StorageOrchestrationService.class.getName();
+ }
+
+ @Override
+ public ConfigKey<?>[] getConfigKeys() {
+ return new ConfigKey<?>[]{ImageStoreImbalanceThreshold};
+ }
+
+ static class MigrateBlockingQueue<T> extends ArrayBlockingQueue<T> {
+
+ MigrateBlockingQueue(int size) {
+ super(size);
+ }
+
+ public boolean offer(T task) {
+ try {
+ this.put(task);
+ } catch (InterruptedException e) {
+ throw new RuntimeException(e);
+ }
+ return true;
+ }
+ }
+
+ @Override
+ public boolean configure(String name, Map<String, Object> params) throws ConfigurationException {
+ numConcurrentCopyTasksPerSSVM = NumbersUtil.parseInt(configDao.getValue(Config.SecStorageMaxMigrateSessions.key()), 2);
+ return true;
+ }
+
+ @Override
+ public MigrationResponse migrateData(Long srcDataStoreId, List<Long> destDatastores, MigrationPolicy migrationPolicy) {
Review comment:
Created a utility class
##########
File path: engine/orchestration/src/main/java/org/apache/cloudstack/engine/orchestration/StorageOrchestrator.java
##########
@@ -0,0 +1,629 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.cloudstack.engine.orchestration;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.Hashtable;
+import java.util.LinkedHashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+import javax.inject.Inject;
+import javax.naming.ConfigurationException;
+
+import org.apache.cloudstack.api.response.MigrationResponse;
+import org.apache.cloudstack.engine.orchestration.service.StorageOrchestrationService;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataObject;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStoreManager;
+import org.apache.cloudstack.engine.subsystem.api.storage.ObjectInDataStoreStateMachine;
+import org.apache.cloudstack.engine.subsystem.api.storage.SecondaryStorageService;
+import org.apache.cloudstack.engine.subsystem.api.storage.SecondaryStorageService.DataObjectResult;
+import org.apache.cloudstack.engine.subsystem.api.storage.SnapshotDataFactory;
+import org.apache.cloudstack.engine.subsystem.api.storage.SnapshotInfo;
+import org.apache.cloudstack.engine.subsystem.api.storage.TemplateDataFactory;
+import org.apache.cloudstack.engine.subsystem.api.storage.VolumeDataFactory;
+import org.apache.cloudstack.framework.async.AsyncCallFuture;
+import org.apache.cloudstack.framework.config.ConfigKey;
+import org.apache.cloudstack.framework.config.Configurable;
+import org.apache.cloudstack.framework.config.dao.ConfigurationDao;
+import org.apache.cloudstack.framework.jobs.AsyncJobManager;
+import org.apache.cloudstack.storage.datastore.db.ImageStoreDao;
+import org.apache.cloudstack.storage.datastore.db.SnapshotDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.SnapshotDataStoreVO;
+import org.apache.cloudstack.storage.datastore.db.TemplateDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.TemplateDataStoreVO;
+import org.apache.cloudstack.storage.datastore.db.VolumeDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.VolumeDataStoreVO;
+import org.apache.commons.math3.stat.descriptive.moment.Mean;
+import org.apache.commons.math3.stat.descriptive.moment.StandardDeviation;
+import org.apache.log4j.Logger;
+
+import com.cloud.configuration.Config;
+import com.cloud.host.HostVO;
+import com.cloud.host.Status;
+import com.cloud.host.dao.HostDao;
+import com.cloud.server.StatsCollector;
+import com.cloud.storage.DataStoreRole;
+import com.cloud.storage.ImageStoreService.MigrationPolicy;
+import com.cloud.storage.SnapshotVO;
+import com.cloud.storage.StorageService;
+import com.cloud.storage.StorageStats;
+import com.cloud.storage.VMTemplateVO;
+import com.cloud.storage.dao.SnapshotDao;
+import com.cloud.storage.dao.VMTemplateDao;
+import com.cloud.utils.NumbersUtil;
+import com.cloud.utils.Pair;
+import com.cloud.utils.StringUtils;
+import com.cloud.utils.component.ManagerBase;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.vm.SecondaryStorageVm;
+import com.cloud.vm.SecondaryStorageVmVO;
+import com.cloud.vm.VirtualMachine;
+import com.cloud.vm.dao.SecondaryStorageVmDao;
+
+public class StorageOrchestrator extends ManagerBase implements StorageOrchestrationService, Configurable {
+
+ private static final Logger s_logger = Logger.getLogger(StorageOrchestrator.class);
+ @Inject
+ TemplateDataStoreDao templateDataStoreDao;
+ @Inject
+ SnapshotDataStoreDao snapshotDataStoreDao;
+ @Inject
+ VolumeDataStoreDao volumeDataStoreDao;
+ @Inject
+ VolumeDataFactory volumeFactory;
+ @Inject
+ VMTemplateDao templateDao;
+ @Inject
+ TemplateDataFactory templateFactory;
+ @Inject
+ SnapshotDao snapshotDao;
+ @Inject
+ SnapshotDataFactory snapshotFactory;
+ @Inject
+ DataStoreManager dataStoreManager;
+ @Inject
+ ImageStoreDao imageStoreDao;
+ @Inject
+ StatsCollector statsCollector;
+ @Inject
+ public StorageService storageService;
+ @Inject
+ SecondaryStorageVmDao secStorageVmDao;
+ @Inject
+ ConfigurationDao configDao;
+ @Inject
+ HostDao hostDao;
+ @Inject
+ private AsyncJobManager jobMgr;
+ @Inject
+ private SecondaryStorageService secStgSrv;
+
+ ConfigKey<Double> ImageStoreImbalanceThreshold = new ConfigKey<>("Advanced", Double.class,
+ "image.store.imbalance.threshold",
+ "0.1",
+ "The storage imbalance threshold that is compared with the standard deviation percentage for a storage utilization metric. " +
+ "The value is a percentage in decimal format.",
+ true, ConfigKey.Scope.Global);
+
+ Integer numConcurrentCopyTasksPerSSVM = 2;
+
+ private double imageStoreCapacityThreshold = 0.90;
+
+ @Override
+ public String getConfigComponentName() {
+ return StorageOrchestrationService.class.getName();
+ }
+
+ @Override
+ public ConfigKey<?>[] getConfigKeys() {
+ return new ConfigKey<?>[]{ImageStoreImbalanceThreshold};
+ }
+
+ static class MigrateBlockingQueue<T> extends ArrayBlockingQueue<T> {
+
+ MigrateBlockingQueue(int size) {
+ super(size);
+ }
+
+ public boolean offer(T task) {
+ try {
+ this.put(task);
+ } catch (InterruptedException e) {
+ throw new RuntimeException(e);
+ }
+ return true;
+ }
+ }
+
+ @Override
+ public boolean configure(String name, Map<String, Object> params) throws ConfigurationException {
+ numConcurrentCopyTasksPerSSVM = NumbersUtil.parseInt(configDao.getValue(Config.SecStorageMaxMigrateSessions.key()), 2);
+ return true;
+ }
+
+ @Override
+ public MigrationResponse migrateData(Long srcDataStoreId, List<Long> destDatastores, MigrationPolicy migrationPolicy) {
+ List<DataObject> files = new LinkedList<>();
+ int successCount = 0;
+ boolean success = true;
+ String message = null;
+
+ if (migrationPolicy == MigrationPolicy.COMPLETE) {
+ if (!filesReady(srcDataStoreId)) {
+ throw new CloudRuntimeException("Complete migration failed as there are data objects which are not Ready");
+ }
+ }
Review comment:
done
##########
File path: engine/orchestration/src/main/java/org/apache/cloudstack/engine/orchestration/StorageOrchestrator.java
##########
@@ -0,0 +1,629 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.cloudstack.engine.orchestration;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.Hashtable;
+import java.util.LinkedHashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+import javax.inject.Inject;
+import javax.naming.ConfigurationException;
+
+import org.apache.cloudstack.api.response.MigrationResponse;
+import org.apache.cloudstack.engine.orchestration.service.StorageOrchestrationService;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataObject;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStoreManager;
+import org.apache.cloudstack.engine.subsystem.api.storage.ObjectInDataStoreStateMachine;
+import org.apache.cloudstack.engine.subsystem.api.storage.SecondaryStorageService;
+import org.apache.cloudstack.engine.subsystem.api.storage.SecondaryStorageService.DataObjectResult;
+import org.apache.cloudstack.engine.subsystem.api.storage.SnapshotDataFactory;
+import org.apache.cloudstack.engine.subsystem.api.storage.SnapshotInfo;
+import org.apache.cloudstack.engine.subsystem.api.storage.TemplateDataFactory;
+import org.apache.cloudstack.engine.subsystem.api.storage.VolumeDataFactory;
+import org.apache.cloudstack.framework.async.AsyncCallFuture;
+import org.apache.cloudstack.framework.config.ConfigKey;
+import org.apache.cloudstack.framework.config.Configurable;
+import org.apache.cloudstack.framework.config.dao.ConfigurationDao;
+import org.apache.cloudstack.framework.jobs.AsyncJobManager;
+import org.apache.cloudstack.storage.datastore.db.ImageStoreDao;
+import org.apache.cloudstack.storage.datastore.db.SnapshotDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.SnapshotDataStoreVO;
+import org.apache.cloudstack.storage.datastore.db.TemplateDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.TemplateDataStoreVO;
+import org.apache.cloudstack.storage.datastore.db.VolumeDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.VolumeDataStoreVO;
+import org.apache.commons.math3.stat.descriptive.moment.Mean;
+import org.apache.commons.math3.stat.descriptive.moment.StandardDeviation;
+import org.apache.log4j.Logger;
+
+import com.cloud.configuration.Config;
+import com.cloud.host.HostVO;
+import com.cloud.host.Status;
+import com.cloud.host.dao.HostDao;
+import com.cloud.server.StatsCollector;
+import com.cloud.storage.DataStoreRole;
+import com.cloud.storage.ImageStoreService.MigrationPolicy;
+import com.cloud.storage.SnapshotVO;
+import com.cloud.storage.StorageService;
+import com.cloud.storage.StorageStats;
+import com.cloud.storage.VMTemplateVO;
+import com.cloud.storage.dao.SnapshotDao;
+import com.cloud.storage.dao.VMTemplateDao;
+import com.cloud.utils.NumbersUtil;
+import com.cloud.utils.Pair;
+import com.cloud.utils.StringUtils;
+import com.cloud.utils.component.ManagerBase;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.vm.SecondaryStorageVm;
+import com.cloud.vm.SecondaryStorageVmVO;
+import com.cloud.vm.VirtualMachine;
+import com.cloud.vm.dao.SecondaryStorageVmDao;
+
+public class StorageOrchestrator extends ManagerBase implements StorageOrchestrationService, Configurable {
+
+ private static final Logger s_logger = Logger.getLogger(StorageOrchestrator.class);
+ @Inject
+ TemplateDataStoreDao templateDataStoreDao;
+ @Inject
+ SnapshotDataStoreDao snapshotDataStoreDao;
+ @Inject
+ VolumeDataStoreDao volumeDataStoreDao;
+ @Inject
+ VolumeDataFactory volumeFactory;
+ @Inject
+ VMTemplateDao templateDao;
+ @Inject
+ TemplateDataFactory templateFactory;
+ @Inject
+ SnapshotDao snapshotDao;
+ @Inject
+ SnapshotDataFactory snapshotFactory;
+ @Inject
+ DataStoreManager dataStoreManager;
+ @Inject
+ ImageStoreDao imageStoreDao;
+ @Inject
+ StatsCollector statsCollector;
+ @Inject
+ public StorageService storageService;
+ @Inject
+ SecondaryStorageVmDao secStorageVmDao;
+ @Inject
+ ConfigurationDao configDao;
+ @Inject
+ HostDao hostDao;
+ @Inject
+ private AsyncJobManager jobMgr;
+ @Inject
+ private SecondaryStorageService secStgSrv;
+
+ ConfigKey<Double> ImageStoreImbalanceThreshold = new ConfigKey<>("Advanced", Double.class,
+ "image.store.imbalance.threshold",
+ "0.1",
+ "The storage imbalance threshold that is compared with the standard deviation percentage for a storage utilization metric. " +
+ "The value is a percentage in decimal format.",
+ true, ConfigKey.Scope.Global);
+
+ Integer numConcurrentCopyTasksPerSSVM = 2;
+
+ private double imageStoreCapacityThreshold = 0.90;
+
+ @Override
+ public String getConfigComponentName() {
+ return StorageOrchestrationService.class.getName();
+ }
+
+ @Override
+ public ConfigKey<?>[] getConfigKeys() {
+ return new ConfigKey<?>[]{ImageStoreImbalanceThreshold};
+ }
+
+ static class MigrateBlockingQueue<T> extends ArrayBlockingQueue<T> {
+
+ MigrateBlockingQueue(int size) {
+ super(size);
+ }
+
+ public boolean offer(T task) {
+ try {
+ this.put(task);
+ } catch (InterruptedException e) {
+ throw new RuntimeException(e);
+ }
+ return true;
+ }
+ }
+
+ @Override
+ public boolean configure(String name, Map<String, Object> params) throws ConfigurationException {
+ numConcurrentCopyTasksPerSSVM = NumbersUtil.parseInt(configDao.getValue(Config.SecStorageMaxMigrateSessions.key()), 2);
+ return true;
+ }
+
+ @Override
+ public MigrationResponse migrateData(Long srcDataStoreId, List<Long> destDatastores, MigrationPolicy migrationPolicy) {
+ List<DataObject> files = new LinkedList<>();
+ int successCount = 0;
+ boolean success = true;
+ String message = null;
+
+ if (migrationPolicy == MigrationPolicy.COMPLETE) {
+ if (!filesReady(srcDataStoreId)) {
+ throw new CloudRuntimeException("Complete migration failed as there are data objects which are not Ready");
+ }
+ }
+
+ DataStore srcDatastore = dataStoreManager.getDataStore(srcDataStoreId, DataStoreRole.Image);
+ Map<DataObject, Pair<List<SnapshotInfo>, Long>> snapshotChains = new HashMap<>();
+ files.addAll(getAllValidTemplates(srcDatastore));
+ files.addAll(getAllValidSnapshotChains(srcDatastore, snapshotChains));
+ files.addAll(getAllValidVolumes(srcDatastore));
Review comment:
done
##########
File path: engine/orchestration/src/main/java/org/apache/cloudstack/engine/orchestration/StorageOrchestrator.java
##########
@@ -0,0 +1,629 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.cloudstack.engine.orchestration;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.Hashtable;
+import java.util.LinkedHashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+import javax.inject.Inject;
+import javax.naming.ConfigurationException;
+
+import org.apache.cloudstack.api.response.MigrationResponse;
+import org.apache.cloudstack.engine.orchestration.service.StorageOrchestrationService;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataObject;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStoreManager;
+import org.apache.cloudstack.engine.subsystem.api.storage.ObjectInDataStoreStateMachine;
+import org.apache.cloudstack.engine.subsystem.api.storage.SecondaryStorageService;
+import org.apache.cloudstack.engine.subsystem.api.storage.SecondaryStorageService.DataObjectResult;
+import org.apache.cloudstack.engine.subsystem.api.storage.SnapshotDataFactory;
+import org.apache.cloudstack.engine.subsystem.api.storage.SnapshotInfo;
+import org.apache.cloudstack.engine.subsystem.api.storage.TemplateDataFactory;
+import org.apache.cloudstack.engine.subsystem.api.storage.VolumeDataFactory;
+import org.apache.cloudstack.framework.async.AsyncCallFuture;
+import org.apache.cloudstack.framework.config.ConfigKey;
+import org.apache.cloudstack.framework.config.Configurable;
+import org.apache.cloudstack.framework.config.dao.ConfigurationDao;
+import org.apache.cloudstack.framework.jobs.AsyncJobManager;
+import org.apache.cloudstack.storage.datastore.db.ImageStoreDao;
+import org.apache.cloudstack.storage.datastore.db.SnapshotDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.SnapshotDataStoreVO;
+import org.apache.cloudstack.storage.datastore.db.TemplateDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.TemplateDataStoreVO;
+import org.apache.cloudstack.storage.datastore.db.VolumeDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.VolumeDataStoreVO;
+import org.apache.commons.math3.stat.descriptive.moment.Mean;
+import org.apache.commons.math3.stat.descriptive.moment.StandardDeviation;
+import org.apache.log4j.Logger;
+
+import com.cloud.configuration.Config;
+import com.cloud.host.HostVO;
+import com.cloud.host.Status;
+import com.cloud.host.dao.HostDao;
+import com.cloud.server.StatsCollector;
+import com.cloud.storage.DataStoreRole;
+import com.cloud.storage.ImageStoreService.MigrationPolicy;
+import com.cloud.storage.SnapshotVO;
+import com.cloud.storage.StorageService;
+import com.cloud.storage.StorageStats;
+import com.cloud.storage.VMTemplateVO;
+import com.cloud.storage.dao.SnapshotDao;
+import com.cloud.storage.dao.VMTemplateDao;
+import com.cloud.utils.NumbersUtil;
+import com.cloud.utils.Pair;
+import com.cloud.utils.StringUtils;
+import com.cloud.utils.component.ManagerBase;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.vm.SecondaryStorageVm;
+import com.cloud.vm.SecondaryStorageVmVO;
+import com.cloud.vm.VirtualMachine;
+import com.cloud.vm.dao.SecondaryStorageVmDao;
+
+public class StorageOrchestrator extends ManagerBase implements StorageOrchestrationService, Configurable {
+
+ private static final Logger s_logger = Logger.getLogger(StorageOrchestrator.class);
+ @Inject
+ TemplateDataStoreDao templateDataStoreDao;
+ @Inject
+ SnapshotDataStoreDao snapshotDataStoreDao;
+ @Inject
+ VolumeDataStoreDao volumeDataStoreDao;
+ @Inject
+ VolumeDataFactory volumeFactory;
+ @Inject
+ VMTemplateDao templateDao;
+ @Inject
+ TemplateDataFactory templateFactory;
+ @Inject
+ SnapshotDao snapshotDao;
+ @Inject
+ SnapshotDataFactory snapshotFactory;
+ @Inject
+ DataStoreManager dataStoreManager;
+ @Inject
+ ImageStoreDao imageStoreDao;
+ @Inject
+ StatsCollector statsCollector;
+ @Inject
+ public StorageService storageService;
+ @Inject
+ SecondaryStorageVmDao secStorageVmDao;
+ @Inject
+ ConfigurationDao configDao;
+ @Inject
+ HostDao hostDao;
+ @Inject
+ private AsyncJobManager jobMgr;
+ @Inject
+ private SecondaryStorageService secStgSrv;
+
+ ConfigKey<Double> ImageStoreImbalanceThreshold = new ConfigKey<>("Advanced", Double.class,
+ "image.store.imbalance.threshold",
+ "0.1",
+ "The storage imbalance threshold that is compared with the standard deviation percentage for a storage utilization metric. " +
+ "The value is a percentage in decimal format.",
+ true, ConfigKey.Scope.Global);
+
+ Integer numConcurrentCopyTasksPerSSVM = 2;
+
+ private double imageStoreCapacityThreshold = 0.90;
+
+ @Override
+ public String getConfigComponentName() {
+ return StorageOrchestrationService.class.getName();
+ }
+
+ @Override
+ public ConfigKey<?>[] getConfigKeys() {
+ return new ConfigKey<?>[]{ImageStoreImbalanceThreshold};
+ }
+
+ static class MigrateBlockingQueue<T> extends ArrayBlockingQueue<T> {
+
+ MigrateBlockingQueue(int size) {
+ super(size);
+ }
+
+ public boolean offer(T task) {
+ try {
+ this.put(task);
+ } catch (InterruptedException e) {
+ throw new RuntimeException(e);
+ }
+ return true;
+ }
+ }
+
+ @Override
+ public boolean configure(String name, Map<String, Object> params) throws ConfigurationException {
+ numConcurrentCopyTasksPerSSVM = NumbersUtil.parseInt(configDao.getValue(Config.SecStorageMaxMigrateSessions.key()), 2);
+ return true;
+ }
+
+ @Override
+ public MigrationResponse migrateData(Long srcDataStoreId, List<Long> destDatastores, MigrationPolicy migrationPolicy) {
+ List<DataObject> files = new LinkedList<>();
+ int successCount = 0;
+ boolean success = true;
+ String message = null;
+
+ if (migrationPolicy == MigrationPolicy.COMPLETE) {
+ if (!filesReady(srcDataStoreId)) {
+ throw new CloudRuntimeException("Complete migration failed as there are data objects which are not Ready");
+ }
+ }
+
+ DataStore srcDatastore = dataStoreManager.getDataStore(srcDataStoreId, DataStoreRole.Image);
+ Map<DataObject, Pair<List<SnapshotInfo>, Long>> snapshotChains = new HashMap<>();
+ files.addAll(getAllValidTemplates(srcDatastore));
+ files.addAll(getAllValidSnapshotChains(srcDatastore, snapshotChains));
+ files.addAll(getAllValidVolumes(srcDatastore));
+
+ Collections.sort(files, new Comparator<DataObject>() {
+ @Override
+ public int compare(DataObject o1, DataObject o2) {
+ Long size1 = o1.getSize();
+ Long size2 = o2.getSize();
+ if (o1 instanceof SnapshotInfo) {
+ size1 = snapshotChains.get(o1).second();
+ }
+ if (o2 instanceof SnapshotInfo) {
+ size2 = snapshotChains.get(o2).second();
+ }
+ //return o2.getSize() > o1.getSize() ? 1 : -1;
+ return size2 > size1 ? 1 : -1;
+ }
+ });
Review comment:
done
##########
File path: engine/orchestration/src/main/java/org/apache/cloudstack/engine/orchestration/StorageOrchestrator.java
##########
@@ -0,0 +1,629 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.cloudstack.engine.orchestration;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.Hashtable;
+import java.util.LinkedHashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+import javax.inject.Inject;
+import javax.naming.ConfigurationException;
+
+import org.apache.cloudstack.api.response.MigrationResponse;
+import org.apache.cloudstack.engine.orchestration.service.StorageOrchestrationService;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataObject;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStoreManager;
+import org.apache.cloudstack.engine.subsystem.api.storage.ObjectInDataStoreStateMachine;
+import org.apache.cloudstack.engine.subsystem.api.storage.SecondaryStorageService;
+import org.apache.cloudstack.engine.subsystem.api.storage.SecondaryStorageService.DataObjectResult;
+import org.apache.cloudstack.engine.subsystem.api.storage.SnapshotDataFactory;
+import org.apache.cloudstack.engine.subsystem.api.storage.SnapshotInfo;
+import org.apache.cloudstack.engine.subsystem.api.storage.TemplateDataFactory;
+import org.apache.cloudstack.engine.subsystem.api.storage.VolumeDataFactory;
+import org.apache.cloudstack.framework.async.AsyncCallFuture;
+import org.apache.cloudstack.framework.config.ConfigKey;
+import org.apache.cloudstack.framework.config.Configurable;
+import org.apache.cloudstack.framework.config.dao.ConfigurationDao;
+import org.apache.cloudstack.framework.jobs.AsyncJobManager;
+import org.apache.cloudstack.storage.datastore.db.ImageStoreDao;
+import org.apache.cloudstack.storage.datastore.db.SnapshotDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.SnapshotDataStoreVO;
+import org.apache.cloudstack.storage.datastore.db.TemplateDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.TemplateDataStoreVO;
+import org.apache.cloudstack.storage.datastore.db.VolumeDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.VolumeDataStoreVO;
+import org.apache.commons.math3.stat.descriptive.moment.Mean;
+import org.apache.commons.math3.stat.descriptive.moment.StandardDeviation;
+import org.apache.log4j.Logger;
+
+import com.cloud.configuration.Config;
+import com.cloud.host.HostVO;
+import com.cloud.host.Status;
+import com.cloud.host.dao.HostDao;
+import com.cloud.server.StatsCollector;
+import com.cloud.storage.DataStoreRole;
+import com.cloud.storage.ImageStoreService.MigrationPolicy;
+import com.cloud.storage.SnapshotVO;
+import com.cloud.storage.StorageService;
+import com.cloud.storage.StorageStats;
+import com.cloud.storage.VMTemplateVO;
+import com.cloud.storage.dao.SnapshotDao;
+import com.cloud.storage.dao.VMTemplateDao;
+import com.cloud.utils.NumbersUtil;
+import com.cloud.utils.Pair;
+import com.cloud.utils.StringUtils;
+import com.cloud.utils.component.ManagerBase;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.vm.SecondaryStorageVm;
+import com.cloud.vm.SecondaryStorageVmVO;
+import com.cloud.vm.VirtualMachine;
+import com.cloud.vm.dao.SecondaryStorageVmDao;
+
+public class StorageOrchestrator extends ManagerBase implements StorageOrchestrationService, Configurable {
+
+ private static final Logger s_logger = Logger.getLogger(StorageOrchestrator.class);
+ @Inject
+ TemplateDataStoreDao templateDataStoreDao;
+ @Inject
+ SnapshotDataStoreDao snapshotDataStoreDao;
+ @Inject
+ VolumeDataStoreDao volumeDataStoreDao;
+ @Inject
+ VolumeDataFactory volumeFactory;
+ @Inject
+ VMTemplateDao templateDao;
+ @Inject
+ TemplateDataFactory templateFactory;
+ @Inject
+ SnapshotDao snapshotDao;
+ @Inject
+ SnapshotDataFactory snapshotFactory;
+ @Inject
+ DataStoreManager dataStoreManager;
+ @Inject
+ ImageStoreDao imageStoreDao;
+ @Inject
+ StatsCollector statsCollector;
+ @Inject
+ public StorageService storageService;
+ @Inject
+ SecondaryStorageVmDao secStorageVmDao;
+ @Inject
+ ConfigurationDao configDao;
+ @Inject
+ HostDao hostDao;
+ @Inject
+ private AsyncJobManager jobMgr;
+ @Inject
+ private SecondaryStorageService secStgSrv;
+
+ ConfigKey<Double> ImageStoreImbalanceThreshold = new ConfigKey<>("Advanced", Double.class,
+ "image.store.imbalance.threshold",
+ "0.1",
+ "The storage imbalance threshold that is compared with the standard deviation percentage for a storage utilization metric. " +
+ "The value is a percentage in decimal format.",
+ true, ConfigKey.Scope.Global);
+
+ Integer numConcurrentCopyTasksPerSSVM = 2;
+
+ private double imageStoreCapacityThreshold = 0.90;
+
+ @Override
+ public String getConfigComponentName() {
+ return StorageOrchestrationService.class.getName();
+ }
+
+ @Override
+ public ConfigKey<?>[] getConfigKeys() {
+ return new ConfigKey<?>[]{ImageStoreImbalanceThreshold};
+ }
+
+ static class MigrateBlockingQueue<T> extends ArrayBlockingQueue<T> {
+
+ MigrateBlockingQueue(int size) {
+ super(size);
+ }
+
+ public boolean offer(T task) {
+ try {
+ this.put(task);
+ } catch (InterruptedException e) {
+ throw new RuntimeException(e);
+ }
+ return true;
+ }
+ }
+
+ @Override
+ public boolean configure(String name, Map<String, Object> params) throws ConfigurationException {
+ numConcurrentCopyTasksPerSSVM = NumbersUtil.parseInt(configDao.getValue(Config.SecStorageMaxMigrateSessions.key()), 2);
+ return true;
+ }
+
+ @Override
+ public MigrationResponse migrateData(Long srcDataStoreId, List<Long> destDatastores, MigrationPolicy migrationPolicy) {
+ List<DataObject> files = new LinkedList<>();
+ int successCount = 0;
+ boolean success = true;
+ String message = null;
+
+ if (migrationPolicy == MigrationPolicy.COMPLETE) {
+ if (!filesReady(srcDataStoreId)) {
+ throw new CloudRuntimeException("Complete migration failed as there are data objects which are not Ready");
+ }
+ }
+
+ DataStore srcDatastore = dataStoreManager.getDataStore(srcDataStoreId, DataStoreRole.Image);
+ Map<DataObject, Pair<List<SnapshotInfo>, Long>> snapshotChains = new HashMap<>();
+ files.addAll(getAllValidTemplates(srcDatastore));
+ files.addAll(getAllValidSnapshotChains(srcDatastore, snapshotChains));
+ files.addAll(getAllValidVolumes(srcDatastore));
+
+ Collections.sort(files, new Comparator<DataObject>() {
+ @Override
+ public int compare(DataObject o1, DataObject o2) {
+ Long size1 = o1.getSize();
+ Long size2 = o2.getSize();
+ if (o1 instanceof SnapshotInfo) {
+ size1 = snapshotChains.get(o1).second();
+ }
+ if (o2 instanceof SnapshotInfo) {
+ size2 = snapshotChains.get(o2).second();
+ }
+ //return o2.getSize() > o1.getSize() ? 1 : -1;
+ return size2 > size1 ? 1 : -1;
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-668401889
@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-667548323
Packaging result: ✔centos7 ✔debian. JID-1632
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-663412459
@Pearl1594 a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-649104295
<b>Trillian test result (tid-1836)</b>
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 28843 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4053-t1836-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_direct_download.py
Intermittent failure detected: /marvin/tests/smoke/test_list_ids_parameter.py
Intermittent failure detected: /marvin/tests/smoke/test_nic.py
Intermittent failure detected: /marvin/tests/smoke/test_templates.py
Intermittent failure detected: /marvin/tests/smoke/test_usage.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_router_nics.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_vpn.py
Intermittent failure detected: /marvin/tests/smoke/test_hostha_kvm.py
Smoke tests completed. 74 look OK, 9 have error(s)
Only failed tests results shown below:
Test | Result | Time (s) | Test File
--- | --- | --- | ---
ContextSuite context=TestDirectDownloadTemplates>:setup | `Error` | 0.00 | test_direct_download.py
ContextSuite context=TestListIdsParams>:setup | `Error` | 0.00 | test_list_ids_parameter.py
test_01_nic | `Error` | 54.26 | test_nic.py
test_01_create_template | `Error` | 28.78 | test_templates.py
test_CreateTemplateWithDuplicateName | `Error` | 29.89 | test_templates.py
test_01_register_template_direct_download_flag | `Error` | 0.12 | test_templates.py
test_02_deploy_vm_from_direct_download_template | `Error` | 0.00 | test_templates.py
test_03_deploy_vm_wrong_checksum | `Error` | 0.10 | test_templates.py
ContextSuite context=TestTemplates>:setup | `Error` | 294.47 | test_templates.py
test_01_template_usage | `Error` | 31.13 | test_usage.py
ContextSuite context=TestVPCRedundancy>:setup | `Error` | 0.00 | test_vpc_redundant.py
ContextSuite context=TestVPCNics>:setup | `Error` | 0.00 | test_vpc_router_nics.py
ContextSuite context=TestRVPCSite2SiteVpn>:setup | `Error` | 0.00 | test_vpc_vpn.py
ContextSuite context=TestVPCSite2SiteVPNMultipleOptions>:setup | `Error` | 0.00 | test_vpc_vpn.py
ContextSuite context=TestVpcRemoteAccessVpn>:setup | `Error` | 0.00 | test_vpc_vpn.py
ContextSuite context=TestVpcSite2SiteVpn>:setup | `Error` | 0.00 | test_vpc_vpn.py
test_disable_oobm_ha_state_ineligible | `Error` | 0.28 | test_hostha_kvm.py
test_hostha_configure_default_driver | `Error` | 0.28 | test_hostha_kvm.py
test_hostha_enable_ha_when_host_disabled | `Error` | 0.28 | test_hostha_kvm.py
test_hostha_enable_ha_when_host_disconected | `Error` | 0.24 | test_hostha_kvm.py
test_hostha_enable_ha_when_host_in_maintenance | `Error` | 0.25 | test_hostha_kvm.py
test_hostha_kvm_host_degraded | `Error` | 0.23 | test_hostha_kvm.py
test_hostha_kvm_host_fencing | `Error` | 0.22 | test_hostha_kvm.py
test_hostha_kvm_host_recovering | `Error` | 0.25 | test_hostha_kvm.py
test_remove_ha_provider_not_possible | `Error` | 0.25 | test_hostha_kvm.py
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-682619315
Packaging result: ✔centos7 ✔centos8 ✔debian. JID-1848
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-671797378
@Pearl1594 a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-674476332
Packaging result: ✔centos7 ✔debian. JID-1763
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-648218518
@Pearl1594 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-689890280
@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-673464023
@blueorangutan package
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-649997581
@Pearl1594 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
rhtyd commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-691447968
@blueorangutan package
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-652758815
@Pearl1594 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-690968607
@blueorangutan package
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-674474689
@blueorangutan package
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-689889238
Packaging result: ✔centos7 ✖centos8 ✖debian. JID-1943
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-675914603
@blueorangutan package
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
rhtyd commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-691018623
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-687655568
Packaging result: ✖centos7 ✖centos8 ✖debian. JID-1902
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] DaanHoogland commented on a change in pull request #4053: Secondary Storage Management
Posted by GitBox <gi...@apache.org>.
DaanHoogland commented on a change in pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#discussion_r422003399
##########
File path: engine/storage/src/main/java/org/apache/cloudstack/storage/image/BaseImageStoreDriverImpl.java
##########
@@ -373,32 +375,26 @@ public void copyAsync(DataObject srcdata, DataObject destData, AsyncCompletionCa
s_logger.error(errMsg);
answer = new Answer(cmd, false, errMsg);
} else {
- boolean sent = false;
- // Find the first endpoint to which the command can be sent to
- for (EndPoint ep : eps) {
- if (getCopyCmdsCountToSpecificSSVM(ep.getId()) >= maxConcurrentCopyOpsPerSSVM) {
- continue;
+ // select endpoint with least number of commands running on them
+ EndPoint endPoint = null;
+ Long epId = ssvmWithLeastMigrateJobs();
+ if (epId == null) {
+ Collections.shuffle(eps);
+ endPoint = eps.get(0);
+ } else {
+ List<EndPoint> remainingEps = eps.stream().filter(ep -> ep.getId() != epId ).collect(Collectors.toList());
+ if (!remainingEps.isEmpty()) {
+ Collections.shuffle(remainingEps);
+ endPoint = remainingEps.get(0);
+ } else {
+ endPoint = _defaultEpSelector.getEndPointFromHostId(epId);
}
-
- CommandExecLogVO execLog = new CommandExecLogVO(ep.getId(), _secStorageVmDao.findByInstanceName(hostDao.findById(ep.getId()).getName()).getId(), cmd.getClass().getSimpleName(), 1);
- Long cmdExecId = _cmdExecLogDao.persist(execLog).getId();
- answer = ep.sendMessage(cmd);
- answer.setContextParam("cmd", cmdExecId.toString());
- sent = true;
- break;
- }
- // If both SSVMs are pre-occupied with tasks, choose the SSVM with least migrate jobs
- if (!sent) {
- // Picking endpoint with least number of copy commands running on it
- Long epId = ssvmWithLeastMigrateJobs();
- EndPoint endPoint = _defaultEpSelector.getEndPointFromHostId(epId);
- CommandExecLogVO execLog = new CommandExecLogVO(epId, _secStorageVmDao.findByInstanceName(hostDao.findById(epId).getName()).getId(), cmd.getClass().getSimpleName(), 1);
- Long cmdExecId = _cmdExecLogDao.persist(execLog).getId();
- answer = endPoint.sendMessage(cmd);
- answer.setContextParam("cmd", cmdExecId.toString());
}
+ CommandExecLogVO execLog = new CommandExecLogVO(endPoint.getId(), _secStorageVmDao.findByInstanceName(hostDao.findById(endPoint.getId()).getName()).getId(), cmd.getClass().getSimpleName(), 1);
+ Long cmdExecId = _cmdExecLogDao.persist(execLog).getId();
+ answer = endPoint.sendMessage(cmd);
+ answer.setContextParam("cmd", cmdExecId.toString());
Review comment:
can you extract this bit like `sendToLeastBusyEndPoint()`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-675253790
<b>Trillian test result (tid-2481)</b>
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 27899 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4053-t2481-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_clusters.py
Intermittent failure detected: /marvin/tests/smoke/test_volumes.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_router_nics.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_vpn.py
Intermittent failure detected: /marvin/tests/smoke/test_host_maintenance.py
Smoke tests completed. 79 look OK, 6 have error(s)
Only failed tests results shown below:
Test | Result | Time (s) | Test File
--- | --- | --- | ---
test_01_deploy_kubernetes_cluster | `Error` | 0.06 | test_kubernetes_clusters.py
test_02_deploy_kubernetes_ha_cluster | `Error` | 0.04 | test_kubernetes_clusters.py
test_04_deploy_and_upgrade_kubernetes_cluster | `Error` | 0.03 | test_kubernetes_clusters.py
test_05_deploy_and_upgrade_kubernetes_ha_cluster | `Error` | 0.03 | test_kubernetes_clusters.py
test_06_deploy_and_invalid_upgrade_kubernetes_cluster | `Error` | 0.03 | test_kubernetes_clusters.py
test_07_deploy_and_scale_kubernetes_cluster | `Error` | 0.03 | test_kubernetes_clusters.py
ContextSuite context=TestCreateVolume>:setup | `Error` | 0.00 | test_volumes.py
ContextSuite context=TestVolumes>:setup | `Error` | 0.00 | test_volumes.py
ContextSuite context=TestVPCRedundancy>:setup | `Error` | 0.00 | test_vpc_redundant.py
ContextSuite context=TestVPCNics>:setup | `Error` | 0.00 | test_vpc_router_nics.py
ContextSuite context=TestRVPCSite2SiteVpn>:setup | `Error` | 0.00 | test_vpc_vpn.py
ContextSuite context=TestVPCSite2SiteVPNMultipleOptions>:setup | `Error` | 0.00 | test_vpc_vpn.py
ContextSuite context=TestVpcRemoteAccessVpn>:setup | `Error` | 0.00 | test_vpc_vpn.py
ContextSuite context=TestVpcSite2SiteVpn>:setup | `Error` | 0.00 | test_vpc_vpn.py
test_01_cancel_host_maintenace_with_no_migration_jobs | `Error` | 0.04 | test_host_maintenance.py
test_02_cancel_host_maintenace_with_migration_jobs | `Error` | 0.03 | test_host_maintenance.py
test_03_cancel_host_maintenace_with_migration_jobs_failure | `Error` | 0.03 | test_host_maintenance.py
test_01_cancel_host_maintenance_ssh_enabled_agent_connected | `Error` | 0.01 | test_host_maintenance.py
test_03_cancel_host_maintenance_ssh_disabled_agent_connected | `Error` | 0.01 | test_host_maintenance.py
test_04_cancel_host_maintenance_ssh_disabled_agent_disconnected | `Error` | 0.01 | test_host_maintenance.py
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
rhtyd commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-672706304
@blueorangutan package
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-672781407
@Pearl1594 a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-688512426
<b>Trillian test result (tid-2691)</b>
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 44869 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4053-t2691-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_clusters.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermittent failure detected: /marvin/tests/smoke/test_hostha_kvm.py
Smoke tests completed. 82 look OK, 3 have error(s)
Only failed tests results shown below:
Test | Result | Time (s) | Test File
--- | --- | --- | ---
test_01_deploy_kubernetes_cluster | `Error` | 0.08 | test_kubernetes_clusters.py
test_02_deploy_kubernetes_ha_cluster | `Error` | 0.04 | test_kubernetes_clusters.py
test_04_deploy_and_upgrade_kubernetes_cluster | `Error` | 0.03 | test_kubernetes_clusters.py
test_05_deploy_and_upgrade_kubernetes_ha_cluster | `Error` | 0.04 | test_kubernetes_clusters.py
test_06_deploy_and_invalid_upgrade_kubernetes_cluster | `Error` | 0.04 | test_kubernetes_clusters.py
test_07_deploy_and_scale_kubernetes_cluster | `Error` | 0.04 | test_kubernetes_clusters.py
test_04_rvpc_network_garbage_collector_nics | `Error` | 3893.80 | test_vpc_redundant.py
test_hostha_enable_ha_when_host_disabled | `Error` | 1.56 | test_hostha_kvm.py
test_hostha_enable_ha_when_host_in_maintenance | `Error` | 301.67 | test_hostha_kvm.py
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
rhtyd commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-691018623
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-691655324
@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-688716411
<b>Trillian test result (tid-2690)</b>
Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server 7
Total time taken: 89670 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4053-t2690-vmware-67u3.zip
Intermittent failure detected: /marvin/tests/smoke/test_deploy_vgpu_enabled_vm.py
Intermittent failure detected: /marvin/tests/smoke/test_deploy_vm_root_resize.py
Intermittent failure detected: /marvin/tests/smoke/test_ssvm.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Smoke tests completed. 77 look OK, 3 have error(s)
Only failed tests results shown below:
Test | Result | Time (s) | Test File
--- | --- | --- | ---
test_3d_gpu_support | `Error` | 2077.22 | test_deploy_vgpu_enabled_vm.py
test_00_deploy_vm_root_resize | `Failure` | 803.09 | test_deploy_vm_root_resize.py
test_07_reboot_ssvm | `Failure` | 63.15 | test_ssvm.py
test_08_reboot_cpvm | `Failure` | 137.85 | test_ssvm.py
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-657965741
@blueorangutan test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] vladimirpetrov commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
vladimirpetrov commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-674925550
@blueorangutan package
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] DaanHoogland commented on a change in pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
DaanHoogland commented on a change in pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#discussion_r429326862
##########
File path: services/secondary-storage/server/src/main/java/org/apache/cloudstack/storage/resource/NfsSecondaryStorageResource.java
##########
@@ -1347,7 +1351,11 @@ protected Answer copyFromNfsToNfs(CopyCommand cmd) {
retObj = newVol;
} else if (destData.getObjectType() == DataObjectType.SNAPSHOT) {
SnapshotObjectTO newSnapshot = new SnapshotObjectTO();
- newSnapshot.setPath(destData.getPath() + File.separator + destFile.getName());
+ if (srcFile.isFile()) {
+ newSnapshot.setPath(destData.getPath() + File.separator + destFile.getName());
+ } else {
+ newSnapshot.setPath(destData.getPath() + File.separator + destFile.getName() + File.separator + destFile.getName());
Review comment:
so `./dir/file/file` without extra `.extension`!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-644575290
Packaging result: ✔centos7 ✔debian. JID-1391
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-693205698
@Pearl1594 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] harikrishna-patnala commented on a change in pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
harikrishna-patnala commented on a change in pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#discussion_r490066335
##########
File path: engine/schema/src/main/resources/META-INF/db/schema-41310to41400.sql
##########
@@ -53,6 +53,33 @@ ALTER TABLE `cloud`.`vm_instance` ADD COLUMN `backup_offering_id` bigint unsigne
ALTER TABLE `cloud`.`vm_instance` ADD COLUMN `backup_external_id` varchar(255) DEFAULT NULL COMMENT 'ID of external backup job or container if any';
ALTER TABLE `cloud`.`vm_instance` ADD COLUMN `backup_volumes` text DEFAULT NULL COMMENT 'details of backedup volumes';
Review comment:
This schema change has to be moved to schema-41400to41500.sql.
@Pearl1594 please raise a separate PR as this is already merged.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-691394188
@blueorangutan package
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-684563132
@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd merged pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
rhtyd merged pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-693251497
@rhtyd post this round of smoke test, the PR will be ready for merging
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan removed a comment on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan removed a comment on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-623262952
Packaging result: ✔centos7 ✔debian. JID-1207
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-691394321
@Pearl1594 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] vladimirpetrov commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
vladimirpetrov commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-638078582
@blueorangutan package
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-668414903
Packaging result: ✔centos7 ✔debian. JID-1637
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] vladimirpetrov commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
vladimirpetrov commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-675385745
@blueorangutan test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-691967791
@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-675406326
@vladimirpetrov a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-691849370
@blueorangutan package
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-691937319
Packaging result: ✔centos7 ✔centos8 ✔debian. JID-1993
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on a change in pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on a change in pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#discussion_r426727506
##########
File path: server/src/main/java/com/cloud/storage/secondary/SecondaryStorageVmManager.java
##########
@@ -31,6 +32,7 @@
public static final int DEFAULT_SS_VM_CPUMHZ = 500; // 500 MHz
public static final int DEFAULT_SS_VM_MTUSIZE = 1500;
public static final int DEFAULT_SS_VM_CAPACITY = 50; // max command execution session per SSVM
+ public static final int DEFAULT_MIGRATE_SS_VM_CAPACITY = 2; // number of concurrent migrate operations to happen per SSVM
Review comment:
It is overridden elsewhere
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-649997185
@blueorangutan package
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
rhtyd commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-648832961
@blueorangutan test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-663411964
@blueorangutan test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-673940571
@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-645928276
@Pearl1594 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-648833273
@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-675953677
@blueorangutan test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
rhtyd commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-673392721
@Pearl1594 can you fix conflicts? thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-674675618
Packaging result: ✔centos7 ✔debian. JID-1765
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-645942052
Packaging result: ✔centos7 ✔debian. JID-1418
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-642583562
@Pearl1594 a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-645766709
@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-682422740
@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-651702384
Packaging result: ✔centos7 ✔debian. JID-1489
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-657963647
Packaging result: ✔centos7 ✔debian. JID-1572
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] DaanHoogland commented on a change in pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
DaanHoogland commented on a change in pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#discussion_r427804902
##########
File path: engine/storage/src/main/java/org/apache/cloudstack/storage/datastore/ObjectInDataStoreManagerImpl.java
##########
@@ -104,6 +104,14 @@ public ObjectInDataStoreManagerImpl() {
// TODO: further investigate why an extra event is sent when it is
// alreay Ready for DownloadListener
stateMachines.addTransition(State.Ready, Event.OperationSuccessed, State.Ready);
+ // State transitions for data object migration
+ stateMachines.addTransition(State.Ready, Event.MigrationRequested, State.Migrating);
+ stateMachines.addTransition(State.Ready, Event.CopyRequested, State.Copying);
+ stateMachines.addTransition(State.Allocated, Event.MigrationRequested, State.Migrating);
+ stateMachines.addTransition(State.Migrating, Event.MigrationFailed, State.Failed);
+ stateMachines.addTransition(State.Migrating, Event.MigrationSucceeded, State.Destroyed);
+ stateMachines.addTransition(State.Migrating, Event.OperationSuccessed, State.Ready);
Review comment:
yes I saw, maybe separate small issue, thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-682832725
@Pearl1594 a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
rhtyd commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-638561264
@blueorangutan test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-652016745
<b>Trillian test result (tid-1908)</b>
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 33671 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4053-t1908-kvm-centos7.zip
Smoke tests completed. 83 look OK, 0 have error(s)
Only failed tests results shown below:
Test | Result | Time (s) | Test File
--- | --- | --- | ---
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-642546077
@blueorangutan package
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-673510612
@Pearl1594 a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-688135004
@Pearl1594 a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on a change in pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on a change in pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#discussion_r422977943
##########
File path: api/src/main/java/com/cloud/storage/ImageStoreService.java
##########
@@ -0,0 +1,29 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package com.cloud.storage;
Review comment:
moved the file to appropriate pkg
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-642546339
@Pearl1594 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] DaanHoogland commented on a change in pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
DaanHoogland commented on a change in pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#discussion_r429326862
##########
File path: services/secondary-storage/server/src/main/java/org/apache/cloudstack/storage/resource/NfsSecondaryStorageResource.java
##########
@@ -1347,7 +1351,11 @@ protected Answer copyFromNfsToNfs(CopyCommand cmd) {
retObj = newVol;
} else if (destData.getObjectType() == DataObjectType.SNAPSHOT) {
SnapshotObjectTO newSnapshot = new SnapshotObjectTO();
- newSnapshot.setPath(destData.getPath() + File.separator + destFile.getName());
+ if (srcFile.isFile()) {
+ newSnapshot.setPath(destData.getPath() + File.separator + destFile.getName());
+ } else {
+ newSnapshot.setPath(destData.getPath() + File.separator + destFile.getName() + File.separator + destFile.getName());
Review comment:
so `./dir/file/file` witout extra `.extension`!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
rhtyd commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-671209518
@blueorangutan package
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-683224824
@blueorangutan test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-693219213
Packaging result: ✔centos7 ✔centos8 ✔debian. JID-2016
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
rhtyd commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-693232958
@Pearl1594 pl advise if/when this is ready for merge, thnx
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-652769255
@blueorangutan test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-688065920
@blueorangutan test matrix
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-690968607
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-675386377
@vladimirpetrov a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] DaanHoogland commented on a change in pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
DaanHoogland commented on a change in pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#discussion_r427257039
##########
File path: engine/orchestration/src/main/java/org/apache/cloudstack/engine/orchestration/DataMigrationUtility.java
##########
@@ -0,0 +1,262 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.cloudstack.engine.orchestration;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+
+import javax.inject.Inject;
+
+import org.apache.cloudstack.engine.subsystem.api.storage.DataObject;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.engine.subsystem.api.storage.ObjectInDataStoreStateMachine;
+import org.apache.cloudstack.engine.subsystem.api.storage.SnapshotDataFactory;
+import org.apache.cloudstack.engine.subsystem.api.storage.SnapshotInfo;
+import org.apache.cloudstack.engine.subsystem.api.storage.TemplateDataFactory;
+import org.apache.cloudstack.engine.subsystem.api.storage.VolumeDataFactory;
+import org.apache.cloudstack.storage.ImageStoreService;
+import org.apache.cloudstack.storage.datastore.db.SnapshotDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.SnapshotDataStoreVO;
+import org.apache.cloudstack.storage.datastore.db.TemplateDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.TemplateDataStoreVO;
+import org.apache.cloudstack.storage.datastore.db.VolumeDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.VolumeDataStoreVO;
+import org.apache.log4j.Logger;
+
+import com.cloud.host.HostVO;
+import com.cloud.host.Status;
+import com.cloud.host.dao.HostDao;
+import com.cloud.storage.DataStoreRole;
+import com.cloud.storage.SnapshotVO;
+import com.cloud.storage.VMTemplateVO;
+import com.cloud.storage.dao.SnapshotDao;
+import com.cloud.storage.dao.VMTemplateDao;
+import com.cloud.utils.Pair;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.vm.SecondaryStorageVm;
+import com.cloud.vm.SecondaryStorageVmVO;
+import com.cloud.vm.VirtualMachine;
+import com.cloud.vm.dao.SecondaryStorageVmDao;
+
+public class DataMigrationUtility {
+ @Inject
+ SecondaryStorageVmDao secStorageVmDao;
+ @Inject
+ TemplateDataStoreDao templateDataStoreDao;
+ @Inject
+ SnapshotDataStoreDao snapshotDataStoreDao;
+ @Inject
+ VolumeDataStoreDao volumeDataStoreDao;
+ @Inject
+ VMTemplateDao templateDao;
+ @Inject
+ VolumeDataFactory volumeFactory;
+ @Inject
+ TemplateDataFactory templateFactory;
+ @Inject
+ SnapshotDataFactory snapshotFactory;
+ @Inject
+ HostDao hostDao;
+ @Inject
+ SnapshotDao snapshotDao;
+
+ private static final Logger s_logger = Logger.getLogger(DataMigrationUtility.class);
+
+ /** This function verifies if the given image store comprises of data objects that are not in either the "Ready" or
+ * "Allocated" state - in such a case, if the migration policy is complete, the migration is terminated
+ */
Review comment:
worth mentioning that here as well. also `validstates` contains more : `{"Ready", "Allocated", "Destroying", "Destroyed", "Failed"}`. seems worth noting in the comment.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-673510138
@blueorangutan test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-673479976
Packaging result: ✔centos7 ✔debian. JID-1742
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-645927785
@blueorangutan package
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-674313921
<b>Trillian test result (tid-2447)</b>
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 44398 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4053-t2447-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_human_readable_logs.py
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_supported_versions.py
Intermittent failure detected: /marvin/tests/smoke/test_outofbandmanagement.py
Intermittent failure detected: /marvin/tests/smoke/test_volumes.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermittent failure detected: /marvin/tests/smoke/test_hostha_kvm.py
Smoke tests completed. 77 look OK, 5 have error(s)
Only failed tests results shown below:
Test | Result | Time (s) | Test File
--- | --- | --- | ---
test_02_enableHumanReadableLogs | `Failure` | 64.08 | test_human_readable_logs.py
test_01_add_delete_kubernetes_supported_version | `Error` | 1808.23 | test_kubernetes_supported_versions.py
test_oobm_zchange_password | `Error` | 6.50 | test_outofbandmanagement.py
test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL | `Failure` | 415.97 | test_vpc_redundant.py
test_04_rvpc_network_garbage_collector_nics | `Error` | 3838.67 | test_vpc_redundant.py
test_hostha_kvm_host_fencing | `Error` | 172.02 | test_hostha_kvm.py
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-657419604
Packaging result: ✔centos7 ✔debian. JID-1570
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] DaanHoogland commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
DaanHoogland commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-632797030
> Yes @DaanHoogland - that's how it's stored in the DB (for vmware)
> Eg:
>
> ```
> install_path: snapshots/2/129/2f6db8ab-9cee-41ef-ac8d-7c9667364539/2f6db8ab-9cee-41ef-ac8d-7c9667364539
> ```
>
> where filename is the snapshot uuid
yes, makes sense but scares me ;)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 closed pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
Pearl1594 closed pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-691619614
@Pearl1594 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-638928520
<b>Trillian test result (tid-1603)</b>
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 43578 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4053-t1603-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_direct_download.py
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_clusters.py
Intermittent failure detected: /marvin/tests/smoke/test_list_ids_parameter.py
Intermittent failure detected: /marvin/tests/smoke/test_ssvm.py
Intermittent failure detected: /marvin/tests/smoke/test_templates.py
Intermittent failure detected: /marvin/tests/smoke/test_usage.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_router_nics.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_vpn.py
Intermittent failure detected: /marvin/tests/smoke/test_hostha_kvm.py
Smoke tests completed. 74 look OK, 9 have error(s)
Only failed tests results shown below:
Test | Result | Time (s) | Test File
--- | --- | --- | ---
ContextSuite context=TestDirectDownloadTemplates>:setup | `Error` | 0.00 | test_direct_download.py
test_04_deploy_and_upgrade_kubernetes_cluster | `Error` | 3617.02 | test_kubernetes_clusters.py
test_05_deploy_and_upgrade_kubernetes_ha_cluster | `Error` | 0.06 | test_kubernetes_clusters.py
test_07_deploy_and_scale_kubernetes_cluster | `Failure` | 1113.12 | test_kubernetes_clusters.py
ContextSuite context=TestKubernetesCluster>:teardown | `Error` | 1149.18 | test_kubernetes_clusters.py
ContextSuite context=TestListIdsParams>:setup | `Error` | 0.00 | test_list_ids_parameter.py
test_01_create_template | `Error` | 47.40 | test_templates.py
test_CreateTemplateWithDuplicateName | `Error` | 50.48 | test_templates.py
test_01_register_template_direct_download_flag | `Error` | 0.16 | test_templates.py
test_02_deploy_vm_from_direct_download_template | `Error` | 0.00 | test_templates.py
test_03_deploy_vm_wrong_checksum | `Error` | 0.13 | test_templates.py
ContextSuite context=TestTemplates>:setup | `Error` | 312.01 | test_templates.py
test_01_template_usage | `Error` | 59.12 | test_usage.py
ContextSuite context=TestVPCRedundancy>:setup | `Error` | 0.00 | test_vpc_redundant.py
ContextSuite context=TestVPCNics>:setup | `Error` | 0.00 | test_vpc_router_nics.py
ContextSuite context=TestRVPCSite2SiteVpn>:setup | `Error` | 0.00 | test_vpc_vpn.py
ContextSuite context=TestVPCSite2SiteVPNMultipleOptions>:setup | `Error` | 0.00 | test_vpc_vpn.py
ContextSuite context=TestVpcRemoteAccessVpn>:setup | `Error` | 0.00 | test_vpc_vpn.py
ContextSuite context=TestVpcSite2SiteVpn>:setup | `Error` | 0.00 | test_vpc_vpn.py
test_disable_oobm_ha_state_ineligible | `Error` | 0.27 | test_hostha_kvm.py
test_hostha_configure_default_driver | `Error` | 0.24 | test_hostha_kvm.py
test_hostha_enable_ha_when_host_disabled | `Error` | 0.26 | test_hostha_kvm.py
test_hostha_enable_ha_when_host_disconected | `Error` | 1.06 | test_hostha_kvm.py
test_hostha_enable_ha_when_host_in_maintenance | `Error` | 1.11 | test_hostha_kvm.py
test_hostha_kvm_host_degraded | `Error` | 1.09 | test_hostha_kvm.py
test_hostha_kvm_host_fencing | `Error` | 0.35 | test_hostha_kvm.py
test_hostha_kvm_host_recovering | `Error` | 0.34 | test_hostha_kvm.py
test_remove_ha_provider_not_possible | `Error` | 0.30 | test_hostha_kvm.py
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] DaanHoogland commented on a change in pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
DaanHoogland commented on a change in pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#discussion_r426669651
##########
File path: server/src/main/java/com/cloud/storage/secondary/SecondaryStorageVmManager.java
##########
@@ -31,6 +32,7 @@
public static final int DEFAULT_SS_VM_CPUMHZ = 500; // 500 MHz
public static final int DEFAULT_SS_VM_MTUSIZE = 1500;
public static final int DEFAULT_SS_VM_CAPACITY = 50; // max command execution session per SSVM
+ public static final int DEFAULT_MIGRATE_SS_VM_CAPACITY = 2; // number of concurrent migrate operations to happen per SSVM
Review comment:
is this being overridden by a global setting somewhere, or a hard-coded restiction
##########
File path: api/src/main/java/org/apache/cloudstack/api/command/admin/storage/MigrateSecondaryStorageDataCmd.java
##########
@@ -0,0 +1,115 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.cloudstack.api.command.admin.storage;
+
+import java.util.List;
+
+import org.apache.cloudstack.acl.RoleType;
+import org.apache.cloudstack.api.APICommand;
+import org.apache.cloudstack.api.ApiConstants;
+import org.apache.cloudstack.api.BaseAsyncCmd;
+import org.apache.cloudstack.api.Parameter;
+import org.apache.cloudstack.api.response.ImageStoreResponse;
+import org.apache.cloudstack.api.response.MigrationResponse;
+import org.apache.cloudstack.context.CallContext;
+import org.apache.log4j.Logger;
+
+import com.cloud.event.EventTypes;
+import com.cloud.utils.StringUtils;
+
+@APICommand(name = MigrateSecondaryStorageDataCmd.APINAME,
+ description = "migrates data objects from one secondary storage to destination image store(s)",
+ responseObject = MigrationResponse.class,
+ requestHasSensitiveInfo = false,
+ responseHasSensitiveInfo = false,
+ since = "4.14.0",
Review comment:
we wont make `"4.14.0"` ;)
##########
File path: api/src/main/java/org/apache/cloudstack/api/command/admin/storage/MigrateSecondaryStorageDataCmd.java
##########
@@ -0,0 +1,115 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.cloudstack.api.command.admin.storage;
+
+import java.util.List;
+
+import org.apache.cloudstack.acl.RoleType;
+import org.apache.cloudstack.api.APICommand;
+import org.apache.cloudstack.api.ApiConstants;
+import org.apache.cloudstack.api.BaseAsyncCmd;
+import org.apache.cloudstack.api.Parameter;
+import org.apache.cloudstack.api.response.ImageStoreResponse;
+import org.apache.cloudstack.api.response.MigrationResponse;
+import org.apache.cloudstack.context.CallContext;
+import org.apache.log4j.Logger;
+
+import com.cloud.event.EventTypes;
+import com.cloud.utils.StringUtils;
+
+@APICommand(name = MigrateSecondaryStorageDataCmd.APINAME,
+ description = "migrates data objects from one secondary storage to destination image store(s)",
+ responseObject = MigrationResponse.class,
+ requestHasSensitiveInfo = false,
+ responseHasSensitiveInfo = false,
+ since = "4.14.0",
+ authorized = {RoleType.Admin})
+public class MigrateSecondaryStorageDataCmd extends BaseAsyncCmd {
+
+ public static final Logger s_logger = Logger.getLogger(MigrateSecondaryStorageDataCmd.class.getName());
Review comment:
this naming convention is not advices for static final fields rather use `LOG` or `LOGGER`
##########
File path: api/src/main/java/org/apache/cloudstack/api/command/admin/storage/ListImageStoresCmd.java
##########
@@ -52,6 +51,9 @@
@Parameter(name = ApiConstants.ID, type = CommandType.UUID, entityType = ImageStoreResponse.class, description = "the ID of the storage pool")
private Long id;
+ @Parameter(name = ApiConstants.READ_ONLY, type = CommandType.BOOLEAN, entityType = ImageStoreResponse.class, description = "read-only status of the image store")
Review comment:
please add `since = "4.15"`
##########
File path: engine/orchestration/src/main/java/org/apache/cloudstack/engine/orchestration/DataMigrationUtility.java
##########
@@ -0,0 +1,262 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.cloudstack.engine.orchestration;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+
+import javax.inject.Inject;
+
+import org.apache.cloudstack.engine.subsystem.api.storage.DataObject;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.engine.subsystem.api.storage.ObjectInDataStoreStateMachine;
+import org.apache.cloudstack.engine.subsystem.api.storage.SnapshotDataFactory;
+import org.apache.cloudstack.engine.subsystem.api.storage.SnapshotInfo;
+import org.apache.cloudstack.engine.subsystem.api.storage.TemplateDataFactory;
+import org.apache.cloudstack.engine.subsystem.api.storage.VolumeDataFactory;
+import org.apache.cloudstack.storage.ImageStoreService;
+import org.apache.cloudstack.storage.datastore.db.SnapshotDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.SnapshotDataStoreVO;
+import org.apache.cloudstack.storage.datastore.db.TemplateDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.TemplateDataStoreVO;
+import org.apache.cloudstack.storage.datastore.db.VolumeDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.VolumeDataStoreVO;
+import org.apache.log4j.Logger;
+
+import com.cloud.host.HostVO;
+import com.cloud.host.Status;
+import com.cloud.host.dao.HostDao;
+import com.cloud.storage.DataStoreRole;
+import com.cloud.storage.SnapshotVO;
+import com.cloud.storage.VMTemplateVO;
+import com.cloud.storage.dao.SnapshotDao;
+import com.cloud.storage.dao.VMTemplateDao;
+import com.cloud.utils.Pair;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.vm.SecondaryStorageVm;
+import com.cloud.vm.SecondaryStorageVmVO;
+import com.cloud.vm.VirtualMachine;
+import com.cloud.vm.dao.SecondaryStorageVmDao;
+
+public class DataMigrationUtility {
+ @Inject
+ SecondaryStorageVmDao secStorageVmDao;
+ @Inject
+ TemplateDataStoreDao templateDataStoreDao;
+ @Inject
+ SnapshotDataStoreDao snapshotDataStoreDao;
+ @Inject
+ VolumeDataStoreDao volumeDataStoreDao;
+ @Inject
+ VMTemplateDao templateDao;
+ @Inject
+ VolumeDataFactory volumeFactory;
+ @Inject
+ TemplateDataFactory templateFactory;
+ @Inject
+ SnapshotDataFactory snapshotFactory;
+ @Inject
+ HostDao hostDao;
+ @Inject
+ SnapshotDao snapshotDao;
+
+ private static final Logger s_logger = Logger.getLogger(DataMigrationUtility.class);
+
+ /** This function verifies if the given image store comprises of data objects that are not in either the "Ready" or
+ * "Allocated" state - in such a case, if the migration policy is complete, the migration is terminated
+ */
Review comment:
So, if there is any file in any other state, the migration is not ready, is it?
##########
File path: engine/orchestration/src/main/java/org/apache/cloudstack/engine/orchestration/DataMigrationUtility.java
##########
@@ -0,0 +1,262 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.cloudstack.engine.orchestration;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+
+import javax.inject.Inject;
+
+import org.apache.cloudstack.engine.subsystem.api.storage.DataObject;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.engine.subsystem.api.storage.ObjectInDataStoreStateMachine;
+import org.apache.cloudstack.engine.subsystem.api.storage.SnapshotDataFactory;
+import org.apache.cloudstack.engine.subsystem.api.storage.SnapshotInfo;
+import org.apache.cloudstack.engine.subsystem.api.storage.TemplateDataFactory;
+import org.apache.cloudstack.engine.subsystem.api.storage.VolumeDataFactory;
+import org.apache.cloudstack.storage.ImageStoreService;
+import org.apache.cloudstack.storage.datastore.db.SnapshotDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.SnapshotDataStoreVO;
+import org.apache.cloudstack.storage.datastore.db.TemplateDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.TemplateDataStoreVO;
+import org.apache.cloudstack.storage.datastore.db.VolumeDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.VolumeDataStoreVO;
+import org.apache.log4j.Logger;
+
+import com.cloud.host.HostVO;
+import com.cloud.host.Status;
+import com.cloud.host.dao.HostDao;
+import com.cloud.storage.DataStoreRole;
+import com.cloud.storage.SnapshotVO;
+import com.cloud.storage.VMTemplateVO;
+import com.cloud.storage.dao.SnapshotDao;
+import com.cloud.storage.dao.VMTemplateDao;
+import com.cloud.utils.Pair;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.vm.SecondaryStorageVm;
+import com.cloud.vm.SecondaryStorageVmVO;
+import com.cloud.vm.VirtualMachine;
+import com.cloud.vm.dao.SecondaryStorageVmDao;
+
+public class DataMigrationUtility {
+ @Inject
+ SecondaryStorageVmDao secStorageVmDao;
+ @Inject
+ TemplateDataStoreDao templateDataStoreDao;
+ @Inject
+ SnapshotDataStoreDao snapshotDataStoreDao;
+ @Inject
+ VolumeDataStoreDao volumeDataStoreDao;
+ @Inject
+ VMTemplateDao templateDao;
+ @Inject
+ VolumeDataFactory volumeFactory;
+ @Inject
+ TemplateDataFactory templateFactory;
+ @Inject
+ SnapshotDataFactory snapshotFactory;
+ @Inject
+ HostDao hostDao;
+ @Inject
+ SnapshotDao snapshotDao;
+
+ private static final Logger s_logger = Logger.getLogger(DataMigrationUtility.class);
+
+ /** This function verifies if the given image store comprises of data objects that are not in either the "Ready" or
+ * "Allocated" state - in such a case, if the migration policy is complete, the migration is terminated
+ */
+ private boolean filesReady(Long srcDataStoreId) {
Review comment:
filesReady in this case means "files ready to migrate" does it? if so can we rename it that please?
##########
File path: services/secondary-storage/controller/src/main/java/org/apache/cloudstack/secondarystorage/PremiumSecondaryStorageManagerImpl.java
##########
@@ -135,13 +152,43 @@ public boolean configure(String name, Map<String, Object> params) throws Configu
}
alreadyRunning = _secStorageVmDao.getSecStorageVmListInStates(null, dataCenterId, State.Running, State.Migrating, State.Starting);
-
List<CommandExecLogVO> activeCmds = findActiveCommands(dataCenterId, cutTime);
+ // Find running copy / migrate commands running arranged in ascending order of their creation time i.e., oldest first
+ List<CommandExecLogVO> copyCmdsInPipeline = findAllActiveCopyCommands(dataCenterId, cutTime);
+ // Count of total hosts
+ Integer hostsCount = _hostDao.countAllByType(Host.Type.Routing);
+ // Maximum number of allowed SSVMs for migration task
+ Integer maxSsvms = (hostsCount < MaxNumberOfSsvmsForMigration.value()) ? hostsCount : MaxNumberOfSsvmsForMigration.value();
+ int halfLimit = Math.round((float) (alreadyRunning.size() * migrateCapPerSSVM) / 2);
+ currentTime = DateUtil.currentGMTTime().getTime();
+
Review comment:
please factor out in a suitably named method.
##########
File path: engine/orchestration/src/main/java/org/apache/cloudstack/engine/orchestration/DataMigrationUtility.java
##########
@@ -0,0 +1,262 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.cloudstack.engine.orchestration;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+
+import javax.inject.Inject;
+
+import org.apache.cloudstack.engine.subsystem.api.storage.DataObject;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.engine.subsystem.api.storage.ObjectInDataStoreStateMachine;
+import org.apache.cloudstack.engine.subsystem.api.storage.SnapshotDataFactory;
+import org.apache.cloudstack.engine.subsystem.api.storage.SnapshotInfo;
+import org.apache.cloudstack.engine.subsystem.api.storage.TemplateDataFactory;
+import org.apache.cloudstack.engine.subsystem.api.storage.VolumeDataFactory;
+import org.apache.cloudstack.storage.ImageStoreService;
+import org.apache.cloudstack.storage.datastore.db.SnapshotDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.SnapshotDataStoreVO;
+import org.apache.cloudstack.storage.datastore.db.TemplateDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.TemplateDataStoreVO;
+import org.apache.cloudstack.storage.datastore.db.VolumeDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.VolumeDataStoreVO;
+import org.apache.log4j.Logger;
+
+import com.cloud.host.HostVO;
+import com.cloud.host.Status;
+import com.cloud.host.dao.HostDao;
+import com.cloud.storage.DataStoreRole;
+import com.cloud.storage.SnapshotVO;
+import com.cloud.storage.VMTemplateVO;
+import com.cloud.storage.dao.SnapshotDao;
+import com.cloud.storage.dao.VMTemplateDao;
+import com.cloud.utils.Pair;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.vm.SecondaryStorageVm;
+import com.cloud.vm.SecondaryStorageVmVO;
+import com.cloud.vm.VirtualMachine;
+import com.cloud.vm.dao.SecondaryStorageVmDao;
+
+public class DataMigrationUtility {
+ @Inject
+ SecondaryStorageVmDao secStorageVmDao;
+ @Inject
+ TemplateDataStoreDao templateDataStoreDao;
+ @Inject
+ SnapshotDataStoreDao snapshotDataStoreDao;
+ @Inject
+ VolumeDataStoreDao volumeDataStoreDao;
+ @Inject
+ VMTemplateDao templateDao;
+ @Inject
+ VolumeDataFactory volumeFactory;
+ @Inject
+ TemplateDataFactory templateFactory;
+ @Inject
+ SnapshotDataFactory snapshotFactory;
+ @Inject
+ HostDao hostDao;
+ @Inject
+ SnapshotDao snapshotDao;
+
+ private static final Logger s_logger = Logger.getLogger(DataMigrationUtility.class);
+
+ /** This function verifies if the given image store comprises of data objects that are not in either the "Ready" or
+ * "Allocated" state - in such a case, if the migration policy is complete, the migration is terminated
+ */
+ private boolean filesReady(Long srcDataStoreId) {
+ String[] validStates = new String[]{"Ready", "Allocated"};
+ boolean isReady = true;
+ List<TemplateDataStoreVO> templates = templateDataStoreDao.listByStoreId(srcDataStoreId);
+ for (TemplateDataStoreVO template : templates) {
+ isReady &= (Arrays.asList(validStates).contains(template.getState().toString()));
+ }
+ List<SnapshotDataStoreVO> snapshots = snapshotDataStoreDao.listByStoreId(srcDataStoreId, DataStoreRole.Image);
+ for (SnapshotDataStoreVO snapshot : snapshots) {
+ isReady &= (Arrays.asList(validStates).contains(snapshot.getState().toString()));
+ }
+ List<VolumeDataStoreVO> volumes = volumeDataStoreDao.listByStoreId(srcDataStoreId);
+ for (VolumeDataStoreVO volume : volumes) {
+ isReady &= (Arrays.asList(validStates).contains(volume.getState().toString()));
+ }
+ return isReady;
+ }
+
+ protected void checkIfCompleteMigrationPossible(ImageStoreService.MigrationPolicy policy, Long srcDataStoreId) {
+ if (policy == ImageStoreService.MigrationPolicy.COMPLETE) {
+ if (!filesReady(srcDataStoreId)) {
+ throw new CloudRuntimeException("Complete migration failed as there are data objects which are not Ready");
+ }
+ }
+ return;
+ }
+
+ protected Long getFileSize(DataObject file, Map<DataObject, Pair<List<SnapshotInfo>, Long>> snapshotChain) {
+ Long size = file.getSize();
+ Pair<List<SnapshotInfo>, Long> chain = snapshotChain.get(file);
+ if (file instanceof SnapshotInfo && chain.first() != null) {
+ size = chain.second();
+ }
+ return size;
+ }
+
+ /**
+ * Sorts the datastores in decreasing order of their free capacities, so as to make
+ * an informed decision of picking the datastore with maximum free capactiy for migration
+ */
+ protected List<Long> sortDataStores(Map<Long, Pair<Long, Long>> storageCapacities) {
+ List<Map.Entry<Long, Pair<Long, Long>>> list =
+ new LinkedList<Map.Entry<Long, Pair<Long, Long>>>((storageCapacities.entrySet()));
+
+ Collections.sort(list, new Comparator<Map.Entry<Long, Pair<Long, Long>>>() {
+ @Override
+ public int compare(Map.Entry<Long, Pair<Long, Long>> e1, Map.Entry<Long, Pair<Long, Long>> e2) {
+ return e2.getValue().first() > e1.getValue().first() ? 1 : -1;
Review comment:
no 0 possible for equal?
##########
File path: services/secondary-storage/controller/src/main/java/org/apache/cloudstack/secondarystorage/PremiumSecondaryStorageManagerImpl.java
##########
@@ -135,13 +152,43 @@ public boolean configure(String name, Map<String, Object> params) throws Configu
}
alreadyRunning = _secStorageVmDao.getSecStorageVmListInStates(null, dataCenterId, State.Running, State.Migrating, State.Starting);
-
List<CommandExecLogVO> activeCmds = findActiveCommands(dataCenterId, cutTime);
+ // Find running copy / migrate commands running arranged in ascending order of their creation time i.e., oldest first
+ List<CommandExecLogVO> copyCmdsInPipeline = findAllActiveCopyCommands(dataCenterId, cutTime);
+ // Count of total hosts
+ Integer hostsCount = _hostDao.countAllByType(Host.Type.Routing);
+ // Maximum number of allowed SSVMs for migration task
+ Integer maxSsvms = (hostsCount < MaxNumberOfSsvmsForMigration.value()) ? hostsCount : MaxNumberOfSsvmsForMigration.value();
+ int halfLimit = Math.round((float) (alreadyRunning.size() * migrateCapPerSSVM) / 2);
+ currentTime = DateUtil.currentGMTTime().getTime();
+
if (alreadyRunning.size() * _capacityPerSSVM - activeCmds.size() < _standbyCapacity) {
s_logger.info("secondary storage command execution standby capactiy low (running VMs: " + alreadyRunning.size() + ", active cmds: " + activeCmds.size() +
"), starting a new one");
return new Pair<AfterScanAction, Object>(AfterScanAction.expand, SecondaryStorageVm.Role.commandExecutor);
}
+ // Scale the number of SSVMs if the number of Copy operations is greater than the number of SSVMs running and the copy operation has been in pipeline for
+ // more than half of the total time allocated for secondary storage operation
+ else if (!copyCmdsInPipeline.isEmpty() && copyCmdsInPipeline.size() >= halfLimit &&
+ (((currentTime - copyCmdsInPipeline.get(halfLimit - 1).getCreated().getTime()) > _maxExecutionTimeMs/2 )) &&
+ (currentTime > nextSpawnTime) && alreadyRunning.size() <= maxSsvms) {
+ nextSpawnTime = currentTime + _maxExecutionTimeMs/2;
+ s_logger.debug("scaling SSVM to handle migration tasks");
+ return new Pair<AfterScanAction, Object>(AfterScanAction.expand, SecondaryStorageVm.Role.templateProcessor);
+ }
+
+ // Scale down the number of SSVMs if the load on them has reduced
+ if ((copyCmdsInPipeline.size() < halfLimit && alreadyRunning.size() * _capacityPerSSVM - activeCmds.size() > _standbyCapacity) && alreadyRunning.size() > 1) {
+ Collections.reverse(alreadyRunning);
+ for(SecondaryStorageVmVO vm : alreadyRunning) {
+ long count = copyCmdsInPipeline.stream().map(cmd -> cmd.getInstanceId() == vm.getId()).count();
+ count += activeCmds.stream().map(cmd -> cmd.getInstanceId() == vm.getId()).count();
+ if (count == 0) {
+ destroySecStorageVm(vm.getId());
+ break;
+ }
+ }
+ }
Review comment:
please factor this out in suitably called methods. This method was already too long to begin with, but let's not add to the lack of maintainability.
##########
File path: server/src/main/java/com/cloud/api/query/ViewResponseHelper.java
##########
@@ -84,8 +82,10 @@
import com.cloud.api.query.vo.UserAccountJoinVO;
import com.cloud.api.query.vo.UserVmJoinVO;
import com.cloud.api.query.vo.VolumeJoinVO;
-import com.cloud.storage.StoragePoolTagVO;
+import com.cloud.configuration.Resource;
+import com.cloud.domain.Domain;
import com.cloud.storage.Storage.ImageFormat;
+import com.cloud.storage.StoragePoolTagVO;
import com.cloud.storage.VolumeStats;
Review comment:
this file is not changed. better remove the diff from the PR
##########
File path: server/src/main/java/com/cloud/storage/ImageStoreDetailsUtil.java
##########
@@ -30,7 +30,6 @@
import com.google.common.base.Preconditions;
public class ImageStoreDetailsUtil {
-
Review comment:
I really like this style but in spite of it looking neater, the removal of the line is a potential conflict before we get a chance to merge. Better have those kind of changes is isolated contributions.
##########
File path: engine/orchestration/src/main/java/org/apache/cloudstack/engine/orchestration/DataMigrationUtility.java
##########
@@ -0,0 +1,262 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.cloudstack.engine.orchestration;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+
+import javax.inject.Inject;
+
+import org.apache.cloudstack.engine.subsystem.api.storage.DataObject;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.engine.subsystem.api.storage.ObjectInDataStoreStateMachine;
+import org.apache.cloudstack.engine.subsystem.api.storage.SnapshotDataFactory;
+import org.apache.cloudstack.engine.subsystem.api.storage.SnapshotInfo;
+import org.apache.cloudstack.engine.subsystem.api.storage.TemplateDataFactory;
+import org.apache.cloudstack.engine.subsystem.api.storage.VolumeDataFactory;
+import org.apache.cloudstack.storage.ImageStoreService;
+import org.apache.cloudstack.storage.datastore.db.SnapshotDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.SnapshotDataStoreVO;
+import org.apache.cloudstack.storage.datastore.db.TemplateDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.TemplateDataStoreVO;
+import org.apache.cloudstack.storage.datastore.db.VolumeDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.VolumeDataStoreVO;
+import org.apache.log4j.Logger;
+
+import com.cloud.host.HostVO;
+import com.cloud.host.Status;
+import com.cloud.host.dao.HostDao;
+import com.cloud.storage.DataStoreRole;
+import com.cloud.storage.SnapshotVO;
+import com.cloud.storage.VMTemplateVO;
+import com.cloud.storage.dao.SnapshotDao;
+import com.cloud.storage.dao.VMTemplateDao;
+import com.cloud.utils.Pair;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.vm.SecondaryStorageVm;
+import com.cloud.vm.SecondaryStorageVmVO;
+import com.cloud.vm.VirtualMachine;
+import com.cloud.vm.dao.SecondaryStorageVmDao;
+
+public class DataMigrationUtility {
+ @Inject
+ SecondaryStorageVmDao secStorageVmDao;
+ @Inject
+ TemplateDataStoreDao templateDataStoreDao;
+ @Inject
+ SnapshotDataStoreDao snapshotDataStoreDao;
+ @Inject
+ VolumeDataStoreDao volumeDataStoreDao;
+ @Inject
+ VMTemplateDao templateDao;
+ @Inject
+ VolumeDataFactory volumeFactory;
+ @Inject
+ TemplateDataFactory templateFactory;
+ @Inject
+ SnapshotDataFactory snapshotFactory;
+ @Inject
+ HostDao hostDao;
+ @Inject
+ SnapshotDao snapshotDao;
+
+ private static final Logger s_logger = Logger.getLogger(DataMigrationUtility.class);
+
+ /** This function verifies if the given image store comprises of data objects that are not in either the "Ready" or
+ * "Allocated" state - in such a case, if the migration policy is complete, the migration is terminated
+ */
+ private boolean filesReady(Long srcDataStoreId) {
+ String[] validStates = new String[]{"Ready", "Allocated"};
+ boolean isReady = true;
+ List<TemplateDataStoreVO> templates = templateDataStoreDao.listByStoreId(srcDataStoreId);
+ for (TemplateDataStoreVO template : templates) {
+ isReady &= (Arrays.asList(validStates).contains(template.getState().toString()));
+ }
+ List<SnapshotDataStoreVO> snapshots = snapshotDataStoreDao.listByStoreId(srcDataStoreId, DataStoreRole.Image);
+ for (SnapshotDataStoreVO snapshot : snapshots) {
+ isReady &= (Arrays.asList(validStates).contains(snapshot.getState().toString()));
+ }
+ List<VolumeDataStoreVO> volumes = volumeDataStoreDao.listByStoreId(srcDataStoreId);
+ for (VolumeDataStoreVO volume : volumes) {
+ isReady &= (Arrays.asList(validStates).contains(volume.getState().toString()));
+ }
+ return isReady;
+ }
+
+ protected void checkIfCompleteMigrationPossible(ImageStoreService.MigrationPolicy policy, Long srcDataStoreId) {
+ if (policy == ImageStoreService.MigrationPolicy.COMPLETE) {
+ if (!filesReady(srcDataStoreId)) {
+ throw new CloudRuntimeException("Complete migration failed as there are data objects which are not Ready");
+ }
+ }
+ return;
+ }
+
+ protected Long getFileSize(DataObject file, Map<DataObject, Pair<List<SnapshotInfo>, Long>> snapshotChain) {
+ Long size = file.getSize();
+ Pair<List<SnapshotInfo>, Long> chain = snapshotChain.get(file);
+ if (file instanceof SnapshotInfo && chain.first() != null) {
+ size = chain.second();
+ }
+ return size;
+ }
+
+ /**
+ * Sorts the datastores in decreasing order of their free capacities, so as to make
+ * an informed decision of picking the datastore with maximum free capactiy for migration
+ */
+ protected List<Long> sortDataStores(Map<Long, Pair<Long, Long>> storageCapacities) {
+ List<Map.Entry<Long, Pair<Long, Long>>> list =
+ new LinkedList<Map.Entry<Long, Pair<Long, Long>>>((storageCapacities.entrySet()));
+
+ Collections.sort(list, new Comparator<Map.Entry<Long, Pair<Long, Long>>>() {
+ @Override
+ public int compare(Map.Entry<Long, Pair<Long, Long>> e1, Map.Entry<Long, Pair<Long, Long>> e2) {
+ return e2.getValue().first() > e1.getValue().first() ? 1 : -1;
+ }
+ });
+ HashMap<Long, Pair<Long, Long>> temp = new LinkedHashMap<>();
+ for (Map.Entry<Long, Pair<Long, Long>> value : list) {
+ temp.put(value.getKey(), value.getValue());
+ }
+
+ return new ArrayList<>(temp.keySet());
+ }
+
+ protected List<DataObject> getSortedValidSourcesList(DataStore srcDataStore, Map<DataObject, Pair<List<SnapshotInfo>, Long>> snapshotChains) {
+ List<DataObject> files = new ArrayList<>();
+ files.addAll(getAllValidTemplates(srcDataStore));
+ files.addAll(getAllValidSnapshotsAndChains(srcDataStore, snapshotChains));
+ files.addAll(getAllValidVolumes(srcDataStore));
+
+ files = sortFilesOnSize(files, snapshotChains);
+
+ return files;
+ }
+
+ protected List<DataObject> sortFilesOnSize(List<DataObject> files, Map<DataObject, Pair<List<SnapshotInfo>, Long>> snapshotChains) {
+ Collections.sort(files, new Comparator<DataObject>() {
+ @Override
+ public int compare(DataObject o1, DataObject o2) {
+ Long size1 = o1.getSize();
+ Long size2 = o2.getSize();
+ if (o1 instanceof SnapshotInfo) {
+ size1 = snapshotChains.get(o1).second();
+ }
+ if (o2 instanceof SnapshotInfo) {
+ size2 = snapshotChains.get(o2).second();
+ }
+ return (int) (size2 - size1);
+ }
+ });
+ return files;
+ }
+
+ // Gets list of all valid templates, i.e, templates in "Ready" state for migration
+ protected List<DataObject> getAllValidTemplates(DataStore srcDataStore) {
+
+ List<DataObject> files = new LinkedList<>();
+ List<TemplateDataStoreVO> templates = templateDataStoreDao.listByStoreId(srcDataStore.getId());
+ for (TemplateDataStoreVO template : templates) {
+ VMTemplateVO templateVO = templateDao.findById(template.getTemplateId());
+ if (template.getState() == ObjectInDataStoreStateMachine.State.Ready && !templateVO.isPublicTemplate()) {
+ files.add(templateFactory.getTemplate(template.getTemplateId(), srcDataStore));
+ }
+ }
+ return files;
+ }
+
+ /** Returns parent snapshots and snapshots that do not have any children; snapshotChains comprises of the snapshot chain info
+ * for each parent snapshot and the cumulative size of the chain - this is done to ensure that all the snapshots in a chain
+ * are migrated to the same datastore
+ */
Review comment:
this triggers the thought: should chain be migrated at all or be coalesced first?
##########
File path: server/src/main/java/com/cloud/configuration/Config.java
##########
@@ -904,6 +905,14 @@
"random",
"'random', 'firstfit', 'userdispersing', 'userconcentratedpod_random', 'userconcentratedpod_firstfit', 'firstfitleastconsumed' : Order in which hosts within a cluster will be considered for VM/volume allocation.",
null),
+ ImageStoreAllocationAlgorithm(
Review comment:
adding keys here is an obsoleted way of creating Configuration details. please use a `Configkey<String>`
##########
File path: server/pom.xml
##########
@@ -89,6 +89,11 @@
<groupId>commons-codec</groupId>
<artifactId>commons-codec</artifactId>
</dependency>
+ <dependency>
+ <groupId>org.apache.commons</groupId>
+ <artifactId>commons-math3</artifactId>
+ <version>3.6.1</version>
Review comment:
version should go in the top level pom and not here.
##########
File path: server/src/main/java/com/cloud/configuration/Config.java
##########
@@ -1808,7 +1817,10 @@
// StatsCollector
StatsOutPutGraphiteHost("Advanced", ManagementServer.class, String.class, "stats.output.uri", "", "URI to additionally send StatsCollector statistics to", null),
- SSVMPSK("Hidden", ManagementServer.class, String.class, "upload.post.secret.key", "", "PSK with SSVM", null);
+ SSVMPSK("Hidden", ManagementServer.class, String.class, "upload.post.secret.key", "", "PSK with SSVM", null),
+
+ SecStorageMaxMigrateSessions(
+ "Advanced", AgentManager.class, Integer.class, "secstorage.max.migrate.sessions","2","The max number of concurrent copy command execution sessions that an SSVM can handle",null);
Review comment:
please use `ConfigKey<>`s
##########
File path: engine/orchestration/src/main/java/org/apache/cloudstack/engine/orchestration/DataMigrationUtility.java
##########
@@ -0,0 +1,262 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.cloudstack.engine.orchestration;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+
+import javax.inject.Inject;
+
+import org.apache.cloudstack.engine.subsystem.api.storage.DataObject;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.engine.subsystem.api.storage.ObjectInDataStoreStateMachine;
+import org.apache.cloudstack.engine.subsystem.api.storage.SnapshotDataFactory;
+import org.apache.cloudstack.engine.subsystem.api.storage.SnapshotInfo;
+import org.apache.cloudstack.engine.subsystem.api.storage.TemplateDataFactory;
+import org.apache.cloudstack.engine.subsystem.api.storage.VolumeDataFactory;
+import org.apache.cloudstack.storage.ImageStoreService;
+import org.apache.cloudstack.storage.datastore.db.SnapshotDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.SnapshotDataStoreVO;
+import org.apache.cloudstack.storage.datastore.db.TemplateDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.TemplateDataStoreVO;
+import org.apache.cloudstack.storage.datastore.db.VolumeDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.VolumeDataStoreVO;
+import org.apache.log4j.Logger;
+
+import com.cloud.host.HostVO;
+import com.cloud.host.Status;
+import com.cloud.host.dao.HostDao;
+import com.cloud.storage.DataStoreRole;
+import com.cloud.storage.SnapshotVO;
+import com.cloud.storage.VMTemplateVO;
+import com.cloud.storage.dao.SnapshotDao;
+import com.cloud.storage.dao.VMTemplateDao;
+import com.cloud.utils.Pair;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.vm.SecondaryStorageVm;
+import com.cloud.vm.SecondaryStorageVmVO;
+import com.cloud.vm.VirtualMachine;
+import com.cloud.vm.dao.SecondaryStorageVmDao;
+
+public class DataMigrationUtility {
+ @Inject
+ SecondaryStorageVmDao secStorageVmDao;
+ @Inject
+ TemplateDataStoreDao templateDataStoreDao;
+ @Inject
+ SnapshotDataStoreDao snapshotDataStoreDao;
+ @Inject
+ VolumeDataStoreDao volumeDataStoreDao;
+ @Inject
+ VMTemplateDao templateDao;
+ @Inject
+ VolumeDataFactory volumeFactory;
+ @Inject
+ TemplateDataFactory templateFactory;
+ @Inject
+ SnapshotDataFactory snapshotFactory;
+ @Inject
+ HostDao hostDao;
+ @Inject
+ SnapshotDao snapshotDao;
+
+ private static final Logger s_logger = Logger.getLogger(DataMigrationUtility.class);
+
+ /** This function verifies if the given image store comprises of data objects that are not in either the "Ready" or
+ * "Allocated" state - in such a case, if the migration policy is complete, the migration is terminated
+ */
+ private boolean filesReady(Long srcDataStoreId) {
+ String[] validStates = new String[]{"Ready", "Allocated"};
+ boolean isReady = true;
+ List<TemplateDataStoreVO> templates = templateDataStoreDao.listByStoreId(srcDataStoreId);
+ for (TemplateDataStoreVO template : templates) {
+ isReady &= (Arrays.asList(validStates).contains(template.getState().toString()));
+ }
+ List<SnapshotDataStoreVO> snapshots = snapshotDataStoreDao.listByStoreId(srcDataStoreId, DataStoreRole.Image);
+ for (SnapshotDataStoreVO snapshot : snapshots) {
+ isReady &= (Arrays.asList(validStates).contains(snapshot.getState().toString()));
+ }
+ List<VolumeDataStoreVO> volumes = volumeDataStoreDao.listByStoreId(srcDataStoreId);
+ for (VolumeDataStoreVO volume : volumes) {
+ isReady &= (Arrays.asList(validStates).contains(volume.getState().toString()));
+ }
+ return isReady;
+ }
+
+ protected void checkIfCompleteMigrationPossible(ImageStoreService.MigrationPolicy policy, Long srcDataStoreId) {
+ if (policy == ImageStoreService.MigrationPolicy.COMPLETE) {
+ if (!filesReady(srcDataStoreId)) {
+ throw new CloudRuntimeException("Complete migration failed as there are data objects which are not Ready");
+ }
+ }
+ return;
+ }
+
+ protected Long getFileSize(DataObject file, Map<DataObject, Pair<List<SnapshotInfo>, Long>> snapshotChain) {
+ Long size = file.getSize();
+ Pair<List<SnapshotInfo>, Long> chain = snapshotChain.get(file);
+ if (file instanceof SnapshotInfo && chain.first() != null) {
+ size = chain.second();
+ }
+ return size;
+ }
+
+ /**
+ * Sorts the datastores in decreasing order of their free capacities, so as to make
+ * an informed decision of picking the datastore with maximum free capactiy for migration
+ */
+ protected List<Long> sortDataStores(Map<Long, Pair<Long, Long>> storageCapacities) {
+ List<Map.Entry<Long, Pair<Long, Long>>> list =
+ new LinkedList<Map.Entry<Long, Pair<Long, Long>>>((storageCapacities.entrySet()));
+
+ Collections.sort(list, new Comparator<Map.Entry<Long, Pair<Long, Long>>>() {
+ @Override
+ public int compare(Map.Entry<Long, Pair<Long, Long>> e1, Map.Entry<Long, Pair<Long, Long>> e2) {
+ return e2.getValue().first() > e1.getValue().first() ? 1 : -1;
+ }
+ });
+ HashMap<Long, Pair<Long, Long>> temp = new LinkedHashMap<>();
+ for (Map.Entry<Long, Pair<Long, Long>> value : list) {
+ temp.put(value.getKey(), value.getValue());
+ }
+
+ return new ArrayList<>(temp.keySet());
+ }
+
+ protected List<DataObject> getSortedValidSourcesList(DataStore srcDataStore, Map<DataObject, Pair<List<SnapshotInfo>, Long>> snapshotChains) {
+ List<DataObject> files = new ArrayList<>();
+ files.addAll(getAllValidTemplates(srcDataStore));
+ files.addAll(getAllValidSnapshotsAndChains(srcDataStore, snapshotChains));
+ files.addAll(getAllValidVolumes(srcDataStore));
+
+ files = sortFilesOnSize(files, snapshotChains);
+
+ return files;
+ }
+
+ protected List<DataObject> sortFilesOnSize(List<DataObject> files, Map<DataObject, Pair<List<SnapshotInfo>, Long>> snapshotChains) {
+ Collections.sort(files, new Comparator<DataObject>() {
+ @Override
+ public int compare(DataObject o1, DataObject o2) {
+ Long size1 = o1.getSize();
+ Long size2 = o2.getSize();
+ if (o1 instanceof SnapshotInfo) {
+ size1 = snapshotChains.get(o1).second();
+ }
+ if (o2 instanceof SnapshotInfo) {
+ size2 = snapshotChains.get(o2).second();
+ }
+ return (int) (size2 - size1);
+ }
+ });
+ return files;
+ }
+
+ // Gets list of all valid templates, i.e, templates in "Ready" state for migration
+ protected List<DataObject> getAllValidTemplates(DataStore srcDataStore) {
+
+ List<DataObject> files = new LinkedList<>();
+ List<TemplateDataStoreVO> templates = templateDataStoreDao.listByStoreId(srcDataStore.getId());
+ for (TemplateDataStoreVO template : templates) {
+ VMTemplateVO templateVO = templateDao.findById(template.getTemplateId());
+ if (template.getState() == ObjectInDataStoreStateMachine.State.Ready && !templateVO.isPublicTemplate()) {
+ files.add(templateFactory.getTemplate(template.getTemplateId(), srcDataStore));
+ }
+ }
+ return files;
+ }
+
+ /** Returns parent snapshots and snapshots that do not have any children; snapshotChains comprises of the snapshot chain info
+ * for each parent snapshot and the cumulative size of the chain - this is done to ensure that all the snapshots in a chain
+ * are migrated to the same datastore
+ */
+ protected List<DataObject> getAllValidSnapshotsAndChains(DataStore srcDataStore, Map<DataObject, Pair<List<SnapshotInfo>, Long>> snapshotChains) {
+ List<SnapshotInfo> files = new LinkedList<>();
+ List<SnapshotDataStoreVO> snapshots = snapshotDataStoreDao.listByStoreId(srcDataStore.getId(), DataStoreRole.Image);
+ for (SnapshotDataStoreVO snapshot : snapshots) {
+ SnapshotVO snapshotVO = snapshotDao.findById(snapshot.getSnapshotId());
+ if (snapshot.getState() == ObjectInDataStoreStateMachine.State.Ready && snapshot.getParentSnapshotId() == 0 ) {
+ SnapshotInfo snap = snapshotFactory.getSnapshot(snapshotVO.getSnapshotId(), DataStoreRole.Image);
+ files.add(snap);
+ }
+ }
+
+ for (SnapshotInfo parent : files) {
+ List<SnapshotInfo> chain = new ArrayList<>();
+ chain.add(parent);
+ for (int i =0; i< chain.size(); i++) {
+ SnapshotInfo child = chain.get(i);
+ List<SnapshotInfo> children = child.getChildren();
+ if (children != null) {
+ chain.addAll(children);
+ }
+ }
+ snapshotChains.put(parent, new Pair<List<SnapshotInfo>, Long>(chain, getSizeForChain(chain)));
+ }
+
+ return (List<DataObject>) (List<?>) files;
+ }
+
+ // Finds the cumulative file size for all data objects in the chain
+ protected Long getSizeForChain(List<SnapshotInfo> chain) {
+ Long size = 0L;
+ for (SnapshotInfo snapshot : chain) {
+ size += snapshot.getSize();
+ }
+ return size;
+ }
+
+ // Returns a list of volumes that are in "Ready" state
Review comment:
rename to `getAllReadyVolumes` and delete comment?
##########
File path: services/secondary-storage/server/src/main/java/org/apache/cloudstack/storage/resource/NfsSecondaryStorageResource.java
##########
@@ -1285,6 +1288,75 @@ protected File findFile(String path) {
return srcFile;
}
+ protected Answer copyFromNfsToNfs(CopyCommand cmd) {
+ final DataTO srcData = cmd.getSrcTO();
+ final DataTO destData = cmd.getDestTO();
+ DataStoreTO srcDataStore = srcData.getDataStore();
+ NfsTO srcStore = (NfsTO)srcDataStore;
+ DataStoreTO destDataStore = destData.getDataStore();
+ final NfsTO destStore = (NfsTO) destDataStore;
+ try {
+ File srcFile = new File(getDir(srcStore.getUrl(), _nfsVersion), srcData.getPath());
+ File destFile = new File(getDir(destStore.getUrl(), _nfsVersion), destData.getPath());
+ ImageFormat format = getTemplateFormat(srcFile.getName());
+
+ if (srcFile == null) {
+ return new CopyCmdAnswer("Can't find src file:" + srcFile);
+ }
+
+ if (srcData instanceof TemplateObjectTO || srcData instanceof VolumeObjectTO) {
+ File srcDir = null;
+ if (srcFile.isFile()) {
+ srcDir = new File(srcFile.getParent());
+ }
+ File destDir = null;
+ if (destFile.isFile()) {
+ destDir = new File(destFile.getParent());
+ }
+
+ try {
+ FileUtils.copyDirectory((srcDir == null ? srcFile : srcDir), (destDir == null? destFile : destDir));
+ } catch (IOException e) {
+ String msg = "Failed to copy file to destination";
+ s_logger.info(msg);
+ return new CopyCmdAnswer(msg);
+ }
+ } else {
+ destFile = new File(destFile, srcFile.getName());
+ try {
+ FileUtils.copyFile(srcFile, destFile);
+ } catch (IOException e) {
+ String msg = "Failed to copy file to destination";
+ s_logger.info(msg);
+ return new CopyCmdAnswer(msg);
+ }
+ }
+
+ DataTO retObj = null;
+ if (destData.getObjectType() == DataObjectType.TEMPLATE) {
+ TemplateObjectTO newTemplate = new TemplateObjectTO();
+ newTemplate.setPath(destData.getPath() + File.separator + srcFile.getName());
+ newTemplate.setSize(getVirtualSize(srcFile, format));
+ newTemplate.setPhysicalSize(srcFile.length());
+ newTemplate.setFormat(format);
+ retObj = newTemplate;
+ } else if (destData.getObjectType() == DataObjectType.VOLUME) {
+ VolumeObjectTO newVol = new VolumeObjectTO();
+ newVol.setPath(destData.getPath() + File.separator + srcFile.getName());
+ newVol.setSize(srcFile.length());
+ retObj = newVol;
+ } else if (destData.getObjectType() == DataObjectType.SNAPSHOT) {
+ SnapshotObjectTO newSnapshot = new SnapshotObjectTO();
+ newSnapshot.setPath(destData.getPath() + File.separator + destFile.getName());
+ retObj = newSnapshot;
+ }
+ return new CopyCmdAnswer(retObj);
+ } catch (Exception e) {
+ s_logger.error("failed to copy file" + srcData.getPath(), e);
+ return new CopyCmdAnswer("failed to copy file" + srcData.getPath() + e.toString());
Review comment:
some kind of indentation problem here. The method is a bit to long for a clear view on what happened. Can you modularise it a bit more, please? Also there is more than one return points, which makes the flow hard to follow.
##########
File path: api/src/main/java/org/apache/cloudstack/api/command/admin/storage/UpdateImageStoreCmd.java
##########
@@ -0,0 +1,89 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.cloudstack.api.command.admin.storage;
+
+import org.apache.cloudstack.api.APICommand;
+import org.apache.cloudstack.api.ApiConstants;
+import org.apache.cloudstack.api.ApiErrorCode;
+import org.apache.cloudstack.api.BaseCmd;
+import org.apache.cloudstack.api.Parameter;
+import org.apache.cloudstack.api.ServerApiException;
+import org.apache.cloudstack.api.response.ImageStoreResponse;
+import org.apache.cloudstack.context.CallContext;
+import org.apache.log4j.Logger;
+
+import com.cloud.storage.ImageStore;
+
+@APICommand(name = UpdateImageStoreCmd.APINAME, description = "Updates image store read-only status", responseObject = ImageStoreResponse.class, entityType = {ImageStore.class},
Review comment:
can you add `since = "4.15"`
##########
File path: engine/orchestration/src/main/java/org/apache/cloudstack/engine/orchestration/DataMigrationUtility.java
##########
@@ -0,0 +1,262 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.cloudstack.engine.orchestration;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+
+import javax.inject.Inject;
+
+import org.apache.cloudstack.engine.subsystem.api.storage.DataObject;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.engine.subsystem.api.storage.ObjectInDataStoreStateMachine;
+import org.apache.cloudstack.engine.subsystem.api.storage.SnapshotDataFactory;
+import org.apache.cloudstack.engine.subsystem.api.storage.SnapshotInfo;
+import org.apache.cloudstack.engine.subsystem.api.storage.TemplateDataFactory;
+import org.apache.cloudstack.engine.subsystem.api.storage.VolumeDataFactory;
+import org.apache.cloudstack.storage.ImageStoreService;
+import org.apache.cloudstack.storage.datastore.db.SnapshotDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.SnapshotDataStoreVO;
+import org.apache.cloudstack.storage.datastore.db.TemplateDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.TemplateDataStoreVO;
+import org.apache.cloudstack.storage.datastore.db.VolumeDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.VolumeDataStoreVO;
+import org.apache.log4j.Logger;
+
+import com.cloud.host.HostVO;
+import com.cloud.host.Status;
+import com.cloud.host.dao.HostDao;
+import com.cloud.storage.DataStoreRole;
+import com.cloud.storage.SnapshotVO;
+import com.cloud.storage.VMTemplateVO;
+import com.cloud.storage.dao.SnapshotDao;
+import com.cloud.storage.dao.VMTemplateDao;
+import com.cloud.utils.Pair;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.vm.SecondaryStorageVm;
+import com.cloud.vm.SecondaryStorageVmVO;
+import com.cloud.vm.VirtualMachine;
+import com.cloud.vm.dao.SecondaryStorageVmDao;
+
+public class DataMigrationUtility {
+ @Inject
+ SecondaryStorageVmDao secStorageVmDao;
+ @Inject
+ TemplateDataStoreDao templateDataStoreDao;
+ @Inject
+ SnapshotDataStoreDao snapshotDataStoreDao;
+ @Inject
+ VolumeDataStoreDao volumeDataStoreDao;
+ @Inject
+ VMTemplateDao templateDao;
+ @Inject
+ VolumeDataFactory volumeFactory;
+ @Inject
+ TemplateDataFactory templateFactory;
+ @Inject
+ SnapshotDataFactory snapshotFactory;
+ @Inject
+ HostDao hostDao;
+ @Inject
+ SnapshotDao snapshotDao;
+
+ private static final Logger s_logger = Logger.getLogger(DataMigrationUtility.class);
+
+ /** This function verifies if the given image store comprises of data objects that are not in either the "Ready" or
+ * "Allocated" state - in such a case, if the migration policy is complete, the migration is terminated
+ */
+ private boolean filesReady(Long srcDataStoreId) {
+ String[] validStates = new String[]{"Ready", "Allocated"};
+ boolean isReady = true;
+ List<TemplateDataStoreVO> templates = templateDataStoreDao.listByStoreId(srcDataStoreId);
+ for (TemplateDataStoreVO template : templates) {
+ isReady &= (Arrays.asList(validStates).contains(template.getState().toString()));
+ }
+ List<SnapshotDataStoreVO> snapshots = snapshotDataStoreDao.listByStoreId(srcDataStoreId, DataStoreRole.Image);
+ for (SnapshotDataStoreVO snapshot : snapshots) {
+ isReady &= (Arrays.asList(validStates).contains(snapshot.getState().toString()));
+ }
+ List<VolumeDataStoreVO> volumes = volumeDataStoreDao.listByStoreId(srcDataStoreId);
+ for (VolumeDataStoreVO volume : volumes) {
+ isReady &= (Arrays.asList(validStates).contains(volume.getState().toString()));
+ }
+ return isReady;
+ }
+
+ protected void checkIfCompleteMigrationPossible(ImageStoreService.MigrationPolicy policy, Long srcDataStoreId) {
+ if (policy == ImageStoreService.MigrationPolicy.COMPLETE) {
+ if (!filesReady(srcDataStoreId)) {
+ throw new CloudRuntimeException("Complete migration failed as there are data objects which are not Ready");
+ }
+ }
+ return;
+ }
+
+ protected Long getFileSize(DataObject file, Map<DataObject, Pair<List<SnapshotInfo>, Long>> snapshotChain) {
+ Long size = file.getSize();
+ Pair<List<SnapshotInfo>, Long> chain = snapshotChain.get(file);
+ if (file instanceof SnapshotInfo && chain.first() != null) {
+ size = chain.second();
+ }
+ return size;
+ }
+
+ /**
+ * Sorts the datastores in decreasing order of their free capacities, so as to make
+ * an informed decision of picking the datastore with maximum free capactiy for migration
+ */
+ protected List<Long> sortDataStores(Map<Long, Pair<Long, Long>> storageCapacities) {
+ List<Map.Entry<Long, Pair<Long, Long>>> list =
+ new LinkedList<Map.Entry<Long, Pair<Long, Long>>>((storageCapacities.entrySet()));
+
+ Collections.sort(list, new Comparator<Map.Entry<Long, Pair<Long, Long>>>() {
+ @Override
+ public int compare(Map.Entry<Long, Pair<Long, Long>> e1, Map.Entry<Long, Pair<Long, Long>> e2) {
+ return e2.getValue().first() > e1.getValue().first() ? 1 : -1;
+ }
+ });
+ HashMap<Long, Pair<Long, Long>> temp = new LinkedHashMap<>();
+ for (Map.Entry<Long, Pair<Long, Long>> value : list) {
+ temp.put(value.getKey(), value.getValue());
+ }
+
+ return new ArrayList<>(temp.keySet());
+ }
+
+ protected List<DataObject> getSortedValidSourcesList(DataStore srcDataStore, Map<DataObject, Pair<List<SnapshotInfo>, Long>> snapshotChains) {
+ List<DataObject> files = new ArrayList<>();
+ files.addAll(getAllValidTemplates(srcDataStore));
+ files.addAll(getAllValidSnapshotsAndChains(srcDataStore, snapshotChains));
+ files.addAll(getAllValidVolumes(srcDataStore));
+
+ files = sortFilesOnSize(files, snapshotChains);
+
+ return files;
+ }
+
+ protected List<DataObject> sortFilesOnSize(List<DataObject> files, Map<DataObject, Pair<List<SnapshotInfo>, Long>> snapshotChains) {
+ Collections.sort(files, new Comparator<DataObject>() {
+ @Override
+ public int compare(DataObject o1, DataObject o2) {
+ Long size1 = o1.getSize();
+ Long size2 = o2.getSize();
+ if (o1 instanceof SnapshotInfo) {
+ size1 = snapshotChains.get(o1).second();
+ }
+ if (o2 instanceof SnapshotInfo) {
+ size2 = snapshotChains.get(o2).second();
+ }
+ return (int) (size2 - size1);
+ }
+ });
+ return files;
+ }
+
+ // Gets list of all valid templates, i.e, templates in "Ready" state for migration
+ protected List<DataObject> getAllValidTemplates(DataStore srcDataStore) {
+
+ List<DataObject> files = new LinkedList<>();
+ List<TemplateDataStoreVO> templates = templateDataStoreDao.listByStoreId(srcDataStore.getId());
+ for (TemplateDataStoreVO template : templates) {
+ VMTemplateVO templateVO = templateDao.findById(template.getTemplateId());
+ if (template.getState() == ObjectInDataStoreStateMachine.State.Ready && !templateVO.isPublicTemplate()) {
+ files.add(templateFactory.getTemplate(template.getTemplateId(), srcDataStore));
+ }
+ }
+ return files;
+ }
+
+ /** Returns parent snapshots and snapshots that do not have any children; snapshotChains comprises of the snapshot chain info
+ * for each parent snapshot and the cumulative size of the chain - this is done to ensure that all the snapshots in a chain
+ * are migrated to the same datastore
+ */
+ protected List<DataObject> getAllValidSnapshotsAndChains(DataStore srcDataStore, Map<DataObject, Pair<List<SnapshotInfo>, Long>> snapshotChains) {
+ List<SnapshotInfo> files = new LinkedList<>();
+ List<SnapshotDataStoreVO> snapshots = snapshotDataStoreDao.listByStoreId(srcDataStore.getId(), DataStoreRole.Image);
+ for (SnapshotDataStoreVO snapshot : snapshots) {
+ SnapshotVO snapshotVO = snapshotDao.findById(snapshot.getSnapshotId());
+ if (snapshot.getState() == ObjectInDataStoreStateMachine.State.Ready && snapshot.getParentSnapshotId() == 0 ) {
+ SnapshotInfo snap = snapshotFactory.getSnapshot(snapshotVO.getSnapshotId(), DataStoreRole.Image);
+ files.add(snap);
+ }
+ }
+
+ for (SnapshotInfo parent : files) {
+ List<SnapshotInfo> chain = new ArrayList<>();
+ chain.add(parent);
+ for (int i =0; i< chain.size(); i++) {
+ SnapshotInfo child = chain.get(i);
+ List<SnapshotInfo> children = child.getChildren();
+ if (children != null) {
+ chain.addAll(children);
+ }
+ }
+ snapshotChains.put(parent, new Pair<List<SnapshotInfo>, Long>(chain, getSizeForChain(chain)));
+ }
+
+ return (List<DataObject>) (List<?>) files;
+ }
+
+ // Finds the cumulative file size for all data objects in the chain
Review comment:
methodname says it all
##########
File path: engine/storage/src/main/java/org/apache/cloudstack/storage/datastore/ObjectInDataStoreManagerImpl.java
##########
@@ -104,6 +104,14 @@ public ObjectInDataStoreManagerImpl() {
// TODO: further investigate why an extra event is sent when it is
// alreay Ready for DownloadListener
stateMachines.addTransition(State.Ready, Event.OperationSuccessed, State.Ready);
+ // State transitions for data object migration
+ stateMachines.addTransition(State.Ready, Event.MigrationRequested, State.Migrating);
+ stateMachines.addTransition(State.Ready, Event.CopyRequested, State.Copying);
+ stateMachines.addTransition(State.Allocated, Event.MigrationRequested, State.Migrating);
+ stateMachines.addTransition(State.Migrating, Event.MigrationFailed, State.Failed);
+ stateMachines.addTransition(State.Migrating, Event.MigrationSucceeded, State.Destroyed);
+ stateMachines.addTransition(State.Migrating, Event.OperationSuccessed, State.Ready);
Review comment:
`Event.OperationSuccessed`? or should this be `Event.OperationSucceeded`?
##########
File path: engine/orchestration/src/main/java/org/apache/cloudstack/engine/orchestration/DataMigrationUtility.java
##########
@@ -0,0 +1,262 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.cloudstack.engine.orchestration;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+
+import javax.inject.Inject;
+
+import org.apache.cloudstack.engine.subsystem.api.storage.DataObject;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.engine.subsystem.api.storage.ObjectInDataStoreStateMachine;
+import org.apache.cloudstack.engine.subsystem.api.storage.SnapshotDataFactory;
+import org.apache.cloudstack.engine.subsystem.api.storage.SnapshotInfo;
+import org.apache.cloudstack.engine.subsystem.api.storage.TemplateDataFactory;
+import org.apache.cloudstack.engine.subsystem.api.storage.VolumeDataFactory;
+import org.apache.cloudstack.storage.ImageStoreService;
+import org.apache.cloudstack.storage.datastore.db.SnapshotDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.SnapshotDataStoreVO;
+import org.apache.cloudstack.storage.datastore.db.TemplateDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.TemplateDataStoreVO;
+import org.apache.cloudstack.storage.datastore.db.VolumeDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.VolumeDataStoreVO;
+import org.apache.log4j.Logger;
+
+import com.cloud.host.HostVO;
+import com.cloud.host.Status;
+import com.cloud.host.dao.HostDao;
+import com.cloud.storage.DataStoreRole;
+import com.cloud.storage.SnapshotVO;
+import com.cloud.storage.VMTemplateVO;
+import com.cloud.storage.dao.SnapshotDao;
+import com.cloud.storage.dao.VMTemplateDao;
+import com.cloud.utils.Pair;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.vm.SecondaryStorageVm;
+import com.cloud.vm.SecondaryStorageVmVO;
+import com.cloud.vm.VirtualMachine;
+import com.cloud.vm.dao.SecondaryStorageVmDao;
+
+public class DataMigrationUtility {
+ @Inject
+ SecondaryStorageVmDao secStorageVmDao;
+ @Inject
+ TemplateDataStoreDao templateDataStoreDao;
+ @Inject
+ SnapshotDataStoreDao snapshotDataStoreDao;
+ @Inject
+ VolumeDataStoreDao volumeDataStoreDao;
+ @Inject
+ VMTemplateDao templateDao;
+ @Inject
+ VolumeDataFactory volumeFactory;
+ @Inject
+ TemplateDataFactory templateFactory;
+ @Inject
+ SnapshotDataFactory snapshotFactory;
+ @Inject
+ HostDao hostDao;
+ @Inject
+ SnapshotDao snapshotDao;
+
+ private static final Logger s_logger = Logger.getLogger(DataMigrationUtility.class);
Review comment:
Can you call this static final something all-caps, please?
##########
File path: engine/storage/image/src/main/java/org/apache/cloudstack/storage/image/TemplateServiceImpl.java
##########
@@ -253,7 +253,8 @@ public void downloadBootstrapSysTemplate(DataStore store) {
@Override
public void handleSysTemplateDownload(HypervisorType hostHyper, Long dcId) {
Set<VMTemplateVO> toBeDownloaded = new HashSet<VMTemplateVO>();
- List<DataStore> stores = _storeMgr.getImageStoresByScope(new ZoneScope(dcId));
+ //List<DataStore> stores = _storeMgr.getImageStoresByScope(new ZoneScope(dcId));
Review comment:
please remove commented code
##########
File path: engine/storage/volume/src/main/java/org/apache/cloudstack/storage/volume/VolumeObject.java
##########
@@ -392,7 +391,8 @@ public void processEvent(ObjectInDataStoreStateMachine.Event event) {
if (event == ObjectInDataStoreStateMachine.Event.CreateOnlyRequested) {
volEvent = Volume.Event.UploadRequested;
} else if (event == ObjectInDataStoreStateMachine.Event.MigrationRequested) {
- volEvent = Volume.Event.CopyRequested;
+ //volEvent = Volume.Event.CopyRequested;
Review comment:
please do not keep comment incode. We have an RCS for keeping save old code.
##########
File path: engine/orchestration/src/main/java/org/apache/cloudstack/engine/orchestration/StorageOrchestrator.java
##########
@@ -0,0 +1,629 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.cloudstack.engine.orchestration;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.Hashtable;
+import java.util.LinkedHashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.Callable;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+
+import javax.inject.Inject;
+import javax.naming.ConfigurationException;
+
+import org.apache.cloudstack.api.response.MigrationResponse;
+import org.apache.cloudstack.engine.orchestration.service.StorageOrchestrationService;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataObject;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStoreManager;
+import org.apache.cloudstack.engine.subsystem.api.storage.ObjectInDataStoreStateMachine;
+import org.apache.cloudstack.engine.subsystem.api.storage.SecondaryStorageService;
+import org.apache.cloudstack.engine.subsystem.api.storage.SecondaryStorageService.DataObjectResult;
+import org.apache.cloudstack.engine.subsystem.api.storage.SnapshotDataFactory;
+import org.apache.cloudstack.engine.subsystem.api.storage.SnapshotInfo;
+import org.apache.cloudstack.engine.subsystem.api.storage.TemplateDataFactory;
+import org.apache.cloudstack.engine.subsystem.api.storage.VolumeDataFactory;
+import org.apache.cloudstack.framework.async.AsyncCallFuture;
+import org.apache.cloudstack.framework.config.ConfigKey;
+import org.apache.cloudstack.framework.config.Configurable;
+import org.apache.cloudstack.framework.config.dao.ConfigurationDao;
+import org.apache.cloudstack.framework.jobs.AsyncJobManager;
+import org.apache.cloudstack.storage.datastore.db.ImageStoreDao;
+import org.apache.cloudstack.storage.datastore.db.SnapshotDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.SnapshotDataStoreVO;
+import org.apache.cloudstack.storage.datastore.db.TemplateDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.TemplateDataStoreVO;
+import org.apache.cloudstack.storage.datastore.db.VolumeDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.VolumeDataStoreVO;
+import org.apache.commons.math3.stat.descriptive.moment.Mean;
+import org.apache.commons.math3.stat.descriptive.moment.StandardDeviation;
+import org.apache.log4j.Logger;
+
+import com.cloud.configuration.Config;
+import com.cloud.host.HostVO;
+import com.cloud.host.Status;
+import com.cloud.host.dao.HostDao;
+import com.cloud.server.StatsCollector;
+import com.cloud.storage.DataStoreRole;
+import com.cloud.storage.ImageStoreService.MigrationPolicy;
+import com.cloud.storage.SnapshotVO;
+import com.cloud.storage.StorageService;
+import com.cloud.storage.StorageStats;
+import com.cloud.storage.VMTemplateVO;
+import com.cloud.storage.dao.SnapshotDao;
+import com.cloud.storage.dao.VMTemplateDao;
+import com.cloud.utils.NumbersUtil;
+import com.cloud.utils.Pair;
+import com.cloud.utils.StringUtils;
+import com.cloud.utils.component.ManagerBase;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.vm.SecondaryStorageVm;
+import com.cloud.vm.SecondaryStorageVmVO;
+import com.cloud.vm.VirtualMachine;
+import com.cloud.vm.dao.SecondaryStorageVmDao;
+
+public class StorageOrchestrator extends ManagerBase implements StorageOrchestrationService, Configurable {
+
+ private static final Logger s_logger = Logger.getLogger(StorageOrchestrator.class);
+ @Inject
+ TemplateDataStoreDao templateDataStoreDao;
+ @Inject
+ SnapshotDataStoreDao snapshotDataStoreDao;
+ @Inject
+ VolumeDataStoreDao volumeDataStoreDao;
+ @Inject
+ VolumeDataFactory volumeFactory;
+ @Inject
+ VMTemplateDao templateDao;
+ @Inject
+ TemplateDataFactory templateFactory;
+ @Inject
+ SnapshotDao snapshotDao;
+ @Inject
+ SnapshotDataFactory snapshotFactory;
+ @Inject
+ DataStoreManager dataStoreManager;
+ @Inject
+ ImageStoreDao imageStoreDao;
+ @Inject
+ StatsCollector statsCollector;
+ @Inject
+ public StorageService storageService;
+ @Inject
+ SecondaryStorageVmDao secStorageVmDao;
+ @Inject
+ ConfigurationDao configDao;
+ @Inject
+ HostDao hostDao;
+ @Inject
+ private AsyncJobManager jobMgr;
+ @Inject
+ private SecondaryStorageService secStgSrv;
+
+ ConfigKey<Double> ImageStoreImbalanceThreshold = new ConfigKey<>("Advanced", Double.class,
+ "image.store.imbalance.threshold",
+ "0.1",
+ "The storage imbalance threshold that is compared with the standard deviation percentage for a storage utilization metric. " +
+ "The value is a percentage in decimal format.",
+ true, ConfigKey.Scope.Global);
+
+ Integer numConcurrentCopyTasksPerSSVM = 2;
+
+ private double imageStoreCapacityThreshold = 0.90;
+
+ @Override
+ public String getConfigComponentName() {
+ return StorageOrchestrationService.class.getName();
+ }
+
+ @Override
+ public ConfigKey<?>[] getConfigKeys() {
+ return new ConfigKey<?>[]{ImageStoreImbalanceThreshold};
+ }
+
+ static class MigrateBlockingQueue<T> extends ArrayBlockingQueue<T> {
+
+ MigrateBlockingQueue(int size) {
+ super(size);
+ }
+
+ public boolean offer(T task) {
+ try {
+ this.put(task);
+ } catch (InterruptedException e) {
+ throw new RuntimeException(e);
+ }
+ return true;
+ }
+ }
+
+ @Override
+ public boolean configure(String name, Map<String, Object> params) throws ConfigurationException {
+ numConcurrentCopyTasksPerSSVM = NumbersUtil.parseInt(configDao.getValue(Config.SecStorageMaxMigrateSessions.key()), 2);
+ return true;
+ }
+
+ @Override
+ public MigrationResponse migrateData(Long srcDataStoreId, List<Long> destDatastores, MigrationPolicy migrationPolicy) {
Review comment:
It is still 112 lines and there are a lot of comments in it that could serve as method names. This would help make the code more prozaic and thus easier to read. This method could do with some modularisation.
##########
File path: engine/orchestration/src/main/java/org/apache/cloudstack/engine/orchestration/DataMigrationUtility.java
##########
@@ -0,0 +1,262 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.cloudstack.engine.orchestration;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+
+import javax.inject.Inject;
+
+import org.apache.cloudstack.engine.subsystem.api.storage.DataObject;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.engine.subsystem.api.storage.ObjectInDataStoreStateMachine;
+import org.apache.cloudstack.engine.subsystem.api.storage.SnapshotDataFactory;
+import org.apache.cloudstack.engine.subsystem.api.storage.SnapshotInfo;
+import org.apache.cloudstack.engine.subsystem.api.storage.TemplateDataFactory;
+import org.apache.cloudstack.engine.subsystem.api.storage.VolumeDataFactory;
+import org.apache.cloudstack.storage.ImageStoreService;
+import org.apache.cloudstack.storage.datastore.db.SnapshotDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.SnapshotDataStoreVO;
+import org.apache.cloudstack.storage.datastore.db.TemplateDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.TemplateDataStoreVO;
+import org.apache.cloudstack.storage.datastore.db.VolumeDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.VolumeDataStoreVO;
+import org.apache.log4j.Logger;
+
+import com.cloud.host.HostVO;
+import com.cloud.host.Status;
+import com.cloud.host.dao.HostDao;
+import com.cloud.storage.DataStoreRole;
+import com.cloud.storage.SnapshotVO;
+import com.cloud.storage.VMTemplateVO;
+import com.cloud.storage.dao.SnapshotDao;
+import com.cloud.storage.dao.VMTemplateDao;
+import com.cloud.utils.Pair;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.vm.SecondaryStorageVm;
+import com.cloud.vm.SecondaryStorageVmVO;
+import com.cloud.vm.VirtualMachine;
+import com.cloud.vm.dao.SecondaryStorageVmDao;
+
+public class DataMigrationUtility {
+ @Inject
+ SecondaryStorageVmDao secStorageVmDao;
+ @Inject
+ TemplateDataStoreDao templateDataStoreDao;
+ @Inject
+ SnapshotDataStoreDao snapshotDataStoreDao;
+ @Inject
+ VolumeDataStoreDao volumeDataStoreDao;
+ @Inject
+ VMTemplateDao templateDao;
+ @Inject
+ VolumeDataFactory volumeFactory;
+ @Inject
+ TemplateDataFactory templateFactory;
+ @Inject
+ SnapshotDataFactory snapshotFactory;
+ @Inject
+ HostDao hostDao;
+ @Inject
+ SnapshotDao snapshotDao;
+
+ private static final Logger s_logger = Logger.getLogger(DataMigrationUtility.class);
+
+ /** This function verifies if the given image store comprises of data objects that are not in either the "Ready" or
+ * "Allocated" state - in such a case, if the migration policy is complete, the migration is terminated
+ */
+ private boolean filesReady(Long srcDataStoreId) {
+ String[] validStates = new String[]{"Ready", "Allocated"};
+ boolean isReady = true;
+ List<TemplateDataStoreVO> templates = templateDataStoreDao.listByStoreId(srcDataStoreId);
+ for (TemplateDataStoreVO template : templates) {
+ isReady &= (Arrays.asList(validStates).contains(template.getState().toString()));
+ }
+ List<SnapshotDataStoreVO> snapshots = snapshotDataStoreDao.listByStoreId(srcDataStoreId, DataStoreRole.Image);
+ for (SnapshotDataStoreVO snapshot : snapshots) {
+ isReady &= (Arrays.asList(validStates).contains(snapshot.getState().toString()));
+ }
+ List<VolumeDataStoreVO> volumes = volumeDataStoreDao.listByStoreId(srcDataStoreId);
+ for (VolumeDataStoreVO volume : volumes) {
+ isReady &= (Arrays.asList(validStates).contains(volume.getState().toString()));
+ }
+ return isReady;
+ }
+
+ protected void checkIfCompleteMigrationPossible(ImageStoreService.MigrationPolicy policy, Long srcDataStoreId) {
+ if (policy == ImageStoreService.MigrationPolicy.COMPLETE) {
+ if (!filesReady(srcDataStoreId)) {
+ throw new CloudRuntimeException("Complete migration failed as there are data objects which are not Ready");
+ }
+ }
+ return;
+ }
+
+ protected Long getFileSize(DataObject file, Map<DataObject, Pair<List<SnapshotInfo>, Long>> snapshotChain) {
+ Long size = file.getSize();
+ Pair<List<SnapshotInfo>, Long> chain = snapshotChain.get(file);
+ if (file instanceof SnapshotInfo && chain.first() != null) {
+ size = chain.second();
+ }
+ return size;
+ }
+
+ /**
+ * Sorts the datastores in decreasing order of their free capacities, so as to make
+ * an informed decision of picking the datastore with maximum free capactiy for migration
+ */
+ protected List<Long> sortDataStores(Map<Long, Pair<Long, Long>> storageCapacities) {
+ List<Map.Entry<Long, Pair<Long, Long>>> list =
+ new LinkedList<Map.Entry<Long, Pair<Long, Long>>>((storageCapacities.entrySet()));
+
+ Collections.sort(list, new Comparator<Map.Entry<Long, Pair<Long, Long>>>() {
+ @Override
+ public int compare(Map.Entry<Long, Pair<Long, Long>> e1, Map.Entry<Long, Pair<Long, Long>> e2) {
+ return e2.getValue().first() > e1.getValue().first() ? 1 : -1;
+ }
+ });
+ HashMap<Long, Pair<Long, Long>> temp = new LinkedHashMap<>();
+ for (Map.Entry<Long, Pair<Long, Long>> value : list) {
+ temp.put(value.getKey(), value.getValue());
+ }
+
+ return new ArrayList<>(temp.keySet());
+ }
+
+ protected List<DataObject> getSortedValidSourcesList(DataStore srcDataStore, Map<DataObject, Pair<List<SnapshotInfo>, Long>> snapshotChains) {
+ List<DataObject> files = new ArrayList<>();
+ files.addAll(getAllValidTemplates(srcDataStore));
+ files.addAll(getAllValidSnapshotsAndChains(srcDataStore, snapshotChains));
+ files.addAll(getAllValidVolumes(srcDataStore));
+
+ files = sortFilesOnSize(files, snapshotChains);
+
+ return files;
+ }
+
+ protected List<DataObject> sortFilesOnSize(List<DataObject> files, Map<DataObject, Pair<List<SnapshotInfo>, Long>> snapshotChains) {
+ Collections.sort(files, new Comparator<DataObject>() {
+ @Override
+ public int compare(DataObject o1, DataObject o2) {
+ Long size1 = o1.getSize();
+ Long size2 = o2.getSize();
+ if (o1 instanceof SnapshotInfo) {
+ size1 = snapshotChains.get(o1).second();
+ }
+ if (o2 instanceof SnapshotInfo) {
+ size2 = snapshotChains.get(o2).second();
+ }
+ return (int) (size2 - size1);
Review comment:
I know I asked for this (or such), but overflow is possible if the diff is huge.
In relation to the comment to the coomparable above, Is 0 useful? Might we want to reuse this in cases where it is?
##########
File path: engine/orchestration/src/main/java/org/apache/cloudstack/engine/orchestration/DataMigrationUtility.java
##########
@@ -0,0 +1,262 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.cloudstack.engine.orchestration;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+
+import javax.inject.Inject;
+
+import org.apache.cloudstack.engine.subsystem.api.storage.DataObject;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.engine.subsystem.api.storage.ObjectInDataStoreStateMachine;
+import org.apache.cloudstack.engine.subsystem.api.storage.SnapshotDataFactory;
+import org.apache.cloudstack.engine.subsystem.api.storage.SnapshotInfo;
+import org.apache.cloudstack.engine.subsystem.api.storage.TemplateDataFactory;
+import org.apache.cloudstack.engine.subsystem.api.storage.VolumeDataFactory;
+import org.apache.cloudstack.storage.ImageStoreService;
+import org.apache.cloudstack.storage.datastore.db.SnapshotDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.SnapshotDataStoreVO;
+import org.apache.cloudstack.storage.datastore.db.TemplateDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.TemplateDataStoreVO;
+import org.apache.cloudstack.storage.datastore.db.VolumeDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.VolumeDataStoreVO;
+import org.apache.log4j.Logger;
+
+import com.cloud.host.HostVO;
+import com.cloud.host.Status;
+import com.cloud.host.dao.HostDao;
+import com.cloud.storage.DataStoreRole;
+import com.cloud.storage.SnapshotVO;
+import com.cloud.storage.VMTemplateVO;
+import com.cloud.storage.dao.SnapshotDao;
+import com.cloud.storage.dao.VMTemplateDao;
+import com.cloud.utils.Pair;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.vm.SecondaryStorageVm;
+import com.cloud.vm.SecondaryStorageVmVO;
+import com.cloud.vm.VirtualMachine;
+import com.cloud.vm.dao.SecondaryStorageVmDao;
+
+public class DataMigrationUtility {
+ @Inject
+ SecondaryStorageVmDao secStorageVmDao;
+ @Inject
+ TemplateDataStoreDao templateDataStoreDao;
+ @Inject
+ SnapshotDataStoreDao snapshotDataStoreDao;
+ @Inject
+ VolumeDataStoreDao volumeDataStoreDao;
+ @Inject
+ VMTemplateDao templateDao;
+ @Inject
+ VolumeDataFactory volumeFactory;
+ @Inject
+ TemplateDataFactory templateFactory;
+ @Inject
+ SnapshotDataFactory snapshotFactory;
+ @Inject
+ HostDao hostDao;
+ @Inject
+ SnapshotDao snapshotDao;
+
+ private static final Logger s_logger = Logger.getLogger(DataMigrationUtility.class);
+
+ /** This function verifies if the given image store comprises of data objects that are not in either the "Ready" or
+ * "Allocated" state - in such a case, if the migration policy is complete, the migration is terminated
+ */
+ private boolean filesReady(Long srcDataStoreId) {
+ String[] validStates = new String[]{"Ready", "Allocated"};
+ boolean isReady = true;
+ List<TemplateDataStoreVO> templates = templateDataStoreDao.listByStoreId(srcDataStoreId);
+ for (TemplateDataStoreVO template : templates) {
+ isReady &= (Arrays.asList(validStates).contains(template.getState().toString()));
+ }
+ List<SnapshotDataStoreVO> snapshots = snapshotDataStoreDao.listByStoreId(srcDataStoreId, DataStoreRole.Image);
+ for (SnapshotDataStoreVO snapshot : snapshots) {
+ isReady &= (Arrays.asList(validStates).contains(snapshot.getState().toString()));
+ }
+ List<VolumeDataStoreVO> volumes = volumeDataStoreDao.listByStoreId(srcDataStoreId);
+ for (VolumeDataStoreVO volume : volumes) {
+ isReady &= (Arrays.asList(validStates).contains(volume.getState().toString()));
+ }
+ return isReady;
+ }
+
+ protected void checkIfCompleteMigrationPossible(ImageStoreService.MigrationPolicy policy, Long srcDataStoreId) {
+ if (policy == ImageStoreService.MigrationPolicy.COMPLETE) {
+ if (!filesReady(srcDataStoreId)) {
+ throw new CloudRuntimeException("Complete migration failed as there are data objects which are not Ready");
+ }
+ }
+ return;
+ }
+
+ protected Long getFileSize(DataObject file, Map<DataObject, Pair<List<SnapshotInfo>, Long>> snapshotChain) {
+ Long size = file.getSize();
+ Pair<List<SnapshotInfo>, Long> chain = snapshotChain.get(file);
+ if (file instanceof SnapshotInfo && chain.first() != null) {
+ size = chain.second();
+ }
+ return size;
+ }
+
+ /**
+ * Sorts the datastores in decreasing order of their free capacities, so as to make
+ * an informed decision of picking the datastore with maximum free capactiy for migration
+ */
+ protected List<Long> sortDataStores(Map<Long, Pair<Long, Long>> storageCapacities) {
+ List<Map.Entry<Long, Pair<Long, Long>>> list =
+ new LinkedList<Map.Entry<Long, Pair<Long, Long>>>((storageCapacities.entrySet()));
+
+ Collections.sort(list, new Comparator<Map.Entry<Long, Pair<Long, Long>>>() {
+ @Override
+ public int compare(Map.Entry<Long, Pair<Long, Long>> e1, Map.Entry<Long, Pair<Long, Long>> e2) {
+ return e2.getValue().first() > e1.getValue().first() ? 1 : -1;
+ }
+ });
+ HashMap<Long, Pair<Long, Long>> temp = new LinkedHashMap<>();
+ for (Map.Entry<Long, Pair<Long, Long>> value : list) {
+ temp.put(value.getKey(), value.getValue());
+ }
+
+ return new ArrayList<>(temp.keySet());
+ }
+
+ protected List<DataObject> getSortedValidSourcesList(DataStore srcDataStore, Map<DataObject, Pair<List<SnapshotInfo>, Long>> snapshotChains) {
+ List<DataObject> files = new ArrayList<>();
+ files.addAll(getAllValidTemplates(srcDataStore));
+ files.addAll(getAllValidSnapshotsAndChains(srcDataStore, snapshotChains));
+ files.addAll(getAllValidVolumes(srcDataStore));
+
+ files = sortFilesOnSize(files, snapshotChains);
+
+ return files;
+ }
+
+ protected List<DataObject> sortFilesOnSize(List<DataObject> files, Map<DataObject, Pair<List<SnapshotInfo>, Long>> snapshotChains) {
+ Collections.sort(files, new Comparator<DataObject>() {
+ @Override
+ public int compare(DataObject o1, DataObject o2) {
+ Long size1 = o1.getSize();
+ Long size2 = o2.getSize();
+ if (o1 instanceof SnapshotInfo) {
+ size1 = snapshotChains.get(o1).second();
+ }
+ if (o2 instanceof SnapshotInfo) {
+ size2 = snapshotChains.get(o2).second();
+ }
+ return (int) (size2 - size1);
+ }
+ });
+ return files;
+ }
+
+ // Gets list of all valid templates, i.e, templates in "Ready" state for migration
Review comment:
not a useful comment given the clear method name. extend the method name to be even more clear? make this a javadoc? remove?
##########
File path: engine/storage/image/src/main/java/org/apache/cloudstack/storage/image/StorageServiceImpl.java
##########
@@ -0,0 +1,165 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.cloudstack.storage.image;
+
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ExecutionException;
+
+import javax.inject.Inject;
+
+import org.apache.cloudstack.engine.subsystem.api.storage.CopyCommandResult;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataMotionService;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataObject;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.engine.subsystem.api.storage.ObjectInDataStoreStateMachine;
+import org.apache.cloudstack.engine.subsystem.api.storage.SecondaryStorageService;
+import org.apache.cloudstack.engine.subsystem.api.storage.SnapshotInfo;
+import org.apache.cloudstack.engine.subsystem.api.storage.TemplateInfo;
+import org.apache.cloudstack.framework.async.AsyncCallFuture;
+import org.apache.cloudstack.framework.async.AsyncCallbackDispatcher;
+import org.apache.cloudstack.framework.async.AsyncCompletionCallback;
+import org.apache.cloudstack.framework.async.AsyncRpcContext;
+import org.apache.cloudstack.storage.command.CopyCmdAnswer;
+import org.apache.cloudstack.storage.datastore.db.TemplateDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.TemplateDataStoreVO;
+import org.apache.log4j.Logger;
+
+import com.cloud.secstorage.CommandExecLogDao;
+import com.cloud.utils.Pair;
+
+public class StorageServiceImpl implements SecondaryStorageService {
Review comment:
Are you sure this is the best naming convention here? a `StorageService` sounds like more basic than a `SecondaryStorageService`. Is it inteded to implement more `Interface`s in the future?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-649995334
@Pearl1594 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-648962479
<b>Trillian test result (tid-1829)</b>
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 37993 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4053-t1829-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_direct_download.py
Intermittent failure detected: /marvin/tests/smoke/test_list_ids_parameter.py
Intermittent failure detected: /marvin/tests/smoke/test_templates.py
Intermittent failure detected: /marvin/tests/smoke/test_usage.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_router_nics.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_vpn.py
Intermittent failure detected: /marvin/tests/smoke/test_hostha_kvm.py
Smoke tests completed. 75 look OK, 8 have error(s)
Only failed tests results shown below:
Test | Result | Time (s) | Test File
--- | --- | --- | ---
ContextSuite context=TestDirectDownloadTemplates>:setup | `Error` | 0.00 | test_direct_download.py
ContextSuite context=TestListIdsParams>:setup | `Error` | 0.00 | test_list_ids_parameter.py
test_01_create_template | `Error` | 57.65 | test_templates.py
test_CreateTemplateWithDuplicateName | `Error` | 65.27 | test_templates.py
test_01_register_template_direct_download_flag | `Error` | 0.15 | test_templates.py
test_02_deploy_vm_from_direct_download_template | `Error` | 0.00 | test_templates.py
test_03_deploy_vm_wrong_checksum | `Error` | 0.14 | test_templates.py
ContextSuite context=TestTemplates>:setup | `Error` | 269.85 | test_templates.py
test_01_template_usage | `Error` | 38.99 | test_usage.py
ContextSuite context=TestVPCRedundancy>:setup | `Error` | 0.00 | test_vpc_redundant.py
ContextSuite context=TestVPCNics>:setup | `Error` | 0.00 | test_vpc_router_nics.py
ContextSuite context=TestRVPCSite2SiteVpn>:setup | `Error` | 0.00 | test_vpc_vpn.py
ContextSuite context=TestVPCSite2SiteVPNMultipleOptions>:setup | `Error` | 0.00 | test_vpc_vpn.py
ContextSuite context=TestVpcRemoteAccessVpn>:setup | `Error` | 0.00 | test_vpc_vpn.py
ContextSuite context=TestVpcSite2SiteVpn>:setup | `Error` | 0.00 | test_vpc_vpn.py
test_disable_oobm_ha_state_ineligible | `Error` | 0.24 | test_hostha_kvm.py
test_hostha_configure_default_driver | `Error` | 0.24 | test_hostha_kvm.py
test_hostha_enable_ha_when_host_disabled | `Error` | 0.26 | test_hostha_kvm.py
test_hostha_enable_ha_when_host_disconected | `Error` | 0.26 | test_hostha_kvm.py
test_hostha_enable_ha_when_host_in_maintenance | `Error` | 0.26 | test_hostha_kvm.py
test_hostha_kvm_host_degraded | `Error` | 0.29 | test_hostha_kvm.py
test_hostha_kvm_host_fencing | `Error` | 0.25 | test_hostha_kvm.py
test_hostha_kvm_host_recovering | `Error` | 0.26 | test_hostha_kvm.py
test_remove_ha_provider_not_possible | `Error` | 0.24 | test_hostha_kvm.py
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
rhtyd commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-682422479
@blueorangutan package
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
rhtyd commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-691863108
@blueorangutan package
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on pull request #4053: Secondary Storage Usage Improvements
Posted by GitBox <gi...@apache.org>.
rhtyd commented on pull request #4053:
URL: https://github.com/apache/cloudstack/pull/4053#issuecomment-687650236
Let's do around of test matrix before merging
@blueorangutan package
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org