You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by GitBox <gi...@apache.org> on 2022/02/11 12:28:56 UTC

[GitHub] [cloudstack] Pearl1594 opened a new pull request #5985: [VMware] Support for removal of NIC on IP disassociation on the VR

Pearl1594 opened a new pull request #5985:
URL: https://github.com/apache/cloudstack/pull/5985


   ### Description
   
   This PR fixes https://github.com/apache/cloudstack/issues/5934
   
   <!--- Describe your changes in DETAIL - And how has behaviour functionally changed. -->
   
   <!-- For new features, provide link to FS, dev ML discussion etc. -->
   <!-- In case of bug fix, the expected and actual behaviours, steps to reproduce. -->
   
   <!-- When "Fixes: #<id>" is specified, the issue/PR will automatically be closed when this PR gets merged -->
   <!-- For addressing multiple issues/PRs, use multiple "Fixes: #<id>" -->
   <!-- Fixes: # -->
   
   <!--- ********************************************************************************* -->
   <!--- NOTE: AUTOMATATION USES THE DESCRIPTIONS TO SET LABELS AND PRODUCE DOCUMENTATION. -->
   <!--- PLEASE PUT AN 'X' in only **ONE** box -->
   <!--- ********************************************************************************* -->
   
   ### Types of changes
   
   - [ ] Breaking change (fix or feature that would cause existing functionality to change)
   - [ ] New feature (non-breaking change which adds functionality)
   - [X] Bug fix (non-breaking change which fixes an issue)
   - [ ] Enhancement (improves an existing feature and functionality)
   - [ ] Cleanup (Code refactoring and cleanup, that may add test cases)
   
   ### Feature/Enhancement Scale or Bug Severity
   #### Bug Severity
   
   - [ ] BLOCKER
   - [ ] Critical
   - [ ] Major
   - [X] Minor
   - [ ] Trivial
   
   
   ### Screenshots (if appropriate):
   
   
   ### How Has This Been Tested?
   <!-- Please describe in detail how you tested your changes. -->
   <!-- Include details of your testing environment, and the tests you ran to -->
   <!-- see how your change affects other areas of the code, etc. -->
   (1) create isolated network
   (2) add new public IP range
   (3) acquire public IP in new IP range
   (4) create firewall rule. a new NIC is added to the network VR
   (5) release the public IP. - if it is the only IP on the NIC, the nic will be removed
   
   Test 1:
   Initial output of `ip a` with default NIC:
   ![image](https://user-images.githubusercontent.com/10495417/153587069-312dd496-7d69-46bc-b826-c3a97375b240.png)
   
   Acquired an IP in another Public IP range and added a firewall rule: New nic was added
   ![image](https://user-images.githubusercontent.com/10495417/153587236-c9f292f3-f23f-4525-87d0-0afc1610d362.png)
   
   On deletion of the firewall rule on the NIC, the nic is removed:
   ![image](https://user-images.githubusercontent.com/10495417/153590796-7d564ee2-a805-48fe-8a7b-5be9bbcd5d37.png)
   
   Test2:
   
   Add another Public IP range in the same VLAN as the one before 
   Acquire & add firewall rule for 2 IPs belonging to 2 different Public IP ranges but the same VLAN, such that there are 2 IPs on a NIC (check eth3)
   ![image](https://user-images.githubusercontent.com/10495417/153591161-bc41c7b2-ed9f-4825-bc13-8e00ff647036.png)
   
   Remove the rule from one of the IPs, and verify that the IP is disassociated from the VRs NIC, but the NIC isn't removed:
   ![image](https://user-images.githubusercontent.com/10495417/153591310-d5083115-00cc-4f6c-b42f-8de50d957f9e.png)
   
   
   <!-- Please read the [CONTRIBUTING](https://github.com/apache/cloudstack/blob/main/CONTRIBUTING.md) document -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [cloudstack] blueorangutan commented on pull request #5985: [VMware] Support for removal of NIC on IP disassociation on the VR

Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #5985:
URL: https://github.com/apache/cloudstack/pull/5985#issuecomment-1038716901


   @Pearl1594 a Trillian-Jenkins test job (centos7 mgmt + vmware-67u3) has been kicked to run smoke tests


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [cloudstack] weizhouapache commented on pull request #5985: [VMware] Support for removal of NIC on IP disassociation on the VR

Posted by GitBox <gi...@apache.org>.
weizhouapache commented on pull request #5985:
URL: https://github.com/apache/cloudstack/pull/5985#issuecomment-1038977302


   > agree @weizhouapache in addition I wonder why this apparently generic feature requires such a vmware specific implementation. Isn't a more hypevisor independent implementation required? I understand that it already works for Xen and KVM, so is there a way to fix it so it works for all, instead of adding ahypervisor specific code?
   
   @DaanHoogland 
   I do not know if there is way to do it.
   The command is sent to host or vcenter and the behaviors are different for hypervisors.
   Please note, this is not a feature. it is a part of IpAssocCommand processing.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [cloudstack] Pearl1594 commented on pull request #5985: [VMware] Support for removal of NIC on IP disassociation on the VR

Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on pull request #5985:
URL: https://github.com/apache/cloudstack/pull/5985#issuecomment-1040030084


   Thanks for the review/testing @weizhouapache . I'll fix the issue.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [cloudstack] blueorangutan commented on pull request #5985: [VMware] Support for removal of NIC on IP disassociation on the VR

Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #5985:
URL: https://github.com/apache/cloudstack/pull/5985#issuecomment-1039342104


   <b>Trillian test result (tid-3322)</b>
   Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server 7
   Total time taken: 36003 seconds
   Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr5985-t3322-vmware-67u3.zip
   Smoke tests completed. 90 look OK, 2 have errors
   Only failed tests results shown below:
   
   
   Test | Result | Time (s) | Test File
   --- | --- | --- | ---
   test_01_offline_migrate_VM_and_root_volume | `Error` | 94.64 | test_vm_life_cycle.py
   test_03_live_migrate_VM_with_two_data_disks | `Error` | 58.53 | test_vm_life_cycle.py
   test_06_disk_offering_strictness_false | `Error` | 143.41 | test_service_offerings.py
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [cloudstack] blueorangutan commented on pull request #5985: [VMware] Support for removal of NIC on IP disassociation on the VR

Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #5985:
URL: https://github.com/apache/cloudstack/pull/5985#issuecomment-1036178024


   @Pearl1594 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [cloudstack] blueorangutan commented on pull request #5985: [VMware] Support for removal of NIC on IP disassociation on the VR

Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #5985:
URL: https://github.com/apache/cloudstack/pull/5985#issuecomment-1039342104


   <b>Trillian test result (tid-3322)</b>
   Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server 7
   Total time taken: 36003 seconds
   Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr5985-t3322-vmware-67u3.zip
   Smoke tests completed. 90 look OK, 2 have errors
   Only failed tests results shown below:
   
   
   Test | Result | Time (s) | Test File
   --- | --- | --- | ---
   test_01_offline_migrate_VM_and_root_volume | `Error` | 94.64 | test_vm_life_cycle.py
   test_03_live_migrate_VM_with_two_data_disks | `Error` | 58.53 | test_vm_life_cycle.py
   test_06_disk_offering_strictness_false | `Error` | 143.41 | test_service_offerings.py
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [cloudstack] blueorangutan commented on pull request #5985: [VMware] Support for removal of NIC on IP disassociation on the VR

Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #5985:
URL: https://github.com/apache/cloudstack/pull/5985#issuecomment-1039126784


   @Pearl1594 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [cloudstack] blueorangutan commented on pull request #5985: [VMware] Support for removal of NIC on IP disassociation on the VR

Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #5985:
URL: https://github.com/apache/cloudstack/pull/5985#issuecomment-1042950225


   @weizhouapache a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [cloudstack] blueorangutan commented on pull request #5985: [VMware] Support for removal of NIC on IP disassociation on the VR

Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #5985:
URL: https://github.com/apache/cloudstack/pull/5985#issuecomment-1043015727


   @weizhouapache a Trillian-Jenkins test job (centos7 mgmt + vmware-70u3) has been kicked to run smoke tests


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [cloudstack] Pearl1594 commented on pull request #5985: [VMware] Support for removal of NIC on IP disassociation on the VR

Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on pull request #5985:
URL: https://github.com/apache/cloudstack/pull/5985#issuecomment-1038884016


   Sure @weizhouapache I am fine with moving to 4.17, I rebased it to 4.16, per a request made earlier : ) 
   @DaanHoogland I am not sure if there is a hypervisor agnostic way to solve this, as each hypervisor has it's own way of handling hot unplugging of NICs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [cloudstack] DaanHoogland commented on pull request #5985: [VMware] Support for removal of NIC on IP disassociation on the VR

Posted by GitBox <gi...@apache.org>.
DaanHoogland commented on pull request #5985:
URL: https://github.com/apache/cloudstack/pull/5985#issuecomment-1039043899


   ok, got it @weizhouapache @Pearl1594 . It doesn't feel ideal, but let's move on.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [cloudstack] Pearl1594 commented on pull request #5985: [VMware] Support for removal of NIC on IP disassociation on the VR

Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on pull request #5985:
URL: https://github.com/apache/cloudstack/pull/5985#issuecomment-1041180088


   @weizhouapache Could you please help verify if the fix addresses your comment. Thanks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [cloudstack] weizhouapache commented on pull request #5985: [VMware] Support for removal of NIC on IP disassociation on the VR

Posted by GitBox <gi...@apache.org>.
weizhouapache commented on pull request #5985:
URL: https://github.com/apache/cloudstack/pull/5985#issuecomment-1036630359


   code works well. @Pearl1594 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [cloudstack] blueorangutan commented on pull request #5985: [VMware] Support for removal of NIC on IP disassociation on the VR

Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #5985:
URL: https://github.com/apache/cloudstack/pull/5985#issuecomment-1038687007


   @Pearl1594 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [cloudstack] Pearl1594 commented on pull request #5985: [VMware] Support for removal of NIC on IP disassociation on the VR

Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on pull request #5985:
URL: https://github.com/apache/cloudstack/pull/5985#issuecomment-1038716337


   @blueorangutan test centos7 vmware-67u3


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [cloudstack] nvazquez merged pull request #5985: [VMware] Support for removal of NIC on IP disassociation on the VR

Posted by GitBox <gi...@apache.org>.
nvazquez merged pull request #5985:
URL: https://github.com/apache/cloudstack/pull/5985


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [cloudstack] weizhouapache commented on pull request #5985: [VMware] Support for removal of NIC on IP disassociation on the VR

Posted by GitBox <gi...@apache.org>.
weizhouapache commented on pull request #5985:
URL: https://github.com/apache/cloudstack/pull/5985#issuecomment-1039051033


   > ok, got it @weizhouapache @Pearl1594 . It doesn't feel ideal, but let's move on.
   
   @DaanHoogland 
   there are many (hundreds of) commands which are processed in hypervisor plugins.
   if you have an idea how to improve the processes and remove the hypervisor specific codes, it would be great.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [cloudstack] Pearl1594 commented on pull request #5985: [VMware] Support for removal of NIC on IP disassociation on the VR

Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on pull request #5985:
URL: https://github.com/apache/cloudstack/pull/5985#issuecomment-1038708488


   @nvazquez I've addressed the comments. @weizhouapache I've rebased the branch off 4.16. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [cloudstack] weizhouapache edited a comment on pull request #5985: [VMware] Support for removal of NIC on IP disassociation on the VR

Posted by GitBox <gi...@apache.org>.
weizhouapache edited a comment on pull request #5985:
URL: https://github.com/apache/cloudstack/pull/5985#issuecomment-1036630359


   code works well. @Pearl1594 
   could you please rebase to 4.16 ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [cloudstack] weizhouapache commented on a change in pull request #5985: [VMware] Support for removal of NIC on IP disassociation on the VR

Posted by GitBox <gi...@apache.org>.
weizhouapache commented on a change in pull request #5985:
URL: https://github.com/apache/cloudstack/pull/5985#discussion_r807635754



##########
File path: plugins/hypervisors/vmware/src/main/java/com/cloud/hypervisor/vmware/resource/VmwareResource.java
##########
@@ -1622,6 +1589,83 @@ private ExecutionResult prepareNetworkElementCommand(IpAssocCommand cmd) {
         return new ExecutionResult(true, null);
     }
 
+    private ExecutionResult cleanupNetworkElementCommand(IpAssocCommand cmd) {
+        VmwareContext context = getServiceContext();
+        try {
+            VmwareHypervisorHost hyperHost = getHyperHost(context);
+            IpAddressTO[] ips = cmd.getIpAddresses();
+            String routerName = cmd.getAccessDetail(NetworkElementCommand.ROUTER_NAME);
+
+            VirtualMachineMO vmMo = hyperHost.findVmOnHyperHost(routerName);
+            // command may sometimes be redirected to a wrong host, we relax
+            // the check and will try to find it within datacenter
+            if (vmMo == null) {
+                if (hyperHost instanceof HostMO) {
+                    final DatacenterMO dcMo = new DatacenterMO(context, hyperHost.getHyperHostDatacenter());
+                    vmMo = dcMo.findVm(routerName);
+                }
+            }
+
+            if (vmMo == null) {
+                String msg = String.format("Router %s no longer exists to execute IPAssoc command ", routerName);
+                s_logger.error(msg);
+                throw new Exception(msg);
+            }
+            final String lastIp = cmd.getAccessDetail(NetworkElementCommand.NETWORK_PUB_LAST_IP);
+            if (ips.length == 1 && !ips[0].isAdd()) {
+                Pair<VirtualDevice, Integer> nicInfo = getVirtualDevice(vmMo, ips[0]);
+
+                if (nicInfo.first() == null && lastIp.equalsIgnoreCase("true")) {
+                    if (nicInfo.second() == 2) {
+                        return new ExecutionResult(false, "Unable to remove eth2 in network VR because it is the public NIC of source NAT");

Review comment:
       @Pearl1594 
   for eth2, we can return a sucess result.

##########
File path: plugins/hypervisors/vmware/src/main/java/com/cloud/hypervisor/vmware/resource/VmwareResource.java
##########
@@ -1622,6 +1589,83 @@ private ExecutionResult prepareNetworkElementCommand(IpAssocCommand cmd) {
         return new ExecutionResult(true, null);
     }
 
+    private ExecutionResult cleanupNetworkElementCommand(IpAssocCommand cmd) {
+        VmwareContext context = getServiceContext();
+        try {
+            VmwareHypervisorHost hyperHost = getHyperHost(context);
+            IpAddressTO[] ips = cmd.getIpAddresses();
+            String routerName = cmd.getAccessDetail(NetworkElementCommand.ROUTER_NAME);
+
+            VirtualMachineMO vmMo = hyperHost.findVmOnHyperHost(routerName);
+            // command may sometimes be redirected to a wrong host, we relax
+            // the check and will try to find it within datacenter
+            if (vmMo == null) {
+                if (hyperHost instanceof HostMO) {
+                    final DatacenterMO dcMo = new DatacenterMO(context, hyperHost.getHyperHostDatacenter());
+                    vmMo = dcMo.findVm(routerName);
+                }
+            }
+
+            if (vmMo == null) {
+                String msg = String.format("Router %s no longer exists to execute IPAssoc command ", routerName);
+                s_logger.error(msg);
+                throw new Exception(msg);
+            }
+            final String lastIp = cmd.getAccessDetail(NetworkElementCommand.NETWORK_PUB_LAST_IP);
+            if (ips.length == 1 && !ips[0].isAdd()) {

Review comment:
       @Pearl1594 
   sometimes there are multiple public Ip addresses in the IpAssocCommand
   maybe it is better to loop all ip addresses, like kvm/xenserver plugins do.

##########
File path: plugins/hypervisors/vmware/src/main/java/com/cloud/hypervisor/vmware/resource/VmwareResource.java
##########
@@ -1622,6 +1589,83 @@ private ExecutionResult prepareNetworkElementCommand(IpAssocCommand cmd) {
         return new ExecutionResult(true, null);
     }
 
+    private ExecutionResult cleanupNetworkElementCommand(IpAssocCommand cmd) {
+        VmwareContext context = getServiceContext();
+        try {
+            VmwareHypervisorHost hyperHost = getHyperHost(context);
+            IpAddressTO[] ips = cmd.getIpAddresses();
+            String routerName = cmd.getAccessDetail(NetworkElementCommand.ROUTER_NAME);
+
+            VirtualMachineMO vmMo = hyperHost.findVmOnHyperHost(routerName);
+            // command may sometimes be redirected to a wrong host, we relax
+            // the check and will try to find it within datacenter
+            if (vmMo == null) {
+                if (hyperHost instanceof HostMO) {
+                    final DatacenterMO dcMo = new DatacenterMO(context, hyperHost.getHyperHostDatacenter());
+                    vmMo = dcMo.findVm(routerName);
+                }
+            }
+
+            if (vmMo == null) {
+                String msg = String.format("Router %s no longer exists to execute IPAssoc command ", routerName);
+                s_logger.error(msg);
+                throw new Exception(msg);
+            }
+            final String lastIp = cmd.getAccessDetail(NetworkElementCommand.NETWORK_PUB_LAST_IP);
+            if (ips.length == 1 && !ips[0].isAdd()) {
+                Pair<VirtualDevice, Integer> nicInfo = getVirtualDevice(vmMo, ips[0]);
+
+                if (nicInfo.first() == null && lastIp.equalsIgnoreCase("true")) {
+                    if (nicInfo.second() == 2) {
+                        return new ExecutionResult(false, "Unable to remove eth2 in network VR because it is the public NIC of source NAT");
+                    }
+                    return new ExecutionResult(false, "Couldn't find NIC");
+                }
+                configureNicDevice(vmMo, nicInfo.first(), VirtualDeviceConfigSpecOperation.REMOVE, "unplugNicCommand");
+            }
+        } catch (Throwable e) {
+            s_logger.error("Unexpected exception: " + e.toString() + " will shortcut rest of IPAssoc commands", e);
+            return new ExecutionResult(false, e.toString());
+        }
+        return new ExecutionResult(true, null);
+    }
+
+    private Pair<VirtualDevice, Integer> getVirtualDevice(VirtualMachineMO vmMo, IpAddressTO ip) throws Exception {
+        NicTO nicTO = ip.getNicTO();
+        URI broadcastUri = BroadcastDomainType.fromString(ip.getBroadcastUri());
+        if (BroadcastDomainType.getSchemeValue(broadcastUri) != BroadcastDomainType.Vlan) {
+            throw new InternalErrorException(String.format("Unable to assign a public IP to a VIF on network %s", ip.getBroadcastUri()));
+        }
+        String vlanId = BroadcastDomainType.getValue(broadcastUri);
+
+        String publicNetworkName = HypervisorHostHelper.getPublicNetworkNamePrefix(vlanId);
+        Pair<Integer, VirtualDevice> publicNicInfo = vmMo.getNicDeviceIndex(publicNetworkName);
+
+        if (publicNicInfo.first() == 2) {

Review comment:
       this is not needed ,because it is already handled in line 1619/1620




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [cloudstack] blueorangutan commented on pull request #5985: [VMware] Support for removal of NIC on IP disassociation on the VR

Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #5985:
URL: https://github.com/apache/cloudstack/pull/5985#issuecomment-1043004549


   @nvazquez a Trillian-Jenkins test job (centos7 mgmt + vmware-67u3) has been kicked to run smoke tests


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [cloudstack] nvazquez commented on a change in pull request #5985: [VMware] Support for removal of NIC on IP disassociation on the VR

Posted by GitBox <gi...@apache.org>.
nvazquez commented on a change in pull request #5985:
URL: https://github.com/apache/cloudstack/pull/5985#discussion_r804896865



##########
File path: plugins/hypervisors/vmware/src/main/java/com/cloud/hypervisor/vmware/resource/VmwareResource.java
##########
@@ -1622,6 +1625,69 @@ private ExecutionResult prepareNetworkElementCommand(IpAssocCommand cmd) {
         return new ExecutionResult(true, null);
     }
 
+    private ExecutionResult cleanupNetworkElementCommand(IpAssocCommand cmd) {
+        VmwareContext context = getServiceContext();
+        try {
+            VmwareHypervisorHost hyperHost = getHyperHost(context);
+            IpAddressTO[] ips = cmd.getIpAddresses();
+            String routerName = cmd.getAccessDetail(NetworkElementCommand.ROUTER_NAME);
+
+            VirtualMachineMO vmMo = hyperHost.findVmOnHyperHost(routerName);
+            // command may sometimes be redirect to a wrong host, we relax
+            // the check and will try to find it within cluster
+            if (vmMo == null) {
+                if (hyperHost instanceof HostMO) {
+                    ClusterMO clusterMo = new ClusterMO(hyperHost.getContext(), ((HostMO) hyperHost).getParentMor());

Review comment:
       What about datacenter instead of cluster search?

##########
File path: plugins/hypervisors/vmware/src/main/java/com/cloud/hypervisor/vmware/resource/VmwareResource.java
##########
@@ -1622,6 +1625,69 @@ private ExecutionResult prepareNetworkElementCommand(IpAssocCommand cmd) {
         return new ExecutionResult(true, null);
     }
 
+    private ExecutionResult cleanupNetworkElementCommand(IpAssocCommand cmd) {
+        VmwareContext context = getServiceContext();
+        try {
+            VmwareHypervisorHost hyperHost = getHyperHost(context);
+            IpAddressTO[] ips = cmd.getIpAddresses();
+            String routerName = cmd.getAccessDetail(NetworkElementCommand.ROUTER_NAME);
+
+            VirtualMachineMO vmMo = hyperHost.findVmOnHyperHost(routerName);
+            // command may sometimes be redirect to a wrong host, we relax
+            // the check and will try to find it within cluster
+            if (vmMo == null) {
+                if (hyperHost instanceof HostMO) {
+                    ClusterMO clusterMo = new ClusterMO(hyperHost.getContext(), ((HostMO) hyperHost).getParentMor());
+                    vmMo = clusterMo.findVmOnHyperHost(routerName);
+                }
+            }
+
+            if (vmMo == null) {
+                String msg = String.format("Router %s no longer exists to execute IPAssoc command ", routerName);
+                s_logger.error(msg);
+                throw new Exception(msg);
+            }
+
+            for (IpAddressTO ip : ips) {

Review comment:
       I think this for loop is not needed since we ensure the logic is executed only when it has one item as is not add. Could be refactored to pre-checks and logic to reduce indentation




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [cloudstack] weizhouapache commented on a change in pull request #5985: [VMware] Support for removal of NIC on IP disassociation on the VR

Posted by GitBox <gi...@apache.org>.
weizhouapache commented on a change in pull request #5985:
URL: https://github.com/apache/cloudstack/pull/5985#discussion_r807953789



##########
File path: plugins/hypervisors/vmware/src/main/java/com/cloud/hypervisor/vmware/resource/VmwareResource.java
##########
@@ -1622,6 +1589,83 @@ private ExecutionResult prepareNetworkElementCommand(IpAssocCommand cmd) {
         return new ExecutionResult(true, null);
     }
 
+    private ExecutionResult cleanupNetworkElementCommand(IpAssocCommand cmd) {
+        VmwareContext context = getServiceContext();
+        try {
+            VmwareHypervisorHost hyperHost = getHyperHost(context);
+            IpAddressTO[] ips = cmd.getIpAddresses();
+            String routerName = cmd.getAccessDetail(NetworkElementCommand.ROUTER_NAME);
+
+            VirtualMachineMO vmMo = hyperHost.findVmOnHyperHost(routerName);
+            // command may sometimes be redirected to a wrong host, we relax
+            // the check and will try to find it within datacenter
+            if (vmMo == null) {
+                if (hyperHost instanceof HostMO) {
+                    final DatacenterMO dcMo = new DatacenterMO(context, hyperHost.getHyperHostDatacenter());
+                    vmMo = dcMo.findVm(routerName);
+                }
+            }
+
+            if (vmMo == null) {
+                String msg = String.format("Router %s no longer exists to execute IPAssoc command ", routerName);
+                s_logger.error(msg);
+                throw new Exception(msg);
+            }
+            final String lastIp = cmd.getAccessDetail(NetworkElementCommand.NETWORK_PUB_LAST_IP);
+            if (ips.length == 1 && !ips[0].isAdd()) {
+                Pair<VirtualDevice, Integer> nicInfo = getVirtualDevice(vmMo, ips[0]);
+
+                if (nicInfo.first() == null && lastIp.equalsIgnoreCase("true")) {
+                    if (nicInfo.second() == 2) {
+                        return new ExecutionResult(false, "Unable to remove eth2 in network VR because it is the public NIC of source NAT");
+                    }
+                    return new ExecutionResult(false, "Couldn't find NIC");
+                }
+                configureNicDevice(vmMo, nicInfo.first(), VirtualDeviceConfigSpecOperation.REMOVE, "unplugNicCommand");

Review comment:
       @Pearl1594 
   nic can be unplugged only if `lastIp.equalsIgnoreCase("true")`
   
   https://github.com/apache/cloudstack/blob/593f1b698836749856f38f706b30b23a660d26a3/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java#L2129-L2136
   

##########
File path: plugins/hypervisors/vmware/src/main/java/com/cloud/hypervisor/vmware/resource/VmwareResource.java
##########
@@ -1622,6 +1589,83 @@ private ExecutionResult prepareNetworkElementCommand(IpAssocCommand cmd) {
         return new ExecutionResult(true, null);
     }
 
+    private ExecutionResult cleanupNetworkElementCommand(IpAssocCommand cmd) {
+        VmwareContext context = getServiceContext();
+        try {
+            VmwareHypervisorHost hyperHost = getHyperHost(context);
+            IpAddressTO[] ips = cmd.getIpAddresses();
+            String routerName = cmd.getAccessDetail(NetworkElementCommand.ROUTER_NAME);
+
+            VirtualMachineMO vmMo = hyperHost.findVmOnHyperHost(routerName);
+            // command may sometimes be redirected to a wrong host, we relax
+            // the check and will try to find it within datacenter
+            if (vmMo == null) {
+                if (hyperHost instanceof HostMO) {
+                    final DatacenterMO dcMo = new DatacenterMO(context, hyperHost.getHyperHostDatacenter());
+                    vmMo = dcMo.findVm(routerName);
+                }
+            }
+
+            if (vmMo == null) {
+                String msg = String.format("Router %s no longer exists to execute IPAssoc command ", routerName);
+                s_logger.error(msg);
+                throw new Exception(msg);
+            }
+            final String lastIp = cmd.getAccessDetail(NetworkElementCommand.NETWORK_PUB_LAST_IP);
+            if (ips.length == 1 && !ips[0].isAdd()) {
+                Pair<VirtualDevice, Integer> nicInfo = getVirtualDevice(vmMo, ips[0]);
+
+                if (nicInfo.first() == null && lastIp.equalsIgnoreCase("true")) {
+                    if (nicInfo.second() == 2) {
+                        return new ExecutionResult(false, "Unable to remove eth2 in network VR because it is the public NIC of source NAT");
+                    }
+                    return new ExecutionResult(false, "Couldn't find NIC");

Review comment:
       return a success if no need to unplug the nic ?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [cloudstack] Pearl1594 commented on pull request #5985: [VMware] Support for removal of NIC on IP disassociation on the VR

Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on pull request #5985:
URL: https://github.com/apache/cloudstack/pull/5985#issuecomment-1039126405


   @blueorangutan package


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [cloudstack] blueorangutan commented on pull request #5985: [VMware] Support for removal of NIC on IP disassociation on the VR

Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #5985:
URL: https://github.com/apache/cloudstack/pull/5985#issuecomment-1036785065


   <b>Trillian test result (tid-3298)</b>
   Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server 7
   Total time taken: 36156 seconds
   Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr5985-t3298-vmware-67u3.zip
   Smoke tests completed. 90 look OK, 2 have errors
   Only failed tests results shown below:
   
   
   Test | Result | Time (s) | Test File
   --- | --- | --- | ---
   test_01_offline_migrate_VM_and_root_volume | `Error` | 101.04 | test_vm_life_cycle.py
   test_03_live_migrate_VM_with_two_data_disks | `Error` | 61.35 | test_vm_life_cycle.py
   test_06_disk_offering_strictness_false | `Error` | 142.03 | test_service_offerings.py
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [cloudstack] weizhouapache commented on pull request #5985: [VMware] Support for removal of NIC on IP disassociation on the VR

Posted by GitBox <gi...@apache.org>.
weizhouapache commented on pull request #5985:
URL: https://github.com/apache/cloudstack/pull/5985#issuecomment-1039964422


   @Pearl1594 
   in my testing, if public ips are used for port forwarding or load balancer, everything seems ok.
   If public ips are used for static nat, the interface will be removed even if there is still public ip in use (for example, pf or lb or nat). This issue happens with eth2 as well which is the interface for source nat.
   
   I think we should use similar code like
   ```
               final String lastIp = cmd.getAccessDetail(NetworkElementCommand.NETWORK_PUB_LAST_IP);
               for (final IpAddressTO ip : ips) {
                   if (!StringUtils.equalsIgnoreCase(lastIp, "true") || ip.isAdd()) {
                       continue;
                   }
   ```
   and
   ```
           if (publicNicInfo.first() == 2) {
               s_logger.debug("Do not remove eth2 in network VR because it is the public NIC of source NAT.");
               continue;
           }
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [cloudstack] weizhouapache commented on pull request #5985: [VMware] Support for removal of NIC on IP disassociation on the VR

Posted by GitBox <gi...@apache.org>.
weizhouapache commented on pull request #5985:
URL: https://github.com/apache/cloudstack/pull/5985#issuecomment-1041541256


   @Pearl1594 
   the following code works in my tesing
   ```
               final String lastIp = cmd.getAccessDetail(NetworkElementCommand.NETWORK_PUB_LAST_IP);
               if (ips.length == 1 && !ips[0].isAdd()) {
                   Pair<VirtualDevice, Integer> nicInfo = getVirtualDevice(vmMo, ips[0]);
   
                   if (nicInfo.first() != null && nicInfo.second() != 2 && lastIp.equalsIgnoreCase("true")) {
                       configureNicDevice(vmMo, nicInfo.first(), VirtualDeviceConfigSpecOperation.REMOVE, "unplugNicCommand");
                   }
               }
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [cloudstack] weizhouapache edited a comment on pull request #5985: [VMware] Support for removal of NIC on IP disassociation on the VR

Posted by GitBox <gi...@apache.org>.
weizhouapache edited a comment on pull request #5985:
URL: https://github.com/apache/cloudstack/pull/5985#issuecomment-1039964422


   @Pearl1594 
   in my testing, if public ips are used for port forwarding or load balancer, everything seems ok.
   If public ips are used for static nat, the interface will be removed even if there is still public ip in use (for example, pf or lb or nat). This issue happens with eth2 as well which is the interface for source nat.
   
   steps to reproduce the issue
   (1) acquire public IP in source NAT ip range or additional IP range
   (2) enable static nat
   (3) disbale static nat
   
   expected result: interface is plugged when enable static nat if needed. interface is unplugged if IP is the last attached IP on the interface, when disable static nat.
   actual result: interface is unplugged every time when disable static nat, no matter if it is the last IP.
   
   I think we should use similar code like
   ```
               final String lastIp = cmd.getAccessDetail(NetworkElementCommand.NETWORK_PUB_LAST_IP);
               for (final IpAddressTO ip : ips) {
                   if (!StringUtils.equalsIgnoreCase(lastIp, "true") || ip.isAdd()) {
                       continue;
                   }
   ```
   and
   ```
           if (publicNicInfo.first() == 2) {
               s_logger.debug("Do not remove eth2 in network VR because it is the public NIC of source NAT.");
               continue;
           }
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [cloudstack] blueorangutan commented on pull request #5985: [VMware] Support for removal of NIC on IP disassociation on the VR

Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #5985:
URL: https://github.com/apache/cloudstack/pull/5985#issuecomment-1043711386


   <b>Trillian test result (tid-3382)</b>
   Environment: vmware-70u3 (x2), Advanced Networking with Mgmt server 7
   Total time taken: 38052 seconds
   Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr5985-t3382-vmware-70u3.zip
   Smoke tests completed. 92 look OK, 0 have errors
   Only failed tests results shown below:
   
   
   Test | Result | Time (s) | Test File
   --- | --- | --- | ---
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [cloudstack] blueorangutan commented on pull request #5985: [VMware] Support for removal of NIC on IP disassociation on the VR

Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #5985:
URL: https://github.com/apache/cloudstack/pull/5985#issuecomment-1036209726


   Packaging result: :heavy_check_mark: el7 :heavy_check_mark: el8 :heavy_check_mark: debian :heavy_check_mark: suse15. SL-JID 2578


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [cloudstack] Pearl1594 commented on pull request #5985: [VMware] Support for removal of NIC on IP disassociation on the VR

Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on pull request #5985:
URL: https://github.com/apache/cloudstack/pull/5985#issuecomment-1036210441


   @blueorangutan test centos7 vmware-67u3


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [cloudstack] blueorangutan commented on pull request #5985: [VMware] Support for removal of NIC on IP disassociation on the VR

Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #5985:
URL: https://github.com/apache/cloudstack/pull/5985#issuecomment-1039174111


   Packaging result: :heavy_check_mark: el7 :heavy_check_mark: el8 :heavy_check_mark: debian :heavy_check_mark: suse15. SL-JID 2601


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [cloudstack] weizhouapache edited a comment on pull request #5985: [VMware] Support for removal of NIC on IP disassociation on the VR

Posted by GitBox <gi...@apache.org>.
weizhouapache edited a comment on pull request #5985:
URL: https://github.com/apache/cloudstack/pull/5985#issuecomment-1036630359


   code works well. @Pearl1594 
   could you please retarget to 4.16 ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [cloudstack] Pearl1594 commented on pull request #5985: [VMware] Support for removal of NIC on IP disassociation on the VR

Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on pull request #5985:
URL: https://github.com/apache/cloudstack/pull/5985#issuecomment-1040030084


   Thanks for the review/testing @weizhouapache . I'll fix the issue.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [cloudstack] DaanHoogland commented on pull request #5985: [VMware] Support for removal of NIC on IP disassociation on the VR

Posted by GitBox <gi...@apache.org>.
DaanHoogland commented on pull request #5985:
URL: https://github.com/apache/cloudstack/pull/5985#issuecomment-1039053603


   > > ok, got it @weizhouapache @Pearl1594 . It doesn't feel ideal, but let's move on.
   > 
   > @DaanHoogland there are many (hundreds of) commands which are processed in hypervisor plugins. if you have an idea how to improve the processes and remove the hypervisor specific codes, it would be great.
   
   always on the lookout @weizhouapache ;)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [cloudstack] Pearl1594 commented on pull request #5985: [VMware] Support for removal of NIC on IP disassociation on the VR

Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on pull request #5985:
URL: https://github.com/apache/cloudstack/pull/5985#issuecomment-1036177251


   @blueorangutan package


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [cloudstack] DaanHoogland commented on pull request #5985: [VMware] Support for removal of NIC on IP disassociation on the VR

Posted by GitBox <gi...@apache.org>.
DaanHoogland commented on pull request #5985:
URL: https://github.com/apache/cloudstack/pull/5985#issuecomment-1038877919


   agree @weizhouapache 
   in addition I wonder why this apparently generic feature requires such a vmware specific implementation. Isn't a more hypevisor independent implementation required?
   I understand that it already works for Xen and KVM, so is there a way to fix it so it works for all, instead of adding ahypervisor specific code?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [cloudstack] blueorangutan commented on pull request #5985: [VMware] Support for removal of NIC on IP disassociation on the VR

Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #5985:
URL: https://github.com/apache/cloudstack/pull/5985#issuecomment-1043694031


   <b>Trillian test result (tid-3378)</b>
   Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server 7
   Total time taken: 37145 seconds
   Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr5985-t3378-vmware-67u3.zip
   Smoke tests completed. 92 look OK, 0 have errors
   Only failed tests results shown below:
   
   
   Test | Result | Time (s) | Test File
   --- | --- | --- | ---
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [cloudstack] DaanHoogland commented on a change in pull request #5985: [VMware] Support for removal of NIC on IP disassociation on the VR

Posted by GitBox <gi...@apache.org>.
DaanHoogland commented on a change in pull request #5985:
URL: https://github.com/apache/cloudstack/pull/5985#discussion_r805800994



##########
File path: plugins/hypervisors/vmware/src/main/java/com/cloud/hypervisor/vmware/resource/VmwareResource.java
##########
@@ -1622,6 +1589,72 @@ private ExecutionResult prepareNetworkElementCommand(IpAssocCommand cmd) {
         return new ExecutionResult(true, null);
     }
 
+    private ExecutionResult cleanupNetworkElementCommand(IpAssocCommand cmd) {
+        VmwareContext context = getServiceContext();
+        try {
+            VmwareHypervisorHost hyperHost = getHyperHost(context);
+            IpAddressTO[] ips = cmd.getIpAddresses();
+            String routerName = cmd.getAccessDetail(NetworkElementCommand.ROUTER_NAME);
+
+            VirtualMachineMO vmMo = hyperHost.findVmOnHyperHost(routerName);
+            // command may sometimes be redirect to a wrong host, we relax
+            // the check and will try to find it within cluster
+            if (vmMo == null) {
+                if (hyperHost instanceof HostMO) {
+                    final DatacenterMO dcMo = new DatacenterMO(context, hyperHost.getHyperHostDatacenter());
+                    vmMo = dcMo.findVm(routerName);
+                }
+            }
+
+            if (vmMo == null) {
+                String msg = String.format("Router %s no longer exists to execute IPAssoc command ", routerName);
+                s_logger.error(msg);
+                throw new Exception(msg);
+            }

Review comment:
       can you factor out this bit?

##########
File path: plugins/hypervisors/vmware/src/main/java/com/cloud/hypervisor/vmware/resource/VmwareResource.java
##########
@@ -1622,6 +1589,72 @@ private ExecutionResult prepareNetworkElementCommand(IpAssocCommand cmd) {
         return new ExecutionResult(true, null);
     }
 
+    private ExecutionResult cleanupNetworkElementCommand(IpAssocCommand cmd) {
+        VmwareContext context = getServiceContext();
+        try {
+            VmwareHypervisorHost hyperHost = getHyperHost(context);
+            IpAddressTO[] ips = cmd.getIpAddresses();
+            String routerName = cmd.getAccessDetail(NetworkElementCommand.ROUTER_NAME);
+
+            VirtualMachineMO vmMo = hyperHost.findVmOnHyperHost(routerName);
+            // command may sometimes be redirect to a wrong host, we relax
+            // the check and will try to find it within cluster
+            if (vmMo == null) {
+                if (hyperHost instanceof HostMO) {
+                    final DatacenterMO dcMo = new DatacenterMO(context, hyperHost.getHyperHostDatacenter());
+                    vmMo = dcMo.findVm(routerName);
+                }
+            }
+
+            if (vmMo == null) {
+                String msg = String.format("Router %s no longer exists to execute IPAssoc command ", routerName);
+                s_logger.error(msg);
+                throw new Exception(msg);
+            }
+
+            if (ips.length == 1 && !ips[0].isAdd()) {
+                IpAddressTO ip = ips[0];
+                NicTO nicTO = ip.getNicTO();
+                URI broadcastUri = BroadcastDomainType.fromString(ip.getBroadcastUri());
+                if (BroadcastDomainType.getSchemeValue(broadcastUri) != BroadcastDomainType.Vlan) {
+                    throw new InternalErrorException(String.format("Unable to assign a public IP to a VIF on network %s", ip.getBroadcastUri()));
+                }
+                String vlanId = BroadcastDomainType.getValue(broadcastUri);
+
+                String publicNetworkName = HypervisorHostHelper.getPublicNetworkNamePrefix(vlanId);
+                Pair<Integer, VirtualDevice> publicNicInfo = vmMo.getNicDeviceIndex(publicNetworkName);
+
+                if (s_logger.isDebugEnabled()) {
+                    s_logger.debug(String.format("Find public NIC index, public network name: %s , index: %s", publicNetworkName, publicNicInfo.first()));
+                }
+
+                VirtualDevice nic = findVirtualNicDevice(vmMo, nicTO.getMac());
+
+                if (nic == null) {
+                    return new ExecutionResult(false, "Couldn't find NIC");
+                }
+                configureNicDevice(vmMo, nic, VirtualDeviceConfigSpecOperation.REMOVE);
+            }

Review comment:
       can you factor out this bit? (maybe if even smaller/nested chunks)

##########
File path: plugins/hypervisors/vmware/src/main/java/com/cloud/hypervisor/vmware/resource/VmwareResource.java
##########
@@ -1622,6 +1589,72 @@ private ExecutionResult prepareNetworkElementCommand(IpAssocCommand cmd) {
         return new ExecutionResult(true, null);
     }
 
+    private ExecutionResult cleanupNetworkElementCommand(IpAssocCommand cmd) {
+        VmwareContext context = getServiceContext();
+        try {
+            VmwareHypervisorHost hyperHost = getHyperHost(context);
+            IpAddressTO[] ips = cmd.getIpAddresses();
+            String routerName = cmd.getAccessDetail(NetworkElementCommand.ROUTER_NAME);
+
+            VirtualMachineMO vmMo = hyperHost.findVmOnHyperHost(routerName);
+            // command may sometimes be redirect to a wrong host, we relax
+            // the check and will try to find it within cluster
+            if (vmMo == null) {
+                if (hyperHost instanceof HostMO) {
+                    final DatacenterMO dcMo = new DatacenterMO(context, hyperHost.getHyperHostDatacenter());
+                    vmMo = dcMo.findVm(routerName);
+                }
+            }
+
+            if (vmMo == null) {
+                String msg = String.format("Router %s no longer exists to execute IPAssoc command ", routerName);
+                s_logger.error(msg);
+                throw new Exception(msg);
+            }
+
+            if (ips.length == 1 && !ips[0].isAdd()) {
+                IpAddressTO ip = ips[0];
+                NicTO nicTO = ip.getNicTO();
+                URI broadcastUri = BroadcastDomainType.fromString(ip.getBroadcastUri());
+                if (BroadcastDomainType.getSchemeValue(broadcastUri) != BroadcastDomainType.Vlan) {
+                    throw new InternalErrorException(String.format("Unable to assign a public IP to a VIF on network %s", ip.getBroadcastUri()));
+                }
+                String vlanId = BroadcastDomainType.getValue(broadcastUri);
+
+                String publicNetworkName = HypervisorHostHelper.getPublicNetworkNamePrefix(vlanId);
+                Pair<Integer, VirtualDevice> publicNicInfo = vmMo.getNicDeviceIndex(publicNetworkName);
+
+                if (s_logger.isDebugEnabled()) {
+                    s_logger.debug(String.format("Find public NIC index, public network name: %s , index: %s", publicNetworkName, publicNicInfo.first()));
+                }
+
+                VirtualDevice nic = findVirtualNicDevice(vmMo, nicTO.getMac());
+
+                if (nic == null) {
+                    return new ExecutionResult(false, "Couldn't find NIC");
+                }
+                configureNicDevice(vmMo, nic, VirtualDeviceConfigSpecOperation.REMOVE);
+            }
+        } catch (Throwable e) {
+            s_logger.error("Unexpected exception: " + e.toString() + " will shortcut rest of IPAssoc commands", e);
+            return new ExecutionResult(false, e.toString());
+        }
+        return new ExecutionResult(true, null);
+    }
+
+    private void configureNicDevice(VirtualMachineMO vmMo, VirtualDevice nic, VirtualDeviceConfigSpecOperation operation) throws Exception {
+        VirtualMachineConfigSpec vmConfigSpec = new VirtualMachineConfigSpec();
+        VirtualDeviceConfigSpec deviceConfigSpec = new VirtualDeviceConfigSpec();
+        deviceConfigSpec.setDevice(nic);
+        deviceConfigSpec.setOperation(operation);
+
+
+        vmConfigSpec.getDeviceChange().add(deviceConfigSpec);
+        if (!vmMo.configureVm(vmConfigSpec)) {
+            throw new Exception("Failed to configure devices when running unplugNicCommand");

Review comment:
       this is not always true I think,
   ```suggestion
               throw new Exception("Failed to configure devices when running a (un/re)PlugNicCommand");
   ```
   maybe even pass it as parameter?
   good extraction otherwise.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [cloudstack] weizhouapache edited a comment on pull request #5985: [VMware] Support for removal of NIC on IP disassociation on the VR

Posted by GitBox <gi...@apache.org>.
weizhouapache edited a comment on pull request #5985:
URL: https://github.com/apache/cloudstack/pull/5985#issuecomment-1039964422


   @Pearl1594 
   in my testing, if public ips are used for port forwarding or load balancer, everything seems ok.
   If public ips are used for static nat, the interface will be removed even if there is still public ip in use (for example, pf or lb or nat). This issue happens with eth2 as well which is the interface for source nat.
   
   steps to reproduce the issue
   (1) acquire public IP in source NAT ip range or additional IP range
   (2) enable static nat
   (3) disbale static nat
   
   expected result: interface is plugged when enable static nat if needed. interface is unplugged if IP is the last attached IP on the interface, when disable static nat.
   actual result: interface is unplugged every time when disable static nat, no matter if it is the last IP.
   
   I think we should use similar code like
   ```
               final String lastIp = cmd.getAccessDetail(NetworkElementCommand.NETWORK_PUB_LAST_IP);
               for (final IpAddressTO ip : ips) {
                   if (!StringUtils.equalsIgnoreCase(lastIp, "true") || ip.isAdd()) {
                       continue;
                   }
   ```
   and
   ```
           if (publicNicInfo.first() == 2) {
               s_logger.debug("Do not remove eth2 in network VR because it is the public NIC of source NAT.");
               continue;
           }
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [cloudstack] weizhouapache commented on a change in pull request #5985: [VMware] Support for removal of NIC on IP disassociation on the VR

Posted by GitBox <gi...@apache.org>.
weizhouapache commented on a change in pull request #5985:
URL: https://github.com/apache/cloudstack/pull/5985#discussion_r808897412



##########
File path: plugins/hypervisors/vmware/src/main/java/com/cloud/hypervisor/vmware/resource/VmwareResource.java
##########
@@ -1622,6 +1589,81 @@ private ExecutionResult prepareNetworkElementCommand(IpAssocCommand cmd) {
         return new ExecutionResult(true, null);
     }
 
+    private ExecutionResult cleanupNetworkElementCommand(IpAssocCommand cmd) {
+        VmwareContext context = getServiceContext();
+        try {
+            VmwareHypervisorHost hyperHost = getHyperHost(context);
+            IpAddressTO[] ips = cmd.getIpAddresses();
+            String routerName = cmd.getAccessDetail(NetworkElementCommand.ROUTER_NAME);
+
+            VirtualMachineMO vmMo = hyperHost.findVmOnHyperHost(routerName);
+            // command may sometimes be redirected to a wrong host, we relax
+            // the check and will try to find it within datacenter
+            if (vmMo == null) {
+                if (hyperHost instanceof HostMO) {
+                    final DatacenterMO dcMo = new DatacenterMO(context, hyperHost.getHyperHostDatacenter());
+                    vmMo = dcMo.findVm(routerName);
+                }
+            }
+
+            if (vmMo == null) {
+                String msg = String.format("Router %s no longer exists to execute IPAssoc command ", routerName);
+                s_logger.error(msg);
+                throw new Exception(msg);
+            }
+            final String lastIp = cmd.getAccessDetail(NetworkElementCommand.NETWORK_PUB_LAST_IP);
+            for (IpAddressTO ip : ips) {
+                if (ip.isAdd() || lastIp.equalsIgnoreCase("false")) {
+                    continue;
+                }
+                Pair<VirtualDevice, Integer> nicInfo = getVirtualDevice(vmMo, ips[0]);

Review comment:
       @Pearl1594 
   use ip instead of `ips[0]` in line 1619 ?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [cloudstack] weizhouapache commented on a change in pull request #5985: [VMware] Support for removal of NIC on IP disassociation on the VR

Posted by GitBox <gi...@apache.org>.
weizhouapache commented on a change in pull request #5985:
URL: https://github.com/apache/cloudstack/pull/5985#discussion_r807950988



##########
File path: plugins/hypervisors/vmware/src/main/java/com/cloud/hypervisor/vmware/resource/VmwareResource.java
##########
@@ -1622,6 +1589,83 @@ private ExecutionResult prepareNetworkElementCommand(IpAssocCommand cmd) {
         return new ExecutionResult(true, null);
     }
 
+    private ExecutionResult cleanupNetworkElementCommand(IpAssocCommand cmd) {
+        VmwareContext context = getServiceContext();
+        try {
+            VmwareHypervisorHost hyperHost = getHyperHost(context);
+            IpAddressTO[] ips = cmd.getIpAddresses();
+            String routerName = cmd.getAccessDetail(NetworkElementCommand.ROUTER_NAME);
+
+            VirtualMachineMO vmMo = hyperHost.findVmOnHyperHost(routerName);
+            // command may sometimes be redirected to a wrong host, we relax
+            // the check and will try to find it within datacenter
+            if (vmMo == null) {
+                if (hyperHost instanceof HostMO) {
+                    final DatacenterMO dcMo = new DatacenterMO(context, hyperHost.getHyperHostDatacenter());
+                    vmMo = dcMo.findVm(routerName);
+                }
+            }
+
+            if (vmMo == null) {
+                String msg = String.format("Router %s no longer exists to execute IPAssoc command ", routerName);
+                s_logger.error(msg);
+                throw new Exception(msg);
+            }
+            final String lastIp = cmd.getAccessDetail(NetworkElementCommand.NETWORK_PUB_LAST_IP);
+            if (ips.length == 1 && !ips[0].isAdd()) {

Review comment:
       @Pearl1594
   from what I have tested, it is good to use `if (ips.length == 1 && !ips[0].isAdd()) {`
   I do not know if there are special case that, there are multiple ip address in IpAssocCommand and we need to remove one of them.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [cloudstack] weizhouapache commented on pull request #5985: [VMware] Support for removal of NIC on IP disassociation on the VR

Posted by GitBox <gi...@apache.org>.
weizhouapache commented on pull request #5985:
URL: https://github.com/apache/cloudstack/pull/5985#issuecomment-1043014738


   @blueorangutan test centos7 vmware-70u3


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [cloudstack] Pearl1594 commented on a change in pull request #5985: [VMware] Support for removal of NIC on IP disassociation on the VR

Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on a change in pull request #5985:
URL: https://github.com/apache/cloudstack/pull/5985#discussion_r807837440



##########
File path: plugins/hypervisors/vmware/src/main/java/com/cloud/hypervisor/vmware/resource/VmwareResource.java
##########
@@ -1622,6 +1589,83 @@ private ExecutionResult prepareNetworkElementCommand(IpAssocCommand cmd) {
         return new ExecutionResult(true, null);
     }
 
+    private ExecutionResult cleanupNetworkElementCommand(IpAssocCommand cmd) {
+        VmwareContext context = getServiceContext();
+        try {
+            VmwareHypervisorHost hyperHost = getHyperHost(context);
+            IpAddressTO[] ips = cmd.getIpAddresses();
+            String routerName = cmd.getAccessDetail(NetworkElementCommand.ROUTER_NAME);
+
+            VirtualMachineMO vmMo = hyperHost.findVmOnHyperHost(routerName);
+            // command may sometimes be redirected to a wrong host, we relax
+            // the check and will try to find it within datacenter
+            if (vmMo == null) {
+                if (hyperHost instanceof HostMO) {
+                    final DatacenterMO dcMo = new DatacenterMO(context, hyperHost.getHyperHostDatacenter());
+                    vmMo = dcMo.findVm(routerName);
+                }
+            }
+
+            if (vmMo == null) {
+                String msg = String.format("Router %s no longer exists to execute IPAssoc command ", routerName);
+                s_logger.error(msg);
+                throw new Exception(msg);
+            }
+            final String lastIp = cmd.getAccessDetail(NetworkElementCommand.NETWORK_PUB_LAST_IP);
+            if (ips.length == 1 && !ips[0].isAdd()) {

Review comment:
       @weizhouapache we only unplug the nic if it is there is only 1 IP on the NIC / last IP right? In that case is it required to loop through all the IPs? sorry, I probably am missing something..




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [cloudstack] nvazquez commented on pull request #5985: [VMware] Support for removal of NIC on IP disassociation on the VR

Posted by GitBox <gi...@apache.org>.
nvazquez commented on pull request #5985:
URL: https://github.com/apache/cloudstack/pull/5985#issuecomment-1043003972


   @blueorangutan test centos7 vmware-67u3


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [cloudstack] weizhouapache commented on pull request #5985: [VMware] Support for removal of NIC on IP disassociation on the VR

Posted by GitBox <gi...@apache.org>.
weizhouapache commented on pull request #5985:
URL: https://github.com/apache/cloudstack/pull/5985#issuecomment-1042949230


   @blueorangutan package


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [cloudstack] weizhouapache commented on pull request #5985: [VMware] Support for removal of NIC on IP disassociation on the VR

Posted by GitBox <gi...@apache.org>.
weizhouapache commented on pull request #5985:
URL: https://github.com/apache/cloudstack/pull/5985#issuecomment-1039964422


   @Pearl1594 
   in my testing, if public ips are used for port forwarding or load balancer, everything seems ok.
   If public ips are used for static nat, the interface will be removed even if there is still public ip in use (for example, pf or lb or nat). This issue happens with eth2 as well which is the interface for source nat.
   
   I think we should use similar code like
   ```
               final String lastIp = cmd.getAccessDetail(NetworkElementCommand.NETWORK_PUB_LAST_IP);
               for (final IpAddressTO ip : ips) {
                   if (!StringUtils.equalsIgnoreCase(lastIp, "true") || ip.isAdd()) {
                       continue;
                   }
   ```
   and
   ```
           if (publicNicInfo.first() == 2) {
               s_logger.debug("Do not remove eth2 in network VR because it is the public NIC of source NAT.");
               continue;
           }
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [cloudstack] blueorangutan commented on pull request #5985: [VMware] Support for removal of NIC on IP disassociation on the VR

Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #5985:
URL: https://github.com/apache/cloudstack/pull/5985#issuecomment-1042999833


   Packaging result: :heavy_check_mark: el7 :heavy_check_mark: el8 :heavy_check_mark: debian :heavy_check_mark: suse15. SL-JID 2642


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [cloudstack] blueorangutan commented on pull request #5985: [VMware] Support for removal of NIC on IP disassociation on the VR

Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #5985:
URL: https://github.com/apache/cloudstack/pull/5985#issuecomment-1036210816


   @Pearl1594 a Trillian-Jenkins test job (centos7 mgmt + vmware-67u3) has been kicked to run smoke tests


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [cloudstack] weizhouapache commented on pull request #5985: [VMware] Support for removal of NIC on IP disassociation on the VR

Posted by GitBox <gi...@apache.org>.
weizhouapache commented on pull request #5985:
URL: https://github.com/apache/cloudstack/pull/5985#issuecomment-1038869612


   @Pearl1594 @nvazquez @sureshanaparti 
   this PR needs to be tested/approved. 
   However 4.16.1.0 is close to be released, let's test/approve/merge this after 4.16.1.0 release, is that ok to you ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [cloudstack] Pearl1594 commented on pull request #5985: [VMware] Support for removal of NIC on IP disassociation on the VR

Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on pull request #5985:
URL: https://github.com/apache/cloudstack/pull/5985#issuecomment-1038686527


   @blueorangutan package


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [cloudstack] blueorangutan commented on pull request #5985: [VMware] Support for removal of NIC on IP disassociation on the VR

Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #5985:
URL: https://github.com/apache/cloudstack/pull/5985#issuecomment-1038715754


   Packaging result: :heavy_check_mark: el7 :heavy_check_mark: el8 :heavy_check_mark: debian :heavy_check_mark: suse15. SL-JID 2597


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org