You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by GitBox <gi...@apache.org> on 2021/07/29 10:50:12 UTC
[GitHub] [cloudstack] davidjumani opened a new pull request #5254: kubernetes: Deploy kubernetes-provider when creating a cluster
davidjumani opened a new pull request #5254:
URL: https://github.com/apache/cloudstack/pull/5254
### Description
This PR automatically deploys the [cloudstack-kubernetes-provider](https://github.com/apache/cloudstack-kubernetes-provider) when deploying a kubernetes cluster
### Types of changes
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] Enhancement (improves an existing feature and functionality)
- [ ] Cleanup (Code refactoring and cleanup, that may add test cases)
#### Feature/Enhancement Scale
- [ ] Major
- [x] Minor
### How Has This Been Tested?
```
kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system cloud-controller-manager-66d6576549-rd5zf 1/1 Running 0 44m
.......
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #5254: kubernetes: Deploy kubernetes-provider when creating a cluster
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #5254:
URL: https://github.com/apache/cloudstack/pull/5254#issuecomment-890189206
<b>Trillian test result (tid-1431)</b>
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 37602 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr5254-t1431-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_clusters.py
Intermittent failure detected: /marvin/tests/smoke/test_outofbandmanagement_nestedplugin.py
Smoke tests completed. 88 look OK, 1 have error(s)
Only failed tests results shown below:
Test | Result | Time (s) | Test File
--- | --- | --- | ---
test_02_deploy_and_upgrade_kubernetes_cluster | `Failure` | 500.36 | test_kubernetes_clusters.py
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #5254: kubernetes: Deploy kubernetes-provider when creating a cluster
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #5254:
URL: https://github.com/apache/cloudstack/pull/5254#issuecomment-891813640
Packaging result: :heavy_check_mark: el7 :heavy_check_mark: el8 :heavy_check_mark: debian. SL-JID 736
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] davidjumani commented on a change in pull request #5254: kubernetes: Deploy kubernetes-provider when creating a cluster
Posted by GitBox <gi...@apache.org>.
davidjumani commented on a change in pull request #5254:
URL: https://github.com/apache/cloudstack/pull/5254#discussion_r681452460
##########
File path: plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/KubernetesClusterManagerImpl.java
##########
@@ -644,7 +652,15 @@ public KubernetesClusterResponse createKubernetesClusterResponse(long kubernetes
return response;
}
+ private void validateEndpointUrl() {
+ String csUrl = ApiServiceConfiguration.ApiServletPath.value();
+ if (csUrl == null || csUrl.contains("localhost")) {
+ throw new InvalidParameterValueException("Global setting endpointe.url has to be set to the Management Server's API end point");
Review comment:
Nope, it's a global variable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] nvazquez commented on pull request #5254: kubernetes: Deploy kubernetes-provider when creating a cluster
Posted by GitBox <gi...@apache.org>.
nvazquez commented on pull request #5254:
URL: https://github.com/apache/cloudstack/pull/5254#issuecomment-891798639
@blueorangutan package
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on a change in pull request #5254: kubernetes: Deploy kubernetes-provider when creating a cluster
Posted by GitBox <gi...@apache.org>.
rhtyd commented on a change in pull request #5254:
URL: https://github.com/apache/cloudstack/pull/5254#discussion_r681451261
##########
File path: plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/KubernetesClusterManagerImpl.java
##########
@@ -1093,13 +1111,39 @@ public boolean startKubernetesCluster(long kubernetesClusterId, boolean onCreate
startWorker = ComponentContext.inject(startWorker);
if (onCreate) {
// Start for Kubernetes cluster in 'Created' state
+ Account owner = accountService.getActiveAccountById(kubernetesCluster.getAccountId());
+ String[] keys = getServiceUserKeys(owner);
+ startWorker.setKeys(keys);
return startWorker.startKubernetesClusterOnCreate();
} else {
// Start for Kubernetes cluster in 'Stopped' state. Resources are already provisioned, just need to be started
return startWorker.startStoppedKubernetesCluster();
}
}
+ private String[] getServiceUserKeys(Account owner) {
+ if (owner == null) {
+ owner = CallContext.current().getCallingAccount();
+ }
+ String username = owner.getAccountName() + "-" + KUBEADMIN_ACCOUNT_NAME;
Review comment:
Do we assume/create a service account/user globally, or is it per account now (a service user just for k8s use-case)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] davidjumani commented on pull request #5254: kubernetes: Deploy kubernetes-provider when creating a cluster
Posted by GitBox <gi...@apache.org>.
davidjumani commented on pull request #5254:
URL: https://github.com/apache/cloudstack/pull/5254#issuecomment-889017975
@blueorangutan package
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] davidjumani commented on pull request #5254: kubernetes: Deploy kubernetes-provider when creating a cluster
Posted by GitBox <gi...@apache.org>.
davidjumani commented on pull request #5254:
URL: https://github.com/apache/cloudstack/pull/5254#issuecomment-893165459
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] davidjumani commented on pull request #5254: kubernetes: Deploy kubernetes-provider when creating a cluster
Posted by GitBox <gi...@apache.org>.
davidjumani commented on pull request #5254:
URL: https://github.com/apache/cloudstack/pull/5254#issuecomment-890752317
@blueorangutan test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #5254: kubernetes: Deploy kubernetes-provider when creating a cluster
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #5254:
URL: https://github.com/apache/cloudstack/pull/5254#issuecomment-893165598
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #5254: kubernetes: Deploy kubernetes-provider when creating a cluster
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #5254:
URL: https://github.com/apache/cloudstack/pull/5254#issuecomment-890292571
@davidjumani a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] davidjumani commented on pull request #5254: kubernetes: Deploy kubernetes-provider when creating a cluster
Posted by GitBox <gi...@apache.org>.
davidjumani commented on pull request #5254:
URL: https://github.com/apache/cloudstack/pull/5254#issuecomment-890292503
@blueorangutan test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #5254: kubernetes: Deploy kubernetes-provider when creating a cluster
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #5254:
URL: https://github.com/apache/cloudstack/pull/5254#issuecomment-891798884
@nvazquez a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] GabrielBrascher commented on a change in pull request #5254: kubernetes: Deploy kubernetes-provider when creating a cluster
Posted by GitBox <gi...@apache.org>.
GabrielBrascher commented on a change in pull request #5254:
URL: https://github.com/apache/cloudstack/pull/5254#discussion_r682037350
##########
File path: plugins/integrations/kubernetes-service/src/main/resources/script/deploy-cloudstack-secret
##########
@@ -0,0 +1,69 @@
+
Review comment:
The first line is blank, instead of containing the bash _shebang #!_.
##########
File path: plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterActionWorker.java
##########
@@ -380,4 +394,108 @@ protected boolean stateTransitTo(long kubernetesClusterId, KubernetesCluster.Eve
return false;
}
}
+
+ protected boolean createCloudStackSecret(String[] keys) {
+ File pkFile = getManagementServerSshPublicKeyFile();
+ Pair<String, Integer> publicIpSshPort = getKubernetesClusterServerIpSshPort(null);
+ publicIpAddress = publicIpSshPort.first();
+ sshPort = publicIpSshPort.second();
+
+ try {
+ Pair<Boolean, String> result = SshHelper.sshExecute(publicIpAddress, sshPort, CLUSTER_NODE_VM_USER,
+ pkFile, null, String.format("sudo /opt/bin/deploy-cloudstack-secret -u '%s' -k '%s' -s '%s'",
+ ApiServiceConfiguration.ApiServletPath.value(), keys[0], keys[1]),
+ 10000, 10000, 60000);
+ return result.first();
+ } catch (Exception e) {
+ String msg = String.format("Failed to add cloudstack-secret to Kubernetes cluster: %s", kubernetesCluster.getName());
+ LOGGER.warn(msg, e);
+ }
+ return false;
+ }
+
+ protected File retrieveScriptFile(String filename) {
+ File file = null;
+ try {
+ String data = readResourceFile("/script/" + filename);
+ file = File.createTempFile(filename, ".sh");
+ BufferedWriter writer = new BufferedWriter(new FileWriter(file));
+ writer.write(data);
+ writer.close();
+ } catch (IOException e) {
+ logAndThrow(Level.ERROR, String.format("Kubernetes Cluster %s : Failed to to fetch script %s",
+ kubernetesCluster.getName(), filename), e);
+ }
+ return file;
+ }
+
+ protected void retrieveScriptFiles() {
+ deploySecretsScriptFile = retrieveScriptFile(deploySecretsScriptFilename);
+ deployProviderScriptFile = retrieveScriptFile(deployProviderScriptFilename);
+ }
+
+ protected void copyScripts(String nodeAddress, final int sshPort) {
+ try {
+ SshHelper.scpTo(nodeAddress, sshPort, CLUSTER_NODE_VM_USER, sshKeyFile, null,
+ "~/", deploySecretsScriptFile.getAbsolutePath(), "0755");
+ SshHelper.scpTo(nodeAddress, sshPort, CLUSTER_NODE_VM_USER, sshKeyFile, null,
+ "~/", deployProviderScriptFile.getAbsolutePath(), "0755");
+ String cmdStr = String.format("sudo mv ~/%s /opt/bin/%s", deploySecretsScriptFile.getName(), deploySecretsScriptFilename);
+ SshHelper.sshExecute(publicIpAddress, sshPort, CLUSTER_NODE_VM_USER, sshKeyFile, null,
+ cmdStr, 10000, 10000, 10 * 60 * 1000);
+ cmdStr = String.format("sudo mv ~/%s /opt/bin/%s", deployProviderScriptFile.getName(), deployProviderScriptFilename);
+ SshHelper.sshExecute(publicIpAddress, sshPort, CLUSTER_NODE_VM_USER, sshKeyFile, null,
+ cmdStr, 10000, 10000, 10 * 60 * 1000);
+ } catch (Exception e) {
+ throw new CloudRuntimeException(e);
+ }
+ }
+
+ protected boolean deployProvider() {
+ Network network = networkDao.findById(kubernetesCluster.getNetworkId());
+ // Since the provider creates IP addresses, don't deploy it unless the underlying network supports it
+ if (network.getGuestType() != GuestType.Isolated) {
+ logMessage(Level.INFO, String.format("Skipping adding the provider as %s is not on an isolated network",
+ kubernetesCluster.getName()), null);
+ return true;
+ }
+ File pkFile = getManagementServerSshPublicKeyFile();
+ Pair<String, Integer> publicIpSshPort = getKubernetesClusterServerIpSshPort(null);
+ publicIpAddress = publicIpSshPort.first();
+ sshPort = publicIpSshPort.second();
+
+ try {
+ String command = "sudo /opt/bin/deploy-provider";
Review comment:
This could also be extracted to a constant, or append the existing String constant `deployProviderScriptFilename`.
##########
File path: plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterActionWorker.java
##########
@@ -380,4 +394,108 @@ protected boolean stateTransitTo(long kubernetesClusterId, KubernetesCluster.Eve
return false;
}
}
+
+ protected boolean createCloudStackSecret(String[] keys) {
+ File pkFile = getManagementServerSshPublicKeyFile();
+ Pair<String, Integer> publicIpSshPort = getKubernetesClusterServerIpSshPort(null);
+ publicIpAddress = publicIpSshPort.first();
+ sshPort = publicIpSshPort.second();
+
+ try {
+ Pair<Boolean, String> result = SshHelper.sshExecute(publicIpAddress, sshPort, CLUSTER_NODE_VM_USER,
+ pkFile, null, String.format("sudo /opt/bin/deploy-cloudstack-secret -u '%s' -k '%s' -s '%s'",
Review comment:
Maybe instead of `String.format("sudo /opt/bin/deploy-cloudstack-secret ..."...)` it could use some of the existing, such as `deploySecretsScriptFilename`.
##########
File path: plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/KubernetesClusterManagerImpl.java
##########
@@ -644,7 +652,15 @@ public KubernetesClusterResponse createKubernetesClusterResponse(long kubernetes
return response;
}
+ private void validateEndpointUrl() {
+ String csUrl = ApiServiceConfiguration.ApiServletPath.value();
+ if (csUrl == null || csUrl.contains("localhost")) {
+ throw new InvalidParameterValueException("Global setting endpointe.url has to be set to the Management Server's API end point");
Review comment:
Would be better to retrieve as @nvazquez said. Thus, if/when this gets fixed it will automatically reflect in this message.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] shwstppr commented on a change in pull request #5254: kubernetes: Deploy kubernetes-provider when creating a cluster
Posted by GitBox <gi...@apache.org>.
shwstppr commented on a change in pull request #5254:
URL: https://github.com/apache/cloudstack/pull/5254#discussion_r679788290
##########
File path: plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterActionWorker.java
##########
@@ -380,4 +394,108 @@ protected boolean stateTransitTo(long kubernetesClusterId, KubernetesCluster.Eve
return false;
}
}
+
+ protected boolean createCloudStackSecret(String[] keys) {
+ File pkFile = getManagementServerSshPublicKeyFile();
+ Pair<String, Integer> publicIpSshPort = getKubernetesClusterServerIpSshPort(null);
+ publicIpAddress = publicIpSshPort.first();
+ sshPort = publicIpSshPort.second();
+
+ try {
+ Pair<Boolean, String> result = SshHelper.sshExecute(publicIpAddress, sshPort, CLUSTER_NODE_VM_USER,
+ pkFile, null, String.format("sudo /opt/bin/deploy-cloudstack-secret -u '%s' -k '%s' -s '%s'",
+ ApiServiceConfiguration.ApiServletPath.value(), keys[0], keys[1]),
+ 10000, 10000, 60000);
+ return result.first();
+ } catch (Exception e) {
+ String msg = String.format("Failed to add cloudstack-secret to Kubernetes cluster: %s", kubernetesCluster.getName());
+ LOGGER.warn(msg, e);
+ }
+ return false;
+ }
+
+ protected File retrieveScriptFile(String filename) {
+ File file = null;
+ try {
+ String data = readResourceFile("/script/" + filename);
+ file = File.createTempFile(filename, ".sh");
+ BufferedWriter writer = new BufferedWriter(new FileWriter(file));
+ writer.write(data);
+ writer.close();
+ } catch (IOException e) {
+ logAndThrow(Level.ERROR, String.format("Failed to upgrade Kubernetes cluster %s, unable to prepare upgrade script %s", kubernetesCluster.getName(), filename), e);
+ }
+ return file;
+ }
+
+ protected void retrieveScriptFiles() {
+ deploySecretsScriptFile = retrieveScriptFile(deploySecretsScriptFilename);
+ deployProviderScriptFile = retrieveScriptFile(deployProviderScriptFilename);
+ }
+
+ protected void copyScripts(String nodeAddress, final int sshPort) {
+ try {
+ SshHelper.scpTo(nodeAddress, sshPort, CLUSTER_NODE_VM_USER, sshKeyFile, null,
+ "~/", deploySecretsScriptFile.getAbsolutePath(), "0755");
+ SshHelper.scpTo(nodeAddress, sshPort, CLUSTER_NODE_VM_USER, sshKeyFile, null,
+ "~/", deployProviderScriptFile.getAbsolutePath(), "0755");
+ String cmdStr = String.format("sudo mv ~/%s /opt/bin/%s", deploySecretsScriptFile.getName(), deploySecretsScriptFilename);
+ SshHelper.sshExecute(publicIpAddress, sshPort, CLUSTER_NODE_VM_USER, sshKeyFile, null,
+ cmdStr, 10000, 10000, 10 * 60 * 1000);
+ cmdStr = String.format("sudo mv ~/%s /opt/bin/%s", deployProviderScriptFile.getName(), deployProviderScriptFilename);
+ SshHelper.sshExecute(publicIpAddress, sshPort, CLUSTER_NODE_VM_USER, sshKeyFile, null,
+ cmdStr, 10000, 10000, 10 * 60 * 1000);
+ } catch (Exception e) {
+ throw new CloudRuntimeException(e);
+ }
+ }
+
+ protected boolean deployProvider() {
+ Network network = networkDao.findById(kubernetesCluster.getNetworkId());
+ // Since the provider creates IP addresses, don't deploy it unless the underlying network supports it
+ if (network.getGuestType() != GuestType.Isolated) {
+ return true;
Review comment:
better to add a log here that provider deployment skipped for networks other than Isolated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #5254: kubernetes: Deploy kubernetes-provider when creating a cluster
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #5254:
URL: https://github.com/apache/cloudstack/pull/5254#issuecomment-889741913
Packaging result: :heavy_check_mark: el7 :heavy_check_mark: el8 :heavy_check_mark: debian. SL-JID 708
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #5254: kubernetes: Deploy kubernetes-provider when creating a cluster
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #5254:
URL: https://github.com/apache/cloudstack/pull/5254#issuecomment-890752760
@davidjumani a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] shwstppr commented on a change in pull request #5254: kubernetes: Deploy kubernetes-provider when creating a cluster
Posted by GitBox <gi...@apache.org>.
shwstppr commented on a change in pull request #5254:
URL: https://github.com/apache/cloudstack/pull/5254#discussion_r679790935
##########
File path: plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterActionWorker.java
##########
@@ -380,4 +394,108 @@ protected boolean stateTransitTo(long kubernetesClusterId, KubernetesCluster.Eve
return false;
}
}
+
+ protected boolean createCloudStackSecret(String[] keys) {
+ File pkFile = getManagementServerSshPublicKeyFile();
+ Pair<String, Integer> publicIpSshPort = getKubernetesClusterServerIpSshPort(null);
+ publicIpAddress = publicIpSshPort.first();
+ sshPort = publicIpSshPort.second();
+
+ try {
+ Pair<Boolean, String> result = SshHelper.sshExecute(publicIpAddress, sshPort, CLUSTER_NODE_VM_USER,
+ pkFile, null, String.format("sudo /opt/bin/deploy-cloudstack-secret -u '%s' -k '%s' -s '%s'",
+ ApiServiceConfiguration.ApiServletPath.value(), keys[0], keys[1]),
+ 10000, 10000, 60000);
+ return result.first();
+ } catch (Exception e) {
+ String msg = String.format("Failed to add cloudstack-secret to Kubernetes cluster: %s", kubernetesCluster.getName());
+ LOGGER.warn(msg, e);
+ }
+ return false;
+ }
+
+ protected File retrieveScriptFile(String filename) {
+ File file = null;
+ try {
+ String data = readResourceFile("/script/" + filename);
+ file = File.createTempFile(filename, ".sh");
+ BufferedWriter writer = new BufferedWriter(new FileWriter(file));
+ writer.write(data);
+ writer.close();
+ } catch (IOException e) {
+ logAndThrow(Level.ERROR, String.format("Failed to upgrade Kubernetes cluster %s, unable to prepare upgrade script %s", kubernetesCluster.getName(), filename), e);
+ }
+ return file;
+ }
+
+ protected void retrieveScriptFiles() {
+ deploySecretsScriptFile = retrieveScriptFile(deploySecretsScriptFilename);
+ deployProviderScriptFile = retrieveScriptFile(deployProviderScriptFilename);
+ }
+
+ protected void copyScripts(String nodeAddress, final int sshPort) {
+ try {
+ SshHelper.scpTo(nodeAddress, sshPort, CLUSTER_NODE_VM_USER, sshKeyFile, null,
+ "~/", deploySecretsScriptFile.getAbsolutePath(), "0755");
+ SshHelper.scpTo(nodeAddress, sshPort, CLUSTER_NODE_VM_USER, sshKeyFile, null,
+ "~/", deployProviderScriptFile.getAbsolutePath(), "0755");
+ String cmdStr = String.format("sudo mv ~/%s /opt/bin/%s", deploySecretsScriptFile.getName(), deploySecretsScriptFilename);
+ SshHelper.sshExecute(publicIpAddress, sshPort, CLUSTER_NODE_VM_USER, sshKeyFile, null,
+ cmdStr, 10000, 10000, 10 * 60 * 1000);
+ cmdStr = String.format("sudo mv ~/%s /opt/bin/%s", deployProviderScriptFile.getName(), deployProviderScriptFilename);
+ SshHelper.sshExecute(publicIpAddress, sshPort, CLUSTER_NODE_VM_USER, sshKeyFile, null,
+ cmdStr, 10000, 10000, 10 * 60 * 1000);
+ } catch (Exception e) {
+ throw new CloudRuntimeException(e);
+ }
+ }
+
+ protected boolean deployProvider() {
+ Network network = networkDao.findById(kubernetesCluster.getNetworkId());
+ // Since the provider creates IP addresses, don't deploy it unless the underlying network supports it
+ if (network.getGuestType() != GuestType.Isolated) {
+ return true;
+ }
+ File pkFile = getManagementServerSshPublicKeyFile();
+ Pair<String, Integer> publicIpSshPort = getKubernetesClusterServerIpSshPort(null);
+ publicIpAddress = publicIpSshPort.first();
+ sshPort = publicIpSshPort.second();
+
+ try {
+ String command = "sudo /opt/bin/deploy-provider";
+ Pair<Boolean, String> result = SshHelper.sshExecute(publicIpAddress, sshPort, CLUSTER_NODE_VM_USER,
+ pkFile, null, command, 10000, 10000, 60000);
+
+ // Maybe the file isn't present. Try and copy it
+ if (!result.first()) {
+ logMessage(Level.INFO, "Provider files missing. Adding them now", null);
+ retrieveScriptFiles();
+ copyScripts(publicIpAddress, sshPort);
+
+ if (!createCloudStackSecret(keys)) {
+ logTransitStateAndThrow(Level.ERROR, String.format("Failed to setup keys for Kubernetes cluster %s",
+ kubernetesCluster.getName()), kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed);
+ }
+
+ // If at first you don't succeed ...
+ result = SshHelper.sshExecute(publicIpAddress, sshPort, CLUSTER_NODE_VM_USER,
+ pkFile, null, command, 10000, 10000, 60000);
+ if (!result.first()) {
+ throw new CloudRuntimeException(result.second());
+ }
+ }
+ return true;
+ } catch (Exception e) {
+ String msg = String.format("Failed to deploy kubernetes provider: %s : %s", kubernetesCluster.getName(), e.getMessage());
+ logAndThrow(Level.ERROR, msg);
+ return false;
+ } finally {
+ // Deploying the provider might fail but it can be deployed manually too, so no need to go to an alert state
+ stateTransitTo(kubernetesCluster.getId(), KubernetesCluster.Event.OperationSucceeded);
Review comment:
Do we need this? Will this preemptively return success for cluster during upgrade?
In startworker, `stateTransitTo` is already present at ln 581
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #5254: kubernetes: Deploy kubernetes-provider when creating a cluster
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #5254:
URL: https://github.com/apache/cloudstack/pull/5254#issuecomment-890750732
Packaging result: :heavy_check_mark: el7 :heavy_check_mark: el8 :heavy_check_mark: debian. SL-JID 722
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #5254: kubernetes: Deploy kubernetes-provider when creating a cluster
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #5254:
URL: https://github.com/apache/cloudstack/pull/5254#issuecomment-889814276
Packaging result: :heavy_check_mark: el7 :heavy_check_mark: el8 :heavy_check_mark: debian. SL-JID 709
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #5254: kubernetes: Deploy kubernetes-provider when creating a cluster
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #5254:
URL: https://github.com/apache/cloudstack/pull/5254#issuecomment-889539508
<b>Trillian test result (tid-1413)</b>
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 39494 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr5254-t1413-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_clusters.py
Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Smoke tests completed. 88 look OK, 1 have error(s)
Only failed tests results shown below:
Test | Result | Time (s) | Test File
--- | --- | --- | ---
test_01_invalid_upgrade_kubernetes_cluster | `Failure` | 0.07 | test_kubernetes_clusters.py
test_02_deploy_and_upgrade_kubernetes_cluster | `Failure` | 0.04 | test_kubernetes_clusters.py
test_03_deploy_and_scale_kubernetes_cluster | `Failure` | 0.04 | test_kubernetes_clusters.py
test_04_basic_lifecycle_kubernetes_cluster | `Failure` | 0.05 | test_kubernetes_clusters.py
test_05_delete_kubernetes_cluster | `Failure` | 0.05 | test_kubernetes_clusters.py
test_07_deploy_kubernetes_ha_cluster | `Failure` | 0.04 | test_kubernetes_clusters.py
test_08_deploy_and_upgrade_kubernetes_ha_cluster | `Failure` | 0.04 | test_kubernetes_clusters.py
test_09_delete_kubernetes_ha_cluster | `Failure` | 0.04 | test_kubernetes_clusters.py
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #5254: kubernetes: Deploy kubernetes-provider when creating a cluster
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #5254:
URL: https://github.com/apache/cloudstack/pull/5254#issuecomment-893175656
Packaging result: :heavy_check_mark: el7 :heavy_check_mark: el8 :heavy_check_mark: debian. SL-JID 756
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] davidjumani commented on a change in pull request #5254: kubernetes: Deploy kubernetes-provider when creating a cluster
Posted by GitBox <gi...@apache.org>.
davidjumani commented on a change in pull request #5254:
URL: https://github.com/apache/cloudstack/pull/5254#discussion_r681452589
##########
File path: plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/KubernetesClusterManagerImpl.java
##########
@@ -1093,13 +1111,39 @@ public boolean startKubernetesCluster(long kubernetesClusterId, boolean onCreate
startWorker = ComponentContext.inject(startWorker);
if (onCreate) {
// Start for Kubernetes cluster in 'Created' state
+ Account owner = accountService.getActiveAccountById(kubernetesCluster.getAccountId());
+ String[] keys = getServiceUserKeys(owner);
+ startWorker.setKeys(keys);
return startWorker.startKubernetesClusterOnCreate();
} else {
// Start for Kubernetes cluster in 'Stopped' state. Resources are already provisioned, just need to be started
return startWorker.startStoppedKubernetesCluster();
}
}
+ private String[] getServiceUserKeys(Account owner) {
+ if (owner == null) {
+ owner = CallContext.current().getCallingAccount();
+ }
+ String username = owner.getAccountName() + "-" + KUBEADMIN_ACCOUNT_NAME;
Review comment:
Per account basis just for kube
##########
File path: test/integration/smoke/test_kubernetes_clusters.py
##########
@@ -75,8 +75,12 @@ def setUpClass(cls):
cls.kubernetes_version_ids = []
if cls.hypervisorNotSupported == False:
- cls.initial_configuration_cks_enabled = Configurations.list(cls.apiclient,
- name="cloud.kubernetes.service.enabled")[0].value
+ cls.endpoint_url = Configurations.list(cls.apiclient, name="endpointe.url")[0].value
+ if "localhost" in cls.endpoint_url:
+ endpoint_url = "http://%s:%d/client/api " %(cls.mgtSvrDetails["mgtSvrIp"], cls.mgtSvrDetails["port"])
+ cls.debug("Setting endpointe.url to %s" %(endpoint_url))
Review comment:
Nope, it's a global variable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #5254: kubernetes: Deploy kubernetes-provider when creating a cluster
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #5254:
URL: https://github.com/apache/cloudstack/pull/5254#issuecomment-889049653
Packaging result: :heavy_check_mark: el7 :heavy_check_mark: el8 :heavy_check_mark: debian. SL-JID 689
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] davidjumani commented on pull request #5254: kubernetes: Deploy kubernetes-provider when creating a cluster
Posted by GitBox <gi...@apache.org>.
davidjumani commented on pull request #5254:
URL: https://github.com/apache/cloudstack/pull/5254#issuecomment-893165459
@blueorangutan package
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] davidjumani commented on a change in pull request #5254: kubernetes: Deploy kubernetes-provider when creating a cluster
Posted by GitBox <gi...@apache.org>.
davidjumani commented on a change in pull request #5254:
URL: https://github.com/apache/cloudstack/pull/5254#discussion_r679799200
##########
File path: plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterActionWorker.java
##########
@@ -380,4 +394,108 @@ protected boolean stateTransitTo(long kubernetesClusterId, KubernetesCluster.Eve
return false;
}
}
+
+ protected boolean createCloudStackSecret(String[] keys) {
+ File pkFile = getManagementServerSshPublicKeyFile();
+ Pair<String, Integer> publicIpSshPort = getKubernetesClusterServerIpSshPort(null);
+ publicIpAddress = publicIpSshPort.first();
+ sshPort = publicIpSshPort.second();
+
+ try {
+ Pair<Boolean, String> result = SshHelper.sshExecute(publicIpAddress, sshPort, CLUSTER_NODE_VM_USER,
+ pkFile, null, String.format("sudo /opt/bin/deploy-cloudstack-secret -u '%s' -k '%s' -s '%s'",
+ ApiServiceConfiguration.ApiServletPath.value(), keys[0], keys[1]),
+ 10000, 10000, 60000);
+ return result.first();
+ } catch (Exception e) {
+ String msg = String.format("Failed to add cloudstack-secret to Kubernetes cluster: %s", kubernetesCluster.getName());
+ LOGGER.warn(msg, e);
+ }
+ return false;
+ }
+
+ protected File retrieveScriptFile(String filename) {
+ File file = null;
+ try {
+ String data = readResourceFile("/script/" + filename);
+ file = File.createTempFile(filename, ".sh");
+ BufferedWriter writer = new BufferedWriter(new FileWriter(file));
+ writer.write(data);
+ writer.close();
+ } catch (IOException e) {
+ logAndThrow(Level.ERROR, String.format("Failed to upgrade Kubernetes cluster %s, unable to prepare upgrade script %s", kubernetesCluster.getName(), filename), e);
+ }
+ return file;
+ }
+
+ protected void retrieveScriptFiles() {
+ deploySecretsScriptFile = retrieveScriptFile(deploySecretsScriptFilename);
+ deployProviderScriptFile = retrieveScriptFile(deployProviderScriptFilename);
+ }
+
+ protected void copyScripts(String nodeAddress, final int sshPort) {
+ try {
+ SshHelper.scpTo(nodeAddress, sshPort, CLUSTER_NODE_VM_USER, sshKeyFile, null,
+ "~/", deploySecretsScriptFile.getAbsolutePath(), "0755");
+ SshHelper.scpTo(nodeAddress, sshPort, CLUSTER_NODE_VM_USER, sshKeyFile, null,
+ "~/", deployProviderScriptFile.getAbsolutePath(), "0755");
+ String cmdStr = String.format("sudo mv ~/%s /opt/bin/%s", deploySecretsScriptFile.getName(), deploySecretsScriptFilename);
+ SshHelper.sshExecute(publicIpAddress, sshPort, CLUSTER_NODE_VM_USER, sshKeyFile, null,
+ cmdStr, 10000, 10000, 10 * 60 * 1000);
+ cmdStr = String.format("sudo mv ~/%s /opt/bin/%s", deployProviderScriptFile.getName(), deployProviderScriptFilename);
+ SshHelper.sshExecute(publicIpAddress, sshPort, CLUSTER_NODE_VM_USER, sshKeyFile, null,
+ cmdStr, 10000, 10000, 10 * 60 * 1000);
+ } catch (Exception e) {
+ throw new CloudRuntimeException(e);
+ }
+ }
+
+ protected boolean deployProvider() {
+ Network network = networkDao.findById(kubernetesCluster.getNetworkId());
+ // Since the provider creates IP addresses, don't deploy it unless the underlying network supports it
+ if (network.getGuestType() != GuestType.Isolated) {
+ return true;
+ }
+ File pkFile = getManagementServerSshPublicKeyFile();
+ Pair<String, Integer> publicIpSshPort = getKubernetesClusterServerIpSshPort(null);
+ publicIpAddress = publicIpSshPort.first();
+ sshPort = publicIpSshPort.second();
+
+ try {
+ String command = "sudo /opt/bin/deploy-provider";
+ Pair<Boolean, String> result = SshHelper.sshExecute(publicIpAddress, sshPort, CLUSTER_NODE_VM_USER,
+ pkFile, null, command, 10000, 10000, 60000);
+
+ // Maybe the file isn't present. Try and copy it
+ if (!result.first()) {
+ logMessage(Level.INFO, "Provider files missing. Adding them now", null);
+ retrieveScriptFiles();
+ copyScripts(publicIpAddress, sshPort);
+
+ if (!createCloudStackSecret(keys)) {
+ logTransitStateAndThrow(Level.ERROR, String.format("Failed to setup keys for Kubernetes cluster %s",
+ kubernetesCluster.getName()), kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed);
+ }
+
+ // If at first you don't succeed ...
+ result = SshHelper.sshExecute(publicIpAddress, sshPort, CLUSTER_NODE_VM_USER,
+ pkFile, null, command, 10000, 10000, 60000);
+ if (!result.first()) {
+ throw new CloudRuntimeException(result.second());
+ }
+ }
+ return true;
+ } catch (Exception e) {
+ String msg = String.format("Failed to deploy kubernetes provider: %s : %s", kubernetesCluster.getName(), e.getMessage());
+ logAndThrow(Level.ERROR, msg);
+ return false;
+ } finally {
+ // Deploying the provider might fail but it can be deployed manually too, so no need to go to an alert state
+ stateTransitTo(kubernetesCluster.getId(), KubernetesCluster.Event.OperationSucceeded);
Review comment:
Fair point, removing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on a change in pull request #5254: kubernetes: Deploy kubernetes-provider when creating a cluster
Posted by GitBox <gi...@apache.org>.
rhtyd commented on a change in pull request #5254:
URL: https://github.com/apache/cloudstack/pull/5254#discussion_r681451705
##########
File path: test/integration/smoke/test_kubernetes_clusters.py
##########
@@ -75,8 +75,12 @@ def setUpClass(cls):
cls.kubernetes_version_ids = []
if cls.hypervisorNotSupported == False:
- cls.initial_configuration_cks_enabled = Configurations.list(cls.apiclient,
- name="cloud.kubernetes.service.enabled")[0].value
+ cls.endpoint_url = Configurations.list(cls.apiclient, name="endpointe.url")[0].value
+ if "localhost" in cls.endpoint_url:
+ endpoint_url = "http://%s:%d/client/api " %(cls.mgtSvrDetails["mgtSvrIp"], cls.mgtSvrDetails["port"])
+ cls.debug("Setting endpointe.url to %s" %(endpoint_url))
Review comment:
Is endpointe a thing or typo?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] davidjumani commented on pull request #5254: kubernetes: Deploy kubernetes-provider when creating a cluster
Posted by GitBox <gi...@apache.org>.
davidjumani commented on pull request #5254:
URL: https://github.com/apache/cloudstack/pull/5254#issuecomment-889819466
@blueorangutan test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] davidjumani commented on pull request #5254: kubernetes: Deploy kubernetes-provider when creating a cluster
Posted by GitBox <gi...@apache.org>.
davidjumani commented on pull request #5254:
URL: https://github.com/apache/cloudstack/pull/5254#issuecomment-889643027
@rhtyd They're both independent, and this is related to the RM for the k8s provider
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #5254: kubernetes: Deploy kubernetes-provider when creating a cluster
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #5254:
URL: https://github.com/apache/cloudstack/pull/5254#issuecomment-893839696
<b>Trillian test result (tid-1485)</b>
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 50493 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr5254-t1485-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_clusters.py
Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Smoke tests completed. 87 look OK, 2 have error(s)
Only failed tests results shown below:
Test | Result | Time (s) | Test File
--- | --- | --- | ---
test_03_vpc_privategw_restart_vpc_cleanup | `Failure` | 372.75 | test_privategw_acl.py
test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL | `Failure` | 549.85 | test_vpc_redundant.py
test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers | `Failure` | 478.00 | test_vpc_redundant.py
test_05_rvpc_multi_tiers | `Failure` | 529.28 | test_vpc_redundant.py
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #5254: kubernetes: Deploy kubernetes-provider when creating a cluster
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #5254:
URL: https://github.com/apache/cloudstack/pull/5254#issuecomment-889727323
@davidjumani a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] davidjumani commented on pull request #5254: kubernetes: Deploy kubernetes-provider when creating a cluster
Posted by GitBox <gi...@apache.org>.
davidjumani commented on pull request #5254:
URL: https://github.com/apache/cloudstack/pull/5254#issuecomment-889776985
@blueorangutan package
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] davidjumani commented on pull request #5254: kubernetes: Deploy kubernetes-provider when creating a cluster
Posted by GitBox <gi...@apache.org>.
davidjumani commented on pull request #5254:
URL: https://github.com/apache/cloudstack/pull/5254#issuecomment-889727003
@blueorangutan package
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] nvazquez commented on a change in pull request #5254: kubernetes: Deploy kubernetes-provider when creating a cluster
Posted by GitBox <gi...@apache.org>.
nvazquez commented on a change in pull request #5254:
URL: https://github.com/apache/cloudstack/pull/5254#discussion_r682231820
##########
File path: plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterStartWorker.java
##########
@@ -571,6 +571,13 @@ public boolean startKubernetesClusterOnCreate() {
if (!isKubernetesClusterDashboardServiceRunning(true, startTimeoutTime)) {
logTransitStateAndThrow(Level.ERROR, String.format("Failed to setup Kubernetes cluster : %s in usable state as unable to get Dashboard service running for the cluster", kubernetesCluster.getName()), kubernetesCluster.getId(),KubernetesCluster.Event.OperationFailed);
}
+ retrieveScriptFiles();
+ copyScripts(publicIpAddress, sshPort);
Review comment:
What about creating a method for these lines on `KubernetesClusterActionWorket` and invoke it here and on `KubernetesClusterActionWorket` instead of lines 475-482
##########
File path: plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterStartWorker.java
##########
@@ -571,6 +571,13 @@ public boolean startKubernetesClusterOnCreate() {
if (!isKubernetesClusterDashboardServiceRunning(true, startTimeoutTime)) {
logTransitStateAndThrow(Level.ERROR, String.format("Failed to setup Kubernetes cluster : %s in usable state as unable to get Dashboard service running for the cluster", kubernetesCluster.getName()), kubernetesCluster.getId(),KubernetesCluster.Event.OperationFailed);
}
+ retrieveScriptFiles();
+ copyScripts(publicIpAddress, sshPort);
Review comment:
What about creating a method for these lines on `KubernetesClusterActionWorket` and invoke it here and on `KubernetesClusterActionWorket` instead of lines 475-482?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #5254: kubernetes: Deploy kubernetes-provider when creating a cluster
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #5254:
URL: https://github.com/apache/cloudstack/pull/5254#issuecomment-891233995
<b>Trillian test result (tid-1445)</b>
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 41561 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr5254-t1445-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_projects.py
Intermittent failure detected: /marvin/tests/smoke/test_routers_network_ops.py
Intermittent failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_router_nics.py
Smoke tests completed. 89 look OK, 0 have error(s)
Only failed tests results shown below:
Test | Result | Time (s) | Test File
--- | --- | --- | ---
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #5254: kubernetes: Deploy kubernetes-provider when creating a cluster
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #5254:
URL: https://github.com/apache/cloudstack/pull/5254#issuecomment-893238516
@davidjumani a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] davidjumani commented on pull request #5254: kubernetes: Deploy kubernetes-provider when creating a cluster
Posted by GitBox <gi...@apache.org>.
davidjumani commented on pull request #5254:
URL: https://github.com/apache/cloudstack/pull/5254#issuecomment-890735799
@blueorangutan package
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] shwstppr closed pull request #5254: kubernetes: Deploy kubernetes-provider when creating a cluster
Posted by GitBox <gi...@apache.org>.
shwstppr closed pull request #5254:
URL: https://github.com/apache/cloudstack/pull/5254
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] davidjumani commented on pull request #5254: kubernetes: Deploy kubernetes-provider when creating a cluster
Posted by GitBox <gi...@apache.org>.
davidjumani commented on pull request #5254:
URL: https://github.com/apache/cloudstack/pull/5254#issuecomment-893238326
@blueorangutan test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #5254: kubernetes: Deploy kubernetes-provider when creating a cluster
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #5254:
URL: https://github.com/apache/cloudstack/pull/5254#issuecomment-890376234
<b>Trillian test result (tid-1439)</b>
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 41359 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr5254-t1439-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_loadbalance.py
Intermittent failure detected: /marvin/tests/smoke/test_routers_network_ops.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Smoke tests completed. 87 look OK, 2 have error(s)
Only failed tests results shown below:
Test | Result | Time (s) | Test File
--- | --- | --- | ---
test_01_RVR_Network_FW_PF_SSH_default_routes_egress_true | `Failure` | 335.65 | test_routers_network_ops.py
test_02_RVR_Network_FW_PF_SSH_default_routes_egress_false | `Failure` | 342.01 | test_routers_network_ops.py
test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL | `Failure` | 532.79 | test_vpc_redundant.py
test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers | `Failure` | 531.30 | test_vpc_redundant.py
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on a change in pull request #5254: kubernetes: Deploy kubernetes-provider when creating a cluster
Posted by GitBox <gi...@apache.org>.
rhtyd commented on a change in pull request #5254:
URL: https://github.com/apache/cloudstack/pull/5254#discussion_r681450943
##########
File path: plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/KubernetesClusterManagerImpl.java
##########
@@ -644,7 +652,15 @@ public KubernetesClusterResponse createKubernetesClusterResponse(long kubernetes
return response;
}
+ private void validateEndpointUrl() {
+ String csUrl = ApiServiceConfiguration.ApiServletPath.value();
+ if (csUrl == null || csUrl.contains("localhost")) {
+ throw new InvalidParameterValueException("Global setting endpointe.url has to be set to the Management Server's API end point");
Review comment:
You may want to fix the typo?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #5254: kubernetes: Deploy kubernetes-provider when creating a cluster
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #5254:
URL: https://github.com/apache/cloudstack/pull/5254#issuecomment-890736014
@davidjumani a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #5254: kubernetes: Deploy kubernetes-provider when creating a cluster
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #5254:
URL: https://github.com/apache/cloudstack/pull/5254#issuecomment-893165598
@davidjumani a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on a change in pull request #5254: kubernetes: Deploy kubernetes-provider when creating a cluster
Posted by GitBox <gi...@apache.org>.
rhtyd commented on a change in pull request #5254:
URL: https://github.com/apache/cloudstack/pull/5254#discussion_r681451461
##########
File path: plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/KubernetesClusterManagerImpl.java
##########
@@ -1093,13 +1111,39 @@ public boolean startKubernetesCluster(long kubernetesClusterId, boolean onCreate
startWorker = ComponentContext.inject(startWorker);
if (onCreate) {
// Start for Kubernetes cluster in 'Created' state
+ Account owner = accountService.getActiveAccountById(kubernetesCluster.getAccountId());
+ String[] keys = getServiceUserKeys(owner);
+ startWorker.setKeys(keys);
return startWorker.startKubernetesClusterOnCreate();
} else {
// Start for Kubernetes cluster in 'Stopped' state. Resources are already provisioned, just need to be started
return startWorker.startStoppedKubernetesCluster();
}
}
+ private String[] getServiceUserKeys(Account owner) {
+ if (owner == null) {
+ owner = CallContext.current().getCallingAccount();
+ }
+ String username = owner.getAccountName() + "-" + KUBEADMIN_ACCOUNT_NAME;
Review comment:
Nevermind read the code, looks like we create if it doesn't exist.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] shwstppr commented on a change in pull request #5254: kubernetes: Deploy kubernetes-provider when creating a cluster
Posted by GitBox <gi...@apache.org>.
shwstppr commented on a change in pull request #5254:
URL: https://github.com/apache/cloudstack/pull/5254#discussion_r679787407
##########
File path: plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterActionWorker.java
##########
@@ -380,4 +394,108 @@ protected boolean stateTransitTo(long kubernetesClusterId, KubernetesCluster.Eve
return false;
}
}
+
+ protected boolean createCloudStackSecret(String[] keys) {
+ File pkFile = getManagementServerSshPublicKeyFile();
+ Pair<String, Integer> publicIpSshPort = getKubernetesClusterServerIpSshPort(null);
+ publicIpAddress = publicIpSshPort.first();
+ sshPort = publicIpSshPort.second();
+
+ try {
+ Pair<Boolean, String> result = SshHelper.sshExecute(publicIpAddress, sshPort, CLUSTER_NODE_VM_USER,
+ pkFile, null, String.format("sudo /opt/bin/deploy-cloudstack-secret -u '%s' -k '%s' -s '%s'",
+ ApiServiceConfiguration.ApiServletPath.value(), keys[0], keys[1]),
+ 10000, 10000, 60000);
+ return result.first();
+ } catch (Exception e) {
+ String msg = String.format("Failed to add cloudstack-secret to Kubernetes cluster: %s", kubernetesCluster.getName());
+ LOGGER.warn(msg, e);
+ }
+ return false;
+ }
+
+ protected File retrieveScriptFile(String filename) {
+ File file = null;
+ try {
+ String data = readResourceFile("/script/" + filename);
+ file = File.createTempFile(filename, ".sh");
+ BufferedWriter writer = new BufferedWriter(new FileWriter(file));
+ writer.write(data);
+ writer.close();
+ } catch (IOException e) {
+ logAndThrow(Level.ERROR, String.format("Failed to upgrade Kubernetes cluster %s, unable to prepare upgrade script %s", kubernetesCluster.getName(), filename), e);
Review comment:
@davidjumani can we generalise error message here as now this can be raised for actions other than upgrade
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] nvazquez commented on a change in pull request #5254: kubernetes: Deploy kubernetes-provider when creating a cluster
Posted by GitBox <gi...@apache.org>.
nvazquez commented on a change in pull request #5254:
URL: https://github.com/apache/cloudstack/pull/5254#discussion_r681920502
##########
File path: plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/KubernetesClusterManagerImpl.java
##########
@@ -644,7 +652,15 @@ public KubernetesClusterResponse createKubernetesClusterResponse(long kubernetes
return response;
}
+ private void validateEndpointUrl() {
+ String csUrl = ApiServiceConfiguration.ApiServletPath.value();
+ if (csUrl == null || csUrl.contains("localhost")) {
+ throw new InvalidParameterValueException("Global setting endpointe.url has to be set to the Management Server's API end point");
Review comment:
What about `ApiServiceConfiguration.ApiServletPath.key()` instead of `endpointe.url`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on pull request #5254: kubernetes: Deploy kubernetes-provider when creating a cluster
Posted by GitBox <gi...@apache.org>.
rhtyd commented on pull request #5254:
URL: https://github.com/apache/cloudstack/pull/5254#issuecomment-889638081
We've another cks PR @davidjumani which one should be reviewed/tested/merged first cc @Pearl1594 or is this result of testing and your RM work on k8s provider?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #5254: kubernetes: Deploy kubernetes-provider when creating a cluster
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #5254:
URL: https://github.com/apache/cloudstack/pull/5254#issuecomment-889018332
@davidjumani a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] shwstppr commented on a change in pull request #5254: kubernetes: Deploy kubernetes-provider when creating a cluster
Posted by GitBox <gi...@apache.org>.
shwstppr commented on a change in pull request #5254:
URL: https://github.com/apache/cloudstack/pull/5254#discussion_r680668376
##########
File path: plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterActionWorker.java
##########
@@ -380,4 +394,108 @@ protected boolean stateTransitTo(long kubernetesClusterId, KubernetesCluster.Eve
return false;
}
}
+
+ protected boolean createCloudStackSecret(String[] keys) {
+ File pkFile = getManagementServerSshPublicKeyFile();
+ Pair<String, Integer> publicIpSshPort = getKubernetesClusterServerIpSshPort(null);
+ publicIpAddress = publicIpSshPort.first();
+ sshPort = publicIpSshPort.second();
+
+ try {
+ Pair<Boolean, String> result = SshHelper.sshExecute(publicIpAddress, sshPort, CLUSTER_NODE_VM_USER,
+ pkFile, null, String.format("sudo /opt/bin/deploy-cloudstack-secret -u '%s' -k '%s' -s '%s'",
+ ApiServiceConfiguration.ApiServletPath.value(), keys[0], keys[1]),
+ 10000, 10000, 60000);
+ return result.first();
+ } catch (Exception e) {
+ String msg = String.format("Failed to add cloudstack-secret to Kubernetes cluster: %s", kubernetesCluster.getName());
+ LOGGER.warn(msg, e);
+ }
+ return false;
+ }
+
+ protected File retrieveScriptFile(String filename) {
+ File file = null;
+ try {
+ String data = readResourceFile("/script/" + filename);
+ file = File.createTempFile(filename, ".sh");
+ BufferedWriter writer = new BufferedWriter(new FileWriter(file));
+ writer.write(data);
+ writer.close();
+ } catch (IOException e) {
+ logAndThrow(Level.ERROR, String.format("Kubernetes Cluster %s : Failed to to fetch script %s",
+ kubernetesCluster.getName(), filename), e);
+ }
+ return file;
+ }
+
+ protected void retrieveScriptFiles() {
+ deploySecretsScriptFile = retrieveScriptFile(deploySecretsScriptFilename);
+ deployProviderScriptFile = retrieveScriptFile(deployProviderScriptFilename);
+ }
+
+ protected void copyScripts(String nodeAddress, final int sshPort) {
+ try {
+ SshHelper.scpTo(nodeAddress, sshPort, CLUSTER_NODE_VM_USER, sshKeyFile, null,
+ "~/", deploySecretsScriptFile.getAbsolutePath(), "0755");
+ SshHelper.scpTo(nodeAddress, sshPort, CLUSTER_NODE_VM_USER, sshKeyFile, null,
+ "~/", deployProviderScriptFile.getAbsolutePath(), "0755");
+ String cmdStr = String.format("sudo mv ~/%s /opt/bin/%s", deploySecretsScriptFile.getName(), deploySecretsScriptFilename);
+ SshHelper.sshExecute(publicIpAddress, sshPort, CLUSTER_NODE_VM_USER, sshKeyFile, null,
+ cmdStr, 10000, 10000, 10 * 60 * 1000);
+ cmdStr = String.format("sudo mv ~/%s /opt/bin/%s", deployProviderScriptFile.getName(), deployProviderScriptFilename);
+ SshHelper.sshExecute(publicIpAddress, sshPort, CLUSTER_NODE_VM_USER, sshKeyFile, null,
+ cmdStr, 10000, 10000, 10 * 60 * 1000);
+ } catch (Exception e) {
+ throw new CloudRuntimeException(e);
+ }
+ }
+
+ protected boolean deployProvider() {
+ Network network = networkDao.findById(kubernetesCluster.getNetworkId());
+ // Since the provider creates IP addresses, don't deploy it unless the underlying network supports it
+ if (network.getGuestType() != GuestType.Isolated) {
+ logMessage(Level.INFO, String.format("Skipping adding the provider as %s is not on an isoalted network",
Review comment:
```suggestion
logMessage(Level.INFO, String.format("Skipping adding the provider as %s is not on an isolated network",
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd merged pull request #5254: kubernetes: Deploy kubernetes-provider when creating a cluster
Posted by GitBox <gi...@apache.org>.
rhtyd merged pull request #5254:
URL: https://github.com/apache/cloudstack/pull/5254
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #5254: kubernetes: Deploy kubernetes-provider when creating a cluster
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #5254:
URL: https://github.com/apache/cloudstack/pull/5254#issuecomment-889819688
@davidjumani a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on pull request #5254: kubernetes: Deploy kubernetes-provider when creating a cluster
Posted by GitBox <gi...@apache.org>.
rhtyd commented on pull request #5254:
URL: https://github.com/apache/cloudstack/pull/5254#issuecomment-895068739
Merging this based on regression tests and lgtms, further review/testing and improvements can be continued at https://github.com/apache/cloudstack/pull/4329
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org